Skip to content

Speed up import and load hvplot lazily#1541

Open
Sevans711 wants to merge 10 commits into
UXARRAY:mainfrom
Sevans711:speedup-import
Open

Speed up import and load hvplot lazily#1541
Sevans711 wants to merge 10 commits into
UXARRAY:mainfrom
Sevans711:speedup-import

Conversation

@Sevans711

@Sevans711 Sevans711 commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Closes #1539

Overview

Speeds up import uxarray from ~1.7 seconds to ~0.7 seconds, by making hvplot.pandas and hvplot.xarray imports lazy, i.e. only import them when actually initializing a PlotAccessor object for the first time.

PR Checklist

General

  • An issue is linked created and linked
  • Add appropriate labels
  • Filled out Overview and Expected Usage (if applicable) sections

Testing

  • Adequate tests are created if there is new functionality
  • Tests cover all possible logical paths in your function
  • Tests are not too basic (such as simply calling a function and nothing else)

Sevans711 and others added 3 commits June 30, 2026 11:34
Small change which reduces time for "import uxarray" on my machine from 1.7 seconds to 0.7 seconds. Also avoids the strange gray bars in Jupyter until actually making a plot for the first time, instead of immediately when importing uxarray. (The bars apparently came from "import hvplot.pandas" and "import hvplot.xarray"). See also: UXARRAY#1260

This also means hvplot import is lazy, which means we should be able to move hvplot to be an optional dependency instead of a requirement. See also: UXARRAY#1224
Comment thread test/test_dependencies.py Outdated
Co-authored-by: Rajeev Jain <rajeeja@gmail.com>
@Sevans711

Copy link
Copy Markdown
Collaborator Author

We might want to delay merging; I just discovered that import hvplot.pandas and import hvplot.xarray always change the backend to bokeh. Thus, with these changes, lazily importing hvplot means the first plot you make from uxarray will be forced into having bokeh backend, regardless of the backend you tried to set originally. I am creating a bug report on hvplot repository now about this.

@Sevans711

Copy link
Copy Markdown
Collaborator Author

I'm very confused about why my latest changes would cause those crashes… I only added an additional import hvplot and an additional call to hvplot.extension. But, the crashes seem to be like AttributeError: 'UxDataArray' object has no attribute 'plot'. Did you mean: 'hvplot'?

Any guesses on how my changes might have led to this? A quick attempt to reproduce it didn't lead me anywhere either, e.g.:

uxds = ux.tutorial.open_dataset("outCSne30-vortex")
hasattr(uxds, 'plot')

gives True. I can chase this down more if I should prioritize it, but wanted to check in first in case you have a guess already. Seems like it might be related to the __getattr__ architecture?

hvplot has no Store attribute, so the previous guard always skipped the
save/restore and the active backend was still forced to bokeh. Read/write the
backend on holoviews.Store, and only restore backends that were actually loaded
(Store.registry) to avoid breaking rendering on a fresh import.
@Sevans711 Sevans711 added the improvement Improvements on existing features or infrastructure label Jul 8, 2026
@rajeeja rajeeja moved this to 👀 In review in UXarray Development Jul 8, 2026
@Sevans711 Sevans711 requested a review from rajeeja July 8, 2026 21:53
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have improved:

Change Before [7c9b487] After [3c16fd7] Ratio Benchmark (Parameter)
- 439M 389M 0.89 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
- 471M 419M 0.89 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
- 629M 391M 0.62 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
- 751M 390M 0.52 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
- 4.83±0.02s 1.03±0s 0.21 import.Imports.timeraw_import_uxarray
- 454M 404M 0.89 mpas_ocean.Gradient.peakmem_gradient('120km')
- 550M 384M 0.7 mpas_ocean.Gradient.peakmem_gradient('480km')
- 405M 352M 0.87 mpas_ocean.Integrate.peakmem_integrate('120km')
- 382M 332M 0.87 mpas_ocean.Integrate.peakmem_integrate('480km')
- 378M 326M 0.86 quad_hexagon.QuadHexagon.peakmem_open_dataset
- 377M 325M 0.86 quad_hexagon.QuadHexagon.peakmem_open_grid

Benchmarks that have stayed the same:

Change Before [7c9b487] After [3c16fd7] Ratio Benchmark (Parameter)
10.8±0.1μs 10.6±0.2μs 0.98 bench_connectivity.Connectivity.time_edge_face('120km')
failed failed n/a bench_connectivity.Connectivity.time_edge_face('15km')
failed failed n/a bench_connectivity.Connectivity.time_edge_face('3.75km')
failed failed n/a bench_connectivity.Connectivity.time_edge_face('30km')
10.9±0.07μs 11.0±0.09μs 1.01 bench_connectivity.Connectivity.time_edge_face('480km')
failed failed n/a bench_connectivity.Connectivity.time_edge_face('7.5km')
10.7±0.08μs 10.8±0.07μs 1.01 bench_connectivity.Connectivity.time_edge_node('120km')
failed failed n/a bench_connectivity.Connectivity.time_edge_node('15km')
failed failed n/a bench_connectivity.Connectivity.time_edge_node('3.75km')
failed failed n/a bench_connectivity.Connectivity.time_edge_node('30km')
11.1±0.09μs 10.9±0.1μs 0.98 bench_connectivity.Connectivity.time_edge_node('480km')
failed failed n/a bench_connectivity.Connectivity.time_edge_node('7.5km')
10.8±0.07μs 10.5±0.2μs 0.97 bench_connectivity.Connectivity.time_face_edge('120km')
failed failed n/a bench_connectivity.Connectivity.time_face_edge('15km')
failed failed n/a bench_connectivity.Connectivity.time_face_edge('3.75km')
failed failed n/a bench_connectivity.Connectivity.time_face_edge('30km')
10.9±0.06μs 11.1±0.06μs 1.02 bench_connectivity.Connectivity.time_face_edge('480km')
failed failed n/a bench_connectivity.Connectivity.time_face_edge('7.5km')
10.6±0.07μs 10.5±0.2μs 0.98 bench_connectivity.Connectivity.time_face_face('120km')
failed failed n/a bench_connectivity.Connectivity.time_face_face('15km')
failed failed n/a bench_connectivity.Connectivity.time_face_face('3.75km')
failed failed n/a bench_connectivity.Connectivity.time_face_face('30km')
11.0±0.1μs 10.8±0.1μs 0.98 bench_connectivity.Connectivity.time_face_face('480km')
failed failed n/a bench_connectivity.Connectivity.time_face_face('7.5km')
21.4±0.3μs 21.3±0.2μs 1.00 bench_connectivity.Connectivity.time_face_node('120km')
failed failed n/a bench_connectivity.Connectivity.time_face_node('15km')
failed failed n/a bench_connectivity.Connectivity.time_face_node('3.75km')
failed failed n/a bench_connectivity.Connectivity.time_face_node('30km')
21.7±0.1μs 21.9±0.3μs 1.01 bench_connectivity.Connectivity.time_face_node('480km')
failed failed n/a bench_connectivity.Connectivity.time_face_node('7.5km')
10.8±0.1μs 10.6±0.6μs 0.99 bench_connectivity.Connectivity.time_node_edge('120km')
failed failed n/a bench_connectivity.Connectivity.time_node_edge('15km')
failed failed n/a bench_connectivity.Connectivity.time_node_edge('3.75km')
failed failed n/a bench_connectivity.Connectivity.time_node_edge('30km')
10.9±0.3μs 11.0±0.1μs 1.01 bench_connectivity.Connectivity.time_node_edge('480km')
failed failed n/a bench_connectivity.Connectivity.time_node_edge('7.5km')
11.1±0.2μs 10.8±0.3μs 0.97 bench_connectivity.Connectivity.time_node_face('120km')
failed failed n/a bench_connectivity.Connectivity.time_node_face('15km')
failed failed n/a bench_connectivity.Connectivity.time_node_face('3.75km')
failed failed n/a bench_connectivity.Connectivity.time_node_face('30km')
11.0±0.2μs 11.1±0.07μs 1.00 bench_connectivity.Connectivity.time_node_face('480km')
failed failed n/a bench_connectivity.Connectivity.time_node_face('7.5km')
14.4±0.04ms 14.6±0.1ms 1.01 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
3.67±0.02ms 3.69±0.09ms 1.01 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
19.0±0.09ms 19.6±0.1ms 1.03 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
2.18±0.01ms 2.19±0.03ms 1.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
939±10ns 960±20ns 1.02 mpas_ocean.CheckNorm.time_check_norm('120km')
898±10ns 897±10ns 1.00 mpas_ocean.CheckNorm.time_check_norm('480km')
823±4ms 826±6ms 1.00 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('120km')
53.6±0.8ms 53.1±0.3ms 0.99 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('480km')
666±6μs 659±10μs 0.99 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km')
593±20μs 598±9μs 1.01 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km')
5.39±0.06ms 5.43±0.03ms 1.01 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('120km')
3.91±0.02ms 3.94±0.04ms 1.01 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('480km')
3.41±0.01s 3.45±0.05s 1.01 mpas_ocean.ConstructFaceLatLon.time_welzl('120km')
222±0.7ms 225±3ms 1.01 mpas_ocean.ConstructFaceLatLon.time_welzl('480km')
18.2±0.03ms 18.1±0.03ms 1.00 mpas_ocean.ConstructTreeStructures.time_ball_tree('120km')
1.03±0.01ms 1.01±0.01ms 0.99 mpas_ocean.ConstructTreeStructures.time_ball_tree('480km')
10.6±0.02ms 10.6±0.01ms 1.00 mpas_ocean.ConstructTreeStructures.time_kd_tree('120km')
721±5μs 698±10μs 0.97 mpas_ocean.ConstructTreeStructures.time_kd_tree('480km')
701±2ms 703±2ms 1.00 mpas_ocean.CrossSections.time_const_lat('120km', 1)
354±1ms 351±2ms 0.99 mpas_ocean.CrossSections.time_const_lat('120km', 2)
184±1ms 183±1ms 1.00 mpas_ocean.CrossSections.time_const_lat('120km', 4)
546±1ms 546±2ms 1.00 mpas_ocean.CrossSections.time_const_lat('480km', 1)
277±0.9ms 275±0.9ms 0.99 mpas_ocean.CrossSections.time_const_lat('480km', 2)
142±0.8ms 141±1ms 0.99 mpas_ocean.CrossSections.time_const_lat('480km', 4)
24.7±0.2ms 24.6±0.2ms 0.99 mpas_ocean.DualMesh.time_dual_mesh_construction('120km')
3.33±0.1ms 3.28±0.09ms 0.99 mpas_ocean.DualMesh.time_dual_mesh_construction('480km')
945±9ms 960±20ms 1.02 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', False)
51.9±0.4ms 51.6±0.7ms 0.99 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', True)
84.0±0.4ms 83.2±1ms 0.99 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', False)
5.60±0.03ms 5.67±0.07ms 1.01 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', True)
174±0.6ms 175±0.3ms 1.00 mpas_ocean.Gradient.time_gradient('120km')
12.4±0.08ms 12.4±0.1ms 1.00 mpas_ocean.Gradient.time_gradient('480km')
224±0.9μs 223±1μs 0.99 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('120km')
132±0.7μs 132±1μs 1.00 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('480km')
214±1μs 214±2μs 1.00 mpas_ocean.Integrate.time_integrate('120km')
197±2μs 198±0.7μs 1.01 mpas_ocean.Integrate.time_integrate('480km')
130±1ms 130±0.4ms 1.00 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'exclude')
131±1ms 128±0.6ms 0.98 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'include')
131±1ms 131±1ms 1.00 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'split')
10.6±0.03ms 10.2±0.1ms 0.96 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'exclude')
10.6±0.1ms 10.2±0.06ms 0.97 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'include')
10.6±0.08ms 10.3±0.06ms 0.97 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'split')
348±0.9μs 352±3μs 1.01 mpas_ocean.PointInPolygon.time_face_search_lonlat('120km')
354±2μs 352±1μs 0.99 mpas_ocean.PointInPolygon.time_face_search_lonlat('480km')
330±1μs 334±3μs 1.01 mpas_ocean.PointInPolygon.time_face_search_xyz('120km')
333±1μs 336±1μs 1.01 mpas_ocean.PointInPolygon.time_face_search_xyz('480km')
241±0.2ms 246±2ms 1.02 mpas_ocean.RemapDownsample.time_bilinear_remapping
299±4ms 298±2ms 1.00 mpas_ocean.RemapDownsample.time_inverse_distance_weighted_remapping
1.44±0.01s 1.43±0.01s 0.99 mpas_ocean.RemapUpsample.time_bilinear_remapping
37.4±0.3ms 36.7±0.5ms 0.98 mpas_ocean.RemapUpsample.time_inverse_distance_weighted_remapping
9.49±0.06ms 9.41±0.1ms 0.99 mpas_ocean.RemapUpsample.time_nearest_neighbor_remapping
28.4±0.3ms 29.2±0.3ms 1.03 mpas_ocean.ZonalAverage.time_zonal_average('120km')
6.63±0.05ms 6.51±0.03ms 0.98 mpas_ocean.ZonalAverage.time_zonal_average('480km')
7.05±0.2ms 6.94±0.08ms 0.99 quad_hexagon.QuadHexagon.time_open_dataset
5.92±0.1ms 5.84±0.07ms 0.99 quad_hexagon.QuadHexagon.time_open_grid

Benchmarks that have got worse:

Change Before [7c9b487] After [3c16fd7] Ratio Benchmark (Parameter)
+ 4.26±0.02ms 4.74±0.03ms 1.11 mpas_ocean.RemapDownsample.time_nearest_neighbor_remapping

@Sevans711 Sevans711 requested a review from cmdupuis3 July 8, 2026 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Improvements on existing features or infrastructure run-benchmark Run ASV benchmark workflow

Projects

Status: 👀 In review

Development

Successfully merging this pull request may close these issues.

speed up import and load hvplot lazily

3 participants