---------------------------------------------------------------------- This is the API documentation for the freezebase library. ---------------------------------------------------------------------- ## MGRS grid 10 km MGRS grid squares to align raster data onto one common grid. ## Raster I/O rasterio helpers that accept local paths and `s3://` URIs via UPath. ## VRT generation Build VRT files without the Python GDAL dependency ## Downloads HTTP downloads with UPath destinations ## S3 S3 `UPath` construction and retry configuration ## Vector data Cached vector datasets and GeoDataFrame helpers ## Utilities Shared helpers used across the package set_loglevel(level: Literal['notset', 'debug', 'info', 'warning', 'error', 'critical']) -> None Configure freezebase's logging levels. Call `set_loglevel("info")` or `set_loglevel("debug")` to get additional debugging information. Parameters ---------- level : {"notset", "debug", "info", "warning", "error", "critical"} The log level of the handler. Notes ----- Copy of `matplotlib.set_loglevel`. ---------------------------------------------------------------------- This is the User Guide documentation for the package. ---------------------------------------------------------------------- ### The MGRS grid ```{python} #| echo: false #| warning: false import warnings warnings.filterwarnings("ignore") ``` `MGRSGrid` provides reproducible 10 km raster targets for multi-source Earth Observation data. It is built on [`odc.geo`](https://odc-geo.readthedocs.io/): each square is a `GeoBox` carrying the CRS, affine transform, and array shape needed to place pixels on the ground. ## Why fix the grid first? A Sentinel-1 scene, a Sentinel-2 composite, a DEM, and a rasterised outline usually arrive in different projections, resolutions, and tiling schemes. If each new layer is aligned to whichever raster happens to be available first, the reference grid becomes an accidental property of the processing order. `MGRSGrid` instead supplies a dataset-independent destination. For a given MGRS code and resolution, every pipeline derives the same CRS, transform, and shape. Sources resampled onto that destination are pixel-aligned by construction and can be stacked without another alignment decision. ## Why MGRS, and why 10 km? The grid uses the [Military Grid Reference System](https://en.wikipedia.org/wiki/Military_Grid_Reference_System) rather than inventing another spatial identifier: **It is established and metric.** Each square lies on the metre-based lattice of a UTM zone. The grid is axis-aligned in its native projection, so a 10 m pixel represents a fixed 10 m grid spacing rather than a varying fraction of a degree. **Sentinel-2 uses the same reference system.** A Sentinel-2 tile such as `T32TMS` corresponds to a 100 km MGRS square; a freezebase code such as `32TMS35` identifies one of its 10 km subdivisions. This makes the parent Sentinel-2 tile directly identifiable. Compatible 10 m data can be retiled without introducing an additional grid shift. **Ten kilometres is a practical unit of work.** At 10 m resolution, a 100 km square contains 10,000 × 10,000 pixels, while a 10 km square contains 1,000 × 1,000. The smaller cells make it practical to download, cache, and process only the squares touched by scattered areas of interest. **Cells do not overlap within one UTM grid.** Unlike Sentinel-2 product tiles, which include wasteful overlap around their 100 km reference squares[^1], freezebase cells step by exactly 10 km. Adjacent UTM zones are separate grids and can overlap; that limitation is described below. [^1]: Bauer-Marschallinger, B., & Falkner, K. (2023). Wasting petabytes: A survey of the Sentinel-2 UTM tiling grid and its spatial overhead. *ISPRS Journal of Photogrammetry and Remote Sensing*, 202, 682–690. ## Select the squares for an AOI Pass an area of interest in WGS84. The example deliberately crosses 6° E, the boundary between UTM zones 31 and 32: ```{python} from shapely import box from freezebase.mgrs import MGRSGrid aoi = box(5.8, 46.0, 6.5, 46.4) # Geneva region; bare Shapely means WGS84 here grid = MGRSGrid(aoi) len(grid) ``` `to_geodataframe()` returns one row per selected square in WGS84: ```{python} gdf = grid.to_geodataframe() gdf.head() ``` The metadata includes the stable MGRS code, UTM zone and hemisphere, native EPSG code, and south-west corner in UTM metres. This AOI needs two local metric grids: ```{python} gdf.groupby(["zone", "epsg"]).size() ``` ::: {.callout-note} ## How AOI selection works - A bare Shapely geometry is assumed to use WGS84 coordinates. A `GeoSeries` or `GeoDataFrame` must declare WGS84 explicitly. - Every square that intersects any input geometry is returned whole. The grid therefore covers slightly more than the exact AOI. - A square that only touches an AOI boundary still counts as an intersection. - `resolution` changes the pixels inside each selected square, not which squares are selected. ::: ## A square is an odc.geo GeoBox Indexing or iterating over the grid yields `MGRSGeoBox` objects, a small subclass of [`odc.geo.geobox.GeoBox`](https://odc-geo.readthedocs.io/en/latest/intro-geobox.html). ```{python} from odc.geo.geobox import GeoBox square = grid[0] display(square) print(f"Is GeoBox: {isinstance(square, GeoBox)}") ``` It combines image width, height, CRS, and an affine transformation to fully define a geo-registered pixel plane: ```{python} from odc.geo.geobox import GeoBox square = grid[0] print(f"mgrs_code: {square.mgrs_code}") print(f"crs: {square.crs}") print(f"shape: {square.shape}") print(f"resolution: {square.resolution}") print(f"transform:\n{square.transform}") print(f"is GeoBox: {isinstance(square, GeoBox)}") ``` It also brings along `odc.geo` operations and properties such as native and geographic extents. Operations that derive a different box—such as padding, slicing, or reprojection—return a plain `GeoBox`. The derived box no longer has the exact footprint named by the MGRS code, so it intentionally does not retain `mgrs_code`. ## Reconstruct a square from its code When a code is already stored in a filename, database, or job description, the AOI does not need to be processed again: ```{python} from freezebase.mgrs import MGRSGeoBox, mgrs_to_crs square = MGRSGeoBox.from_mgrs("32TMS35", resolution=10.0) print(square) print(mgrs_to_crs(square.mgrs_code)) ``` This is why the grid is a specification rather than a dataset: the code and resolution are enough to reconstruct its georeferencing. The resolution must be positive and divide 10 km evenly. Resolutions such as 10 m, 20 m, and 100 m therefore produce whole-pixel, nested grids with an integer scale factor: ```{python} for resolution in (10.0, 20.0, 100.0): cell = MGRSGeoBox.from_mgrs("32TMS35", resolution=resolution) print(f"{resolution:>5.0f} m -> {cell.shape}") ``` ## Geographic limits and zone boundaries freezebase implements the UTM portion of MGRS: 80° S to 84° N. It does not implement the polar UPS grids, so it does not cover every location on Earth. Each UTM zone is projected independently. Cells within one zone share a non-overlapping 10 km lattice, but full cells retained along a zone edge can extend into the neighbouring zone. An AOI crossing that edge can consequently select cells from both zones that cover some of the same ground. This preserves coverage and keeps every cell square and axis-aligned in its own UTM CRS. The structure is visible when the Geneva example is drawn in one display CRS: ```{python} #| code-fold: true #| fig-cap: "The 10 km grids of UTM zones 31 and 32 over the Geneva AOI. Each is axis-aligned in its own projection, so they appear rotated and overlap near the boundary when displayed together." #| warning: false import contextily as cx import geopandas as gpd import matplotlib.pyplot as plt from matplotlib.patches import Patch import pyproj web_mercator = pyproj.CRS(3857) squares = gdf.to_crs(web_mercator) aoi_gdf = gpd.GeoDataFrame(geometry=[aoi], crs=4326).to_crs(web_mercator) zone_colors = {31: "tab:blue", 32: "tab:red"} fig, ax = plt.subplots(figsize=(9, 6)) for zone, color in zone_colors.items(): squares[squares.zone == zone].plot( ax=ax, facecolor=color, alpha=0.25, edgecolor=color, linewidth=1.0 ) aoi_gdf.boundary.plot(ax=ax, color="black", linewidth=2.0, linestyle="--") cx.add_basemap(ax, source=cx.providers.CartoDB.Positron, crs=web_mercator, attribution_size=6) ax.legend( handles=[ *( Patch(facecolor=color, alpha=0.25, edgecolor=color, label=f"UTM zone {zone}") for zone, color in zone_colors.items() ), Patch(facecolor="none", edgecolor="black", linestyle="--", label="AOI"), ], loc="lower right", ) ax.set_axis_off() ``` ## What's next? The [Rasters on S3](02-rasters-on-s3.qmd) guide shows how the same path abstraction can read a remote raster and resample it onto one of these boxes. ### Rasters on S3 ```{python} #| echo: false #| warning: false import warnings warnings.filterwarnings("ignore") ``` freezebase connects two libraries that normally see object storage independently: [Universal Pathlib](https://universal-pathlib.readthedocs.io/) provides pathlib-style S3 paths through `UPath`, while [Rasterio](https://rasterio.readthedocs.io/) performs the raster I/O. freezebase enables both layers to use the same per-path authentication and endpoint configuration. ## The two-client problem Rasterio can open an `s3://` URI using GDAL's `/vsis3/` driver, and s3fs can list the same bucket through a `UPath`. Both support credentials, but they do not ordinarily share a per-path configuration. This becomes especially awkward when one process uses multiple accounts or S3-compatible services such as MinIO or Ceph. `make_s3_upath()` attaches storage options to a `UPath`. Filesystem operations read them through s3fs, while `rasterio_open()` translates them into a scoped Rasterio environment for the duration of its `with` block: ```python from freezebase.raster import rasterio_open from freezebase.s3 import make_s3_upath # The named profile travels with this path and all paths derived from it. path = make_s3_upath("s3://my-bucket/scene.tif", profile="research") with rasterio_open(path) as src: data = src.read(1) ``` A local `str` or `Path` works through the same raster helper without any S3 configuration. The wrapper deliberately supports local and S3 paths; other `UPath` protocols are not accepted by Rasterio helpers. ::: {.callout-note} ## Install the S3 extra The S3 integration requires s3fs, fsspec, and boto3: ```bash pip install "freezebase[s3]" ``` These dependencies are imported lazily, so local raster operations work with the base installation. ::: ## Authentication patterns Credentials and endpoint configuration travel with each S3 path. Paths can select different named AWS profiles, including profiles used with custom S3-compatible endpoints: ```python from freezebase.s3 import make_s3_upath aws_path = make_s3_upath("s3://aws-bucket/data", profile="research") ceph_path = make_s3_upath( "s3://ceph-bucket/data", profile="ceph-research", endpoint_url="https://objects.example.org", ) ``` The profile name—not its credentials—is stored on the path and inherited by child paths. s3fs and Rasterio independently load that same profile when they access the object store. Selecting a profile explicitly also makes boto skip its environment-variable credential provider. A profiled path is therefore never signed by whatever `AWS_ACCESS_KEY_ID` or `AWS_SECRET_ACCESS_KEY` happens to be set in the shell. Besides making multiple stores usable in one process, this prevents ambient credentials intended for one service from being sent to another. Resolving a profile means reading the AWS configuration files, so the resulting Rasterio session is cached per configuration and reused by every subsequent read and write. Refreshable credentials—STS or SSO profiles—still rotate on their own. Static credentials are held for the lifetime of the process; if the credentials file is rewritten underneath a long-running process, call `clear_aws_session_cache()` to force the next access to resolve it again. Explicit credentials and temporary session tokens are also supported. This is useful when credentials come from an application-specific secret store rather than an AWS configuration file: ```python custom = make_s3_upath( "s3://imagery/scene.tif", key=..., secret=..., token=..., # optional STS/session token endpoint_url="https://objects.example.org", region="eu-central-1", ) ``` When neither a profile nor explicit credentials are attached, both layers fall back to boto's standard credential chain. This supports environment variables and workload roles, but the selected identity is no longer explicit on the path. Public buckets require explicit anonymous access: ```python public = make_s3_upath("s3://public-bucket/scene.tif", anon=True) ``` Profiles, explicit credentials, and anonymous access are mutually exclusive. The helper also configures retries for both the s3fs/botocore and GDAL paths, plus checksum defaults that avoid trailers rejected by some S3-compatible gateways. ::: {.callout-warning} ## Treat configured paths as sensitive Explicit `key`, `secret`, and `token` values are stored in the `UPath`'s `storage_options`. Do not log, publish, or serialize those options. A named profile avoids storing the credential values on the path. ::: ## Read a public raster The [Copernicus DEM GLO-30](https://registry.opendata.aws/copernicus-dem/) is a public collection of Cloud-Optimized GeoTIFFs. The same configured path can be inspected through Universal Pathlib and opened through Rasterio: ```{python} from freezebase.raster import rasterio_open from freezebase.s3 import make_s3_upath tile = "Copernicus_DSM_COG_10_N46_00_E008_00_DEM" dem_path = make_s3_upath( f"s3://copernicus-dem-30m/{tile}/{tile}.tif", anon=True, region="eu-central-1", ) # s3fs print(f"exists: {dem_path.exists()}") # GDAL with rasterio_open(dem_path) as src: print(f"driver: {src.driver}") print(f"crs: {src.crs}") print(f"shape: {src.shape}") print(f"bounds: {tuple(round(value) for value in src.bounds)}") print(f"overviews: {src.overviews(1)}") ``` Methods such as `.exists()`, `.stat()`, `.iterdir()`, and `.glob()` use the same path configuration as the raster read. Avoid wrapping a potentially large bucket listing in `list()` merely to inspect its first few entries; use `itertools.islice()` when only a sample is needed. ## Write and copy rasters `write_cog()` writes an array as a Cloud-Optimized GeoTIFF to either a local path or an S3 `UPath`. A GeoBox from the MGRS guide supplies the spatial part of the Rasterio profile: ```python import numpy as np from freezebase.mgrs import MGRSGeoBox from freezebase.raster import write_cog square = MGRSGeoBox.from_mgrs("32TMS35", resolution=10.0) data = np.zeros(square.shape.yx, dtype=np.float32) profile = { "dtype": "float32", "count": 1, "width": square.shape.x, "height": square.shape.y, "crs": square.crs, "transform": square.transform, "nodata": np.nan, } destination = make_s3_upath(f"s3://my-bucket/dem/{square.mgrs_code}.tif") write_cog(data, destination, profile, band_names=["elevation"]) ``` `rewrite_tiff()` handles existing files. It can recompress or retile them and copy between local storage and S3. Source and destination credentials are resolved independently, so two S3 paths may refer to different stores: ```python from freezebase.raster import COG_PROFILE, rewrite_tiff # local to S3 rewrite_tiff("scene.tif", destination, profile=COG_PROFILE) # S3 to S3 rewrite_tiff(source_on_store_a, destination_on_store_b) rewrite_tiff("scene.tif", "scene.cog.tif", move=True) ``` Both functions also carry metadata into the file. `tags` sets dataset-level tags, `band_tags` sets per-band tags (one mapping per band), and `units` sets each band's unit string. `rewrite_tiff()` merges them onto whatever the source already carried, so a rewrite can add a key without discarding the rest: ```python write_cog( data, destination, profile, band_names=["elevation"], tags={"PRODUCT_TYPE": "DEM", "VERTICAL_DATUM": "EGM2008"}, band_tags=[{"SOURCE": "Copernicus GLO-30"}], units=["m"], ) ``` ## Putting S3 and MGRS together The public DEM is in geographic coordinates and does not share the MGRS pixel grid. An `MGRSGeoBox` provides the complete destination geometry for `rasterio.warp.reproject`: ```{python} import numpy as np from rasterio import band from rasterio.warp import Resampling, reproject from freezebase.mgrs import MGRSGeoBox square = MGRSGeoBox.from_mgrs("32TMS35", resolution=10.0) dem = np.full(square.shape.yx, np.nan, dtype=np.float32) with rasterio_open(dem_path) as src: reproject( source=band(src, 1), destination=dem, src_transform=src.transform, src_crs=src.crs, dst_transform=square.transform, dst_crs=square.crs, dst_nodata=np.nan, resampling=Resampling.cubic, ) print(f"{square.mgrs_code}: {dem.shape} pixels in {square.crs}") print(f"elevation: {np.nanmin(dem):.0f} m to {np.nanmax(dem):.0f} m") ``` Passing `rasterio.band(src, 1)` lets GDAL read the source as needed instead of first loading the complete source tile into a NumPy array. The result has the same CRS, transform, and shape as every other 10 m layer resampled onto `32TMS35`. ```{python} #| code-fold: true #| fig-cap: "Copernicus DEM GLO-30 resampled onto MGRS square 32TMS35 in the Bernese Alps." #| warning: false import matplotlib.pyplot as plt left, bottom, right, top = square.boundingbox fig, ax = plt.subplots(figsize=(6.5, 5.5)) image = ax.imshow(dem, cmap="terrain", extent=(left, right, bottom, top)) fig.colorbar(image, ax=ax, label="elevation (m)", shrink=0.85) ax.set(title=f"MGRS {square.mgrs_code}", xlabel="easting (m)", ylabel="northing (m)") ax.ticklabel_format(style="plain") ``` For complete signatures and the additional mosaic and profile helpers, see the [API reference](../reference/index.qmd). ### Contributing freezebase is a small core library maintained on a best-effort basis. Bug reports and questions belong on the [issue tracker](https://github.com/lqgentner/freezebase/issues). ## Setting up The project uses [uv](https://docs.astral.sh/uv/). One command creates the venv and installs all (development) dependencies: ```bash git clone https://github.com/lqgentner/freezebase.git cd freezebase uv sync --all-extras ``` `--all-extras` matters: the `s3` extra is optional at runtime but the test suite imports `freezebase.s3`, so a bare `uv sync` leaves those tests skipped. ## The checks These are the same four commands CI runs, so a clean run locally means a clean run there: ```bash uv run pytest uv run ruff format && uv run ruff check --fix uv run mypy src/freezebase tests ``` A few things worth knowing about each. **Tests.** The suite is fast (a few seconds) and offline. Tests marked `integration` need a live S3-compatible service and are skipped unless the `FREEZEBASE_TEST_S3_*` environment variables are set — see [S3 integration tests](#s3-integration-tests) below. **Lint.** `ruff` runs its expanded default rule set plus project-specific checks and numpydoc docstring conventions, all configured in `pyproject.toml`. It also formats Python code blocks in Markdown and Quarto files. Docstrings on public functions need `Parameters`, `Returns`, and `Raises` sections; the linter enforces it. **Types.** The package ships a `py.typed` marker, so its annotations are part of the public contract. `mypy` must pass on `src/freezebase` and `tests` before a change lands. ## S3 integration tests The credentialed S3 read/write paths cannot be exercised by unit tests — they need a real object store. CI runs them against MinIO in a container, and you can do the same: ```bash docker run -d --name minio -p 9000:9000 \ -e MINIO_ROOT_USER=testkey -e MINIO_ROOT_PASSWORD=testsecret \ minio/minio server /data FREEZEBASE_TEST_S3_ENDPOINT=http://localhost:9000 \ FREEZEBASE_TEST_S3_KEY=testkey \ FREEZEBASE_TEST_S3_SECRET=testsecret \ uv run pytest -q -m integration ``` Run these whenever you touch `freezebase.s3` or the S3 branches of `freezebase.raster`. The unit tests mock the object store, so they will not catch a broken credential or endpoint path. ## Dependency lower bounds Every dependency carries a lower bound chosen as the oldest release that actually installs and works on Python 3.12. CI has a dedicated job that resolves with `--resolution lowest-direct` and runs the suite against those pins: ```bash uv sync --all-extras --resolution lowest-direct --python 3.12 uv run --no-sync pytest -q ``` If you use a newer API, raise the corresponding bound in `pyproject.toml` and note why in the comment above the dependency list. `--no-sync` matters here: without it, `uv run` re-resolves the environment back to the latest versions before running. ## Building the docs The documentation is built with [Great Docs](https://posit-dev.github.io/great-docs/), which renders through [Quarto](https://quarto.org/docs/get-started/). Quarto is a separate install — see [its getting-started page](https://quarto.org/docs/get-started/) — and is not managed by uv. ```bash uv sync --group docs uv run great-docs build # writes great-docs/_site/ uv run great-docs preview # serves great-docs/_site/ on port 3000 ``` ::: {.callout-important} ## `preview` does not rebuild `great-docs preview` is a static file server over `great-docs/_site/`. It only builds when that directory is missing, so it will happily serve a stale site after you edit a page or `great-docs.yml`. Use `great-docs build` to refresh, or `great-docs build --watch` in a second terminal to rebuild on change. ::: `great-docs.yml` is committed; the `great-docs/` build directory is ephemeral and gitignored. ### The freeze cache The user guide executes its code at build time — it reads a public S3 bucket and fetches basemap tiles. Those outputs are cached in the tracked `_freeze/` directory, so a page is only re-executed when its own `.qmd` changes. Editing the README or a docstring rebuilds without touching the network, which also keeps CI builds off the tile server. That caching is keyed on page source, not on library behaviour. If you change something that alters a page's *output* without changing the page itself, refresh it explicitly: ```bash uv run great-docs freeze --info # what is cached and stale uv run great-docs freeze user_guide/01-mgrs-grid.qmd # re-execute one page git add _freeze/ # commit the new outputs ``` ## Releasing Versions come from git tags via `hatch-vcs`, so there is no version string to edit. Record changes in [`CHANGELOG.md`](https://github.com/lqgentner/freezebase/blob/main/CHANGELOG.md) under `[Unreleased]` as you go, following [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The project is pre-1.0, so minor releases may contain breaking changes — but they must be labelled. Mark anything that changes existing behaviour with **Breaking:** and show the before/after, so the entry is enough on its own to fix a downstream call site.