freezebase connects two libraries that normally see object storage independently: Universal Pathlib provides pathlib-style S3 paths through UPath, while Rasterio performs the raster I/O. freezebase enables both layers to use the same per-path authentication and endpoint configuration.
The two-client problem
Rasterio can open an s3:// URI using GDAL’s /vsis3/ driver, and s3fs can list the same bucket through a UPath. Both support credentials, but they do not ordinarily share a per-path configuration. This becomes especially awkward when one process uses multiple accounts or S3-compatible services such as MinIO or Ceph.
make_s3_upath() attaches storage options to a UPath. Filesystem operations read them through s3fs, while rasterio_open() translates them into a scoped Rasterio environment for the duration of its with block:
from freezebase.raster import rasterio_open
from freezebase.s3 import make_s3_upath
# The named profile travels with this path and all paths derived from it.
path = make_s3_upath("s3://my-bucket/scene.tif", profile="research")
with rasterio_open(path) as src:
data = src.read(1)
A local str or Path works through the same raster helper without any S3 configuration. The wrapper deliberately supports local and S3 paths; other UPath protocols are not accepted by Rasterio helpers.
The S3 integration requires s3fs, fsspec, and boto3:
pip install "freezebase[s3]"
These dependencies are imported lazily, so local raster operations work with the base installation.
Authentication patterns
Credentials and endpoint configuration travel with each S3 path. Paths can select different named AWS profiles, including profiles used with custom S3-compatible endpoints:
from freezebase.s3 import make_s3_upath
aws_path = make_s3_upath("s3://aws-bucket/data", profile="research")
ceph_path = make_s3_upath(
"s3://ceph-bucket/data",
profile="ceph-research",
endpoint_url="https://objects.example.org",
)
The profile name—not its credentials—is stored on the path and inherited by child paths. s3fs and Rasterio independently load that same profile when they access the object store.
Selecting a profile explicitly also makes boto skip its environment-variable credential provider. A profiled path is therefore never signed by whatever AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY happens to be set in the shell. Besides making multiple stores usable in one process, this prevents ambient credentials intended for one service from being sent to another.
Resolving a profile means reading the AWS configuration files, so the resulting Rasterio session is cached per configuration and reused by every subsequent read and write. Refreshable credentials—STS or SSO profiles—still rotate on their own. Static credentials are held for the lifetime of the process; if the credentials file is rewritten underneath a long-running process, call clear_aws_session_cache() to force the next access to resolve it again.
Explicit credentials and temporary session tokens are also supported. This is useful when credentials come from an application-specific secret store rather than an AWS configuration file:
custom = make_s3_upath(
"s3://imagery/scene.tif",
key=...,
secret=...,
token=..., # optional STS/session token
endpoint_url="https://objects.example.org",
region="eu-central-1",
)
When neither a profile nor explicit credentials are attached, both layers fall back to boto’s standard credential chain. This supports environment variables and workload roles, but the selected identity is no longer explicit on the path.
Public buckets require explicit anonymous access:
public = make_s3_upath("s3://public-bucket/scene.tif", anon=True)
Profiles, explicit credentials, and anonymous access are mutually exclusive. The helper also configures retries for both the s3fs/botocore and GDAL paths, plus checksum defaults that avoid trailers rejected by some S3-compatible gateways.
Explicit key, secret, and token values are stored in the UPath’s storage_options. Do not log, publish, or serialize those options. A named profile avoids storing the credential values on the path.
Read a public raster
The Copernicus DEM GLO-30 is a public collection of Cloud-Optimized GeoTIFFs. The same configured path can be inspected through Universal Pathlib and opened through Rasterio:
from freezebase.raster import rasterio_open
from freezebase.s3 import make_s3_upath
tile = "Copernicus_DSM_COG_10_N46_00_E008_00_DEM"
dem_path = make_s3_upath(
f"s3://copernicus-dem-30m/{tile}/{tile}.tif",
anon=True,
region="eu-central-1",
)
# s3fs
print(f"exists: {dem_path.exists()}")
# GDAL
with rasterio_open(dem_path) as src:
print(f"driver: {src.driver}")
print(f"crs: {src.crs}")
print(f"shape: {src.shape}")
print(f"bounds: {tuple(round(value) for value in src.bounds)}")
print(f"overviews: {src.overviews(1)}")
exists: True
driver: GTiff
crs: EPSG:4326
shape: (3600, 3600)
bounds: (8, 46, 9, 47)
overviews: [2, 4, 8]
Methods such as .exists(), .stat(), .iterdir(), and .glob() use the same path configuration as the raster read. Avoid wrapping a potentially large bucket listing in list() merely to inspect its first few entries; use itertools.islice() when only a sample is needed.
Write and copy rasters
write_cog() writes an array as a Cloud-Optimized GeoTIFF to either a local path or an S3 UPath. A GeoBox from the MGRS guide supplies the spatial part of the Rasterio profile:
import numpy as np
from freezebase.mgrs import MGRSGeoBox
from freezebase.raster import write_cog
square = MGRSGeoBox.from_mgrs("32TMS35", resolution=10.0)
data = np.zeros(square.shape.yx, dtype=np.float32)
profile = {
"dtype": "float32",
"count": 1,
"width": square.shape.x,
"height": square.shape.y,
"crs": square.crs,
"transform": square.transform,
"nodata": np.nan,
}
destination = make_s3_upath(f"s3://my-bucket/dem/{square.mgrs_code}.tif")
write_cog(data, destination, profile, band_names=["elevation"])
rewrite_tiff() handles existing files. It can recompress or retile them and copy between local storage and S3. Source and destination credentials are resolved independently, so two S3 paths may refer to different stores:
from freezebase.raster import COG_PROFILE, rewrite_tiff
# local to S3
rewrite_tiff("scene.tif", destination, profile=COG_PROFILE)
# S3 to S3
rewrite_tiff(source_on_store_a, destination_on_store_b)
rewrite_tiff("scene.tif", "scene.cog.tif", move=True)
Both functions also carry metadata into the file. tags sets dataset-level tags, band_tags sets per-band tags (one mapping per band), and units sets each band’s unit string. rewrite_tiff() merges them onto whatever the source already carried, so a rewrite can add a key without discarding the rest:
write_cog(
data,
destination,
profile,
band_names=["elevation"],
tags={"PRODUCT_TYPE": "DEM", "VERTICAL_DATUM": "EGM2008"},
band_tags=[{"SOURCE": "Copernicus GLO-30"}],
units=["m"],
)
Putting S3 and MGRS together
The public DEM is in geographic coordinates and does not share the MGRS pixel grid. An MGRSGeoBox provides the complete destination geometry for rasterio.warp.reproject:
import numpy as np
from rasterio import band
from rasterio.warp import Resampling, reproject
from freezebase.mgrs import MGRSGeoBox
square = MGRSGeoBox.from_mgrs("32TMS35", resolution=10.0)
dem = np.full(square.shape.yx, np.nan, dtype=np.float32)
with rasterio_open(dem_path) as src:
reproject(
source=band(src, 1),
destination=dem,
src_transform=src.transform,
src_crs=src.crs,
dst_transform=square.transform,
dst_crs=square.crs,
dst_nodata=np.nan,
resampling=Resampling.cubic,
)
print(f"{square.mgrs_code}: {dem.shape} pixels in {square.crs}")
print(f"elevation: {np.nanmin(dem):.0f} m to {np.nanmax(dem):.0f} m")
32TMS35: (1000, 1000) pixels in EPSG:32632
elevation: 1930 m to 4086 m
Passing rasterio.band(src, 1) lets GDAL read the source as needed instead of first loading the complete source tile into a NumPy array. The result has the same CRS, transform, and shape as every other 10 m layer resampled onto 32TMS35.
Code
import matplotlib.pyplot as plt
left, bottom, right, top = square.boundingbox
fig, ax = plt.subplots(figsize=(6.5, 5.5))
image = ax.imshow(dem, cmap="terrain", extent=(left, right, bottom, top))
fig.colorbar(image, ax=ax, label="elevation (m)", shrink=0.85)
ax.set(title=f"MGRS {square.mgrs_code}", xlabel="easting (m)", ylabel="northing (m)")
ax.ticklabel_format(style="plain")
For complete signatures and the additional mosaic and profile helpers, see the API reference.
