freezebase connects two libraries that normally see object storage independently: Universal Pathlib provides pathlib-style S3 paths through UPath, while Rasterio performs the raster I/O. freezebase enables both layers to use the same per-path authentication and endpoint configuration.
The two-client problem
Rasterio can open an s3:// URI using GDAL’s /vsis3/ driver, and s3fs can list the same bucket through a UPath. Both support credentials, but they do not ordinarily share a per-path configuration. This becomes especially awkward when one process uses multiple accounts or S3-compatible services such as MinIO or Ceph.
make_s3_upath() attaches storage options to a UPath. Filesystem operations read them through s3fs, while rasterio_open() translates them into a scoped Rasterio environment for the duration of its with block:
from freezebase.raster import rasterio_open
from freezebase.s3 import make_s3_upath
# The named profile travels with this path and all paths derived from it.
path = make_s3_upath("s3://my-bucket/scene.tif", profile="research")
with rasterio_open(path) as src:
data = src.read(1)
A local str or Path works through the same raster helper without any S3 configuration. The wrapper deliberately supports local and S3 paths; other UPath protocols are not accepted by Rasterio helpers.
The S3 integration requires s3fs, fsspec, boto3, and pydantic-settings (the last for freezebase.settings, below):
pip install "freezebase[s3]"
These dependencies are imported lazily, so local raster operations work with the base installation.
Authentication patterns
Credentials and endpoint configuration travel with each S3 path. Paths can select different named AWS profiles, including profiles used with custom S3-compatible endpoints:
from freezebase.s3 import make_s3_upath
aws_path = make_s3_upath("s3://aws-bucket/data", profile="research")
ceph_path = make_s3_upath(
"s3://ceph-bucket/data",
profile="ceph-research",
endpoint_url="https://objects.example.org",
)
The profile name—not its credentials—is stored on the path and inherited by child paths. s3fs and Rasterio independently load that same profile when they access the object store.
The endpoint may also come from the AWS configuration instead of an explicit argument. botocore’s own lookup—AWS_ENDPOINT_URL_S3, then AWS_ENDPOINT_URL, then the services section and the endpoint_url key of the profile’s section in the AWS config file—applies to a path built from the profile alone:
# ~/.aws/config
[profile ceph-research]
region = eu-central-1
endpoint_url = https://objects.example.org
ceph_path = make_s3_upath("s3://ceph-bucket/data", profile="ceph-research")
An explicit endpoint_url argument wins over all of that, and one set through client_kwargs={"endpoint_url": ...} wins over the configured value too. resolve_endpoint_url(path) returns whichever applies, and is what s3_env and subprocess_s3_env point GDAL at.
Settings from the environment
An application usually takes the profile and endpoint from its own environment variables rather than hard-coding them. S3Settings reads them under a prefix of your choosing and resolves a configured value into a Path or a UPath:
from freezebase.settings import S3Settings
# Reads MYAPP_S3_PROFILE, MYAPP_S3_ENDPOINT_URL, and the two
# MYAPP_S3_*_CHECKSUM_* policies for S3-compatible gateways.
s3 = S3Settings.from_env("MYAPP_S3_")
data_dir = s3.resolve_path("s3://my-bucket/data") # a UPath
scratch = s3.resolve_path("~/scratch") # a Path
Only the profile is required for an s3:// value; the endpoint may come from the AWS configuration instead, as above.
Selecting a profile explicitly also makes boto skip its environment-variable credential provider. A profiled path is therefore never signed by whatever AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY happens to be set in the shell. Besides making multiple stores usable in one process, this prevents ambient credentials intended for one service from being sent to another.
Resolving a profile means reading the AWS configuration files, so the resulting Rasterio session and the configured endpoint are each cached per profile and reused by every subsequent read and write. Refreshable credentials—STS or SSO profiles—still rotate on their own. Static credentials and configured endpoints are held for the lifetime of the process; if the credentials or config file is rewritten underneath a long-running process, call clear_aws_session_cache() to force the next access to resolve both again.
Explicit credentials and temporary session tokens are also supported. This is useful when credentials come from an application-specific secret store rather than an AWS configuration file:
custom = make_s3_upath(
"s3://imagery/scene.tif",
key=...,
secret=...,
token=..., # optional STS/session token
endpoint_url="https://objects.example.org",
region="eu-central-1",
)
When neither a profile nor explicit credentials are attached, both layers fall back to boto’s standard credential chain. This supports environment variables and workload roles, but the selected identity is no longer explicit on the path.
Public buckets require explicit anonymous access:
public = make_s3_upath("s3://public-bucket/scene.tif", anon=True)
Profiles, explicit credentials, and anonymous access are mutually exclusive. The helper also configures retries for both the s3fs/botocore and GDAL paths, plus checksum defaults that avoid trailers rejected by some S3-compatible gateways.
Explicit key, secret, and token values are stored in the UPath’s storage_options. Do not log, publish, or serialize those options. A named profile avoids storing the credential values on the path.
Read a public raster
The Copernicus DEM GLO-30 is a public collection of Cloud-Optimized GeoTIFFs. The same configured path can be inspected through Universal Pathlib and opened through Rasterio:
from freezebase.raster import rasterio_open
from freezebase.s3 import make_s3_upath
tile = "Copernicus_DSM_COG_10_N46_00_E008_00_DEM"
dem_path = make_s3_upath(
f"s3://copernicus-dem-30m/{tile}/{tile}.tif",
anon=True,
region="eu-central-1",
)
# s3fs
print(f"exists: {dem_path.exists()}")
# GDAL
with rasterio_open(dem_path) as src:
print(f"driver: {src.driver}")
print(f"crs: {src.crs}")
print(f"shape: {src.shape}")
print(f"bounds: {tuple(round(value) for value in src.bounds)}")
print(f"overviews: {src.overviews(1)}")
exists: True
driver: GTiff
crs: EPSG:4326
shape: (3600, 3600)
bounds: (8, 46, 9, 47)
overviews: [2, 4, 8]
Methods such as .exists(), .stat(), .iterdir(), and .glob() use the same path configuration as the raster read. Avoid wrapping a potentially large bucket listing in list() merely to inspect its first few entries; use itertools.islice() when only a sample is needed.
Write and copy rasters
write_cog() writes an array as a Cloud-Optimized GeoTIFF to either a local path or an S3 UPath. A GeoBox from the MGRS guide supplies the spatial part of the Rasterio profile:
import numpy as np
from freezebase.mgrs import MGRSGeoBox
from freezebase.raster import write_cog
square = MGRSGeoBox.from_mgrs("32TMS35", resolution=10.0)
data = np.zeros(square.shape.yx, dtype=np.float32)
profile = {
"dtype": "float32",
"count": 1,
"width": square.shape.x,
"height": square.shape.y,
"crs": square.crs,
"transform": square.transform,
"nodata": np.nan,
}
destination = make_s3_upath(f"s3://my-bucket/dem/{square.mgrs_code}.tif")
write_cog(data, destination, profile, band_names=["elevation"])
GDAL only encodes: the raster is written to memory or a local temporary file and uploaded to S3 in one s3fs put, which declares its content type and raises if the store refuses it. The stored size is read back, so an upload that was accepted but not kept fails too. rasterio_open() in a write mode on S3 works on a local copy that is uploaded when the dataset closes without an error.
rewrite_tiff() handles existing files. It can recompress or retile them and copy between local storage and S3. Source and destination credentials are resolved independently, so two S3 paths may refer to different stores:
from freezebase.raster import COG_PROFILE, rewrite_tiff
# local to S3
rewrite_tiff("scene.tif", destination, profile=COG_PROFILE)
# S3 to S3
rewrite_tiff(source_on_store_a, destination_on_store_b)
rewrite_tiff("scene.tif", "scene.cog.tif", move=True)
Both functions also carry metadata into the file. tags sets dataset-level tags, band_tags sets per-band tags (one mapping per band), and units sets each band’s unit string. rewrite_tiff() merges them onto whatever the source already carried, so a rewrite can add a key without discarding the rest:
write_cog(
data,
destination,
profile,
band_names=["elevation"],
tags={"PRODUCT_TYPE": "DEM", "VERTICAL_DATUM": "EGM2008"},
band_tags=[{"SOURCE": "Copernicus GLO-30"}],
units=["m"],
)
Putting S3 and MGRS together
The public DEM is in geographic coordinates and does not share the MGRS pixel grid. An MGRSGeoBox provides the complete destination geometry for rasterio.warp.reproject:
import numpy as np
from rasterio import band
from rasterio.warp import Resampling, reproject
from freezebase.mgrs import MGRSGeoBox
square = MGRSGeoBox.from_mgrs("32TMS35", resolution=10.0)
dem = np.full(square.shape.yx, np.nan, dtype=np.float32)
with rasterio_open(dem_path) as src:
reproject(
source=band(src, 1),
destination=dem,
src_transform=src.transform,
src_crs=src.crs,
dst_transform=square.transform,
dst_crs=square.crs,
dst_nodata=np.nan,
resampling=Resampling.cubic,
)
print(f"{square.mgrs_code}: {dem.shape} pixels in {square.crs}")
print(f"elevation: {np.nanmin(dem):.0f} m to {np.nanmax(dem):.0f} m")
32TMS35: (1000, 1000) pixels in EPSG:32632
elevation: 1930 m to 4086 m
Passing rasterio.band(src, 1) lets GDAL read the source as needed instead of first loading the complete source tile into a NumPy array. The result has the same CRS, transform, and shape as every other 10 m layer resampled onto 32TMS35.
Code
import matplotlib.pyplot as plt
left, bottom, right, top = square.boundingbox
fig, ax = plt.subplots(figsize=(6.5, 5.5))
image = ax.imshow(dem, cmap="terrain", extent=(left, right, bottom, top))
fig.colorbar(image, ax=ax, label="elevation (m)", shrink=0.85)
ax.set(title=f"MGRS {square.mgrs_code}", xlabel="easting (m)", ylabel="northing (m)")
ax.ticklabel_format(style="plain")
For complete signatures and the additional mosaic and profile helpers, see the API reference.
