# Rasters on S3

freezebase connects two libraries that normally see object storage independently: [Universal Pathlib](https://universal-pathlib.readthedocs.io/) provides pathlib-style S3 paths through `UPath`, while [Rasterio](https://rasterio.readthedocs.io/) performs the raster I/O. freezebase enables both layers to use the same per-path authentication and endpoint configuration.


# The two-client problem

Rasterio can open an `s3://` URI using GDAL's `/vsis3/` driver, and s3fs can list the same bucket through a `UPath`. Both support credentials, but they do not ordinarily share a per-path configuration. This becomes especially awkward when one process uses multiple accounts or S3-compatible services such as MinIO or Ceph.

[make_s3_upath()](../reference/s3.make_s3_upath.md#freezebase.s3.make_s3_upath) attaches storage options to a `UPath`. Filesystem operations read them through s3fs, while [rasterio_open()](../reference/raster.rasterio_open.md#freezebase.raster.rasterio_open) translates them into a scoped Rasterio environment for the duration of its `with` block:

``` python
from freezebase.raster import rasterio_open
from freezebase.s3 import make_s3_upath

# The named profile travels with this path and all paths derived from it.
path = make_s3_upath("s3://my-bucket/scene.tif", profile="research")
with rasterio_open(path) as src:
    data = src.read(1)
```

A local `str` or `Path` works through the same raster helper without any S3 configuration. The wrapper deliberately supports local and S3 paths; other `UPath` protocols are not accepted by Rasterio helpers.

> **Note: Install the S3 extra**
>
> The S3 integration requires s3fs, fsspec, and boto3:
>
> ``` bash
> pip install "freezebase[s3]"
> ```
>
> These dependencies are imported lazily, so local raster operations work with the base installation.


# Authentication patterns

Credentials and endpoint configuration travel with each S3 path. Paths can select different named AWS profiles, including profiles used with custom S3-compatible endpoints:

``` python
from freezebase.s3 import make_s3_upath

aws_path = make_s3_upath("s3://aws-bucket/data", profile="research")
ceph_path = make_s3_upath(
    "s3://ceph-bucket/data",
    profile="ceph-research",
    endpoint_url="https://objects.example.org",
)
```

The profile name--not its credentials--is stored on the path and inherited by child paths. s3fs and Rasterio independently load that same profile when they access the object store.

Selecting a profile explicitly also makes boto skip its environment-variable credential provider. A profiled path is therefore never signed by whatever `AWS_ACCESS_KEY_ID` or `AWS_SECRET_ACCESS_KEY` happens to be set in the shell. Besides making multiple stores usable in one process, this prevents ambient credentials intended for one service from being sent to another.

Resolving a profile means reading the AWS configuration files, so the resulting Rasterio session is cached per configuration and reused by every subsequent read and write. Refreshable credentials--STS or SSO profiles--still rotate on their own. Static credentials are held for the lifetime of the process; if the credentials file is rewritten underneath a long-running process, call `clear_aws_session_cache()` to force the next access to resolve it again.

Explicit credentials and temporary session tokens are also supported. This is useful when credentials come from an application-specific secret store rather than an AWS configuration file:

``` python
custom = make_s3_upath(
    "s3://imagery/scene.tif",
    key=...,
    secret=...,
    token=...,  # optional STS/session token
    endpoint_url="https://objects.example.org",
    region="eu-central-1",
)
```

When neither a profile nor explicit credentials are attached, both layers fall back to boto's standard credential chain. This supports environment variables and workload roles, but the selected identity is no longer explicit on the path.

Public buckets require explicit anonymous access:

``` python
public = make_s3_upath("s3://public-bucket/scene.tif", anon=True)
```

Profiles, explicit credentials, and anonymous access are mutually exclusive. The helper also configures retries for both the s3fs/botocore and GDAL paths, plus checksum defaults that avoid trailers rejected by some S3-compatible gateways.

> **Warning: Treat configured paths as sensitive**
>
> Explicit `key`, `secret`, and `token` values are stored in the `UPath`'s `storage_options`. Do not log, publish, or serialize those options. A named profile avoids storing the credential values on the path.


# Read a public raster

The [Copernicus DEM GLO-30](https://registry.opendata.aws/copernicus-dem/) is a public collection of Cloud-Optimized GeoTIFFs. The same configured path can be inspected through Universal Pathlib and opened through Rasterio:


``` python
from freezebase.raster import rasterio_open
from freezebase.s3 import make_s3_upath

tile = "Copernicus_DSM_COG_10_N46_00_E008_00_DEM"
dem_path = make_s3_upath(
    f"s3://copernicus-dem-30m/{tile}/{tile}.tif",
    anon=True,
    region="eu-central-1",
)
# s3fs
print(f"exists:    {dem_path.exists()}")
# GDAL
with rasterio_open(dem_path) as src:
    print(f"driver:    {src.driver}")
    print(f"crs:       {src.crs}")
    print(f"shape:     {src.shape}")
    print(f"bounds:    {tuple(round(value) for value in src.bounds)}")
    print(f"overviews: {src.overviews(1)}")
```


    exists:    True
    driver:    GTiff
    crs:       EPSG:4326
    shape:     (3600, 3600)
    bounds:    (8, 46, 9, 47)
    overviews: [2, 4, 8]


Methods such as `.exists()`, `.stat()`, `.iterdir()`, and `.glob()` use the same path configuration as the raster read. Avoid wrapping a potentially large bucket listing in `list()` merely to inspect its first few entries; use `itertools.islice()` when only a sample is needed.


# Write and copy rasters

[write_cog()](../reference/raster.write_cog.md#freezebase.raster.write_cog) writes an array as a Cloud-Optimized GeoTIFF to either a local path or an S3 `UPath`. A GeoBox from the MGRS guide supplies the spatial part of the Rasterio profile:

``` python
import numpy as np

from freezebase.mgrs import MGRSGeoBox
from freezebase.raster import write_cog

square = MGRSGeoBox.from_mgrs("32TMS35", resolution=10.0)
data = np.zeros(square.shape.yx, dtype=np.float32)
profile = {
    "dtype": "float32",
    "count": 1,
    "width": square.shape.x,
    "height": square.shape.y,
    "crs": square.crs,
    "transform": square.transform,
    "nodata": np.nan,
}

destination = make_s3_upath(f"s3://my-bucket/dem/{square.mgrs_code}.tif")
write_cog(data, destination, profile, band_names=["elevation"])
```

[rewrite_tiff()](../reference/raster.rewrite_tiff.md#freezebase.raster.rewrite_tiff) handles existing files. It can recompress or retile them and copy between local storage and S3. Source and destination credentials are resolved independently, so two S3 paths may refer to different stores:

``` python
from freezebase.raster import COG_PROFILE, rewrite_tiff

# local to S3
rewrite_tiff("scene.tif", destination, profile=COG_PROFILE)
# S3 to S3
rewrite_tiff(source_on_store_a, destination_on_store_b)

rewrite_tiff("scene.tif", "scene.cog.tif", move=True)
```

Both functions also carry metadata into the file. `tags` sets dataset-level tags, `band_tags` sets per-band tags (one mapping per band), and `units` sets each band's unit string. [rewrite_tiff()](../reference/raster.rewrite_tiff.md#freezebase.raster.rewrite_tiff) merges them onto whatever the source already carried, so a rewrite can add a key without discarding the rest:

``` python
write_cog(
    data,
    destination,
    profile,
    band_names=["elevation"],
    tags={"PRODUCT_TYPE": "DEM", "VERTICAL_DATUM": "EGM2008"},
    band_tags=[{"SOURCE": "Copernicus GLO-30"}],
    units=["m"],
)
```


# Putting S3 and MGRS together

The public DEM is in geographic coordinates and does not share the MGRS pixel grid. An [MGRSGeoBox](../reference/mgrs.MGRSGeoBox.md#freezebase.mgrs.MGRSGeoBox) provides the complete destination geometry for `rasterio.warp.reproject`:


``` python
import numpy as np
from rasterio import band
from rasterio.warp import Resampling, reproject

from freezebase.mgrs import MGRSGeoBox

square = MGRSGeoBox.from_mgrs("32TMS35", resolution=10.0)
dem = np.full(square.shape.yx, np.nan, dtype=np.float32)

with rasterio_open(dem_path) as src:
    reproject(
        source=band(src, 1),
        destination=dem,
        src_transform=src.transform,
        src_crs=src.crs,
        dst_transform=square.transform,
        dst_crs=square.crs,
        dst_nodata=np.nan,
        resampling=Resampling.cubic,
    )

print(f"{square.mgrs_code}: {dem.shape} pixels in {square.crs}")
print(f"elevation: {np.nanmin(dem):.0f} m to {np.nanmax(dem):.0f} m")
```


    32TMS35: (1000, 1000) pixels in EPSG:32632
    elevation: 1930 m to 4086 m


Passing `rasterio.band(src, 1)` lets GDAL read the source as needed instead of first loading the complete source tile into a NumPy array. The result has the same CRS, transform, and shape as every other 10 m layer resampled onto `32TMS35`.


Code

``` python
import matplotlib.pyplot as plt

left, bottom, right, top = square.boundingbox
fig, ax = plt.subplots(figsize=(6.5, 5.5))
image = ax.imshow(dem, cmap="terrain", extent=(left, right, bottom, top))
fig.colorbar(image, ax=ax, label="elevation (m)", shrink=0.85)
ax.set(title=f"MGRS {square.mgrs_code}", xlabel="easting (m)", ylabel="northing (m)")
ax.ticklabel_format(style="plain")
```


<figure class="figure">
<p><img src="rasters-on-s3_files/figure-html/cell-5-output-1.png" class="figure-img" width="598" height="468" /></p>
<figcaption>Copernicus DEM GLO-30 resampled onto MGRS square 32TMS35 in the Bernese Alps.</figcaption>
</figure>


For complete signatures and the additional mosaic and profile helpers, see the [API reference](../reference/index.md).