download.HTTPDownloader

Download manager for fetching files over HTTP/HTTPS.

Usage

Source

download.HTTPDownloader(
    *,
    method="GET",
    auth=None,
    trusted_hosts=None,
    timeout=DEFAULT_TIMEOUT,
    progress=True
)

Built upon requests. Inspired by pooch.HTTPDownloader. Supports downloading with GET and POST requests.

Response bodies are always streamed to disk in chunks, so memory use stays flat no matter how large the file is, and redirects are always followed by the downloader itself, so trusted_hosts governs where credentials go.

Holds an open requests.Session, which keeps a connection to the server alive between downloads. Call close() when done, or use the downloader as a context manager. If neither happens, the session is closed once Python discards the downloader.

Methods

Name Description
__call__() Download a file from a URL to a directory with optional progress bar.
__del__() Close the session when Python discards the downloader.
__enter__() Return self for use as a context manager.
__exit__() Close the session on context exit.
__init__() Initialize an HTTPDownloader instance.
close() Close the session and its connections to the server.

__call__()

Download a file from a URL to a directory with optional progress bar.

Usage

Source

__call__(self, url: str, save_dir: str | Path, filename: str | None = None, overwrite: bool = False) -> Path
__call__(self, url: str, save_dir: UPath, filename: str | None = None, overwrite: bool = False) -> UPath

Supports both local paths and remote paths (e.g. UPath for S3).

Parameters
url: str

The URL of the file to download.

save_dir: str, Path, or UPath

The directory to save the downloaded file. Can be a local path or a remote UPath (e.g. S3).

filename: str = None

The filename of the downloaded file. If not provided, the filename will be inferred from the HTML header or URL. Whether provided or inferred, the name is validated to be a single path component confined to save_dir; absolute paths, directory separators, .., control characters, and reserved device names are rejected.

overwrite: bool = False
If False (the default), raise FileExistsError when the target already exists. If True, replace it.
Returns
Path or UPath
The path of the downloaded file.
Raises
ValueError

If the resolved filename is unsafe or escapes save_dir.

FileExistsError
If the target exists and overwrite is False.

__del__()

Close the session when Python discards the downloader.

Usage

Source

__del__()

Errors are ignored: this can run while the interpreter is shutting down, or on a downloader whose __init__ did not finish.

__enter__()

Return self for use as a context manager.

Usage

Source

__enter__()

__exit__()

Close the session on context exit.

Usage

Source

__exit__(*exc_info)

__init__()

Initialize an HTTPDownloader instance.

Usage

Source

__init__(
    *,
    method="GET",
    auth=None,
    trusted_hosts=None,
    timeout=DEFAULT_TIMEOUT,
    progress=True
)
Parameters
method: ("GET", "POST") = "GET"

HTTP method used to request the file.

auth: tuple[str, str] or instance of AuthBase subclass = None

HTTP authentication object (default is None). For HTTP Basic Authentication, provide (user, pass) tuple

trusted_hosts: str or iterable of str = None

Hostnames for which auth should be preserved during redirects. If not specified, auth is only sent to the original host.

timeout: float or tuple[float, float] = DEFAULT_TIMEOUT

Seconds to wait for the server, as a single value or a (connect, read) pair.

progress: bool = True
If True, show a progress bar during download.

close()

Close the session and its connections to the server.

Usage

Source

close()