GeoGenius.Downloaders.Req (geo_genius v0.2.0)

Copy Markdown View Source

Streams an artifact to disk with Req, hashing as it goes.

Req is an optional dependency: a host whose artifacts are all operator-supplied never downloads anything and need not install it. available?/0 reports whether it is loaded, and fetch/3 says so plainly rather than raising UndefinedFunctionError deep inside a phase.

The body is written to destination <> ".part" as it streams and renamed onto destination only once a 200 response has fully arrived. Any other status, a transport failure, a raised exception, or an exit/throw removes the partial file and leaves destination untouched.

Req-level retries are fixed off and cannot be re-enabled through opts. The destination file handle is opened once and written to as chunks arrive; a Req-level retry re-enters the adapter with a fresh response, so the hash state and byte count carried on that response reset while the handle keeps appending. The result is a file containing both attempts concatenated together, with a reported sha256/bytes covering only the last one: corruption that passes checksum verification.

Rate limiting is retried here instead, from the top and over a fresh handle. A 429 or 503 response closes the handle, removes the partial file, waits, and starts the request again, up to opts[:max_attempts] times (default 5). The wait honors a Retry-After header given in seconds and otherwise doubles from 1000 ms per attempt, capped at 60000 ms. Any other non-200 status is final. opts[:sleep] replaces Process.sleep/1 for tests.