# Ingesting a collection An import turns a reviewed document into a verified release. You write a manifest naming the collection, the release, the provider that parses it, and every file it is built from. `GeoGenius.import/1` registers that manifest in the catalog, claims a durable import run, and hands the run to a runner. The pipeline downloads, checks, stages, normalizes, relates, analyzes, and verifies. Nothing any host reads changes until you publish, which is a separate call. This guide covers the manifest, the provider contract, the four adapters, the runner backends, the phases, and the five mix tasks. For reading a published catalog see [`reading.md`](reading.md); for installing the schema see [`installation.md`](installation.md). ## The manifest A manifest is the whole description of one release. Nothing discovers a release and nothing follows a mutable "latest" pointer: a new release becomes available because somebody reviewed a document and committed it. ```json { "collection": "demo", "collection_name": "Demo Territories", "description": "Operator-drawn delivery zones", "release": "r1", "provider": "geojson", "requires_geometry": true, "source_date": "2026-01-15", "authorities": [{ "key": "demo", "name": "Demo Operations" }], "area_types": [{ "key": "territory", "rank": 100 }], "sources": [ { "source_key": "demo:territories", "provider": "geojson", "license": "CC0-1.0", "attribution": "Demo Corp", "release_key": "2026-01", "source_date": "2026-01-15", "artifacts": [ { "logical_name": "territories.geojson", "url": "https://example.test/territories.geojson", "operator_supplied": false, "format": "geojson", "required": true, "sha256": "0000000000000000000000000000000000000000000000000000000000000000", "bytes": 4096, "members": ["territories.geojson"] } ] } ], "options": { "code_property": "territory_id", "name_property": "territory_name", "area_type": "territory" } } ``` `collection` and `release` are required, and both must match `~r/\A[a-z0-9][a-z0-9_.-]*\z/`. That is a path-traversal control rather than a naming style: `GeoGenius.Manifest.load/3` joins the two into a filesystem path, and the character class excludes the separator while requiring an alphanumeric first character, so neither `..` nor an absolute path can match. `collection_name` and `description` are what the collection row carries; leaving `collection_name` out names the collection after its key. `provider` must resolve through the registry described under [Providers](#providers). Collection-level `requires_geometry` is a promise about the whole collection, checked at publication time: with it set, `verify_release` refuses a release in which any area lacks a boundary. An `area_types` entry may also carry `"requires_geometry": true`; that leaves the collection usable for metadata-only types while requiring boundaries for areas of the marked type. Omitted area-type flags default to false, and an explicit false is preserved when the manifest is stored on the release row. A collection-level true still requires boundaries for every area, regardless of its type. `authorities` names who is responsible for the identifiers; an authority key is the first segment of every `area_key` keyed under it, so `demo` above yields `demo:territory:west`. It is a list because a collection may draw on more than one: a US release keyed partly by the Census, partly by the USPS, and partly by its own vendor identifiers names all three, and a provider may emit areas under any of them. Every authority an area is keyed under must be declared here, since an area naming one the collection does not carry is refused. `authorities` is required and must name at least one. That is stricter than `area_types`, which may be declared as an empty list: a manifest declaring no area types registers none, and a release that normalizes an area whose type it never declared fails. Nothing else supplies either. A manifest with no authorities can therefore register no area at all, and rejecting it at load names the field rather than failing partway through normalization. `area_types` declares the ranked types this collection uses, low rank containing high: ```json [ { "key": "bounded_zone", "rank": 10, "requires_geometry": true }, { "key": "metadata_record", "rank": 20 } ] ``` `GeoGenius.Providers.GeoJSON` and `GeoGenius.Providers.CSV` key every area they emit under one authority, which they take from `options["authority"]` or, when that is absent, from the manifest's own single authority; a manifest declaring several must name which one under that option. `sources` is a list because a release can draw on more than one feed: an authority's own publication plus an operator-supplied correction, for instance. Each source needs a `source_key`, a `provider`, a `license`, a `release_key` naming the vintage, and at least one artifact. `attribution` is the credit line the license asks you to carry. Each artifact is either downloadable or operator-supplied, never both and never neither. A downloadable artifact carries a `url` and `operator_supplied: false`; an operator-supplied one omits the `url` and sets `operator_supplied: true`, and you place the file in the cache yourself before the run. Both carry `sha256` and `bytes`, which are expectations rather than observations: the pipeline hashes whatever it actually got, cache hit and fresh download alike, and PostgreSQL refuses the mismatch. `required` defaults to true. `required: false` means exactly one thing: an artifact with no `url`, and no copy in the cache, is counted in the `optional_missing` metric instead of failing the run. It is not a promise that the artifact is skippable in general. An optional artifact that names a `url` is downloaded like any other, and a download that fails ends the import, because a URL the manifest declares and the network refuses is a condition the operator has to see rather than a file the release can quietly go without. An optional artifact that has to tolerate a failing source is one that belongs in the cache, not on a URL. `members` names the files expected inside an archive, which is what the shapefile provider reads to find its `.shp` among the rest. It is validated as a list of non-empty strings. An artifact may also carry a `cache_key`. Use it for a licensed file no URL can serve, so the operator chooses the name they will place the file under rather than reverse engineering the one the pipeline would have derived. When you leave it out, the key is `///`. Either way the segments are validated, because a segment carrying a separator would put the file outside the cache root. `options` is the provider's own configuration and nothing outside the provider reads it. Manifest validation checks that every key `required_options/0` names is present, and then hands the whole map to `validate_options/1`, if it exports one, to check the options only it understands. Both run for **every provider the manifest names** -- the release's own and each source's -- and both run before any release row exists, so a manifest that is missing a required key or carries a malformed option is rejected at load rather than several phases into a run. `GeoGenius.Manifest.to_map/1` produces exactly the document above, and `from_map(to_map(manifest))` is the identity. That is what makes storing the manifest on the release row lossless: the pipeline rebuilds the manifest it was given from `release.manifest` without reading the file again. ## Where manifests are found `GeoGenius.import(collection: "demo", release: "r1")` looks for `/demo/r1.json` under each directory in the search path, in order, and uses the first one that is a regular file. The search path is `opts[:manifest_paths]`, then `config :geo_genius, :manifest_paths`, and always the package's own `priv/geo_genius/manifests` last: ```elixir config :geo_genius, manifest_paths: ["priv/geo_genius/manifests"] ``` The package's directory is searched last on purpose, so a host shipping a corrected manifest under the same name takes precedence over one the package carries. A lookup that finds nothing returns `{:error, %GeoGenius.ManifestError{}}` naming the candidate path it checked, which is what tells "rejected by name" apart from "the file is not there". You can skip the search entirely by handing `GeoGenius.import/1` a manifest you built yourself: ```elixir {:ok, manifest} = GeoGenius.Manifest.from_map(decoded) GeoGenius.import(manifest: manifest) ``` ## Providers A provider adapts one source format. It is called for parsing and never for a write: every write the import makes is the pipeline's, through `GeoGenius.Catalog`. Four providers ship: `GeoGenius.Providers.GeoJSON` (`"geojson"`), `GeoGenius.Providers.CSV` (`"csv"`), `GeoGenius.Providers.Shapefile` (`"shapefile"`), which converts its archive with `ogr2ogr -f GeoJSONSeq -t_srs EPSG:4326 -lco RS=NO` before streaming the resulting GeoJSON Feature sequence in bounded batches, and `GeoGenius.Providers.SimpleMaps` (`"simplemaps"`), which has a section of its own [below](#the-simplemaps-provider). Shapefile staging is the only shipped provider that requires GDAL's `ogr2ogr` binary. The conversion explicitly reprojects to EPSG:4326 and writes one Feature per line; the sequence reader decodes and emits each configured batch before reading further, so neither the converted artifact nor all of its staged rows are retained in memory. `GeoGenius.Provider` declares six callbacks every provider implements and two optional validation callbacks. A source may name its own `provider`, and a source that names none inherits the release's. That is what lets one release draw on sources in different formats: a delimited file staged by the CSV provider beside an archive staged by the shapefile provider, converging on `area_key` so one area carries what each source contributed. The pipeline asks `artifacts/1` once per source with the manifest narrowed to that source, dispatches `stage/5` on the artifact, and dispatches `normalize/2` and `asserted_relations/2` on the row's own artifact. `relations/1` is the union: relations are rebuilt when any of a release's providers asks for it, so a source whose areas nest spatially is still measured beside a source that carries no geometry at all. One caveat follows from `options` being release-level while the option vocabulary is per-provider: an `implied_areas` entry names its code key in one provider's suffix, so a release cannot share one between a CSV source and a GeoJSON source. SimpleMaps requires no options, so its optional `"non_census_state_area_type"` key can coexist with the required options of another provider in a composed release. `required_options/0` returns the `options` keys manifest validation insists on. The GeoJSON provider requires `"area_type"` and `"code_property"`; the CSV provider requires `"area_type"` and `"code_column"`. SimpleMaps reads its columns by name and requires none; the optional key documented below changes only the type of its six USPS-only state codes. `validate_options/1` is optional. A provider that exports it is handed the whole `options` map at load, once the required keys are known to be present, and the error it returns becomes a manifest error naming the field. This is where a provider checks what only it understands -- the shipped generic providers validate their `implied_areas` entries here. A provider that exports none contributes no checks beyond the required keys. `validate_manifest/1` is the second optional validation seam. It receives the fully decoded manifest after structural and option validation, so a provider can check a relationship between fields without putting source-specific policy in the manifest parser. Every provider named by the release or one of its sources receives the same complete manifest. SimpleMaps uses this to ensure its effective non-Census state type -- the configured `"non_census_state_area_type"`, or `"state"` when omitted -- names a declared area type whose effective `requires_geometry` value is false. `artifacts/1` returns the artifacts, across the manifest's sources, this provider will stage. The shipped providers stage every declared artifact and so delegate to `GeoGenius.Provider.all_artifacts/1`. `stage/5` receives the manifest, one artifact, the local path the artifact was resolved to, an `emit` function, and an options list. It parses the file and hands `emit` lists of `%GeoGenius.Staging.Row{}`. Emit in chunks rather than once for the whole file: each call bounds the size of a single insert, and each call is also where the run's lease is renewed, so a provider that emits once at the end of a two-hour parse lets its own lease go stale. The options list carries `:work_dir`, a directory resolved per run and removed when the run ends, and `:command`, the command adapter to shell out through. Return `{:error, reason}` for a file you cannot parse; raise only for a defect in the provider itself. `normalize/2` receives the manifest and one staged row and returns `{:ok, %GeoGenius.Provider.Area{}}`. It is pure: no adapters, no environment, no database. Return `:skip` for a row that carries no area, such as a summary record or a feature with no usable code, since that is an expected shape rather than a failure. A row may also imply areas identified by *other* columns on itself. A record carrying a grouping code -- a statistical grouping on an administrative record, an administrative parent on a place record -- describes that grouping as well as itself, and the GeoJSON and CSV providers read those columns from an `implied_areas` manifest option rather than requiring a provider of your own: ```json "options": { "area_type": "place", "code_property": "code", "implied_areas": [ { "area_type": "cluster", "code_property": "CLUSTER", "names": { "1": "Outer Cluster", "2": "Inner Cluster" }, "relation": "contains" } ] } ``` Each entry names the area type to imply, the column carrying its code, a `names` map from code to display name, an optional `authority` override, and an optional `relation` defaulting to `"contains"`. The code key follows the provider's own vocabulary: `code_property` under GeoJSON and shapefile manifests, `code_column` under CSV ones. These entries are validated at load, through `validate_options/1`, so a missing `area_type`, an unknown `relation`, a `names` map that is not an object, or an entry written in the other provider's vocabulary is rejected before anything is downloaded. The providers validate again as they read, because a `%GeoGenius.Manifest{}` built in Elixir never passes through `GeoGenius.Manifest.from_map/2`. An implied area carries no geometry and no centroid: a grouping a source names only by code has no shape of its own there, and deriving one from the row's geometry would claim the grouping's boundary is one member's boundary. A blank code implies nothing and is not an error, since a column populated for most rows and blank for a few is ordinary. A code that is present but missing from `names` fails the release, because keying an area with no name would publish an unlabelled area. The same entries drive `asserted_relations/2`, so each implied area also gets an edge to the row's own area. That composes with `relations/1` rather than replacing it: a provider that rebuilds measured relations from geometry can assert membership the geometry cannot express, and both land in one release. It may also return `{:ok, [%GeoGenius.Provider.Area{}, ...]}`, which is what a source that denormalises a hierarchy into every row needs: a city row that also carries its county and its state describes all three, and returning them together is cheaper and truer than staging the same row once per level. Areas repeated across rows converge on `area_key`, so emit an implied parent from every row that implies it rather than tracking which ones you have already emitted. `relations/1` returns `:rebuild` when the areas nest spatially or by code and `:none` when the collection carries no hierarchy. The GeoJSON, CSV and shapefile providers delegate to `GeoGenius.Provider.always_rebuild/1`; SimpleMaps returns `:none`, because it carries centroids and no boundaries and so has no overlap to measure. Note that a rebuild pairs a lower `area_type.rank` against a higher one, so a release built from one manifest of one area type produces no measured relations at all; hierarchy appears when a release composes several sources of different types. `asserted_relations/2` returns the edges one staged row states outright, as `{parent_area_key, child_area_key, relation_type}` tuples with `relation_type` one of `"contains"`, `"mostly_contains"`, or `"overlaps"`. It is for the hierarchy no overlap test can derive: a county FIPS on every city row, an admin parent code on every place, or a source with no geometry at all. Both keys are the strings `GeoGenius.Provider.Area.key/1` composes -- `authority_key`, `area_type_key` and `code` joined by `:`, the same composition PostgreSQL stores in `area.area_key`. Build the area an edge names and take its key from there rather than joining the parts by hand, so an edge can only name a key `normalize/2` produced. It composes with `relations/1` rather than replacing it: a provider whose areas nest spatially may both rebuild measured relations and assert edges the geometry does not carry, and both land in the same release. Asserted edges are written after every area exists, so an edge may name any area the run produced rather than only the ones its own row emitted. A provider whose source carries no hierarchy in its columns delegates to `GeoGenius.Provider.no_asserted_relations/2`. To write your own, implement the behaviour and register the module under the name your manifests will use: ```elixir config :geo_genius, providers: %{"acme" => MyApp.AcmeProvider} ``` Shipped providers are merged ahead of your registrations, so registering your own module under `"geojson"` replaces the one this package carries. A name pointing at a module that does not load is a configuration error and says so, rather than being read as a provider with no requirements, which would make manifest validation accept every options block written for it. The spec these providers were built against writes `stage/2` and `normalize/1`. The shipped arities are `stage/5` and `normalize/2`, because a provider needs the manifest on every call and `stage` needs somewhere to emit to and an adapter to shell out through. The callbacks and their responsibilities are otherwise unchanged. `asserted_relations/2` is required, not optional. A provider written against the earlier six-callback behaviour keeps compiling by adding one line: ```elixir @impl GeoGenius.Provider defdelegate asserted_relations(manifest, row), to: GeoGenius.Provider, as: :no_asserted_relations ``` ## The SimpleMaps provider `GeoGenius.Providers.SimpleMaps` (`"simplemaps"`) parses the SimpleMaps US cities and US ZIP codes datasets: two comma-delimited files, `uscities` and `uszips`, named by the artifact's `logical_name` rather than by a manifest option. It requires no options. The files are licensed downloads, so the manifest this package ships, `us_simplemaps`, declares both artifacts `operator_supplied` with a `cache_key` and no `url`; place your copies in the cache under those keys, and replace each artifact's `sha256` and `bytes` with the digest and size of the copy you licensed. The shipped `license` and `attribution` describe the paid tier, whose row counts the validation rules below are measured against, so replace those two as well if you licensed a different tier -- the free files are CC BY 4.0. A host-shipped manifest wins over the package's own, so the usual route is to copy it into your own `:manifest_paths` directory and edit it there. ### Four area types | Area type | Rank | Comes from | |-----------|------|---------------------------------------------------------| | `state` | 10 | the `state_id`/`state_name` columns of both files | | `county` | 20 | the `county_fips_all` and county-name columns of both | | `city` | 30 | a `uscities` row | | `zip` | 40 | a `uszips` row | Only cities and ZIPs exist as rows. Counties and states are read out of columns, so they carry only what those columns say: a code, a name, and a FIPS or ANSI code. They get no centroid and no attributes, because a row's `lat`/`lng` and its demographic columns measure the city or ZIP and nothing else. Both files denormalise the whole hierarchy into every row, so `normalize/2` returns several areas per row: a `uscities` row is its city, every county it falls in, and its state; a `uszips` row is every county it touches, the ZIP, and its state. A ZIP row also names a city, but only as the mailing name the USPS prefers for that ZIP, so that becomes a `:mailing` name on the ZIP rather than a city area of its own -- the `uscities` file is where a city gets its identity. A row whose counties do not all lie in the state it names yields those states too, one area each, read from the county FIPS prefixes. They carry a code and no name: `state_name` on the row describes the row's state, not theirs. ### Three authorities, and the six state codes that key under the USPS The manifest must declare all three. An area naming an authority the collection does not carry is refused when the area is registered. | Authority | Keys | |--------------|------------------------------------------------| | `simplemaps` | cities, under the vendor's own `id` | | `census` | counties, and states that have an ANSI code | | `usps` | ZIPs, and the six state codes that do not | `AA`, `AE` and `AP` are USPS constructs for military mail -- Armed Forces Americas, Europe and Pacific. `FM`, `PW` and `MH` are the Freely Associated States -- Micronesia, Palau and the Marshall Islands -- sovereign countries the USPS serves rather than states of any kind. The Census defines none of the six and assigns none of them an ANSI code, so keying them under `census` with an `ansi_state` code would assert two things no source says. They key under `usps` and carry a `usps_state` external code instead. By default those six still use the `state` area type, preserving the area keys emitted by existing SimpleMaps manifests. In a collection where ordinary states require a Census boundary, opt only those six into a separately declared non-geometric type with: ```json { "area_types": [ { "key": "state", "rank": 10, "requires_geometry": true }, { "key": "postal_region", "rank": 15, "requires_geometry": false }, { "key": "county", "rank": 20 }, { "key": "city", "rank": 30 }, { "key": "zip", "rank": 40 } ], "options": { "non_census_state_area_type": "postal_region" } } ``` `postal_region` is an example, not a reserved name. The option accepts any non-blank area-type key and the manifest must declare that key with `requires_geometry` false. As with every area type, omitting `requires_geometry` has the same effective false value; spelling it out makes a mixed geometric manifest easier to audit. The option changes only `AA`, `AE`, `AP`, `FM`, `MH`, and `PW`: their area keys and their asserted state-like-to-ZIP edges use the selected type. Ordinary ANSI states, counties, cities, ZIPs, authorities, and external codes are unchanged. **A host looking a state up by code must query both code types.** `ansi_state` alone silently misses those six, and every ZIP under them; the two together are the whole set. `AA`, `AE` and `AP` also publish nameless: `state_name` is blank in every one of their `uszips` rows and `uscities` carries no row at all for any of the six codes, so those three areas carry no name. Nothing invents one -- supplying "Armed Forces Europe" out of a table in this library would manufacture a fact the source does not carry. They exist so their ZIPs have a parent; a host that wants a label supplies its own. ### Area keys `GeoGenius.Provider.Area.key/1` composes every one of them: | Area type | Form | Example | |-----------|--------------------------|----------------------| | `city` | `simplemaps:city:` | `simplemaps:city:1840021543` | | `county` | `census:county:` | `census:county:06075` | | `state` | `census:state:` | `census:state:CA` | | `state` (default) or configured type | `usps::` | `usps:state:AE` or `usps:postal_region:AE` | | `zip` | `usps:zip:` | `usps:zip:94110` | ### Relations SimpleMaps carries centroids and no boundaries, so `relations/1` is `:none` and the whole hierarchy comes from `asserted_relations/2`, read from the FIPS columns. A row asserts a state-to-county edge per county it names, and a county-to-child edge per county: `contains` when the row names one county, `overlaps` when it names several, since a place crossing county lines is contained by neither. No edge is ever asserted between a city and a ZIP in either direction -- a ZIP crosses city lines and neither one contains the other. **A county's state comes from its own FIPS, never from the row's `state_id` column.** A five-digit county FIPS begins with the two-digit FIPS of the state that assigns it, and that is the only statement about the county's state a row cannot get wrong: ZIP `20041` is a District of Columbia address for county `51107`, which is in Virginia, and 150 of the source's 3,233 counties are named on rows of more than one state. A county whose prefix names no state `GeoGenius.Providers.SimpleMaps.Fips` carries gets no state parent at all, since the row's own column is the value already known to be describing something else. A row naming no county asserts one edge instead: its state contains the ZIP directly. That is the only parent such a row names, and hanging it there is truer than leaving it with no parent at all. ### The county rules that fail a release `GeoGenius.Providers.SimpleMaps.Validation` runs before anything is built from a row. A row that fails ends the import, naming the column and enough of the row -- `zip`, `city`, `state_id` -- to find it in a file of tens of thousands. Both files are checked for: - `county_fips` present but absent from `county_fips_all`. The source is contradicting itself about which counties the row belongs to. - `county_fips_all` and the county-name column (`county_name_all` on `uscities`, `county_names_all` on `uszips`) of different lengths. The two are paired positionally, so unequal lengths mean at least one FIPS is paired with the wrong name. `uscities` is additionally checked for a row naming **no** county, and `uszips` for `county_weights` that is blank beside a populated `county_fips_all`, that does not decode as a JSON object, or whose keys are not exactly `county_fips_all`. A blank code in `id`, `zip` or `state_id` is an error too, rather than the `:skip` the CSV provider returns for a blank code column. SimpleMaps populates those columns in every row of both files, so a blank one is a corrupt or truncated download, and `area_key` is `::` -- a blank code would put every such row under one shared, meaningless key. **The county-less asymmetry is measured, not assumed.** A `uscities` row naming no county fails the release; a `uszips` row naming none is ordinary data. Every one of `uscities`' 109,071 rows names at least one county, because every city sits in one, so a row naming none is corruption. 617 of `uszips`' 41,551 rows name none, because a military APO/FPO ZIP delivers to an overseas address belonging to no US county; failing them would abort a real import on the first one and never reach the other 616. ### What is not checked **Every rule reads one staged row.** Nothing compares across rows or across the two files. Cross-file agreement -- one FIPS mapping to two different states, or `uscities` and `uszips` spelling one county's name two different ways -- cannot be seen from a single row and is **not** checked here. GeoGenius accepts several official names for one area, so a disagreement of that kind lands in the catalog as two names rather than as an error. A host that needs cross-file agreement enforced verifies it itself, after the import. ## The adapters Four adapter roles are resolved the same way: `opts` on the call, then application environment, then the shipped default. | Role | Config key | Default | What it does | |---------------|--------------|--------------------------------|---------------------------------------------------| | `:cache` | `:cache` | `GeoGenius.Caches.FileSystem` | Where artifacts live between runs | | `:downloader` | `:downloader`| `GeoGenius.Downloaders.Req` | Streams a remote artifact to a local path | | `:command` | `:command` | `GeoGenius.Commands.System` | Invokes an external executable | | `:notifier` | `:notifier` | `GeoGenius.Notifiers.Noop` | Delivers import lifecycle events | Substitute one for every import by configuring it, or for a single call by passing it: ```elixir config :geo_genius, notifier: MyApp.GeoGeniusNotifier GeoGenius.import(collection: "demo", release: "r1", cache: MyApp.S3Cache) ``` The cache is checked before the downloader is ever consulted, which is what makes a rerun cheap and what makes an operator-supplied artifact work at all. The shipped cache stores artifacts as files under `opts[:cache_dir]`, then `config :geo_genius, :cache_dir`, then a `geo_genius` directory under the system temporary directory. A cache hit is hashed on every run, including one whose artifact row already carries a `validated_at`, because trusting that column would mean a corrupted or swapped cache entry is never noticed again. `GeoGenius.Downloaders.Req` needs `req`, which is an optional dependency: add `{:req, "~> 0.7"}` to your own `mix.exs` if you import anything downloadable. A downloader reports only what it observed, never whether that matched an expectation. The comparison lives in `record_artifact_observation` in PostgreSQL, and only there, which is what keeps the two from disagreeing. The command adapter is only reached by a provider that shells out. What the pipeline actually hands a provider is `GeoGenius.Pipeline.CommandAllowlist`, which accepts `ogr2ogr` and refuses everything else by name. That is a guardrail rather than a sandbox: it bounds what a provider does by accident, not what it does deliberately. A notifier is called for side effects only and its return value is ignored. It cannot fail an import: the pipeline wraps every call, so one that raises or exits has its event logged at warning level and dropped, and the phase continues. The events are `:import_started`, `:phase_advanced`, `:import_completed`, `:import_failed`, `:release_published`, and `:release_rolled_back`; `GeoGenius.Notifier.events/0` returns that fixed list so your notifier can match exhaustively. ## Choosing a runner A runner decides where `GeoGenius.Pipeline.execute/3` actually runs. It owns none of the run's state: `enqueue/3` starts the work and returns, and the run's status, progress, and error all live in PostgreSQL. Three backends ship, and with nothing configured the first available automatic backend wins: `Runners.Task`, then `Runners.Inline`. `GeoGenius.Runners.PgFlow` is explicit even when a PgFlow engine happens to be running. PgFlow's finite job deadline is a host decision for bounded work; merely installing it for an unrelated workflow must not make it the execution policy for an import that can run for hours. GeoGenius declares `pgflow ~> 0.7.0` as an optional dependency to establish compile order when a host opts in, without installing PgFlow for other hosts. Pin the runner per call or in application configuration after choosing a suitable deadline. The deadline is compile-time PgFlow job definition data, so it must be configured before GeoGenius compiles: ```elixir config :geo_genius, runner: GeoGenius.Runners.PgFlow, pgflow_job_timeout_seconds: 14_400 ``` `14_400` is only an example, not a package default. Choose a finite bound for the host's complete workload. It must exceed the effective `stale_after_seconds` on every import (900 by default), but that inequality alone does not prove that the whole import fits inside the deadline. `GeoGenius.Runners.Task` is what a host gets with no runner configuration. It runs each import as a supervised task under `GeoGenius.TaskSupervisor`, a `Task.Supervisor` the `:geo_genius` application starts itself. `GeoGenius.import/1` returns as soon as the task starts, and you read the outcome back with `GeoGenius.status/2` or `GeoGenius.await/3`. One exception is worth knowing: setting `config :geo_genius, :task_supervisor` selects this backend whether or not anything is actually running under that name. A host that named a supervisor and never started it gets a named error from `enqueue/3` saying so, rather than a silent downgrade to `Runners.Inline`, which would block the caller for the length of an import without ever saying why. **If you care about shutdown ordering, configure your own supervisor:** ```elixir config :geo_genius, :task_supervisor, MyApp.TaskSupervisor ``` with `MyApp.TaskSupervisor` listed in your own children *after* `MyApp.Repo`. Here is why. Applications start in dependency order and stop in the reverse of it, so `:geo_genius` starts before your application and therefore stops after it. With no configuration, the library's own supervisor sits outside your tree: on shutdown your Repo finishes stopping first, and an import still running at that moment keeps running against a Repo that is already gone. Its execution guardian cannot record a later task death once that Repo is unavailable. A supervisor you place after your Repo does not have that problem, because a supervisor terminates its children in the reverse of their start order, so the tasks under it are stopped before the Repo is. Setting the key at boot also tells GeoGenius to start no task supervisor of its own; its execution-guardian supervisor remains active. Killing a running import is not graceful: a `Task` does not trap exits and cannot clean up its staging table. Under the host-owned ordering, the guardian sees that death while the Repo is still alive and records a durable failure with the exact executor identity. Whole-node loss, or package-owned shutdown after the Repo has already stopped, leaves no process able to record the outcome. A stale heartbeat never authorizes another executor to take over; an operator must fail that abandoned attempt explicitly before creating a replacement with `GeoGenius.retry_failed/2`. `GeoGenius.Runners.Inline` runs the import in the calling process and does not return until it has finished. Choose it deliberately for tests and for scripts, where blocking is the point and the outcome is already durable by the time the call returns. It is also where automatic resolution ends up when Task cannot accept work, which for a national vintage means blocking the caller for hours. More on `GeoGenius.Runners.PgFlow`: it runs the import as a durable PgFlow job. The optional dependency edge ensures this library's job submodule compiles after `PgFlow.Job` when the host adds PgFlow directly. PgFlow 0.7.0 compiles its dashboard modules even when the host does not mount the dashboard, so a current PgFlow consumer must also declare PgFlow's optional `phoenix`, `phoenix_live_view`, and `livefilter` dependencies. This is a PgFlow packaging requirement, not a dependency GeoGenius imposes on hosts using another runner. This backend reports itself available once `pgflow` and the job submodule are loaded and `PgFlow.Supervisor` is running; short of that it returns an explicit not-enqueued error naming what is missing rather than raising deep inside `PgFlow.enqueue/2`. It also verifies that PgFlow and the import use the same Repo, that the stored flow and `execute` step carry the compiled timeout, and that a healthy queue worker is polling before enqueue. This keeps readiness, durable enqueue, and execution in one database. Hosts upgrading from an earlier PgFlow move their PgFlow schema to core V03 and helpers V07 first, following PgFlow's [Upgrading to 0.7](https://github.com/agoodway/pgflow/blob/v0.7.0/README.md#upgrading-to-07) and full [upgrade guide](https://github.com/agoodway/pgflow/blob/v0.7.0/docs/UPGRADING_UPSTREAM_2026_10.md). After setting the compile-time timeout, generate and run the PgFlow job migration: ```console mix pgflow.gen.job_migration GeoGenius.Runners.PgFlow.Job mix ecto.migrate ``` Then start `{PgFlow, repo: MyApp.Repo, jobs: [GeoGenius.Runners.PgFlow.Job]}` in the host's supervision tree. Changing `:pgflow_job_timeout_seconds` later requires forcing GeoGenius to recompile and generating another PgFlow job migration; enqueue fails before acceptance if the compiled module and stored definition disagree. Pin a backend for every import or for one call: ```elixir config :geo_genius, runner: GeoGenius.Runners.Inline GeoGenius.import(collection: "demo", release: "r1", runner: GeoGenius.Runners.Inline) GeoGenius.import( collection: "demo", release: "r1", runner: GeoGenius.Runners.PgFlow, stale_after_seconds: 900 ) ``` The backend a run was claimed under is recorded on `import_run.runner_backend`, so a run stays readable after you change the configuration. There is no Oban backend; a host that wants one implements `GeoGenius.Runner`'s three callbacks, which is the reason the behaviour exists rather than a fixed backend list. ## The phases An import walks a fixed sequence, advancing the durable run row at each boundary. The catalog accepts only the next edge in that sequence: callers cannot skip ahead or use a phase advance to create a terminal status. A phase that fails stops the sequence and records the failure through `fail_import`; `complete_import` records a verified non-publishing completion, while `publish_import` records completion and swaps publication atomically. `downloading` resolves every artifact the release composes to a local file. The cache is checked first, a miss is downloaded, and whatever came back is hashed and recorded against the manifest's expectation. Measures `artifacts`, `downloaded`, `cached`, and `bytes`. `validating` re-reads the artifacts and checks that every required one carries a `validated_at`, counting the optional ones nothing could obtain. It re-reads rather than trusting what the previous phase believed, which is what makes it a check. `staging` calls the provider's `stage/5` for each artifact and writes the emitted rows into an unlogged table of this run's own. The table is emptied before the attempt stages. A replacement attempt gets a new run id and a separate table; an abandoned attempt's rows are evidence to sweep after that attempt is terminal, never input another executor resumes. `normalizing` reads the staged rows back in batches, calls the provider's `normalize/2` on each, and writes the areas, names, codes, membership, and boundaries through `GeoGenius.Catalog`. A batch is collected in full and then written as a set: one statement for its areas, one for its names, one for its codes, one for its memberships, through the [set writes](sql_api.md#set-writes), and one for all its boundaries. A source that denormalises a hierarchy names the same county in every city row of it, so a batch describes far fewer distinct areas than it has rows, and one statement per area spends a round trip on every repeat. Measured on the US SimpleMaps import -- 150,622 staged rows, 466,262 area writes over 153,917 distinct areas -- this phase costs 2,449 statements and 39.6 s written as sets, against 1,755,328 statements and 994.8 s written one area at a time. Collecting before writing also decides what a provider's own error leaves behind: an illegal name kind, or a staged row naming an artifact this run did not stage, fails the phase with none of that batch written, rather than partway through it. Earlier batches are already committed to the candidate during that attempt; nothing opens a transaction around a batch. A failed attempt is not resumed. `retry_failed` resets the candidate from the replacement manifest before the new run writes it. Calling `GeoGenius.import/1` again does not perform that reset. An identical failed candidate returns `%GeoGenius.CandidateError{reason: :failed}` with the failed run id, while a changed manifest returns `:manifest_changed`; only `GeoGenius.retry_failed/2` creates the next attempt. `relating` rebuilds measured relations from boundary overlap when `relations/1` asks for it, and then writes every edge `asserted_relations/2` returns for each staged row, one set write per batch the same way `normalizing` writes areas. The two compose: a release can carry both. Measures `relations` (only when a rebuild ran) and `asserted_relations`, which counts edges asserted rather than distinct edges written -- two rows asserting the same edge each add one, though the edge itself upserts to a single row. `indexing` runs the run-fenced `analyze_import`, so the release has statistics before anything plans a query against it. `verifying` runs the run-fenced `verify_import` and fails the run when the report is not `ok`. This is the gate: a release that has no areas, that lacks a boundary somewhere the collection or area type requires one, that carries a relation or a membership belonging to another collection, or that declares no source releases, does not become publishable. `verify_release` also reports boundaries with invalid geometry, though nothing an import does can produce one: `put_boundary` repairs its input with `ST_MakeValid` before storing it, and the `boundary_geom_valid_chk` constraint refuses an invalid geometry written to the table by any other route. That check is a backstop against a writer that bypasses both, not a condition an import reaches. `publishing` runs only when you passed `publish: true`. `publish_import` requires a validated observation for every required selected artifact, verifies and publishes the candidate, marks the attempt completed, and removes its lease in one transaction. An optional artifact that was not available remains part of the attempt snapshot without blocking publication. Without `publish: true`, the pipeline ends after `verifying` and calls `complete_import`. That operation re-checks the same required observations and release invariants, then completes the run and removes its lease without changing publication. A later `publish_release` call verifies the completed release again before making it visible. The import opens no transaction around the whole pipeline. Every catalog function is atomic on its own, so a long run does not hold one connection for the length of its slowest phase. The staging table is dropped on success and on failure alike. Every staging insert, set-based normalization or asserted-relation write, measured relation rebuild, analyze, verification, and publication statement disables DBConnection checkout retries. A timeout or dropped connection leaves a mutating statement's outcome ambiguous: automatically checking out a fresh connection could run the whole statement twice. GeoGenius surfaces the first failure instead. Once that failure is recorded, recovery is an explicit `GeoGenius.retry_failed/2` call that creates a new attempt. This does not change the configured `:timeout` or `:stale_after_seconds`; both keep their existing per-run meanings. A retry runs every phase from `downloading` under a new `run_id`, rather than picking up at the failed attempt's phase. What makes that cheap is the artifact cache, which skips the network. The new run owns its exact manifest, artifact selection, observations, and staging table; the failed run's manifest, observations, status, and error remain immutable evidence. ## Reading a run `GeoGenius.import/1` returns `{:ok, run_id}` once the run is claimed and enqueued. The work may be executing on another node, so both ways of reading it back go to PostgreSQL rather than to a process. ```elixir {:ok, run_id} = GeoGenius.import(collection: "demo", release: "r1") GeoGenius.status(run_id) # %GeoGenius.ImportRun{status: "normalizing", stage_metrics: %{"staged" => 1200}, ...} GeoGenius.await(run_id) # {:ok, %GeoGenius.ImportRun{status: "completed"}} ``` `status/2` returns one `%GeoGenius.ImportRun{}` snapshot, or `nil` for a run id the catalog does not carry. `await/3` polls every 250ms until the run finishes or the timeout elapses, returning `{:ok, run}`, `{:error, run}` for a run that finished and failed, or `{:error, :timeout}`. Both read the `import_run_status` view described in [`sql_api.md`](sql_api.md#views). `await/3`'s `timeout` resolves the explicit argument first, then `config :geo_genius, :await_timeout`, then a library default of 1,800,000ms (thirty minutes). A full US SimpleMaps import, which carries no boundaries, runs in well under two minutes; the thirty are sized for a boundary-carrying collection, whose geometry repair, subdivision, and indexing can still dominate a national import even with set-based writes. Pass `:infinity` at either level for a caller willing to wait as long as it takes: ```elixir GeoGenius.await(run_id, :infinity) ``` `stage_metrics` accumulates what each phase measured; `progress` is the lease's rolling progress object, updated by heartbeats during a phase. A failed run carries the reason on `error`. A terminal run holds no lease, so its `progress` reads as an empty map and its `heartbeat_at` falls back to the run's own column. `:owner` defaults to the node name and identifies who prepared the attempt. Re-enqueueing a live attempt for the same owner is safe, but it is not takeover: `claim_import_execution` latches the first executor, and every later delivery receives `occupied` and returns a no-op. A different owner is refused while the lease is live. Once an attempt fails, retry it explicitly; repeating `GeoGenius.import/1` returns the failed attempt rather than creating another, and an executor is never replaced in place. ```elixir {:ok, retry_run_id} = GeoGenius.retry_failed(failed.run_id, collection: failed.collection_key, release: failed.release_key ) ``` ## Publishing and rolling back An import produces a verified candidate. Making it visible is a separate act, and that is deliberate: you can import a vintage on Friday, look at it, and publish it on Monday without either step implying the other. ```elixir {:ok, release_id} = GeoGenius.publish(release_id) {:ok, previous_id} = GeoGenius.rollback("demo") GeoGenius.published_release("demo") ``` `publish/2` is the explicit operator path. It verifies and publishes a release only when there is no active lease and its latest import, if any, is completed; it returns `{:error, exception}` rather than raising when those guards or verification fail. Publishing the release that is already published is a no-op that leaves the rollback target untouched, so a rollback still reaches the last release actually swapped away from. `rollback/2` swaps the publication pointer back to the previous release, not to the oldest one, and returns `{:error, reason}` without clearing the publication when there is nothing to roll back to. Both are pointer swaps under MVCC: readers see one release or the other, never a partial catalog, and a published release is immutable, so changing what hosts see always means building and publishing a new release. `publish: true` on `GeoGenius.import/1` is the ingestion path for a caller that wants both in one go. Its final phase uses `publish_import`, not the operator shortcut, so completion, lease release, verification, and the publication swap are atomic. ## Enqueuing a release at boot `GeoGenius.Bootstrap` is an optional supervised child that enqueues a configured release when your application starts. GeoGenius does not add it to its own tree, because starting an application must never download or import anything on its own; you place it in yours, after your Repo, the way you place `GeoGenius.Preflight`: ```elixir children = [ MyApp.Repo, {GeoGenius.Preflight, repo: MyApp.Repo}, {GeoGenius.Bootstrap, repo: MyApp.Repo}, MyAppWeb.Endpoint ] config :geo_genius, :bootstrap, enabled: true, collection: "us_counties", release: "2026", publish: true ``` It is disabled by default: without `enabled: true` you get a no-op child. Registering a manifest is idempotent end to end, so a node booting against a release the catalog already carries re-registers the same rows rather than duplicating them, and only the first node's claim takes the lease. A node that is refused logs at error and returns `:ignore` like every other outcome, because nothing here may stop a host from booting: whatever release is already published is still readable. ## The mix tasks Five tasks drive the same public API from a shell or a deploy step. Each takes `--repo` and `--prefix`, and each resolves the Repo from your application configuration when you leave `--repo` out. `mix geo_genius.import` registers a release from its manifest and enqueues the run. Without `--await` it prints the run id and returns immediately, because the work may run elsewhere. With `--await` it waits and exits non-zero for a run that failed or did not finish in time, so a deploy step can gate on the outcome. ```console $ mix geo_genius.import --collection us_counties --release 2026 --await --publish ``` `mix geo_genius.publish` publishes a verified release, named either by id or by collection and release key. Supplying both selectors is refused rather than resolved, since the two can name different releases. A release that fails verification exits non-zero. `--timeout` bounds the publication statement, which re-runs the release's verification inside itself, and defaults to 900000 milliseconds. ```console $ mix geo_genius.publish --collection us_counties --release 2026 ``` A known limitation: `mix geo_genius.import --publish` has no equivalent switch. Its publishing phase is bounded by the window the run was claimed under, 900 seconds by default, and the task exposes no way to move it. A release whose verification needs longer is imported without `--publish` and published afterwards with `mix geo_genius.publish --timeout`. `mix geo_genius.rollback` swaps a collection back to its previous release. It changes what every host reading that collection sees, so it prints what it would do and does nothing without `--yes`. ```console $ mix geo_genius.rollback --collection us_counties --yes ``` A rollback that committed exits zero even when the release it landed on could not be read back afterwards; the unreadable outcome goes to stderr naming the rollback as done. That is deliberate. The publication has already moved by then, and a non-zero exit under a "failed" label would invite a deploy script retrying on exit code to roll the collection back a second step. `GeoGenius.rollback/2` reports the same case as `{:error, {:unread, message}}`, a shape distinct from the exception a failed rollback returns and from the string an unknown collection returns. `mix geo_genius.status` reports one run with `--run-id`, or every run for a collection newest first, plus what that collection currently publishes, with `--collection`. The two are mutually exclusive. ```console $ mix geo_genius.status --collection us_counties ``` `mix geo_genius.sweep_staging` drops per-run staging tables left behind by runs that have already finished. The pipeline drops its own table in an `after` clause, and that drop is deliberately rescued so a database failure during cleanup cannot destroy the run's failure record, which means a host whose database dies mid-cleanup leaks a table that nothing else reclaims. Like the rollback task, it prints what it would drop and does nothing without `--yes`. ```console $ mix geo_genius.sweep_staging --yes ``` The installation tasks (`mix geo_genius.setup`, `mix geo_genius.gen.migration`, `mix geo_genius.check_schema`, and `mix geo_genius.uninstall`) are covered in [`installation.md`](installation.md).