Behaviour a format adapter implements to bring one kind of source into the catalog.
A provider is called for its parsing, never for a write: stage/5 receives
a file path and an emit function and reports only whether it succeeded;
normalize/2 receives one staged row and returns a described area. Every
write is the pipeline's, made through GeoGenius.Catalog. That is the
contract this behaviour expresses -- callers should not hand a provider a
Repo or a GeoGenius.Context, and a provider that follows this
behaviour never needs one. Elixir has no module-level enforcement of that,
so it is a discipline the pipeline and the tests hold providers to, not a
guarantee this module itself can make.
emit, passed to stage/5, is called with a list of rows rather than a
single row so a provider can batch; the pipeline supplies a function that
writes through GeoGenius.Staging.insert/4 and heartbeats the run. A
provider stages a large artifact in chunks rather than one emit call for
the whole file, both to bound the size of a single insert and because each
emit call is also where the run's lease gets renewed.
stage/5 also takes an opts keyword list, which carries adapter modules
resolved by the pipeline -- [command: ..., work_dir: ...] for a provider
that shells out to convert its source first. The :command a provider is
handed is GeoGenius.Pipeline.CommandAllowlist, which accepts ogr2ogr and
refuses everything else; it reads the adapter it wraps from
:command_target, also in opts, and a provider passing opts through to
command.run/3 unchanged (as GeoGenius.Providers.Shapefile does) carries
that along without needing to know it is there. Resolving an adapter inside a
provider would make it impossible to override per call and would give a
provider a reason to read application environment, which nothing else in
this layer does. A provider that needs neither entry in opts ignores it.
normalize/2 takes no such list: it is pure, so a caller derives an area
from a staged row with no environment and no adapter to substitute.
artifacts/1, relations/1, and asserted_relations/2 have one body
most providers share, so those bodies live here as all_artifacts/1,
always_rebuild/1, and no_asserted_relations/2; a provider implements
the callback with a one-line defdelegate, keeping its own @impl and
@doc. A provider whose relations need none rebuilt, or whose rows carry
a hierarchy in their columns, implements the callback directly instead of
delegating -- GeoGenius.Providers.SimpleMaps does both.
Option vocabulary across providers
Each shipped provider names its manifest options keys for the shape its
format actually carries -- a GeoJSON feature has properties, a CSV row has
columns -- rather than a shared, format-neutral name that would read
wrong on the page for one format or the other. The concepts underneath are
the same one; this table is the one place to learn the correspondence:
| Concept | GeoGenius.Providers.GeoJSON | GeoGenius.Providers.CSV | Default when omitted |
|---|---|---|---|
| the area's code | "code_property" | "code_column" | required, no default |
| the area's area type | "area_type" (shared key) | "area_type" (shared key) | required, no default |
| the area's official name | "name_property" | "name_column" | "name" |
| alias names | "alias_properties" | "alias_columns" | [] |
| free-form attributes | "attribute_properties" | "attribute_columns" | [] |
| external codes | "code_properties" | "code_columns" | [] |
| authority | "authority" (shared key) | "authority" (shared key) | the manifest's own single authority |
| delimiter | n/a | "delimiter" | "," (CSV only) |
| parent areas the row implies | "implied_areas" (shared key) | "implied_areas" (shared key) | [] |
"implied_areas" is a list of entries, and each entry names its own code
key in the calling provider's suffix: "code_property" under
GeoGenius.Providers.GeoJSON, "code_column" under
GeoGenius.Providers.CSV. An entry also names an "area_type", a "names"
map from code to display name, an optional "authority", and an optional
"relation" defaulting to "contains". A row whose code column carries
"1" under
%{"area_type" => "cluster", "code_property" => "CLUSTER",
"names" => %{"1" => "Outer Cluster", "2" => "Inner Cluster"}}implies the area cluster:1 and asserts contains from it to the row's own
area. Both generic providers validate these entries at manifest load through
validate_options/1, so an entry written in the other's vocabulary is
rejected before anything is downloaded. See
GeoGenius.Providers.ImpliedAreas.
GeoGenius.Providers.Shapefile reads the GeoJSON vocabulary, since it
converts its archive before parsing it. GeoGenius.Providers.SimpleMaps is
outside this table: it parses two fixed files and reads their columns by
name. It requires no options, but accepts "non_census_state_area_type" so
the six USPS-only state codes can use a separately declared non-geometric
type while ordinary Census states keep state.
Every string value this vocabulary reads out of a payload -- a code, a
name, an attribute, an external code -- is trimmed before use, and a
whitespace-only value is treated the same as an absent one: see
GeoGenius.Providers.Fields.presence/1. A fixed-width source that pads a
FIPS or ZIP code with spaces would otherwise carry that padding straight
into area_key, a public, stable identifier, and a later re-import of the
same source without the padding would create a second area rather than
updating the first.
Summary
Types
The failure a provider reports for a file it cannot parse or a manifest option it cannot resolve.
The adapters stage/5 may use, resolved by the pipeline rather than read from application environment.
Callbacks
The artifacts, across a manifest's sources, this provider will stage.
Relation edges this row asserts, which geometry cannot measure.
Describes one staged row as an area, or as several.
Whether the areas this provider stages need parent_area_id relations
rebuilt after normalization.
The manifest options keys this provider requires.
Parses path -- the downloaded or operator-supplied copy of artifact --
and hands each resulting batch of rows to emit.
Validates relationships between fields of a fully decoded manifest.
Validates the manifest options map as a whole, beyond the presence of the
keys required_options/0 names.
Functions
Every artifact declared across a manifest's sources.
Always rebuilds relations; the collection carries no hierarchy of its own to preserve.
Asserts no relations; the source carries no hierarchy in its columns.
Types
@type reason() :: String.t()
The failure a provider reports for a file it cannot parse or a manifest option it cannot resolve.
The adapters stage/5 may use, resolved by the pipeline rather than read from application environment.
Callbacks
@callback artifacts(manifest :: GeoGenius.Manifest.t()) :: [ GeoGenius.Manifest.Artifact.t() ]
The artifacts, across a manifest's sources, this provider will stage.
The pipeline asks this once per source, with the manifest narrowed to that
one source, so a provider delegating to all_artifacts/1 answers with its
own source's artifacts. A release may name a different provider per source,
and each logical name is staged by the provider of the source that declared
it.
@callback asserted_relations( manifest :: GeoGenius.Manifest.t(), row :: GeoGenius.Staging.Row.t() ) :: [ {parent_area_key :: String.t(), child_area_key :: String.t(), relation_type :: String.t()} ]
Relation edges this row asserts, which geometry cannot measure.
A source that carries its hierarchy in columns -- a county FIPS on every
city row, an admin parent code on every place -- knows edges no overlap
test can derive, and a source with no geometry at all has no other way to
express one. Each edge is {parent_area_key, child_area_key, relation_type}, where relation_type is "contains",
"mostly_contains", or "overlaps".
Both keys are the strings GeoGenius.Provider.Area.key/1 composes. Build
the %GeoGenius.Provider.Area{} an edge names and take its key from there,
rather than joining the three parts by hand: an edge composed a second way
can name a key normalize/2 never produced, and GeoGenius.Catalog.put_relation/4
refuses an area the release does not carry.
Composes with relations/1 rather than replacing it: a provider whose
areas nest spatially may both rebuild measured relations and assert edges
the geometry does not carry. Edges are written after every area exists, so
an edge may reference an area any row emitted, not only its own.
Asserting an edge for a pair relations/1 also rebuilds from geometry
overwrites that measurement: GeoGenius.Catalog.put_relation/4 upserts on
the same (parent_area_id, child_area_id) pair, nulling
intersection_area_m2, parent_coverage, and child_coverage and
replacing the geometry-classified relation_type with the asserted one,
with no warning that a measurement was discarded. A provider returning
:rebuild should assert only pairs geometry cannot derive, keeping the two
sets of edges disjoint.
@callback normalize(manifest :: GeoGenius.Manifest.t(), row :: GeoGenius.Staging.Row.t()) :: {:ok, GeoGenius.Provider.Area.t()} | {:ok, [GeoGenius.Provider.Area.t()]} | :skip | {:error, reason()}
Describes one staged row as an area, or as several.
Returns :skip for a row that carries no area -- a header, a summary
record, a feature with no usable code -- rather than an error, since that
is an expected shape for some rows rather than a failure.
A list is for a source that denormalises a hierarchy into every row: a
city row that also carries its county and state describes all three, and
returning them together is cheaper and truer than staging the same row
once per level. Areas repeated across rows converge on area_key, so a
provider emits an implied parent from every row that implies it rather
than tracking which it has already emitted.
@callback relations(manifest :: GeoGenius.Manifest.t()) :: :rebuild | :none
Whether the areas this provider stages need parent_area_id relations
rebuilt after normalization.
:rebuild for a provider whose areas nest spatially or by code; :none
for a provider whose collection carries no hierarchy.
@callback required_options() :: [String.t()]
The manifest options keys this provider requires.
Manifest validation checks that every key named here is present in a
manifest's options map before the manifest is accepted.
@callback stage( manifest :: GeoGenius.Manifest.t(), artifact :: GeoGenius.Manifest.Artifact.t(), path :: Path.t(), emit :: ([GeoGenius.Staging.Row.t()] -> :ok), opts :: stage_opts() ) :: :ok | {:error, reason()}
Parses path -- the downloaded or operator-supplied copy of artifact --
and hands each resulting batch of rows to emit.
opts carries the adapters described above. Returns {:error, reason}
for a file this provider cannot parse; never raises for a malformed file,
only for a defect in the provider itself.
@callback validate_manifest(manifest :: GeoGenius.Manifest.t()) :: :ok | {:error, reason()}
Validates relationships between fields of a fully decoded manifest.
Optional. GeoGenius.Manifest calls this after its own structural checks and
validate_options/1, so a provider can enforce a policy spanning options,
area types, authorities, or sources without putting source-specific rules in
the manifest parser. Every provider named by the release or one of its
sources receives the same complete manifest.
Validates the manifest options map as a whole, beyond the presence of the
keys required_options/0 names.
Optional. A provider that exports it is called at manifest load, once every
required key is known to be present, and an error it returns becomes a
GeoGenius.ManifestError naming the field. That is what makes a malformed
option fail before a release row exists, rather than partway through
normalizing an artifact already downloaded and staged.
This is where a provider checks the options only it understands. The option
vocabulary is per-provider, so the same document can be valid under one
provider and invalid under another: GeoGenius.Providers.GeoJSON reads an
implied_areas entry's code key as "code_property" and
GeoGenius.Providers.CSV reads it as "code_column", and neither accepts
the other's.
A provider still validates what it reads at the point it reads it. A
%GeoGenius.Manifest{} built in Elixir never passes through
GeoGenius.Manifest.from_map/2, so this is a gate on loaded documents, not a
guarantee normalize/2 may rely on.
Functions
@spec all_artifacts(GeoGenius.Manifest.t()) :: [GeoGenius.Manifest.Artifact.t()]
Every artifact declared across a manifest's sources.
@spec always_rebuild(GeoGenius.Manifest.t()) :: :rebuild
Always rebuilds relations; the collection carries no hierarchy of its own to preserve.
@spec no_asserted_relations(GeoGenius.Manifest.t(), GeoGenius.Staging.Row.t()) :: []
Asserts no relations; the source carries no hierarchy in its columns.