GeoGenius.Provider behaviour (geo_genius v0.2.0)

Copy Markdown View Source

Behaviour a format adapter implements to bring one kind of source into the catalog.

A provider is called for its parsing, never for a write: stage/5 receives a file path and an emit function and reports only whether it succeeded; normalize/2 receives one staged row and returns a described area. Every write is the pipeline's, made through GeoGenius.Catalog. That is the contract this behaviour expresses -- callers should not hand a provider a Repo or a GeoGenius.Context, and a provider that follows this behaviour never needs one. Elixir has no module-level enforcement of that, so it is a discipline the pipeline and the tests hold providers to, not a guarantee this module itself can make.

emit, passed to stage/5, is called with a list of rows rather than a single row so a provider can batch; the pipeline supplies a function that writes through GeoGenius.Staging.insert/4 and heartbeats the run. A provider stages a large artifact in chunks rather than one emit call for the whole file, both to bound the size of a single insert and because each emit call is also where the run's lease gets renewed.

stage/5 also takes an opts keyword list, which carries adapter modules resolved by the pipeline -- [command: ..., work_dir: ...] for a provider that shells out to convert its source first. The :command a provider is handed is GeoGenius.Pipeline.CommandAllowlist, which accepts ogr2ogr and refuses everything else; it reads the adapter it wraps from :command_target, also in opts, and a provider passing opts through to command.run/3 unchanged (as GeoGenius.Providers.Shapefile does) carries that along without needing to know it is there. Resolving an adapter inside a provider would make it impossible to override per call and would give a provider a reason to read application environment, which nothing else in this layer does. A provider that needs neither entry in opts ignores it. normalize/2 takes no such list: it is pure, so a caller derives an area from a staged row with no environment and no adapter to substitute.

artifacts/1, relations/1, and asserted_relations/2 have one body most providers share, so those bodies live here as all_artifacts/1, always_rebuild/1, and no_asserted_relations/2; a provider implements the callback with a one-line defdelegate, keeping its own @impl and @doc. A provider whose relations need none rebuilt, or whose rows carry a hierarchy in their columns, implements the callback directly instead of delegating -- GeoGenius.Providers.SimpleMaps does both.

Option vocabulary across providers

Each shipped provider names its manifest options keys for the shape its format actually carries -- a GeoJSON feature has properties, a CSV row has columns -- rather than a shared, format-neutral name that would read wrong on the page for one format or the other. The concepts underneath are the same one; this table is the one place to learn the correspondence:

ConceptGeoGenius.Providers.GeoJSONGeoGenius.Providers.CSVDefault when omitted
the area's code"code_property""code_column"required, no default
the area's area type"area_type" (shared key)"area_type" (shared key)required, no default
the area's official name"name_property""name_column""name"
alias names"alias_properties""alias_columns"[]
free-form attributes"attribute_properties""attribute_columns"[]
external codes"code_properties""code_columns"[]
authority"authority" (shared key)"authority" (shared key)the manifest's own single authority
delimitern/a"delimiter""," (CSV only)
parent areas the row implies"implied_areas" (shared key)"implied_areas" (shared key)[]

"implied_areas" is a list of entries, and each entry names its own code key in the calling provider's suffix: "code_property" under GeoGenius.Providers.GeoJSON, "code_column" under GeoGenius.Providers.CSV. An entry also names an "area_type", a "names" map from code to display name, an optional "authority", and an optional "relation" defaulting to "contains". A row whose code column carries "1" under

%{"area_type" => "cluster", "code_property" => "CLUSTER",
  "names" => %{"1" => "Outer Cluster", "2" => "Inner Cluster"}}

implies the area cluster:1 and asserts contains from it to the row's own area. Both generic providers validate these entries at manifest load through validate_options/1, so an entry written in the other's vocabulary is rejected before anything is downloaded. See GeoGenius.Providers.ImpliedAreas.

GeoGenius.Providers.Shapefile reads the GeoJSON vocabulary, since it converts its archive before parsing it. GeoGenius.Providers.SimpleMaps is outside this table: it parses two fixed files and reads their columns by name. It requires no options, but accepts "non_census_state_area_type" so the six USPS-only state codes can use a separately declared non-geometric type while ordinary Census states keep state.

Every string value this vocabulary reads out of a payload -- a code, a name, an attribute, an external code -- is trimmed before use, and a whitespace-only value is treated the same as an absent one: see GeoGenius.Providers.Fields.presence/1. A fixed-width source that pads a FIPS or ZIP code with spaces would otherwise carry that padding straight into area_key, a public, stable identifier, and a later re-import of the same source without the padding would create a second area rather than updating the first.

Summary

Types

The failure a provider reports for a file it cannot parse or a manifest option it cannot resolve.

The adapters stage/5 may use, resolved by the pipeline rather than read from application environment.

Callbacks

The artifacts, across a manifest's sources, this provider will stage.

Relation edges this row asserts, which geometry cannot measure.

Describes one staged row as an area, or as several.

Whether the areas this provider stages need parent_area_id relations rebuilt after normalization.

The manifest options keys this provider requires.

Parses path -- the downloaded or operator-supplied copy of artifact -- and hands each resulting batch of rows to emit.

Validates relationships between fields of a fully decoded manifest.

Validates the manifest options map as a whole, beyond the presence of the keys required_options/0 names.

Functions

Every artifact declared across a manifest's sources.

Always rebuilds relations; the collection carries no hierarchy of its own to preserve.

Asserts no relations; the source carries no hierarchy in its columns.

Types

reason()

@type reason() :: String.t()

The failure a provider reports for a file it cannot parse or a manifest option it cannot resolve.

stage_opts()

@type stage_opts() :: [
  command: module(),
  command_target: module(),
  work_dir: Path.t()
]

The adapters stage/5 may use, resolved by the pipeline rather than read from application environment.

Callbacks

artifacts(manifest)

@callback artifacts(manifest :: GeoGenius.Manifest.t()) :: [
  GeoGenius.Manifest.Artifact.t()
]

The artifacts, across a manifest's sources, this provider will stage.

The pipeline asks this once per source, with the manifest narrowed to that one source, so a provider delegating to all_artifacts/1 answers with its own source's artifacts. A release may name a different provider per source, and each logical name is staged by the provider of the source that declared it.

asserted_relations(manifest, row)

@callback asserted_relations(
  manifest :: GeoGenius.Manifest.t(),
  row :: GeoGenius.Staging.Row.t()
) :: [
  {parent_area_key :: String.t(), child_area_key :: String.t(),
   relation_type :: String.t()}
]

Relation edges this row asserts, which geometry cannot measure.

A source that carries its hierarchy in columns -- a county FIPS on every city row, an admin parent code on every place -- knows edges no overlap test can derive, and a source with no geometry at all has no other way to express one. Each edge is {parent_area_key, child_area_key, relation_type}, where relation_type is "contains", "mostly_contains", or "overlaps".

Both keys are the strings GeoGenius.Provider.Area.key/1 composes. Build the %GeoGenius.Provider.Area{} an edge names and take its key from there, rather than joining the three parts by hand: an edge composed a second way can name a key normalize/2 never produced, and GeoGenius.Catalog.put_relation/4 refuses an area the release does not carry.

Composes with relations/1 rather than replacing it: a provider whose areas nest spatially may both rebuild measured relations and assert edges the geometry does not carry. Edges are written after every area exists, so an edge may reference an area any row emitted, not only its own.

Asserting an edge for a pair relations/1 also rebuilds from geometry overwrites that measurement: GeoGenius.Catalog.put_relation/4 upserts on the same (parent_area_id, child_area_id) pair, nulling intersection_area_m2, parent_coverage, and child_coverage and replacing the geometry-classified relation_type with the asserted one, with no warning that a measurement was discarded. A provider returning :rebuild should assert only pairs geometry cannot derive, keeping the two sets of edges disjoint.

normalize(manifest, row)

@callback normalize(manifest :: GeoGenius.Manifest.t(), row :: GeoGenius.Staging.Row.t()) ::
  {:ok, GeoGenius.Provider.Area.t()}
  | {:ok, [GeoGenius.Provider.Area.t()]}
  | :skip
  | {:error, reason()}

Describes one staged row as an area, or as several.

Returns :skip for a row that carries no area -- a header, a summary record, a feature with no usable code -- rather than an error, since that is an expected shape for some rows rather than a failure.

A list is for a source that denormalises a hierarchy into every row: a city row that also carries its county and state describes all three, and returning them together is cheaper and truer than staging the same row once per level. Areas repeated across rows converge on area_key, so a provider emits an implied parent from every row that implies it rather than tracking which it has already emitted.

relations(manifest)

@callback relations(manifest :: GeoGenius.Manifest.t()) :: :rebuild | :none

Whether the areas this provider stages need parent_area_id relations rebuilt after normalization.

:rebuild for a provider whose areas nest spatially or by code; :none for a provider whose collection carries no hierarchy.

required_options()

@callback required_options() :: [String.t()]

The manifest options keys this provider requires.

Manifest validation checks that every key named here is present in a manifest's options map before the manifest is accepted.

stage(manifest, artifact, path, emit, opts)

@callback stage(
  manifest :: GeoGenius.Manifest.t(),
  artifact :: GeoGenius.Manifest.Artifact.t(),
  path :: Path.t(),
  emit :: ([GeoGenius.Staging.Row.t()] -> :ok),
  opts :: stage_opts()
) :: :ok | {:error, reason()}

Parses path -- the downloaded or operator-supplied copy of artifact -- and hands each resulting batch of rows to emit.

opts carries the adapters described above. Returns {:error, reason} for a file this provider cannot parse; never raises for a malformed file, only for a defect in the provider itself.

validate_manifest(manifest)

(optional)
@callback validate_manifest(manifest :: GeoGenius.Manifest.t()) ::
  :ok | {:error, reason()}

Validates relationships between fields of a fully decoded manifest.

Optional. GeoGenius.Manifest calls this after its own structural checks and validate_options/1, so a provider can enforce a policy spanning options, area types, authorities, or sources without putting source-specific rules in the manifest parser. Every provider named by the release or one of its sources receives the same complete manifest.

validate_options(options)

(optional)
@callback validate_options(options :: map()) :: :ok | {:error, reason()}

Validates the manifest options map as a whole, beyond the presence of the keys required_options/0 names.

Optional. A provider that exports it is called at manifest load, once every required key is known to be present, and an error it returns becomes a GeoGenius.ManifestError naming the field. That is what makes a malformed option fail before a release row exists, rather than partway through normalizing an artifact already downloaded and staged.

This is where a provider checks the options only it understands. The option vocabulary is per-provider, so the same document can be valid under one provider and invalid under another: GeoGenius.Providers.GeoJSON reads an implied_areas entry's code key as "code_property" and GeoGenius.Providers.CSV reads it as "code_column", and neither accepts the other's.

A provider still validates what it reads at the point it reads it. A %GeoGenius.Manifest{} built in Elixir never passes through GeoGenius.Manifest.from_map/2, so this is a gate on loaded documents, not a guarantee normalize/2 may rely on.

Functions

all_artifacts(manifest)

@spec all_artifacts(GeoGenius.Manifest.t()) :: [GeoGenius.Manifest.Artifact.t()]

Every artifact declared across a manifest's sources.

always_rebuild(manifest)

@spec always_rebuild(GeoGenius.Manifest.t()) :: :rebuild

Always rebuilds relations; the collection carries no hierarchy of its own to preserve.

no_asserted_relations(manifest, row)

@spec no_asserted_relations(GeoGenius.Manifest.t(), GeoGenius.Staging.Row.t()) :: []

Asserts no relations; the source carries no hierarchy in its columns.