v0.2.0
Dependencies
- Optional
pgflownow requires~> 0.7.0(core schema V03, helpers V07). Optional Phoenix edge raised to~> 1.8. GeoGenius ships no PgFlow schema and itsgeo_genius_importjob needs no new migration. Hosts usingGeoGenius.Runners.PgFlowupgrade their PgFlow schema first, following PgFlow's Upgrading to 0.7 and full upgrade guide. - Schema version 2 repairs the display geometry both boundary writes derive, and is the
package's first adjacent upgrade.
boundary.display_geomisST_QuantizeCoordinates(canonical, 6), and the column carries the sameST_IsValidandNOT ST_IsEmptychecks the canonical column does. Quantizing moves every coordinate by up to half a unit in the sixth decimal place, so a valid canonical geometry can quantize into an invalid one -- a ring whose three distinct points differ only past that place collapses onto a single point, and the row is rejected withviolates check constraint "boundary_display_geom_valid_chk". Because the write is a batch in thenormalizingphase, one such area failed a whole national import.put_boundaryandput_boundariesnow repair the quantized geometry the way they already repair the canonical one --ST_MakeValidthenST_CollectionExtract(..., 3)-- and keep the canonical geometry when the repair leaves no polygon at all. The check constraints are unchanged: a display geometry still has to be a valid, nonempty polygon, and now it is one. Adds theboundary_display_repaircapability. - Version 2 is a function replacement, not a table change.
v02_up.sqlandv02_down.sqlareCREATE OR REPLACE FUNCTIONover the two boundary writes plus the two marker views, so a schema at version 1 upgrades in place and keeps every row it holds, and either direction can be applied to a live schema. Hosts on version 1 must apply it: generate the pinned wrapper withmix geo_genius.gen.migration --from 1 --to 2, or render the SQL withmix geo_genius.migration_sql --prefix <prefix> --from 1 --to 2and its reverse. Upgrade before the next import.
Installation
- Postgrex
>= 0.20.0 and < 0.23.0is supported. This includes hosts locked to Postgrex 0.20 as well as the package's Postgrex 0.22 development baseline. GeoGenius.Migration.render_sql/1andmix geo_genius.migration_sqlrender a deterministic schema transition without starting a Repo or executing SQL. The task requires--prefix,--from, and--to, writes only the rendered SQL to standard output, substitutes the quoted host-selected schema, and renders the consolidated v1 install or uninstall. GeoGenius still never migrates a host automatically; the host commits and applies the rendered SQL or generated Ecto wrapper itself.- Pre-production installs use a return-and-reinstall policy. Schema version 1 describes only
the package's current schema contract; no reconciliation API, task, or compatibility edge is
published for older development snapshots that also called themselves version 1. Return the
host-owned setup migration, update GeoGenius, reinstall v1, and run
check_schema. This is a destructive pre-production policy for catalogs reproducible from reviewed manifests and checksummed artifacts, not a production migration strategy. - PgFlow
~> 0.7.0is an optional integration dependency. This establishes dependency compile order whenever a host opts into PgFlow, soGeoGenius.Runners.PgFlow.Jobis compiled afterPgFlow.Job; hosts without PgFlow still compile without any optional dependencies.
Reading
:limitacceptsnilto mean no cap onGeoGenius.areas_near/4,GeoGenius.search_areas/2andGeoGenius.Query.search_areas/2, andresult_limitaccepts an explicitNULLon theareas_nearandsearch_areasSQL functions. Both defaulted anilto 50, so a caller that narrows the result afterwards -- by a scope the catalog does not model -- had to name a number large enough to stand in for the whole set. Omitting the option, or the argument, still means 50. Reinstall the schema.Four set-keyed reads:
children_of_many/2,ancestors_of_many/2,related_areas_many/2, andareas_by_code_many/3, each taking a list where the singular beside it takes a string, and each backed by a SQL function of the same name. Resolving a list previously cost one round trip per element; a page issuing one call per city paid several thousand. The plural result for[a, b]is the singular results foraand thenb, concatenated in the array's order; a seed that matched nothing contributes no rows; an empty list returns nothing in one call; anillist, or anilelement inside one, raises. Measured over a 4,000-area release with 1,000 seeds on loopback: 445ms for 1,000 singular calls against 52ms for one plural call.%GeoGenius.SeededMatch{}pairs the seed a row came from with the ordinary%GeoGenius.AreaMatch{}, which is unchanged. The plural reads return it because one result mixes rows from many seeds.Enum.group_by(result, & &1.seed_key, & &1.match)regroups them onto the caller's list.GeoGenius.Storegains four callbacks --children_of_many/3,ancestors_of_many/3,related_areas_many/3, andareas_by_code_many/4. An out-of-tree store must implement them to keep compiling.seeded_area_matchis installed besidearea_match:seed_key textfollowed byarea_match's sixteen columns in order.Three release-scoped base views --
release_area_codes,release_area_namesandrelease_relations-- joinrelease_areas, and the fourpublished_*views are now each nothing but their base joined to the publication pointer. A pair projects identical columns in identical order, so one read shape works either side of publication.published_area_codesandpublished_area_namesgain arelease_idcolumn: their bases carry every release, sorelease_idis what tells apart the rows of an area that several releases carry, and it is the second half of any join back to an areas view. Reinstall the schema.release_idis inserted at ordinal 2 of both views, matching wherepublished_areashas carried it since the catalog was first installed, so the four columns after it shift down by one rather than the new column being appended. A host reading these views by name -- which every read in this library does -- is unaffected. A host doing positionalSELECT *,INSERT ... SELECT *, or reading result columns by index sees those four move. Until the schema is reinstalled,GeoGenius.Published.AreaCodeandGeoGenius.Published.AreaNameselect a column the installed views do not have, and every read through them fails withundefined_column.No index is shipped on
release_area.data, now as a tested decision rather than an omission. No installed function filters or orders by it, GIN cannot serve the ordered read a host actually wants there, and the keys are vendor-defined.guides/sql_api.mddocuments both index shapes a host can add, which predicate each serves, and the guarantee that an index on the partitioned parent reaches every partition a later import creates -- pinned againstcreate_release_partitionsin pgTAP.GeoGenius.Published, a view-backed queryable. Read-only Ecto schemas overpublished_areas,published_area_codes,published_area_namesandpublished_area_relations, plus composable query functions --areas/1,children_of/2,ancestors_of/2,areas_by_code/3,codes/1,names/1,relations/1-- each returning anEcto.Queryselecting whole structs. Two things it fixes for a host that joins or aggregates against the catalog: a plpgsqlSETOFfunction is an optimizer barrier, so the whole set materialises before the host's ownWHEREruns, andGeoGenius.Query's selects project five ofarea_match's sixteen columns. Every column of every view is exposed here,release_idincluded, so a host projection table keyed(release_id, area_key)is joinable at last. Set-keyed: where the function-backed API takes onearea_key, these take a list. Sources carry Ecto bindings (:area,:relation,:code,:name) for composing further. Its:release_idoption reads a release whether or not its collection publishes it, by swapping every source in the query onto the matching release-scoped base view.GeoGenius.Queryis not the join API. Its moduledoc said it was. It stays correct and unchanged for what a view has no form for::max_depthwalks and trigramsearch_areas/2. An unpublished release is served either way.
Projections
GeoGenius.ReleaseArtifactsanswers what files a release was built from and where each one is on this machine.list/2returns every artifact of a release,fetch/3one by logical name, andpath/3a local file. Resolution goes throughGeoGenius.Cachefor both downloaded and operator-supplied artifacts, so a licensed file placed by hand -- which carries acache_keyand nourl-- resolves the same way a download does.:release_idreads a release other than the published one, which is how a host fills a projection before the release goes live.GeoGenius.ArtifactErrorcarries:reason--:no_published_release,:unknown_artifact,:invalid_cache_key, or:not_cached-- and, for:not_cached, the cache key and the path the artifact was expected at, so an operator reading the message knows where to put the file.GeoGenius.Pipeline.Artifactsderives its cache keys throughGeoGenius.ReleaseArtifacts.cache_key/1, so a release is addressed the same way by the import that wrote it and the host that reads it afterwards.guides/projections.mddocuments the pattern: a host-owned table keyed(release_id, area_key)holding a source's own columns, why it is keyed that way, how to repopulate it for a new release, and how to prune rows for retired ones.
Provider contract
asserted_relations/2is a required callback. A provider returns the relation edges one staged row states outright, as{parent_area_key, child_area_key, relation_type}tuples, for the hierarchy no overlap test can derive: a county FIPS on every city row, an admin parent code on every place, or a source with no geometry at all. It composes withrelations/1rather than replacing it, and both land in the same release. An out-of-tree provider must add one line to keep compiling:@impl GeoGenius.Provider defdelegate asserted_relations(manifest, row), to: GeoGenius.Provider, as: :no_asserted_relationsnormalize/2may return a list of areas. A source that denormalises a hierarchy into every row -- a city row carrying its county and its state -- describes several areas, and returning them together is cheaper and truer than staging the same row once per level. Areas repeated across rows converge onarea_key. Returning a single{:ok, area}is unchanged.GeoGenius.Provider.Area.key/1is public. It composes an area's catalog key --authority_key,area_type_keyandcodejoined by:-- which is the same composition PostgreSQL stores inarea.area_keyand the string form both ends of an asserted edge take. A provider asserting edges composes keys through it rather than re-deriving the format.GeoGenius.Manifestenforces:authorities. The field was already required and non-empty at load; it is now in@enforce_keystoo, so a%GeoGenius.Manifest{}built in Elixir cannot default it toniland reach registration as aProtocol.UndefinedErrorinstead of an error naming the field.The
relatingphase writes asserted edges as well as rebuilding measured relations, streaming staged rows in pages and heartbeating the run's lease between them. It measuresasserted_relationsbesiderelations.
Providers
- Shapefile staging now streams a GeoJSON Feature sequence in bounded batches. The
provider runs GDAL's
ogr2ogr -f GeoJSONSeq -t_srs EPSG:4326 -lco RS=NO, then decodes one newline-delimited feature at a time instead of loading a converted FeatureCollection into memory. Plain LF-delimited and RFC 8142 record-separator sequences are accepted; the shapefile provider requires an installedogr2ogrbinary. implied_areas, a manifest option on both generic providers, lets a source row describe parent areas it names only by code in other columns, and asserts an edge from each of them to the row's own area. A source that carries a grouping code on every record describes that grouping as well as the record, and staging the same file a second time under a second manifest to reach it was the only way to get those areas before. Each entry names an"area_type", the column carrying the code --"code_property"forGeoGenius.Providers.GeoJSON,"code_column"forGeoGenius.Providers.CSV-- a"names"map from code to display name, an optional"authority", and an optional"relation"defaulting to"contains". An implied area carries no geometry and no centroid, since a grouping a source names only by code has no shape of its own there, and areas implied by several rows converge onarea_keyexactly as areas returned bynormalize/2always have. A blank code implies nothing and is not an error, because a column populated for most rows and blank for a few is ordinary; a populated code with no entry in"names"fails the release, because keying an area with no name would publish an unlabelled one.validate_options/1, an optional provider callback, is handed a manifest's wholeoptionsmap at load, once every keyrequired_options/0names is known to be present, and the error it returns becomes aGeoGenius.ManifestErrornaming the field. This is where a provider checks the options only it understands: the option vocabulary is per-provider, so the same document can be valid under one provider and invalid under another, and manifest validation cannot check a provider's own keys on its behalf. The shipped generic providers validate theirimplied_areasentries through it, so a missing"area_type", an unknown"relation", a malformed"names"map, or an entry written in the other provider's vocabulary now fails at load rather than per row, partway through normalizing an artifact already downloaded and staged.GeoGenius.Providers.Shapefiledelegates toGeoGenius.Providers.GeoJSON, since it names its options in the converted document's vocabulary. The callback is optional: a provider that exports none is unaffected. Providers still validate what they read at the point they read it, because a%GeoGenius.Manifest{}built in Elixir never passes throughGeoGenius.Manifest.from_map/2.- A source names the provider that stages it, and a source that names none inherits the
release's own
provider, so every existing manifest is unchanged. One release can now draw on sources in different formats -- a delimited file and a shapefile archive in the same release, converging onarea_key.artifacts/1is asked once per source with the manifest narrowed to that source;stage/5dispatches on the artifact;normalize/2andasserted_relations/2dispatch on the row's artifact;relations/1is the union, so relations are rebuilt when any provider asks. Manifest validation now checksrequired_options/0andvalidate_options/1for every provider a manifest names, and the missing-key message names the provider that wanted it. A source'sproviderwas previously registered and round-tripped but never read.
Breaking
GeoGenius.Pipeline.Stateno longer carries:provider. A run resolves several, so the field is replaced byproviders(the distinct modules, in source declaration order) andartifact_providers(logical name to module). This is an internal pipeline struct with no documented construction path outside the library.The
area_types/0provider callback is removed. It promised that a manifest declaring noarea_typeswould fall back to the provider's own, and nothing anywhere read it: no call site existed in the pipeline, in registration, or in any shipped provider, and the documented fallback was never implemented, so a manifest that relied on it registered no area types at all. An out-of-tree provider should delete itsarea_types/0implementation, whose@implnow warns that the behaviour specifies no such callback, and any manifest that leaned on the fallback must declare itsarea_typesitself. The sharedno_area_types/0body the shipped providers delegated to is removed with it.
Ingestion
Downloaders.Reqretries rate-limited downloads from the top. A429or503response closes the handle, removes the partial file, waits, and requests the artifact again over a fresh handle, up to:max_attemptstimes (default 5), honoring aRetry-Afterheader in seconds and otherwise doubling from one second. Req-level retry stays off because it concatenates attempts into one file. Sequential national imports against Census, which answers a burst of a few dozen requests with429, previously failed the whole run.Downloaders.Reqcarries the reviewed byte cap throughReq.Request.put_private/3. Passing the cap as aprivate:request option was rejected by Req before any bytes moved, so every required artifact failed atdownloadingwithunknown option :private. The downloader now builds the request explicitly and enforces the cap while streaming; a body that exceeds it is an error that leaves no partial file behind.Import ownership is exact-attempt and run-fenced.
prepare_import/2registers the exact manifest and returns a candidate decision;retry_failed/3is the only replacement path and creates a new run while preserving the failed attempt's manifest, artifact observations, metrics, and error. Release-owned ingestion writes now takerun_id, verify the latest lease and permitted phase, and refuse failed or superseded attempts.Executor ownership is single-flight.
claim_import_execution/2latches the first executor for a pending attempt and returnsoccupiedto duplicate deliveries. A stale heartbeat is diagnostic and never authorizes takeover; an abandoned attempt is failed explicitly and then retried as a new attempt.Import publication is atomic.
publish_import/2completes the publishing run, removes its lease, verifies the release, and swaps publication in one transaction.publish_release/1remains the operator API and refuses an active or incompletely imported release.Lost-reply and retry policy are explicit. Catalog mutations disable DBConnection checkout retries. An unresolved executor claim terminalizes with the same executor id or keeps the guardian armed.
fail_importcan claim an unclaimed pending lease in the same statement.retry_failedis idempotent against its immediate pending successor.complete_importlost replies reread completed instead of recording failure. Duplicatelogical_namevalues across sources are rejected.prepare_importno longer contains an unreachable reset branch.Six set writes:
upsert_area_many,put_area_name_many,put_area_code_many,put_area_in_release_many,put_relation_many, andput_boundaries, each taking one array per column and pairing them by position, withGeoGenius.Catalogwrappers of the same names taking a list of maps. The consolidated schema v1 includes the plural boundary write: it validates and repairs each geometry, replaces each area's boundary and subdivided parts, and recomputes its centroid in one round trip per batch.put_boundaryremains compatible; zero tolerance uses the plural path with one element, while nonzero display simplification retains the singular SQL implementation.The normalizing and relating phases write a page as a set. Normalizing collects a page of staged rows, then issues one statement for its areas, one for its names, one for its codes, one for its memberships, and one for its boundaries; relating issues one for a page's asserted edges. Measured on the US SimpleMaps import (150,622 staged rows, 153,917 areas, 466,262 area writes, 330,297 asserted edges), against the same data on the same machine:
Phase Before After normalizing 994.8 s, 1,755,328 statements 39.6 s, 2,449 statements relating 162.7 s, 331,751 statements 20.4 s, 1,254 statements whole run 1183.5 s 85.5 s The catalog the two runs left is identical, row for row.
A batch that fails leaves none of itself written. A page is collected before any of it is written, so a provider returning an illegal name kind, or a staged row naming an artifact the run did not stage, fails the phase with nothing of that page written rather than with the areas ahead of the failure already in the catalog. Earlier pages stay committed during that attempt; a failed attempt is preserved and retried under a new run id.
An unresolved key now names itself. The plural writes resolve keys by joining, and report an unresolved one through the new
assert_resolvedguard, which raises the sameP0002thatSELECT ... INTO STRICTraised but appends the key:query returned no rows for authority key acme. Because the scalar writes delegate, they carry the named message too.put_boundaryandassert_release_mutablestill raise the bare message.assert_write_arraysrefuses a malformed batch. Arrays that disagree in length raise22023, and a null element in a column that requires one raises22004.unnestpads a short array with nulls rather than failing, so without this a batch assembled from two sources of different lengths would write rows the caller never described. The columns that accept a null element areput_area_name_many'slocalesandput_area_in_release_many'scentroidsanddata.One lock order for the whole write path, through two new helpers:
area_lock_key, the single definition of what an area serializes on, andlock_areas, which takes a batch's locks ascending by that key before the batch touches any row. Bothupsert_area_manyandput_area_name_manygo through it, and both walk theirarearows inarea_keyorder, so the two agree at both levels. A plural write locks many rows per statement where a scalar locks one, and import leases are scoped to a release, so two releases of one collection normalize at once and the normalizing phase issues an area upsert and a name write for every page: two functions each sorting their own way deadlock over any pair of areas they share.put_area_code_many,put_area_in_release_manyandput_relation_manytake no advisory lock -- they reachareaonly through foreign keys, which takeFOR KEY SHAREand cannot conflict with theFOR NO KEY UPDATEthe other two take -- and each orders its own insert on its own conflict key. A batch holds one advisory lock per distinct area for the length of its statement, so the batch size a caller chooses bounds its share ofmax_locks_per_transaction.Repeats inside one batch are deduplicated on the constraint key, since
ON CONFLICT DO UPDATEraises21000when one statement presents the same conflict key twice and a denormalised source repeats the same county in every city row. Where a write is last-write-wins -- release membership and relations -- the last occurrence in the arrays wins, matching what a loop of scalar calls in that order would have left.GeoGenius.await/3's default timeout is now 1,800,000ms (thirty minutes), up from 300,000ms (five minutes) -- this changes default behaviour for existing callers. The previous default did not cover the library's own flagship workload: a full US SimpleMaps import measured at ~17 minutes (1015 seconds), so any caller awaiting one with no explicittimeoutfailed on the default alone. The set writes above have since brought that same import under two minutes, but the default stays where it is: a boundary-carrying collection still spends significant time repairing, subdividing, and indexing geometry even though boundaries are written once per batch, which is the shape the thirty minutes are for. Resolution order is now thetimeoutargument, thenconfig :geo_genius, :await_timeout, then the 1,800,000ms library default, via the newawait_timeout/1resolver in the library's internal config module -- the same per-call-opts-then-app-env-then-default patternmanifest_paths/1and the adapter resolvers already use.:infinityremains valid at every level. A caller that already passes an explicittimeoutis unaffected.mix geo_genius.import --awaitparticipates in that resolution. The task carried a300_000default of its own and passed it toawait/3as an explicit argument, which wins the order above, soconfig :geo_genius, :await_timeoutcould not reach the library's own CLI and it still gave up after five minutes.--timeoutnow parses to nil when absent, and the value the task reports on a timeout is the one the wait used.A staging pass empties the run's table before it writes to it.
create_stagingisCREATE UNLOGGED TABLE IF NOT EXISTS, and nothing truncated, so an attempt that died whereGeoGenius.Pipeline's cleanup could not run -- a killed VM, a lost machine -- left its rows staged under the same run id and the next attempt appended to them. A measured import reported 150,622 staged rows into a table holding 301,244. Doubling the work is the mild half; the other half is that the stale rows are normalized too, so a row deleted from the source between attempts is resurrected into the release. The newGeoGenius.Staging.reset/3drops and recreates the table, and the staging phase calls it in place ofcreate/3. No SQL changed:create_stagingstill keeps what it finds, and a host driving staging through the SQL API callsdrop_stagingbefore it.Every
authoritiesandarea_typesentry is validated field by field. An entry naming nokey, an authority naming noname, or an area type whoserankis not a positive integer is refused byGeoGenius.Manifest.load/3andfrom_map/2with a message naming both the field and the list it came from. Such an entry passed validation and failed several phases later as aGeoGenius.CatalogErrorwrapping a NOT NULL violation or a failed cast insideupsert_authority/upsert_area_type. A manifest carrying one now fails at load rather than partway through an import.
SimpleMaps provider
GeoGenius.Providers.SimpleMaps("simplemaps") parses the SimpleMaps US cities and US ZIP codes datasets into states, counties, cities and ZIPs, ranked 10/20/30/40. Both files denormalise the hierarchy into every row, so counties and states are read out of columns rather than off rows of their own, and the whole hierarchy is asserted from the FIPS columns -- the data carries centroids and no boundaries, sorelations/1is:none. It requires no manifestoptions; the artifact'slogical_name,uscitiesoruszips, selects the parser.- Three authorities. Cities key under
simplemaps, counties and most states undercensus, and ZIPs underusps. Six state codes key underuspsas well:AA,AEandAPare USPS military-mail constructs andFM,PWandMHare the Freely Associated States, and the Census assigns none of the six an ANSI code. A host looking a state up by code queries bothansi_stateandusps_state, or misses those six. - A county hangs under the state its own FIPS names. A five-digit county FIPS begins
with the two-digit FIPS of the state that assigns it, and that prefix -- not the row's
state_idcolumn -- is whatasserted_relations/2reads a county's state parent from. The two disagree wherever a mailing address crosses a state line: ZIP20041carriesstate_idDC andcounty_fips51107, which is Loudoun County, Virginia, and 150 of the source's 3,233 counties are named on rows of more than one state. A row whose county lies in another state also yields that state as an area of its own, sinceput_relationrequires both ends of an edge to be members of the release. A county whose prefix names no stateGeoGenius.Providers.SimpleMaps.Fipscarries gets no state parent at all rather than a false one. GeoGenius.Providers.SimpleMaps.Fipsstates the state FIPS to postal code table the above reads: the fifty states, DC, and the five inhabited territories the Census assigns a state-level FIPS to. Neither source file carries the pair and no single row can establish it, so it is stated in the provider, where the rest of this vendor's US knowledge already lives.- Row-scoped county validation. A row whose
county_fips,county_fips_all, county-name orcounty_weightscolumns contradict each other fails the release, naming the column and the row. Cross-file agreement is deliberately not checked: it cannot be seen from a single row. Auscitiesrow naming no county fails; auszipsrow naming none hangs its ZIP off its state directly. - A shipped manifest,
us_simplemaps, declaring the collection, the three authorities, the four ranked area types and both artifacts as operator-supplied cache keys. The files are licensed downloads, so an operator places their copies in the cache and replaces each artifact'ssha256andbyteswith those of the copy they licensed.
v0.1.0
Initial release: a versioned catalog of named geographic areas, installed into a host-selected PostgreSQL schema through EctoEvolver.
- Catalog identity — collections, authorities, area types (ranked), and areas with a
stable, human-readable
area_key. Areas carry names (official, alias, mailing, abbreviation), external codes, and an optionalretired_atwith asuccessor_id. - Releases and membership — versioned releases with sources, source releases, and
checksummed artifacts.
put_area_in_release/put_boundaryrecordrelease_areamembership rows; geometry is an optional attachment to that membership, not a condition of it. A collection can setrequires_geometryto restore the strict boundary check for release verification. - Boundaries and relations — partitioned
boundary/boundary_parttables (subdivided index geometry for spatial joins), plusrelationrows that are either measured from geometry (rebuild_relations, populatingintersection_area_m2/parent_coverage/child_coverage) or asserted from source data (put_relation, leaving those columnsNULL). Both traverse identically. - Publication lifecycle —
verify_release,publish_release,rollback_publication, andretire_releases, guarded by transaction-level advisory locks and a constraint trigger that only allows a completed release to be published. Every publish or rollback appends a row to the durablepublication_eventtable. - Reads —
published_*views that resolve through the publication pointer, so a release swap changes every view's results atomically under MVCC with no refresh step.published_areasexposes retired areas (withretired_at) rather than hiding them. - Resolution —
areas_for_point,areas_for_geometry,areas_near,areas_by_code(scoped to a parent area's descendants viaparent_area_key/parent_max_depth), andsearch_areas(trigram-ranked name search), pluschildren_of,ancestors_of, andrelated_areasfor hierarchy traversal. Theresolvecascade tries containment, code, name, and proximity in order against a single JSON input and returns the first strategy's matches, withparent_area_keyscoping the code and name strategies. - Import runs — a durable exact-attempt state machine (
prepare_import,retry_failed,claim_import_execution,heartbeat_import,advance_import,fail_import) with one executor per attempt and explicit retry. A stale run is never taken over or rewritten; its manifest, artifact observations, metrics, and error remain evidence while a retry gets a new run id.completedandfailedare terminal, so a late error cannot overwrite history. - Host integration —
GeoGenius.Preflightfor a startup-time prerequisite check (extensions, geometry and jsonb decoding, and schema version), andmix geo_genius.setup,mix geo_genius.gen.migration,mix geo_genius.check_schema, andmix geo_genius.uninstallfor install, upgrade, verification, and (opt-in,--yes-gated) teardown. - Elixir read layer —
GeoGeniuswraps all ten reads in%GeoGenius.AreaMatch{}structs, andGeoGenius.Querycomposes a catalog read into a host's own Ecto query for a single-round-trip join. Requiresgeo_postgis: geometry crosses the boundary asGeostructs (centroidas%Geo.Point{},areas_for_geometry's argument as%Geo.Polygon{}/%Geo.MultiPolygon{}), never as WKT/EWKT text, and the host Repo must be configured with a Postgrex types module registeringGeo.PostGIS.Extension. - Manifests — a reviewed JSON document per release, describing the collection, the
provider, the authorities its areas are keyed under, the ranked area types, every source
with its license and attribution, and every artifact with its URL or operator-supplied
cache key, expected SHA-256, expected byte count, archive members, and whether it is
required.
GeoGenius.Manifestvalidates every field and names the one that was wrong;to_map/1round-trips, so the manifest stored on the release row rebuilds without touching the filesystem. Manifests are searched for underconfig :geo_genius, :manifest_paths, with the package's own directory last so a host-shipped correction wins.authoritiesis a required, non-empty list, because a collection may draw on more than one — a US release keyed partly by the Census, partly by the USPS, and partly by a vendor's own identifiers names all three — and every authority its areas are keyed under is registered when the release is opened. - Artifact acquisition — a cache checked before any download, so a rerun costs
nothing and an operator-supplied file is the same lookup as a fetched one. Cache hits
are hashed on every run, and
record_artifact_observationin PostgreSQL is the only place an expectation meets an observation. - Providers —
GeoGenius.Provider(required_options/0,validate_options/1,validate_manifest/1,artifacts/1,stage/5,normalize/2,relations/1, andasserted_relations/2) with four implementations:Providers.GeoJSON,Providers.CSV,Providers.Shapefile, andProviders.SimpleMaps; the shapefile provider converts its archive throughogr2ogr. A host registers its own underconfig :geo_genius, :providers. - Adapters —
GeoGenius.Cache(Caches.FileSystem),GeoGenius.Downloader(Downloaders.Req, requiring the optionalreqdependency),GeoGenius.Command(Commands.System, narrowed toogr2ogrbyPipeline.CommandAllowlistbefore a provider sees it), andGeoGenius.Notifier(Notifiers.Noop), each resolved from options, then application environment, then the shipped default. A notifier cannot fail an import. - The import pipeline —
GeoGenius.Pipelinewalks a claimed run through downloading, validating, staging, normalizing, relating, indexing, and verifying, advancing the durable run row at each boundary and heartbeating its lease from inside long phases. It opens no transaction around the full run, drops its staging table on success and failure alike, records every failure durably, and usespublish_importfor atomic completion and publication. - Runner backends —
GeoGenius.RunnerwithRunners.Task(the zero-configuration default, running each import under a supervisedTask.Supervisor),Runners.Inline(the calling process), andRunners.PgFlow(available only to a host that has installedpgflowitself). The backend a run was claimed under is recorded onimport_run.runner_backendand resolves back to a module throughGeoGenius.Runner.module_for_backend/1. Nothing in the library reads it back: run cancellation, the feature that would have, is not shipped, and the column stands as the durable record of which backend claimed a run. - Elixir ingestion API —
GeoGenius.import/1,retry_failed/2,status/2,await/3,publish/2,rollback/2, andpublished_release/2. Registration is idempotent end to end, failed replacement is explicit, and importing never publishes unless asked. - Operational mix tasks —
mix geo_genius.import,mix geo_genius.publish,mix geo_genius.rollback,mix geo_genius.status, andmix geo_genius.sweep_staging. The two that change what hosts see, and the one that drops tables, do nothing without--yes. - Boot-time enqueue —
GeoGenius.Bootstrap, an optional child a host places in its own supervision tree, disabled unlessconfig :geo_genius, :bootstrapsetsenabled: true. - New SQL —
prepare_import,retry_failed,claim_import_execution,publish_import,upsert_source,upsert_source_release,put_artifact,record_artifact_observation,open_release,attach_source_release,staging_table_name,create_staging,insert_staging_many,drop_staging,analyze_release,analyze_import, andpublished_release, plus theimport_run_status,run_artifacts, andrelease_artifactsviews.
Registration
GeoGenius.Registration.prepare_import/3andretry_failed/4translate the catalog's closed candidate decisions into the public import lifecycle. Registration and attempt creation are one transaction; retry targets one failed latest run explicitly rather than reopening it.
Deliberate deviations from the specification
GeoGenius.Providers.Shapefileships in place ofProviders.Census. Nothing in it is Census-specific: it unpacks an archive, converts it withogr2ogr, and parses the result. A Census collection is a manifest, not a module, and naming the provider after one publisher would have implied a second module was needed for the next one.GeoGenius.Runners.Inlineships in place ofRunners.Test. It is not a test double. It is the backend a host with no PgFlow and noTask.Supervisoractually runs imports under, and a name saying otherwise would have steered hosts away from the only backend available to them.Runners.Obanis not shipped;Runners.PgFlowis the only durable backend. TheGeoGenius.Runnerbehaviour is the extension point for exactly this case: a host that wants an Oban-backed runner implementsname/0,available?/0, andenqueue/3itself, which is the reason the behaviour exists rather than a fixed backend list.- The library declares a
mod:application callback, where the specification says it starts no supervision tree of its own. The tree holds at most one child, aTask.Supervisorregistered asGeoGenius.TaskSupervisor, and starts nothing at all whenconfig :geo_genius, :task_supervisoris already set at boot. Without it,Runners.Taskrequired a host to both configure a supervisor and start it, and configuring only the first selected the Task backend and then failed to enqueue; the practical default was thereforeRunners.Inline, which blocks the caller for the length of an import. ATask.Supervisorcarries none of the boot-order risk that rule exists to avoid: it needs nothing to start and does no work until a host callsGeoGenius.import/1. Its shutdown is the caveat.:geo_geniusstops after its host, so the library's own supervisor is stopped only once the host's Repo has already gone, and an import still running then keeps running against a Repo that is no longer there. A host that cares about shutdown ordering setsconfig :geo_genius, :task_supervisorto a supervisor it places after its own Repo, which this library cannot arrange from the outside.GeoGenius.Preflightdeliberately does not join this tree. - The provider callback arities are
stage/5,normalize/2, andrelations/1, where the specification writesstage/2andnormalize/1. A provider needs the manifest on every call, andstageneeds an emit function to hand batches to and an adapter list to shell out through. The six callbacks and their responsibilities are otherwise unchanged. This is the deviation that matters most to a reader, because every host writing its own provider implements these exact signatures. Runners.Taskresolves its supervisor from two tiers, not three, andRunners.Task.available?/1does not exist. The specification's per-calloptstier is gone: resolution readsconfig :geo_genius, :task_supervisorand falls back toGeoGenius.TaskSupervisor, and the behaviour callback isavailable?/0. A per-call supervisor would have let one import run under a supervisor a later import knew nothing about, while the durable backend it competes with,Runners.PgFlow, has no per-call equivalent to offer. The visible consequence is inGeoGenius.Runner.configured/1:Runners.Taskis selected whenever the config key names anything at all, live or not, so a typo'd or not-yet-started supervisor surfaces as a namedenqueue/3error instead of falling through toRunners.Inlineand blocking the caller for a whole import.