The import phase that turns staged rows into catalog rows.
This is the only part of the pipeline that knows
GeoGenius.Provider.Area -- what a provider may describe, and what each
part of it becomes: an area, its names and external codes, its membership in
the release, and its boundary.
A provider's result is checked before it reaches SQL. The four legal
name kinds are atoms so a provider pattern-matches them, but Dialyzer does
not catch a misspelled one: normalize/2 returns a union and names is a
list, so a mis-typed success typing always overlaps the spec and
invalid_contract never fires. This is therefore the only place a bad kind
can be caught while the provider and the row are still known; one row
further on it is a Postgrex encode or check error naming neither.
Staging is read in pages, so no connection is held across the phase, and each page is one heartbeat: a batch is well inside the lease's staleness window, so no background heartbeat process is needed -- and not having one means there is no second process whose death could be mistaken for a dead worker.
A page is written as a set, not as rows. The page is turned into areas
first and written afterwards, as one statement for the batch's areas, one
for its names, one for its codes and one for its memberships, through the
*_many counterparts of the scalar catalog writes. A source that
denormalises a hierarchy describes the same county in every city row of it,
so a batch names far fewer distinct areas than it has rows, and one statement
per area would spend a round trip on every repeat. Boundaries use one plural
write for the page as well: GeoGenius.Catalog.put_boundaries/4 validates
and repairs the geometries, replaces the areas' boundaries and subdivided
parts, and recomputes their centroids from what it stored.
Collecting the whole page before writing any of it also means a provider's own errors -- an illegal name kind, an unstaged artifact -- are found before the batch is written rather than partway through it.
Summary
Functions
Streams the run's staged rows through the provider that staged each one and writes what comes back.
Functions
@spec normalize(GeoGenius.Pipeline.State.t()) :: GeoGenius.Pipeline.State.result()
Streams the run's staged rows through the provider that staged each one and writes what comes back.
Measures "areas", "boundaries", and "skipped".