GeoGenius.Pipeline.Normalize (geo_genius v0.2.0)

Copy Markdown View Source

The import phase that turns staged rows into catalog rows.

This is the only part of the pipeline that knows GeoGenius.Provider.Area -- what a provider may describe, and what each part of it becomes: an area, its names and external codes, its membership in the release, and its boundary.

A provider's result is checked before it reaches SQL. The four legal name kinds are atoms so a provider pattern-matches them, but Dialyzer does not catch a misspelled one: normalize/2 returns a union and names is a list, so a mis-typed success typing always overlaps the spec and invalid_contract never fires. This is therefore the only place a bad kind can be caught while the provider and the row are still known; one row further on it is a Postgrex encode or check error naming neither.

Staging is read in pages, so no connection is held across the phase, and each page is one heartbeat: a batch is well inside the lease's staleness window, so no background heartbeat process is needed -- and not having one means there is no second process whose death could be mistaken for a dead worker.

A page is written as a set, not as rows. The page is turned into areas first and written afterwards, as one statement for the batch's areas, one for its names, one for its codes and one for its memberships, through the *_many counterparts of the scalar catalog writes. A source that denormalises a hierarchy describes the same county in every city row of it, so a batch names far fewer distinct areas than it has rows, and one statement per area would spend a round trip on every repeat. Boundaries use one plural write for the page as well: GeoGenius.Catalog.put_boundaries/4 validates and repairs the geometries, replaces the areas' boundaries and subdivided parts, and recomputes their centroids from what it stored.

Collecting the whole page before writing any of it also means a provider's own errors -- an illegal name kind, an unstaged artifact -- are found before the batch is written rather than partway through it.

Summary

Functions

Streams the run's staged rows through the provider that staged each one and writes what comes back.

Functions

normalize(state)

Streams the run's staged rows through the provider that staged each one and writes what comes back.

Measures "areas", "boundaries", and "skipped".