GeoGenius.Catalog (geo_genius v0.2.0)

Copy Markdown View Source

Writes and publishes the catalog: collections, authorities, area types, areas, their names and codes, releases, sources, artifacts, and import runs.

GeoGenius.Store is a behaviour because the read side names a host-repo seam as its alternative: a host may want reads routed through its own layer. Ingestion has no such alternative -- there is exactly one way to write this catalog, and it is the shipped SQL. A behaviour here would be one callback list with one implementation and no second caller, so this is a plain module instead.

Every function that wraps more than a couple of positional SQL arguments takes (context, identifier, attrs). Most of attrs's keys are named for the SQL argument they fill, but a few are named for what they mean on the Elixir side instead, because that name reads better at every call site that will use it: put_area_in_release's :attributes (SQL data), put_boundary's :geometry (SQL input_geom) and :source_release_id (SQL target_source_release_id). Candidate registration and failed-attempt retry accept :stale_after_seconds inside their claim maps (SQL stale_after, expressed in seconds rather than as an interval). This shape keeps every wrapper's arity under Credo's cap and means a caller cannot silently transpose two same-typed arguments by getting their position wrong.

Every uuid crosses this module as the hyphenated string the read layer already returns and accepts, never as Postgrex's raw sixteen-byte binary.

Summary

Functions

Advances a run to its next status, merging a metrics patch into its stage metrics.

Runs ANALYZE against the candidate owned by the current leased import attempt.

Runs ANALYZE against a release's partitions.

Attaches one immutable artifact definition to the exact release that selected it.

Attaches a source release to a release, so the release composes it.

Atomically claims the one executor allowed to run a pending import attempt.

Completes the verified current import attempt without publishing its release.

Creates a release's staging table, returning its name.

Drops a terminal or orphaned run's staging table, if it exists.

Drops the current executor's active staging table, if it exists.

Marks a run failed, storing its error detail.

Merges a progress patch into a run's active lease.

One import run, or nil when no run carries that id.

Every import run recorded for a collection, most recently started first.

Opens a release for a collection with its reviewed manifest, returning its id. Reopening the same, still-unpublished release returns the same id.

Atomically registers or diagnoses one exact manifest and its import attempt.

Publishes the candidate owned by the current leased import attempt.

Publishes a release, returning the id of the collection it publishes into.

The currently published release for a collection, or nil if none is published.

Sets one of an area's external codes, returning the code row's id.

Sets every external code in codes, returning the code rows' ids in the order they were given.

Places an area into a release with its centroid and release-scoped attributes.

Places every area in memberships into one release, in one round trip.

Sets one of an area's names, returning the name row's id.

Sets every name in names, returning the name rows' ids in the order they were given.

Records an artifact belonging to a source release, returning its id.

Attaches every boundary in boundaries within one release, in one round trip.

Attaches a boundary geometry to an area within a release, attributed to the source release it came from.

Asserts a relation between two areas within a release.

Asserts every relation in relations within one release, in one round trip.

Rebuilds a release's transitive relations, returning how many rows were written.

Records what a download actually produced, checking it against the artifact's manifest expectation.

Every artifact the release composes, with the source release each is attributed to. Reads the release_artifacts view.

The key of the collection a release belongs to, or nil for a release id the catalog does not carry.

The manifest a release was opened with, as a decoded map.

Drops the bulk data of every release beyond the newest keep, returning how many were retired.

Atomically resets one failed latest attempt to an exact replacement manifest.

Rolls a collection's publication back to its previous release, returning the collection's id on success.

The exact artifact definitions and observations for one import attempt.

Creates or updates an area within a collection, returning its id.

Creates or updates every area in areas, returning their ids in the order they were given.

Creates or updates an area type within a collection, returning its id.

Creates or updates an authority within a collection, returning its id.

Creates or updates a collection, returning its id.

Creates or updates a source within a collection, returning its id.

Creates or updates a source release, returning its id.

Checks the current leased import attempt's structural invariants.

Checks a release's structural invariants, returning a report rather than raising.

Functions

advance_import(context, run_id, executor_id, next_status, metrics_patch)

@spec advance_import(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  String.t(),
  map()
) :: :ok

Advances a run to its next status, merging a metrics patch into its stage metrics.

analyze_import(context, run_id, executor_id, opts \\ [])

@spec analyze_import(GeoGenius.Context.t(), Ecto.UUID.t(), Ecto.UUID.t(), keyword()) ::
  :ok

Runs ANALYZE against the candidate owned by the current leased import attempt.

analyze_release(context, release_id, opts \\ [])

@spec analyze_release(GeoGenius.Context.t(), Ecto.UUID.t(), keyword()) :: :ok

Runs ANALYZE against a release's partitions.

A national release analyzes tables holding hundreds of thousands of rows in one statement, so opts reaches Repo.query/3 here too.

attach_artifact(context, release_id, artifact_id)

@spec attach_artifact(GeoGenius.Context.t(), Ecto.UUID.t(), Ecto.UUID.t()) :: :ok

Attaches one immutable artifact definition to the exact release that selected it.

attach_source_release(context, release_id, source_release_id)

@spec attach_source_release(GeoGenius.Context.t(), Ecto.UUID.t(), Ecto.UUID.t()) ::
  :ok

Attaches a source release to a release, so the release composes it.

claim_import_execution(context, run_id, executor_id)

@spec claim_import_execution(GeoGenius.Context.t(), Ecto.UUID.t(), Ecto.UUID.t()) ::
  :claimed | :occupied | :completed | :failed | :missing | :superseded

Atomically claims the one executor allowed to run a pending import attempt.

complete_import(context, run_id, executor_id, metrics, opts \\ [])

@spec complete_import(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  map(),
  keyword()
) ::
  Ecto.UUID.t()

Completes the verified current import attempt without publishing its release.

create_staging(context, run_id, executor_id)

@spec create_staging(GeoGenius.Context.t(), Ecto.UUID.t(), Ecto.UUID.t()) ::
  String.t()

Creates a release's staging table, returning its name.

drop_staging(context, run_id)

@spec drop_staging(GeoGenius.Context.t(), Ecto.UUID.t()) :: :ok

Drops a terminal or orphaned run's staging table, if it exists.

drop_staging(context, run_id, executor_id)

@spec drop_staging(GeoGenius.Context.t(), Ecto.UUID.t(), Ecto.UUID.t()) :: :ok

Drops the current executor's active staging table, if it exists.

fail_import(context, run_id, executor_id, error_detail)

@spec fail_import(GeoGenius.Context.t(), Ecto.UUID.t(), Ecto.UUID.t(), map()) :: :ok

Marks a run failed, storing its error detail.

Raises 55000 for a run that has completed. Idempotent on one that already failed.

heartbeat_import(context, run_id, executor_id, progress_patch)

@spec heartbeat_import(GeoGenius.Context.t(), Ecto.UUID.t(), Ecto.UUID.t(), map()) ::
  :ok

Merges a progress patch into a run's active lease.

import_run(context, run_id)

@spec import_run(GeoGenius.Context.t(), Ecto.UUID.t()) ::
  GeoGenius.ImportRun.t() | nil

One import run, or nil when no run carries that id.

import_runs(context, collection_key)

@spec import_runs(GeoGenius.Context.t(), String.t()) :: [GeoGenius.ImportRun.t()]

Every import run recorded for a collection, most recently started first.

open_release(context, collection_key, attrs)

@spec open_release(GeoGenius.Context.t(), String.t(), map()) :: Ecto.UUID.t()

Opens a release for a collection with its reviewed manifest, returning its id. Reopening the same, still-unpublished release returns the same id.

prepare_import(context, manifest, claim)

@spec prepare_import(GeoGenius.Context.t(), map(), map()) :: map()

Atomically registers or diagnoses one exact manifest and its import attempt.

publish_import(context, run_id, executor_id, opts \\ [])

@spec publish_import(GeoGenius.Context.t(), Ecto.UUID.t(), Ecto.UUID.t(), keyword()) ::
  Ecto.UUID.t()

Publishes the candidate owned by the current leased import attempt.

publish_release(context, release_id, opts \\ [])

@spec publish_release(GeoGenius.Context.t(), Ecto.UUID.t(), keyword()) ::
  Ecto.UUID.t()

Publishes a release, returning the id of the collection it publishes into.

The SQL function re-runs verify_release as its first act, so a publication costs whatever that release's verification costs. opts reaches Repo.query/3 for the same reason it does on rebuild_relations/3, verify_release/3, and analyze_release/3: a release large enough to need more than DBConnection's fifteen-second default has no other way to ask for it.

published_release(context, collection_key)

@spec published_release(GeoGenius.Context.t(), String.t()) :: Ecto.UUID.t() | nil

The currently published release for a collection, or nil if none is published.

put_area_code(context, run_id, executor_id, area_key, attrs)

@spec put_area_code(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  String.t(),
  map()
) ::
  Ecto.UUID.t()

Sets one of an area's external codes, returning the code row's id.

put_area_code_many(context, run_id, executor_id, codes, opts \\ [])

@spec put_area_code_many(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  [map()],
  keyword()
) :: [
  Ecto.UUID.t()
]

Sets every external code in codes, returning the code rows' ids in the order they were given.

Each map carries :area_key, :code_type, and :code_value. This is put_area_code/3 over a whole batch, in one round trip.

Returns [] without touching the database for an empty list.

put_area_in_release(context, run_id, executor_id, area_key, attrs)

@spec put_area_in_release(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  String.t(),
  map()
) :: :ok

Places an area into a release with its centroid and release-scoped attributes.

put_area_in_release_many(context, run_id, executor_id, memberships, opts \\ [])

@spec put_area_in_release_many(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  [map()],
  keyword()
) :: :ok

Places every area in memberships into one release, in one round trip.

Each map carries :area_key, :centroid, and optional :attributes. Membership is last-write-wins, so an area named twice in one batch keeps the last of the two, exactly as two put_area_in_release/5 calls would leave it.

Returns :ok without touching the database for an empty list, having validated release_id first, so a malformed release id fails the way it would for a batch that had rows.

put_area_name(context, run_id, executor_id, area_key, attrs)

@spec put_area_name(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  String.t(),
  map()
) ::
  Ecto.UUID.t()

Sets one of an area's names, returning the name row's id.

put_area_name_many(context, run_id, executor_id, names, opts \\ [])

@spec put_area_name_many(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  [map()],
  keyword()
) :: [
  Ecto.UUID.t()
]

Sets every name in names, returning the name rows' ids in the order they were given.

Each map carries :area_key, :name, :kind, and an optional :locale. This is put_area_name/5 over a whole batch, in one round trip; :locale is the one field that may be nil.

Returns [] without touching the database for an empty list.

put_artifact(context, source_release_id, attrs)

@spec put_artifact(GeoGenius.Context.t(), Ecto.UUID.t(), map()) :: Ecto.UUID.t()

Records an artifact belonging to a source release, returning its id.

:expected_sha256 and :expected_bytes come from the reviewed manifest and are what record_artifact_observation/5 later checks a download against.

put_boundaries(context, run_id, executor_id, boundaries, opts \\ [])

@spec put_boundaries(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  [map()],
  keyword()
) :: :ok

Attaches every boundary in boundaries within one release, in one round trip.

Each map carries :area_key, :source_release_id, and :geometry, with optional :display_tier and :source_properties. A repeated area key keeps its last boundary, matching the state left by scalar writes in caller order.

Returns :ok without touching the database for an empty list, having validated release_id first.

put_boundary(context, run_id, executor_id, area_key, attrs)

@spec put_boundary(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  String.t(),
  map()
) :: :ok

Attaches a boundary geometry to an area within a release, attributed to the source release it came from.

A zero or omitted :simplify_tolerance uses the set-based boundary path. A nonzero tolerance retains the singular SQL path so existing callers keep its display-geometry simplification behavior.

put_relation(context, run_id, executor_id, attrs)

@spec put_relation(GeoGenius.Context.t(), Ecto.UUID.t(), Ecto.UUID.t(), map()) :: :ok

Asserts a relation between two areas within a release.

put_relation_many(context, run_id, executor_id, relations, opts \\ [])

@spec put_relation_many(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  [map()],
  keyword()
) :: :ok

Asserts every relation in relations within one release, in one round trip.

Each map carries :parent_area_key, :child_area_key, and :relation_type. A pair asserted twice in one batch keeps the last :relation_type, the way two put_relation/4 calls would leave it.

Returns :ok without touching the database for an empty list, having validated release_id first.

rebuild_relations(context, run_id, executor_id, opts \\ [])

@spec rebuild_relations(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  keyword()
) :: integer()

Rebuilds a release's transitive relations, returning how many rows were written.

One statement measures every pair in the release, so this is among the longest calls in the library. opts reaches Repo.query/3, which is how a caller raises :timeout past DBConnection's fifteen-second default for a release large enough to need it.

record_artifact_observation(context, run_id, executor_id, artifact_id, attrs)

@spec record_artifact_observation(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  map()
) :: :ok

Records what a download actually produced, checking it against the artifact's manifest expectation.

release_artifacts(context, release_id)

@spec release_artifacts(GeoGenius.Context.t(), Ecto.UUID.t()) :: [map()]

Every artifact the release composes, with the source release each is attributed to. Reads the release_artifacts view.

Each map carries string keys matching the view's columns: "release_id", "artifact_id", "source_release_id", "source_key", "source_release_key", "collection_key", "logical_name", "url", "operator_supplied", "format", "expected_sha256", "expected_bytes", "observed_sha256", "observed_bytes", "validated_at", and "metadata". The two uuid columns are cast to text. Ordered by logical_name, so a caller reading a release twice sees the same sequence: the view carries no ordering of its own, and a caller that needs one has no way to add it.

release_collection_key(context, release_id)

@spec release_collection_key(GeoGenius.Context.t(), Ecto.UUID.t()) :: String.t() | nil

The key of the collection a release belongs to, or nil for a release id the catalog does not carry.

A release id reaching the catalog from outside -- one a host stored, or one passed as a :release_id option -- says nothing about which collection it belongs to, and every release-scoped read filters on the id alone. This is how a caller checks that the id it was handed is a release of the collection it means, rather than reading another collection's rows under its own name.

release_manifest(context, release_id)

@spec release_manifest(GeoGenius.Context.t(), Ecto.UUID.t()) :: map() | nil

The manifest a release was opened with, as a decoded map.

This is the release's current declaration. Import execution instead rebuilds from import_run.manifest, preserving the exact document selected by that attempt even when a corrected retry replaces the candidate declaration.

retire_releases(context, collection_key, keep)

@spec retire_releases(GeoGenius.Context.t(), String.t(), pos_integer()) ::
  non_neg_integer()

Drops the bulk data of every release beyond the newest keep, returning how many were retired.

The release rows survive: publication_event and import_run reference them, and both are kept indefinitely. Only the partitions holding geometry, membership, and relations go.

retry_failed(context, failed_run_id, manifest, claim)

@spec retry_failed(GeoGenius.Context.t(), Ecto.UUID.t(), map(), map()) :: map()

Atomically resets one failed latest attempt to an exact replacement manifest.

rollback_publication(context, collection_key)

@spec rollback_publication(GeoGenius.Context.t(), String.t()) :: Ecto.UUID.t() | nil

Rolls a collection's publication back to its previous release, returning the collection's id on success.

Returns nil for a collection key the catalog does not carry, since a caller cannot distinguish that from "nothing to roll back" by the key alone. Raises when the collection exists but has no previous release: that is an operator error worth surfacing rather than a silent no-op.

run_artifacts(context, run_id)

@spec run_artifacts(GeoGenius.Context.t(), Ecto.UUID.t()) :: [map()]

The exact artifact definitions and observations for one import attempt.

upsert_area(context, collection_key, attrs)

@spec upsert_area(GeoGenius.Context.t(), String.t(), map()) :: Ecto.UUID.t()

Creates or updates an area within a collection, returning its id.

upsert_area_many(context, run_id, executor_id, areas, opts \\ [])

@spec upsert_area_many(
  GeoGenius.Context.t(),
  Ecto.UUID.t(),
  Ecto.UUID.t(),
  [map()],
  keyword()
) :: [
  Ecto.UUID.t()
]

Creates or updates every area in areas, returning their ids in the order they were given.

Each map carries :authority_key, :area_type_key, and :code. The three cross as parallel arrays and are written by one statement, so a batch costs one round trip rather than one per area, and upsert_area/3 is this function with a one-element list. An area named twice in one batch -- the county every city row of that county describes -- is written once and its id returned at both positions.

Returns [] without touching the database for an empty list.

upsert_area_type(context, collection_key, attrs)

@spec upsert_area_type(GeoGenius.Context.t(), String.t(), map()) :: Ecto.UUID.t()

Creates or updates an area type within a collection, returning its id.

upsert_authority(context, collection_key, attrs)

@spec upsert_authority(GeoGenius.Context.t(), String.t(), map()) :: Ecto.UUID.t()

Creates or updates an authority within a collection, returning its id.

upsert_collection(context, attrs)

@spec upsert_collection(GeoGenius.Context.t(), map()) :: Ecto.UUID.t()

Creates or updates a collection, returning its id.

upsert_source(context, collection_key, attrs)

@spec upsert_source(GeoGenius.Context.t(), String.t(), map()) :: Ecto.UUID.t()

Creates or updates a source within a collection, returning its id.

upsert_source_release(context, collection_key, attrs)

@spec upsert_source_release(GeoGenius.Context.t(), String.t(), map()) :: Ecto.UUID.t()

Creates or updates a source release, returning its id.

verify_import(context, run_id, executor_id, opts \\ [])

@spec verify_import(GeoGenius.Context.t(), Ecto.UUID.t(), Ecto.UUID.t(), keyword()) ::
  map()

Checks the current leased import attempt's structural invariants.

verify_release(context, release_id, opts \\ [])

@spec verify_release(GeoGenius.Context.t(), Ecto.UUID.t(), keyword()) :: map()

Checks a release's structural invariants, returning a report rather than raising.

Every check runs in one statement across every area and boundary in the release, so opts reaches Repo.query/3 the same way rebuild_relations/3 passes a :timeout through.