GeoGenius.Providers.Shapefile (geo_genius v0.2.0)

Copy Markdown View Source

Provider for a shapefile archive.

A shapefile is several sibling files (.shp, .dbf, .shx, and usually .prj) that must travel together, so the artifact this provider stages is a zip archive rather than a single file. stage/5:

  1. Unzips the archive into a private subdirectory of opts[:work_dir] with :zip.unzip/2 from OTP -- no dependency, since the archive format is fixed.
  2. Finds exactly one .shp member. Zero is an error naming the archive; more than one is an error naming both, because picking one silently would publish half a dataset.
  3. Converts it with the configured GeoGenius.Command adapter: ogr2ogr -f GeoJSONSeq -t_srs EPSG:4326 -lco RS=NO <out.geojsonl> <in.shp>. The -t_srs flag is not optional: a shapefile in a projected coordinate system would otherwise stage coordinates in metres that PostGIS accepts as degrees -- geometry that is silently, plausibly wrong rather than obviously broken.
  4. Delegates the parse of the converted sequence to the GeoJSON Sequence provider's staging implementation.

normalize/2, required_options/0, artifacts/1, relations/1, and asserted_relations/2 all delegate to GeoGenius.Providers.GeoJSON unchanged: a converted shapefile is a sequence of GeoJSON Feature records by the time anything downstream of ogr2ogr sees it, so the manifest options a release names are the same ones GeoGenius.Providers.GeoJSON reads.

This is the only shipped provider that needs GDAL installed -- a host publishing GeoJSON or CSV collections never installs it. available?/2 on the configured GeoGenius.Command is checked before ogr2ogr runs, so a host missing GDAL gets a named error rather than a raised exception or a bare non-zero exit.

This module ships in place of the module name Providers.Census used elsewhere: what a Census-specific provider would actually contain is this format reader plus a vendor's file-naming and code conventions, and the second half is vendor knowledge a host-neutral catalog library has no business carrying. Shipping the format half means a Census release is an ordinary manifest naming "shapefile" with Census column names in its options, and a release from any other authority that ships shapefiles is another one.

Working directory. opts[:work_dir] is resolved once per pipeline run and may be shared across every artifact in a manifest -- a downloaded archive can already be sitting in it, and another shapefile artifact in the same release may extract into it around the same time. This provider never removes work_dir itself: it creates its own uniquely-named subdirectory under it (System.unique_integer/1-suffixed, the shape used elsewhere in this codebase) to unzip and convert into, and removes only that subdirectory when the call returns. stage/5 is eager -- emit has already run by the time this function returns -- so nothing outside the call needs the extracted or converted files afterward, and cleanup runs on both the success and error paths. Cleanup does not run if the process is killed by an exit signal rather than returning or raising; the default work_dir lives under System.tmp_dir!/0, so the OS reaps it regardless.

Member matching is case-sensitive and unfiltered by path. TERRITORIES.SHP in an archive is read as "no .shp member", and an archive zipped by macOS that carries __MACOSX/._territories.shp alongside the real member is read as "more than one .shp member". Both fail loudly with a named error rather than silently choosing wrong, so this is a limit to know about, not a defect to route around silently.

Streaming conversion. -f GeoJSONSeq -lco RS=NO writes one Feature per newline. The GeoJSON Sequence provider reads and emits the resulting sequence in bounded batches, so this provider never retains the converted dataset as a whole document.

Summary

Functions

Delegates to GeoGenius.Providers.GeoJSON.artifacts/1: every artifact declared across the manifest's sources, all of them shapefile archives.

Delegates to GeoGenius.Providers.GeoJSON.asserted_relations/2: a converted shapefile carries no hierarchy in its columns beyond what its geometry already expresses.

Delegates to GeoGenius.Providers.GeoJSON.normalize/2: a converted shapefile's staged rows carry the same shape as a GeoJSON feature's.

Delegates to GeoGenius.Providers.GeoJSON.relations/1: a converted shapefile's areas carry no hierarchy of their own; relations are always rebuilt.

Delegates to GeoGenius.Providers.GeoJSON.required_options/0: the manifest options a shapefile release names are read from the converted GeoJSON document.

Unzips path into a private subdirectory of opts[:work_dir], converts its single .shp member to GeoJSON Sequence with ogr2ogr, and delegates the parse to the GeoJSON Sequence provider's staging implementation.

Delegates to GeoGenius.Providers.GeoJSON.validate_options/1: a shapefile release names its options in the converted GeoJSON document's vocabulary.

Functions

artifacts(manifest)

Delegates to GeoGenius.Providers.GeoJSON.artifacts/1: every artifact declared across the manifest's sources, all of them shapefile archives.

asserted_relations(manifest, row)

@spec asserted_relations(GeoGenius.Manifest.t(), GeoGenius.Staging.Row.t()) :: []

Delegates to GeoGenius.Providers.GeoJSON.asserted_relations/2: a converted shapefile carries no hierarchy in its columns beyond what its geometry already expresses.

normalize(manifest, row)

Delegates to GeoGenius.Providers.GeoJSON.normalize/2: a converted shapefile's staged rows carry the same shape as a GeoJSON feature's.

relations(manifest)

@spec relations(GeoGenius.Manifest.t()) :: :rebuild | :none

Delegates to GeoGenius.Providers.GeoJSON.relations/1: a converted shapefile's areas carry no hierarchy of their own; relations are always rebuilt.

required_options()

@spec required_options() :: [String.t()]

Delegates to GeoGenius.Providers.GeoJSON.required_options/0: the manifest options a shapefile release names are read from the converted GeoJSON document.

stage(manifest, artifact, path, emit, opts)

Unzips path into a private subdirectory of opts[:work_dir], converts its single .shp member to GeoJSON Sequence with ogr2ogr, and delegates the parse to the GeoJSON Sequence provider's staging implementation.

The subdirectory this call creates is removed once the call returns, whether it succeeds or fails; opts[:work_dir] itself, and anything else already in it, is left untouched.

validate_options(options)

@spec validate_options(map()) :: :ok | {:error, String.t()}

Delegates to GeoGenius.Providers.GeoJSON.validate_options/1: a shapefile release names its options in the converted GeoJSON document's vocabulary.