Provider for a shapefile archive.
A shapefile is several sibling files (.shp, .dbf, .shx, and usually
.prj) that must travel together, so the artifact this provider stages is
a zip archive rather than a single file. stage/5:
- Unzips the archive into a private subdirectory of
opts[:work_dir]with:zip.unzip/2from OTP -- no dependency, since the archive format is fixed. - Finds exactly one
.shpmember. Zero is an error naming the archive; more than one is an error naming both, because picking one silently would publish half a dataset. - Converts it with the configured
GeoGenius.Commandadapter:ogr2ogr -f GeoJSONSeq -t_srs EPSG:4326 -lco RS=NO <out.geojsonl> <in.shp>. The-t_srsflag is not optional: a shapefile in a projected coordinate system would otherwise stage coordinates in metres that PostGIS accepts as degrees -- geometry that is silently, plausibly wrong rather than obviously broken. - Delegates the parse of the converted sequence to the GeoJSON Sequence provider's staging implementation.
normalize/2, required_options/0, artifacts/1,
relations/1, and asserted_relations/2 all delegate to
GeoGenius.Providers.GeoJSON unchanged: a converted shapefile is a sequence
of GeoJSON Feature records by the time anything downstream of ogr2ogr
sees it, so the manifest options a release names are the same ones
GeoGenius.Providers.GeoJSON reads.
This is the only shipped provider that needs GDAL installed -- a host
publishing GeoJSON or CSV collections never installs it. available?/2 on
the configured GeoGenius.Command is checked before ogr2ogr runs, so a
host missing GDAL gets a named error rather than a raised exception or a
bare non-zero exit.
This module ships in place of the module name Providers.Census used
elsewhere: what a Census-specific provider would actually contain is this
format reader plus a vendor's file-naming and code conventions, and the
second half is vendor knowledge a host-neutral catalog library has no
business carrying. Shipping the format half means a Census release is an
ordinary manifest naming "shapefile" with Census column names in its
options, and a release from any other authority that ships shapefiles is
another one.
Working directory. opts[:work_dir] is resolved once per pipeline run
and may be shared across every artifact in a manifest -- a downloaded
archive can already be sitting in it, and another shapefile artifact in
the same release may extract into it around the same time. This provider
never removes work_dir itself: it creates its own uniquely-named
subdirectory under it (System.unique_integer/1-suffixed, the shape used
elsewhere in this codebase) to unzip and convert into, and removes only
that subdirectory when the call returns. stage/5 is eager -- emit has
already run by the time this function returns -- so nothing outside the
call needs the extracted or converted files afterward, and cleanup runs on
both the success and error paths. Cleanup does not run if the process is
killed by an exit signal rather than returning or raising; the default
work_dir lives under System.tmp_dir!/0, so the OS reaps it regardless.
Member matching is case-sensitive and unfiltered by path.
TERRITORIES.SHP in an archive is read as "no .shp member", and an
archive zipped by macOS that carries __MACOSX/._territories.shp
alongside the real member is read as "more than one .shp member". Both
fail loudly with a named error rather than silently choosing wrong, so
this is a limit to know about, not a defect to route around silently.
Streaming conversion. -f GeoJSONSeq -lco RS=NO writes one Feature per
newline. The GeoJSON Sequence provider reads and emits the resulting
sequence in bounded batches, so this provider never retains the converted
dataset as a whole document.
Summary
Functions
Delegates to GeoGenius.Providers.GeoJSON.artifacts/1: every artifact declared across the manifest's sources, all of them shapefile archives.
Delegates to GeoGenius.Providers.GeoJSON.asserted_relations/2: a converted shapefile carries no hierarchy in its columns beyond what its geometry already expresses.
Delegates to GeoGenius.Providers.GeoJSON.normalize/2: a converted shapefile's staged rows carry the same shape as a GeoJSON feature's.
Delegates to GeoGenius.Providers.GeoJSON.relations/1: a converted shapefile's areas carry no hierarchy of their own; relations are always rebuilt.
Delegates to GeoGenius.Providers.GeoJSON.required_options/0: the manifest options a shapefile release names are read from the converted GeoJSON document.
Unzips path into a private subdirectory of opts[:work_dir], converts
its single .shp member to GeoJSON Sequence with ogr2ogr, and delegates
the parse to the GeoJSON Sequence provider's staging implementation.
Delegates to GeoGenius.Providers.GeoJSON.validate_options/1: a shapefile release names its options in the converted GeoJSON document's vocabulary.
Functions
@spec artifacts(GeoGenius.Manifest.t()) :: [GeoGenius.Manifest.Artifact.t()]
Delegates to GeoGenius.Providers.GeoJSON.artifacts/1: every artifact declared across the manifest's sources, all of them shapefile archives.
@spec asserted_relations(GeoGenius.Manifest.t(), GeoGenius.Staging.Row.t()) :: []
Delegates to GeoGenius.Providers.GeoJSON.asserted_relations/2: a converted shapefile carries no hierarchy in its columns beyond what its geometry already expresses.
@spec normalize(GeoGenius.Manifest.t(), GeoGenius.Staging.Row.t()) :: {:ok, GeoGenius.Provider.Area.t()} | :skip | {:error, GeoGenius.Provider.reason()}
Delegates to GeoGenius.Providers.GeoJSON.normalize/2: a converted shapefile's staged rows carry the same shape as a GeoJSON feature's.
@spec relations(GeoGenius.Manifest.t()) :: :rebuild | :none
Delegates to GeoGenius.Providers.GeoJSON.relations/1: a converted shapefile's areas carry no hierarchy of their own; relations are always rebuilt.
@spec required_options() :: [String.t()]
Delegates to GeoGenius.Providers.GeoJSON.required_options/0: the manifest options a shapefile release names are read from the converted GeoJSON document.
@spec stage( GeoGenius.Manifest.t(), GeoGenius.Manifest.Artifact.t(), Path.t(), ([GeoGenius.Staging.Row.t()] -> :ok), GeoGenius.Provider.stage_opts() ) :: :ok | {:error, GeoGenius.Provider.reason()}
Unzips path into a private subdirectory of opts[:work_dir], converts
its single .shp member to GeoJSON Sequence with ogr2ogr, and delegates
the parse to the GeoJSON Sequence provider's staging implementation.
The subdirectory this call creates is removed once the call returns,
whether it succeeds or fails; opts[:work_dir] itself, and anything else
already in it, is left untouched.
Delegates to GeoGenius.Providers.GeoJSON.validate_options/1: a shapefile release names its options in the converted GeoJSON document's vocabulary.