Theo,
you indeed spotted an hardcoded limit in the the GeoJSONSeq driver.
Addressed per https://github.com/OSGeo/gdal/pull/6177
Even
Le 09/08/2022 à 05:20, Theo Linnemann a écrit :
Hi There,
I'm working with a set of fairly large geojsonl files, one for
each state and territory in the United States. I'm using ogr2ogr
to ingest this data into PostGIS, generally without issue, to
speed up an expensive geospatial join.
However, one of my files, specifically the Commonwealth of
Pennsylvania, is about 6.5 GB and has been throwing an odd error
that I can't seem to find much information on. "ERROR 6: Too large
feature". Below is the command and output from running ogrinfo
with the debug flag set to on, which returns the same error.
I'm just hoping for some guidance in terms of what if anything I
can do to sidestep the issue.
So far, I've taken a look at what seems to be the GDAL source code
that generates this error
<https://github.com/OSGeo/gdal/blob/030ff40cf8340273bcc797e90c938cc32d14a34f/ogr/ogrsf_frmts/geojson/ogrgeojsonseqdriver.cpp#L410>,
but it's unclear to me if it's possible to adjust the size
constraint that is specified in the condition that throws the
error or if the constraint is arbitrary or indicative of a larger
problem with the file. I've already tried using the -gt flag with
various sizes as well, but none seem to resolve the issue. I ran
into an issue previously that required me to set
OGR_GEOJSON_MAX_OBJ_SIZE=2048MB but that doesn't seem to have any
effect here for some reason. Perhaps that only effects the GeoJSON
driver and not the GeoJSONSeq driver? For what it's worth this is
running on a node with ~350GB of RAM, so generally memory
shouldn't be an issue.
ogrinfo --version
GDAL 3.5.0, released 2022/05/10
------------------------------------------------------
ogrinfo --build
PAM_ENABLED=YES
OGR_ENABLED=YES
GEOS_ENABLED=YES
GEOS_VERSION=3.10.2-CAPI-1.16.0
PROJ_BUILD_VERSION=9.0.0
PROJ_RUNTIME_VERSION=9.0.0
------------------------------------------------------
ogrinfo crs4326_fema_42_2022-07-22.geojsonl --debug ON
GeoJSONSeq: First pass: 0.06 %
GeoJSONSeq: First pass: 0.12 %
GeoJSONSeq: First pass: 0.18 %
...
GeoJSONSeq: First pass: 92.08 %
GeoJSONSeq: First pass: 92.14 %
GeoJSONSeq: First pass: 92.20 %
ERROR 6: Too large feature
GDAL: GDALOpen(crs4326_fema_42_2022-07-22.geojsonl,
this=0x564966fae8b0) succeeds as GeoJSONSeq.
INFO: Open of `crs4326_fema_42_2022-07-22.geojsonl'
using driver `GeoJSONSeq' successful.
OGR: GetLayerCount() = 1
1: crs4326_fema_42_2022-07-22 (Multi Polygon)
GDAL: GDALClose(crs4326_fema_42_2022-07-22.geojsonl,
this=0x564966fae8b0)
GDAL: In GDALDestroy - unloading GDAL shared library.
Thank you for any help you're able to offer, it's much appreciated.
TL
Theo Linnemann <https://www.theolinnemann.com/>
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
