So this (https://github.com/OSGeo/gdal/pull/6804) optimization by Even might be helpful here.
In any case nice to see the limits of current FlatGeobuf impl. being put to the test. 🙂 I suppose one could do an on disk sorting implementation but I'm not sure it's worth the complexity of it and speed trade off. If i can get to it would like to get some measurements done to properly document the amount of RAM required as a function of number of features to index. Björn Den tors 24 nov. 2022 19:38Even Rouault <even.roua...@spatialys.com> skrev: > > Le 24/11/2022 à 19:18, michael.smith.e...@gmail.com a écrit : > > I don’t believe these are huge features, its OSM building for the > > world. Must be the memory constraint since it is a lot of features. > > This doesn’t happen (i think, am retesting now) if created from the > > osm pbf file directly but does from the fgb created from the pbf. > Hum, it might be that the FGB reader is a bit gready in its memory usage > then. Björn might be able to comment more > > > > Do either of the columnar formats (arrow, geoparquet) support spatial > > access window selections yet (via ogr2ogr) or is that still forthcoming? > > There is no spatial index specification for those formats (yet ? it > isn't obvious how to stick one into them, in a clean and backward > compatible way, but I believe this is a subject of research). So for now > spatial filtering involves scanning all features geometries (there are a > few optimizations to make it as efficient as possible, but you > definitely won't get a great performance if repeatedly selecting small > amounts of features with different extents) > > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev