I'm 0 on this. While this is always nice to see a perf improvement, I'm
somewhat concerned by the duplication of code between MapServer and GDAL.
And from a purely technical point of view, one of my worry is that
symbol clashes might occur between GDAL and MapServer on the common code
they share in the FlatGeobuf and flatbuffers C++ namespaces. If the
implementations are not kept in sync (and that will necessarily happen
at some point because we cannot control which GDAL version is used by
MapServer), weird things (crashes) could possibly happen at runtime (or
at build time if doing static builds). That worry could be solved by
possibly having a mapserver:: prefix in front of those namespace names
(can be done with some #define trickery. See
https://github.com/OSGeo/PROJ/blob/9f89c7288e44c64ba234a299b1d5528a54d9d61e/include/proj/util.hpp#L104
for potential inspiration)
The interesting take-away from my perspective is that the OGR overhead
for FlatGeobuf is only 5 ms, but 13 ms for Shapefile. So one
interpretation would be that there's some inefficiency in the OGR
Shapefile driver vs the native MapServer provider.
Even
Le 17/05/2022 à 15:19, Jeff McKenna a écrit :
Update: Björn has completed the effort (which now includes several
msautotests) and we've created an RFC (137) for the native FlatGeobuf
support: https://mapserver.org/development/rfc/ms-rfc-137.html
I therefore motion to include FlatGeobuf as a native MapServer driver,
per RFC 137, and maintained by Björn Harrtell.
starting with my +1
Some other interesting updates:
- great to see changes here made through testing benefiting so many
other projects
- here is an updated benchmark result (that is mentioned in the RFC) :
FlatGeobuf (native) 0.008s
Shapefile (native) 0.010s
FlatGeobuf (OGR) 0.013s
Shapefile (OGR) 0.023s
GeoPackage (OGR) 0.042s
SpatiaLite (OGR) 0.045s
PostGIS (native) 0.053s
GeoJSON (OGR) 0.089s
(that was with yesterday's main, GDAL 3.4.3, with MS4W on Windows)
- with all this testing done, and the existing FlatGeobuf
documentation effort, it will be very easy to add this to the main
documentation now...
Thanks,
-Björn & Jeff
On 2022-04-27 3:07 p.m., Jeff McKenna wrote:
Great point Even, here is an updated result:
Shapefile 0.011s
FlatGeobuf 0.014s
Shapefile (OGR) 0.024s
GeoPackage 0.042s
SpatiaLite 0.045s
PostGIS 0.053s
GeoJSON 0.089s
-jeff
On 2022-04-27 2:09 p.m., Even Rouault wrote:
Jeff,
as a data point, perhaps you could enhance your bench with
shapefiles through OGR ? It would be interesting to have a sense of
the OGR overhead (that will be the same for any OGR supported
datasource) to have an idea if it is really worth the effort to have
a native FlatGeobuf provider for MapServer w.r.t going through OGR
(it is hard to beat shapefile, and the current difference between
shapefile and flatgeobuf is small)
Even
Le 26/04/2022 à 12:57, Jeff McKenna a écrit :
(to followup from our chat on IRC yesterday)
To clarify: this would be a new native format in MapServer, such as
Shapefile or PostGIS, and the goal would be to not connect to
FlatGeobuf through OGR, but instead directly, such as:
DATA countries.fgb
instead of:
CONNECTIONTYPE OGR
CONNECTION "countries.fgb"
DATA "countries"
This should improve the performance even more. And it will be easy
to update the documentation, as the existing files just need to be
updated (see the new page at
https://mapserver.org/input/vector/flatgeobuf.html )
Great plan Björn! I look forward to your Pull Request, ha, and
will tackle the documentation changes for you.
This is also great news for MapServer, and will give us some nice
publicity.
Thanks so much!
-jeff
On 2022-04-25 5:57 p.m., Björn Harrtell wrote:
Hi mapserver devs!
I got interested in the subject because of the tests made recently
by Jeff that shows the potential of the format. I believe it
should be possible to get significant additional performance out
of FlatGeobuf in MapServer if it was built in just like Shapefile
support is.
FlatGeobuf C++ implementation is rather small, doesn't require any
dependencies, and I don't expect it to require any invasive
changes to existing code.
Would love to get started on this ASAP, if there are no
objections. :)
Regards,
Björn Harrtell
______
_______________________________________________
MapServer-dev mailing list
MapServer-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
MapServer-dev mailing list
MapServer-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
