Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-21 Thread Björn Harrtell
I can live with not being elegant, my concern is only that it involves changing upstream sources. But yes it seems to be that there is no other viable solution so I will go ahead with it. Den fre 20 maj 2022 kl 19:40 skrev Even Rouault : > > Le 20/05/2022 à 19:00, Björn Harrtell a écrit : > > Fo

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-20 Thread Even Rouault
Le 20/05/2022 à 19:00, Björn Harrtell a écrit : For the record I can't see the #define trick as reasonable/acceptable if i understand it correctly. If you're saying it is not elegant, obviously, it isn't, but that works. Or is your concern something else ? All that pertains to the https://en

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-20 Thread Björn Harrtell
For the record I can't see the #define trick as reasonable/acceptable if i understand it correctly. I also have no other ideas, so I'm (very) sorry to say this is now dead in the water if nothing new comes to light. Den tors 19 maj 2022 09:13Björn Harrtell skrev: > Thanks Even, I was not aware o

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-19 Thread Björn Harrtell
Thanks Even, I was not aware of the risk of symbol clash. I will try and apply the #define trickery. Den ons 18 maj 2022 kl 13:31 skrev Even Rouault : > I'm 0 on this. While this is always nice to see a perf improvement, I'm > somewhat concerned by the duplication of code between MapServer and GD

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-19 Thread Björn Harrtell
1. Undefined behavior more or less. There is a verifier that could mitigate security risks in some capacity but it would cost a small bit of performance. I've not implemented support to use the verifier at this point (to keep the initial implementation as simple as possible). 2. It contains mapserv

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-18 Thread Jeff McKenna
Ah I see your point, I will expand that point more in the RFC now... Your assumption is correct. Thanks Lars! -jeff On 2022-05-18 5:09 p.m., Jeff McKenna wrote: Thanks for the positive support Lars, your testing feedback is important, and merry Christmas ha. Regarding your question, pleas

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-18 Thread Jeff McKenna
Thanks for the positive support Lars, your testing feedback is important, and merry Christmas ha. Regarding your question, please see the section "3.3 Limitations" of the RFC which is supposed to handle your question. -jeff On 2022-05-18 4:07 p.m., Lars Schylberg wrote: Thanks, Björn and

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-18 Thread Lars Schylberg
Thanks, Björn and Jeff for the effort to do this.I am eager to start testing soon.As a user I feel that Santa is coming early this year. I also have one question.Is the native FlatGeobuf driver doing the reading without checks, like "VERIFY_BUFFERS" "NO" that you can set in the OGR driver? M

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-18 Thread Steve Lime
Nice work all - esp. Björn. Evan has a good point that sounds like it could be addressed. Couple of comments: 1. I'd like to see a security impact section added just so it's clear that it has been thought about. For example, what is behavior if a corrupt, invalid or truncated file is acce

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-18 Thread Tom Kralidis
+1 ..Tom >> >>>> On May 18, 2022, at 06:54, jbo-...@mailo.com wrote: >>>> >>> +1 >>> Jérome >>> >>> >>>> De : Jeff McKenna >>>> À : mapserver-dev@lists.osgeo.org >>>> Sujet : Re: [mapserv

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-18 Thread Even Rouault
I'm 0 on this. While this is always nice to see a perf improvement, I'm somewhat concerned by the duplication of code between MapServer and GDAL. And from a purely technical point of view, one of my worry is that symbol clashes might occur between GDAL and MapServer on the common code they sha

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-18 Thread Seth G
t;>> De : Jeff McKenna >>> À : mapserver-dev@lists.osgeo.org >>> Sujet : Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as >>> built-in format >>> Date : 17/05/2022 15:19:06 Europe/Paris >>> >>> Update: Björn has completed the effo

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-18 Thread Tom Kralidis
+1 ..Tom > On May 18, 2022, at 06:54, jbo-...@mailo.com wrote: > > +1 > Jérome > > >> De : Jeff McKenna >> À : mapserver-dev@lists.osgeo.org >> Sujet : Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as >> built-in format

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-18 Thread jbo-ads
+1 Jérome De : Jeff McKenna À : mapserver-dev@lists.osgeo.org Sujet : Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format Date : 17/05/2022 15:19:06 Europe/Paris Update: Björn has completed the effort (which now includes several msautotests) and we've creat

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-05-17 Thread Jeff McKenna
Update: Björn has completed the effort (which now includes several msautotests) and we've created an RFC (137) for the native FlatGeobuf support: https://mapserver.org/development/rfc/ms-rfc-137.html I therefore motion to include FlatGeobuf as a native MapServer driver, per RFC 137, and mainta

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-04-27 Thread Jeff McKenna
Great point Even, here is an updated result: Shapefile 0.011s FlatGeobuf 0.014s Shapefile (OGR) 0.024s GeoPackage 0.042s SpatiaLite 0.045s PostGIS 0.053s GeoJSON 0.089s -jeff On 2022-04-27 2:09 p.m., Even Rouault wrote: Jeff, as a data po

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-04-27 Thread Even Rouault
Jeff, as a data point, perhaps you could enhance your bench with shapefiles through OGR ? It would be interesting to have a sense of the OGR overhead (that will be the same for any OGR supported datasource) to have an idea if it is really worth the effort to have a native FlatGeobuf provider

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-04-26 Thread Steve Lime
Seems like a small RFC is in order... --Steve On Tue, Apr 26, 2022 at 5:57 AM Jeff McKenna wrote: > (to followup from our chat on IRC yesterday) > > To clarify: this would be a new native format in MapServer, such as > Shapefile or PostGIS, and the goal would be to not connect to FlatGeobuf > th

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-04-26 Thread Rahkonen Jukka (MML)
+1 -Jukka Rahkonen- Lähettäjä: MapServer-dev Puolesta Björn Harrtell Lähetetty: maanantai 25. huhtikuuta 2022 23.58 Vastaanottaja: mapserver-dev@lists.osgeo.org Aihe: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format Hi mapserver devs! I got interested in the

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-04-26 Thread Jeff McKenna
(to followup from our chat on IRC yesterday) To clarify: this would be a new native format in MapServer, such as Shapefile or PostGIS, and the goal would be to not connect to FlatGeobuf through OGR, but instead directly, such as: DATA countries.fgb instead of: CONNECTIONTYPE OGR CONNE

Re: [mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-04-26 Thread Seth G
Hi Bjorn, +1 That sounds great! This would be a new "native" MapServer data source rather than going through GDAL/OGR, is that correct? It would need a few msautotests and docs, but I can try and help/test. Seth -- web:https://geographika.net twitter: @geographika On Mon, Apr 25, 2022, at 10

[mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

2022-04-25 Thread Björn Harrtell
Hi mapserver devs! I got interested in the subject because of the tests made recently by Jeff that shows the potential of the format. I believe it should be possible to get significant additional performance out of FlatGeobuf in MapServer if it was built in just like Shapefile support is. FlatGeo