Re: [Geoserver-users] reorder static gpkg by geohash for faster reads?
Hi Andrea, Thanks for the help. I feel like I should be able to use part of your example http://www.opengis.net/gpkg"; name="zoomstack-styles-sort-geohash"> land EPSG:27700 oszoom:land http://www.opengis.net/fes/2.0";> geom true true In the WPS request builder (gs:GeoPackage) on your test environment https://tb16.geo-solutions.it/geoserver/web/ ? The exception is: Process failed during execution java.lang.NoClassDefFoundError: ch/hsr/geohash/GeoHash ch/hsr/geohash/GeoHash My local modified attempt fails with Process failed during execution java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper com/fasterxml/jackson/databind/ObjectMapper com.fasterxml.jackson.databind.ObjectMapper I wanted to please check the direction I am taking, before I went looking for missing dependencies. I have installed geoserver-2.19-SNAPSHOT-wps-plugin.zip geoserver-2.19-SNAPSHOT-wps-download-plugin.zip geoserver-2.19-SNAPSHOT-geopkg-plugin.zip As a side note I think there is something wrong with the development build of the platform independent binary. Apologies if this is a known issue. I tested Windows and Ubuntu. The .war is fine. Please wait while loading GeoServer... ERROR : Nothing to start, exiting ... Usage: java -jar $JETTY_HOME/start.jar [options] [properties] [configs] java -jar $JETTY_HOME/start.jar --help # for more information Press any key to continue . . . I want to also try the PostGIS approach but this is a bigger hurdle for now. Cheers. From: Andrea Aime Sent: Friday, 8 January 2021 6:29 PM To: Russell Grew Cc: GeoServer Mailing List List Subject: Re: [Geoserver-users] reorder static gpkg by geohash for faster reads? On Fri, Jan 8, 2021 at 4:42 AM Russell Grew mailto:russell.g...@douglaspartners.com.au>> wrote: All, I quite enjoyed the two OGC Testbed 16 videos prepared by GeoSolutions Group. https://www.youtube.com/watch?v=hc81tlFMDbA<https://protect-au.mimecast.com/s/B_vLCwVLJVhMjD6tVSKYR?domain=youtube.com> Thanks! I have a few large (5+ GB) geopackage files hosted read only by GeoServer. They could be shapefiles but I went with geopackage to do something progressive, and avoid the 2 GB limit which some software has. In our experience GeoHash sorting helped greatly on the 200GB GeoPackage, and only in a smaller amount on the 10GB ZoomStack package, where the generalized table extension had a far greater effect*. Your case might vary, depends a lot on the number of records (the more records, the more the effect is visible) and the type of disks (less visible on SSDs unless the size of the package is huge, more visible on spinning disks... I'm guessing network mounted disks also benefit). Following Andre’s talk I wonder if I should attempt to reorder the data in these files by geohash. To help it load faster. Is my understanding correct? Does anyone know of any tools to enable this? I had to build the machinery from scratch in Java. Thinking out loud, if you want to use something else, maybe you can dump the data in postgresql, add the geohash column, and create a view that returns data sorted by geohash by default. Then use ogr2ogr to dump a geopackage from the view, that hopefully retains the right record order. Perhaps I could use GeoServer 2.19 RC hosting my current files and then dump them with gs:GeoPackage? If this is the way forward any advice on how to tell it what layer I want to dump would be great. The WPS I have looked at before all had vector layer process input boxes. I've attached a request sorting by GeoHash against the ZoomStack data set, hopefully you can use it as a reference to build your request. If you try it out, let us know how the results perform. Cheers Andrea *: As a note, GeoServer has been extended to produce the packages with the new extensions, but has not ability to read them (the read part was in a different Testbed component, developed by another company). That said, the existing work on the write side would help develop the read portion quicker. == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V<https://protect-au.mimecast.com/s/M_oWCxnMKnc6l28HvhP6T?domain=goo.gl> for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it<https://protect-au.mimecast.com/s/RjLKCyoNLoIv1lDsQYVJO?domain=geo-solutions.it> http://twitter.com/geosolutions_it<https://protect-au.mimecast.com/s/wwezCzvOMvul10DiKmD18?domain=twitter.com> --- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”),
Re: [Geoserver-users] reorder static gpkg by geohash for faster reads?
On Fri, Jan 8, 2021 at 4:42 AM Russell Grew < russell.g...@douglaspartners.com.au> wrote: > All, > > > > I quite enjoyed the two OGC Testbed 16 videos prepared by GeoSolutions > Group. > > > > https://www.youtube.com/watch?v=hc81tlFMDbA > Thanks! > > > I have a few large (5+ GB) geopackage files hosted read only by GeoServer. > They could be shapefiles but I went with geopackage to do something > progressive, and avoid the 2 GB limit which some software has. > In our experience GeoHash sorting helped greatly on the 200GB GeoPackage, and only in a smaller amount on the 10GB ZoomStack package, where the generalized table extension had a far greater effect*. Your case might vary, depends a lot on the number of records (the more records, the more the effect is visible) and the type of disks (less visible on SSDs unless the size of the package is huge, more visible on spinning disks... I'm guessing network mounted disks also benefit). > Following Andre’s talk I wonder if I should attempt to reorder the data > in these files by geohash. To help it load faster. > > > > Is my understanding correct? Does anyone know of any tools to enable this? > I had to build the machinery from scratch in Java. Thinking out loud, if you want to use something else, maybe you can dump the data in postgresql, add the geohash column, and create a view that returns data sorted by geohash by default. Then use ogr2ogr to dump a geopackage from the view, that hopefully retains the right record order. > Perhaps I could use GeoServer 2.19 RC hosting my current files and then > dump them with gs:GeoPackage? If this is the way forward any advice on how > to tell it what layer I want to dump would be great. The WPS I have looked > at before all had vector layer process input boxes. > I've attached a request sorting by GeoHash against the ZoomStack data set, hopefully you can use it as a reference to build your request. If you try it out, let us know how the results perform. Cheers Andrea *: As a note, GeoServer has been extended to produce the packages with the new extensions, but has not ability to read them (the read part was in a different Testbed component, developed by another company). That said, the existing work on the write side would help develop the read portion quicker. == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- *Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.* http://www.opengis.net/wps/1.0.0"; xmlns="http://www.opengis.net/wps/1.0.0"; xmlns:gml="http://www.opengis.net/gml"; xmlns:ogc="http://www.opengis.net/ogc"; xmlns:ows="http://www.opengis.net/ows/1.1"; xmlns:wcs="http://www.opengis.net/wcs/1.1.1"; xmlns:wfs="http://www.opengis.net/wfs"; xmlns:xlink="http://www.w3.org/1999/xlink"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; version="1.0.0" service="WPS" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd";> gs:GeoPackage contents geopackage ___ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listin
[Geoserver-users] reorder static gpkg by geohash for faster reads?
All, I quite enjoyed the two OGC Testbed 16 videos prepared by GeoSolutions Group. https://www.youtube.com/watch?v=hc81tlFMDbA I have a few large (5+ GB) geopackage files hosted read only by GeoServer. They could be shapefiles but I went with geopackage to do something progressive, and avoid the 2 GB limit which some software has. Following Andre's talk I wonder if I should attempt to reorder the data in these files by geohash. To help it load faster. Is my understanding correct? Does anyone know of any tools to enable this? Perhaps I could use GeoServer 2.19 RC hosting my current files and then dump them with gs:GeoPackage? If this is the way forward any advice on how to tell it what layer I want to dump would be great. The WPS I have looked at before all had vector layer process input boxes. Thanks. ___ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
