[Geotools-devel] Geopackage re-initializing the table structure like crazy
Hi, looking at the extreme slowness of the geopackage tests (takes a few minutes to run them in online mode, like, Oracle level slow...) I found that the code is running the full test of "create tables" for geopackage support every time a connection is grabbed: https://github.com/geotools/geotools/blob/master/modules/unsupported/geopkg/src/main/java/org/geotools/geopkg/GeoPackage.java#L232 This is a bit crazy to me, shouldn't we doing that only once instead? Or have a fast way to check if the geopackage is already properly initialized? Wondering if there was already any thought in that direction. Like, what about running the table initialization only once on store creation for example? Cheers Andrea -- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003* Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc. --- -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
Re: [Geotools-devel] Upgrading xerial-jdbc driver version in geopackage and mbtiles
Right, I thought that was going to be the direction too. It seems the Linux distributions have started to collaborate, mine ships with spatialite 4.3.0 which should allow dynamic linking. Now... we just need someone to do the work :-p Cheers Andrea On Wed, Oct 19, 2016 at 2:59 PM, Justin Deoliveira wrote: > No suggestion unfortunately. > > That spatialite custom build of xerial was a pain to build, and probably > not worth the effort. The intention was to have all of the library loading, > etc… be transparent, in the same way that users of xerial don’t have to > worry about loading the sqlite library. But it required forking the build > chain of the xerial driver, and those changes are probably long past > compatible with the current upstream version. That said I think newer > versions (4+) of spatialite allow for a more dynamic library loading so I > think it is possible to do this without really changing anything upstream. > But I haven’t really investigated that. > > For the geopackage and mbtiles drivers we explicitly wanted to avoid this > kind of mess, which we were allowed to do since spatialite was not a > requirement. > > As for the conflict between the two… yes that is an issue and I don’t have > any solution there besides changing the spatialite module to use the base > xerial driver and load the spatialite libraries dynamically at runtime. > > Not sure if that helps. > > -Justin > > On Wed, Oct 19, 2016 at 6:48 AM Andrea Aime > wrote: > >> Hi, >> working on the geopackage module I've found a test failing due to a bug >> in the >> xerial driver itself: >> >> https://bitbucket.org/xerial/sqlite-jdbc/issues/134/ >> metadatagetindexinfo-throws-exception >> >> The latest version of the driver has this issue fixed, so I wanted to >> propose an upgrade >> from 3.8.6, the currently used version, to 3.14.2.1, the latest version. >> >> I see that mbtiles also uses the same driver, and upgrading it causes no >> issues. >> >> I was worried about the spatialite, but I see it's already using a >> different version (a customized >> 3.7.something) so I guess changing the version of the other two modules >> is hopefully >> not going to cause "new" troubles, in the end, if the two modules are put >> togther in the >> same classpath there is no saying which libsqlitejdbc.so/dll will be >> picked up... >> >> Justin, the spatialite module lists you as the maintainer, any >> suggestion? >> >> Cheers >> Andrea >> >> -- >> == >> GeoServer Professional Services from the experts! Visit >> http://goo.gl/it488V for more information. >> == >> >> Ing. Andrea Aime >> @geowolf >> Technical Lead >> >> GeoSolutions S.A.S. >> Via di Montramito 3/A >> 55054 Massarosa (LU) >> phone: +39 0584 962313 <+39%200584%20962313> >> fax: +39 0584 1660272 <+39%200584%20166%200272> >> mob: +39 339 8844549 <+39%20339%20884%204549> >> >> http://www.geo-solutions.it >> http://twitter.com/geosolutions_it >> >> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003* >> >> Le informazioni contenute in questo messaggio di posta elettronica e/o >> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il >> loro utilizzo è consentito esclusivamente al destinatario del messaggio, >> per le finalità indicate nel messaggio stesso. Qualora riceviate questo >> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di >> darcene notizia via e-mail e di procedere alla distruzione del messaggio >> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, >> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od >> utilizzarlo per finalità diverse, costituisce comportamento contrario ai >> principi dettati dal D.Lgs. 196/2003. >> >> >> >> The information in this message and/or attachments, is intended solely >> for the attention and use of the named addressee(s) and may be confidential >> or proprietary in nature or covered by the provisions of privacy act >> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection >> Code).Any use not in accord with its purpose, any disclosure, reproduction, >> copying, distribution, or either dissemination, either whole or partial, is >> strictly forbidden except previous formal approval of the named >> addressee(s). If you are not the intended recipient, please contact >> immediately the sender by telephone, fax or e-mail and delete the >> information in this message that has been received in error. The sender >> does not give any warranty or accept liability as the content, accuracy or >> completeness of sent messages and accepts no responsibility for changes >> made after they were sent or for other risks which arise as a result of >> e-mail transmission, viruses, etc. >> >> --- >> > -- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) pho
Re: [Geotools-devel] Upgrading xerial-jdbc driver version in geopackage and mbtiles
No suggestion unfortunately. That spatialite custom build of xerial was a pain to build, and probably not worth the effort. The intention was to have all of the library loading, etc… be transparent, in the same way that users of xerial don’t have to worry about loading the sqlite library. But it required forking the build chain of the xerial driver, and those changes are probably long past compatible with the current upstream version. That said I think newer versions (4+) of spatialite allow for a more dynamic library loading so I think it is possible to do this without really changing anything upstream. But I haven’t really investigated that. For the geopackage and mbtiles drivers we explicitly wanted to avoid this kind of mess, which we were allowed to do since spatialite was not a requirement. As for the conflict between the two… yes that is an issue and I don’t have any solution there besides changing the spatialite module to use the base xerial driver and load the spatialite libraries dynamically at runtime. Not sure if that helps. -Justin On Wed, Oct 19, 2016 at 6:48 AM Andrea Aime wrote: > Hi, > working on the geopackage module I've found a test failing due to a bug in > the > xerial driver itself: > > > https://bitbucket.org/xerial/sqlite-jdbc/issues/134/metadatagetindexinfo-throws-exception > > The latest version of the driver has this issue fixed, so I wanted to > propose an upgrade > from 3.8.6, the currently used version, to 3.14.2.1, the latest version. > > I see that mbtiles also uses the same driver, and upgrading it causes no > issues. > > I was worried about the spatialite, but I see it's already using a > different version (a customized > 3.7.something) so I guess changing the version of the other two modules is > hopefully > not going to cause "new" troubles, in the end, if the two modules are put > togther in the > same classpath there is no saying which libsqlitejdbc.so/dll will be > picked up... > > Justin, the spatialite module lists you as the maintainer, any suggestion? > > Cheers > Andrea > > -- > == > GeoServer Professional Services from the experts! Visit > http://goo.gl/it488V for more information. > == > > Ing. Andrea Aime > @geowolf > Technical Lead > > GeoSolutions S.A.S. > Via di Montramito 3/A > 55054 Massarosa (LU) > phone: +39 0584 962313 <+39%200584%20962313> > fax: +39 0584 1660272 <+39%200584%20166%200272> > mob: +39 339 8844549 <+39%20339%20884%204549> > > http://www.geo-solutions.it > http://twitter.com/geosolutions_it > > *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003* > > Le informazioni contenute in questo messaggio di posta elettronica e/o > nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il > loro utilizzo è consentito esclusivamente al destinatario del messaggio, > per le finalità indicate nel messaggio stesso. Qualora riceviate questo > messaggio senza esserne il destinatario, Vi preghiamo cortesemente di > darcene notizia via e-mail e di procedere alla distruzione del messaggio > stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, > divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od > utilizzarlo per finalità diverse, costituisce comportamento contrario ai > principi dettati dal D.Lgs. 196/2003. > > > > The information in this message and/or attachments, is intended solely for > the attention and use of the named addressee(s) and may be confidential or > proprietary in nature or covered by the provisions of privacy act > (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection > Code).Any use not in accord with its purpose, any disclosure, reproduction, > copying, distribution, or either dissemination, either whole or partial, is > strictly forbidden except previous formal approval of the named > addressee(s). If you are not the intended recipient, please contact > immediately the sender by telephone, fax or e-mail and delete the > information in this message that has been received in error. The sender > does not give any warranty or accept liability as the content, accuracy or > completeness of sent messages and accepts no responsibility for changes > made after they were sent or for other risks which arise as a result of > e-mail transmission, viruses, etc. > > --- > -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
[Geotools-devel] Upgrading xerial-jdbc driver version in geopackage and mbtiles
Hi, working on the geopackage module I've found a test failing due to a bug in the xerial driver itself: https://bitbucket.org/xerial/sqlite-jdbc/issues/134/metadatagetindexinfo-throws-exception The latest version of the driver has this issue fixed, so I wanted to propose an upgrade from 3.8.6, the currently used version, to 3.14.2.1, the latest version. I see that mbtiles also uses the same driver, and upgrading it causes no issues. I was worried about the spatialite, but I see it's already using a different version (a customized 3.7.something) so I guess changing the version of the other two modules is hopefully not going to cause "new" troubles, in the end, if the two modules are put togther in the same classpath there is no saying which libsqlitejdbc.so/dll will be picked up... Justin, the spatialite module lists you as the maintainer, any suggestion? Cheers Andrea -- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003* Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc. --- -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
