Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?
Hi, playing lately with more complex scenarios with ogr2ogr, that time involving both a lot of features and multiple layers being converted at once to GeoPackage, I've seen error messages being emitted during conversions and some slowness due to SQLite3 self-locking itself for periods of a few seconds. This is still due to the multithreaded RTree implementation. I didn't notice data corruption though, but I won't swear it couldn't happen. My conclusion was that is isn't safe to enable it when multiple layers are created in a GeoPackage file. I've thus disabled it in such scenarios per https://github.com/OSGeo/gdal/pull/6959. So in the meantime setting the OGR_GPKG_ALLOW_THREADED_RTREE environment variable/configuration option to NO is still a recommended workaround (if you write multiple layers in the same GeoPackage file). I'll probably issue a 3.6.2 release at the beginning of next year when everyone is back from their break. Even Le 13/12/2022 à 16:18, Even Rouault via gdal-dev a écrit : Hi, https://github.com/qgis/QGIS/issues/51188 has been brought to my attention. The issue is that the new background building of the RTree of GeoPackage files introduced in 3.6.0 didn't work well with committing transactions in between, which is easily triggered by ogr2ogr. All features were inserted but with default settings ~ 10% of them lacked an entry in the spatial index, which can be enough to break interactive display and workflows relying on spatial filtering. When using ogr2ogr, I believe the issue can only be seen when creating layers with more than 100 000 features, since that's the default value for the interval at which transactions are committed. I've a fix ready in https://github.com/OSGeo/gdal/pull/6911. The fix itself is a simple two-liner: https://github.com/OSGeo/gdal/pull/6911/commits/3f5f6225fe82e0c2e0241e4f66bfb861cdf4fe9d Given the status of GeoPackage being the default format for QGIS, I believe this is a severe enough issue to warrant an advanced 3.6.1 release, and an official retraction of 3.6.0. Thoughts ? Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?
Le 15/12/2022 à 21:48, Daniel Evans a écrit : Similar thoughts here, too. Thinking particularly of the Docker image, the main use case for them is ensuring a reproducible build on the user's side. Those using the `latest` tag (indicating they're not so bothered about versions) will start building with 3.6.1 automatically anyway. Actually no: the :latest Docker image actually correspond to master builds, at least as currently setup, and they are infrequently refreshed (too long to refresh them nightly now they involve doing arm64 cross building) -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?
Similar thoughts here, too. Thinking particularly of the Docker image, the main use case for them is ensuring a reproducible build on the user's side. Those using the `latest` tag (indicating they're not so bothered about versions) will start building with 3.6.1 automatically anyway. Those who have pinned builds to 3.6.0 presumably want to avoid unexpected changes - and given it's short life, I expect those are users who follow GDAL quite closely. I also don't get the impression this is *significantly* worse than the normal bugfixes we see with any release - minor bugs like coordinate rounding and interpolation issues have affected plenty of raster datasets in the past, but we wouldn't consider a retraction of all GDAL versions up to that point. Regards, Daniel On Thu, 15 Dec 2022 at 20:11, Greg Troxel wrote: > > Even Rouault writes: > > > Question: now that 3.6.1 is out, what to do with the > > osgeo/gdal:-3.6.0 docker images: > > > > - keep them > > > > - remove them > > > > - alias them to the 3.6.1 ones which are now available ? > > Basically I'm with Bas on this. > > But even more strongly, do not alias 3.6.0 to 3.6.1. If someone asks > for 3.6.0, getting it is fine, and getting an error is ok, but getting > something else is not ok. > > As a packger I don't really care, because I was fortunate to still be > waiting to get around to 3.6.0 (part cmake, part life in general, part > being a little shy about .0). So I will, when I get to it, go right to > 3.6.1. > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?
Even Rouault writes: > Question: now that 3.6.1 is out, what to do with the > osgeo/gdal:-3.6.0 docker images: > > - keep them > > - remove them > > - alias them to the 3.6.1 ones which are now available ? Basically I'm with Bas on this. But even more strongly, do not alias 3.6.0 to 3.6.1. If someone asks for 3.6.0, getting it is fine, and getting an error is ok, but getting something else is not ok. As a packger I don't really care, because I was fortunate to still be waiting to get around to 3.6.0 (part cmake, part life in general, part being a little shy about .0). So I will, when I get to it, go right to 3.6.1. signature.asc Description: PGP signature ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?
On 12/15/22 20:35, Even Rouault wrote: The same holds with source tarballs of 3.6.0 available on download.osgeo.org/gdal/3.6.0 or as github release. Digging into history, we retracted 1.7.0 in https://trac.osgeo.org/gdal/wiki/DownloadSource#a1.7.0retracted-January2010 and as far as I can see in http://download.osgeo.org/gdal/old_releases/ , the 1.7.0 tarballs were also removed from download. Why try to scrub buggy releases from the history? They are not infectious diseases that need to be avoided like the plague. The release happened so the artifacts associated with it should remain, because of a significant bug a new release was published sooner than expected to reduce the time the latest release is affected. Kind Regards, Bas -- GPG Key ID: 4096R/6750F10AE88D4AF1 Fingerprint: 8182 DE41 7056 408D 6146 50D1 6750 F10A E88D 4AF1 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?
Question: now that 3.6.1 is out, what to do with the osgeo/gdal:-3.6.0 docker images: - keep them - remove them - alias them to the 3.6.1 ones which are now available ? The same holds with source tarballs of 3.6.0 available on download.osgeo.org/gdal/3.6.0 or as github release. Digging into history, we retracted 1.7.0 in https://trac.osgeo.org/gdal/wiki/DownloadSource#a1.7.0retracted-January2010 and as far as I can see in http://download.osgeo.org/gdal/old_releases/ , the 1.7.0 tarballs were also removed from download. Even Le 13/12/2022 à 16:18, Even Rouault via gdal-dev a écrit : Hi, https://github.com/qgis/QGIS/issues/51188 has been brought to my attention. The issue is that the new background building of the RTree of GeoPackage files introduced in 3.6.0 didn't work well with committing transactions in between, which is easily triggered by ogr2ogr. All features were inserted but with default settings ~ 10% of them lacked an entry in the spatial index, which can be enough to break interactive display and workflows relying on spatial filtering. When using ogr2ogr, I believe the issue can only be seen when creating layers with more than 100 000 features, since that's the default value for the interval at which transactions are committed. I've a fix ready in https://github.com/OSGeo/gdal/pull/6911. The fix itself is a simple two-liner: https://github.com/OSGeo/gdal/pull/6911/commits/3f5f6225fe82e0c2e0241e4f66bfb861cdf4fe9d Given the status of GeoPackage being the default format for QGIS, I believe this is a severe enough issue to warrant an advanced 3.6.1 release, and an official retraction of 3.6.0. Thoughts ? Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?
+1 > On Dec 13, 2022, at 9:47 AM, Daniel Morissette > wrote: > > +1 for 3.6.1 with this fix. > > Daniel > > > On 2022-12-13 10:18, Even Rouault via gdal-dev wrote: >> Hi, >> https://github.com/qgis/QGIS/issues/51188 has been brought to my attention. >> The issue is that the new background building of the RTree of GeoPackage >> files introduced in 3.6.0 didn't work well with committing transactions in >> between, which is easily triggered by ogr2ogr. All features were inserted >> but with default settings ~ 10% of them lacked an entry in the spatial >> index, which can be enough to break interactive display and workflows >> relying on spatial filtering. When using ogr2ogr, I believe the issue can >> only be seen when creating layers with more than 100 000 features, since >> that's the default value for the interval at which transactions are >> committed. >> I've a fix ready in https://github.com/OSGeo/gdal/pull/6911. The fix itself >> is a simple two-liner: >> https://github.com/OSGeo/gdal/pull/6911/commits/3f5f6225fe82e0c2e0241e4f66bfb861cdf4fe9d >> Given the status of GeoPackage being the default format for QGIS, I believe >> this is a severe enough issue to warrant an advanced 3.6.1 release, and an >> official retraction of 3.6.0. >> Thoughts ? >> Even > > -- > Daniel Morissette > Mapgears Inc > T: +1 418-696-5056 #201 > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?
On 12/13/22 16:18, Even Rouault via gdal-dev wrote: Given the status of GeoPackage being the default format for QGIS, I believe this is a severe enough issue to warrant an advanced 3.6.1 release, and an official retraction of 3.6.0. Thoughts ? Yes, please. Kind Regards, Bas -- GPG Key ID: 4096R/6750F10AE88D4AF1 Fingerprint: 8182 DE41 7056 408D 6146 50D1 6750 F10A E88D 4AF1 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?
+1 for 3.6.1 with this fix. Daniel On 2022-12-13 10:18, Even Rouault via gdal-dev wrote: Hi, https://github.com/qgis/QGIS/issues/51188 has been brought to my attention. The issue is that the new background building of the RTree of GeoPackage files introduced in 3.6.0 didn't work well with committing transactions in between, which is easily triggered by ogr2ogr. All features were inserted but with default settings ~ 10% of them lacked an entry in the spatial index, which can be enough to break interactive display and workflows relying on spatial filtering. When using ogr2ogr, I believe the issue can only be seen when creating layers with more than 100 000 features, since that's the default value for the interval at which transactions are committed. I've a fix ready in https://github.com/OSGeo/gdal/pull/6911. The fix itself is a simple two-liner: https://github.com/OSGeo/gdal/pull/6911/commits/3f5f6225fe82e0c2e0241e4f66bfb861cdf4fe9d Given the status of GeoPackage being the default format for QGIS, I believe this is a severe enough issue to warrant an advanced 3.6.1 release, and an official retraction of 3.6.0. Thoughts ? Even -- Daniel Morissette Mapgears Inc T: +1 418-696-5056 #201 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?
+1 for the hotfix. -Jukka Rahkonen- -Alkuperäinen viesti- Lähettäjä: gdal-dev Puolesta Even Rouault via gdal-dev Lähetetty: tiistai 13. joulukuuta 2022 17.19 Vastaanottaja: gdal-dev@lists.osgeo.org Aihe: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ? Hi, https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqgis%2FQGIS%2Fissues%2F51188data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C2761f315f6bd4eb6c91f08dadd1d6bbd%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638065415688838146%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=I%2BM6iKoyeESbOVeR9csXkA6ATaKh4GNbRKldfhF0pJs%3Dreserved=0 has been brought to my attention. The issue is that the new background building of the RTree of GeoPackage files introduced in 3.6.0 didn't work well with committing transactions in between, which is easily triggered by ogr2ogr. All features were inserted but with default settings ~ 10% of them lacked an entry in the spatial index, which can be enough to break interactive display and workflows relying on spatial filtering. When using ogr2ogr, I believe the issue can only be seen when creating layers with more than 100 000 features, since that's the default value for the interval at which transactions are committed. I've a fix ready in https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOSGeo%2Fgdal%2Fpull%2F6911data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C2761f315f6bd4eb6c91f08dadd1d6bbd%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638065415688838146%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=sII0PhAG2VreyRvIn5e8cfjNLq5ZSlunDTJm9FpMdmU%3Dreserved=0. The fix itself is a simple two-liner: https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOSGeo%2Fgdal%2Fpull%2F6911%2Fcommits%2F3f5f6225fe82e0c2e0241e4f66bfb861cdf4fe9ddata=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C2761f315f6bd4eb6c91f08dadd1d6bbd%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638065415688838146%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=vpl4%2FHFkLRyjtBlgGULabBjsZuxLnhBzRK%2BOwFA6RAE%3Dreserved=0 Given the status of GeoPackage being the default format for QGIS, I believe this is a severe enough issue to warrant an advanced 3.6.1 release, and an official retraction of 3.6.0. Thoughts ? Even -- https://eur06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com%2Fdata=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C2761f315f6bd4eb6c91f08dadd1d6bbd%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638065415688838146%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=W1%2BeCIrV8xPlGUfw3h%2BeO4dGKryoyh94CwJWE0uQ1s4%3Dreserved=0 My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgdal-devdata=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C2761f315f6bd4eb6c91f08dadd1d6bbd%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638065415688838146%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=rX4HSCp1kbHMMMuttm8kKl91BRK9JE6K%2BQk3pvzc6Zs%3Dreserved=0 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Advanced 3.6.1 release and retraction of 3.6.0 ?
No objection from me! On Tue, Dec 13, 2022, 8:19 AM Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Hi, > > https://github.com/qgis/QGIS/issues/51188 has been brought to my > attention. The issue is that the new background building of the RTree of > GeoPackage files introduced in 3.6.0 didn't work well with committing > transactions in between, which is easily triggered by ogr2ogr. All > features were inserted but with default settings ~ 10% of them lacked an > entry in the spatial index, which can be enough to break interactive > display and workflows relying on spatial filtering. When using ogr2ogr, > I believe the issue can only be seen when creating layers with more than > 100 000 features, since that's the default value for the interval at > which transactions are committed. > > I've a fix ready in https://github.com/OSGeo/gdal/pull/6911. The fix > itself is a simple two-liner: > > https://github.com/OSGeo/gdal/pull/6911/commits/3f5f6225fe82e0c2e0241e4f66bfb861cdf4fe9d > > Given the status of GeoPackage being the default format for QGIS, I > believe this is a severe enough issue to warrant an advanced 3.6.1 > release, and an official retraction of 3.6.0. > > Thoughts ? > > Even > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
