Re: [gdal-dev] Un-vendoring a number of third-party libraries?
On 12/15/23 15:35, Even Rouault via gdal-dev wrote: Thoughts ? (given the length of the email, it should probably be formalized as a RFC. I'll do that, unless there is a massive uprising against the proposal...) LERC doesn't support big endian architectures currently, only using that on little endian architectures using the internal copy currently works as expected. Using the external library would require conditionals in the packaging which I'm not in favor of. Kind Regards, Bas -- GPG Key ID: 4096R/6750F10AE88D4AF1 Fingerprint: 8182 DE41 7056 408D 6146 50D1 6750 F10A E88D 4AF1 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Un-vendoring a number of third-party libraries?
Le 15/12/2023 à 15:49, Sebastiaan Couwenberg via gdal-dev a écrit : On 12/15/23 15:35, Even Rouault via gdal-dev wrote: Thoughts ? (given the length of the email, it should probably be formalized as a RFC. I'll do that, unless there is a massive uprising against the proposal...) LERC doesn't support big endian architectures currently, only using that on little endian architectures using the internal copy currently works as expected. Using the external library would require conditionals in the packaging which I'm not in favor of. Bas, - The Debian libtiff package already handles that conditional, so there must certainly be a way of using the same trick for the GDAL build recipee - The only user of LERC in GDAL (except for its libtiff internal cpoy) is the MRF driver. I doubt MRF is used widely except in Esri data centers... So even if you don't want to change the GDAL build recipee to include a conditional liblerc dependency, that wouldn't be the end of the world. Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Un-vendoring a number of third-party libraries?
> On Dec 15, 2023, at 8:35 AM, Even Rouault via gdal-dev > wrote: > > Thoughts ? (given the length of the email, it should probably be formalized > as a RFC. I'll do that, unless there is a massive uprising against the > proposal...) If the scope of this were to unvendor just these four (libjpeg, libpng, zlib, giflib), I think it is enough to start, but it brings up the question of whether or not JPEG, PNG, and GIF support are hard dependencies in GDAL afterward. They're always available no matter how you build now, so would we make them hard dependencies or relax them to optional after unvendoring? I assume zlib would move to a hard dependency of GDAL. Howard ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Un-vendoring a number of third-party libraries?
If the scope of this were to unvendor just these four (libjpeg, libpng, zlib, giflib), I think it is enough to start, but it brings up the question of whether or not JPEG, PNG, and GIF support are hard dependencies in GDAL afterward. They're always available no matter how you build now, so would we make them hard dependencies or relax them to optional after unvendoring? I assume zlib would move to a hard dependency of GDAL. I would let libjpeg, libpng, giflib, liblerc as optional dependencies. Our CMake scripts already handle them as optional. Someone doing a GDAL build only for vector might want to skip them. But strongly recommended at least for libjpeg and libpng, since beyond being standalone formats, they are also used for example for GeoPackage, MBTiles, MRF, OGCAPI/WMS/WMTS, etc. Yes zlib would move to a hard dependency, like PROJ (and if you build PROJ with libtiff or libcurl support, you must certainly already have zlib as it is a canonical dependency of those). There are lots of places in the code base where we assume deflate compression/decompression to be available. -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Un-vendoring a number of third-party libraries?
On Fri, 15 Dec 2023, Even Rouault via gdal-dev wrote: If the scope of this were to unvendor just these four (libjpeg, libpng, zlib, giflib), I think it is enough to start, but it brings up the question of whether or not JPEG, PNG, and GIF support are hard dependencies in GDAL afterward. They're always available no matter how you build now, so would we make them hard dependencies or relax them to optional after unvendoring? I assume zlib would move to a hard dependency of GDAL. I would let libjpeg, libpng, giflib, liblerc as optional dependencies. Our CMake scripts already handle them as optional. Someone doing a GDAL build only for vector might want to skip them. But strongly recommended at least for libjpeg and libpng, since beyond being standalone formats, they are also used for example for GeoPackage, MBTiles, MRF, OGCAPI/WMS/WMTS, etc. Yes zlib would move to a hard dependency, like PROJ (and if you build PROJ with libtiff or libcurl support, you must certainly already have zlib as it is a canonical dependency of those). There are lots of places in the code base where we assume deflate compression/decompression to be available. I don't see a problem but libpng also uses zlib. Do we need to do anything to ensure that gdal uses the same zlib as libpng ? -- Andrew C. Aitchison Kendal, UK and...@aitchison.me.uk___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Un-vendoring a number of third-party libraries?
Andrew, I don't see a problem but libpng also uses zlib. Do we need to do anything to ensure that gdal uses the same zlib as libpng ? It is not the role to GDAL to check this (there's no portable way of doing that AFAIK), but it is obviously the responsibility of the user to make sure that all libraries (libpng, libcurl, libtiff, etc. and GDAL) that link to zlib link to the same one. This is basic dependency hygiene. Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Un-vendoring a number of third-party libraries?
Even Rouault via gdal-dev writes: > I'm considering removing from the source tree a number of third-party > libraries that we have vendored over the years: zlib, libpng, libjpeg, > giflib, liblerc I'm basically strongly in favor of un-vendoring. I view vendoring as a bug, even if it is a workaround for other bugs. > I believe the main reason for having them vendored is now mostly > historical, dating back to times where there was no packaging system > on Windows. Now we have Conda-Forge or vcpkg, it is easy to have those > dependencies installed. From the pkgrsc viewpoint, I don't see any reasons why vendoring helps. It looks to me like the build is using normal dependencies, including zlib, jpeg, tiff, geotiff. But apparently not shapelib and json-c. $ ldd /usr/pkg/lib/libgdal.so /usr/pkg/lib/libgdal.so: -lgeos_c.1 => /usr/pkg/lib/libgeos_c.so.1 -lgeos.3.12.1 => /usr/pkg/lib/libgeos.so.3.12.1 -lstdc++.9 => /usr/lib/libstdc++.so.9 -lm.0 => /usr/lib/libm.so.0 -lc.12 => /usr/lib/libc.so.12 -lgcc_s.1 => /usr/lib/libgcc_s.so.1 -lwebp.7 => /usr/pkg/lib/libwebp.so.7 -lsharpyuv.0 => /usr/pkg/lib/libsharpyuv.so.0 -lpthread.1 => /usr/lib/libpthread.so.1 -lexpat.2 => /usr/lib/libexpat.so.2 -lxerces-c-3.2 => /usr/pkg/lib/libxerces-c-3.2.so -lopenjp2.7 => /usr/pkg/lib/libopenjp2.so.7 -lnetcdf.19 => /usr/pkg/lib/libnetcdf.so.19 -lexecinfo.0 => /usr/lib/libexecinfo.so.0 -lelf.2 => /usr/lib/libelf.so.2 -lhdf5_hl.200 => /usr/pkg/lib/libhdf5_hl.so.200 -lhdf5.200 => /usr/pkg/lib/libhdf5.so.200 -lsz.2 => /usr/pkg/lib/libsz.so.2 -laec.0 => /usr/pkg/lib/libaec.so.0 -lz.1 => /usr/lib/libz.so.1 -lbz2.1 => /usr/lib/libbz2.so.1 -lxml2.2 => /usr/pkg/lib/libxml2.so.2 -llzma.2 => /usr/lib/liblzma.so.2 -lcurl.4 => /usr/pkg/lib/libcurl.so.4 -lnghttp2.14 => /usr/pkg/lib/libnghttp2.so.14 -lidn2.0 => /usr/pkg/lib/libidn2.so.0 -lunistring.5 => /usr/pkg/lib/libunistring.so.5 -lintl.1 => /usr/lib/libintl.so.1 -lssl.15 => /usr/lib/libssl.so.15 -lcrypto.15 => /usr/lib/libcrypto.so.15 -lcrypt.1 => /lib/libcrypt.so.1 -lgssapi.12 => /usr/lib/libgssapi.so.12 -lkrb5.28 => /usr/lib/libkrb5.so.28 -lhx509.7 => /usr/lib/libhx509.so.7 -lasn1.10 => /usr/lib/libasn1.so.10 -lcom_err.8 => /usr/lib/libcom_err.so.8 -lroken.20 => /usr/lib/libroken.so.20 -lutil.7 => /usr/lib/libutil.so.7 -lwind.1 => /usr/lib/libwind.so.1 -lheimbase.2 => /usr/lib/libheimbase.so.2 -lheimntlm.6 => /usr/lib/libheimntlm.so.6 -lgif.7 => /usr/pkg/lib/libgif.so.7 -lgeotiff.5 => /usr/pkg/lib/libgeotiff.so.5 -lproj.19 => /usr/pkg/lib/libproj.so.19 -lsqlite3.1 => /usr/lib/libsqlite3.so.1 -ltiff.6 => /usr/pkg/lib/libtiff.so.6 -ljbig.2 => /usr/pkg/lib/libjbig.so.2 -ljpeg.8 => /usr/pkg/lib/libjpeg.so.8 -lpng16.16 => /usr/pkg/lib/libpng16.so.16 -lrt.1 => /usr/lib/librt.so.1 -lpcre.1 => /usr/pkg/lib/libpcre.so.1 -lqhull_r.8.0 => /usr/pkg/lib/libqhull_r.so.8.0 > For libjpeg, there was a particular history related to 12-bit JPEG So documentation of prereqs will say "you need jpeg, and you really should use jpeg-turbo so you have 12-bit support"? If so, sounds good. > Benefits of un-vendoring those libraries: > > - currently, we must take care of updating them regularly, in > particular to make sure they integrate the latest fixes for their > vulnerabilities. And not just that, but to have a gdal point release within days of any vendored upstream release that has a security fix, documented as such or not. That's a tall order, and ~nobody achieves it. This point is strongly in favor of unvendoring. Also, if there is a portablity or other bug fix, that needs to be imported and apoint release, so people can get those fixes. > Looking a bit around in different open source build recipees of GDAL > (Debian, Conda-Forge, vcpkg, OSGeo4W, gisinternals, rasterio-wheels), > those proposed changes should have modest impact, as they already > mostly use external libraries. What I've identified (I may have missed > things) to require changes from the maintainers of those distributions > to keep the same level of functionality: > > - gisinternals doesn't seem to have a liblerc build > > - rasterio-wheels doesn't seem to have libpng and giflib builds I am pretty sure pkgsrc will have little to no trouble. There's no liblerc, but it's optional and surely it can be packaged if somebody cares. > Potential candidates, but would remain in-tree for now: > > - libtiff: compulsory dependency. GDAL has been the main driver for > most libtiff development over the last 10 years, and GDAL autotest > suite tortures libtiff much more than libtiff own testsuite, hence > it is quite convenie
Re: [gdal-dev] Un-vendoring a number of third-party libraries?
+1 for un-vendoring. Long term, I think that will be a big win. Looks like others have covered all of the issues that I can think of. On Fri, Dec 15, 2023 at 4:45 PM Greg Troxel via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Even Rouault via gdal-dev writes: > > > I'm considering removing from the source tree a number of third-party > > libraries that we have vendored over the years: zlib, libpng, libjpeg, > > giflib, liblerc > > I'm basically strongly in favor of un-vendoring. I view vendoring as a > bug, even if it is a workaround for other bugs. > > > I believe the main reason for having them vendored is now mostly > > historical, dating back to times where there was no packaging system > > on Windows. Now we have Conda-Forge or vcpkg, it is easy to have those > > dependencies installed. > > From the pkgrsc viewpoint, I don't see any reasons why vendoring helps. > It looks to me like the build is using normal dependencies, including > zlib, jpeg, tiff, geotiff. But apparently not shapelib and json-c. > > $ ldd /usr/pkg/lib/libgdal.so > /usr/pkg/lib/libgdal.so: > -lgeos_c.1 => /usr/pkg/lib/libgeos_c.so.1 > -lgeos.3.12.1 => /usr/pkg/lib/libgeos.so.3.12.1 > -lstdc++.9 => /usr/lib/libstdc++.so.9 > -lm.0 => /usr/lib/libm.so.0 > -lc.12 => /usr/lib/libc.so.12 > -lgcc_s.1 => /usr/lib/libgcc_s.so.1 > -lwebp.7 => /usr/pkg/lib/libwebp.so.7 > -lsharpyuv.0 => /usr/pkg/lib/libsharpyuv.so.0 > -lpthread.1 => /usr/lib/libpthread.so.1 > -lexpat.2 => /usr/lib/libexpat.so.2 > -lxerces-c-3.2 => /usr/pkg/lib/libxerces-c-3.2.so > -lopenjp2.7 => /usr/pkg/lib/libopenjp2.so.7 > -lnetcdf.19 => /usr/pkg/lib/libnetcdf.so.19 > -lexecinfo.0 => /usr/lib/libexecinfo.so.0 > -lelf.2 => /usr/lib/libelf.so.2 > -lhdf5_hl.200 => /usr/pkg/lib/libhdf5_hl.so.200 > -lhdf5.200 => /usr/pkg/lib/libhdf5.so.200 > -lsz.2 => /usr/pkg/lib/libsz.so.2 > -laec.0 => /usr/pkg/lib/libaec.so.0 > -lz.1 => /usr/lib/libz.so.1 > -lbz2.1 => /usr/lib/libbz2.so.1 > -lxml2.2 => /usr/pkg/lib/libxml2.so.2 > -llzma.2 => /usr/lib/liblzma.so.2 > -lcurl.4 => /usr/pkg/lib/libcurl.so.4 > -lnghttp2.14 => /usr/pkg/lib/libnghttp2.so.14 > -lidn2.0 => /usr/pkg/lib/libidn2.so.0 > -lunistring.5 => /usr/pkg/lib/libunistring.so.5 > -lintl.1 => /usr/lib/libintl.so.1 > -lssl.15 => /usr/lib/libssl.so.15 > -lcrypto.15 => /usr/lib/libcrypto.so.15 > -lcrypt.1 => /lib/libcrypt.so.1 > -lgssapi.12 => /usr/lib/libgssapi.so.12 > -lkrb5.28 => /usr/lib/libkrb5.so.28 > -lhx509.7 => /usr/lib/libhx509.so.7 > -lasn1.10 => /usr/lib/libasn1.so.10 > -lcom_err.8 => /usr/lib/libcom_err.so.8 > -lroken.20 => /usr/lib/libroken.so.20 > -lutil.7 => /usr/lib/libutil.so.7 > -lwind.1 => /usr/lib/libwind.so.1 > -lheimbase.2 => /usr/lib/libheimbase.so.2 > -lheimntlm.6 => /usr/lib/libheimntlm.so.6 > -lgif.7 => /usr/pkg/lib/libgif.so.7 > -lgeotiff.5 => /usr/pkg/lib/libgeotiff.so.5 > -lproj.19 => /usr/pkg/lib/libproj.so.19 > -lsqlite3.1 => /usr/lib/libsqlite3.so.1 > -ltiff.6 => /usr/pkg/lib/libtiff.so.6 > -ljbig.2 => /usr/pkg/lib/libjbig.so.2 > -ljpeg.8 => /usr/pkg/lib/libjpeg.so.8 > -lpng16.16 => /usr/pkg/lib/libpng16.so.16 > -lrt.1 => /usr/lib/librt.so.1 > -lpcre.1 => /usr/pkg/lib/libpcre.so.1 > -lqhull_r.8.0 => /usr/pkg/lib/libqhull_r.so.8.0 > > > For libjpeg, there was a particular history related to 12-bit JPEG > > So documentation of prereqs will say "you need jpeg, and you really > should use jpeg-turbo so you have 12-bit support"? If so, sounds good. > > > Benefits of un-vendoring those libraries: > > > > - currently, we must take care of updating them regularly, in > > particular to make sure they integrate the latest fixes for their > > vulnerabilities. > > And not just that, but to have a gdal point release within days of any > vendored upstream release that has a security fix, documented as such or > not. That's a tall order, and ~nobody achieves it. This point is > strongly in favor of unvendoring. > > Also, if there is a portablity or other bug fix, that needs to be > imported and apoint release, so people can get those fixes. > > > Looking a bit around in different open source build recipees of GDAL > > (Debian, Conda-Forge, vcpkg, OSGeo4W, gisinternals, rasterio-wheels), > > those proposed changes should have modest impact, as they already > > mostly use external libraries. What I've identified (I may have missed > > things) to require changes from the maintainers of those distributions > > to keep the same level of functionality: > > > > - gisinternals doesn't seem to have a liblerc build > > > > - rasterio-wheels doesn't seem to have libpng and giflib builds > > I am pretty sur
Re: [gdal-dev] Un-vendoring a number of third-party libraries?
On Sat, 16 Dec 2023, 1:03 am Even Rouault via gdal-dev, < gdal-dev@lists.osgeo.org> wrote: > > Le 15/12/2023 à 15:49, Sebastiaan Couwenberg via gdal-dev a écrit : > > On 12/15/23 15:35, Even Rouault via gdal-dev wrote: > >> Thoughts ? (given the length of the email, it should probably be > >> formalized as a RFC. I'll do that, unless there is a massive uprising > >> against the proposal...) > > > > LERC doesn't support big endian architectures currently, only using > > that on little endian architectures using the internal copy currently > > works as expected. Using the external library would require > > conditionals in the packaging which I'm not in favor of. > Bas, > > - The Debian libtiff package already handles that conditional, so there > must certainly be a way of using the same trick for the GDAL build recipee > > - The only user of LERC in GDAL (except for its libtiff internal cpoy) > is the MRF driver. I doubt MRF is used widely except in Esri data > centers... So even if you don't want to change the GDAL build recipee to > include a conditional liblerc dependency, that wouldn't be the end of > the world. > Some ESRI image server layers are served up using lerc compression, eg the landcover tiles: https://tiledimageservices.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/Esri_2020_Land_Cover_V2/ImageServer?f=pjson I was planning on doing some future work in qgis to support reading these layers by using gdal to do the lerc decompression, so I'd be disappointed if the lerc compression support is dropped. Nyall > Even > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Un-vendoring a number of third-party libraries?
Le 29/12/2023 à 22:10, Nyall Dawson a écrit : On Sat, 16 Dec 2023, 1:03 am Even Rouault via gdal-dev, wrote: Le 15/12/2023 à 15:49, Sebastiaan Couwenberg via gdal-dev a écrit : > On 12/15/23 15:35, Even Rouault via gdal-dev wrote: >> Thoughts ? (given the length of the email, it should probably be >> formalized as a RFC. I'll do that, unless there is a massive uprising >> against the proposal...) > > LERC doesn't support big endian architectures currently, only using > that on little endian architectures using the internal copy currently > works as expected. Using the external library would require > conditionals in the packaging which I'm not in favor of. Bas, - The Debian libtiff package already handles that conditional, so there must certainly be a way of using the same trick for the GDAL build recipee - The only user of LERC in GDAL (except for its libtiff internal cpoy) is the MRF driver. I doubt MRF is used widely except in Esri data centers... So even if you don't want to change the GDAL build recipee to include a conditional liblerc dependency, that wouldn't be the end of the world. Some ESRI image server layers are served up using lerc compression, eg the landcover tiles: https://tiledimageservices.arcgis.com/P3ePLMYs2RVChkJx/arcgis/rest/services/Esri_2020_Land_Cover_V2/ImageServer?f=pjson I was planning on doing some future work in qgis to support reading these layers by using gdal to do the lerc decompression, so I'd be disappointed if the lerc compression support is dropped. Hi Nyall, Interesting. Digging a bit more, I now see that the MRF driver can open Lerc blobs too (independently of the MRF extra stuff), so that can potentially be a way to get Lerc decompression working. And actually there's is a test in the WMS driver that uses MRF/LERC (https://github.com/OSGeo/gdal/blob/master/autotest/gdrivers/wms.py#L872) which might be just what you need. It would be nice if the GDAL Debian package could link liblerc in the future. Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
