Even Rouault via gdal-dev <gdal-dev@lists.osgeo.org> writes: > I'm considering removing from the source tree a number of third-party > libraries that we have vendored over the years: zlib, libpng, libjpeg, > giflib, liblerc
I'm basically strongly in favor of un-vendoring. I view vendoring as a bug, even if it is a workaround for other bugs. > I believe the main reason for having them vendored is now mostly > historical, dating back to times where there was no packaging system > on Windows. Now we have Conda-Forge or vcpkg, it is easy to have those > dependencies installed. From the pkgrsc viewpoint, I don't see any reasons why vendoring helps. It looks to me like the build is using normal dependencies, including zlib, jpeg, tiff, geotiff. But apparently not shapelib and json-c. $ ldd /usr/pkg/lib/libgdal.so /usr/pkg/lib/libgdal.so: -lgeos_c.1 => /usr/pkg/lib/libgeos_c.so.1 -lgeos.3.12.1 => /usr/pkg/lib/libgeos.so.3.12.1 -lstdc++.9 => /usr/lib/libstdc++.so.9 -lm.0 => /usr/lib/libm.so.0 -lc.12 => /usr/lib/libc.so.12 -lgcc_s.1 => /usr/lib/libgcc_s.so.1 -lwebp.7 => /usr/pkg/lib/libwebp.so.7 -lsharpyuv.0 => /usr/pkg/lib/libsharpyuv.so.0 -lpthread.1 => /usr/lib/libpthread.so.1 -lexpat.2 => /usr/lib/libexpat.so.2 -lxerces-c-3.2 => /usr/pkg/lib/libxerces-c-3.2.so -lopenjp2.7 => /usr/pkg/lib/libopenjp2.so.7 -lnetcdf.19 => /usr/pkg/lib/libnetcdf.so.19 -lexecinfo.0 => /usr/lib/libexecinfo.so.0 -lelf.2 => /usr/lib/libelf.so.2 -lhdf5_hl.200 => /usr/pkg/lib/libhdf5_hl.so.200 -lhdf5.200 => /usr/pkg/lib/libhdf5.so.200 -lsz.2 => /usr/pkg/lib/libsz.so.2 -laec.0 => /usr/pkg/lib/libaec.so.0 -lz.1 => /usr/lib/libz.so.1 -lbz2.1 => /usr/lib/libbz2.so.1 -lxml2.2 => /usr/pkg/lib/libxml2.so.2 -llzma.2 => /usr/lib/liblzma.so.2 -lcurl.4 => /usr/pkg/lib/libcurl.so.4 -lnghttp2.14 => /usr/pkg/lib/libnghttp2.so.14 -lidn2.0 => /usr/pkg/lib/libidn2.so.0 -lunistring.5 => /usr/pkg/lib/libunistring.so.5 -lintl.1 => /usr/lib/libintl.so.1 -lssl.15 => /usr/lib/libssl.so.15 -lcrypto.15 => /usr/lib/libcrypto.so.15 -lcrypt.1 => /lib/libcrypt.so.1 -lgssapi.12 => /usr/lib/libgssapi.so.12 -lkrb5.28 => /usr/lib/libkrb5.so.28 -lhx509.7 => /usr/lib/libhx509.so.7 -lasn1.10 => /usr/lib/libasn1.so.10 -lcom_err.8 => /usr/lib/libcom_err.so.8 -lroken.20 => /usr/lib/libroken.so.20 -lutil.7 => /usr/lib/libutil.so.7 -lwind.1 => /usr/lib/libwind.so.1 -lheimbase.2 => /usr/lib/libheimbase.so.2 -lheimntlm.6 => /usr/lib/libheimntlm.so.6 -lgif.7 => /usr/pkg/lib/libgif.so.7 -lgeotiff.5 => /usr/pkg/lib/libgeotiff.so.5 -lproj.19 => /usr/pkg/lib/libproj.so.19 -lsqlite3.1 => /usr/lib/libsqlite3.so.1 -ltiff.6 => /usr/pkg/lib/libtiff.so.6 -ljbig.2 => /usr/pkg/lib/libjbig.so.2 -ljpeg.8 => /usr/pkg/lib/libjpeg.so.8 -lpng16.16 => /usr/pkg/lib/libpng16.so.16 -lrt.1 => /usr/lib/librt.so.1 -lpcre.1 => /usr/pkg/lib/libpcre.so.1 -lqhull_r.8.0 => /usr/pkg/lib/libqhull_r.so.8.0 > For libjpeg, there was a particular history related to 12-bit JPEG So documentation of prereqs will say "you need jpeg, and you really should use jpeg-turbo so you have 12-bit support"? If so, sounds good. > Benefits of un-vendoring those libraries: > > - currently, we must take care of updating them regularly, in > particular to make sure they integrate the latest fixes for their > vulnerabilities. And not just that, but to have a gdal point release within days of any vendored upstream release that has a security fix, documented as such or not. That's a tall order, and ~nobody achieves it. This point is strongly in favor of unvendoring. Also, if there is a portablity or other bug fix, that needs to be imported and apoint release, so people can get those fixes. > Looking a bit around in different open source build recipees of GDAL > (Debian, Conda-Forge, vcpkg, OSGeo4W, gisinternals, rasterio-wheels), > those proposed changes should have modest impact, as they already > mostly use external libraries. What I've identified (I may have missed > things) to require changes from the maintainers of those distributions > to keep the same level of functionality: > > - gisinternals doesn't seem to have a liblerc build > > - rasterio-wheels doesn't seem to have libpng and giflib builds I am pretty sure pkgsrc will have little to no trouble. There's no liblerc, but it's optional and surely it can be packaged if somebody cares. > Potential candidates, but would remain in-tree for now: > > - libtiff: compulsory dependency. GDAL has been the main driver for > most libtiff development over the last 10 years, and GDAL autotest > suite tortures libtiff much more than libtiff own testsuite, hence > it is quite convenient to have the capability of vendoring it. Plus > the fact that for "staging codecs" (that is codecs not yet > integrated in official libtiff), currently JPEG-XL (a few years ago > this was the LERC codec), we can't build them against an external > libtiff. I see this as a workaround and it would be better if it weren't needed, but getting most of the gain and leaving the harder issues for later is a great strategy. FWIW pkgsrc's gdal build uses pkgsrc libtiff and I read normal tiffs just fine. > - shapelib. compulsory dependency. External default shapelib build > uses 32-bit file offset, whereas the internal shapelib is built with > 64-bit offset support (to use .DBF files > 2 GB). We don't have > build support for using it as external lib. Wow, that seems like quite a mess. I wonder if that's only for operating systems that have a LFS notion, vs those that just have 64-bit off_t? Overall, I would say unvendor as much as you feel you can do without real trouble, and then repeat, and from my viewpoint, the more and sooner the better, but I realize others have issues. _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev