+1 for un-vendoring. Long term, I think that will be a big win. Looks like others have covered all of the issues that I can think of.
On Fri, Dec 15, 2023 at 4:45 PM Greg Troxel via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Even Rouault via gdal-dev <gdal-dev@lists.osgeo.org> writes: > > > I'm considering removing from the source tree a number of third-party > > libraries that we have vendored over the years: zlib, libpng, libjpeg, > > giflib, liblerc > > I'm basically strongly in favor of un-vendoring. I view vendoring as a > bug, even if it is a workaround for other bugs. > > > I believe the main reason for having them vendored is now mostly > > historical, dating back to times where there was no packaging system > > on Windows. Now we have Conda-Forge or vcpkg, it is easy to have those > > dependencies installed. > > From the pkgrsc viewpoint, I don't see any reasons why vendoring helps. > It looks to me like the build is using normal dependencies, including > zlib, jpeg, tiff, geotiff. But apparently not shapelib and json-c. > > $ ldd /usr/pkg/lib/libgdal.so > /usr/pkg/lib/libgdal.so: > -lgeos_c.1 => /usr/pkg/lib/libgeos_c.so.1 > -lgeos.3.12.1 => /usr/pkg/lib/libgeos.so.3.12.1 > -lstdc++.9 => /usr/lib/libstdc++.so.9 > -lm.0 => /usr/lib/libm.so.0 > -lc.12 => /usr/lib/libc.so.12 > -lgcc_s.1 => /usr/lib/libgcc_s.so.1 > -lwebp.7 => /usr/pkg/lib/libwebp.so.7 > -lsharpyuv.0 => /usr/pkg/lib/libsharpyuv.so.0 > -lpthread.1 => /usr/lib/libpthread.so.1 > -lexpat.2 => /usr/lib/libexpat.so.2 > -lxerces-c-3.2 => /usr/pkg/lib/libxerces-c-3.2.so > -lopenjp2.7 => /usr/pkg/lib/libopenjp2.so.7 > -lnetcdf.19 => /usr/pkg/lib/libnetcdf.so.19 > -lexecinfo.0 => /usr/lib/libexecinfo.so.0 > -lelf.2 => /usr/lib/libelf.so.2 > -lhdf5_hl.200 => /usr/pkg/lib/libhdf5_hl.so.200 > -lhdf5.200 => /usr/pkg/lib/libhdf5.so.200 > -lsz.2 => /usr/pkg/lib/libsz.so.2 > -laec.0 => /usr/pkg/lib/libaec.so.0 > -lz.1 => /usr/lib/libz.so.1 > -lbz2.1 => /usr/lib/libbz2.so.1 > -lxml2.2 => /usr/pkg/lib/libxml2.so.2 > -llzma.2 => /usr/lib/liblzma.so.2 > -lcurl.4 => /usr/pkg/lib/libcurl.so.4 > -lnghttp2.14 => /usr/pkg/lib/libnghttp2.so.14 > -lidn2.0 => /usr/pkg/lib/libidn2.so.0 > -lunistring.5 => /usr/pkg/lib/libunistring.so.5 > -lintl.1 => /usr/lib/libintl.so.1 > -lssl.15 => /usr/lib/libssl.so.15 > -lcrypto.15 => /usr/lib/libcrypto.so.15 > -lcrypt.1 => /lib/libcrypt.so.1 > -lgssapi.12 => /usr/lib/libgssapi.so.12 > -lkrb5.28 => /usr/lib/libkrb5.so.28 > -lhx509.7 => /usr/lib/libhx509.so.7 > -lasn1.10 => /usr/lib/libasn1.so.10 > -lcom_err.8 => /usr/lib/libcom_err.so.8 > -lroken.20 => /usr/lib/libroken.so.20 > -lutil.7 => /usr/lib/libutil.so.7 > -lwind.1 => /usr/lib/libwind.so.1 > -lheimbase.2 => /usr/lib/libheimbase.so.2 > -lheimntlm.6 => /usr/lib/libheimntlm.so.6 > -lgif.7 => /usr/pkg/lib/libgif.so.7 > -lgeotiff.5 => /usr/pkg/lib/libgeotiff.so.5 > -lproj.19 => /usr/pkg/lib/libproj.so.19 > -lsqlite3.1 => /usr/lib/libsqlite3.so.1 > -ltiff.6 => /usr/pkg/lib/libtiff.so.6 > -ljbig.2 => /usr/pkg/lib/libjbig.so.2 > -ljpeg.8 => /usr/pkg/lib/libjpeg.so.8 > -lpng16.16 => /usr/pkg/lib/libpng16.so.16 > -lrt.1 => /usr/lib/librt.so.1 > -lpcre.1 => /usr/pkg/lib/libpcre.so.1 > -lqhull_r.8.0 => /usr/pkg/lib/libqhull_r.so.8.0 > > > For libjpeg, there was a particular history related to 12-bit JPEG > > So documentation of prereqs will say "you need jpeg, and you really > should use jpeg-turbo so you have 12-bit support"? If so, sounds good. > > > Benefits of un-vendoring those libraries: > > > > - currently, we must take care of updating them regularly, in > > particular to make sure they integrate the latest fixes for their > > vulnerabilities. > > And not just that, but to have a gdal point release within days of any > vendored upstream release that has a security fix, documented as such or > not. That's a tall order, and ~nobody achieves it. This point is > strongly in favor of unvendoring. > > Also, if there is a portablity or other bug fix, that needs to be > imported and apoint release, so people can get those fixes. > > > Looking a bit around in different open source build recipees of GDAL > > (Debian, Conda-Forge, vcpkg, OSGeo4W, gisinternals, rasterio-wheels), > > those proposed changes should have modest impact, as they already > > mostly use external libraries. What I've identified (I may have missed > > things) to require changes from the maintainers of those distributions > > to keep the same level of functionality: > > > > - gisinternals doesn't seem to have a liblerc build > > > > - rasterio-wheels doesn't seem to have libpng and giflib builds > > I am pretty sure pkgsrc will have little to no trouble. There's no > liblerc, but it's optional and surely it can be packaged if somebody > cares. > > > Potential candidates, but would remain in-tree for now: > > > > - libtiff: compulsory dependency. GDAL has been the main driver for > > most libtiff development over the last 10 years, and GDAL autotest > > suite tortures libtiff much more than libtiff own testsuite, hence > > it is quite convenient to have the capability of vendoring it. Plus > > the fact that for "staging codecs" (that is codecs not yet > > integrated in official libtiff), currently JPEG-XL (a few years ago > > this was the LERC codec), we can't build them against an external > > libtiff. > > I see this as a workaround and it would be better if it weren't needed, > but getting most of the gain and leaving the harder issues for later is > a great strategy. FWIW pkgsrc's gdal build uses pkgsrc libtiff and I > read normal tiffs just fine. > > > - shapelib. compulsory dependency. External default shapelib build > > uses 32-bit file offset, whereas the internal shapelib is built with > > 64-bit offset support (to use .DBF files > 2 GB). We don't have > > build support for using it as external lib. > > Wow, that seems like quite a mess. I wonder if that's only for > operating systems that have a LFS notion, vs those that just have 64-bit > off_t? > > > Overall, I would say unvendor as much as you feel you can do without > real trouble, and then repeat, and from my viewpoint, the more and > sooner the better, but I realize others have issues. > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev