On Fri, 24 Jun 2022 15:14:23 -0400 Nathan Hartman <hartman.nat...@gmail.com> wrote:
> On Thu, Jun 23, 2022 at 5:58 AM Sergei Trofimovich <sly...@gmail.com> wrote: > > > > Hi subversion developers! > > > > I'm working on fixing parallel build (or install) faliures in upstream > > packages used in NixOS linux distribution. Mostly to speed package builds > > up on continuous build farm. > > > > That usually means that > > $ ./configure && make && make install > > is changed to > > $ ./configure && make -j$(nproc) && make -j$(nproc) install > > > > To make the error more obvioud GNU make recently added --shuffle option > > to better expose missing dependencies across Makefile targets: > > https://savannah.gnu.org/bugs/index.php?62100 > > > > 'subversion-1.14.2' came as one of the packages that sometimes fails on > > 'make install' > > > > There is a build log snippet that exposed missing install dependencies: > > > > $ ./configure --disable-static --prefix=/<<NIX>>/subversion-1.14.2 > > --bindir=/<<NIX>>/subversion-1.14.2/bin > > --sbindir=/<<NIX>>/subversion-1.14.2/sbin > > --includedir=/<<NIX>>/subversion-1.14.2-dev/include > > --oldincludedir=/<<NIX>>/subversion-1.14.2-dev/include > > --mandir=/<<NIX>>/subversion-1.14.2-man/share/man > > --infodir=/<<NIX>>/subversion-1.14.2/share/info > > --docdir=/<<NIX>>/subversion-1.14.2/share/doc/subversion > > --libdir=/<<NIX>>/subversion-1.14.2/lib > > --libexecdir=/<<NIX>>/subversion-1.14.2/libexec > > --localedir=/<<NIX>>/subversion-1.14.2/share/locale --with-berkeley-db > > --without-apxs --without-swig --without-sasl > > --with-serf=/<<NIX>>/serf-1.3.9 --with-zlib=/<<NIX>>/zlib-1.2.12-dev > > --with-sqlite=/<<NIX>>/sqlite-3.38.5-dev --build=x86_64-unknown-linux-gnu > > --host=x86_64-unknown-linux-gnu > > ... > > $ make -j16 -l16 SHELL=/<<NIX>>/bash-5.1-p16/bin/bash > > APACHE_LIBEXECDIR=/<<NIX>>/subversion-1.14.2/modules > > ... > > $ make --shuffle APACHE_LIBEXECDIR=/<<NIX>>/subversion-1.14.2/modules > > pkgconfigdir=/<<NIX>>/subversion-1.14.2-dev/lib/pkgconfig > > m4datadir=/<<NIX>>/subversion-1.14.2-dev/share/aclocal > > aclocaldir=/<<NIX>>/subversion-1.14.2-dev/share/aclocal install > > test -d /<<NIX>>/subversion-1.14.2-dev/include/subversion-1 || \ > > /<<NIX>>/coreutils-9.1/bin/install -c -d > > /<<NIX>>/subversion-1.14.2-dev/include/subversion-1 > > (subversion/svnversion/svnversion . 2> /dev/null || \ > > svnversion . 2> /dev/null || \ > > echo "unknown"; \ > > ) > /<<NIX>>/subversion-1.14.2-dev/include/subversion-1/svn-revision.txt > > /<<NIX>>/coreutils-9.1/bin/install -c -d /<<NIX>>/subversion-1.14.2/lib > > /<<NIX>>/subversion-1.14.2/share/pkgconfig > > cd subversion/libsvn_fs_base ; /<<NIX>>/bash-5.1-p16/bin/bash > > "/build/subversion/libtool" --mode=install > > /<<NIX>>/coreutils-9.1/bin/install -c libsvn_fs_base-1.la > > /<<NIX>>/subversion-1.14.2/lib/libsvn_fs_base-1.la > > libtool: warning: relinking 'libsvn_fs_base-1.la' > > libtool: install: (cd /build/subversion/subversion/libsvn_fs_base; > > /<<NIX>>/bash-5.1-p16/bin/bash "/build/subversion/libtool" --tag CC > > --silent --mode=relink gcc -shared -g -O2 -g -O2 > > -L/<<NIX>>/openssl-1.1.1o/lib -L/<<NIX>>/db-5.3.28/lib > > -L/<<NIX>>/expat-2.4.8/lib -L/<<NIX>>/serf-1.3.9/lib -rpath > > /<<NIX>>/subversion-1.14.2/lib -version-info 0 -Wl,--no-undefined -o > > libsvn_fs_base-1.la bdb/bdb-err.lo bdb/bdb_compat.lo bdb/changes-table.lo > > bdb/checksum-reps-table.lo bdb/copies-table.lo bdb/dbt.lo bdb/env.lo > > bdb/lock-tokens-table.lo bdb/locks-table.lo bdb/miscellaneous-table.lo > > bdb/node-origins-table.lo bdb/nodes-table.lo bdb/reps-table.lo > > bdb/rev-table.lo bdb/strings-table.lo bdb/txn-table.lo bdb/uuids-table.lo > > dag.lo err.lo fs.lo id.lo key-gen.lo lock.lo node-rev.lo reps-strings.lo > > revs-txns.lo trail.lo tree.lo util/fs_skels.lo uuid.lo > > ../../subversion/libsvn_delta/libsvn_delta-1.la > > ../../subversion/libsvn_subr/libsvn_subr-1.la -L/<<NIX>>/apr-util-1.6.1/lib > > -laprutil-1 -L/<<N IX>>/apr-1.7.0/lib -lapr-1 -ldb-5.3 ../../subversion/libsvn_fs_util/libsvn_fs_util-1.la ) > > /<<NIX>>/binutils-2.38/bin/ld: cannot find -lsvn_delta-1: No such file > > or directory > > /<<NIX>>/binutils-2.38/bin/ld: cannot find -lsvn_fs_util-1: No such > > file or directory > > collect2: error: ld returned 1 exit status > > libtool: error: error: relink 'libsvn_fs_base-1.la' with the above > > command before installing it > > make: *** [build-outputs.mk:1422: install-bdb-lib] Error 1 > > shuffle=1656015674 > > > > The failure as I understand it: > > > > 'install-bdb-lib' install target assumes that '-lsvn_delta-1' and > > '-lsvn_fs_util-1' > > are already installed into 'DESTDIR'. But they are not. 'libtool' fails to > > relink > > 'libsvn_fs_base' against 'DESTDIR'. > > > > I think the dependencies usually happen to be present if > > 'install-fsmod-lib' target > > was lucky to be executed first. But nothing in 'Makefile.in' or > > 'build-outputs.mk' > > seems to have an equivalent of dependency like: > > > > # hypothetical fix > > install-bdb-lib : install-fsmod-lib > > > > It looks like 'build.conf' does contain some build dependency information. > > > > I also see special cases like fs-lib to handle some of install deps: > > > > # manual > > Makefile.in:SVN_FS_LIB_INSTALL_DEPS = @SVN_FS_LIB_INSTALL_DEPS@ > > > > # generated: > > build-outputs.mk:install-ramod-lib: $(SVN_FS_LIB_INSTALL_DEPS) > > > > Would it make sense to add 'install-bdb-lib : install-fsmod-lib' dependency > > explicitly? > > > > Thank you! > > > > -- > > > > Sergei > > > Hi Sergei, > > This is a very useful feature for GNU Make to have! This is the first > I've heard of it and it should allow many, many projects, not just > Subversion, to fix these kinds of pesky issues in parallel builds. > > It seems like this landed in Make literally a few days ago so isn't in > any released version. Is that correct? That is correct: there are no releases or snapshots of GNU make with a feature yet. I'm using locally built make from tip of the git tree. > If so, I don't currently have a (convenient) setup to experiment with > this at the moment; are you able to just hack your hypothetical fix > into your SVN makefile, i.e., `install-bdb-lib : install-fsmod-lib`, > and run make with shuffle=1656015674 and tell us if that appears to > fix it? Then I can go digging to see where to add it correctly. Before the hack my installs were failing 60% of the time (6 of 10 attmpts failed). After the change below 100 of 100 install succeeded. I had to add two depdendencies to get there: install-bdb-lib: install-fsmod-lib install-serf-lib: install-fsmod-lib The change itself (with linkage errors they fix): $ svn diff Index: Makefile.in =================================================================== --- Makefile.in (revision 1902174) +++ Makefile.in (working copy) @@ -431,6 +431,36 @@ @INCLUDE_OUTPUTS@ +# A hack to test https://issues.apache.org/jira/browse/SVN-4901: +# orkaround missing dependencies on installing libtool libraries +# that depend on other libraries: +# +## case 1: fs_base depend on fs_util and delta: +# +# libtool: install: (cd /build/subversion/subversion/libsvn_fs_base; bash "/build/subversion/libtool" \ +# --tag CC --silent --mode=relink gcc -shared -g -O2 -g -O2 -L...-openssl-1.1.1p/lib \ +# -L...-db-5.3.28/lib -L...-expat-2.4.8/lib -L...-serf-1.3.9/lib -rpath ...-subversion-1.14.2/lib \ +# -version-info 0 -Wl,--no-undefined \ +# -o libsvn_fs_base-1.la bdb/bdb-err.lo \ +# ... ../../subversion/libsvn_delta/libsvn_delta-1.la ../../subversion/libsvn_subr/libsvn_subr-1.la \ +# -L...-apr-util-1.6.1/lib -laprutil-1 -L...-apr-1.7.0/lib -lapr-1 -ldb-5.3 \ +# ../../subversion/libsvn_fs_util/libsvn_fs_util-1.la ) +# ld: cannot find -lsvn_delta-1: No such file or directory +# ld: cannot find -lsvn_fs_util-1: No such file or directory +install-bdb-lib: install-fsmod-lib +# +## case 2: ra-serf depends on delta: +# +# libtool: install: (cd /build/subversion/subversion/libsvn_ra_serf; bash "/build/subversion/libtool" \ +# --tag CC --silent --mode=relink gcc -shared -g -O2 -g -O2 -L...-openssl-1.1.1p/lib \ +# -L...-db-5.3.28/lib -L...-expat-2.4.8/lib -L...-serf-1.3.9/lib -rpath ...-subversion-1.14.2/lib \ +# -version-info 0 -Wl,--no-undefined +# -o libsvn_ra_serf-1.la blame.lo \ +# ... xml.lo ../../subversion/libsvn_delta/libsvn_delta-1.la ../../subversion/libsvn_subr/libsvn_subr-1.la \ +# -L...-apr-util-1.6.1/lib -laprutil-1 -L...-apr-1.7.0/lib -lapr-1 -lserf-1 -L...-zlib-1.2.12-dev/lib -lz ) +# ld: cannot find -lsvn_delta-1: No such file or directory +install-serf-lib: install-fsmod-lib + local-all: @BUILD_RULES@ @TRANSFORM_LIBTOOL_SCRIPTS@ transform-libtool-scripts: @BUILD_RULES@ > Thanks for bringing this (and the new/upcoming GNU Make feature) to > our attention. That's exciting news. Glad you see it useful! Thank you! -- Sergei
