testing dist tarballs
Hi, On 2023-05-23 14:51:03 -0400, Tom Lane wrote: > Andres Freund writes: > > I guess I need to go and check how long the "release" tarball generation > > takes... > > It's quick except for the documentation-generating steps. Maybe > we could test that part only once? First thing I noticed that 'make dist' doesn't work in a vpath, failing in a somewhat obscure way (likely because in a vpath build the the copy from the source dir doesn't include GNUMakefile). Do we expect it to work? Besides docs, the slowest part appears to be gzip --best and then bzip2, as those runs serially and takes 11 and 13 seconds respectively here... The first thing I tried was: make -j8 dist GZIP=pigz BZIP2=pbzip2 unfortunately that results in pigz: abort: cannot provide files in GZIP environment variable echo GZIP=pigz >> src/Makefile.custom echo BZIP2=pbzip2 >> src/Makefile.custom reduces that to real1m6.472s user1m28.316s sys 0m5.340s real0m54.811s user1m42.078s sys 0m6.183s still not great... OTOH, we currently already build the docs as part of the CompilerWarnings test. I don't think there's a reason to test that twice? For me make distcheck currently fails: In file included from ../../src/include/postgres.h:46, from hashfn.c:24: ../../src/include/utils/elog.h:79:10: fatal error: utils/errcodes.h: No such file or directory 79 | #include "utils/errcodes.h" | ^~ compilation terminated. make[3]: *** [: hashfn.o] Error 1 at first I thought it was due to my use of -j8 - but it doesn't even work without that. That's due to MAKELEVEL: submake-generated-headers: ifndef NO_GENERATED_HEADERS ifeq ($(MAKELEVEL),0) $(MAKE) -C $(top_builddir)/src/backend generated-headers endif endif So the distcheck target needs to reset MAKELEVEL=0 - unless somebody has a better idea? Separately, it's somewhat confusing that we include errcodes.h etc in src/backend/utils, rather than its final location, in src/include/utils. It works, even without perl, because copying the file doesn't require perl, it's just generating it... Greetings, Andres Freund
Re: testing dist tarballs
Andres Freund writes: > First thing I noticed that 'make dist' doesn't work in a vpath, failing in a > somewhat obscure way (likely because in a vpath build the the copy from the > source dir doesn't include GNUMakefile). Do we expect it to work? Don't see how it could possibly be useful in a vpath, because you'd have the real source files and the generated files in different trees. regards, tom lane
Re: testing dist tarballs
Re: Andres Freund > That's due to MAKELEVEL: > > submake-generated-headers: > ifndef NO_GENERATED_HEADERS > ifeq ($(MAKELEVEL),0) > $(MAKE) -C $(top_builddir)/src/backend generated-headers > endif > endif > > So the distcheck target needs to reset MAKELEVEL=0 - unless somebody has a > better idea? Fwiw, I've had that problem as well in the Debian packages where debian/rules is already a Makefile and calling $(MAKE) from there trips up that logic. The workaround I used is: override_dh_auto_build-arch: # set MAKELEVEL to 0 to force building submake-generated-headers in src/Makefile.global(.in) MAKELEVEL=0 $(MAKE) -C build/src all ... override_dh_auto_test-arch: ifeq (, $(findstring nocheck, $(DEB_BUILD_OPTIONS))) # when tests fail, print newest log files # initdb doesn't like LANG and LC_ALL to contradict, unset LANG and LC_CTYPE here # temp-install wants to be invoked from a top-level make, unset MAKELEVEL here # tell pg_upgrade to create its sockets in /tmp to avoid too long paths unset LANG LC_CTYPE MAKELEVEL; ulimit -c unlimited; \ if ! make -C build check-world \ $(TEMP_CONFIG) \ PGSOCKETDIR="/tmp" \ PG_TEST_EXTRA='ssl' \ PROVE_FLAGS="--verbose"; \ ... (Just mentioning this, not asking it to be changed.) Re: Tom Lane > Andres Freund writes: > > First thing I noticed that 'make dist' doesn't work in a vpath, failing in a > > somewhat obscure way (likely because in a vpath build the the copy from the > > source dir doesn't include GNUMakefile). Do we expect it to work? > > Don't see how it could possibly be useful in a vpath, because you'd have > the real source files and the generated files in different trees. I don't think "make dist" is generally expected to work in vpath builds, that's probably one indirection layer too much. (The "make distcheck" rule generated by automake tests vpath builds, though.) Christoph
Re: testing dist tarballs
On 24.05.23 23:24, Andres Freund wrote: First thing I noticed that 'make dist' doesn't work in a vpath, failing in a somewhat obscure way (likely because in a vpath build the the copy from the source dir doesn't include GNUMakefile). Do we expect it to work? I don't think so. Separately, it's somewhat confusing that we include errcodes.h etc in src/backend/utils, rather than its final location, in src/include/utils. It works, even without perl, because copying the file doesn't require perl, it's just generating it... The "copying" is actually a symlink, right? I don't think we want to ship symlinks in the tarball?
Re: testing dist tarballs
Hi, On 2023-05-26 09:02:33 +0200, Peter Eisentraut wrote: > On 24.05.23 23:24, Andres Freund wrote: > > First thing I noticed that 'make dist' doesn't work in a vpath, failing in a > > somewhat obscure way (likely because in a vpath build the the copy from the > > source dir doesn't include GNUMakefile). Do we expect it to work? > > I don't think so. Maybe we should just error out in that case, instead of failing in an obscure way down the line? > > Separately, it's somewhat confusing that we include errcodes.h etc in > > src/backend/utils, rather than its final location, in src/include/utils. It > > works, even without perl, because copying the file doesn't require perl, > > it's > > just generating it... > > The "copying" is actually a symlink, right? I don't think we want to ship > symlinks in the tarball? Fair point - still seems we should just create the files in the right directory instead of doing it in the wrong place and then creating symlinks to make them accessible... Greetings, Andres Freund
Re: testing dist tarballs
On 27.05.23 14:47, Andres Freund wrote: Separately, it's somewhat confusing that we include errcodes.h etc in src/backend/utils, rather than its final location, in src/include/utils. It works, even without perl, because copying the file doesn't require perl, it's just generating it... The "copying" is actually a symlink, right? I don't think we want to ship symlinks in the tarball? Fair point - still seems we should just create the files in the right directory instead of doing it in the wrong place and then creating symlinks to make them accessible... Right. I think the reason this was set up this way is that with make it is generally dubious to create target files outside of the current directory.
