Re: [RFC] building postgres with meson - v13
> From 680ff3f7b4da1dbf21d0c7cd87af9bb5ee8b230c Mon Sep 17 00:00:00 2001 > From: Andres Freund > Date: Wed, 21 Sep 2022 20:36:36 -0700 > Subject: [PATCH v17 01/23] meson: ci: wip: move compilerwarnings task to meson >always: > gcc_warning_script: | > - time ./configure \ > ---cache gcc.cache \ > ---enable-dtrace \ > -${LINUX_CONFIGURE_FEATURES} \ > -CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" > - make -s -j${BUILD_JOBS} clean > - time make -s -j${BUILD_JOBS} world-bin > + mkdir build && cd build > + CC="ccache gcc" CXX="ccache g++" \ > +meson setup \ > + -Dwerror=true \ > + -Dcassert=false \ > + -Ddtrace=enabled \ > + ${LINUX_MESON_FEATURES} \ > + .. > + time ninja -j${BUILD_JOBS} With gcc, autoconf uses -O2, so I think this should specify buildtype=debugoptimized, or pass -Doptimization=2. Otherwise it ends up in release mode with -O3.
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-26 14:15:35 -0700, Peter Geoghegan wrote: > On Mon, Sep 26, 2022 at 1:27 PM Andres Freund wrote: > > > Some feedback: > > > * I gather that "running" as it appears in commands like "meson test > > > --setup running" refers to a particular setup named "running", that > > > you invented as part of creating a meson-ish substitute for > > > installcheck. Can "running" be renamed to something that makes it > > > obvious that it's a Postgres thing, and not a generic meson thing? > > > > Yes. The only caveat is that it makes lines longer, because it's included in > > the printed test line (there's no real requirement to have the test suite > > and > > the setup named the same,b ut it seems confusing not to) > > Probably doesn't have to be too long. And I'm not sure of the details. > Just a small thing from my point of view. Attached is an updated version of that patch. I left the name as 'running' because a postgres- or pg- prefix felt to awkward. This just adds fairly minimal documentation for the 'running' setup, while we now have some basic docs for building with meson, we don't yet have a "translation" of regress.sgml. Not sure how to structure that best, either. I plan to commit that soon. This likely isn't the be-all-end-all, but it's quite useful as-is. Greetings, Andres Freund >From a50cc9ac3045b0ef1cfd3cc76b3ff5759bdb85ef Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Tue, 6 Dec 2022 19:24:44 -0800 Subject: [PATCH v2] meson: Add 'running' test setup, as a replacement for installcheck To run all tests that support running against existing server: $ meson test --setup running To run just the main pg_regress tests against existing server: $ meson test --setup running regress-running/regress To ensure the 'running' setup continues to work, test it as part of the freebsd CI task. Discussion: https://postgr.es/m/CAH2-Wz=xdqcmloo7rr_i6fkqddmcyb9q5gstnfuuqxroghb...@mail.gmail.com --- meson.build | 91 --- contrib/basic_archive/meson.build | 3 + contrib/pg_freespacemap/meson.build | 3 + contrib/pg_stat_statements/meson.build| 4 + contrib/pg_walinspect/meson.build | 3 + contrib/test_decoding/meson.build | 5 + src/interfaces/ecpg/test/meson.build | 1 + doc/src/sgml/installation.sgml| 6 ++ .cirrus.yml | 15 +++ src/test/isolation/meson.build| 1 + src/test/modules/commit_ts/meson.build| 3 + src/test/modules/snapshot_too_old/meson.build | 3 + src/test/modules/test_oat_hooks/meson.build | 1 + src/test/modules/test_pg_dump/meson.build | 2 + src/test/modules/test_slru/meson.build| 1 + src/test/modules/worker_spi/meson.build | 4 +- src/test/regress/meson.build | 1 + 17 files changed, 132 insertions(+), 15 deletions(-) diff --git a/meson.build b/meson.build index 39fc3ddab26..3cb50c0b172 100644 --- a/meson.build +++ b/meson.build @@ -2920,6 +2920,20 @@ endif # Test Generation ### +# When using a meson version understanding exclude_suites, define a +# 'tmp_install' test setup (the default) that excludes tests running against a +# pre-existing install and a 'running' setup that conflicts with creation of +# the temporary installation and tap tests (which don't support running +# against a running server). + +running_suites = [] +install_suites = [] +if meson.version().version_compare('>=0.57') + runningcheck = true +else + runningcheck = false +endif + testwrap = files('src/tools/testwrap') foreach test_dir : tests @@ -2927,7 +2941,6 @@ foreach test_dir : tests testwrap, '--basedir', meson.build_root(), '--srcdir', test_dir['sd'], -'--testgroup', test_dir['name'], ] foreach kind, v : test_dir @@ -2940,55 +2953,94 @@ foreach test_dir : tests if kind in ['regress', 'isolation', 'ecpg'] if kind == 'regress' runner = pg_regress +fallback_dbname = 'regression_@0@' elif kind == 'isolation' runner = pg_isolation_regress +fallback_dbname = 'isolation_regression_@0@' elif kind == 'ecpg' runner = pg_regress_ecpg +fallback_dbname = 'ecpg_regression_@0@' endif - test_output = test_result_dir / test_dir['name'] / kind + test_group = test_dir['name'] + test_group_running = test_dir['name'] + '-running' - test_command = [ + test_output = test_result_dir / test_group / kind + test_output_running = test_result_dir / test_group_running/ kind + + # Unless specified by the test, choose a non-conflicting database name, + # to avoid conflicts when running against existing server. + dbname = t.get('dbname', +fallback_dbname.format(test_dir['name'])) + + test_command_base = [ runner.full_path(),
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-22 04:29:15 -0400, Andrew Dunstan wrote: > Now I'll start on buildfarm support. Given my current commitments, this will > take me a while, but I hope to have a working client by about the beginning > of November. Just checking: Any progress on this? Anything I can help with? I'd like to move towards dropping src/tools/msvc at some point not too far away, and we can't do so before having buildfarm support. I was just reminded of this by looking at the windows-arm support patch... Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
> From 680ff3f7b4da1dbf21d0c7cd87af9bb5ee8b230c Mon Sep 17 00:00:00 2001 > From: Andres Freund > Date: Wed, 21 Sep 2022 20:36:36 -0700 > Subject: [PATCH v17 01/23] meson: ci: wip: move compilerwarnings task to meson > > --- > .cirrus.yml| 92 +- > src/tools/ci/linux-mingw-w64-64bit.txt | 13 > 2 files changed, 59 insertions(+), 46 deletions(-) > create mode 100644 src/tools/ci/linux-mingw-w64-64bit.txt > > diff --git a/.cirrus.yml b/.cirrus.yml > index 7b5cb021027..eb33fdc4855 100644 > --- a/.cirrus.yml > +++ b/.cirrus.yml > @@ -465,6 +465,10 @@ task: >ccache_cache: > folder: $CCACHE_DIR > > + ccache_stats_start_script: > +ccache -s > +ccache -z I realized that ccache -z clears out not only the global stats, but the per-file cache stats (from which the global stats are derived) - which obviously makes the cache work poorly. Newer ccache has CCACHE_STATSLOG, and --show-log-stats, which I think can do what's wanted. I'll update my ci branch with that.
Re: [RFC] building postgres with meson - v13
Andres Freund writes: > Seems like we should have a different pg_config flag for meson options than > for configure, and perhaps separately an option to show the buildsystem? Yeah, probably a good idea, given that shoving the options for one buildsystem into the other isn't likely to work. regards, tom lane
Re: [RFC] building postgres with meson - v13
Hi, On 2022-10-13 23:35:14 -0400, Tom Lane wrote: > "shiy.f...@fujitsu.com" writes: > > On Fri, Oct 14, 2022 12:40 AM Andres Freund wrote: > >> It'd be a fair amount of work, both initially and to maintain it, to > >> generate > >> something compatible. I can see some benefit in showing some feature > >> influencing output in --configure, but compatible output doesn't seem > >> worth it > >> to me. > > > And it's ok for me that the output is not exactly the same as before. > > Yeah, the output doesn't have to be exactly the same. Seems like we should have a different pg_config flag for meson options than for configure, and perhaps separately an option to show the buildsystem? Maybe --buildsystem -> autoconf|meson, --configure -> existing CONFIGURE_ARGS for autoconf, empty otherwise, --meson-options -> meson options? Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
"shiy.f...@fujitsu.com" writes: > On Fri, Oct 14, 2022 12:40 AM Andres Freund wrote: >> It'd be a fair amount of work, both initially and to maintain it, to generate >> something compatible. I can see some benefit in showing some feature >> influencing output in --configure, but compatible output doesn't seem worth >> it >> to me. > And it's ok for me that the output is not exactly the same as before. Yeah, the output doesn't have to be exactly the same. But it had better show all the options influencing the compilation, or else we will be looking for other ways to reconstruct that information. regards, tom lane
RE: [RFC] building postgres with meson - v13
On Fri, Oct 14, 2022 12:40 AM Andres Freund wrote: > > Hi, > > On 2022-10-13 09:24:51 +, shiy.f...@fujitsu.com wrote: > > I noticed that `pg_config --configure` didn't show the options given when > > building with meson. > > Yes, that was noted somewhere on this thread. > > > > Maybe it would be better if pg_config can output this information, to be > > consistent with the output when building with `./configure` and `make`. > > > > The output when building with `./configure` and `make`: > > $ pg_config --configure > > '--prefix=/home/postgres/install/' '--cache' 'gcc.cache' '--enable-dtrace' > > '-- > with-icu' '--enable-cassert' > > It'd be a fair amount of work, both initially and to maintain it, to generate > something compatible. I can see some benefit in showing some feature > influencing output in --configure, but compatible output doesn't seem worth it > to me. > I agree that there are some benefits to showing that, which helps to confirm the build options. Although that can be confirmed from the compile log, but the log may not be available all the time. And it's ok for me that the output is not exactly the same as before. Regards, Shi yu
Re: [RFC] building postgres with meson - v13
Hi, On 2022-10-13 09:24:51 +, shiy.f...@fujitsu.com wrote: > I noticed that `pg_config --configure` didn't show the options given when > building with meson. Yes, that was noted somewhere on this thread. > Maybe it would be better if pg_config can output this information, to be > consistent with the output when building with `./configure` and `make`. > > The output when building with `./configure` and `make`: > $ pg_config --configure > '--prefix=/home/postgres/install/' '--cache' 'gcc.cache' '--enable-dtrace' > '--with-icu' '--enable-cassert' It'd be a fair amount of work, both initially and to maintain it, to generate something compatible. I can see some benefit in showing some feature influencing output in --configure, but compatible output doesn't seem worth it to me. Greetings, Andres Freund
RE: [RFC] building postgres with meson - v13
Hi, I noticed that `pg_config --configure` didn't show the options given when building with meson. For example, meson setup build -Dcache=gcc.cache -Ddtrace=enabled -Dicu=enabled -Dcassert=true -Dprefix=/home/postgres/install_meson/ meson compile -C build meson install -C build $ pg_config --configure The options I specified (like dtrace) are not shown. I found they actually work in compilation. When specifying `-Ddtrace=enabled`, there is a log like this. And with `-Ddtrace=disabled`, no such log. [120/1834] /usr/bin/dtrace -C -h -s ../src/include/utils/../../backend/utils/probes.d -o src/include/utils/probes.h.tmp Maybe it would be better if pg_config can output this information, to be consistent with the output when building with `./configure` and `make`. The output when building with `./configure` and `make`: $ pg_config --configure '--prefix=/home/postgres/install/' '--cache' 'gcc.cache' '--enable-dtrace' '--with-icu' '--enable-cassert' Regards, Shi yu
Re: [RFC] building postgres with meson - v13
Hi, On 2022-10-05 10:16:06 +0200, Peter Eisentraut wrote: > On 04.10.22 05:25, Andres Freund wrote: > > I've attached a revised version of the xml-tools dependency wrapper (0001): > > Cleanups, minor error handling improvements, and bit of comment polishing. > > I'd > > welcome review. But as it fixes a build-dependency bug / FIXME, I'm planning > > to push it relatively soon otherwise. > > > > 0002 fixes libpq's .pc file (static dependencies didn't show up anymore) and > > AIX compilation. AIX doesn't yet support link_whole (support was merged into > > meson yesterday though). On the way it also improves comments and a bit of > > generic infrastructure. The price for now is that the static libpq is built > > separately from the shared one, not reusing any objects. I felt that the > > complexity of reusing the objects isn't worth it for now. > > > > Peter, it'd be great if you could have a look at 0002. > > > > 0003 mirrors the setup of libpq to the various ecpg libraries. This is a > > prerequisite to adding resource files. > > > > 0004 adds the resource files > > These patches look ok to me. Thanks for checking. With that I'm closing the original meson CF entry. Wohoo! I'll post two new threads, one about pgxs compatibility, one about precompiled headers in a bit. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On 04.10.22 05:25, Andres Freund wrote: I've attached a revised version of the xml-tools dependency wrapper (0001): Cleanups, minor error handling improvements, and bit of comment polishing. I'd welcome review. But as it fixes a build-dependency bug / FIXME, I'm planning to push it relatively soon otherwise. 0002 fixes libpq's .pc file (static dependencies didn't show up anymore) and AIX compilation. AIX doesn't yet support link_whole (support was merged into meson yesterday though). On the way it also improves comments and a bit of generic infrastructure. The price for now is that the static libpq is built separately from the shared one, not reusing any objects. I felt that the complexity of reusing the objects isn't worth it for now. Peter, it'd be great if you could have a look at 0002. 0003 mirrors the setup of libpq to the various ecpg libraries. This is a prerequisite to adding resource files. 0004 adds the resource files These patches look ok to me.
Re: [RFC] building postgres with meson - v13
On 03.10.22 09:39, samay sharma wrote: 9f5be26c1215 meson: Add docs for building with meson I do like the overall layout of this. The "Supported Platforms" section should be moved back to near the end of the chapter. I don't see a reason to move it forward, at least none that is related to the meson issue. Agreed that it's unrelated to meson. However, I think it's better to move it in the front as it's generally useful to know if your platform is supported before you start performing the installation steps and get stuck somewhere. The way it is currently organized is that 17.2 says "In general, a modern Unix-compatible platform should be able to run PostgreSQL. The platforms that had received specific testing at the time of release are described in Section 17.6 below." So basically, it says, don't worry about it, your platform is probably supported, but check below if you are interested in the details. I don't see a reason to turn this around. Do you think I should submit that as a separate commit in the same patch-set or just move it out to a completely different patch submission? The changes to the "Getting the Source" section are also not appropriate for this patch. Given that many developers are now using Git for downloading the source code, I think it makes sense to be in the Getting the source section. Also, meson today doesn't cleanly build via the tarballs. Hence, I added it to the section (and patchset). Section 17.3 already contains a link to section I.1 about using Git. Do you think I should move this to a different patch? If you wanted to pursue these changes, then yes, but I think they are not clear improvements, as mentioned above. I suggest focusing on getting the actual meson documentation finished and then considering polishing the overall flow if desired.
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-30 15:35:26 -0700, Andres Freund wrote: > I was thinking of starting at least the following threads / CF entries once a > few of the remaining things are resolved: > > - PGXS compatibility, plus related autoconf simplification patches > - pkg-config files for building postgres extensions > - relative rpath support > > I am a bit on the fence about whether it's worth doing so for: > > - installcheck equivalent > - precompiled header support (would like it soon, because it reduces > compile-test times substantially) > > and, for no really tangible reason, considered > - resource files generation > - docs > - docs dependency > > to be part of this thread / CF entry. > > Now that I think about it more, I am inclined to also push the docs changes to > a new thread, just for wider visibility. > > I think it'd be ok to commit the docs dependency fix soon, without a separate > thread, as it really fixes a "build bug". I've not yet posted these different threads, but I've split up the meson tree into subtrees corresponding to pretty much the above. The meson tree now mainly merges those subtrees together. It still directly contains the xml-tools dependency wrapper (to be merged soon) and the CI changes (either later or never). I've attached a revised version of the xml-tools dependency wrapper (0001): Cleanups, minor error handling improvements, and bit of comment polishing. I'd welcome review. But as it fixes a build-dependency bug / FIXME, I'm planning to push it relatively soon otherwise. 0002 fixes libpq's .pc file (static dependencies didn't show up anymore) and AIX compilation. AIX doesn't yet support link_whole (support was merged into meson yesterday though). On the way it also improves comments and a bit of generic infrastructure. The price for now is that the static libpq is built separately from the shared one, not reusing any objects. I felt that the complexity of reusing the objects isn't worth it for now. Peter, it'd be great if you could have a look at 0002. 0003 mirrors the setup of libpq to the various ecpg libraries. This is a prerequisite to adding resource files. 0004 adds the resource files I think after that we could close the CF entry (and create a bunch of followup entries, as discussed above). Although it somehow seems frivolous to start a separate thread for "installcheck equivalent" :) Greetings, Andres Freund >From b21850b49a9dd9f4a60eb9c243ba64be61e7e9c0 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Mon, 3 Oct 2022 18:26:24 -0700 Subject: [PATCH v18 01/22] meson: docs: Add xml{lint,proc} wrapper to collect dependencies meson/ninja do not support specifying dependencies via globs (as those make it significantly more expensive to check the current build state). Instead targets should emit dependency information when running that then can be cheaply re-checked during future builds. To handle xmllint and xsltproc invocations in the docs, add and use a wrapper that uses --load-trace to collect dependency information. Author: Nazir Bilal Yavuz Author: Andres Freund Discussion: https://postgr.es/m/c5736f70-bb6d-8d25-e35c-e3d886e4e...@enterprisedb.com --- doc/src/sgml/meson.build | 41 --- doc/src/sgml/xmltools_dep_wrapper | 54 +++ 2 files changed, 84 insertions(+), 11 deletions(-) create mode 100644 doc/src/sgml/xmltools_dep_wrapper diff --git a/doc/src/sgml/meson.build b/doc/src/sgml/meson.build index ba2a261e7a4..65fd6131344 100644 --- a/doc/src/sgml/meson.build +++ b/doc/src/sgml/meson.build @@ -2,7 +2,7 @@ docs = [] alldocs = [] doc_generated = [] -xmllint = find_program(get_option('XMLLINT'), native: true, required: false) +xmllint_bin = find_program(get_option('XMLLINT'), native: true, required: false) version_sgml = configure_file( @@ -60,14 +60,23 @@ doc_generated += custom_target('keywords-table.sgml', ) # For everything else we need at least xmllint -if not xmllint.found() +if not xmllint_bin.found() subdir_done() endif pandoc = find_program('pandoc', native: true, required: false) -xsltproc = find_program(get_option('XSLTPROC'), native: true, required: false) +xsltproc_bin = find_program(get_option('XSLTPROC'), native: true, required: false) fop = find_program('fop', native: true, required: false) +xmltools_wrapper = [ + python, files('xmltools_dep_wrapper'), + '--targetname', '@OUTPUT@', '--depfile', '@DEPFILE@' +] + +xmllint = xmltools_wrapper + [ + '--tool', xmllint_bin, '--', +] + # Run validation only once, common to all subsequent targets. While # we're at it, also resolve all entities (that is, copy all included # files into one big file). This helps tools that don't understand @@ -75,6 +84,7 @@ fop = find_program('fop', native: true, required: false) postgres_full_xml = custom_target('postgres-full.xml', input: 'postgres.sgml', output: 'postgres-full.xml', + depfile: 'postgres-full.xml.d', command: [xmllint, '--noent',
Re: [RFC] building postgres with meson - v13
Hi, On Mon, Sep 26, 2022 at 6:02 AM Peter Eisentraut < peter.eisentr...@enterprisedb.com> wrote: > On 24.09.22 20:09, Andres Freund wrote: > > On 2022-09-24 13:52:29 -0400, Tom Lane wrote: > >> ... btw, shouldn't the CF entry [1] get closed now? > > > > Unfortunately not - there's quite a few followup patches that haven't > been > > [fully] reviewed and thus not applied yet. > > Here is some review of the remaining ones (might not match exactly what > you attached, I was working off your branch): > > > 9f789350a7a7 meson: ci: wip: move compilerwarnings task to meson > > This sounds reasonable to me in principle, but I haven't followed the > cirrus stuff too closely, and it doesn't say why it's "wip". Perhaps > others could take a closer look. > > > ccf20a68f874 meson: ci: Add additional CI coverage > > IIUC, this is just for testing your branch, not meant for master? > > > 02d84c21b227 meson: prereq: win: remove date from version number in > win32ver.rc > > do it > > > 5c42b3e7812e meson: WIP: Add some of the windows resource files > > What is the thinking on this now? What does this change over the > current state? > > > 9bc60bccfd10 meson: Add support for relative rpaths, fixing tests on > MacOS w/ SIP > > I suggest a separate thread and/or CF entry for this. There have been > various attempts to deal with SIP before, with varying results. This > is not part of the meson transition as such. > > > 9f5be26c1215 meson: Add docs for building with meson > > I do like the overall layout of this. > > The "Supported Platforms" section should be moved back to near the end > of the chapter. I don't see a reason to move it forward, at least > none that is related to the meson issue. > Agreed that it's unrelated to meson. However, I think it's better to move it in the front as it's generally useful to know if your platform is supported before you start performing the installation steps and get stuck somewhere. Do you think I should submit that as a separate commit in the same patch-set or just move it out to a completely different patch submission? > > The changes to the "Getting the Source" section are also not > appropriate for this patch. > > Given that many developers are now using Git for downloading the source code, I think it makes sense to be in the Getting the source section. Also, meson today doesn't cleanly build via the tarballs. Hence, I added it to the section (and patchset). Do you think I should move this to a different patch? > In the section "Building and Installation with meson": > > - Remove the "git clone" stuff. > - The "Running tests" section should be moved to Chapter 33. Regression > Tests. > The autoconf / make section also has a small section on how to run the regression tests. The "Running tests" section is meant to the be equivalent of that for meson (i.e. brief overview). I do intend to add a detailed section to Chapter 33 with more info on how to interpret test results etc. Do you think the current section is too verbose for where it is? > Some copy-editing will probably be suitable, but I haven't focused on > that yet. > > > 9c00d355d0e9 meson: Add PGXS compatibility > > This looks like a reasonable direction to me. How complete is it? It > says it works for some extensions but not others. How do we define > the target line here? > > > 3fd5e13dcad3 meson: Add postgresql-extension.pc for building extension > libraries > > Separate thread for this as well. This is good and important, but we > must also add it to the make build. > > > 4b5bfa1c19aa meson: Add LLVM bitcode emission > > still in progress > > > eb40f6e53104 meson: Add support for building with precompiled headers > > Any reason not to enable this by default? The benefits on non-Windows > appear to be less dramatic, but they are not zero. Might be better to > enable it consistently so that for example any breakage is easier > caught. > > > 377bfdea6042 meson: Add xmllint/xsltproc wrapper script to handle > dependencies automatically > > Is this part of the initial transition, required for correctness, or > is it an optional optimization? Could use more explanation. Maybe > move to separate thread also? > >
Re: [RFC] building postgres with meson - v13
On Sun, Oct 02, 2022 at 01:38:37PM -0500, Justin Pryzby wrote: > On Sun, Oct 02, 2022 at 11:05:30AM -0700, Andres Freund wrote: > > > Also, you wrote "rm -fr build" between building for gcc and clang, but > > > since they run in an "always" block, it'd be better to use separate > > > dirs, to allow seeing logs for the the all (failed) tasks, in case the > > > last one succeeds. > > > > Hm, when are logs important for CompilerWarnings? I don't think we even > > collect any? Using a different builddir for the "sibling" tests (i.e. the > > two > > gcc and the two clang tests) would increase the times a bit because we'd > > regenerate the bison files etc. > > > > I guess it'll look a bit cleaner to use a build-gcc and a build-clang, just > > to > > get rid of the irregularity of needing that rm -rf. > > The build logs are important when hacking on .cirrus.yml itself. > > You're right that we don't normally save logs for CompilerWarnings; one or > another (unpublished) patch of mine adds that, and then also needed to change > to use separate dirs in order to debug building while experimenting with your > patch to use meson. FYI, this is what led me to make that suggestion. https://cirrus-ci.com/task/5920691940753408 I had a patch laying around to change the "compiler warnings" task to use debian "testing", which seems to have added some new flags in -Wall, which caused me to add (for now) some compiler flags like -Wno-error=... But when I added them to the task's CFLAGS, it broke "clang" (which doesn't support the warnings) in an obscure way[0], and no logs available to show why. [0] Header "uuid/uuid.h" has symbol "uuid_generate" with dependency uuid: NO So, I think it's worth reporting meson's build logs, even though no tests are run here. -- Justin
Re: [RFC] building postgres with meson - v13
On Sun, Oct 02, 2022 at 11:05:30AM -0700, Andres Freund wrote: > > Also, you wrote "rm -fr build" between building for gcc and clang, but > > since they run in an "always" block, it'd be better to use separate > > dirs, to allow seeing logs for the the all (failed) tasks, in case the > > last one succeeds. > > Hm, when are logs important for CompilerWarnings? I don't think we even > collect any? Using a different builddir for the "sibling" tests (i.e. the two > gcc and the two clang tests) would increase the times a bit because we'd > regenerate the bison files etc. > > I guess it'll look a bit cleaner to use a build-gcc and a build-clang, just to > get rid of the irregularity of needing that rm -rf. The build logs are important when hacking on .cirrus.yml itself. You're right that we don't normally save logs for CompilerWarnings; one or another (unpublished) patch of mine adds that, and then also needed to change to use separate dirs in order to debug building while experimenting with your patch to use meson. -- Justin
Re: [RFC] building postgres with meson - v13
Hi, On 2022-10-02 12:25:20 -0500, Justin Pryzby wrote: > On Mon, Sep 26, 2022 at 06:19:51PM -0700, Andres Freund wrote: > > From 680ff3f7b4da1dbf21d0c7cd87af9bb5ee8b230c Mon Sep 17 00:00:00 2001 > > From: Andres Freund > > Date: Wed, 21 Sep 2022 20:36:36 -0700 > > Subject: [PATCH v17 01/23] meson: ci: wip: move compilerwarnings task to > > meson > > This patch isn't finished, but this part looks like a rebase conflict: > > - make -s -j${BUILD_JOBS} clean > + make -s -j${BUILD_JOBS} world-bin I don't think so - it's the first task building with autoconf / in-tree. I however shouldn't added ccache to CC, that was an accident. I think I'll convert it to a vpath build, seems cleaner. > Also, you wrote "rm -fr build" between building for gcc and clang, but > since they run in an "always" block, it'd be better to use separate > dirs, to allow seeing logs for the the all (failed) tasks, in case the > last one succeeds. Hm, when are logs important for CompilerWarnings? I don't think we even collect any? Using a different builddir for the "sibling" tests (i.e. the two gcc and the two clang tests) would increase the times a bit because we'd regenerate the bison files etc. I guess it'll look a bit cleaner to use a build-gcc and a build-clang, just to get rid of the irregularity of needing that rm -rf. > On Mon, Sep 26, 2022 at 06:19:51PM -0700, Andres Freund wrote: > > From 6025cb80d65fd7a8414241931df9f003a292052f Mon Sep 17 00:00:00 2001 > > From: Andres Freund > > Date: Sun, 25 Sep 2022 12:07:29 -0700 > > Subject: [PATCH v17 16/23] windows: adjust FD_SETSIZE via commandline > > define > > > +++ b/src/bin/pgbench/meson.build > > @@ -27,6 +27,8 @@ pgbench = executable('pgbench', > >pgbench_sources, > >dependencies: [frontend_code, libpq, thread_dep], > >include_directories: include_directories('.'), > > + c_pch: pch_postgres_fe_h, > > + c_args: host_system == 'windows' ? ['-DFD_SETSIZE=1024'] : [], > >kwargs: default_bin_args, > > ) > > This puts PCH into the preparatory commit. > Also, src/tools/msvc/Mkvcbuild.pm seems to use spaces rather than tabs. Oops, will fix. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On Mon, Sep 26, 2022 at 06:19:51PM -0700, Andres Freund wrote: > From 680ff3f7b4da1dbf21d0c7cd87af9bb5ee8b230c Mon Sep 17 00:00:00 2001 > From: Andres Freund > Date: Wed, 21 Sep 2022 20:36:36 -0700 > Subject: [PATCH v17 01/23] meson: ci: wip: move compilerwarnings task to meson This patch isn't finished, but this part looks like a rebase conflict: - make -s -j${BUILD_JOBS} clean + make -s -j${BUILD_JOBS} world-bin Also, you wrote "rm -fr build" between building for gcc and clang, but since they run in an "always" block, it'd be better to use separate dirs, to allow seeing logs for the the all (failed) tasks, in case the last one succeeds. On Mon, Sep 26, 2022 at 06:19:51PM -0700, Andres Freund wrote: > From 6025cb80d65fd7a8414241931df9f003a292052f Mon Sep 17 00:00:00 2001 > From: Andres Freund > Date: Sun, 25 Sep 2022 12:07:29 -0700 > Subject: [PATCH v17 16/23] windows: adjust FD_SETSIZE via commandline > define > +++ b/src/bin/pgbench/meson.build > @@ -27,6 +27,8 @@ pgbench = executable('pgbench', >pgbench_sources, >dependencies: [frontend_code, libpq, thread_dep], >include_directories: include_directories('.'), > + c_pch: pch_postgres_fe_h, > + c_args: host_system == 'windows' ? ['-DFD_SETSIZE=1024'] : [], >kwargs: default_bin_args, > ) This puts PCH into the preparatory commit. Also, src/tools/msvc/Mkvcbuild.pm seems to use spaces rather than tabs. -- Justin
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-30 23:51:04 +0200, Peter Eisentraut wrote: > On 27.09.22 03:19, Andres Freund wrote: > > Attaches is version 17. Other changes: > [23 attachments] > > How shall we proceed here? The more progress we make, the more patches > appear. ;-) > Maybe close this commitfest entry now, and start new threads for each > subsequent topic. I was thinking of starting at least the following threads / CF entries once a few of the remaining things are resolved: - PGXS compatibility, plus related autoconf simplification patches - pkg-config files for building postgres extensions - relative rpath support I am a bit on the fence about whether it's worth doing so for: - installcheck equivalent - precompiled header support (would like it soon, because it reduces compile-test times substantially) and, for no really tangible reason, considered - resource files generation - docs - docs dependency to be part of this thread / CF entry. Now that I think about it more, I am inclined to also push the docs changes to a new thread, just for wider visibility. I think it'd be ok to commit the docs dependency fix soon, without a separate thread, as it really fixes a "build bug". Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On 27.09.22 03:19, Andres Freund wrote: Attaches is version 17. Other changes: [23 attachments] How shall we proceed here? The more progress we make, the more patches appear. ;-) Maybe close this commitfest entry now, and start new threads for each subsequent topic.
Re: [RFC] building postgres with meson - v13
On Tue, Sep 27, 2022 at 2:41 PM John Naylor wrote: > > On Tue, Sep 27, 2022 at 2:06 AM Andres Freund wrote: > > > > On 2022-09-26 15:18:29 +0700, John Naylor wrote: > > Yea, it's /usr/local on x86-64, based on what was required to make macos CI > > work. I updated the wiki page, half-blindly - it'd be nice if you could > > confirm that that works? > > Not sure if you intended for me to try the full script in your last response or just what's in the wiki page, but for the latter (on commit bed0927aeb0c6), it fails at > > [1656/2199] Linking target src/bin/psql/psql > FAILED: src/bin/psql/psql Per off-list discussion with Andres, the linking failure was caused by some env variables set in my bash profile for the sake of Homebrew. After removing those, the recipe in the wiki worked fine. -- John Naylor EDB: http://www.enterprisedb.com
Re: [RFC] building postgres with meson - v13
On 26.09.22 18:35, Andres Freund wrote: 9f5be26c1215 meson: Add docs for building with meson I do like the overall layout of this. The "Supported Platforms" section should be moved back to near the end of the chapter. I don't see a reason to move it forward, at least none that is related to the meson issue. The changes to the "Getting the Source" section are also not appropriate for this patch. We don't really support building from a tarball with meson yet (you'd need to confiure, maintainer-clean, configure meson), so it does make some sense... Okay, interesting point. I suggest that we write it as if that were fixed, and for the time being insert a (or similar) explaining the above restriction. Otherwise we'll have to rewrite it again later.
Re: [RFC] building postgres with meson - v13
On Tue, Sep 27, 2022 at 2:06 AM Andres Freund wrote: > > On 2022-09-26 15:18:29 +0700, John Naylor wrote: > > Either way it doesn't exist on this machine. I was able to get a working > > build with > > > > /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig > > Hm - what did you need this path for - I don't think that should be needed. I just cargo-culted the pattern from Arm (before I figured out it was Arm) and used the "find" command to look for the directories by name. I tried again without specifying any of the three directory flags, and I can run the tests getting: Ok: 233 Expected Fail: 0 Fail: 0 Unexpected Pass:0 Skipped:2 Timeout:0 ...which is fine for me since I don't do much development on MacOS nowadays. > > 1) /opt/homebrew/ seems to be an "Apple silicon" path? > > Yea, it's /usr/local on x86-64, based on what was required to make macos CI > work. I updated the wiki page, half-blindly - it'd be nice if you could > confirm that that works? Not sure if you intended for me to try the full script in your last response or just what's in the wiki page, but for the latter (on commit bed0927aeb0c6), it fails at [1656/2199] Linking target src/bin/psql/psql FAILED: src/bin/psql/psql clang -o src/bin/psql/psql src/bin/psql/psql.p/meson-generated_.._psqlscanslash.c.o src/bin/psql/psql.p/meson-generated_.._sql_help.c.o src/bin/psql/psql.p/command.c.o src/bin/psql/psql.p/common.c.o src/bin/psql/psql.p/copy.c.o src/bin/psql/psql.p/crosstabview.c.o src/bin/psql/psql.p/describe.c.o src/bin/psql/psql.p/help.c.o src/bin/psql/psql.p/input.c.o src/bin/psql/psql.p/large_obj.c.o src/bin/psql/psql.p/mainloop.c.o src/bin/psql/psql.p/prompt.c.o src/bin/psql/psql.p/startup.c.o src/bin/psql/psql.p/stringutils.c.o src/bin/psql/psql.p/tab-complete.c.o src/bin/psql/psql.p/variables.c.o -L/usr/local/opt/readline/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/zlib/lib -L/usr/local/opt/openssl/lib -I/usr/local/opt/readline/include -I/usr/local/opt/gettext/include -I/usr/local/opt/zlib/include -I/usr/local/opt/openssl/include -Wl,-dead_strip_dylibs -Wl,-headerpad_max_install_names -Wl,-undefined,error -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk -Wl,-rpath,@loader_path/../../interfaces/libpq -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/Cellar/zstd/1.5.2/lib src/fe_utils/libpgfeutils.a src/common/libpgcommon.a src/common/libpgcommon_ryu.a src/common/libpgcommon_config_info.a src/port/libpgport.a src/port/libpgport_crc.a src/interfaces/libpq/libpq.5.dylib -lm /usr/local/lib/libintl.dylib -ledit -lz /usr/local/Cellar/zstd/1.5.2/lib/libzstd.dylib -lz -lz -lz Undefined symbols for architecture x86_64: "_rl_completion_suppress_quote", referenced from: _psql_completion in tab-complete.c.o _quote_file_name in tab-complete.c.o _complete_from_files in tab-complete.c.o "_rl_filename_dequoting_function", referenced from: _initialize_readline in tab-complete.c.o "_rl_filename_quote_characters", referenced from: _initialize_readline in tab-complete.c.o "_rl_filename_quoting_function", referenced from: _initialize_readline in tab-complete.c.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) -- John Naylor EDB: http://www.enterprisedb.com
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-27 17:29:27 +1300, Thomas Munro wrote: > On Tue, Sep 27, 2022 at 2:19 PM Andres Freund wrote: > > Subject: [PATCH v17 15/23] windows: Set UMDF_USING_NTSTATUS globally, > > include ntstatus.h > > No Windows expertise here, but this looks reasonable. I originally > tried to contain UMDF_USING_NTSTATUS to small translation units for > fear of unintended consequences, but PCH requires you not to mess with > macros that affect the compilation of a header as seen by different > translation units, which is an incompatible goal. If this is passing > on MSVC and MingGW then +1 from me. Yes, passes both. > You mentioned WIN32_NO_STATUS in the commit message -- a mistake? Argh. An earlier iteration. Works on mingw, but making it work with msvc required a lot more modifications IIRC. > Digging out my old emails/notes... that's another way to be allowed to > include both ntstatus.h and windows.h/etc in the same translation > unit, but not the one we're using. I assume it's worse because you > have to define it and then undefine it, which sounds more antithetical > to the PCH dream. Admittedly UMDF_USING_NTSTATUS -- from the > "User-Mode Driver Framework" -- is a weird thing to be getting tangled > up with because we aren't writing a driver here, but it seems to be a > well known and widely used alternative, and is nicer because you only > have to define it. It's definitely weird. But it appears to be widely used... > > Subject: [PATCH v17 16/23] windows: adjust FD_SETSIZE via commandline define > > Right, we have to fix that across translation units for the same > reason. But why as -D and not in win32_port.h? I followed the > discussion from 9acda73118 to try to find the answer to that and saw > that Michael wanted to put it there, but wanted to minimise the blast > radius at the time: > > https://www.postgresql.org/message-id/20190826054000.GE7005%40paquier.xyz I guess a similar consideration. I was a bit worried about the references to FD_SETSIZE in src/backend/port/win32/socket.c. Multi kB on-stack arrays in postmaster seem like they could cause issues. ISTM we really ought to move away from stuff using FD_SETSIZE on windows... Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On Tue, Sep 27, 2022 at 2:19 PM Andres Freund wrote: > Subject: [PATCH v17 15/23] windows: Set UMDF_USING_NTSTATUS globally, include > ntstatus.h No Windows expertise here, but this looks reasonable. I originally tried to contain UMDF_USING_NTSTATUS to small translation units for fear of unintended consequences, but PCH requires you not to mess with macros that affect the compilation of a header as seen by different translation units, which is an incompatible goal. If this is passing on MSVC and MingGW then +1 from me. You mentioned WIN32_NO_STATUS in the commit message -- a mistake? Digging out my old emails/notes... that's another way to be allowed to include both ntstatus.h and windows.h/etc in the same translation unit, but not the one we're using. I assume it's worse because you have to define it and then undefine it, which sounds more antithetical to the PCH dream. Admittedly UMDF_USING_NTSTATUS -- from the "User-Mode Driver Framework" -- is a weird thing to be getting tangled up with because we aren't writing a driver here, but it seems to be a well known and widely used alternative, and is nicer because you only have to define it. > Subject: [PATCH v17 16/23] windows: adjust FD_SETSIZE via commandline define Right, we have to fix that across translation units for the same reason. But why as -D and not in win32_port.h? I followed the discussion from 9acda73118 to try to find the answer to that and saw that Michael wanted to put it there, but wanted to minimise the blast radius at the time: https://www.postgresql.org/message-id/20190826054000.GE7005%40paquier.xyz
Re: [RFC] building postgres with meson - v13
On Mon, Sep 26, 2022 at 1:27 PM Andres Freund wrote: > > Some feedback: > > * I gather that "running" as it appears in commands like "meson test > > --setup running" refers to a particular setup named "running", that > > you invented as part of creating a meson-ish substitute for > > installcheck. Can "running" be renamed to something that makes it > > obvious that it's a Postgres thing, and not a generic meson thing? > > Yes. The only caveat is that it makes lines longer, because it's included in > the printed test line (there's no real requirement to have the test suite and > the setup named the same,b ut it seems confusing not to) Probably doesn't have to be too long. And I'm not sure of the details. Just a small thing from my point of view. > > * It would be nice if failed tests told me exactly which "diffs" file > > I needed to look at, without my having to look for the message through > > the meson log (or running with -v). Is this possible? > > You can use --print-errorlogs to print the log output iff a test fails. It's a > bit painful that some tests have very verbose output :(. I don't really see a > way that meson can help us here, this is pretty much on "our" end. Makes sense. Thanks. > Hm. I'm not entirely sure what you mean here. The only thing that you can't > change in a existing build-dir with meson configure is the compiler. I do understand that it doesn't particularly matter to meson itself. The point I was making was one about how I personally find it convenient to set those things that I know will never change in practice (because they're fundamentally things that I know that I won't ever need to change) during "meson setup", while doing everything else using "meson configure". I like to automate everything using shell scripts. I will very occasionally have to run "meson setup" via a zsh function anyway, so why not couple that process with the process of setting "immutable for this build directory" settings? With autoconf, I will run one of various zsh functions that run configure in some specific way -- there are various "build types", such as debug, release, and Valgrind. But with meson it makes sense to split it in two -- have a generic zsh function for generic once-off build directory setup (including even the mkdir), that also sets generic, "immutable" settings, and a specialized zsh function that changes things in a way that is particular to that kind of build (like whether asserts are enabled, optimization level, and so on). > I personally have different types of build dirs set up in parallel > (e.g. assert, optimize, assert-32, assert-w64). I'll occasionally > enable/disable I know that other experienced hackers do it that way. I have found that ccache works well enough that I don't feel the need for multiple build directories per branch. Perhaps I've assumed more than I should about my approach being broadly representative. It might ultimately be easier to just have multiple build directories per branch/source directory -- one per "build type" per branch. That has the advantage of not requiring each "build type" zsh function to remember to reset anything that might have been set by one of its sibling zsh functions for some other build type (there is no need to "reset the setting to its default"). That approach is more like scripting autoconf/configure would be, in that you basically never change any settings for a given build directory in practice (you maybe invent a new kind of build type instead, or you update the definition of an existing standard build type based on a new requirement for that build type). It's really handy that meson lets you quickly change one setting against an existing build directory. I'm slightly worried that that will allow me to shoot myself in the foot, though. Perhaps I'll change some exotic setting in an ad-hoc way, and then forget to unset it afterwards, leading to (say) a mysterious performance degradation for what is supposed to be one of my known standard build types. There is no risk of that with my autoconf/configure workflow, because I'll rerun the relevant configure zsh function before long anyway, making it impossible for me to accidentally keep something that I never meant to keep. I like being able to throw everything away and quickly rebuild "from scratch" (in reality rebuild using ccache and a cache for configure) due to superstition/defensive paranoia/learned helplessness. This has always worked well enough because ccache works fairly well. I'm not sure how useful that kind of mindset will be with meson just yet, and if I'm just thinking about it in the wrong way, so forgive me for rambling like this. -- Peter Geoghegan
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-26 12:47:14 -0700, Peter Geoghegan wrote: > On Sun, Sep 25, 2022 at 5:38 PM Andres Freund wrote: > > # run just the main pg_regress tests against existing server > > meson test --setup running main/regress-running > > > Peter, would this address your use case? > > I tried out your v16 patchset, which seems to mostly work as I'd hoped > it would. Thanks & cool. > Some feedback: > * I gather that "running" as it appears in commands like "meson test > --setup running" refers to a particular setup named "running", that > you invented as part of creating a meson-ish substitute for > installcheck. Can "running" be renamed to something that makes it > obvious that it's a Postgres thing, and not a generic meson thing? Yes. The only caveat is that it makes lines longer, because it's included in the printed test line (there's no real requirement to have the test suite and the setup named the same,b ut it seems confusing not to) > Maybe some kind of consistent naming convention would work best here. > This setup could be "pg_against_running_server", or something along > those lines. > * It would be nice if failed tests told me exactly which "diffs" file > I needed to look at, without my having to look for the message through > the meson log (or running with -v). Is this possible? You can use --print-errorlogs to print the log output iff a test fails. It's a bit painful that some tests have very verbose output :(. I don't really see a way that meson can help us here, this is pretty much on "our" end. > BTW the meson wiki page iencourages users to think of "meson setup" > and "meson configure" as equivalent to autoconf configure. I get why > you explained it like that, but that confused me at first. What I > since figured out (which will be absurdly obvious to you) is that you > really need to decouple the generic from the specific -- very much > unlike autoconf. I found it useful to separate stuff that I know will > never change for a given build directory (such as the prefix install > path) from other things that are variable configuration settings > (things like the optimization level used by GCC). I now have a > scripted way of running "meson setup" for the former stuff (which is > generic), and a scripted way of running "meson configure" for the > latter set of stuff (with variations for "standard" release and debug > builds, building Valgrind, etc). Hm. I'm not entirely sure what you mean here. The only thing that you can't change in a existing build-dir with meson configure is the compiler. I personally have different types of build dirs set up in parallel (e.g. assert, optimize, assert-32, assert-w64). I'll occasionally enable/disable Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On Sun, Sep 25, 2022 at 5:38 PM Andres Freund wrote: > # run just the main pg_regress tests against existing server > meson test --setup running main/regress-running > Peter, would this address your use case? I tried out your v16 patchset, which seems to mostly work as I'd hoped it would. Some feedback: * I gather that "running" as it appears in commands like "meson test --setup running" refers to a particular setup named "running", that you invented as part of creating a meson-ish substitute for installcheck. Can "running" be renamed to something that makes it obvious that it's a Postgres thing, and not a generic meson thing? Maybe some kind of consistent naming convention would work best here. This setup could be "pg_against_running_server", or something along those lines. * It would be nice if failed tests told me exactly which "diffs" file I needed to look at, without my having to look for the message through the meson log (or running with -v). Is this possible? To be fair I should probably just be running -v when I run tests against an existing running server, anyway -- so maybe I'm asking for the wrong thing. It would at least be slightly better if I always got to see a path to a .diffs file for failed tests, even without -v. But it's just a "nice to have" thing -- it's not worth going out of your way to make it work like that * Just FYI, there are considerations about the libpq that we link to here (actually this isn't particularly related to the new installcheck work, but thought I'd mention it in passing). I'm using Debian Unstable here. Like Nathan, I found that I needed a custom -Dextra_lib_dirs just so that binaries would link against the installation's own libpq, rather than the system libpq. This is important-ish because linking to the wrong libpq means that I get an error about not being able to connect via trust authentication to a unix socket from the directory /var/run/postgresql -- confusion over where to look for sockets visibly breaks many things. The workaround that I have is fine, but this still seems like something that should "just work". I believe that there is a pending patch for this already, so enough said here. > I think it'd make sense to add a few toplevel targets to run tests in certain > ways, but I've not done that here. I usually run "installcheck-world" (not just installcheck) when I want to do a generic smoke test with Vaglrind. Sometimes that will fail relatively early for very silly reasons, for example because I don't have exactly the expected plan in some harmless way (I try to account for this by running Valgrind in a shellscript that tries to match "make check", but that doesn't always work). It is nice that I won't have to worry about such minor issues derailing everything for a long running and unsupervised Valgrind test. (Maybe I could have worked around this before now, but I guess I never tried.) More generally, I think that we should be encouraging users to think of the tests as something that you can run in any order. People should be encouraged to think in terms of the meson abstractions, such as test setups. I found that "meson test --setup running --list" will show me what tests I'll be running if I want to do "installcheck" style testing, without having to run any tests at all -- another small but important improvement. This seems worth drawing attention to on the meson Wiki page as a non-obvious improvement over "installcheck". I might even find it useful to hard-code some of these tests in a shellscript, that runs only a subset of "--setup running" tests that happen to be interesting for Valgrind testing right now. BTW the meson wiki page iencourages users to think of "meson setup" and "meson configure" as equivalent to autoconf configure. I get why you explained it like that, but that confused me at first. What I since figured out (which will be absurdly obvious to you) is that you really need to decouple the generic from the specific -- very much unlike autoconf. I found it useful to separate stuff that I know will never change for a given build directory (such as the prefix install path) from other things that are variable configuration settings (things like the optimization level used by GCC). I now have a scripted way of running "meson setup" for the former stuff (which is generic), and a scripted way of running "meson configure" for the latter set of stuff (with variations for "standard" release and debug builds, building Valgrind, etc). -- Peter Geoghegan
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-26 09:35:16 -0700, Andres Freund wrote: > > 9c00d355d0e9 meson: Add PGXS compatibility > > > > This looks like a reasonable direction to me. How complete is it? It > > says it works for some extensions but not others. How do we define > > the target line here? > > Yea, those are good questions. > > > > How complete is it? > > It's a bit hard to know. I think the most important stuff is there. But > there's no clear "API" around pgxs. E.g. we don't (yet?) have an exactly > equivalent definition of 'host', because that's very config.guess specific. > > There's lots of shortcuts - e.g. with meson we don't need an equivalent to > PGAC_CHECK_STRIP, so we need to make up something for Makefile.global. > > Noah suggested using $(error something), but that only works if $variable is > only used in recursively expanded variables - the errors end up confusing. Looking through a few of the not-nicely-replaced things, I think we can simplify at least some away: - RANLIB: most platforms use AROPT = crs, making ranlib unnecessary. {free, net, open}bsd don't currently, but all support it from what I know - with_gnu_ld: this is only used on solaris, to set export_dynamic = -Wl,-E when using a gnu ld. How about moving this to configure instead, and just checking if -Wl,-E links? - FLEXFLAGS: As a configure input this is afaict unused and undocumented - and it's not clear why it'd be useful? Not that an empty replacement is a meaningful effort I'm not sure what to do about: - autodepend - I'm inclined to set it to true when using a gcc like compiler. I think extension authors won't be happy if suddenly their extensions don't rebuild reliably anymore. An --enable-depend like setting doesn't make sense for meson, so we don't have anything to source it from. - {LDAP,UUID,ICU}_{LIBS,CFLAGS} - might some extension need them? For some others I think it's ok to not have replacement. Would be good for somebody to check my thinking though: - LIBOBJS, PG_CRC32C_OBJS, TAS: Not needed because we don't build the server / PLs with the generated makefile - ZIC: only needed to build tzdata as part of server build - MSGFMT et al: translation doesn't appear to be supported by pgxs, correct? - XMLLINT et al: docs don't seem to be supported by pgxs - GENHTML et al: supporting coverage for pgxs-in-meson build doesn't seem worth it - WINDRES: I don't think extensions are bothering to generate rc files on windows I'll include an updated pgxs-compat patch in the next post of the series (in a few hours). Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-26 15:18:29 +0700, John Naylor wrote: > On Wed, Sep 21, 2022 at 7:11 AM Andres Freund wrote: > > > > I added installation instructions for meson for a bunch of platforms, but > > A couple more things for the wiki: > > 1) /opt/homebrew/ seems to be an "Apple silicon" path? Yea, it's /usr/local on x86-64, based on what was required to make macos CI work. I updated the wiki page, half-blindly - it'd be nice if you could confirm that that works? I needed something like the below to get (nearly?) all dependencies working: brewpath="/usr/local" PKG_CONFIG_PATH="${brewpath}/lib/pkgconfig:${PKG_CONFIG_PATH}" for pkg in icu4c krb5 openldap openssl zstd ; do pkgpath="${brewpath}/opt/${pkg}" PKG_CONFIG_PATH="${pkgpath}/lib/pkgconfig:${PKG_CONFIG_PATH}" PATH="${pkgpath}/bin:${pkgpath}/sbin:$PATH" done export PKG_CONFIG_PATH PATH meson setup \ --buildtype=debug \ -Dextra_include_dirs=${brewpath}/include \ -Dextra_lib_dirs=${brewpath}/lib \ -Dcassert=true \ -Dssl=openssl -Duuid=e2fs -Ddtrace=auto \ -DPG_TEST_EXTRA="$PG_TEST_EXTRA" \ build the per-package stuff is needed because some libraries aren't installed into /usr/local (or /opt/homebrew), but only in a subdirectory within that. > Either way it doesn't exist on this machine. I was able to get a working > build with > > /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig Hm - what did you need this path for - I don't think that should be needed. > (My homebrew install doesn't seem to have anything relevant for > extra_include_dirs or extra_lib_dirs.) I think libintl.h / libintl.dylib are only in there. With meson that's the only need for extra_include_dirs / extra_lib_dirs I found on arm apple. > 2) Also, "ninja -v install" has the same line count as "ninja install" -- > are there versions that do something different? Yea, that looks like a copy-and-pasto (not even from me :)). Think I fixed it. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-26 15:01:56 +0200, Peter Eisentraut wrote: > Here is some review of the remaining ones (might not match exactly what you > attached, I was working off your branch): Thanks, and makes sense. > 9f789350a7a7 meson: ci: wip: move compilerwarnings task to meson > > This sounds reasonable to me in principle, but I haven't followed the > cirrus stuff too closely, and it doesn't say why it's "wip". Perhaps > others could take a closer look. It's mostly WIP because it doesn't yet convert all the checks, specifically headerscheck/cpluspluscheck isn't converted yet. > ccf20a68f874 meson: ci: Add additional CI coverage > > IIUC, this is just for testing your branch, not meant for master? Yes. I think we might want to add openbsd / netbsd at some point, but that'll be a separate thread. Until then it just catches a bunch of mistakes more easily. > 02d84c21b227 meson: prereq: win: remove date from version number in > win32ver.rc > > do it The newest version has evolved a bit, changing Project.pm as well. > 5c42b3e7812e meson: WIP: Add some of the windows resource files > > What is the thinking on this now? What does this change over the > current state? The newest commit has a lot more rc files added and has this summary: meson: Add windows resource files The generated resource files aren't exactly the same ones as the old buildsystems generate. Previously "InternalName" and "OriginalFileName" were mostly wrong / not set (despite being required), but that was hard to fix in at least the make build. Additionally, the meson build falls back to a "auto-generated" description when not set, and doesn't set it in a few cases - unlikely that anybody looks at these descriptions in detail. The only thing missing rc files is the various ecpg libraries. The issue is that we shouldn't add resource file to static libraries, so we need to split the definitions. I'll go and do that next. > 9bc60bccfd10 meson: Add support for relative rpaths, fixing tests on MacOS > w/ SIP > > I suggest a separate thread and/or CF entry for this. There have been > various attempts to deal with SIP before, with varying results. This > is not part of the meson transition as such. I think I might need to split this one more time. We don't add all the rpaths we add with autoconf before this commit, even not on macOS, which is not great... Nor do we have a --disable-rpath equivalent yet - I suspect we'll need that. https://postgr.es/m/2022093729.GA721620%40nathanxps13 > 9f5be26c1215 meson: Add docs for building with meson > > I do like the overall layout of this. > > The "Supported Platforms" section should be moved back to near the end > of the chapter. I don't see a reason to move it forward, at least > none that is related to the meson issue. > > The changes to the "Getting the Source" section are also not > appropriate for this patch. We don't really support building from a tarball with meson yet (you'd need to confiure, maintainer-clean, configure meson), so it does make some sense... > 9c00d355d0e9 meson: Add PGXS compatibility > > This looks like a reasonable direction to me. How complete is it? It > says it works for some extensions but not others. How do we define > the target line here? Yea, those are good questions. > How complete is it? It's a bit hard to know. I think the most important stuff is there. But there's no clear "API" around pgxs. E.g. we don't (yet?) have an exactly equivalent definition of 'host', because that's very config.guess specific. There's lots of shortcuts - e.g. with meson we don't need an equivalent to PGAC_CHECK_STRIP, so we need to make up something for Makefile.global. Noah suggested using $(error something), but that only works if $variable is only used in recursively expanded variables - the errors end up confusing. > It says it works for some extensions but not others. I think that's slightly outdated - IIRC it was about pgbouncer, but after a fix the remaining failure is shared between autoconf and meson builds. > 3fd5e13dcad3 meson: Add postgresql-extension.pc for building extension > libraries > > Separate thread for this as well. This is good and important, but we > must also add it to the make build. Makes sense. > eb40f6e53104 meson: Add support for building with precompiled headers > > Any reason not to enable this by default? The benefits on non-Windows > appear to be less dramatic, but they are not zero. Might be better to > enable it consistently so that for example any breakage is easier > caught. There's no real reason not to - the wins are small on linux, so introducing PCH didn't seem necessary. I'm also not sure how well pch works across random compiler versions - it's so crucial on windows that it seems like a more well worn path there. linux, gcc 12: b_pch=false: real0m16.233s user6m40.375s sys 0m48.953s b_pch=true: real0m15.983s user6m20.357s sys 0m49.967s freebsd VM,
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-26 10:41:01 +0200, Alvaro Herrera wrote: > On 2022-Sep-25, Andres Freund wrote: > > > From 3eb0ca196084da314d94d1e51c7b775012a4773c Mon Sep 17 00:00:00 2001 > > From: Andres Freund > > Date: Wed, 21 Sep 2022 11:03:07 -0700 > > Subject: [PATCH v16 04/16] meson: Add windows resource files > > > diff --git a/src/backend/jit/llvm/meson.build > > b/src/backend/jit/llvm/meson.build > > index de2e624ab58..5fb63768358 100644 > > --- a/src/backend/jit/llvm/meson.build > > +++ b/src/backend/jit/llvm/meson.build > > @@ -20,6 +20,12 @@ llvmjit_sources += files( > >'llvmjit_expr.c', > > ) > > > > +if host_system == 'windows' > > + llvmjit_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ > > +'--NAME', 'llvmjit', > > +'--FILEDESC', 'llvmjit - JIT using LLVM',]) > > +endif > > This is tediously imperative. Isn't there a more declarative way to > have it? I tried to come up with something better, without success. I think it's acceptable, even if not great. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On 24.09.22 20:09, Andres Freund wrote: On 2022-09-24 13:52:29 -0400, Tom Lane wrote: ... btw, shouldn't the CF entry [1] get closed now? Unfortunately not - there's quite a few followup patches that haven't been [fully] reviewed and thus not applied yet. Here is some review of the remaining ones (might not match exactly what you attached, I was working off your branch): 9f789350a7a7 meson: ci: wip: move compilerwarnings task to meson This sounds reasonable to me in principle, but I haven't followed the cirrus stuff too closely, and it doesn't say why it's "wip". Perhaps others could take a closer look. ccf20a68f874 meson: ci: Add additional CI coverage IIUC, this is just for testing your branch, not meant for master? 02d84c21b227 meson: prereq: win: remove date from version number in win32ver.rc do it 5c42b3e7812e meson: WIP: Add some of the windows resource files What is the thinking on this now? What does this change over the current state? 9bc60bccfd10 meson: Add support for relative rpaths, fixing tests on MacOS w/ SIP I suggest a separate thread and/or CF entry for this. There have been various attempts to deal with SIP before, with varying results. This is not part of the meson transition as such. 9f5be26c1215 meson: Add docs for building with meson I do like the overall layout of this. The "Supported Platforms" section should be moved back to near the end of the chapter. I don't see a reason to move it forward, at least none that is related to the meson issue. The changes to the "Getting the Source" section are also not appropriate for this patch. In the section "Building and Installation with meson": - Remove the "git clone" stuff. - The "Running tests" section should be moved to Chapter 33. Regression Tests. Some copy-editing will probably be suitable, but I haven't focused on that yet. 9c00d355d0e9 meson: Add PGXS compatibility This looks like a reasonable direction to me. How complete is it? It says it works for some extensions but not others. How do we define the target line here? 3fd5e13dcad3 meson: Add postgresql-extension.pc for building extension libraries Separate thread for this as well. This is good and important, but we must also add it to the make build. 4b5bfa1c19aa meson: Add LLVM bitcode emission still in progress eb40f6e53104 meson: Add support for building with precompiled headers Any reason not to enable this by default? The benefits on non-Windows appear to be less dramatic, but they are not zero. Might be better to enable it consistently so that for example any breakage is easier caught. 377bfdea6042 meson: Add xmllint/xsltproc wrapper script to handle dependencies automatically Is this part of the initial transition, required for correctness, or is it an optional optimization? Could use more explanation. Maybe move to separate thread also?
Re: [RFC] building postgres with meson - v13
On 2022-Sep-25, Andres Freund wrote: > From 3eb0ca196084da314d94d1e51c7b775012a4773c Mon Sep 17 00:00:00 2001 > From: Andres Freund > Date: Wed, 21 Sep 2022 11:03:07 -0700 > Subject: [PATCH v16 04/16] meson: Add windows resource files > diff --git a/src/backend/jit/llvm/meson.build > b/src/backend/jit/llvm/meson.build > index de2e624ab58..5fb63768358 100644 > --- a/src/backend/jit/llvm/meson.build > +++ b/src/backend/jit/llvm/meson.build > @@ -20,6 +20,12 @@ llvmjit_sources += files( >'llvmjit_expr.c', > ) > > +if host_system == 'windows' > + llvmjit_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ > +'--NAME', 'llvmjit', > +'--FILEDESC', 'llvmjit - JIT using LLVM',]) > +endif This is tediously imperative. Isn't there a more declarative way to have it? -- Álvaro HerreraBreisgau, Deutschland — https://www.EnterpriseDB.com/ "Pensar que el espectro que vemos es ilusorio no lo despoja de espanto, sólo le suma el nuevo terror de la locura" (Perelandra, C.S. Lewis)
Re: [RFC] building postgres with meson - v13
On Wed, Sep 21, 2022 at 7:11 AM Andres Freund wrote: > > I added installation instructions for meson for a bunch of platforms, but A couple more things for the wiki: 1) /opt/homebrew/ seems to be an "Apple silicon" path? Either way it doesn't exist on this machine. I was able to get a working build with /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig (My homebrew install doesn't seem to have anything relevant for extra_include_dirs or extra_lib_dirs.) 2) Also, "ninja -v install" has the same line count as "ninja install" -- are there versions that do something different? -- John Naylor EDB: http://www.enterprisedb.com
RE: [RFC] building postgres with meson - v13
On Mon, Sep 26, 2022 at 14:47 PM Andres Freund wrote: > Hi, > > On 2022-09-26 06:24:42 +, wangw.f...@fujitsu.com wrote: > > I tried to use meson and ninja and they are really efficient. > > But when I tried to specify "c_args", it did not take effect. > > They should take effect, but won't be shown in the summary section > currently. That currently only shows the flags chosen by the configure step, > rather than user specified ones. > > > > After I made the below modifications, the specified "c_args" took effect. > > ``` > > @@ -2439,6 +2439,10 @@ endif > > > > # Set up compiler / linker arguments to be used everywhere, individual > targets > > # can add further args directly, or indirectly via dependencies > > + > > +tmp_c_args = get_option('c_args') > > +cflags += tmp_c_args > > + > > add_project_arguments(cflags, language: ['c']) > > add_project_arguments(cppflags, language: ['c']) > > add_project_arguments(cflags_warn, language: ['c']) > > ``` > > That'll likely end up with the same cflags added multiple times. You should > see them when building with ninja -v. Thanks for sharing the information. I saw the user specified CFLAG when building with `ninja -v`. But, after installing PG with command `ninja -v install`, pg_config does not show the user specified CFLAG. Should we print this information there? > How about adding c_args to the summary, in a separate line? I think that'd > clarify what's happening? Yes, I think it might be better. Regards, Wang wei
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-26 06:24:42 +, wangw.f...@fujitsu.com wrote: > I tried to use meson and ninja and they are really efficient. > But when I tried to specify "c_args", it did not take effect. They should take effect, but won't be shown in the summary section currently. That currently only shows the flags chosen by the configure step, rather than user specified ones. > After I made the below modifications, the specified "c_args" took effect. > ``` > @@ -2439,6 +2439,10 @@ endif > > # Set up compiler / linker arguments to be used everywhere, individual > targets > # can add further args directly, or indirectly via dependencies > + > +tmp_c_args = get_option('c_args') > +cflags += tmp_c_args > + > add_project_arguments(cflags, language: ['c']) > add_project_arguments(cppflags, language: ['c']) > add_project_arguments(cflags_warn, language: ['c']) > ``` That'll likely end up with the same cflags added multiple times. You should see them when building with ninja -v. How about adding c_args to the summary, in a separate line? I think that'd clarify what's happening? Greetings, Andres Freund
RE: [RFC] building postgres with meson - v13
Hi, I tried to use meson and ninja and they are really efficient. But when I tried to specify "c_args", it did not take effect. Attached my steps: [In the HEAD (7d708093b7)] $ meson setup build --prefix /home/wangw/install/parallel_apply/ -Dcassert=true -Dtap_tests=enabled -Dicu=enabled -Dc_args='-fno-omit-frame-pointer' Log: .. Compiler Flags CPP FLAGS : -D_GNU_SOURCE C FLAGS, functional: -fno-strict-aliasing -fwrapv -fexcess-precision=standard C FLAGS, warnings : -Wmissing-prototypes -Wpointer-arith -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -Wdeclaration-after-statement -Wno-format-truncation -Wno-stringop-truncation .. After I made the below modifications, the specified "c_args" took effect. ``` @@ -2439,6 +2439,10 @@ endif # Set up compiler / linker arguments to be used everywhere, individual targets # can add further args directly, or indirectly via dependencies + +tmp_c_args = get_option('c_args') +cflags += tmp_c_args + add_project_arguments(cflags, language: ['c']) add_project_arguments(cppflags, language: ['c']) add_project_arguments(cflags_warn, language: ['c']) ``` I might missed something. Just to confirm is there another way to add CFLAG ? Regards, Wang wei
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-24 17:33:49 -0700, Peter Geoghegan wrote: > On Sat, Sep 24, 2022 at 5:13 PM Andres Freund wrote: > > > One more question about this, that wasn't covered by the Wiki page: is > > > there some equivalent to "make installcheck" with meson builds? > > > > Not yet. Nothing impossible, just not done yet. Partially because > > installcheck > > is so poorly defined (run against an already running server for pg_regress > > vs > > using "system" installed binaries for tap tests). > > Got it. I can work around that by just having an old autoconf-based > vpath build directory. I'll need to do this when I run Valgrind. > > My workaround would be annoying if I needed to run "installcheck" > anywhere near as frequently as I run "make check-world". But that > isn't the case. meson delivers a significant improvement in the metric > that really matters to me, so I can't really complain. My gut feeling is that we should use this opportunity to split 'installcheck' into two. "test a running server" and "test installed binaries". I think the cleanest way to do this with meson would be to utilize meson tests's "setups". $ meson test --setup 'running-server' would run all [selected] tests compatible with running against a running server. And $ meson test --setup 'installed' would test installed binaries. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On Sat, Sep 24, 2022 at 5:13 PM Andres Freund wrote: > > One more question about this, that wasn't covered by the Wiki page: is > > there some equivalent to "make installcheck" with meson builds? > > Not yet. Nothing impossible, just not done yet. Partially because installcheck > is so poorly defined (run against an already running server for pg_regress vs > using "system" installed binaries for tap tests). Got it. I can work around that by just having an old autoconf-based vpath build directory. I'll need to do this when I run Valgrind. My workaround would be annoying if I needed to run "installcheck" anywhere near as frequently as I run "make check-world". But that isn't the case. meson delivers a significant improvement in the metric that really matters to me, so I can't really complain. -- Peter Geoghegan
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-24 16:56:20 -0700, Peter Geoghegan wrote: > On Thu, Sep 22, 2022 at 2:50 PM Andres Freund wrote: > > meson: > > > > time meson test > > real0m42.178s > > user7m8.533s > > sys 2m17.711s > > I find that a more or less comparable test run on my workstation > (which has a Ryzen 9 5950X) takes just over 38 seconds. I think that > the improvement is far more pronounced on that machine compared to a > much older workstation. Cool! > One more question about this, that wasn't covered by the Wiki page: is > there some equivalent to "make installcheck" with meson builds? Not yet. Nothing impossible, just not done yet. Partially because installcheck is so poorly defined (run against an already running server for pg_regress vs using "system" installed binaries for tap tests). Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On Thu, Sep 22, 2022 at 2:50 PM Andres Freund wrote: > meson: > > time meson test > real0m42.178s > user7m8.533s > sys 2m17.711s I find that a more or less comparable test run on my workstation (which has a Ryzen 9 5950X) takes just over 38 seconds. I think that the improvement is far more pronounced on that machine compared to a much older workstation. One more question about this, that wasn't covered by the Wiki page: is there some equivalent to "make installcheck" with meson builds? -- Peter Geoghegan
Re: [RFC] building postgres with meson - v13
... btw, shouldn't the CF entry [1] get closed now? The cfbot's unhappy that the last patch no longer applies. regards, tom lane [1] https://commitfest.postgresql.org/39/3395/
Re: [RFC] building postgres with meson - v13
On Thu, Sep 22, 2022 at 01:28:09PM -0700, Andres Freund wrote: > On 2022-09-22 13:05:33 -0700, Nathan Bossart wrote: >> * I'm using an Ubuntu-based distribution, and the version of meson that apt >> installed was not new enough for Postgres. I ended up cloning meson [0] >> and using the newest tag. This is no big deal. > > I assume this is 20.04 LTS? If so, we're missing it by one version of meson > currently. There's unfortunately a few features that'd be a bit painful to not > have. Yes. I imagine I'll upgrade to 22.04 LTS soon, which appears to provide a new enough version of meson. >> * The installed binaries were unable to locate libraries like libpq. I >> ended up setting the extra_lib_dirs option to the directory where these >> libraries were installed to fix this. This one is probably worth >> investigating further. > > I think that should be "fixed" in a later commit in the meson tree - any > chance you could try that? Yup, after cherry-picking 9bc60bc, this is fixed. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com
Re: [RFC] building postgres with meson - v13
On Thu, Sep 22, 2022 at 2:50 PM Andres Freund wrote: > I'm likely the most biased person on this, but for me the reliable incremental > builds and the readability of the test output are big enough wins that the > answer is pretty clear... Doesn't hurt that running all tests is faster too. It's nice that things are much more discoverable now. For example, if you want to run some random test on its own then you just...do it in the obvious, discoverable way. It took me about 2 minutes to figure out how to do that, without reading any documentation. OTOH doing the same thing with the old autoconf-based build system requires the user to know the exact magical incantation for Postgres tests. You just have to know to run the 2 or 3 tests that are undocumented (or poorly documented) dependencies first. That seems like an enormous usability improvement, especially for those of us that haven't been working on Postgres for years. > time to run all tests (cassert, -Og), in a fully built tree: > time make -j48 -s -Otarget check-world PROVE_FLAGS='-j4' > real1m1.577s > user7m32.579s > sys 2m17.767s > time meson test > real0m42.178s > user7m8.533s > sys 2m17.711s Sold! -- Peter Geoghegan
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-22 13:21:28 -0700, Peter Geoghegan wrote: > Is it generally recommended that individual hackers mostly switch over > to Meson for their day to day work soon? I'm guessing that this > question doesn't really have a clear answer yet, but thought I'd ask, > just in case. It'll probably depend on who you ask ;) I'm likely the most biased person on this, but for me the reliable incremental builds and the readability of the test output are big enough wins that the answer is pretty clear... Doesn't hurt that running all tests is faster too. The currently existing limitations are imo mostly around making it usable for production, particularly on windows. time to run all tests (cassert, -Og), in a fully built tree: make: time make -j48 -s -Otarget check-world real2m44.206s user6m29.121s sys 1m54.069s time make -j48 -s -Otarget check-world PROVE_FLAGS='-j4' real1m1.577s user7m32.579s sys 2m17.767s meson: time meson test real0m42.178s user7m8.533s sys 2m17.711s FWIW, I just rebased my older patch to cache and copy initdb during the tests. The %user saved is impressive enough to pursue it again... time make -j48 -s -Otarget check-world PROVE_FLAGS='-j4' real0m52.655s user2m19.504s sys 1m26.264s time meson test: real0m36.370s user2m14.748s sys 1m36.741s Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-22 13:05:33 -0700, Nathan Bossart wrote: > I gave the meson build system a try, and it seems to work nicely. It > didn't take long at all to adapt my workflow. > > A few notes from my experience: > > * I'm using an Ubuntu-based distribution, and the version of meson that apt > installed was not new enough for Postgres. I ended up cloning meson [0] > and using the newest tag. This is no big deal. I assume this is 20.04 LTS? If so, we're missing it by one version of meson currently. There's unfortunately a few features that'd be a bit painful to not have. > * The installed binaries were unable to locate libraries like libpq. I > ended up setting the extra_lib_dirs option to the directory where these > libraries were installed to fix this. This one is probably worth > investigating further. I think that should be "fixed" in a later commit in the meson tree - any chance you could try that? https://github.com/anarazel/postgres/tree/meson > * meson really doesn't like it when there are things leftover from > configure/make. Whenever I switch from make to meson, I have to run 'make > maintainer-clean'. Yes. I recommend building out-of-tree with autoconf as well. > Otherwise, all of my usual build options, ccache, etc. are working just > like before. Nice work! Cool! Thanks for testing, Andres Freund
Re: [RFC] building postgres with meson - v13
On Thu, Sep 22, 2022 at 1:05 PM Nathan Bossart wrote: > Otherwise, all of my usual build options, ccache, etc. are working just > like before. Nice work! +1 Is it generally recommended that individual hackers mostly switch over to Meson for their day to day work soon? I'm guessing that this question doesn't really have a clear answer yet, but thought I'd ask, just in case. -- Peter Geoghegan
Re: [RFC] building postgres with meson - v13
I gave the meson build system a try, and it seems to work nicely. It didn't take long at all to adapt my workflow. A few notes from my experience: * I'm using an Ubuntu-based distribution, and the version of meson that apt installed was not new enough for Postgres. I ended up cloning meson [0] and using the newest tag. This is no big deal. * The installed binaries were unable to locate libraries like libpq. I ended up setting the extra_lib_dirs option to the directory where these libraries were installed to fix this. This one is probably worth investigating further. * meson really doesn't like it when there are things leftover from configure/make. Whenever I switch from make to meson, I have to run 'make maintainer-clean'. Otherwise, all of my usual build options, ccache, etc. are working just like before. Nice work! [0] https://github.com/mesonbuild/meson -- Nathan Bossart Amazon Web Services: https://aws.amazon.com
Re: [RFC] building postgres with meson - v13
On 2022-Sep-22, Tom Lane wrote: > Ah, right, the joys of maintaining multiple build systems. I wonder > if there's any way to avoid that by scraping file lists from one > group to the other. Or maybe we could have a file common to both, say OBJS, which both scrape in their own way. That could be easier than one scraping the other. > We got a little spoiled perhaps by the MSVC scripts managing to do > that in most cases. Right ... and it was so annoying in the cases it couldn't. -- Álvaro HerreraBreisgau, Deutschland — https://www.EnterpriseDB.com/ "Aprender sin pensar es inútil; pensar sin aprender, peligroso" (Confucio)
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-22 04:29:15 -0400, Andrew Dunstan wrote: > Great. Now I'll start on buildfarm support. Given my current > commitments, this will take me a while, but I hope to have a working > client by about the beginning of November. Great! Let me know if there's something I can do to help. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-22 16:56:57 +0200, Alvaro Herrera wrote: > On 2022-Sep-22, Tom Lane wrote: > > Initial reports from the cfbot are mostly promising, but there are a few > > patches where all the meson builds fail while all the autoconf ones pass, > > so there's something for you to look at. So far CF entries 3464, 3733, > > 3771, 3808 look that way. > > Hmm, but those patches add files, which means they're now outdated: they > need to add these files to the respective meson.build file. Yea, I looked through all of these and they all need need a simple addition of a file to be built or installed. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
Alvaro Herrera writes: > On 2022-Sep-22, Tom Lane wrote: >> Initial reports from the cfbot are mostly promising, but there are a few >> patches where all the meson builds fail while all the autoconf ones pass, >> so there's something for you to look at. So far CF entries 3464, 3733, >> 3771, 3808 look that way. > Hmm, but those patches add files, which means they're now outdated: they > need to add these files to the respective meson.build file. Ah, right, the joys of maintaining multiple build systems. I wonder if there's any way to avoid that by scraping file lists from one group to the other. We got a little spoiled perhaps by the MSVC scripts managing to do that in most cases. regards, tom lane
Re: [RFC] building postgres with meson - v13
On 2022-Sep-22, Tom Lane wrote: > Initial reports from the cfbot are mostly promising, but there are a few > patches where all the meson builds fail while all the autoconf ones pass, > so there's something for you to look at. So far CF entries 3464, 3733, > 3771, 3808 look that way. Hmm, but those patches add files, which means they're now outdated: they need to add these files to the respective meson.build file. -- Álvaro HerreraBreisgau, Deutschland — https://www.EnterpriseDB.com/ "La experiencia nos dice que el hombre peló millones de veces las patatas, pero era forzoso admitir la posibilidad de que en un caso entre millones, las patatas pelarían al hombre" (Ijon Tichy)
Re: [RFC] building postgres with meson - v13
Andres Freund writes: > On 2022-09-21 09:46:30 -0700, Andres Freund wrote: >> I'm planning to commit this today, unless somebody wants to argue against >> that. > And done! Yay! Initial reports from the cfbot are mostly promising, but there are a few patches where all the meson builds fail while all the autoconf ones pass, so there's something for you to look at. So far CF entries 3464, 3733, 3771, 3808 look that way. regards, tom lane
Re: [RFC] building postgres with meson - v13
On 2022-09-22 Th 01:57, Andres Freund wrote: > Hi, > > On 2022-09-21 09:46:30 -0700, Andres Freund wrote: >> I'm planning to commit this today, unless somebody wants to argue against >> that. > And done! > > Changes: > - fixed a few typos (thanks Thomas) > - less duplication in the CI tasks > - removed an incomplete implementation of the target for abbrevs.txt - do we > even want to have that? > - plenty hand wringing on my part > > > I also rebased my meson git tree, which still has plenty additional test > platforms (netbsd, openbsd, debian sid, fedora rawhide, centos 8, centos 7, > opensuse tumbleweed), but without the autoconf versions of those targets. I > also added a commit that translates most of the CompilerWarnings task to > meson. Still need to add a headerscheck / cpluspluscheck target. > Great. Now I'll start on buildfarm support. Given my current commitments, this will take me a while, but I hope to have a working client by about the beginning of November. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-21 09:46:30 -0700, Andres Freund wrote: > I'm planning to commit this today, unless somebody wants to argue against > that. And done! Changes: - fixed a few typos (thanks Thomas) - less duplication in the CI tasks - removed an incomplete implementation of the target for abbrevs.txt - do we even want to have that? - plenty hand wringing on my part I also rebased my meson git tree, which still has plenty additional test platforms (netbsd, openbsd, debian sid, fedora rawhide, centos 8, centos 7, opensuse tumbleweed), but without the autoconf versions of those targets. I also added a commit that translates most of the CompilerWarnings task to meson. Still need to add a headerscheck / cpluspluscheck target. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-21 09:46:30 -0700, Andres Freund wrote: > After that I am planning to split the "ci" commit so that it converts a few of > the CI tasks to use meson, without adding all the other platforms I added for > development. I think that's important to get in soon, given that it'll > probably take a bit until the buildfarm grows meson coverage and because it > provides cfbot coverage which seems important for now as well. > > I think we should: > > - convert windows to build with ninja - it builds faster, runs all tests, > parallelizes tests. That means that msbuild based builds don't have coverage > via CI / cfbot, but we don't currently have the resources to test both. I was working on that and hit an issue that took me a while to resolve: Once I tested only the "main" meson commit plus CI the windows task was running out of memory. There was an outage of the CI provider at the same time, so I first blamed it on that. But it turns out to be "legitimately" high memory usage related to debug symbols - the only reason CI didn't show that before was that it's incidentally fixed as a indirect consequence of using precompiled headers, in a later commit. Argh. It can also be fixed by the option required to use ccache at some point, so I'll do that for now. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On Wed, Sep 21, 2022 at 09:46:30AM -0700, Andres Freund wrote: > I think we should: > > - convert windows to build with ninja - it builds faster, runs all tests, > parallelizes tests. That means that msbuild based builds don't have coverage > via CI / cfbot, but we don't currently have the resources to test both. +1 If multiple Windows (or other) tasks are going to exist, I think they should have separate "ci-os-only" conditions, like windows-msvc, windows-ninja, ... It should be possible to run only one.
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-21 13:56:37 -0400, Tom Lane wrote: > Andres Freund writes: > > I think we should: > > > - convert windows to build with ninja - it builds faster, runs all tests, > > parallelizes tests. That means that msbuild based builds don't have > > coverage > > via CI / cfbot, but we don't currently have the resources to test both. > > Check. The sooner we can get rid of the custom MSVC scripts, the better, > because now we'll be on the hook to maintain *three* build systems. Agreed. I think the only "major" missing thing is the windows resource file generation stuff, which is mostly done in one of the "later" commits. Also need to test a few more of the optional dependencies (ICU, gettext, ...) on windows (I did test zlib, lz4, zstd). And of course get a bit of wider exposure than "just me and CI". > > I'm less clear on whether we should convert macos / freebsd to meson at this > > point? > > We certainly could debug/polish the meson stuff just on linux and windows > for now, but is there a reason to wait on the others? No - freebsd and macos have worked in CI for a long time. I was wondering whether we want more coverage for autoconf in CI, but thinking about it futher, it's more important to have the meson coverage. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
Andres Freund writes: > I think we should: > - convert windows to build with ninja - it builds faster, runs all tests, > parallelizes tests. That means that msbuild based builds don't have coverage > via CI / cfbot, but we don't currently have the resources to test both. Check. The sooner we can get rid of the custom MSVC scripts, the better, because now we'll be on the hook to maintain *three* build systems. > - add a linux build using meson, we currently can afford building both with > autoconf and meson for linux Right. > I'm less clear on whether we should convert macos / freebsd to meson at this > point? We certainly could debug/polish the meson stuff just on linux and windows for now, but is there a reason to wait on the others? regards, tom lane
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-21 09:52:48 +0700, John Naylor wrote: > On Wed, Sep 21, 2022 at 7:11 AM Andres Freund wrote: > > > > Hi, > > > > On 2022-09-19 19:16:30 -0700, Andres Freund wrote: > > > To have some initial "translation" for other developers I've started a > wiki > > > page with a translation table. Still very WIP: > > > https://wiki.postgresql.org/wiki/Meson > > > > > > For now, a bit of polishing aside, I'm just planning to add a minimal > > > explanation of what's happening, and a reference to this thread. > > > > I added installation instructions for meson for a bunch of platforms, but > > Small typo: The homebrew section is still labeled with "find MacPorts > libraries". Thanks, fixed. I wrote these blindly, so there's probably more wrong than this - although Thomas was helpful enough to provide some information / testing. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On Wed, Sep 21, 2022 at 7:11 AM Andres Freund wrote: > > Hi, > > On 2022-09-19 19:16:30 -0700, Andres Freund wrote: > > To have some initial "translation" for other developers I've started a wiki > > page with a translation table. Still very WIP: > > https://wiki.postgresql.org/wiki/Meson > > > > For now, a bit of polishing aside, I'm just planning to add a minimal > > explanation of what's happening, and a reference to this thread. > > I added installation instructions for meson for a bunch of platforms, but Small typo: The homebrew section is still labeled with "find MacPorts libraries". -- John Naylor EDB: http://www.enterprisedb.com
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-19 19:16:30 -0700, Andres Freund wrote: > To have some initial "translation" for other developers I've started a wiki > page with a translation table. Still very WIP: > https://wiki.postgresql.org/wiki/Meson > > For now, a bit of polishing aside, I'm just planning to add a minimal > explanation of what's happening, and a reference to this thread. I added installation instructions for meson for a bunch of platforms, but failed to figure out how to do so in a rhel9 container. I don't have a rhel subscription, and apparently the repos with developer tools now require a subscription. Great way to make it easy for projects to test anything on RHEL. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-19 05:25:59 -0400, Peter Eisentraut wrote: > IMO, the following commits are ready to be pushed now: Slowly working through them. To have some initial "translation" for other developers I've started a wiki page with a translation table. Still very WIP: https://wiki.postgresql.org/wiki/Meson For now, a bit of polishing aside, I'm just planning to add a minimal explanation of what's happening, and a reference to this thread. > b7d7fe009731 Remove DLLTOOL, DLLWRAP from configure / Makefile.global.in > 979f26889544 Don't hardcode tmp_check/ as test directory for tap tests > 9fc657fbb7e2 Split TESTDIR into TESTLOGDIR and TESTDATADIR > 6de8f1de0ffa meson: prereq: Extend gendef.pl in preparation for meson > 5a9731dcc2e6 meson: prereq: port: Include c.h instead of postgres.h in > *p{read,write}*.c Done > 7054861f0fef meson: prereq: Add src/tools/gen_export.pl This one I'm planning to merge with the "main" commit, given there's no other user. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On 19.09.22 02:29, Andres Freund wrote: Hi, On September 18, 2022 5:24:06 PM PDT, Peter Eisentraut wrote: On 15.09.22 04:26, Andres Freund wrote: Attached is v13 of the meson patchset. The biggest changes are: Did something about warning flags change from the previous patch set? I see it's building with -Wextra now, which combined with -Werror causes the build to fail for me. I have never encountered that with any of the previous patch sets. In older versions of the patch the default warning level was set to include Wextra, and I had added my local flags to suppress uninteresting warnings. Comparing the warning flags I reduced the warning level and removed the suppressing flags - but changing default options only affects new build trees. To change existing ones do meson configure -Dwarning_level=1 Ok that was the reason. It works now. IMO, the following commits are ready to be pushed now: b7d7fe009731 Remove DLLTOOL, DLLWRAP from configure / Makefile.global.in 979f26889544 Don't hardcode tmp_check/ as test directory for tap tests 9fc657fbb7e2 Split TESTDIR into TESTLOGDIR and TESTDATADIR 6de8f1de0ffa meson: prereq: Extend gendef.pl in preparation for meson 7054861f0fef meson: prereq: Add src/tools/gen_export.pl 1aa586f2921c meson: prereq: Refactor PG_TEST_EXTRA logic in autoconf build 5a9731dcc2e6 meson: prereq: port: Include c.h instead of postgres.h in *p{read,write}*.c 1939bdcfbfea meson: Add meson based buildsystem
Re: [RFC] building postgres with meson - v13
Hi, On September 18, 2022 5:24:06 PM PDT, Peter Eisentraut wrote: >On 15.09.22 04:26, Andres Freund wrote: >> Attached is v13 of the meson patchset. The biggest changes are: > >Did something about warning flags change from the previous patch set? I see >it's building with -Wextra now, which combined with -Werror causes the build >to fail for me. I have never encountered that with any of the previous patch >sets. In older versions of the patch the default warning level was set to include Wextra, and I had added my local flags to suppress uninteresting warnings. Comparing the warning flags I reduced the warning level and removed the suppressing flags - but changing default options only affects new build trees. To change existing ones do meson configure -Dwarning_level=1 -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [RFC] building postgres with meson - v13
On 15.09.22 04:26, Andres Freund wrote: Attached is v13 of the meson patchset. The biggest changes are: Did something about warning flags change from the previous patch set? I see it's building with -Wextra now, which combined with -Werror causes the build to fail for me. I have never encountered that with any of the previous patch sets.
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-16 16:22:35 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2022-09-16 09:14:20 +1200, Thomas Munro wrote: > >> On Thu, Sep 15, 2022 at 2:26 PM Andres Freund wrote: > >>> - noticed that libpgport.a had and needed a dependency on errcodes.h - > >>> that > >>> seemed wrong. The dependency is due to src/port/*p{read,write}v?.c > >>> including > >>> postgres.h - which seems wrong. So I added a patch changing them to > >>> include > >>> c.h. > > >> Oops. +1 > > > Looks like this has been the case since > > 0d56acfbaa799553c0c6ea350fd6e68d81025994 in 14. Any opinions on whether we > > should backpatch the "fix"? > > +1, those files have no business including all of postgres.h Done. I've been wondering whether we should protect against this kind of issue on the buildsystem level. Whenever building frontend code, add something like -DBUILDING_FRONTEND, and error out if postgres.h is included without going through postgres_fe.h. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
Andres Freund writes: > On 2022-09-16 09:14:20 +1200, Thomas Munro wrote: >> On Thu, Sep 15, 2022 at 2:26 PM Andres Freund wrote: >>> - noticed that libpgport.a had and needed a dependency on errcodes.h - that >>> seemed wrong. The dependency is due to src/port/*p{read,write}v?.c including >>> postgres.h - which seems wrong. So I added a patch changing them to include >>> c.h. >> Oops. +1 > Looks like this has been the case since > 0d56acfbaa799553c0c6ea350fd6e68d81025994 in 14. Any opinions on whether we > should backpatch the "fix"? +1, those files have no business including all of postgres.h regards, tom lane
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-16 09:14:20 +1200, Thomas Munro wrote: > On Thu, Sep 15, 2022 at 2:26 PM Andres Freund wrote: > > - noticed that libpgport.a had and needed a dependency on errcodes.h - that > > seemed wrong. The dependency is due to src/port/*p{read,write}v?.c > > including > > postgres.h - which seems wrong. So I added a patch changing them to > > include > > c.h. > > Oops. +1 Looks like this has been the case since 0d56acfbaa799553c0c6ea350fd6e68d81025994 in 14. Any opinions on whether we should backpatch the "fix"? Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-16 09:14:20 +1200, Thomas Munro wrote: > GCC 12 produces a bunch of warnings by default with meson, and that > turned out to be because the default optimisation level is -O3. > That's a change from the make build, which uses -O2. Should we set a > default of 2, or is there some meson-way-of-doing-things reason why > not? We can change the defaults - the only downside is that there's a convenience setting 'buildtype' (debug, debugoptimized, release, minsize, custom, plain) that changes multiple settings (optimization level, amount of debugging information) and that doesn't work as nicely if you change the default compiler optimization setting. They made a similar discovery as us, deriving the defaults of settings based on other settings quickly can become confusing. I think they're looking at how to make that UI a bit nicer. I'd prefer to defer fine-tuning the default settings till a bunch of this has gone in, but I won't insist on that course. Their default warning flags passed to compilers trigger a bunch of warnings in our build (irrespective of -O*), so I lowered the warning level. But I think their set of settings likely is sensible, an we should just disable a bunch of warnings we don't care about. But I haven't done that for now, to keep the set of warning flags the same between meson and autoconf. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
On Thu, Sep 15, 2022 at 2:26 PM Andres Freund wrote: > - noticed that libpgport.a had and needed a dependency on errcodes.h - that > seemed wrong. The dependency is due to src/port/*p{read,write}v?.c including > postgres.h - which seems wrong. So I added a patch changing them to include > c.h. Oops. +1 GCC 12 produces a bunch of warnings by default with meson, and that turned out to be because the default optimisation level is -O3. That's a change from the make build, which uses -O2. Should we set a default of 2, or is there some meson-way-of-doing-things reason why not?
Re: [RFC] building postgres with meson - v13
Hi, On 2022-09-15 01:10:16 -0400, Tom Lane wrote: > Andres Freund writes: > > I'm inclined to build the static lib on windows as long as we do it on other > > platforms. > > Maybe I spent too much time working for Red Hat, but I'm kind of > unhappy that we build static libraries at all. Yea, I have been wondering about that too. Oddly enough, given our current behaviour, the strongest case for static libraries IMO is on windows, due to the lack of a) rpath b) a general library search path. Peter IIRC added the static libraries to the meson port just to keep the set of installed files the same, which makes sense. > They are maintenance hazards and therefore security hazards by definition, > because if you find a problem in $package_x you will have to find and > rebuild every other package that has statically-embedded code from > $package_x. So Red Hat has, or least had, a policy against packages > exporting such libraries. It obviously is a bad idea for widely used system packages. I think there are a few situations, e.g. a downloadable self-contained and relocatable application, where shared libraries provide less of a benefit. > I realize that there are people for whom other considerations outweigh > that, but I don't think that we should install static libraries by > default. Long ago it was pretty common for configure scripts to > offer --enable-shared and --enable-static options ... should we > resurrect that? It'd be easy enough. I don't really have an opinion on whether it's worth having the options. I think most packaging systems have ways of not including files even if $software installs them. Greetings, Andres Freund
Re: [RFC] building postgres with meson - v13
Andres Freund writes: > I'm inclined to build the static lib on windows as long as we do it on other > platforms. Maybe I spent too much time working for Red Hat, but I'm kind of unhappy that we build static libraries at all. They are maintenance hazards and therefore security hazards by definition, because if you find a problem in $package_x you will have to find and rebuild every other package that has statically-embedded code from $package_x. So Red Hat has, or least had, a policy against packages exporting such libraries. I realize that there are people for whom other considerations outweigh that, but I don't think that we should install static libraries by default. Long ago it was pretty common for configure scripts to offer --enable-shared and --enable-static options ... should we resurrect that? regards, tom lane