Re: Guiding principle for dropping LLVM versions?
On 1/25/24 13:41, Thomas Munro wrote: On Thu, Jan 25, 2024 at 4:44 PM Thomas Munro wrote: ... A few build farm animals will now fail in the configure step as discussed, and need some adjustment (ie disable LLVM or upgrade to LLVM 10+ for the master branch). Owners pinged. I think I fixed up the 4 or 6 under my name... Regards, Mark
Re: Guiding principle for dropping LLVM versions?
On Thu, Jan 25, 2024 at 4:44 PM Thomas Munro wrote: > ... A few build farm animals will > now fail in the configure step as discussed, and need some adjustment > (ie disable LLVM or upgrade to LLVM 10+ for the master branch). Owners pinged.
Re: Guiding principle for dropping LLVM versions?
Thanks all for the discussion. Pushed. A few build farm animals will now fail in the configure step as discussed, and need some adjustment (ie disable LLVM or upgrade to LLVM 10+ for the master branch). Next year I think we should be able to do a much bigger cleanup, by moving to LLVM 14+.
Re: Guiding principle for dropping LLVM versions?
2024-01 Commitfest. Hi, This patch has a CF status of "Ready for Committer", but it looks like it failed when the CFbot test for it was last run [1]. Please have a look and post an updated version.. == [1] https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/46/4640 Kind Regards, Peter Smith.
Re: Guiding principle for dropping LLVM versions?
On 25.10.23 07:47, Thomas Munro wrote: Ideally more distros would be present in this vacuum-horizon decision table, but I don't think it'd change the conclusion: 10 is the trailing edge. Therefore the attached patch scales back its ambition to that release. Tested on LLVM 10-18. This patch and the associated reasoning look good to me. I think this is good to go for PG17.
Re: Guiding principle for dropping LLVM versions?
Hi, On 2023-11-02 10:46:52 +1300, Thomas Munro wrote: > So it sounds like we're in agreement that it is time to require LLVM > 10+ in master. Could the owners (CC'd) of the following animals > please turn off --with-llvm on master (and future 17+ branches), or > consider upgrading to a modern OS release? Otherwise they'll turn > red. > > And of course Andres would need to do the same for his coverage > animals in that range: > >animal|arch| llvm_version | os | os_release > | end_of_support > -++--+++ > dragonet| x86_64 | 3.9.1| Debian | sid| > phycodurus | x86_64 | 3.9.1| Debian | sid| > desmoxytes | x86_64 | 4.0.1| Debian | sid| > petalura| x86_64 | 4.0.1| Debian | sid| > idiacanthus | x86_64 | 5.0.2| Debian | sid| > pogona | x86_64 | 5.0.2| Debian | sid| > komodoensis | x86_64 | 6.0.1| Debian | sid| > xenodermus | x86_64 | 6.0.1| Debian | sid| Would you want me to do this now or just before you apply the patch? I think I should stand up a few more replacement animals to cover older llvm versions with assertions enabled... Greetings, Andres Freund
Re: Guiding principle for dropping LLVM versions?
So it sounds like we're in agreement that it is time to require LLVM 10+ in master. Could the owners (CC'd) of the following animals please turn off --with-llvm on master (and future 17+ branches), or consider upgrading to a modern OS release? Otherwise they'll turn red. animal|arch| llvm_version | os | os_release | end_of_support -++--+++ mantid | x86_64 | 5.0.1| CentOS | 7 | 2019-08-06 cotinga | s390x | 6.0.0| Ubuntu | 18.04 | 2023-06-01 vimba | aarch64| 6.0.0| Ubuntu | 18.04 | 2023-06-01 topminnow | mips64el; -mabi=32 | 6.0.1| Debian | 8 | 2018-06-17 alimoche| aarch64| 7.0.1| Debian | 10 | 2022-09-10 blackneck | aarch64| 7.0.1| Debian | 10 | 2022-09-10 And of course Andres would need to do the same for his coverage animals in that range: animal|arch| llvm_version | os | os_release | end_of_support -++--+++ dragonet| x86_64 | 3.9.1| Debian | sid| phycodurus | x86_64 | 3.9.1| Debian | sid| desmoxytes | x86_64 | 4.0.1| Debian | sid| petalura| x86_64 | 4.0.1| Debian | sid| idiacanthus | x86_64 | 5.0.2| Debian | sid| pogona | x86_64 | 5.0.2| Debian | sid| komodoensis | x86_64 | 6.0.1| Debian | sid| xenodermus | x86_64 | 6.0.1| Debian | sid|
Re: Guiding principle for dropping LLVM versions?
Hi, On Thu, 2023-10-19 at 08:13 +1300, Thomas Munro wrote: > If we used Debian as a yardstick, PostgreSQL 17 wouldn't need anything > older than LLVM 14 AFAICS. Who else do we need to ask? LLVM 15 is the minimum one for the platforms that I build the packages on. So LLVM >= 14 is great for HEAD. Regards, -- Devrim Gündüz Open Source Solution Architect, PostgreSQL Major Contributor Twitter: @DevrimGunduz , @DevrimGunduzTR
Re: Guiding principle for dropping LLVM versions?
On Wed, Oct 25, 2023 at 7:12 PM Tom Lane wrote: > Thomas Munro writes: > > 3. We exclude OSes that don't bless an LLVM release (eg macOS running > > an arbitrarily picked version), and animals running only to cover > > ancient LLVM compiled from source for coverage (Andres's sid > > menagerie). > > Seems generally reasonable. Maybe rephrase 3 as "We consider only > an OS release's default LLVM version"? Or a bit more forgivingly, > "... only LLVM versions available from the OS vendor"? Also, > what's an OS vendor? You rejected macOS which is fine, but > I think the packages available from MacPorts or Homebrew should > be considered. OK. For me the key differences are that they are independent of OS releases and time lines, they promptly add new releases, they have a wide back-catalogue of the old releases and they let the user decide which to use. So I don't think they constrain us and it makes no sense to try to apply 'end of support' logic to them. https://formulae.brew.sh/formula/llvm https://ports.macports.org/search/?q=llvm&name=on (Frustratingly, the ancient releases of clang don't actually seem to work on MacPorts at least on aarch64, and they tell you so when you try to install them.) The BSDs may be closer to macOS in that respect too, since they have ports separate from OS releases and they offer a rolling and wide range of LLVMs and generally default to a very new one, so I don't think they constrain us either. It's really Big Linux that is drawing the lines in the sand here, though (unusually) not RHEL-and-frenemies as they've opted for rolling to the latest in every minor release as Devrim explained.
Re: Guiding principle for dropping LLVM versions?
Thomas Munro writes: > Here are some systematic rules I'd like to propose to anchor this > stuff to reality and avoid future doubt and litigation: > 1. Build farm animals testing LLVM determine the set of OSes and LLVM > versions we consider. > 2. We exclude OSes that will be out of full vendor support when a > release ships. > 3. We exclude OSes that don't bless an LLVM release (eg macOS running > an arbitrarily picked version), and animals running only to cover > ancient LLVM compiled from source for coverage (Andres's sid > menagerie). Seems generally reasonable. Maybe rephrase 3 as "We consider only an OS release's default LLVM version"? Or a bit more forgivingly, "... only LLVM versions available from the OS vendor"? Also, what's an OS vendor? You rejected macOS which is fine, but I think the packages available from MacPorts or Homebrew should be considered. You could imagine somebody trying to game the system by standing up a buildfarm animal running some really arbitrary combination of versions --- but what would be the point? I think we can deal with that when/if it happens. But "macOS running an LLVM version available from MacPorts" doesn't seem arbitrary. regards, tom lane
Re: Guiding principle for dropping LLVM versions?
Here are some systematic rules I'd like to propose to anchor this stuff to reality and avoid future doubt and litigation: 1. Build farm animals testing LLVM determine the set of OSes and LLVM versions we consider. 2. We exclude OSes that will be out of full vendor support when a release ships. 3. We exclude OSes that don't bless an LLVM release (eg macOS running an arbitrarily picked version), and animals running only to cover ancient LLVM compiled from source for coverage (Andres's sid menagerie). By these rules we can't require LLVM 14 for another year, because Ubuntu and Amazon Linux are standing in the way*: animal | arch | llvm_version | os | os_release | end_of_support ---+-+--+++ branta| s390x | 10.0.0 | Ubuntu | 20.04 | 2025-04-01 splitfin | aarch64 | 10.0.0 | Ubuntu | 20.04 | 2025-04-01 urutau| s390x | 10.0.0 | Ubuntu | 20.04 | 2025-04-01 massasauga| aarch64 | 11.1.0 | Amazon | 2 | 2025-06-30 snakefly | aarch64 | 11.1.0 | Amazon | 2 | 2025-06-30 sarus | s390x | 14.0.0 | Ubuntu | 22.04 | 2027-06-01 shiner| aarch64 | 14.0.0 | Ubuntu | 22.04 | 2027-06-01 turbot| aarch64 | 14.0.0 | Ubuntu | 22.04 | 2027-06-01 lora | s390x | 15.0.7 | RHEL | 9 | 2027-05-31 mamushi | s390x | 15.0.7 | RHEL | 9 | 2027-05-31 nicator | ppc64le | 15.0.7 | Alma | 9 | 2027-05-31 oystercatcher | aarch64 | 15.0.7 | Alma | 9 | 2027-05-31 Ideally more distros would be present in this vacuum-horizon decision table, but I don't think it'd change the conclusion: 10 is the trailing edge. Therefore the attached patch scales back its ambition to that release. Tested on LLVM 10-18. If I pushed this we'd need to disable or upgrade the following to avoid failure in configure on master: animal|arch| llvm_version | os | os_release | end_of_support -++--+++ dragonet| x86_64 | 3.9.1| Debian | sid| phycodurus | x86_64 | 3.9.1| Debian | sid| desmoxytes | x86_64 | 4.0.1| Debian | sid| petalura| x86_64 | 4.0.1| Debian | sid| mantid | x86_64 | 5.0.1| CentOS | 7 | 2019-08-06 idiacanthus | x86_64 | 5.0.2| Debian | sid| pogona | x86_64 | 5.0.2| Debian | sid| cotinga | s390x | 6.0.0| Ubuntu | 18.04 | 2023-06-01 vimba | aarch64| 6.0.0| Ubuntu | 18.04 | 2023-06-01 komodoensis | x86_64 | 6.0.1| Debian | sid| topminnow | mips64el; -mabi=32 | 6.0.1| Debian | 8 | 2018-06-17 xenodermus | x86_64 | 6.0.1| Debian | sid| alimoche| aarch64| 7.0.1| Debian | 10 | 2022-09-10 blackneck | aarch64| 7.0.1| Debian | 10 | 2022-09-10 bonito | ppc64le| 7.0.1| Fedora | 29 | 2019-11-26 *Some distros announce EOL date by month without saying which day, so in my data collecting operation I just punched in the first of the month, *shrug* From 581b88879316065dab113d1512aeac7e9932b0af Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 19 Oct 2023 04:45:46 +1300 Subject: [PATCH v4] jit: Require at least LLVM 10. Remove support for older LLVM versions. The default on common software distributions will be at least LLVM 10 when PostgreSQL 17 ships. Discussion: https://postgr.es/m/CA%2BhUKGLhNs5geZaVNj2EJ79Dx9W8fyWUU3HxcpZy55sMGcY%3DiA%40mail.gmail.com --- config/llvm.m4 | 10 ++-- configure | 43 ++--- doc/src/sgml/installation.sgml | 4 +- meson.build | 2 +- src/backend/jit/llvm/llvmjit.c | 53 ++--- src/backend/jit/llvm/llvmjit_error.cpp | 10 src/backend/jit/llvm/llvmjit_expr.c | 6 +-- src/backend/jit/llvm/llvmjit_inline.cpp | 38 +-- src/backend/jit/llvm/llvmjit_wrap.cpp | 61 - src/include/jit/llvmjit.h | 17 --- src/include/pg_config.h.in | 12 - src/tools/msvc/Solution.pm | 3 -- 12 files changed, 17 insertions(+), 242 deletions(-) diff --git a/config/llvm.m4 b/config/llvm.m4 index 21d8cd4f90..44769d819a 100644 --- a/config/llvm.m4 +++ b/config/llvm.m4 @@ -13,7 +13,7 @@ AC_DEFUN([PGAC_LLVM_SUPPORT], AC_REQUIRE([AC_PROG_AWK]) AC_ARG_VAR(LLVM_CONFIG, [path to llvm-config command]) - PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-7 llvm-config-6.0 llvm-config-5.0 llvm-confi
Re: Guiding principle for dropping LLVM versions?
Hi, On October 21, 2023 7:46:17 PM PDT, Xing Guo wrote: >Can we also check if the clang's version is compatible with llvm's version >in llvm.m4? I have multiple llvm toolchains installed on my system and I >have to specify the $CLANG and $LLVM_CONFIG variables each time I build the >server against a toolchain that is not present in $PATH. If one of the >variables is missing, the build system will pick up a default one whose >version might not be compatible with the other. E.g., If we use clang-16 >and llvm-config-15, there will be issues when creating indexes for bitcodes >at the end of installation. It's unfortunately not that obvious to figure out what is compatible and what not. Older clang versions work, except if too old. Newer versions sometimes work. We could perhaps write a script that will find many, but not all, incompatibilities. For the meson build I made it just use clang belonging to the llvm install - but that's very painful when building against an assert enabled llvm, clang is slower by an order of magnitude or so. I wonder if we should change the search order to 1) CLANG, iff explicitly specified, 2) use explicitly specified or inferred llvm-config, 3) only if that didn't find clang, search path. >wrote: > >> Rebased. I also noticed this woefully out of date line: >> >> - PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-7 >> llvm-config-6.0 llvm-config-5.0 llvm-config-4.0 llvm-config-3.9) >> + PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-17 >> llvm-config-16 llvm-config-15 llvm-config-14) >> It's outdated, but not completely absurd - back then often no llvm-config -> llvm-config-XY was installed, but these days there pretty much always is. Andres -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Guiding principle for dropping LLVM versions?
On Sun, Oct 22, 2023 at 3:46 PM Xing Guo wrote: > Can we also check if the clang's version is compatible with llvm's version in > llvm.m4? I have multiple llvm toolchains installed on my system and I have to > specify the $CLANG and $LLVM_CONFIG variables each time I build the server > against a toolchain that is not present in $PATH. If one of the variables is > missing, the build system will pick up a default one whose version might not > be compatible with the other. E.g., If we use clang-16 and llvm-config-15, > there will be issues when creating indexes for bitcodes at the end of > installation. Hmm. Problems that occur to me: 1. We need to decide if our rule is that clang must be <= llvm, or ==. I think this question has been left unanswered in the past when it has come up. So far I think <= would be enough to avoid the error you showed but can we find where this policy (ie especially commitments for future releases) is written down in LLVM literature? 2. Apple's clang lies about its version (I don't know the story behind that, but my wild guess is that someone from marketing wanted the compiler's version numbers to align with xcode's version numbers? they're off by 1 or something like that). Another idea could be to produce some bitcode with clang, and then check if a relevant LLVM tool can deal with it.
Re: Guiding principle for dropping LLVM versions?
Hi, Can we also check if the clang's version is compatible with llvm's version in llvm.m4? I have multiple llvm toolchains installed on my system and I have to specify the $CLANG and $LLVM_CONFIG variables each time I build the server against a toolchain that is not present in $PATH. If one of the variables is missing, the build system will pick up a default one whose version might not be compatible with the other. E.g., If we use clang-16 and llvm-config-15, there will be issues when creating indexes for bitcodes at the end of installation. There will be errors look like ``` LLVM ERROR: ThinLTO cannot create input file: Unknown attribute kind (86) (Producer: 'LLVM16.0.6' Reader: 'LLVM 15.0.7') PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace. Stack dump: 0. Program arguments: /usr/lib/llvm15/bin/llvm-lto -thinlto -thinlto-action=thinlink -o postgres.index.bc postgres/access/brin/brin.bc postgres/access/brin/brin_bloom.bc postgres/acces s/brin/brin_inclusion.bc postgres/access/brin/brin_minmax.bc postgres/access/brin/brin_minmax_multi.bc postgres/access/brin/brin_pageops.bc postgres/access/brin/brin_revmap.bc postgres/acce ss/brin/brin_tuple.bc postgres/access/brin/brin_validate.bc postgres/access/brin/brin_xlog.bc postgres/access/common/attmap.bc postgres/access/common/bufmask.bc postgres/access/common/detoa st.bc postgres/access/common/heaptuple.bc postgres/access/common/indextuple.bc postgres/access/common/printsimple.bc postgres/access/common/printtup.bc postgres/access/common/relation.bc po stgres/access/common/reloptions.bc postgres/access/common/scankey.bc postgres/access/common/session.bc postgres/access/common/syncscan.bc postgres/access/common/toast_compression.bc postgre s/access/common/toast_internals.bc postgres/access/common/tupconvert.bc postgres/access/common/tupdesc.bc postgres/access/gin/ginarrayproc.bc postgres/access/gin/ginbtree.bc postgres/access /gin/ginbulk.bc postgres/access/gin/gindatapage.bc postgres/access/gin/ginentrypage.bc postgres/access/gin/ginfast.bc postgres/access/gin/ginget.bc postgres/access/gin/gininsert.bc postgres /access/gin/ginlogic.bc postgres/access/gin/ginpostinglist.bc postgres/access/gin/ginscan.bc postgres/access/gin/ginutil.bc postgres/access/gin/ginvacuum.bc postgres/access/gin/ginvalidate. bc postgres/access/gin/ginxlog.bc postgres/access/gist/gist.bc postgres/access/gist/gistbuild.bc postgres/access/gist/gistbuildbuffers.bc postgres/access/gist/gistget.bc postgres/access/gis t/gistproc.bc postgres/access/gist/gistscan.bc postgres/access/gist/gistsplit.bc postgres/access/gist/gistutil.bc postgres/access/gist/gistvacuum.bc postgres/access/gist/gistvalidate.bc pos tgres/access/gist/gistxlog.bc postgres/access/hash/hash.bc postgres/access/hash/hash_xlog.bc postgres/access/hash/hashfunc.bc postgres/access/hash/hashinsert.bc postgres/access/hash/hashovf l.bc postgres/access/hash/hashpage.bc postgres/access/hash/hashsearch.bc postgres/access/hash/hashsort.bc postgres/access/hash/hashutil.bc postgres/access/hash/hashvalidate.bc postgres/acce ss/heap/heapam.bc postgres/access/heap/heapam_handler.bc postgres/access/heap/heapam_visibility.bc postgres/access/heap/heaptoast.bc postgres/access/heap/hio.bc postgres/access/heap/prunehe ``` If we can check the llvm-config versions and clang versions at the configuration phase we can detect the problem earlier. Best Regards, Xing On Sun, Oct 22, 2023 at 10:07 AM Thomas Munro wrote: > Rebased. I also noticed this woefully out of date line: > > - PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-7 > llvm-config-6.0 llvm-config-5.0 llvm-config-4.0 llvm-config-3.9) > + PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-17 > llvm-config-16 llvm-config-15 llvm-config-14) >
Re: Guiding principle for dropping LLVM versions?
Rebased. I also noticed this woefully out of date line:
- PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-7
llvm-config-6.0 llvm-config-5.0 llvm-config-4.0 llvm-config-3.9)
+ PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-17
llvm-config-16 llvm-config-15 llvm-config-14)
From 522a4f3744e691f42f43bb1ca91df60b12465bff Mon Sep 17 00:00:00 2001
From: Thomas Munro
Date: Thu, 19 Oct 2023 04:45:46 +1300
Subject: [PATCH v3 1/2] jit: Require at least LLVM 14, if enabled.
Remove support for a lot of older LLVM versions dating back to 3.9. The
default on common software distritbutions will be at least LLVM 14 when
PostgreSQL 17 ships.
Discussion: https://postgr.es/m/CA%2BhUKGLhNs5geZaVNj2EJ79Dx9W8fyWUU3HxcpZy55sMGcY%3DiA%40mail.gmail.com
---
config/llvm.m4 | 10 +-
configure | 43 +--
doc/src/sgml/installation.sgml | 4 +-
meson.build | 2 +-
src/backend/jit/llvm/llvmjit.c | 158 +---
src/backend/jit/llvm/llvmjit_error.cpp | 35 --
src/backend/jit/llvm/llvmjit_expr.c | 6 +-
src/backend/jit/llvm/llvmjit_inline.cpp | 51 +---
src/backend/jit/llvm/llvmjit_wrap.cpp | 65 --
src/include/jit/llvmjit.h | 17 ---
src/include/pg_config.h.in | 12 --
src/tools/msvc/Solution.pm | 3 -
12 files changed, 15 insertions(+), 391 deletions(-)
diff --git a/config/llvm.m4 b/config/llvm.m4
index 21d8cd4f90..ede5e25e91 100644
--- a/config/llvm.m4
+++ b/config/llvm.m4
@@ -13,7 +13,7 @@ AC_DEFUN([PGAC_LLVM_SUPPORT],
AC_REQUIRE([AC_PROG_AWK])
AC_ARG_VAR(LLVM_CONFIG, [path to llvm-config command])
- PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-7 llvm-config-6.0 llvm-config-5.0 llvm-config-4.0 llvm-config-3.9)
+ PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-17 llvm-config-16 llvm-config-15 llvm-config-14)
# no point continuing if llvm wasn't found
if test -z "$LLVM_CONFIG"; then
@@ -25,14 +25,14 @@ AC_DEFUN([PGAC_LLVM_SUPPORT],
AC_MSG_ERROR([$LLVM_CONFIG does not work])
fi
# and whether the version is supported
- if echo $pgac_llvm_version | $AWK -F '.' '{ if ([$]1 >= 4 || ([$]1 == 3 && [$]2 >= 9)) exit 1; else exit 0;}';then
-AC_MSG_ERROR([$LLVM_CONFIG version is $pgac_llvm_version but at least 3.9 is required])
+ if echo $pgac_llvm_version | $AWK -F '.' '{ if ([$]1 >= 14) exit 1; else exit 0;}';then
+AC_MSG_ERROR([$LLVM_CONFIG version is $pgac_llvm_version but at least 14 is required])
fi
AC_MSG_NOTICE([using llvm $pgac_llvm_version])
# need clang to create some bitcode files
AC_ARG_VAR(CLANG, [path to clang compiler to generate bitcode])
- PGAC_PATH_PROGS(CLANG, clang clang-7 clang-6.0 clang-5.0 clang-4.0 clang-3.9)
+ PGAC_PATH_PROGS(CLANG, clang clang-17 clang-16 clang-15 clang-14)
if test -z "$CLANG"; then
AC_MSG_ERROR([clang not found, but required when compiling --with-llvm, specify with CLANG=])
fi
@@ -115,8 +115,6 @@ AC_DEFUN([PGAC_CHECK_LLVM_FUNCTIONS],
# Check which functionality is present
SAVE_CPPFLAGS="$CPPFLAGS"
CPPFLAGS="$CPPFLAGS $LLVM_CPPFLAGS"
- AC_CHECK_DECLS([LLVMOrcGetSymbolAddressIn], [], [], [[#include ]])
- AC_CHECK_DECLS([LLVMGetHostCPUName, LLVMGetHostCPUFeatures], [], [], [[#include ]])
AC_CHECK_DECLS([LLVMCreateGDBRegistrationListener, LLVMCreatePerfJITEventListener], [], [], [[#include ]])
CPPFLAGS="$SAVE_CPPFLAGS"
])# PGAC_CHECK_LLVM_FUNCTIONS
diff --git a/configure b/configure
index c2cb1b1b24..2d92e3dea3 100755
--- a/configure
+++ b/configure
@@ -5056,7 +5056,7 @@ if test "$with_llvm" = yes; then :
if test -z "$LLVM_CONFIG"; then
- for ac_prog in llvm-config llvm-config-7 llvm-config-6.0 llvm-config-5.0 llvm-config-4.0 llvm-config-3.9
+ for ac_prog in llvm-config llvm-config-17 llvm-config-16 llvm-config-15 llvm-config-14
do
# Extract the first word of "$ac_prog", so it can be a program name with args.
set dummy $ac_prog; ac_word=$2
@@ -5120,8 +5120,8 @@ fi
as_fn_error $? "$LLVM_CONFIG does not work" "$LINENO" 5
fi
# and whether the version is supported
- if echo $pgac_llvm_version | $AWK -F '.' '{ if ($1 >= 4 || ($1 == 3 && $2 >= 9)) exit 1; else exit 0;}';then
-as_fn_error $? "$LLVM_CONFIG version is $pgac_llvm_version but at least 3.9 is required" "$LINENO" 5
+ if echo $pgac_llvm_version | $AWK -F '.' '{ if ($1 >= 14) exit 1; else exit 0;}';then
+as_fn_error $? "$LLVM_CONFIG version is $pgac_llvm_version but at least 14 is required" "$LINENO" 5
fi
{ $as_echo "$as_me:${as_lineno-$LINENO}: using llvm $pgac_llvm_version" >&5
$as_echo "$as_me: using llvm $pgac_llvm_version" >&6;}
@@ -5129,7 +5129,7 @@ $as_echo "$as_me: using llvm $pgac_llvm_version" >&6;}
# need clang to create some bitcode files
if test -z "$CLANG"; then
- for ac_prog in clang clang-7 clang-6.0 clang-5.0 clang-4.0 clang-3.9
+ for ac_prog in clang clang-17 clang-16 clang-15 cl
Re: Guiding principle for dropping LLVM versions?
We could go further. With LLVM 14 as the minimum we can just use
opaque pointers everywhere, and delete more conditional code in
master. Tested on 14-18.
I explored using the new pass manager everywhere too. It almost
worked, but I couldn't see how to override the inlining threshold
before LLVM 16[1], even in C++, so we couldn't fix that with a
llvmjit_wrap.cpp hack.
I like this. How can I find out if someone would shout at me for
dropping LLVM 13?
[1]
https://github.com/llvm/llvm-project/commit/4fa328074efd7eefdbb314b8f6e9f855e443ca20
From dd6005de803fb621cf1e21d9d7d31589e903e35b Mon Sep 17 00:00:00 2001
From: Thomas Munro
Date: Thu, 19 Oct 2023 04:45:46 +1300
Subject: [PATCH v2 1/2] jit: Require at least LLVM 14, if enabled.
Remove support for a lot of older LLVM versions dating back to 3.9. The
default on common software distritbutions will be at least LLVM 14 when
PostgreSQL 17 ships.
Discussion: https://postgr.es/m/CA%2BhUKGLhNs5geZaVNj2EJ79Dx9W8fyWUU3HxcpZy55sMGcY%3DiA%40mail.gmail.com
---
config/llvm.m4 | 6 +-
configure | 39 +-
doc/src/sgml/installation.sgml | 4 +-
meson.build | 2 +-
src/backend/jit/llvm/llvmjit.c | 158 +---
src/backend/jit/llvm/llvmjit_error.cpp | 35 --
src/backend/jit/llvm/llvmjit_expr.c | 6 +-
src/backend/jit/llvm/llvmjit_inline.cpp | 51 +---
src/backend/jit/llvm/llvmjit_wrap.cpp | 65 --
src/include/jit/llvmjit.h | 17 ---
src/include/pg_config.h.in | 12 --
src/tools/msvc/Solution.pm | 3 -
12 files changed, 11 insertions(+), 387 deletions(-)
diff --git a/config/llvm.m4 b/config/llvm.m4
index 3a75cd8b4d..a99f29ffdc 100644
--- a/config/llvm.m4
+++ b/config/llvm.m4
@@ -25,8 +25,8 @@ AC_DEFUN([PGAC_LLVM_SUPPORT],
AC_MSG_ERROR([$LLVM_CONFIG does not work])
fi
# and whether the version is supported
- if echo $pgac_llvm_version | $AWK -F '.' '{ if ([$]1 >= 4 || ([$]1 == 3 && [$]2 >= 9)) exit 1; else exit 0;}';then
-AC_MSG_ERROR([$LLVM_CONFIG version is $pgac_llvm_version but at least 3.9 is required])
+ if echo $pgac_llvm_version | $AWK -F '.' '{ if ([$]1 >= 14) exit 1; else exit 0;}';then
+AC_MSG_ERROR([$LLVM_CONFIG version is $pgac_llvm_version but at least 14 is required])
fi
# need clang to create some bitcode files
@@ -114,8 +114,6 @@ AC_DEFUN([PGAC_CHECK_LLVM_FUNCTIONS],
# Check which functionality is present
SAVE_CPPFLAGS="$CPPFLAGS"
CPPFLAGS="$CPPFLAGS $LLVM_CPPFLAGS"
- AC_CHECK_DECLS([LLVMOrcGetSymbolAddressIn], [], [], [[#include ]])
- AC_CHECK_DECLS([LLVMGetHostCPUName, LLVMGetHostCPUFeatures], [], [], [[#include ]])
AC_CHECK_DECLS([LLVMCreateGDBRegistrationListener, LLVMCreatePerfJITEventListener], [], [], [[#include ]])
CPPFLAGS="$SAVE_CPPFLAGS"
])# PGAC_CHECK_LLVM_FUNCTIONS
diff --git a/configure b/configure
index d47e0f8b26..f4a04485e1 100755
--- a/configure
+++ b/configure
@@ -5120,8 +5120,8 @@ fi
as_fn_error $? "$LLVM_CONFIG does not work" "$LINENO" 5
fi
# and whether the version is supported
- if echo $pgac_llvm_version | $AWK -F '.' '{ if ($1 >= 4 || ($1 == 3 && $2 >= 9)) exit 1; else exit 0;}';then
-as_fn_error $? "$LLVM_CONFIG version is $pgac_llvm_version but at least 3.9 is required" "$LINENO" 5
+ if echo $pgac_llvm_version | $AWK -F '.' '{ if ($1 >= 14) exit 1; else exit 0;}';then
+as_fn_error $? "$LLVM_CONFIG version is $pgac_llvm_version but at least 14 is required" "$LINENO" 5
fi
# need clang to create some bitcode files
@@ -16571,41 +16571,6 @@ if test "$with_llvm" = yes; then
# Check which functionality is present
SAVE_CPPFLAGS="$CPPFLAGS"
CPPFLAGS="$CPPFLAGS $LLVM_CPPFLAGS"
- ac_fn_c_check_decl "$LINENO" "LLVMOrcGetSymbolAddressIn" "ac_cv_have_decl_LLVMOrcGetSymbolAddressIn" "#include
-"
-if test "x$ac_cv_have_decl_LLVMOrcGetSymbolAddressIn" = xyes; then :
- ac_have_decl=1
-else
- ac_have_decl=0
-fi
-
-cat >>confdefs.h <<_ACEOF
-#define HAVE_DECL_LLVMORCGETSYMBOLADDRESSIN $ac_have_decl
-_ACEOF
-
- ac_fn_c_check_decl "$LINENO" "LLVMGetHostCPUName" "ac_cv_have_decl_LLVMGetHostCPUName" "#include
-"
-if test "x$ac_cv_have_decl_LLVMGetHostCPUName" = xyes; then :
- ac_have_decl=1
-else
- ac_have_decl=0
-fi
-
-cat >>confdefs.h <<_ACEOF
-#define HAVE_DECL_LLVMGETHOSTCPUNAME $ac_have_decl
-_ACEOF
-ac_fn_c_check_decl "$LINENO" "LLVMGetHostCPUFeatures" "ac_cv_have_decl_LLVMGetHostCPUFeatures" "#include
-"
-if test "x$ac_cv_have_decl_LLVMGetHostCPUFeatures" = xyes; then :
- ac_have_decl=1
-else
- ac_have_decl=0
-fi
-
-cat >>confdefs.h <<_ACEOF
-#define HAVE_DECL_LLVMGETHOSTCPUFEATURES $ac_have_decl
-_ACEOF
-
ac_fn_c_check_decl "$LINENO" "LLVMCreateGDBRegistrationListener" "ac_cv_have_decl_LLVMCreateGDBRegistrationListener" "#include
"
if test "x$ac_cv_have_decl_LLVMCreateGDBRegistrationListener" = xyes; then :
diff --git a/doc/src/sgml/installa
Re: Guiding principle for dropping LLVM versions?
*rouge
Re: Guiding principle for dropping LLVM versions?
On Thu, Sep 21, 2023 at 12:27 PM Devrim Gündüz wrote: > Even though older LLVM versions exist on both RHEL and Fedora, they > don't provide older Clang packages, which means we have to link to the > latest release anyway (like currently Fedora 38 packages are waiting for > LLVM 16 patch, as they cannot be linked against LLVM 15) That's quite interesting, because it means that RHEL doesn't act as the "lanterne route" for this, ie the most conservative relevant distribution. If we used Debian as a yardstick, PostgreSQL 17 wouldn't need anything older than LLVM 14 AFAICS. Who else do we need to ask? Where could we find this sort of information in machine-readable form (that is feedback I got discussing the wiki page idea with people, ie that it would be bound to become stale and abandoned)? Fresh from doing battle with this stuff, I wanted to see what it would look like if we dropped 3.9...13 in master: 11 files changed, 12 insertions(+), 367 deletions(-) I noticed in passing that the LLVMOrcRegisterJITEventListener configure probes are not present in meson. 0001-jit-Require-at-least-LLVM-14-if-enabled.patch Description: Binary data
Re: Guiding principle for dropping LLVM versions?
> On 21 Sep 2023, at 07:28, Tom Lane wrote: > > Thomas Munro writes: >> I wonder if there is a good way to make this sort of thing more >> systematic. If we could agree on a guiding principle vaguely like the >> above, then perhaps we just need a wiki page that lists relevant >> distributions, versions and EOL dates, that could be used to reduce >> the combinations of stuff we have to consider and make the pruning >> decisions into no-brainers. As someone who on occasion poke at OpenSSL compat code I would very much like a more structured approach around dealing with dependencies. > Thus, I think it's worthwhile to spend effort on back-patching > new-LLVM compatibility fixes into old PG branches, but I agree > that newer PG branches can drop compatibility with obsolete > LLVM versions. +1 > LLVM is maybe not the poster child for these concerns -- for > either direction of compatibility problems, someone could build > without JIT support and not really be dead in the water. Right, OpenSSL on the other hand might be better example since removing TLS support is likely a no-show. I can see both the need to use an old OpenSSL version in a backbranch due to certifications etc, as well as a requirement in other cases to use the latest version due to CVE's. > In any case, I agree with your prior decision to not touch v11 > for this. With that branch's next release being its last, > I think the odds of introducing a bug we can't fix later > outweigh any arguable portability gain. Agreed. -- Daniel Gustafsson
Re: Guiding principle for dropping LLVM versions?
Thomas Munro writes: > I wonder if there is a good way to make this sort of thing more > systematic. If we could agree on a guiding principle vaguely like the > above, then perhaps we just need a wiki page that lists relevant > distributions, versions and EOL dates, that could be used to reduce > the combinations of stuff we have to consider and make the pruning > decisions into no-brainers. FWIW, I think "compile older Postgres on newer infrastructure" is a more common and more defensible scenario than "compile newer Postgres on older infrastructure". We've spent a ton of effort on the latter scenario (and I've helped lead the charge in many cases), but I think the real-world demand for it isn't truly that high once you get beyond a year or two back. On the other hand, if you have an app that depends on PG 11 behavioral details and you don't want to update it right now, you might nonetheless need to put that server onto recent infrastructure for practical reasons. Thus, I think it's worthwhile to spend effort on back-patching new-LLVM compatibility fixes into old PG branches, but I agree that newer PG branches can drop compatibility with obsolete LLVM versions. LLVM is maybe not the poster child for these concerns -- for either direction of compatibility problems, someone could build without JIT support and not really be dead in the water. In any case, I agree with your prior decision to not touch v11 for this. With that branch's next release being its last, I think the odds of introducing a bug we can't fix later outweigh any arguable portability gain. regards, tom lane
Re: Guiding principle for dropping LLVM versions?
Hi, On Thu, 2023-09-21 at 10:54 +1200, Thomas Munro wrote: > I'm trying to understand the practical constraints. Perhaps a package > maintainer could correct me if I have this wrong. Distros typically > support a range of releases from the past few years, and then bless > one as 'default' by making it the one you get if you install a meta > package eg 'llvm' without a number (for example, on Debian 12 this is > LLVM 14, though LLVM 13 is still available). Having a default > encourages sharing, eg one LLVM library can be used by many different > things. The maintainer of the PostgreSQL package then chooses which > one to link against, and it's usually the default one unless we can't > use that one yet for technical reasons (a situation that might arise > from time to time in bleeding edge distros). So if we just knew the > *oldest default* on every live distro at release time, I assume no > package maintainer would get upset if we ripped out support for > everything older, and that'd let us vacuum a lot of old versions out > of our tree. RPM packager speaking: Even though older LLVM versions exist on both RHEL and Fedora, they don't provide older Clang packages, which means we have to link to the latest release anyway (like currently Fedora 38 packages are waiting for LLVM 16 patch, as they cannot be linked against LLVM 15) Regards, Regards, -- Devrim Gündüz Open Source Solution Architect, PostgreSQL Major Contributor Twitter: @DevrimGunduz , @DevrimGunduzTR
Guiding principle for dropping LLVM versions?
Hi, Currently we claim to support all versions of LLVM from 3.9 up. It's now getting quite inconvenient to test changes on older releases with single digit major versions, because they aren't available through usual package channels on current distributions, and frankly it feels like pointless busy-work to build those older versions from source (not to mention that it takes huuurrs to compile that much C++). At the other end of the window, we've also been back-patching support for the latest LLVM versions into all supported releases, which might make slightly more sense, but I don't know. For the trailing end of the window, would it make sense to say that when PostgreSQL 17 ships, it doesn't need to support any LLVM versions that are no longer available in the default package repositories of current major distros? I'm trying to understand the practical constraints. Perhaps a package maintainer could correct me if I have this wrong. Distros typically support a range of releases from the past few years, and then bless one as 'default' by making it the one you get if you install a meta package eg 'llvm' without a number (for example, on Debian 12 this is LLVM 14, though LLVM 13 is still available). Having a default encourages sharing, eg one LLVM library can be used by many different things. The maintainer of the PostgreSQL package then chooses which one to link against, and it's usually the default one unless we can't use that one yet for technical reasons (a situation that might arise from time to time in bleeding edge distros). So if we just knew the *oldest default* on every live distro at release time, I assume no package maintainer would get upset if we ripped out support for everything older, and that'd let us vacuum a lot of old versions out of our tree. A more conservative horizon would be: which is the *oldest* LLVM you can still get through the usual channels on every relevant distro, for the benefit of people compiling from source, who for some reason want to use a version older then the default on their distro? I don't know what the motivation would be. What reason could there be to be more conservative than that? I wonder if there is a good way to make this sort of thing more systematic. If we could agree on a guiding principle vaguely like the above, then perhaps we just need a wiki page that lists relevant distributions, versions and EOL dates, that could be used to reduce the combinations of stuff we have to consider and make the pruning decisions into no-brainers.
