Re: Idea + question regarding the build targets
On 6/15/2019 2:54 AM, Willy Tarreau wrote: Actually maybe we should have some super-options separate from the target to decide what feature set to enable. Instead of having just TARGET being mandatory, we could have both TARGET and OPTIONS for example. Then one could just build like this : make TARGET=linux-glibc OPTIONS=all or : make TARGET=freebsd OPTIONS=ssl,zlib,pcre or : make TARGET=cygwin OPTIONS=ssl Sounds pretty good to me. I do have an idea there, described below. At this point we'll need to pursue this discussion for 2.1 I guess, and this will not prevent us from backporting some improvements to help users of 2.0. But let's not forget that novices should definitely use their distro's packages first. The discussion could turn into bikeshedding, but I bet the core team maintains enough authority that somebody can be the benevolent dictator. The only reason that I build haproxy from source is so that I can run the newest release in whichever X.Y version I'm going for -- distros do lag a bit in that regard, otherwise I would use the distro package. I do use distro packages for most software. Ubuntu doesn't lag as far behind as some of the other distros, and Ubuntu is my preferred flavor right now. At a previous job, I also compiled my own openssl, with static linking, because the load balancer machines were running CentOS 6 and the distro's openssl was ANCIENT. I'm currently using 1.8, because the bugs I've read about in 1.9 scare me a bit and I'm waiting for it to stabilize. I think I might switch to 1.9 after another release or two ... my interpretation of the grapevine says that the latest releases have worked out most of the problems. At the moment, here's the only shed colors I have thought of beyond the ideas I've read so far: * Have OPTIONS=default be possible. MAYBE have that be what the build does if the user doesn't include OPTIONS. With clear documentation in the README about what the default options are, I think an implicit OPTIONS=default could work well. * Have the build print out what options have been enabled, pause for some reasonably short time period so the user can see it, and then proceed. 5 seconds, maybe up to 10. If there's a sane cross-platform way to do it, provide an option for the user to press some key that skips the pause. The check I mentioned could happen right before that pause point, and if problems are detected, print an error and exit. You'll need to figure out whether or not to add "force" to the options, to go ahead and try a build even when the check detects a problem. I wish you and the community the best of luck in finding the precise path that's right for haproxy. Thanks, Shawn
Re: Idea + question regarding the build targets
On Sat, Jun 15, 2019 at 05:29:49PM +0200, Tim Düsterhus wrote: > Willy, > William, > > Am 14.06.19 um 22:41 schrieb Willy Tarreau: > > Maybe we could have a "recommended" variant for each of these based on what > > people "usually" enable in such environments. > > > > I've explained in Message-ID > 67c868d5-32bc-1a98-658e-486676099...@bastelstu.be why I consider this a > bad idea / useless effort. OK at least this means that this discussion deserves to be pursued beyond the release. > Regarding patches: They look good to me. Thanks! I'm currently applying minor cleanups and am going to merge them. Willy
Re: Idea + question regarding the build targets
Willy, William, Am 14.06.19 um 22:41 schrieb Willy Tarreau: > Maybe we could have a "recommended" variant for each of these based on what > people "usually" enable in such environments. > I've explained in Message-ID 67c868d5-32bc-1a98-658e-486676099...@bastelstu.be why I consider this a bad idea / useless effort. Regarding patches: They look good to me. Best regards Tim Düsterhus
Re: Idea + question regarding the build targets
Hi Shawn, On Fri, Jun 14, 2019 at 09:03:33PM -0600, Shawn Heisey wrote: > What I've noticed is that for the most part, you can classify source code > downloaders in one of two camps: Either the complete novice, or the > experienced user. The complete novice wants steps that are very simple, and > the experienced user wants the build to be customizable with myriad knobs > and switches, and wants them all documented. I find that membership in an > "intermediate" camp is very small ... people tend to either remain novices > or gravitate towards expert. I totally agree. > Dividing the README into two parts, one for the novice and one for the > expert, would be reasonable to me. This is a good proposal. I think it's what I tried to do recently but maybe it's not enough. It starts with a "quick build & install" section proposing a command to type on most common platforms. > The haproxy build is one of the more complex that I've encountered. This is interesting because it's not the feedback I got nor my experience either. I got many times some in person greetings like "thank you for not pissing us off with autoconf", which I personally totally share. But I agree that the doc on the build process is probably lacking a lot. And what is not easy is to add extra defines, CFLAGS or LDFLAGS because we tried to remain mostly compatible with command lines people have been using for a while, resulting in multiple ways to do this. > I've > written a little shell script to handle building and installing it, because > I'll never remember all the make options I need without checking the docs: > > - > root@smeagol:/usr/local/src# cat new-haproxy > #!/bin/sh > if [ -e "$1" ]; then > if [ -d "$1" ]; then > cd $1 > make clean && > make TARGET=linux2628 USE_PCRE_JIT=1 USE_OPENSSL=1 USE_ZLIB=1 > USE_SYSTEMD=1 CPU=native > RETVAL="$?" > if [ "$RETVAL" = 0 ]; then > make install > else > echo Build failed\! skipping install\! > fi > else > echo location \($1\) is not a directory. > fi > else > echo location \($1\) does not exist. > fi > - So the really useful part of it is : make clean && make TARGET=linux2628 USE_PCRE_JIT=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 CPU=native && make install Right ? Thus I guess the complex aspect you're seeing here is to remember the names of the USE_* options and the target. In a patchset I'm going to merge now, I've improved this a bit so that the "help" target lists these options and the ones enabled or disabled on your current target. > It would be really nice if a novice could simply type "make install" and > have it build with "sane" options for whatever OS they're on. That's about what William proposed with the "recommended" options. But if we do so we need to do it cross-systems, as Linux clearly is the most used one but not the only one. > Maybe having > one static command (or set of commands) for almost every user isn't > achievable without additional software that you don't want to use, such as > autoconf. The problem with autoconf precisely is that it does *not* set the options the user *wants* but instead will prevent the user from having more options than what autoconf *believes* his *current* system supports. That makes a lot of wrongs resulting in every user having to spend countless hours reading horrible configure scripts to figure the secret "ac_cv_*" variables that have to be forced by hand to make the script believe it previously checked and found a usable result, or patch the script. And while this could be acceptable for a lot of software that you're going to run locally, it makes no sense for something like haproxy that you are rarely going to build from sources to use on your desktop machine or laptop : if you're just a novice user or a web application developer, and have an instance on your machine to test your application, you just want to pick the last version shipped with your distro. Tricking autoconf scripts to force them to build for your servers' options from your desktop is another level of difficulty that's a real pain even for advanced users. > That said, some simplification would be welcome. Aliases so that > "TARGET=linux" produces a generally useful build on most current Linux > distros would be AWESOME. That was William's proposal initially and I'd tend to agree on this. I'd like to see openssl enable by default on most platforms for example. > The build should, as much as possible, check that > all options requested are possible, and fail fast if they're not, with > useful error messages. We could possibly create a "check" target in the makefile to validate if it thinks it will be able to build, and possibly to suggest *some* package names depending on the system. > General statement, not directed specifically at haproxy: I find it > frustrating to have a build get started, run for several minutes, Often if it takes several minutes it'
Re: Idea + question regarding the build targets
On 6/14/2019 7:01 AM, Willy Tarreau wrote:
OK. When discussing this with William, we figured it could be interesting
instead to have some aliases which are maybe more symbolic, such as :
- linux-complete : full set of supported features, will simply fail
if you don't have all libs installed, useful primarily for devs ;
- linux-recent : common set of features present by default on latest
release of LTS distros at the time of releasing. I.e. could be the
default if you have a reasonably up-to-date system ;
- linux-ancient : common set of features present by default on oldest
supported LTS distros at the time of releasing. Should be a safe
fallback for everyone who doesn't want to care more ;
- linux-minimal : basically just oldest supported LTS kernel + equivalent
libc, would serve as a starting point to manually add USE_*..
In fact I'm still a bit concerned by the fact that we continue to
advertise 2.6.28 as the split point while no older kernels are still
supported, and that some of the features placed there very likely still
don't work well in embedded environments (openwrt for example).
What I've noticed is that for the most part, you can classify source
code downloaders in one of two camps: Either the complete novice, or
the experienced user. The complete novice wants steps that are very
simple, and the experienced user wants the build to be customizable with
myriad knobs and switches, and wants them all documented. I find that
membership in an "intermediate" camp is very small ... people tend to
either remain novices or gravitate towards expert.
Dividing the README into two parts, one for the novice and one for the
expert, would be reasonable to me.
The haproxy build is one of the more complex that I've encountered.
I've written a little shell script to handle building and installing it,
because I'll never remember all the make options I need without checking
the docs:
-
root@smeagol:/usr/local/src# cat new-haproxy
#!/bin/sh
if [ -e "$1" ]; then
if [ -d "$1" ]; then
cd $1
make clean &&
make TARGET=linux2628 USE_PCRE_JIT=1 USE_OPENSSL=1 USE_ZLIB=1
USE_SYSTEMD=1 CPU=native
RETVAL="$?"
if [ "$RETVAL" = 0 ]; then
make install
else
echo Build failed\! skipping install\!
fi
else
echo location \($1\) is not a directory.
fi
else
echo location \($1\) does not exist.
fi
-
It would be really nice if a novice could simply type "make install" and
have it build with "sane" options for whatever OS they're on. Maybe
having one static command (or set of commands) for almost every user
isn't achievable without additional software that you don't want to use,
such as autoconf.
That said, some simplification would be welcome. Aliases so that
"TARGET=linux" produces a generally useful build on most current Linux
distros would be AWESOME. The build should, as much as possible, check
that all options requested are possible, and fail fast if they're not,
with useful error messages.
General statement, not directed specifically at haproxy: I find it
frustrating to have a build get started, run for several minutes, and
then suddenly fail with a compiler message that may or may not be easily
translatable into "you don't have the XXX development library
installed." I can usually figure out what those errors mean, but the
novice user probably won't have any clue.
A new "linux" target should probably internally choose the most common
numbered variant. At the moment, that's the latest (2628), but at some
point in the future it might not be the latest. It does seem reasonable
to have the "linux" alias *not* enable any other options, and have other
aliases that enable things like ZLIB, SSL, PCRE, etc.
I'd like very much to get rid of this laughing "2.6.28" now but if we
only use "linux" nobody will update it and we'll be back to the same
situation in one or two versions. With the split above we can have
fast moving targets ("complete", updated during dev; "recent", updated
with new distro announces) and slow moving ones ("ancient", "minimal")
and that might be a sweet spot.
I've wondered about that, actually. Occasionally I wonder whether 3.x
or 4.x kernels (and now 5.x) might offer something new that could
improve haproxy. Because the target name hasn't changed in such a long
time, I have to assume that there hasn't been anything earth-shattering,
and that the options for 2.6.28 really do offer the best possible
performance on recent kernels.
I personally have no use for older Linux targets, but I feel pretty sure
that there are still plenty of systems running on ancient kernels, with
admins that want to build haproxy.
Thanks,
Shawn
Re: Idea + question regarding the build targets
On Fri, Jun 14, 2019 at 10:37:09PM +0200, William Lallemand wrote: > On Fri, Jun 14, 2019 at 05:04:51PM +0200, Willy Tarreau wrote: > > Are you guys fine with these patches ? > > > > Thanks, > > willy > > Looks fine to me. Thanks! > However, in my opinion we should have in addition a target > which evolves more frequently and contains the latest features. (lua, openssl, > etc) > > Could be something called "linux-recommended". So after this discussion I looked again and saw that none of these are enabled either on other operating systems. Thus in all cases we currently only cover the kernel+libc combination. Maybe we could have a "recommended" variant for each of these based on what people "usually" enable in such environments. Willy
Re: Idea + question regarding the build targets
On Fri, Jun 14, 2019 at 05:04:51PM +0200, Willy Tarreau wrote: > Are you guys fine with these patches ? > > Thanks, > willy Looks fine to me. However, in my opinion we should have in addition a target which evolves more frequently and contains the latest features. (lua, openssl, etc) Could be something called "linux-recommended". -- William Lallemand
Re: Idea + question regarding the build targets
Are you guys fine with these patches ? Thanks, willy >From 573604dd22843805c0d8e47befc1453c2da80872 Mon Sep 17 00:00:00 2001 From: Willy Tarreau Date: Fri, 14 Jun 2019 16:32:09 +0200 Subject: BUILD: makefile: rename "linux2628" to "linux-glibc" and remove older targets The linux targets have become more than confusing over time. We used to have "linux2628" to match the features available in kernels 2.6.28 and above, without consideration for the libc, and due to many new features appearing later in kernels, some other options were added that are not enabled by default in linux2628, so this target doesn't make any sense anymore. The older ones (linux 2.2, linux 2.4, ...) do not make sense either since these versions are not supported anymore. Let's clean things up by creating a new "linux-glibc" target that matches what is available by default on Linux kernels and glibc present on supported distros at the time of release. Other libc implementation may use a custom or generic target or be added later if needed. All the older linux targets were removed. --- Makefile | 41 +++-- 1 file changed, 7 insertions(+), 34 deletions(-) diff --git a/Makefile b/Makefile index 0e123e937..3405db3c6 100644 --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ # # When in doubt, invoke help, possibly with a known target : # [g]make help -# [g]make help TARGET=linux +# [g]make help TARGET=linux-glibc # # By default the detailed commands are hidden for a cleaner output, but you may # see them by appending "V=1" to the make command. @@ -142,8 +142,8 @@ DOCDIR = $(PREFIX)/doc/haproxy TARGET system # Use TARGET= to optimize for a specifc target OS among the # following list (use the default "generic" if uncertain) : -#generic, linux22, linux24, linux24e, linux26, solaris, -#freebsd, openbsd, netbsd, cygwin, haiku, custom, aix51, aix52 +#linux-glibc, solaris, freebsd, openbsd, netbsd, cygwin, +#haiku, aix51, aix52, osx, generic, custom TARGET = TARGET CPU @@ -319,34 +319,8 @@ ifeq ($(TARGET),haiku) set_target_defaults = $(call default_opts,USE_POLL USE_TPROXY) endif -# Linux 2.2 -ifeq ($(TARGET),linux22) - set_target_defaults = $(call default_opts, \ -USE_POLL USE_TPROXY USE_LIBCRYPT USE_DL USE_RT) -endif - -# Standard Linux 2.4 with netfilter but without epoll() -ifeq ($(TARGET),linux24) - set_target_defaults = $(call default_opts, \ -USE_POLL USE_TPROXY USE_LIBCRYPT USE_DL USE_RT USE_CRYPT_H USE_NETFILTER) -endif - -# Enhanced Linux 2.4 with netfilter and epoll() patch > 0.21 -ifeq ($(TARGET),linux24e) - set_target_defaults = $(call default_opts, \ -USE_POLL USE_TPROXY USE_LIBCRYPT USE_DL USE_RT USE_CRYPT_H USE_NETFILTER \ -USE_EPOLL USE_MY_EPOLL) -endif - -# Standard Linux 2.6 with netfilter and standard epoll() -ifeq ($(TARGET),linux26) - set_target_defaults = $(call default_opts, \ -USE_POLL USE_TPROXY USE_LIBCRYPT USE_DL USE_RT USE_CRYPT_H USE_NETFILTER \ -USE_EPOLL USE_FUTEX USE_PRCTL) -endif - -# Standard Linux >= 2.6.28 with netfilter, epoll, tproxy and splice -ifeq ($(TARGET),linux2628) +# For linux >= 2.6.28 and glibc +ifeq ($(TARGET),linux-glibc) set_target_defaults = $(call default_opts, \ USE_POLL USE_TPROXY USE_LIBCRYPT USE_DL USE_RT USE_CRYPT_H USE_NETFILTER \ USE_CPU_AFFINITY USE_THREAD USE_EPOLL USE_FUTEX USE_LINUX_TPROXY \ @@ -759,9 +733,8 @@ all: @echo @echo "Please choose the target among the following supported list :" @echo - @echo " linux2628, linux26, linux24, linux24e, linux22, solaris," - @echo " freebsd, netbsd, osx, openbsd, aix51, aix52, cygwin, haiku," - @echo " generic, custom" + @echo " linux-glibc, solaris, freebsd, openbsd, netbsd, cygwin," + @echo " haiku, aix51, aix52, osx, generic, custom" @echo @echo "Use \"generic\" if you don't want any optimization, \"custom\" if you" @echo "want to precisely tweak every option, or choose the target which" -- 2.20.1 >From 46d2f00d8f54173519518cf3f6c19fe7042bbcfa Mon Sep 17 00:00:00 2001 From: Willy Tarreau Date: Fri, 14 Jun 2019 16:44:49 +0200 Subject: BUILD: makefile: detect and reject recently removed linux targets We've just removed old linux targets "linux22", "linux24", "linux24e", "linux26" and "linux2628" and it's likely that many build scripts and packages will still reference these. So let's have the makefile detect these and reject with instructions instead of silently building with incorrect options. --- Makefile | 8 1 file changed, 8 insertions(+) diff --git a/Makefile b/Makefile index 3405db3c6..299e707e3 100644 --- a/Makefile +++ b/Makefile @@ -742,6 +742,14 @@ all: @echo "out of it. Please check the Makefile in case of doubts." @echo @exit 1 +else ifneq ($(filter $(TARGET), linux22 linux24 linux24e linux26 linux2628),) +all: + @echo + @echo "Target '$(TARGET)' was re
Re: Idea + question regarding the build targets
On Fri, Jun 14, 2019 at 03:30:19PM +0200, Lukas Tribus wrote: > I agree with Tim. > > I don't think anyone still deploys "heavily patched 2.4 kernels" and > 2.6.28 is ancient itself, but a dependency. We can just call it Linux > at this point. > > This removes the strange "2628" number which may be confusing, without > introducing build issues by including additional external libraries. Thanks Lukas as well for this brain storming. Are you fine with my previous proposal of calling the target "linux-glibc" so that we do enable what glibc needs without pretending that everything is enforced by the kernel and creating trouble to other libcs ? thanks, Willy
Re: Idea + question regarding the build targets
On Fri, Jun 14, 2019 at 03:11:55PM +0200, Tim Düsterhus wrote: > These aliases still have the issue that the development headers are not > necessarily installed, even if the distro has the headers available. OK. > I'm > not sure whether there even is a need to change anything at all: > > - The uneducated user will simply use whatever their distro maintainer > (or third party repository maintainer) selects for them. They don't > touch `make` at all. > - The distro maintainer can be expected to be smart enough to take a > look into the Makefile and enable all the options that are available on > the distro. > - The power user most likely wants to customize his build to include > exactly what they need (e.g. disable Lua). > > None of these groups would really benefit from a `linux-recent` target! Indeed. > Instead I suggest to: > > 1. Add a `linux` target that is equivalent to `linux2628` (thus being > equivalent to the linux-minimal from your suggestion above). Then I'd rather call it "linux-glibc" in this case. I'm really having a problem with conflating the kernel and the libc in fact. I'm fine with considering they are both "recent enough" but seeing that the linux target becomes totally unusable with uclibc, musl, dietlibc or any possible alternative is a concern to me. And the USE_DL, USE_CRYPT, USE_RT, USE_THREAD, USE_CRYPT_H are exclusively libc stuff and not related to linux itself. > 2. Remove all the other linux targets that target kernels from a > bazillion years ago. If anyone actually has a need to run these AND also > run a bleeding edge HAProxy they can be expected to be smart enough to > use TARGET=generic and select all the other options. I definitely agree. I think I've been keeping them for a long time because I knew places where they were being used, and later just because it took more effort to remove them than to leave them. > If we really want to make it easy for users then we should add an > autoconf script Bah no thanks. You'll find another maintainer before we sink into this. I already hate its face beyond imagination as a user, I don't even want to meet its ass as a developer. I think it is the only tool ever created in computer history which has consumed more human time to work around than to write. > that automatically spits out a line with all the > supported options (and possibly `apt-get` commands to install the > remaining headers that are not supported). That's doable as suggestions for certain operating systems if needed, though history shows that such suggestions generally don't remain valid for a long time. In the mean time I've just added this to "make help" at the end : $ make help TARGET not set. Enabled features for TARGET '' (disable with 'USE_xxx=') : POLL Disabled features for TARGET '' (enable with 'USE_xxx=1') : EPOLL KQUEUE MY_EPOLL MY_SPLICE NETFILTER PCRE PCRE_JIT PCRE2 PCRE2_JIT PRIVATE_CACHE THREAD PTHREAD_PSHARED REGPARM STATIC_PCRE STATIC_PCRE2 TPROXY LINUX_TPROXY LINUX_SPLICE LIBCRYPT CRYPT_H VSYSCALL GETADDRINFO OPENSSL LUA FUTEX ACCEPT4 MY_ACCEPT4 ZLIB SLZ CPU_AFFINITY TFO NS DL RT DEVICEATLAS 51DEGREES WURFL SYSTEMD OBSOLETE_LINKER PRCTL THREAD_DUMP EVPORTS $ make help TARGET=linux2628 Current TARGET: linux2628 Enabled features for TARGET 'linux2628' (disable with 'USE_xxx=') : EPOLL NETFILTER POLL THREAD TPROXY LINUX_TPROXY LINUX_SPLICE LIBCRYPT CRYPT_H FUTEX ACCEPT4 CPU_AFFINITY DL RT PRCTL THREAD_DUMP Disabled features for TARGET 'linux2628' (enable with 'USE_xxx=1') : KQUEUE MY_EPOLL MY_SPLICE PCRE PCRE_JIT PCRE2 PCRE2_JIT PRIVATE_CACHE PTHREAD_PSHARED REGPARM STATIC_PCRE STATIC_PCRE2 VSYSCALL GETADDRINFO OPENSSL LUA MY_ACCEPT4 ZLIB SLZ TFO NS DEVICEATLAS 51DEGREES WURFL SYSTEMD OBSOLETE_LINKER EVPORTS $ make help TARGET=mylinux Current TARGET: mylinux (custom target) Enabled features for TARGET 'mylinux' (disable with 'USE_xxx=') : POLL Disabled features for TARGET 'mylinux' (enable with 'USE_xxx=1') : EPOLL KQUEUE MY_EPOLL MY_SPLICE NETFILTER PCRE PCRE_JIT PCRE2 PCRE2_JIT PRIVATE_CACHE THREAD PTHREAD_PSHARED REGPARM STATIC_PCRE STATIC_PCRE2 TPROXY LINUX_TPROXY LINUX_SPLICE LIBCRYPT CRYPT_H VSYSCALL GETADDRINFO OPENSSL LUA FUTEX ACCEPT4 MY_ACCEPT4 ZLIB SLZ CPU_AFFINITY TFO NS DL RT DEVICEATLAS 51DEGREES WURFL SYSTEMD OBSOLETE_LINKER PRCTL THREAD_DUMP EVPORTS It helped me immediately spot that I didn't enable DEVICEATLAS nor 51DEGREES yet in my default build scripts :-) Ideas and opinions still welcome of course! Willy
Re: Idea + question regarding the build targets
Hello, On Fri, 14 Jun 2019 at 15:12, Tim Düsterhus wrote: > > So does anyone have an opinion about the proposal above. Do not try > > to be gentle, "this is stupid" or "don't change anything at this point" > > are fine to me. I'd just want to be sure that whatever choice we make, > > it will have been given some though and will not just be the result of > > "oops we again forgot to change this". > > These aliases still have the issue that the development headers are not > necessarily installed, even if the distro has the headers available. I'm > not sure whether there even is a need to change anything at all: > > - The uneducated user will simply use whatever their distro maintainer > (or third party repository maintainer) selects for them. They don't > touch `make` at all. > - The distro maintainer can be expected to be smart enough to take a > look into the Makefile and enable all the options that are available on > the distro. > - The power user most likely wants to customize his build to include > exactly what they need (e.g. disable Lua). > > None of these groups would really benefit from a `linux-recent` target! > > Instead I suggest to: > > 1. Add a `linux` target that is equivalent to `linux2628` (thus being > equivalent to the linux-minimal from your suggestion above). > 2. Remove all the other linux targets that target kernels from a > bazillion years ago. If anyone actually has a need to run these AND also > run a bleeding edge HAProxy they can be expected to be smart enough to > use TARGET=generic and select all the other options. > > If we really want to make it easy for users then we should add an > autoconf script that automatically spits out a line with all the > supported options (and possibly `apt-get` commands to install the > remaining headers that are not supported). I agree with Tim. I don't think anyone still deploys "heavily patched 2.4 kernels" and 2.6.28 is ancient itself, but a dependency. We can just call it Linux at this point. This removes the strange "2628" number which may be confusing, without introducing build issues by including additional external libraries. Lukas
Re: Idea + question regarding the build targets
Willy,
Am 14.06.19 um 15:01 schrieb Willy Tarreau:
> Hi again,
>
> On Wed, Jun 12, 2019 at 07:17:29PM +0200, Willy Tarreau wrote:
>>> To be honest, I don't think this is very common in OSS projects; most
>>> of them use configure scripts that just include the library if the
>>> headers are detected, or not link against it at all if it isn't there.
>>> But we would brake the build here, which is different.
>>
>> OK. When discussing this with William, we figured it could be interesting
>> instead to have some aliases which are maybe more symbolic, such as :
>>
>> - linux-complete : full set of supported features, will simply fail
>> if you don't have all libs installed, useful primarily for devs ;
>>
>> - linux-recent : common set of features present by default on latest
>> release of LTS distros at the time of releasing. I.e. could be the
>> default if you have a reasonably up-to-date system ;
>>
>> - linux-ancient : common set of features present by default on oldest
>> supported LTS distros at the time of releasing. Should be a safe
>> fallback for everyone who doesn't want to care more ;
>>
>> - linux-minimal : basically just oldest supported LTS kernel + equivalent
>> libc, would serve as a starting point to manually add USE_*..
>>
>> In fact I'm still a bit concerned by the fact that we continue to
>> advertise 2.6.28 as the split point while no older kernels are still
>> supported, and that some of the features placed there very likely still
>> don't work well in embedded environments (openwrt for example).
>>
>> I'd like very much to get rid of this laughing "2.6.28" now but if we
>> only use "linux" nobody will update it and we'll be back to the same
>> situation in one or two versions. With the split above we can have
>> fast moving targets ("complete", updated during dev; "recent", updated
>> with new distro announces) and slow moving ones ("ancient", "minimal")
>> and that might be a sweet spot.
>
> So does anyone have an opinion about the proposal above. Do not try
> to be gentle, "this is stupid" or "don't change anything at this point"
> are fine to me. I'd just want to be sure that whatever choice we make,
> it will have been given some though and will not just be the result of
> "oops we again forgot to change this".
These aliases still have the issue that the development headers are not
necessarily installed, even if the distro has the headers available. I'm
not sure whether there even is a need to change anything at all:
- The uneducated user will simply use whatever their distro maintainer
(or third party repository maintainer) selects for them. They don't
touch `make` at all.
- The distro maintainer can be expected to be smart enough to take a
look into the Makefile and enable all the options that are available on
the distro.
- The power user most likely wants to customize his build to include
exactly what they need (e.g. disable Lua).
None of these groups would really benefit from a `linux-recent` target!
Instead I suggest to:
1. Add a `linux` target that is equivalent to `linux2628` (thus being
equivalent to the linux-minimal from your suggestion above).
2. Remove all the other linux targets that target kernels from a
bazillion years ago. If anyone actually has a need to run these AND also
run a bleeding edge HAProxy they can be expected to be smart enough to
use TARGET=generic and select all the other options.
If we really want to make it easy for users then we should add an
autoconf script that automatically spits out a line with all the
supported options (and possibly `apt-get` commands to install the
remaining headers that are not supported).
Best regards
Tim Düsterhus
Re: Idea + question regarding the build targets
Hi again,
On Wed, Jun 12, 2019 at 07:17:29PM +0200, Willy Tarreau wrote:
> > To be honest, I don't think this is very common in OSS projects; most
> > of them use configure scripts that just include the library if the
> > headers are detected, or not link against it at all if it isn't there.
> > But we would brake the build here, which is different.
>
> OK. When discussing this with William, we figured it could be interesting
> instead to have some aliases which are maybe more symbolic, such as :
>
> - linux-complete : full set of supported features, will simply fail
> if you don't have all libs installed, useful primarily for devs ;
>
> - linux-recent : common set of features present by default on latest
> release of LTS distros at the time of releasing. I.e. could be the
> default if you have a reasonably up-to-date system ;
>
> - linux-ancient : common set of features present by default on oldest
> supported LTS distros at the time of releasing. Should be a safe
> fallback for everyone who doesn't want to care more ;
>
> - linux-minimal : basically just oldest supported LTS kernel + equivalent
> libc, would serve as a starting point to manually add USE_*..
>
> In fact I'm still a bit concerned by the fact that we continue to
> advertise 2.6.28 as the split point while no older kernels are still
> supported, and that some of the features placed there very likely still
> don't work well in embedded environments (openwrt for example).
>
> I'd like very much to get rid of this laughing "2.6.28" now but if we
> only use "linux" nobody will update it and we'll be back to the same
> situation in one or two versions. With the split above we can have
> fast moving targets ("complete", updated during dev; "recent", updated
> with new distro announces) and slow moving ones ("ancient", "minimal")
> and that might be a sweet spot.
So does anyone have an opinion about the proposal above. Do not try
to be gentle, "this is stupid" or "don't change anything at this point"
are fine to me. I'd just want to be sure that whatever choice we make,
it will have been given some though and will not just be the result of
"oops we again forgot to change this".
Thanks!
Willy
Re: Idea + question regarding the build targets
as for popular distro, for example fedora, I'd spent some time on packaging
rpm in fedora copr (rather than telling people proper Makefile options).
if there's an interest in "official" package distro, I'd take part with
fedora copr and maybe few others
ср, 12 июн. 2019 г. в 10:41, Willy Tarreau :
> Hi all,
>
> I'm currently re-reading my notes for the upcoming release and found
> something I noted not that long ago regarding the TARGET variable of
> the makefile. The list of operating systems hasn't changed in a while
> and in parallel we've added a bunch of build options that do not solely
> depend on the kernel anymore, but more on features related to the whole
> environment (pcre, openssl, lua, systemd, zlib, and so on).
>
> I thought that now that the targets definition is very simple in the
> makefile, we could simply add distro names and versions and adjust the
> set of default USE_xxx variables based on this. We could thus have
> rhel{8,7,6,5}, debian{9,8,7,6}, ubuntu{18,16}, fedora{31,30,29} and so
> on, just like we already have for solaris/openbsd/freebsd for example,
> except that we generally only reference one version there. It would
> become easier to add new variants even to stable distros to ease the
> build for new users, who would just have to issue "make TARGET=debian9"
> for example, without having to figure if PCRE2 has to be specified or
> OPENSSL or any such thing.
>
> This this is fairly minor and basically just requires a few lines to
> be added to the makefile and to the doc, so I'm fine with including
> this to the final release.
>
> So my questions are :
> - does anybody think it's a bad idea ?
> - and among those who support the idea (if any), do you use
> particular options for a given distro (pcre2, lua version, systemd)
> - should we consider that using these pre-set distros enables more
> features by default given that packages are readily available ?
> (openssl seems obvious, zlib and pcre{1,2} probably, lua maybe)
>
> Note that it is also something we can add as backports after the release.
> It's just that if we want to teach people to start using a distro name
> over a kernel version, we'd rather propose a few entries first.
>
> Thanks for any hint,
> Willy
>
>
Re: Idea + question regarding the build targets
Hi,
Le mer. 12 juin 2019 à 19:19, Willy Tarreau a écrit :
> Hi guys,
>
> On Wed, Jun 12, 2019 at 04:27:42PM +0200, Lukas Tribus wrote:
> (...)
> > I think it's a bad idea.
> >
> > Basically what Tim says (I was interrupted several times while writing
> > this email).
>
> OK, and this morning William told me he thought the same for the same
> reasons, so definitely I'm the one wrong here, which justifies that I
> asked. And I totally agree with your arguments.
>
> (...)
> > Those will be the interactions, with a lot of round-trips back and
> > forth. At least the user understands that USE_LUA=1 means that LUA
> > will be included. With TARGET=ubuntu18 the user may not even know what
> > LUA is, let alone that it's a dependency.
>
> I was actually thinking about enabling what I expect to be installed
> by default, i.e. zlib & openssl mainly. But at first I didn't want to
> address libraries as much as the default OS which involves a minimal
> kernel version and a libc. In the past we've had to guess quite a few
> times some settings, which were nicely addressed by the split on kernel
> 2.6.28, but it seems cleaner to support a kernel+libc combination, which
> is what made me think about distros, hence their respective libs.
>
> > So do we brake the build by
> > default and just document in INSTALL that in Ubuntu 18.04 the LUA
> > dependency must be fulfilled by installing the source package, which
> > can be either liblua5.1-0-dev, liblua5.2-dev or liblua5.3-dev?
>
> I agree.
>
> > INSTALL already contains suggestions about what USE_ flags to include,
> > as a matter of fact all 5 external libraries mentioned above are
> > suggested:
> >
> >
> http://git.haproxy.org/?p=haproxy.git;a=blob;f=INSTALL;h=9df17caf68c4f00cdaaaec2ec929f0af66e1d297;hb=HEAD#l35
> >
> >
> > To be honest, I don't think this is very common in OSS projects; most
> > of them use configure scripts that just include the library if the
> > headers are detected, or not link against it at all if it isn't there.
> > But we would brake the build here, which is different.
>
> OK. When discussing this with William, we figured it could be interesting
> instead to have some aliases which are maybe more symbolic, such as :
>
> - linux-complete : full set of supported features, will simply fail
> if you don't have all libs installed, useful primarily for devs ;
>
> - linux-recent : common set of features present by default on latest
> release of LTS distros at the time of releasing. I.e. could be the
> default if you have a reasonably up-to-date system ;
>
> - linux-ancient : common set of features present by default on oldest
> supported LTS distros at the time of releasing. Should be a safe
> fallback for everyone who doesn't want to care more ;
>
> - linux-minimal : basically just oldest supported LTS kernel + equivalent
> libc, would serve as a starting point to manually add USE_*..
>
> In fact I'm still a bit concerned by the fact that we continue to
> advertise 2.6.28 as the split point while no older kernels are still
> supported, and that some of the features placed there very likely still
> don't work well in embedded environments (openwrt for example).
>
> I'd like very much to get rid of this laughing "2.6.28" now but if we
> only use "linux" nobody will update it and we'll be back to the same
> situation in one or two versions. With the split above we can have
> fast moving targets ("complete", updated during dev; "recent", updated
> with new distro announces) and slow moving ones ("ancient", "minimal")
> and that might be a sweet spot.
>
> > > just like we already have for solaris/openbsd/freebsd for example
> >
> > None of these include external libraries though, like OpenSSL, LUA,
> > zlib, PCRE1/2. Only kernel features and "very core" libraries are
> > included with the solaris/BSD targets (just like linux2628). So the
> > build may brake because a basic and crucial threading lib is missing,
> > but not because LUA is not there. That's very different kind of build
> > failure.
>
> Agreed, probably that I was a bit too enthousiast by this idea and
> conflated several things. We should probably get back to platform and
> features separately. openssl, lua, zlib, pcre are in fact features if
> they are not present by default. I was happy to place some of them by
> default but if they are not necessarily present, let's not force :-)
>
> thanks guys for fueling the discussion.
>
> Willy
>
Sorry if I'm totally "out" on this point, but I was wondering why HAProxy
did not provide a simple "configure" script : all available options would
be listed (I know it's in the doc, but hey, how many people do actually
read it ? :p). and script will check if options chosen can be compiled (if
pcre2 is available, if openssl-devel is available, and so on ...).
You may then pick a default config, for example SSL always compiled in, LUA
not, ... if no other option are provided.
Olivier
Re: Idea + question regarding the build targets
Hi guys,
On Wed, Jun 12, 2019 at 04:27:42PM +0200, Lukas Tribus wrote:
(...)
> I think it's a bad idea.
>
> Basically what Tim says (I was interrupted several times while writing
> this email).
OK, and this morning William told me he thought the same for the same
reasons, so definitely I'm the one wrong here, which justifies that I
asked. And I totally agree with your arguments.
(...)
> Those will be the interactions, with a lot of round-trips back and
> forth. At least the user understands that USE_LUA=1 means that LUA
> will be included. With TARGET=ubuntu18 the user may not even know what
> LUA is, let alone that it's a dependency.
I was actually thinking about enabling what I expect to be installed
by default, i.e. zlib & openssl mainly. But at first I didn't want to
address libraries as much as the default OS which involves a minimal
kernel version and a libc. In the past we've had to guess quite a few
times some settings, which were nicely addressed by the split on kernel
2.6.28, but it seems cleaner to support a kernel+libc combination, which
is what made me think about distros, hence their respective libs.
> So do we brake the build by
> default and just document in INSTALL that in Ubuntu 18.04 the LUA
> dependency must be fulfilled by installing the source package, which
> can be either liblua5.1-0-dev, liblua5.2-dev or liblua5.3-dev?
I agree.
> INSTALL already contains suggestions about what USE_ flags to include,
> as a matter of fact all 5 external libraries mentioned above are
> suggested:
>
> http://git.haproxy.org/?p=haproxy.git;a=blob;f=INSTALL;h=9df17caf68c4f00cdaaaec2ec929f0af66e1d297;hb=HEAD#l35
>
>
> To be honest, I don't think this is very common in OSS projects; most
> of them use configure scripts that just include the library if the
> headers are detected, or not link against it at all if it isn't there.
> But we would brake the build here, which is different.
OK. When discussing this with William, we figured it could be interesting
instead to have some aliases which are maybe more symbolic, such as :
- linux-complete : full set of supported features, will simply fail
if you don't have all libs installed, useful primarily for devs ;
- linux-recent : common set of features present by default on latest
release of LTS distros at the time of releasing. I.e. could be the
default if you have a reasonably up-to-date system ;
- linux-ancient : common set of features present by default on oldest
supported LTS distros at the time of releasing. Should be a safe
fallback for everyone who doesn't want to care more ;
- linux-minimal : basically just oldest supported LTS kernel + equivalent
libc, would serve as a starting point to manually add USE_*..
In fact I'm still a bit concerned by the fact that we continue to
advertise 2.6.28 as the split point while no older kernels are still
supported, and that some of the features placed there very likely still
don't work well in embedded environments (openwrt for example).
I'd like very much to get rid of this laughing "2.6.28" now but if we
only use "linux" nobody will update it and we'll be back to the same
situation in one or two versions. With the split above we can have
fast moving targets ("complete", updated during dev; "recent", updated
with new distro announces) and slow moving ones ("ancient", "minimal")
and that might be a sweet spot.
> > just like we already have for solaris/openbsd/freebsd for example
>
> None of these include external libraries though, like OpenSSL, LUA,
> zlib, PCRE1/2. Only kernel features and "very core" libraries are
> included with the solaris/BSD targets (just like linux2628). So the
> build may brake because a basic and crucial threading lib is missing,
> but not because LUA is not there. That's very different kind of build
> failure.
Agreed, probably that I was a bit too enthousiast by this idea and
conflated several things. We should probably get back to platform and
features separately. openssl, lua, zlib, pcre are in fact features if
they are not present by default. I was happy to place some of them by
default but if they are not necessarily present, let's not force :-)
thanks guys for fueling the discussion.
Willy
Re: Idea + question regarding the build targets
Hello Willy,
On Wed, 12 Jun 2019 at 07:39, Willy Tarreau wrote:
>
> Hi all,
>
> I'm currently re-reading my notes for the upcoming release and found
> something I noted not that long ago regarding the TARGET variable of
> the makefile. The list of operating systems hasn't changed in a while
> and in parallel we've added a bunch of build options that do not solely
> depend on the kernel anymore, but more on features related to the whole
> environment (pcre, openssl, lua, systemd, zlib, and so on).
>
> I thought that now that the targets definition is very simple in the
> makefile, we could simply add distro names and versions and adjust the
> set of default USE_xxx variables based on this. We could thus have
> rhel{8,7,6,5}, debian{9,8,7,6}, ubuntu{18,16}, fedora{31,30,29} and so
> on, just like we already have for solaris/openbsd/freebsd for example,
> except that we generally only reference one version there. It would
> become easier to add new variants even to stable distros to ease the
> build for new users, who would just have to issue "make TARGET=debian9"
> for example, without having to figure if PCRE2 has to be specified or
> OPENSSL or any such thing.
>
> This this is fairly minor and basically just requires a few lines to
> be added to the makefile and to the doc, so I'm fine with including
> this to the final release.
>
> So my questions are :
> - does anybody think it's a bad idea ?
I think it's a bad idea.
Basically what Tim says (I was interrupted several times while writing
this email).
For the build not to fail, all those libraries *and the source package
of it* have to be installed first. Even if (I imagine) some of those
libraries in some of the OS configurations are installed by default,
the source packages with the header files are still missing and the
build would fail for every single dependency by default, which are at
least 5 (PCRE and variants, OpenSSL, LUA, ZLIB, SYSTEMD).
By not requiring the user to think about the libraries, we are hiding
complexity, which will then come up on the mailing list, discourse and
github issues.
"my build fails with a libssl.h error" --> install libssl-dev
"now the build still fails, but with a libz.h error" --> install libz-dev
"now the build still fails, but with a systemd.h error" --> install
systemd-dev libraries
Those will be the interactions, with a lot of round-trips back and
forth. At least the user understands that USE_LUA=1 means that LUA
will be included. With TARGET=ubuntu18 the user may not even know what
LUA is, let alone that it's a dependency. So do we brake the build by
default and just document in INSTALL that in Ubuntu 18.04 the LUA
dependency must be fulfilled by installing the source package, which
can be either liblua5.1-0-dev, liblua5.2-dev or liblua5.3-dev?
INSTALL already contains suggestions about what USE_ flags to include,
as a matter of fact all 5 external libraries mentioned above are
suggested:
http://git.haproxy.org/?p=haproxy.git;a=blob;f=INSTALL;h=9df17caf68c4f00cdaaaec2ec929f0af66e1d297;hb=HEAD#l35
To be honest, I don't think this is very common in OSS projects; most
of them use configure scripts that just include the library if the
headers are detected, or not link against it at all if it isn't there.
But we would brake the build here, which is different.
> just like we already have for solaris/openbsd/freebsd for example
None of these include external libraries though, like OpenSSL, LUA,
zlib, PCRE1/2. Only kernel features and "very core" libraries are
included with the solaris/BSD targets (just like linux2628). So the
build may brake because a basic and crucial threading lib is missing,
but not because LUA is not there. That's very different kind of build
failure.
cheers,
lukas
Re: Idea + question regarding the build targets
Willy, Am 12.06.19 um 07:38 schrieb Willy Tarreau: > So my questions are : > - does anybody think it's a bad idea ? I do. Even if the Linux distribution in question theoretically supports a certain feature you are: 1. Not guaranteed that the development headers are installed. As a specific example: Debian ships systemd (and by extension libsystemd) by default. But I would not be able to compile HAProxy with USE_SYSTEMD without installing libsystemd-dev first. 2. Not guaranteed that the user is interested in certain features. I guess most are interested in OpenSSL support, but Lua might be more questionable. Why should I carry the Lua interpreter along when I'm not using Lua scripts anyway? Personally I would leave Lua out on most of my servers if I would compile my own HAProxy [1]. This is because I realized, by doing stupid but fun things with Lua, that Lua support is way less reliable than HAProxy's C code. Possibly because most people don't need Lua and thus it gets less exposure in production. (1) could be solved with feature detection (e.g. autoconf, but you probably don't want to touch that). (2) not so much. Both could be solved with some kind of wizard that asks the user what they want and detects the appropriate options and flags based on the system. I believe something similar exists for the Linux kernel (I never compiled my own kernel). Best regards Tim Düsterhus [1] Big if there. I'm super happy with the job the Debian HAProxy team is doing by providing a separate APT repository for the newest versions.
Idea + question regarding the build targets
Hi all,
I'm currently re-reading my notes for the upcoming release and found
something I noted not that long ago regarding the TARGET variable of
the makefile. The list of operating systems hasn't changed in a while
and in parallel we've added a bunch of build options that do not solely
depend on the kernel anymore, but more on features related to the whole
environment (pcre, openssl, lua, systemd, zlib, and so on).
I thought that now that the targets definition is very simple in the
makefile, we could simply add distro names and versions and adjust the
set of default USE_xxx variables based on this. We could thus have
rhel{8,7,6,5}, debian{9,8,7,6}, ubuntu{18,16}, fedora{31,30,29} and so
on, just like we already have for solaris/openbsd/freebsd for example,
except that we generally only reference one version there. It would
become easier to add new variants even to stable distros to ease the
build for new users, who would just have to issue "make TARGET=debian9"
for example, without having to figure if PCRE2 has to be specified or
OPENSSL or any such thing.
This this is fairly minor and basically just requires a few lines to
be added to the makefile and to the doc, so I'm fine with including
this to the final release.
So my questions are :
- does anybody think it's a bad idea ?
- and among those who support the idea (if any), do you use
particular options for a given distro (pcre2, lua version, systemd)
- should we consider that using these pre-set distros enables more
features by default given that packages are readily available ?
(openssl seems obvious, zlib and pcre{1,2} probably, lua maybe)
Note that it is also something we can add as backports after the release.
It's just that if we want to teach people to start using a distro name
over a kernel version, we'd rather propose a few entries first.
Thanks for any hint,
Willy
