Re: Linking coreutils against OpenSSL
Hi! On Thu, 2023-11-09 at 17:38:05 -0500, Benjamin Barenblat wrote: > coreutils can link against OpenSSL, yielding a substantial speed boost > in sha256sum etc. For many years, this was inadvisable due to license > conflicts. However, as of bookworm, coreutils requires GPL-3+ and > OpenSSL is Apache-2.0, so I believe all license compatibility questions > have been resolved. > > What would you think about having coreutils Depend on libssl3? This > would make the libssl3 package essential, which is potentially > undesirable, but it also has the potential for serious user time savings > (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than > coreutils’ internal implementation). This increases the pseudo-essential-set by around 6 MiB, to increase performance for the digest tools, which do not seem like something that gets run too often? So the trade-off here does not seem worth it to me TBH. The other thing is that the OpenSSL support seems to be coming from gnulib, which is using the deprecated low-level digest functions from OpenSSL (supposedly to be removed in the future), and where the supported high-level ones have the problem (AFAIUI) of being affected by the OpenSSL policies (such as FIPS or other config or similar) which could then render those commands unusable once and if they switch API. This is one of the reasons for why I switched dpkg from its embedded digest functions to use libmd (currently pseudo-essential) instead one of the other crypto libs. Of course libmd does not currently contain optimized functions, but I'd be happy to take patches for that, or might eventually look into adding support to it to use the Linux Kernel Crypto API or similar. Thanks, Guillem
Re: Linking coreutils against OpenSSL
On 2023-11-11 09:32, Julian Andres Klode wrote: > While libraries are dependencies of Essential packages, they > themselves are distinctively not Essential, they are pseudo-essential. Fair enough, but still the general point of being very careful about what we make (pseudo-)essential is valid and important I think. We should only make the set bigger if the benefits are absolutely clear, and I argue that improving the performance of /usr/bin/sha256sum is not a good example.
Re: Linking coreutils against OpenSSL
Luca Boccassi wrote on 10/11/2023 at 16:08:30+0100:
> On Fri, 10 Nov 2023 at 14:22, Pierre-Elliott Bécue wrote:
>>
>> Luca Boccassi wrote on 10/11/2023 at 15:00:24+0100:
>>
>> > On Fri, 10 Nov 2023 at 13:45, Bjørn Mork wrote:
>> >>
>> >> Luca Boccassi writes:
>> >>
>> >> > If we want to start nitpicking, then let's be exact: 0.61% of popcon.
>> >> > Whether that qualifies as "plenty" we'll leave as an exercise for the
>> >> > readers.
>> >>
>> >> I wonder if this sort of arrogant "don't care about minorities" attitude
>> >> will ever stop? Was sort of hoping that things would quiet down when
>> >> everyone just gave up on the systemd and usr-merge crazyness. But it
>> >> seems the bullies continue on and on and on and on.
>> >>
>> >> I guess those 0.61% are run bt the most valuable Debian users out there.
>> >> I'm sorry to not be one of them, but that's the way things have become.
>> >> Not by choice.
>> >
>> > Having a 5x performance improvement in core utilities at zero cost for
>> > 99.39% of users and at the cost of an extra library that is already
>> > widely used for 0.61% --> bullying. Noted.
>>
>> The problem is not the argument but clearly the way you state it, which
>> is quite coherent with your way of answering most of the time.
>>
>> I know it's not mandatory, but a bit of empathy, a bit of caring and a
>> bit of "just because I can act bluntly with a pinch of contempt doesn't
>> mean I should" would be really appreciable.
>
> You know what else would be really appreciable? Not starting the
> usual, tired, old flamewar, with heavily emotionally charged language
> ("bullying", "contempt", "lack of empathy", "lack of caring", etc),
> when talking about something as trivial and harmless as installing a
> widely used and common library to get a 5x performance boost out of
> the core system. How about we do that instead? Because, just in case
> perspective was already lost, this is what started the tirade:
>
>> This sounds great. systemd also uses OpenSSL for various things, so
>> libssl3 is pretty much a given on any bootable installation anyway
>> already.
>
> IE: a supportive and appreciative comment, followed by a relevant and
> objective piece of information. Where's the bullying?
I'd like to extend an apology to Luca, my mail was intended to be
private, I used the wrong shortcut to reply. I'll therefore reply to
this second mail privately as it was first intended.
Regards,
--
PEB
signature.asc
Description: PGP signature
Re: Linking coreutils against OpenSSL
* Theodore Ts'o: > If you can get upstream a patch so that coreutils could try to dlopen > OpenSSL and use it if it is available, but skip it if it is not, that > might be one way to avoid OpenSSL going into essential. The challenge > is that OpenSSL is not known for its ability to maintain a stable ABI, > but if we only care about supporting a specific version of OpenSSL > (the one which is shipped with coreutils) and given that the fallback > is a slower sha256sum, which IMHO is *not* a disaster, perhaps it's > doable? I think the OpenSSL 3 ABI should be stable for quite some time. It's probably not even necessary to use dlvsym instead of dlsym because I think OpenSSL upstream just adds new functions if they need new behaviors.
Re: Linking coreutils against OpenSSL
On Sat, Nov 11, 2023 at 09:32:46AM +0200, Julian Andres Klode wrote: > > WRT dlopen(), this is never an appealing solution because you do not > get any type-checking, you have to make sourceful changes for an soname > bump. It is an interface you can use for loading plugins (that is, you > should be in control of what the interface is that you are loading from > the library object), but it's not really usable for stuff that is outside > of your control. There are some caveats, yes, but I *have* made it work. For example, see [1]. This allows debugfs in e2fsprogs to use GNU readline (or BSD's libedit), which is *super* convenient, without requiring forcing GNU readline to be placed in emergency boot floppies or being added to essential. [1] https://github.com/tytso/e2fsprogs/blob/master/lib/ss/get_readline.c For things which are optional, if there is a soname bump, things won't stop working. In the case of coreutils and OpenSSL, sha1sum will just git a bit slower --- in fact, the speeds that it has today, which has never been a problem for me, FWIW. And yes, to add accomodate a soname bump you might need to make some sourceful changes, but it's often not that hard to do. At least for the libreadline functionality used by debugfs, it's mainly doing a bit of testing, concluding "nope, the ABI didn't change for the functions that libss cares about", and just adding the additional SONAME to: #define DEFAULT_LIBPATH "libreadline.so.8:libreadline.so.7:libreadline.so.6:libreadline.so.5:libreadline.so.4:libreadline.so:libedit.so.2:libedit.so:libeditline.so.0:libeditline.so" :-) As another example, I have a pending patch to e2fsprogs that I will shortly integrating which allows mke2fs to optionally use libarchive, so that mke2fs -d can take a tarball, but again, without expanding the dependencies for e2fsprogs. It's not *that* bad. - Ted
Re: Linking coreutils against OpenSSL
On 2023-11-11 Michael Stone wrote:
> On Sat, Nov 11, 2023 at 11:50:31AM +0100, Andreas Metzler wrote:
> > you seem to have missed/deleted the paragraph where Ansgar suggested how
> > to do this *without* tradeoff. ("explicitly disable/enable build options
> > per arch")
> No, I didn't. That was in my original email and is one of the possibilities
> for future versions depending on the feedback from people testing to guide
> whether it makes sense to make this per-arch rather than global.
Hm. Would you mind explaining why you chose to not implement this but
instead only used a per-arch build-dep? (With the downside of wrong
builds on dirty chroots and slent breakage when a respective feature
is not found)?
cu Andreas
--
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'
Re: Linking coreutils against OpenSSL
On Sat, Nov 11, 2023 at 11:50:31AM +0100, Andreas Metzler wrote:
you seem to have missed/deleted the paragraph where Ansgar suggested how
to do this *without* tradeoff. ("explicitly disable/enable build options
per arch")
No, I didn't. That was in my original email and is one of the
possibilities for future versions depending on the feedback from people
testing to guide whether it makes sense to make this per-arch rather
than global.
Re: Linking coreutils against OpenSSL
On 2023-11-10 Michael Stone wrote:
> On Fri, Nov 10, 2023 at 03:10:42PM +0100, Ansgar wrote:
>> Please avoid producing different results depending on the build
>> environment. That just results in non-reproducible issues in unclean
>> environments (suddenly different dependencies, different features,
>> ...).
> I think that is an acceptable tradeoff at this time; the only difference
> will be the dependencies, but that is the intent. Automated buildd packages
> should be stable. Based on further experience and feedback, one of the other
> options could be chosen instead. (I'm particularly interested in hearing
> from people who compare the different builds on arm, as that is where
> there's been an assertion of a performance regression.)
Hello,
you seem to have missed/deleted the paragraph where Ansgar suggested how
to do this *without* tradeoff. ("explicitly disable/enable build options
per arch")
cu Andreas
--
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'
diff -Nru coreutils-9.4/debian/rules coreutils-9.4/debian/rules
--- coreutils-9.4/debian/rules 2023-11-10 14:31:21.0 +0100
+++ coreutils-9.4/debian/rules 2023-11-11 11:37:05.0 +0100
@@ -13,13 +13,20 @@
DEB_CFLAGS_MAINT_APPEND += -mieee
endif
+ifeq ($(DEB_TARGET_MULTIARCH),x86_64-linux-gnu)
+opensslconf = --with-openssl=auto-gpl-compat
+else
+opensslconf = --with-openssl=no
+endif
+
BIN_PROGS = cat chgrp chmod chown cp date dd df dir echo false ln ls mkdir \
mknod mv pwd readlink rm rmdir vdir sleep stty sync touch true uname \
mktemp
d=debian/coreutils
override_dh_auto_configure:
- dh_auto_configure -- --enable-install-program=arch --with-openssl=auto-gpl-compat
+ dh_auto_configure -- --enable-install-program=arch \
+ $(opensslconf)
%:
dh $@ --with autoreconf
Re: Linking coreutils against OpenSSL
On Sat, Nov 11, 2023 at 06:52:50AM +0100, Emanuele Rocca wrote: > Hi, > > On 2023-11-09 07:31, Theodore Ts'o wrote: > > If you can get upstream a patch so that coreutils could try to dlopen > > OpenSSL and use it if it is available, but skip it if it is not, that > > might be one way to avoid OpenSSL going into essential. The challenge > > is that OpenSSL is not known for its ability to maintain a stable ABI, > > but if we only care about supporting a specific version of OpenSSL > > (the one which is shipped with coreutils) and given that the fallback > > is a slower sha256sum, which IMHO is *not* a disaster, perhaps it's > > doable? > > This idea seems appealing to me. > WRT dlopen(), this is never an appealing solution because you do not get any type-checking, you have to make sourceful changes for an soname bump. It is an interface you can use for loading plugins (that is, you should be in control of what the interface is that you are loading from the library object), but it's not really usable for stuff that is outside of your control. > In terms of costs vs benefits, adding OpenSSL to essential has a fairly > high cost for unclear benefits. > > Costs, from Policy 3.8 [1]: > > "Maintainers should take great care in adding any programs, interfaces, > or functionality to essential packages. Packages may assume that > functionality provided by essential packages is always available without > declaring explicit dependencies, which means that removing functionality > from the Essential set is very difficult and is almost never done. Any > capability added to an essential package therefore creates an obligation > to support that capability as part of the Essential set in perpetuity." > > Can we quantify the benefits? Not in terms of artificial benchmarks, but > real use cases if possible. > > [1] https://www.debian.org/doc/debian-policy/ch-binary.html#essential-packages I do not think this is a reasonable argument to make. While libraries are dependencies of Essential packages, they themselves are distinctively not Essential, they are pseudo-essential. The rules here don't apply to pseudo-essential library packages because quite frankly, they would have to apply to the specific soname because that is the provided interface. But since libraries frequently change their soname, they can't be considered subject to essential clauses because every soname bump, you'd be removing a package from the essential set and adding a new one to it. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
Re: Linking coreutils against OpenSSL
Hi, On 2023-11-09 07:31, Theodore Ts'o wrote: > If you can get upstream a patch so that coreutils could try to dlopen > OpenSSL and use it if it is available, but skip it if it is not, that > might be one way to avoid OpenSSL going into essential. The challenge > is that OpenSSL is not known for its ability to maintain a stable ABI, > but if we only care about supporting a specific version of OpenSSL > (the one which is shipped with coreutils) and given that the fallback > is a slower sha256sum, which IMHO is *not* a disaster, perhaps it's > doable? This idea seems appealing to me. In terms of costs vs benefits, adding OpenSSL to essential has a fairly high cost for unclear benefits. Costs, from Policy 3.8 [1]: "Maintainers should take great care in adding any programs, interfaces, or functionality to essential packages. Packages may assume that functionality provided by essential packages is always available without declaring explicit dependencies, which means that removing functionality from the Essential set is very difficult and is almost never done. Any capability added to an essential package therefore creates an obligation to support that capability as part of the Essential set in perpetuity." Can we quantify the benefits? Not in terms of artificial benchmarks, but real use cases if possible. [1] https://www.debian.org/doc/debian-policy/ch-binary.html#essential-packages
Re: Linking coreutils against OpenSSL
On Thu, Nov 09, 2023 at 11:13:51PM +, Luca Boccassi wrote: > > Alternatively, what would you think about making sha256sum etc. > > divertible and providing implementations both with and without the > > OpenSSL dependency? > > Please, no, no more diversion/alternatives/shenanigans, it's just huge > and convoluted complications for no real gain. Agreed, let's not. If you can get upstream a patch so that coreutils could try to dlopen OpenSSL and use it if it is available, but skip it if it is not, that might be one way to avoid OpenSSL going into essential. The challenge is that OpenSSL is not known for its ability to maintain a stable ABI, but if we only care about supporting a specific version of OpenSSL (the one which is shipped with coreutils) and given that the fallback is a slower sha256sum, which IMHO is *not* a disaster, perhaps it's doable? - Ted
Re: Linking coreutils against OpenSSL
https://lists.debian.org/debian-arm/2021/06/msg1.html is the start of a thread about support for encryption in HW, which AFAIK uses the Crypto API. That's available on a number of arm64 devices. It may be useful to specify/detail a test so that people can (uniformly) test the performance on their devices (on various architectures). I did an OpenSSH/OpenSSL test, but I actually don't know if those were good tests. Still far from being an expert, but I was recently involved with testing a Crypto implementation for rk3566 (based) devices and noticed the results were rather mixed. It was much slower with 16 bytes, but significantly faster with 16384 bytes and the 'turnover' point was around 1024 bytes. In the discussion that followed, the point was made that HW-based Crypto would be much energy efficient and it would free up the CPU to do other things. So that could also be a factor that could be considered. Personally I don't have a problem with linking to OpenSSL, but I'm not a subject matter expert. HTH PS: The 'argument' wrt a certain package of Priority: important (not Essential) makes my blood boil, so this will likely be my only contribution to this thread. signature.asc Description: This is a digitally signed message part.
Re: Linking coreutils against OpenSSL
Hi, On 11/11/23 03:34, Julian Andres Klode wrote: 1) we use libgcrypt in libapt-pkg, which needs global initialization. Libraries should not be doing the initialization, we're basically misusing it. I remember that KiCad had problems with OpenSSL for this exact reason -- we were using libcurl from multiple threads from different plugins, and OpenSSL required us to initialize some locking mechanism from the KiCad main program so plugins could operate safely, breaking the abstraction. For this reason, KiCad had a direct dependency on OpenSSL for years, even if libcurl was provided by libcurl-gnutls -- can't know whether libcurl pulls in openssl, so we need to initialize it. We removed that code a while ago after the offending plugins were removed, and it seems the OpenSSL API has removed the CRYPTO_set_locking_functions call since then so there is a good chance that they've cleaned up a bit. There are still some globals, but mostly for malloc replacements, so I guess these can be ignored. Simon
Re: Linking coreutils against OpenSSL
On Sat, Nov 11, 2023 at 12:55:16AM +0900, Simon Richter wrote: > Hi, > > On 11/10/23 21:07, Stephan Verbücheln wrote: > > > In my opinion, this is yet another reason to use a proper cryptography > > library (openssl, gnutls or gcrypt) instead of a custom implementation > > for this kind of algorithm. > > Yes and no. The reason several of our core tools bring their own functions > is to specifically reduce the Essential package set. Hence this thread: we > need to weigh the benefits against the drawbacks here. > > In coreutils' case, I think the benefits are kind of qualified by the number > of direct users. Basically, few people have use cases that require them to > routinely verify checksums with the tools from coreutils[1]. > > The main benefit of this move is that container images will shrink because > libssl becomes part of the base layer, so fewer copies of it will be kept in > stacked layers. I would disregard this as a benefit, otherwise we could make > a case that more packages should be Essential. > > The actual drawbacks for users are minimal too: > - systemd pulls it in anyway > - apt will pull it in on the remaining systems > > I don't quite see the appeal of OpenSSL as a dependency for apt either. I > have 2 Gbps Internet at home, and a laptop I bought in 2012, and apt is > never CPU bound. I could see the benefit of gzip offloading into the kernel > crypto engine, that would be noticeable to me and at least two other people. It's not a performance issue for APT, but: 1) we use libgcrypt in libapt-pkg, which needs global initialization. Libraries should not be doing the initialization, we're basically misusing it. The reason we use a library is to not have to vendorize the hashing algorithms. But more importantly, OpenSSL is the right choice because: 2) We use GnuTLS for the https support and that has various little incompatibilities that break it with some servers, certificates, or oppressive government firewalls, causing people not to be able to download their updates via https. Hence want to replace that with OpenSSL -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
Re: Linking coreutils against OpenSSL
Hi, On 11/10/23 21:07, Stephan Verbücheln wrote: In my opinion, this is yet another reason to use a proper cryptography library (openssl, gnutls or gcrypt) instead of a custom implementation for this kind of algorithm. Yes and no. The reason several of our core tools bring their own functions is to specifically reduce the Essential package set. Hence this thread: we need to weigh the benefits against the drawbacks here. In coreutils' case, I think the benefits are kind of qualified by the number of direct users. Basically, few people have use cases that require them to routinely verify checksums with the tools from coreutils[1]. The main benefit of this move is that container images will shrink because libssl becomes part of the base layer, so fewer copies of it will be kept in stacked layers. I would disregard this as a benefit, otherwise we could make a case that more packages should be Essential. The actual drawbacks for users are minimal too: - systemd pulls it in anyway - apt will pull it in on the remaining systems I don't quite see the appeal of OpenSSL as a dependency for apt either. I have 2 Gbps Internet at home, and a laptop I bought in 2012, and apt is never CPU bound. I could see the benefit of gzip offloading into the kernel crypto engine, that would be noticeable to me and at least two other people. We already have two other such libraries in the quasi-essential set: libcrypt, and the Linux kernel. libcrypt: + already in the quasi-essential set (no extra space) - still slow kernel: + No extra space needed + Support for offload engines that use privileged access - Invisible dependency OpenSSL: + Handwritten optimized assembler functions for a set of architectures - Horrible code The optimized assembler function brings a massive speedup on amd64, which is what triggered this thread. The ARM NEON assembler code gives a moderate speedup for hashing compared to autovectorized generic code, but in general vector units are the wrong tool for cryptographic hashes, so I'm not surprised it isn't an order of magnitude. Over time, when these libraries add support for cryptography acceleration instructions for more architectures, all programs will benefit from it. Yes, but crypto acceleration in instruction form is difficult to implement in RISC architectures -- which is why these usually have separate DMA capable devices, and work queues managed in the kernel. I would expect that many rich ARM SoCs for phones, laptops and servers already have something and that openssl supports it already. What device did you run your benchmark on? I used a Zynq SoC, and just tested a random file I had that fit into memory, running sha256sum and kcapi-dgst -c sha256 five times each. OpenSSL is a bit faster (going from 1 minute to 45 seconds), but in a real world application I'd give this an offload engine on the FPGA side if it was a hot path, and ignore it if it was not. In summary, I don't believe this change has any measurable effect beyond growing the Essential set and improving artificial benchmarks, so I'm pretty lukewarm about this. Simon [1] "debootstrap george" is an outlier and should not have been counted
Re: Linking coreutils against OpenSSL
On Fri, 10 Nov 2023 at 14:22, Pierre-Elliott Bécue wrote:
>
> Luca Boccassi wrote on 10/11/2023 at 15:00:24+0100:
>
> > On Fri, 10 Nov 2023 at 13:45, Bjørn Mork wrote:
> >>
> >> Luca Boccassi writes:
> >>
> >> > If we want to start nitpicking, then let's be exact: 0.61% of popcon.
> >> > Whether that qualifies as "plenty" we'll leave as an exercise for the
> >> > readers.
> >>
> >> I wonder if this sort of arrogant "don't care about minorities" attitude
> >> will ever stop? Was sort of hoping that things would quiet down when
> >> everyone just gave up on the systemd and usr-merge crazyness. But it
> >> seems the bullies continue on and on and on and on.
> >>
> >> I guess those 0.61% are run bt the most valuable Debian users out there.
> >> I'm sorry to not be one of them, but that's the way things have become.
> >> Not by choice.
> >
> > Having a 5x performance improvement in core utilities at zero cost for
> > 99.39% of users and at the cost of an extra library that is already
> > widely used for 0.61% --> bullying. Noted.
>
> The problem is not the argument but clearly the way you state it, which
> is quite coherent with your way of answering most of the time.
>
> I know it's not mandatory, but a bit of empathy, a bit of caring and a
> bit of "just because I can act bluntly with a pinch of contempt doesn't
> mean I should" would be really appreciable.
You know what else would be really appreciable? Not starting the
usual, tired, old flamewar, with heavily emotionally charged language
("bullying", "contempt", "lack of empathy", "lack of caring", etc),
when talking about something as trivial and harmless as installing a
widely used and common library to get a 5x performance boost out of
the core system. How about we do that instead? Because, just in case
perspective was already lost, this is what started the tirade:
> This sounds great. systemd also uses OpenSSL for various things, so
> libssl3 is pretty much a given on any bootable installation anyway
> already.
IE: a supportive and appreciative comment, followed by a relevant and
objective piece of information. Where's the bullying?
Re: Linking coreutils against OpenSSL
On Fri, Nov 10, 2023 at 03:10:42PM +0100, Ansgar wrote: Please avoid producing different results depending on the build environment. That just results in non-reproducible issues in unclean environments (suddenly different dependencies, different features, ...). I think that is an acceptable tradeoff at this time; the only difference will be the dependencies, but that is the intent. Automated buildd packages should be stable. Based on further experience and feedback, one of the other options could be chosen instead. (I'm particularly interested in hearing from people who compare the different builds on arm, as that is where there's been an assertion of a performance regression.)
Re: Linking coreutils against OpenSSL
Luca Boccassi wrote on 10/11/2023 at 15:00:24+0100: > On Fri, 10 Nov 2023 at 13:45, Bjørn Mork wrote: >> >> Luca Boccassi writes: >> >> > If we want to start nitpicking, then let's be exact: 0.61% of popcon. >> > Whether that qualifies as "plenty" we'll leave as an exercise for the >> > readers. >> >> I wonder if this sort of arrogant "don't care about minorities" attitude >> will ever stop? Was sort of hoping that things would quiet down when >> everyone just gave up on the systemd and usr-merge crazyness. But it >> seems the bullies continue on and on and on and on. >> >> I guess those 0.61% are run bt the most valuable Debian users out there. >> I'm sorry to not be one of them, but that's the way things have become. >> Not by choice. > > Having a 5x performance improvement in core utilities at zero cost for > 99.39% of users and at the cost of an extra library that is already > widely used for 0.61% --> bullying. Noted. The problem is not the argument but clearly the way you state it, which is quite coherent with your way of answering most of the time. I know it's not mandatory, but a bit of empathy, a bit of caring and a bit of "just because I can act bluntly with a pinch of contempt doesn't mean I should" would be really appreciable. -- PEB signature.asc Description: PGP signature
Re: Linking coreutils against OpenSSL
On Fri, 10 Nov 2023 at 13:48, Michael Stone wrote: > > On Fri, Nov 10, 2023 at 10:38:13AM +, Luca Boccassi wrote: > >Per-architecture dependencies are possible though, so maybe starting > >to add the libssl dependency only on amd64 is a good starting point, > >and then users of other architectures can request to be added too if > >it is beneficial for them. > > I haven't seen any objections to the basic idea, so I'm starting here: > coreutils 9.4-2 will link to libcrypto if there's a gpl-compatible > version available at build time, but I've added the build-dependency as > linux-amd64 only for now. That should make it fairly straightforward for > people to control the linking on other architectures by controlling > their build environment. Going forward, depending on feedback, I can > roll this back, expand the build-dep, and/or make the configure option > also depend on the arch. Very nice, thank you for this work!
Re: Linking coreutils against OpenSSL
On Fri, 2023-11-10 at 08:48 -0500, Michael Stone wrote: > On Fri, Nov 10, 2023 at 10:38:13AM +, Luca Boccassi wrote: > > Per-architecture dependencies are possible though, so maybe starting > > to add the libssl dependency only on amd64 is a good starting point, > > and then users of other architectures can request to be added too if > > it is beneficial for them. > > I haven't seen any objections to the basic idea, so I'm starting here: > coreutils 9.4-2 will link to libcrypto if there's a gpl-compatible > version available at build time, but I've added the build-dependency as > linux-amd64 only for now. That should make it fairly straightforward for > people to control the linking on other architectures by controlling > their build environment. Going forward, depending on feedback, I can > roll this back, expand the build-dep, and/or make the configure option > also depend on the arch. Please avoid producing different results depending on the build environment. That just results in non-reproducible issues in unclean environments (suddenly different dependencies, different features, ...). Please consider to just use openssl everywhere or also explicitly disable/enable build options per arch. (Personally I would in this case probably just enable openssl everywhere and recommend people to improve openssl in case it is slower than the built-in implementation; openssl is probably use widely enough to warrant that.) Ansgar
Re: Linking coreutils against OpenSSL
Hi, On Fri, 2023-11-10 at 14:45 +0100, Bjørn Mork wrote: > I guess those 0.61% are run bt the most valuable Debian users out > there. I'm sorry to not be one of them, but that's the way things > have become. Do you have any data to back up this claim? Or is it just a totally made up guess and one could just as well claim that those are the least valuable 0.61%? Ansgar
Re: Linking coreutils against OpenSSL
On Fri, 10 Nov 2023 at 13:45, Bjørn Mork wrote: > > Luca Boccassi writes: > > > If we want to start nitpicking, then let's be exact: 0.61% of popcon. > > Whether that qualifies as "plenty" we'll leave as an exercise for the > > readers. > > I wonder if this sort of arrogant "don't care about minorities" attitude > will ever stop? Was sort of hoping that things would quiet down when > everyone just gave up on the systemd and usr-merge crazyness. But it > seems the bullies continue on and on and on and on. > > I guess those 0.61% are run bt the most valuable Debian users out there. > I'm sorry to not be one of them, but that's the way things have become. > Not by choice. Having a 5x performance improvement in core utilities at zero cost for 99.39% of users and at the cost of an extra library that is already widely used for 0.61% --> bullying. Noted.
Re: Linking coreutils against OpenSSL
On Fri, Nov 10, 2023 at 10:38:13AM +, Luca Boccassi wrote: Per-architecture dependencies are possible though, so maybe starting to add the libssl dependency only on amd64 is a good starting point, and then users of other architectures can request to be added too if it is beneficial for them. I haven't seen any objections to the basic idea, so I'm starting here: coreutils 9.4-2 will link to libcrypto if there's a gpl-compatible version available at build time, but I've added the build-dependency as linux-amd64 only for now. That should make it fairly straightforward for people to control the linking on other architectures by controlling their build environment. Going forward, depending on feedback, I can roll this back, expand the build-dep, and/or make the configure option also depend on the arch.
Re: Linking coreutils against OpenSSL
Luca Boccassi writes: > If we want to start nitpicking, then let's be exact: 0.61% of popcon. > Whether that qualifies as "plenty" we'll leave as an exercise for the > readers. I wonder if this sort of arrogant "don't care about minorities" attitude will ever stop? Was sort of hoping that things would quiet down when everyone just gave up on the systemd and usr-merge crazyness. But it seems the bullies continue on and on and on and on. I guess those 0.61% are run bt the most valuable Debian users out there. I'm sorry to not be one of them, but that's the way things have become. Not by choice. Bjørn
Re: Linking coreutils against OpenSSL
On Fri, 10 Nov 2023 at 11:54, Matthew Vernon wrote: > > Luca Boccassi writes: > > > On Thu, 9 Nov 2023 at 22:54, Benjamin Barenblat wrote: > > >> What would you think about having coreutils Depend on libssl3? This > >> would make the libssl3 package essential, which is potentially > >> undesirable, but it also has the potential for serious user time savings > >> (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than > >> coreutils’ internal implementation). > > > > This sounds great. systemd also uses OpenSSL for various things, so > > libssl3 is pretty much a given on any bootable installation anyway > > already. > > I have no particular opinion on the proposal to make coreutils Depend > on libssl3; but there are plenty of bootable Debian systems without > systemd, so I'd like to gently push back on the idea that the only > bootable Debian systems are those using systemd as init. > > Thanks, > > Matthew > [commenting here, but I see at least one repetition of the claim in the > thread] If we want to start nitpicking, then let's be exact: 0.61% of popcon. Whether that qualifies as "plenty" we'll leave as an exercise for the readers.
Re: Linking coreutils against OpenSSL
In my opinion, this is yet another reason to use a proper cryptography library (openssl, gnutls or gcrypt) instead of a custom implementation for this kind of algorithm. Over time, when these libraries add support for cryptography acceleration instructions for more architectures, all programs will benefit from it. I would expect that many rich ARM SoCs for phones, laptops and servers already have something and that openssl supports it already. What device did you run your benchmark on? Regards Stephan signature.asc Description: This is a digitally signed message part
Re: Linking coreutils against OpenSSL
Luca Boccassi writes: > On Thu, 9 Nov 2023 at 22:54, Benjamin Barenblat wrote: >> What would you think about having coreutils Depend on libssl3? This >> would make the libssl3 package essential, which is potentially >> undesirable, but it also has the potential for serious user time savings >> (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than >> coreutils’ internal implementation). > > This sounds great. systemd also uses OpenSSL for various things, so > libssl3 is pretty much a given on any bootable installation anyway > already. I have no particular opinion on the proposal to make coreutils Depend on libssl3; but there are plenty of bootable Debian systems without systemd, so I'd like to gently push back on the idea that the only bootable Debian systems are those using systemd as init. Thanks, Matthew [commenting here, but I see at least one repetition of the claim in the thread] -- "At least you know where you are with Microsoft." "True. I just wish I'd brought a paddle." http://www.debian.org
Re: Linking coreutils against OpenSSL
On Fri, Nov 10, 2023 at 10:38:13AM +, Luca Boccassi wrote: > On Fri, 10 Nov 2023 at 02:26, Simon Richter wrote: > > > > Hi, > > > > > What would you think about having coreutils Depend on libssl3? This > > > would make the libssl3 package essential, which is potentially > > > undesirable, but it also has the potential for serious user time savings > > > (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than > > > coreutils’ internal implementation). > > > > That is only on amd64 though. > > > > On ARM and riscv64, OpenSSL is slightly slower than coreutils' > > sha256sum, so this would introduce an additional dependency and degrade > > performance. The best choice there is the kernel crypto API, which knows > > about offload hardware and special CPU instructions, both of which are > > common. > > Per-architecture dependencies are possible though, so maybe starting > to add the libssl dependency only on amd64 is a good starting point, > and then users of other architectures can request to be added too if > it is beneficial for them. > Per architecture does cause problems if we end up with a two tier system where some architectures can never catch up or if we have users who assume that Debian will work similarly on every architecture. It may also introduce code complexity which is undesirable in the longer term. Just my 0.02 - I have no particular viewpoint in the best course of action overall. Andy [amaca...@debian.org]
Re: Linking coreutils against OpenSSL
On Fri, 10 Nov 2023 at 02:26, Simon Richter wrote: > > Hi, > > > What would you think about having coreutils Depend on libssl3? This > > would make the libssl3 package essential, which is potentially > > undesirable, but it also has the potential for serious user time savings > > (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than > > coreutils’ internal implementation). > > That is only on amd64 though. > > On ARM and riscv64, OpenSSL is slightly slower than coreutils' > sha256sum, so this would introduce an additional dependency and degrade > performance. The best choice there is the kernel crypto API, which knows > about offload hardware and special CPU instructions, both of which are > common. Per-architecture dependencies are possible though, so maybe starting to add the libssl dependency only on amd64 is a good starting point, and then users of other architectures can request to be added too if it is beneficial for them.
Re: Linking coreutils against OpenSSL
On Thu, Nov 09, 2023 at 05:38:05PM -0500, Benjamin Barenblat wrote: > Dear Debian folks, > > coreutils can link against OpenSSL, yielding a substantial speed boost > in sha256sum etc. For many years, this was inadvisable due to license > conflicts. However, as of bookworm, coreutils requires GPL-3+ and > OpenSSL is Apache-2.0, so I believe all license compatibility questions > have been resolved. > > What would you think about having coreutils Depend on libssl3? This > would make the libssl3 package essential, which is potentially > undesirable, but it also has the potential for serious user time savings > (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than > coreutils’ internal implementation). I plan to switch APT to OpenSSL for trixie too, so this is not a huge concern to me. It was rightfully pointed out that systemd already uses it, so it's already on bootable systems, with APT using it, it will also be around in chroots. > > Alternatively, what would you think about making sha256sum etc. > divertible and providing implementations both with and without the > OpenSSL dependency? Oh please no. There should be a single build of coreutils. There is no point in doing multiple ones. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
Re: Linking coreutils against OpenSSL
On Fri, Nov 10, 2023 at 11:25:34AM +0900, Simon Richter wrote: > Hi, > > > What would you think about having coreutils Depend on libssl3? This > > would make the libssl3 package essential, which is potentially > > undesirable, but it also has the potential for serious user time savings > > (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than > > coreutils’ internal implementation). > > That is only on amd64 though. > > On ARM and riscv64, OpenSSL is slightly slower than coreutils' sha256sum, so > this would introduce an additional dependency and degrade performance. The > best choice there is the kernel crypto API, which knows about offload > hardware and special CPU instructions, both of which are common. OpenSSL should have the same hardware acceleration features as the kernel crypto API, without the socket overhead. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
Re: Linking coreutils against OpenSSL
On Thu, Nov 09, 2023 at 11:13:51PM +, Luca Boccassi wrote: > On Thu, 9 Nov 2023 at 22:54, Benjamin Barenblat wrote: > > > > Dear Debian folks, > > > > coreutils can link against OpenSSL, yielding a substantial speed boost > > in sha256sum etc. For many years, this was inadvisable due to license > > conflicts. However, as of bookworm, coreutils requires GPL-3+ and > > OpenSSL is Apache-2.0, so I believe all license compatibility questions > > have been resolved. > > > > What would you think about having coreutils Depend on libssl3? This > > would make the libssl3 package essential, which is potentially > > undesirable, but it also has the potential for serious user time savings > > (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than > > coreutils’ internal implementation). > > This sounds great. systemd also uses OpenSSL for various things, so > libssl3 is pretty much a given on any bootable installation anyway > already. Strongly agree. In addition to possible performance gains I think there are other benefits. Fewer better reviewed crypto implementations and a single upgrade path would be two, especially since like you say libcrypto is pretty much always there anyway. > > > Alternatively, what would you think about making sha256sum etc. > > divertible and providing implementations both with and without the > > OpenSSL dependency? > > Please, no, no more diversion/alternatives/shenanigans, it's just huge > and convoluted complications for no real gain. > +1
Re: Linking coreutils against OpenSSL
Hi, What would you think about having coreutils Depend on libssl3? This would make the libssl3 package essential, which is potentially undesirable, but it also has the potential for serious user time savings (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than coreutils’ internal implementation). That is only on amd64 though. On ARM and riscv64, OpenSSL is slightly slower than coreutils' sha256sum, so this would introduce an additional dependency and degrade performance. The best choice there is the kernel crypto API, which knows about offload hardware and special CPU instructions, both of which are common. Alternatively, what would you think about making sha256sum etc. divertible and providing implementations both with and without the OpenSSL dependency? Makes sense to me, especially if it is a diversion only, not full alternatives (which could not have sensible priorities, because what is fastest is system dependent. Simon
Re: Linking coreutils against OpenSSL
On Thu, 9 Nov 2023 at 22:54, Benjamin Barenblat wrote: > > Dear Debian folks, > > coreutils can link against OpenSSL, yielding a substantial speed boost > in sha256sum etc. For many years, this was inadvisable due to license > conflicts. However, as of bookworm, coreutils requires GPL-3+ and > OpenSSL is Apache-2.0, so I believe all license compatibility questions > have been resolved. > > What would you think about having coreutils Depend on libssl3? This > would make the libssl3 package essential, which is potentially > undesirable, but it also has the potential for serious user time savings > (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than > coreutils’ internal implementation). This sounds great. systemd also uses OpenSSL for various things, so libssl3 is pretty much a given on any bootable installation anyway already. > Alternatively, what would you think about making sha256sum etc. > divertible and providing implementations both with and without the > OpenSSL dependency? Please, no, no more diversion/alternatives/shenanigans, it's just huge and convoluted complications for no real gain.
Linking coreutils against OpenSSL
Dear Debian folks, coreutils can link against OpenSSL, yielding a substantial speed boost in sha256sum etc. For many years, this was inadvisable due to license conflicts. However, as of bookworm, coreutils requires GPL-3+ and OpenSSL is Apache-2.0, so I believe all license compatibility questions have been resolved. What would you think about having coreutils Depend on libssl3? This would make the libssl3 package essential, which is potentially undesirable, but it also has the potential for serious user time savings (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than coreutils’ internal implementation). Alternatively, what would you think about making sha256sum etc. divertible and providing implementations both with and without the OpenSSL dependency? Best, Benjamin
