RE: about build flavours
>From: Arturo Borrero Gonzalez [mailto:art...@debian.org]
>
>@Matthew, we are wondering whether hyperscan is able to detect SSE3 support at
>runtime in a given machine.
In Hyperscan v4.3 we don't have those cpuid tests.
The next release (v4.4, due before the end of the year) will have these checks,
with an API call to check that the current machine supports what target the
Hyperscan library was built for. So means that Suricata could be linked to
Hyperscan, and at runtime the machine can be checked to see if Hyperscan can be
used.
The next release will also have the ability to be built as a "fat" library. The
Hyperscan runtime can be built in a number of configurations for different
levels of instruction set features (SSSE3, SSSE4.2+POPCNT, AVX2), and combined
in the one library, and then use the GCC ifunc ("resolver") attribute to select
the correct implementation at runtime, or if there is no SSSE3 a version of the
runtime
API that returns an error instead of ungracious SIGILL.
We still require a minimum of SSSE3, but the hope is that this will simplify
things for users. I'm hoping to get that into our devel branch on Github soon
so it can be tested more widely.
Matt.
Re: about build flavours
Thibaut Paumard writes ("Re: about build flavours"):
> Exactly, plus a pair of symlinks with a default value and sysadmin
> control. The standard way is using the alternatives system:
> https://wiki.debian.org/DebianAlternatives
This is good advice if there is no way to do this without user
configuration.
I understand that libhyperscan just depends on SSE, so there can't be
runtime switching there, but there is an alternative possibility:
Provide a /usr/bin/suricata which detects the presence of SSE right
now, and exec's one of two appropriate suricata binaries.
The difficulty with this is that the system would have to have a
nonfunctional libhyperscan installed on it, for dependency reasons,
but if there are not many callers of libhyperscan all of the callers
could do whatever check or fallback is appropriate; if there are many
callers of libhyperscan, libhyperscan.deb could provide a checking
facility to be used by callers.
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: about build flavours
On 30 November 2016 at 11:33, Paul Wise wrote: > On Wed, 2016-11-30 at 11:11 +0100, Arturo Borrero Gonzalez wrote: > >> I don't think hyperscan currently is able to detect SSE3 support at runtime. > > Sounds like a bug. > >> Do you think that the warning at install-time is not enough? > > Sounds like a reasonable workaround for the bug. > CC'ing Matthew, who is upstream developer at Intel. Perhaps he could share some comments on this topic. @Matthew, we are wondering whether hyperscan is able to detect SSE3 support at runtime in a given machine. We are planning to integrate hyperscan with other packages (such as suricata) and we are evaluating possible alternatives regarding this integration. (please, keep debian-devel in CC) regards
Re: about build flavours
Hi Paul, >> Yes, the libhyperscan package alerts the user at pre-install time if >> SSE3 is not supported on the target system. That's one of the reasons >> why I think there should still be a version of suricata that works >> without Hyperscan. > > Sounds like they are doing it wrong. The detection should happen at > runtime not pre-install time. It looks like SSE3 support is decided in libhyperscan at compile time. To be exact it's actually a hard requirement with no alternative fallback code path, see [1]. In this light I think it's just fair to test usability of the code before it's even installed, to keep the user from facing broken software depending on the non-usable library. Cheers Sascha (BTW my 2 cents as someone with a scientific computing background: for specific fast implementations of common tasks I consider such requirements absolutely acceptable, especially if there are slower but generic alternatives available. Often the careful hardware-optimised implementation is the main raison d'etre for the tool or library in question. This may be not exactly the case with Hyperscan but I've encountered this issue before while packaging and it tends to make me twitch a bit.) [1] http://01org.github.io/hyperscan/dev-reference/getting_started.html#requirements
Re: about build flavours
On Wed, 2016-11-30 at 11:11 +0100, Arturo Borrero Gonzalez wrote: > I don't think hyperscan currently is able to detect SSE3 support at runtime. Sounds like a bug. > Do you think that the warning at install-time is not enough? Sounds like a reasonable workaround for the bug. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: about build flavours
On 30 November 2016 at 11:09, Paul Wise wrote: > On Wed, Nov 30, 2016 at 6:01 PM, Sascha Steinbiss wrote: > >> Yes, the libhyperscan package alerts the user at pre-install time if >> SSE3 is not supported on the target system. That's one of the reasons >> why I think there should still be a version of suricata that works >> without Hyperscan. > > Sounds like they are doing it wrong. The detection should happen at > runtime not pre-install time. > I don't think hyperscan currently is able to detect SSE3 support at runtime. Do you think that the warning at install-time is not enough?
Re: about build flavours
On Wed, Nov 30, 2016 at 6:01 PM, Sascha Steinbiss wrote: > Yes, the libhyperscan package alerts the user at pre-install time if > SSE3 is not supported on the target system. That's one of the reasons > why I think there should still be a version of suricata that works > without Hyperscan. Sounds like they are doing it wrong. The detection should happen at runtime not pre-install time. -- bye, pabs https://wiki.debian.org/PaulWise
Re: about build flavours
Hi Paul, [...] >> we are trying to create a build flavour for the suricata package [0], >> which we would like to link to hyperscan, which uses SSSE3. > > I hope it detects the presence of SSE3 at runtime. Yes, the libhyperscan package alerts the user at pre-install time if SSE3 is not supported on the target system. That's one of the reasons why I think there should still be a version of suricata that works without Hyperscan. Cheers Sascha signature.asc Description: OpenPGP digital signature
Re: about build flavours
On Tue, Nov 29, 2016 at 8:35 PM, Arturo Borrero Gonzalez wrote: > we are trying to create a build flavour for the suricata package [0], > which we would like to link to hyperscan, which uses SSSE3. I hope it detects the presence of SSE3 at runtime. -- bye, pabs https://wiki.debian.org/PaulWise
Re: about build flavours
Dear Sascha, Le 29/11/2016 à 15:24, Sascha Steinbiss a écrit : >>> >>> Then, we conflict the packages with each other and dpkg-divert the >>> /usr/bin/suricata file in suricata-hyperscan. >> >> If both packages conflict with each other, then you don't need >> dpkg-divert. > > One could think of multiple approaches with varying modularity. You can thing of many ways, but: >> And dpkg-divert is mainly useful when you want to replace a >> file that is not under your own control, otherwise using the alternatives >> system is usually preferred. > > Interesting. I assume this would require using unique installation paths > for both the Hyperscan-enabled and non-enabled /usr/bin/suricata > versions, neither of which should be /usr/bin/suricata? Exactly, plus a pair of symlinks with a default value and sysadmin control. The standard way is using the alternatives system: https://wiki.debian.org/DebianAlternatives Don't complicate life for your users, use this system. Regards, Thibaut.
Re: about build flavours
Hi Raphael, thanks, here are a few more thoughts. >> Currently, our main idea is to create a suricata-hyperscan package >> which includes the dependency to hyperscan, and the binary is build >> linked to the library. >> >> We distribute two /usr/bin/suricata binaries in two binary packages: >> * suricata, without hyperscan >> * suricata-hyperscan, with hyperscan >> >> Then, we conflict the packages with each other and dpkg-divert the >> /usr/bin/suricata file in suricata-hyperscan. > > If both packages conflict with each other, then you don't need > dpkg-divert. One could think of multiple approaches with varying modularity. The first approach builds two full copies of the suricata package with all package contents in each, largely identical except for /usr/bin/suricata which differs w.r.t. Hyperscan support compiled in (and dependencies, obviously). These two packages would indeed be set to Conflict: with each other to avoid file clashes. In the other (IMHO better) approach one wouldn't set Conflicts. The idea is to have an original suricata package containing everything including a generic /usr/bin/suricata binary without Hyperscan support. This file would get dpkg-diverted via the suricata-hyperscan package by a version of this file that has Hyperscan support. In this case the suricata-hyperscan package would only contain /usr/bin/suricata to reduce duplication of content. It would depend on libhyperscan4 (as it should) as well as on the original suricata package, to make sure the 'rest' of the Suricata distribution is installed. This way one could 'patch' Hyperscan support in by just installing suricata-hyperscan. At least these are the two alternatives linked in #846143. > And dpkg-divert is mainly useful when you want to replace a > file that is not under your own control, otherwise using the alternatives > system is usually preferred. Interesting. I assume this would require using unique installation paths for both the Hyperscan-enabled and non-enabled /usr/bin/suricata versions, neither of which should be /usr/bin/suricata? Cheers Sascha signature.asc Description: OpenPGP digital signature
Re: about build flavours
Hello, On Tue, 29 Nov 2016, Arturo Borrero Gonzalez wrote: > Currently, our main idea is to create a suricata-hyperscan package > which includes the dependency to hyperscan, and the binary is build > linked to the library. > > We distribute two /usr/bin/suricata binaries in two binary packages: > * suricata, without hyperscan > * suricata-hyperscan, with hyperscan > > Then, we conflict the packages with each other and dpkg-divert the > /usr/bin/suricata file in suricata-hyperscan. If both packages conflict with each other, then you don't need dpkg-divert. And dpkg-divert is mainly useful when you want to replace a file that is not under your own control, otherwise using the alternatives system is usually preferred. Cheers, -- Raphaël Hertzog ◈ Debian Developer Support Debian LTS: http://www.freexian.com/services/debian-lts.html Learn to master Debian: http://debian-handbook.info/get/
