Re: Is detecting endianness at compile-time unworkable?
Ævar Arnfjörð Bjarmason writes: > And, as an aside, the reason we can't easily make it better ourselves is > because the build process for git.git doesn't have a facility to run > code to detect this type of stuff (the configure script is always > optional). So we can't just run this test ourselves. It won't help those who cross-compile anyway. I thought we declared "we make a reasonable effort to guess the target endianness from the system header by inspecting usual macros, but will not aim to cover every system on the planet---instead there is a knob to tweak it for those on exotic platforms" last time we discussed this?
Re: Is detecting endianness at compile-time unworkable?
Junio C Hamano writes: > Ævar Arnfjörð Bjarmason writes: > >> And, as an aside, the reason we can't easily make it better ourselves is >> because the build process for git.git doesn't have a facility to run >> code to detect this type of stuff (the configure script is always >> optional). So we can't just run this test ourselves. > > It won't help those who cross-compile anyway. I thought we declared > "we make a reasonable effort to guess the target endianness from the > system header by inspecting usual macros, but will not aim to cover > every system on the planet---instead there is a knob to tweak it for > those on exotic platforms" last time we discussed this? Well, having said all that, I do not think I personally mind if ./configure learned to include a "compile small program and run it to determine byte order on the build machine" as part of "we make a reasonable effort" as long as it cleanly excludes cross building case (and the result is made overridable just in case we misdetect the "cross-ness" of the build).
Re: Is detecting endianness at compile-time unworkable?
The change was definitely made for performance. Removing the if statements, conditioned upon endianess was an approx 10% improvement, which was very important to getting this library accepted into git. Thanks, Dan On Mon, Jul 30, 2018 at 11:32 AM, Junio C Hamano wrote: > Junio C Hamano writes: > > > Ævar Arnfjörð Bjarmason writes: > > > >> And, as an aside, the reason we can't easily make it better ourselves is > >> because the build process for git.git doesn't have a facility to run > >> code to detect this type of stuff (the configure script is always > >> optional). So we can't just run this test ourselves. > > > > It won't help those who cross-compile anyway. I thought we declared > > "we make a reasonable effort to guess the target endianness from the > > system header by inspecting usual macros, but will not aim to cover > > every system on the planet---instead there is a knob to tweak it for > > those on exotic platforms" last time we discussed this? > > Well, having said all that, I do not think I personally mind if > ./configure learned to include a "compile small program and run it > to determine byte order on the build machine" as part of "we make a > reasonable effort" as long as it cleanly excludes cross building > case (and the result is made overridable just in case we misdetect > the "cross-ness" of the build). > >
Re: Is detecting endianness at compile-time unworkable?
I have just replied to
https://github.com/cr-marcstevens/sha1collisiondetection/pull/42
I checked a gcc compiler on AIX, and I have the defines for vac.
I do not have access yet to SLES or RHEL (or Ubuntu), just a "free
Debian" on my Power6.
* my conclusions|recommendations:
a) AIX is always Big Endian, the define _AIX can be used to determine if AIX
b) POWER7 and earlier are always Big Endian
c) assuming lscpu is always available on Linux systems a command (in
configure?) could be used:
root@x074:/usr/bin# lscpu | grep -i endian
Byte Order: Big Endian
d) some linux systems (in any case latest versions of RHEL and SLES
enterprise) should have a file named lparcfg in /proc
(/proc/{powerppc|ppc64|ppc64le|ppc64el}/lparcfg - and it might be in
that file. Need to get onto a (POWER8|POWER9) system to check.
Hope this helps:
details re: define of _AIX
root@x068:[/data/httpd/gcc]gcc -dM -E - < /dev/null | grep AIX | head -1
#define _AIX 1
michael@x071:[/home/michael]/usr/bin/grep -p DEFLT:
/etc/vac.cfg.[567][123] | grep options\
options =
-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_IBMR2,-D_POWER
options =
-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_AIX61,-D_IBMR2,-D_POWER
options =
-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_AIX61,-D_AIX71,-D_IBMR2,-D_POWER
options =
-D_AIX,-D_AIX32,-D_AIX41,-D_AIX43,-D_AIX50,-D_AIX51,-D_AIX52,-D_AIX53,-D_AIX61,-D_AIX71,-D_AIX72,-D_IBMR2,-D_POWER
michael@x071:[/home/michael]ls /etc/vac.cfg.[567][123]
/etc/vac.cfg.53 /etc/vac.cfg.61 /etc/vac.cfg.71 /etc/vac.cfg.72
On 7/30/2018 8:39 PM, Daniel Shumow wrote:
The change was definitely made for performance. Removing the if
statements, conditioned upon endianess was an approx 10% improvement,
which was very important to getting this library accepted into git.
Thanks,
Dan
On Mon, Jul 30, 2018 at 11:32 AM, Junio C Hamano > wrote:
Junio C Hamano mailto:gits...@pobox.com>> writes:
> Ævar Arnfjörð Bjarmason mailto:ava...@gmail.com>> writes:
>
>> And, as an aside, the reason we can't easily make it better
ourselves is
>> because the build process for git.git doesn't have a facility
to run
>> code to detect this type of stuff (the configure script is always
>> optional). So we can't just run this test ourselves.
>
> It won't help those who cross-compile anyway. I thought we declared
> "we make a reasonable effort to guess the target endianness from the
> system header by inspecting usual macros, but will not aim to cover
> every system on the planet---instead there is a knob to tweak it for
> those on exotic platforms" last time we discussed this?
Well, having said all that, I do not think I personally mind if
./configure learned to include a "compile small program and run it
to determine byte order on the build machine" as part of "we make a
reasonable effort" as long as it cleanly excludes cross building
case (and the result is made overridable just in case we misdetect
the "cross-ness" of the build).
Re: Is detecting endianness at compile-time unworkable?
A small step back...
On 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote:
On Sun, Jul 29 2018, Michael wrote:
On 29/07/2018 22:06, brian m. carlson wrote:
On Sun, Jul 29, 2018 at 09:48:43PM +0200, Michael wrote:
On 29/07/2018 21:27, brian m. carlson wrote:
Well, that explains it. I would recommend submitting a patch to
https://github.com/cr-marcstevens/sha1collisiondetection, and the we can
pull in the updated submodule with that fix.
Not sure I am smart enough to do that. I'll have to download, build, and see
what it says.
The issue is that somewhere in lib/sha1.c, you need to cause
SHA1DC_BIGENDIAN to be set. That means you need to figure out what
compiler macro might indicate that.
I remember - roughly - a few decades back - having an assignment to
write code to determine endianness. PDP and VAC were different iirc,
and many other micro-processors besides the 8088/8086/z85/68k/etc..
If you are looking for a compiler macro as a way to determine this -
maybe you have one for gcc, but not for xlc. I do not know it - currently :)
I'm not familiar with AIX, but from searching around I found this
porting manual from IBM:
http://www.redbooks.ibm.com/redbooks/pdfs/sg246034.pdf
This is from July 2001 - when AIX 5L, for Linux affinity, was new. AIX
was (nearly) the #1 posix system, and linux was a minor player in the
data center (in or out (now as IAAS)). IMHO, the recommendations made in
2001 are probably no longer applicable (64-bit was fairly new, e.g.,
rather than common).
There they suggest either defining your own macros, or testing the
memory layout at runtime (see section "2.2.2.3 Technique 3: Testing
memory layout" and surrounding sections).
Perhaps it's worth taking a step back here and thinking about whether
this whole thing is unworkable. It was hard enough to get this to work
on the combination of Linux, *BSD and Solaris, but I suspect we'll run
into increasingly obscure platforms where this is hard or impossible
(AIX, HP/UX etc.)
The reason we're in this hole is because we use this
sha1collisiondetection library to do SHA-1, and the reason we have
issues with it specifically (not OpenSSL et al) is because its only
method of detecting endianness is at compile time.
Cannot speak for the "others", but as I have mentioned before - as AIX
in only on POWER it is also only Big Endian - so a compiletime #if
testing for _AIX will work fine
This didn't use to be the case, it was changed in this commit:
https://github.com/cr-marcstevens/sha1collisiondetection/commit/d597672
Dan Shumow: Since the commit message doesn't say why, can you elaborate
a bit on why this was done, i.e. is determining this at runtime harmful
for performance? If not, perhaps it would be best to bring this back, at
least as an option.
And, as an aside, the reason we can't easily make it better ourselves is
because the build process for git.git doesn't have a facility to run
code to detect this type of stuff (the configure script is always
optional). So we can't just run this test ourselves.
On AIX - I am required to run configure, and frankly, I am amazed that
not everyone is running it. Among other things I an modifying the prefix
(to /opt) and many of the others to different /var/git/* areas as I do
not want to "polute" the BOS (base OS) and/or other packages/packagers.
Officially, according the Linux FHS-3.0 I sould be using /opt/aixtools
as prefix.
FYI: my current build process is:
wget git-{version}.tar.xz
xz -dc git-{version}.tar.xz
cd git-{version}.tar.xz
mv Makefile Makefile.git #my build scripts only run configure when
Makefile does not exist
./configure ...
ln Makefile.git Makefile
make
I am amazed, as it rarely happens (maybe git is my first encounter) -
that configure does not create a Makefile. This also complicates
building git "out of tree".
Junio: I've barked up that particular tree before in
https://public-inbox.org/git/87a7x3kmh5@evledraar.gmail.com/ and I
won't bore you all by repeating myself, except to say that this is yet
another case where I wish we had a hard dependency on some way of doing
checks via compiled code in our build system.
For AIX: again - the determination is simple. If _AIX is set to 1 then
use BigEndian, or, use:
michael@x071:[/home/michael]uname
AIX
i.e., something like:
$(uname) == "AIX" && BigEndian=1
Re: Is detecting endianness at compile-time unworkable?
On 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote: > The reason we're in this hole is because we use this > sha1collisiondetection library to do SHA-1, and the reason we have > issues with it specifically (not OpenSSL et al) is because its only > method of detecting endianness is at compile time. When using gcc (no xlc available for Linux on Power) POWER6 (Big Endian by definition) root@x068:[/data/httpd/gcc]gcc -dM -E - < /dev/null | grep -i end #define __ORDER_LITTLE_ENDIAN__ 1234 #define __BIG_ENDIAN__ 1 #define __FLOAT_WORD_ORDER__ __ORDER_BIG_ENDIAN__ #define __ORDER_PDP_ENDIAN__ 3412 #define _BIG_ENDIAN 1 #define __ORDER_BIG_ENDIAN__ 4321 #define __BYTE_ORDER__ __ORDER_BIG_ENDIAN__ SLES12 on POWER8 suse12test:~ # gcc -dM -E - < /dev/null | grep -i end #define __ORDER_LITTLE_ENDIAN__ 1234 #define _LITTLE_ENDIAN 1 #define __FLOAT_WORD_ORDER__ __ORDER_LITTLE_ENDIAN__ #define __ORDER_PDP_ENDIAN__ 3412 #define __LITTLE_ENDIAN__ 1 #define __ORDER_BIG_ENDIAN__ 4321 #define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__ *So, for compile time tests, when gcc is the compiler it seems the following defines are available** **__BIG_ENDIAN__, _BIG_ENDIAN, __LITTLE__ENDIAN__, _LITTLE_ENDIAN** **or something based on the value of __BYTE_ORDER__* I'll see if I can find something similar for xlc, but will only be able to test xlc on AIX. > > This didn't use to be the case, it was changed in this commit: > https://github.com/cr-marcstevens/sha1collisiondetection/commit/d597672 > > Dan Shumow: Since the commit message doesn't say why, can you elaborate > a bit on why this was done, i.e. is determining this at runtime harmful > for performance? If not, perhaps it would be best to bring this back, at > least as an option.
Re: Is detecting endianness at compile-time unworkable?
I hope a I have a "leap forward" On 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote: > Perhaps it's worth taking a step back here and thinking about whether > this whole thing is unworkable. It was hard enough to get this to work > on the combination of Linux, *BSD and Solaris, but I suspect we'll run > into increasingly obscure platforms where this is hard or impossible > (AIX, HP/UX etc.) While I still cannot say for HP/UX it does seem there is a potential solution based on the status for _LITTLE_ENDIAN and _BIG_ENDIAN. At least, gcc on POWER and xlc on POWER provides one or the other - and my hope is that gcc on other platforms also provides them. For "other" compilers that do not provide them - a modification to CFLAGS to define one or the other should make "make" work. Details (note - I am not a programmer, so by definition at least one of my "macros" will be wrong :) AIX and xlc root@x066:[/]xlc -qshowmacros -E /dev/null | grep -i endi 1506-297 (S) Unable to open input file null. A file or directory in the path name does not exist.. #define __HHW_BIG_ENDIAN__ 1 #define __BIG_ENDIAN__ 1 #define __THW_BIG_ENDIAN__ 1 #define _BIG_ENDIAN 1 On SLES12 (le) and xlc suse12test:~/images/littleEndian/sles # xlc -qshowmacros -dM -E x.c | grep -i endi #define _LITTLE_ENDIAN 1 #define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__ #define __FLOAT_WORD_ORDER__ __ORDER_LITTLE_ENDIAN__ #define __LITTLE_ENDIAN__ 1 #define __ORDER_BIG_ENDIAN__ 4321 #define __ORDER_LITTLE_ENDIAN__ 1234 #define __ORDER_PDP_ENDIAN__ 3412 #define __VEC_ELEMENT_REG_ORDER__ __ORDER_LITTLE_ENDIAN__ Based on what I can see on gcc on POWER and xlc on POWER I think an approach (simplified) can be: #if undefined(_BIG_ENDIAN) && undef(_LITTLE_ENDIAN) #error "one of _BIG_ENDIAN or _LITTLE_ENDIAN must be defined. Try adding the correct value to CFLAGS" #else defined(_BIG_ENDIAN) && defined(_LITTLE_ENDIAN) #error "Only one of _BIG_ENDIAN and _LITTLE_ENDIAN may be defined, not both" #endif And then logic based on the value set. This should also make cross-compile possible by unsetting an incorrect default and setting the correct value. p.s. Is there a setting I need to set somewhere so I receive a copy of the email sent after it is received by the list. I could send myself a copy, but I much prefer it comes from the maillist - as verification it was received.
Re: Is detecting endianness at compile-time unworkable?
On Tue, Jul 31 2018, Michael Felt wrote: > I hope a I have a "leap forward" > > > On 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote: >> Perhaps it's worth taking a step back here and thinking about whether >> this whole thing is unworkable. It was hard enough to get this to work >> on the combination of Linux, *BSD and Solaris, but I suspect we'll run >> into increasingly obscure platforms where this is hard or impossible >> (AIX, HP/UX etc.) > While I still cannot say for HP/UX it does seem there is a potential > solution based on the status for _LITTLE_ENDIAN and _BIG_ENDIAN. At > least, gcc on POWER and xlc on POWER provides one or the other - and my > hope is that gcc on other platforms also provides them. Yeah with GCC this is relatively easy, see https://github.com/cr-marcstevens/sha1collisiondetection/blame/c3e1304/lib/sha1.c#L29-L115 > For "other" compilers that do not provide them - a modification to > CFLAGS to define one or the other should make "make" work. > > Details (note - I am not a programmer, so by definition at least one of > my "macros" will be wrong :) > > AIX and xlc > root@x066:[/]xlc -qshowmacros -E /dev/null | grep -i endi > 1506-297 (S) Unable to open input file null. A file or directory in the > path name does not exist.. > #define __HHW_BIG_ENDIAN__ 1 > #define __BIG_ENDIAN__ 1 > #define __THW_BIG_ENDIAN__ 1 > #define _BIG_ENDIAN 1 > > On SLES12 (le) and xlc > suse12test:~/images/littleEndian/sles # xlc -qshowmacros -dM -E x.c | > grep -i endi > #define _LITTLE_ENDIAN 1 > #define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__ > #define __FLOAT_WORD_ORDER__ __ORDER_LITTLE_ENDIAN__ > #define __LITTLE_ENDIAN__ 1 > #define __ORDER_BIG_ENDIAN__ 4321 > #define __ORDER_LITTLE_ENDIAN__ 1234 > #define __ORDER_PDP_ENDIAN__ 3412 > #define __VEC_ELEMENT_REG_ORDER__ __ORDER_LITTLE_ENDIAN__ > > > Based on what I can see on gcc on POWER and xlc on POWER I think an > approach (simplified) can be: > > #if undefined(_BIG_ENDIAN) && undef(_LITTLE_ENDIAN) > #error "one of _BIG_ENDIAN or _LITTLE_ENDIAN must be defined. Try adding > the correct value to CFLAGS" > #else defined(_BIG_ENDIAN) && defined(_LITTLE_ENDIAN) > #error "Only one of _BIG_ENDIAN and _LITTLE_ENDIAN may be defined, not both" > #endif > > And then logic based on the value set. > This should also make cross-compile possible by unsetting an incorrect > default and setting the correct value. ...the real trick is using these macros outside of GCC / glibc and on older GCC versions. See the github link above, you basically end up with a whitelist of how it looks on different systems / compilers. Sometimes both are defined, sometimes only both etc. It can be done, but as that code shows it's somewhat complex macro soup to get right. > p.s. Is there a setting I need to set somewhere so I receive a copy of > the email sent after it is received by the list. I could send myself a > copy, but I much prefer it comes from the maillist - as verification it > was received. You should get that, but maybe your mailer ignores Message-Ids it already has, but you can go to https://public-inbox.org/git/ and search for the Message-Id or your name to see E-Mails you've sent that made it to the list, e.g.: https://public-inbox.org/git/?q=aixtools%40felt.demon.nl
Re: Is detecting endianness at compile-time unworkable?
On 31/07/2018 16:25, Ævar Arnfjörð Bjarmason wrote: ...the real trick is using these macros outside of GCC / glibc and on older GCC versions. See the github link above, you basically end up with a whitelist of how it looks on different systems / compilers. Sometimes both are defined, sometimes only both etc. It can be done, but as that code shows it's somewhat complex macro soup to get right. FYI - the gcc I was using is 4.7.4. And, the reason I suggest the test for both not being defined is so that 'make' stops and whoever is running make just sets one or the other. Let them 'file a bug' When they come with a compiler that does not work - and find out what could be used. For example, _AIX is the same as _BIG_ENDIAN. In the meantime, the code to test is simple. Either one of _BIG_ENDIAN or _LITTLE_ENDIAN is provided by the compiler or the builder supplies one of the two using CFLAGS. I assume there is also a "undefine" flag, maybe -U - so hopefully a -U and a -D combination could be used for cross-compiling. re: my mailer blocking things - it would only be for this list, as other lists come through with no extra work from me. At least I am not aware of anything special I could do.
Re: Is detecting endianness at compile-time unworkable?
Junio C Hamano wrote: > Well, having said all that, I do not think I personally mind if > ./configure learned to include a "compile small program and run it > to determine byte order on the build machine" as part of "we make a > reasonable effort" as long as it cleanly excludes cross building > case (and the result is made overridable just in case we misdetect > the "cross-ness" of the build). No need to run the program for cross-compiles, grepping a object file seems to work for autoconf: git clone https://git.sv.gnu.org/git/autoconf.git $PAGER autoconf/lib/autoconf/c.m4 # look for "BIGenDianSyS"
Re: Is detecting endianness at compile-time unworkable?
On Mon, Jul 30 2018, Junio C Hamano wrote: > Ævar Arnfjörð Bjarmason writes: > >> And, as an aside, the reason we can't easily make it better ourselves is >> because the build process for git.git doesn't have a facility to run >> code to detect this type of stuff (the configure script is always >> optional). So we can't just run this test ourselves. > > It won't help those who cross-compile anyway. I was being unclear, what I mean by having "a hard dependency on some way of doing checks via compiled code in our build system" is that we would do some equivalent of this: diff --git a/Makefile b/Makefile index 08e5c54549..b021b6e1b6 100644 --- a/Makefile +++ b/Makefile @@ -1107,7 +1107,7 @@ DC_SHA1_SUBMODULE = auto endif include config.mak.uname --include config.mak.autogen +include config.mak.autogen -include config.mak ifdef DEVELOPER And document that in order to build git you needed to do './configure && make' instead of just 'make', and we'd error out by default if config.mak.autogen wasn't there. Now obviously that would need some sort of escape hatch. I.e. you could invoke 'make' like this: make CONFIGURE_MAK_AUTOGEN_FILE=some-file That's how you would do cross-compilation, you'd arrange to run './configure' on some system, save the output, and ferry over this 'some-file' to where you're building git, or you would manually prepare a file that had all the settings we'd expect to have been set already set. Now, whether we do this with autoconf or not is just an implementation detail. Looking at this some more I think since we already use the $(shell) construct we could just have some 'configure-detect' Makefile target which would compile various test programs, and we'd use their output to set various settings, a sort of home-grown autoconf (because people are bound to have objections to a hard dependency on it...). > I thought we declared "we make a reasonable effort to guess the target > endianness from the system header by inspecting usual macros, but will > not aim to cover every system on the planet---instead there is a knob > to tweak it for those on exotic platforms" last time we discussed > this? Yes, but I think it's worth re-visiting that decision, which was made with the constraints that we don't have a build system that can do checks via compiled code, so we need this hack in the first place instead of things Just Working. And as I pointed out in the linked E-Mail this also impacts us in other ways, and will cause other issues in the future, so it's worth thinking about if this is the right path to take.
Re: Is detecting endianness at compile-time unworkable?
On Tue, Jul 31 2018, Michael Felt wrote: > For AIX: again - the determination is simple. If _AIX is set to 1 then > use BigEndian, or, use: > michael@x071:[/home/michael]uname > AIX > i.e., something like: > $(uname) == "AIX" && BigEndian=1 In lieu of some "let's test this with a compile-test" solution, that seems like a viable way forward for now. Can you please test the merge request I have at https://github.com/cr-marcstevens/sha1collisiondetection/pull/45 by manually applying that change to the relevant file in sha1dc/ and building/testing git on AIX? It should work, but as noted in the MR please test it so we can make sure, and then (if you have a GitHub account) comment on the MR saying it works for you.
