Re: odd behaviour of some programs on i386 cross-built from amd64
At Wed, 22 Apr 2020 21:08:46 -0700, "Greg A. Woods" wrote: Subject: odd behaviour of some programs on i386 cross-built from amd64 > > # od > od: "8/2 " %06o " "\n"": bad format > # file /usr/bin/od > /usr/bin/od: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), > statically linked, for NetBSD 8.99.32, stripped So at least the problems with 'awk' and 'od' seem to be related somehow to the kernels I built. I found this by installing a stock 9.0/i386 on real i386 hardware (after repairing said hardware -- it had not run in years) and finding it could run the static-linked 'od' I had previously built without problems. Curious I decided to cross-build the same source tree on the newly installed 9.0 system and, perhaps without surprise, it generated an identical binary for, e.g. 'od': $ cd $MY_DESTDIR $ cmp /future/build/woods/future/current-i386-ppro-destdir/usr/bin/od usr/bin/od -r-xr-xr-x 2 woods wheel 244288 Apr 16 15:38 /future/build/woods/future/current-i386-ppro-destdir/usr/bin/od -r-xr-xr-x 2 woods wheel 244288 Apr 26 02:36 usr/bin/od $ Similarly 'awk' looks the same works just fine. One of the kernels seems to be the same too, assuming one takes into account the obvious difference in vers.o: textdata bss dec hex filename 20222422 515508 926144 2166407414a914a sys/arch/i386/compile/MONOLITHIC/netbsd 20222382 515508 926144 2166403414a9122 /future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/MONOLITHIC/netbsd textdata bss dec hex filename 795 0 0 795 31b sys/arch/i386/compile/MONOLITHIC/vers.o 755 0 0 755 2f3 /future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/MONOLITHIC/vers.o However the other kernel I've tested is somehow not the same: textdata bss dec hex filename 4776885 80232 1347584 6204701 5ead1d sys/arch/i386/compile/XEN3PAE_DOMU/netbsd 4776585 80232 1347584 6204401 5eabf1 /future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/netbsd textdata bss dec hex filename 799 0 0 799 31f sys/arch/i386/compile/XEN3PAE_DOMU/vers.o 791 0 0 791 317 /future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/vers.o Next I'll try testing these kernels on Xen and the Soekris box. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpKTUf0bQ9F4.pgp Description: OpenPGP Digital Signature
Re: odd behaviour of some programs on i386 cross-built from amd64
At Mon, 27 Apr 2020 21:17:04 -0700, "Greg A. Woods" wrote:
Subject: Re: odd behaviour of some programs on i386 cross-built from amd64
>
> One of the kernels seems to be the same too, assuming one takes into
> account the obvious difference in vers.o:
>
>textdata bss dec hex filename
> 20222422 515508 926144 2166407414a914a
> sys/arch/i386/compile/MONOLITHIC/netbsd
> 20222382 515508 926144 2166403414a9122
> /future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/MONOLITHIC/netbsd
>
>textdata bss dec hex filename
> 795 0 0 795 31b
> sys/arch/i386/compile/MONOLITHIC/vers.o
> 755 0 0 755 2f3
> /future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/MONOLITHIC/vers.o
Indeed with a direct "cmp" of the .o files in each build, all are
identical except for vers.o and debugsyms.o -- the latter differs only
in one pathname to the compile directory, which is also to be expected
since the path prefixes differ between the two build environments.
Unfortunately this native-built kernel also works identically on the
Soekris hardware (as expected, since there are no difference in the
object code), i.e. causing the same odd behaviour to a few odd binaries.
I guess now I'll have to dig into my local kernel changes to see what
might be incompatible with a 32-bit system. Maybe I can also try an ARM
build to see if the same problems happen on an RPi or Beaglebone.
> However the other kernel I've tested is somehow not the same:
The differences are few, but inexplicable. Here are all the object
files which differ in the XEN3PAE_DOMU builds:
/build/woods/once.local/current-i386-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/arc4.o
/future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/arc4.o
differ: char 33, line 1
/build/woods/once.local/current-i386-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/bf_ecb.o
/future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/bf_ecb.o
differ: char 33, line 1
/build/woods/once.local/current-i386-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/camellia-api.o
/future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/camellia-api.o
differ: char 33, line 1
/build/woods/once.local/current-i386-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/debugsyms.o
/future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/debugsyms.o
differ: char 180, line 1
/build/woods/once.local/current-i386-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/quota1_subr.o
/future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/quota1_subr.o
differ: char 33, line 1
/build/woods/once.local/current-i386-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/vers.o
/future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/sys/arch/i386/compile/XEN3PAE_DOMU/vers.o
differ: char 33, line 1
Again debugsyms.o and vers.o differences can be ignored, but the rest
are so far inexplicably different.
All are built from the same source tree, with the same build options.
The only difference is one build host ("future") is amd64 (running under
Xen, also built from the same source tree), and the other ("once") is
running on native i386 hardware (an old Dell PE2650, but running stock
9.0).
(this sdiff output shows best at 132 column width)
sys/arch/i386/compile/XEN3PAE_DOMU/arc4.o: file format elf3 |
/future/build/woods/future/current-amd64-i386-ppro-obj/usr/src/
Disassembly of section .text: Disassembly
of section .text:
:
:
0: 55 push %ebp |0: b8 08
04 00 00 mov$0x408,%eax
1: b8 08 04 00 00 mov$0x408,%eax |5: c3
ret
6: 89 e5 mov%esp,%ebp|6: 8d b4
26 00 00 00 00lea0x0(%esi,%eiz,1),%esi
8: 5d pop%ebp |d: 8d 76
00lea0x0(%esi),%esi
9: c3 ret <
a: 8d b6 00 00 00 00 lea0x0(%esi),%esi <
0010 : 0010
:
10: 55 push %ebp 10: 55
push %ebp
11: 31 c0 xor%eax,%eax11: 31 c0
xor%eax,%eax
13: 89 e5 mov%esp,%ebp| 13: 57
Re: odd behaviour of some programs on i386 cross-built from amd64
At Tue, 28 Apr 2020 16:05:10 -0700, "Greg A. Woods" wrote:
Subject: Re: odd behaviour of some programs on i386 cross-built from amd64
>
> I guess now I'll have to dig into my local kernel changes to see what
> might be incompatible with a 32-bit system. Maybe I can also try an ARM
> build to see if the same problems happen on an RPi or Beaglebone.
I'm still working on the evbarm build -- there were a number of weird
little botches I have had to fix to get the build to go all the way
through to a complete release, but it's nearly there.
In the mean time I finally remembered to try running old binaries from
previous releases, and they work A-OK, so this may give me a way forward
to debugging what's going wrong with the new binaries (and maybe make
the Soekris machine more usable in the mean time too):
# pwd
/building/build/woods/building/netbsd-5-i386-ppro-destdir
# ll usr/bin/od
368 -r-xr-xr-x 2 woods wheel 187404 Jun 5 2016 usr/bin/od
# file usr/bin/od
usr/bin/od: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
statically linked, for NetBSD 5.2, stripped
# usr/bin/od
asdlkfj
000 071541 066144 063153 005152
010
# /usr/bin/od
od: "8/2 " %06o " "\n"": bad format
However at the moment while the old gdb (6.5 from 5.2_STABLE) works, it
won't read symbols from a new binary ("Dwarf Error"):
# file hexdump
hexdump: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically
linked, for NetBSD 8.99.32, with debug_info, not stripped
# old-gdb hexdump
GNU gdb 6.5
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386--netbsdelf"...Dwarf Error: wrong version in
compilation unit header (is 4, should be 2) [in module
/future/build/woods/future/current-amd64-i386-ppro-obj/more/work/woods/m-NetBSD-current/usr.bin/hexdump/hexdump]
(gdb) run
Starting program:
/future/build/woods/future/current-amd64-i386-ppro-obj/more/work/woods/m-NetBSD-current/usr.bin/hexdump/hexdump
warning: shared library handler failed to enable breakpoint
hexdump: ""%07.7_ax " 8/2 "%04x " "\n"": bad format
Program exited with code 01.
(gdb) break add
No symbol table is loaded. Use the "file" command.
(gdb)
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpZGfUOA5T1T.pgp
Description: OpenPGP Digital Signature
SOLVED (mostly) Re: odd behaviour of some programs on i386 cross-built from amd64
So, I can now report I've been a victim of my own aging eyes and
clumsiness. :-)
In summary the problem was due to accidentally typing an errant
character in a source file while browsing it (sometime back in
February), and worse yet I saved it without knowing I had done so, and
further having the bad luck for that character to not trigger any errors
during compilation.
The errant character was a tilde ('~'), and it landed at the beginning
of line #122 in src/lib/libc/gen/ctype_.c. Obviously, but unfortunately
for me, this did not generate a syntax error, but instead just changed
the value of one entry in the _ctype_tab_ table (the one for the space
character).
The long version of the story is that after all the previously mentioned
problems with debuggers, etc., I started debugging by inserting some
better error messages in usr.bin/hexdump/parse.c to see if I could
discover exactly what line the problem was occurring on, and sure enough
it seemed to be with the macros. Then I found I was able to
work around the problem by locally defining a naive version of isdigit()
(probably (I have not verified) this worked because the new value for
the space character in _ctype_tab_ was now identified as a digit, and my
naive replacement avoided this problem).
The final mystery is why the affected programs work when run with either
a newer kernel, or on amd64. Although I can reproduce the bug in
hexdump, I cannot seem to reproduce it exactly. I.e. if I reproduce the
bug by locally defining _ctype_tab_ et al with the errant value then
hexdump, when compiled for i386, exhibits the same problem on both i386
and amd64 with matching and newer kernels. I.e. the reproduced bug does
not disappear in the scenarios where it disappeared before. The old
buggy binary still only exhibits the bug only on a real i386 with a
matching kernel, and of course it still works OK on both amd64 with a
matching kernel and on a real i386 with a newer kernel. Keep in mind
this is a static-linked binary.
Here's the buggy version working fine on a real i386 with a newer kernel:
$ uname -a
NetBSD once.local 9.0 NetBSD 9.0 (GENERIC) #0: Fri Feb 14 00:06:28 UTC 2020
mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/i386/compile/GENERIC i386
$ /more/home/more/woods/tmp/hexdump-
asdf
000 7361 6664 000a
005
$ file /more/home/more/woods/tmp/hexdump-
/more/home/more/woods/tmp/hexdump-: ELF 32-bit LSB executable, Intel 80386,
version 1 (SYSV), statically linked, for NetBSD 8.99.32, stripped
Here's the buggy version working fine on amd64 (with a matching kernel):
$ uname -a
NetBSD future 8.99.32 NetBSD 8.99.32 (XEN3_DOMU) #1: Thu Nov 28 18:31:36 PST
2019
woods@future:/build/woods/future/current-amd64-amd64-obj/more/work/woods/m-NetBSD-current/sys/arch/amd64/compile/XEN3_DOMU
amd64
$ ~/tmp/hexdump-
asdf
000 7361 6664 000a
005
Here's the buggy version failing on a real i386 with a matching kernel:
$ uname -a
NetBSD lilbit 8.99.32 NetBSD 8.99.32 (NET5501) #3: Fri May 1 16:55:04 PDT 2020
woods@once.local:/build/woods/once.local/current-i386-i386-ppro-obj/more/work/woods/m-NetBSD-current/sys/arch/i386/compile/NET5501
i386
$ /more/home/more/woods/tmp/hexdump-
hexdump-: ""%07.7_ax " 8/2 "%04x " "\n"": bad format
I guess the most interesting test would be to step instruction by
instruction through the execution on the real i386 with a newer kernel
and see if I can understand how it manages to work. I don't think I
kept a copy of hexdump.debug though -- I may have to rebuild the whole
tree with the original error to make that less arduous to do. Oh well,
I guess it only takes about 4 hours on my speediest build machine.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpL_axIbcpDS.pgp
Description: OpenPGP Digital Signature
