Bug#982794: firefox-esr: illegal instruction in libxul.so on armhf
Control: tags -1 + patch
On Fri, Mar 03, 2023 at 03:13:15PM +0100, Emanuele Rocca wrote:
> Firefox seems to erroneously enable NEON in places where it shouldn't. Trying
> to figure out exactly where and what's the best way to address this.
Patch attached.
According to the large disclaimer in moz.build one should not touch that file
but instead modify generate_mozbuild.py. It seems to me however that changes to
the python script are not automatically applied to moz.build?
In any case with the attached patch firefox-esr builds properly for armhf and
works fine here emulating a cortex-r5f CPU.
Thanks,
ema
Index: firefox-esr-102.8.0esr/gfx/skia/moz.build
===
--- firefox-esr-102.8.0esr.orig/gfx/skia/moz.build
+++ firefox-esr-102.8.0esr/gfx/skia/moz.build
@@ -455,8 +455,6 @@ if CONFIG['INTEL_ARCHITECTURE']:
SOURCES['skia/src/opts/SkOpts_sse42.cpp'].flags += ['-msse4.2']
SOURCES['skia/src/opts/SkOpts_avx.cpp'].flags += ['-mavx']
SOURCES['skia/src/opts/SkOpts_hsw.cpp'].flags += ['-mavx2', '-mf16c', '-mfma']
-elif CONFIG['CPU_ARCH'] == 'arm' and CONFIG['CC_TYPE'] in ('clang', 'gcc'):
-CXXFLAGS += CONFIG['NEON_FLAGS']
elif CONFIG['CPU_ARCH'] == 'aarch64' and CONFIG['CC_TYPE'] in ('clang', 'gcc'):
SOURCES['skia/src/opts/SkOpts_crc32.cpp'].flags += ['-march=armv8-a+crc']
Bug#982794: firefox-esr: illegal instruction in libxul.so on armhf
Hi, On Sun, Feb 14, 2021 at 02:12:17PM +, Vincent Arkesteijn wrote: > Firefox is killed with SIGILL shortly after startup: > $ firefox-esr -safe-mode > Illegal instruction > $ This is due to the fact that some armhf CPUs do not have support for NEON instructions. skia used to detect such support at runtime, but that behavior was removed in https://github.com/google/skia/commit/809ccf37ec836d0df64afd0b13023fd968d505a4 Firefox seems to erroneously enable NEON in places where it shouldn't. Trying to figure out exactly where and what's the best way to address this. Meanwhile, to reproduce and debug this issue on a amd64 machine: apt install debootstrap qemu-user-static binfmt-support schroot Trying to run a armhf binary on the x86 host will invoke qemu-arm-static, see: /usr/sbin/update-binfmts --display qemu-arm ls -l /usr/libexec/qemu-binfmt/arm-binfmt-P Create a armhf chroot: debootstrap --arch=armhf sid /srv/sid-armhf printf '[armhf]\ntype=directory\ndirectory=/srv/sid-armhf\n' >> /etc/schroot/schroot.conf Install and run firefox-esr in the chroot: schroot -c armhf (armhf)root@ariel:/home/ema# apt install --no-install-recommends firefox-esr Firefox seems to be working: (armhf)root@ariel:/home/ema# firefox --help | head -1 Usage: firefox-esr [ options ... ] [URL] The reason why firefox does not crash here is that the default armhf CPU emulated by qemu-arm-static has NEON support. We can override that by setting QEMU_CPU to cortex-r5f (which cannot execute NEON instructions) and reproduce the issue: (armhf)root@ariel:/home/ema# QEMU_CPU=cortex-r5f firefox qemu: uncaught target signal 4 (Illegal instruction) - core dumped Illegal instruction To get a backtrace, install Firefox's debugging symbols in the chroot: (armhf)root@ariel:/home/ema# echo 'deb http://deb.debian.org/debian-debug sid-debug main' > /etc/apt/sources.list.d/debug.list (armhf)root@ariel:/home/ema# apt update && apt install firefox-esr-dbgsym And do the following on the x86 host: dpkg --add-architecture armhf apt install libc6:armhf libc6-dbg:armhf gdb-multiarch LD_LIBRARY_PATH=/srv/sid-armhf/usr/lib/arm-linux-gnueabihf qemu-arm-static -g 1234 -cpu cortex-r5f /srv/sid-armhf/usr/bin/firefox-esr --private-window In another terminal, again on the host, this should give you a backtrace: gdb-multiarch -q /srv/sid-armhf/usr/bin/firefox-esr -ex 'set architecture arm' -ex 'target remote :1234' -ex 'set debug-file-directory /srv/sid-armhf/usr/lib/debug' -ex 'set pagination off' -ex 'continue' -ex 'bt' -ex 'continue' -ex 'exit' Something like: Program received signal SIGILL, Illegal instruction. 0x37071dc6 in _GLOBAL__sub_I_SkOpts.cpp () from /srv/sid-armhf/usr/lib/firefox-esr/libxul.so #0 0x37071dc6 in _GLOBAL__sub_I_SkOpts.cpp () from /srv/sid-armhf/usr/lib/firefox-esr/libxul.so #1 0x3f7d144c in call_init (env=0x3f208340, argv=0x3c94, argc=1, l=) at dl-init.c:70 #2 call_init (l=, argc=1, argv=0x3c94, env=0x3f208340) at dl-init.c:26 #3 0x3f7d14f2 in _dl_init (main_map=0x3f245f00, argc=1, argv=0x3c94, env=0x3f208340) at dl-init.c:117 #4 0x3f56664a in _dl_catch_exception () from /srv/sid-armhf/usr/lib/arm-linux-gnueabihf/libc.so.6 #5 0x3f7d5b60 in dl_open_worker (a=0x3fffd7b0) at dl-open.c:808 #6 0x3f566614 in _dl_catch_exception () from /srv/sid-armhf/usr/lib/arm-linux-gnueabihf/libc.so.6 #7 0x3f7d5da2 in _dl_open (file=0x3fffda64 "/srv/sid-armhf/usr/lib/firefox-esr/libxul.so", mode=-2147483391, caller_dlopen=0x4000af81 <_start+4560>, nsid=-2, argc=1, argv=0x3c94, env=0x3f208340) at dl-open.c:884 #8 0x3f4d9da0 in ?? () from /srv/sid-armhf/usr/lib/arm-linux-gnueabihf/libc.so.6
Bug#982794: firefox-esr: illegal instruction in libxul.so on armhf
Control: found -1 78.11.0esr-1 Hi Hideki and Jochen, Thank you for both of your responses. On Thu, Jul 01, 2021 at 08:08:44AM +0200, Jochen Sprickerhof wrote: > * Hideki Yamane [2021-06-28 22:35]: > > Can you reproduce it on freshly installed bullseye sytem? After apt upgrade (firefox now at 78.11.0esr-1), the issue is still there. The offending instruction is the same and the backtrace looks very similar. Given that the cause seems well understood (use of NEON instructions on a non-NEON system), I don't think a fresh install would give us any new information. > I only found this reference for NEON on armhf: > > "NEON and VFP/VFP2/VFP3 remain an optional part of the architecture." > > https://wiki.debian.org/ArmHardFloatPort#VFP In addition: "VFPv3-D16 is the common denominator of the processors to support here (therefore the recommended build option is -mfpu=vfpv3-d16)" https://wiki.debian.org/ArmHardFloatPort/VfpComparison#FPU I couldn't find a more authoritative definition of the supported architecture subset for the armhf port. > If this is still reproducible, I see two options: > - Disable NEON code. > - Depend on the neon-support dummy package. Agreed. > > > Kernel: Linux 3.5.7-14-ARCH (PREEMPT) > > It seems that is not the kernel bullseye provides. Correct. The default Debian armhf kernel doesn't give me video, and I forgot whether it even boots. > > And it maybe help to provide its hardware information, too. > The bug author wrote: > > > > This is on a Marvell Dove system, with VFPv3-D16. From /proc/cpuinfo: > > > Features : swp half thumb fastmult vfp edsp iwmmxt thumbee vfpv3 > > > vfpv3d16 tls More specifically, this is on a SolidRun CuBox (first generation, so not the CuBox-i or CuBox-M). I noticed that some time ago, the severity of this bug was raised from normal to serious. While it is serious on my system, I had set it to normal because it likely affects only a relatively small number of systems. And while I would appreciate this bug getting resolved, making it release critical seems unnecessary. Regards, Vincent.
Bug#982794: firefox-esr: illegal instruction in libxul.so on armhf
(somehow my message didn't made it to the BTS, sorry for double post.) Hi Hideki and Vincent, just some information from a bystander. Also Ccing the bug author for more information. @Vincent: would be great to get your answers as well. * Hideki Yamane [2021-06-28 22:35]: On Sun, 14 Feb 2021 14:12:17 + Vincent Arkesteijn wrote: Firefox is killed with SIGILL shortly after startup: $ firefox-esr -safe-mode Illegal instruction Can you reproduce it on freshly installed bullseye sytem? I guess this is still possible given: The referenced (NEON?) register Q8 is not available here, nor even the VFPv3-D32 registers D16-D17 that it maps to. I only found this reference for NEON on armhf: "NEON and VFP/VFP2/VFP3 remain an optional part of the architecture." https://wiki.debian.org/ArmHardFloatPort#VFP There are strings like -DBUILD_ARM_NEON=1 and -DHAVE_ARM_NEON=1 in the buildd logs here: https://buildd.debian.org/status/fetch.php?pkg=firefox-esr=armhf=78.11.0esr-1=1622606659=0 (I didn't look what they actually do.) If this is still reproducible, I see two options: - Disable NEON code. - Depend on the neon-support dummy package. -- System Information: Debian Release: bullseye/sid APT prefers testing APT policy: (500, 'testing') Architecture: armhf (armv7l) However, Kernel: Linux 3.5.7-14-ARCH (PREEMPT) It seems that is not the kernel bullseye provides. And it maybe help to provide its hardware information, too. The bug author wrote: This is on a Marvell Dove system, with VFPv3-D16. From /proc/cpuinfo: Features: swp half thumb fastmult vfp edsp iwmmxt thumbee vfpv3 vfpv3d16 tls Cheers Jochen signature.asc Description: PGP signature
Bug#982794: firefox-esr: illegal instruction in libxul.so on armhf
Hi Hideki and Vincent, just some information from a bystander. Also Ccing the bug author for more information. @Vincent: would be great to get your answers as well. * Hideki Yamane [2021-06-28 22:35]: On Sun, 14 Feb 2021 14:12:17 + Vincent Arkesteijn wrote: Firefox is killed with SIGILL shortly after startup: $ firefox-esr -safe-mode Illegal instruction Can you reproduce it on freshly installed bullseye sytem? I guess this is still possible given: The referenced (NEON?) register Q8 is not available here, nor even the VFPv3-D32 registers D16-D17 that it maps to. I only found this reference for NEON on armhf: "NEON and VFP/VFP2/VFP3 remain an optional part of the architecture." https://wiki.debian.org/ArmHardFloatPort#VFP There are strings like -DBUILD_ARM_NEON=1 and -DHAVE_ARM_NEON=1 in the buildd logs here: https://buildd.debian.org/status/fetch.php?pkg=firefox-esr=armhf=78.11.0esr-1=1622606659=0 (I didn't look what they actually do.) If this is still reproducible, I see two options: - Disable NEON code. - Depend on the neon-support dummy package. -- System Information: Debian Release: bullseye/sid APT prefers testing APT policy: (500, 'testing') Architecture: armhf (armv7l) However, Kernel: Linux 3.5.7-14-ARCH (PREEMPT) It seems that is not the kernel bullseye provides. And it maybe help to provide its hardware information, too. The bug author wrote: This is on a Marvell Dove system, with VFPv3-D16. From /proc/cpuinfo: Features: swp half thumb fastmult vfp edsp iwmmxt thumbee vfpv3 vfpv3d16 tls Cheers Jochen signature.asc Description: PGP signature
Bug#982794: firefox-esr: illegal instruction in libxul.so on armhf
Package: firefox-esr Version: 78.7.0esr-1 Severity: normal Dear Maintainer, Firefox is killed with SIGILL shortly after startup: $ firefox-esr -safe-mode Illegal instruction $ A gdb session on a dumped core to investigate: [...] Core was generated by `firefox-esr'. Program terminated with signal SIGILL, Illegal instruction. #0 0xaf6f0ab0 in ?? () from /usr/lib/firefox-esr/libxul.so (gdb) backtrace #0 0xaf6f0ab0 in ?? () from /usr/lib/firefox-esr/libxul.so #1 0xb6f32f40 in call_init (l=, argc=argc@entry=1, argv=argv@entry=0xbeb44584, env=env@entry=0xb6a0a240) at dl-init.c:72 #2 0xb6f32fe2 in call_init (env=, argv=, argc=, l=) at dl-init.c:30 #3 _dl_init (main_map=0xb6a30c00, argc=1, argv=0xbeb44584, env=0xb6a0a240) at dl-init.c:119 #4 0xb6cec52e in __GI__dl_catch_exception (exception=exception@entry=0x0, operate=0xb6f352c1 , args=0xbeb41ed8, args@entry=0xbeb41f10) at dl-error-skeleton.c:182 #5 0xb6f35d04 in dl_open_worker (a=) at dl-open.c:758 #6 0xb6cec4f4 in __GI__dl_catch_exception (exception=exception@entry=0xbeb420d8, operate=0xb6f35869 , args=args@entry=0xbeb420e4) at dl-error-skeleton.c:208 #7 0xb6f355cc in _dl_open (file=0xbeb4237c "/usr/lib/firefox-esr/libxul.so", mode=-2147483391, caller_dlopen=0xb6f5df85 <_start+2424>, nsid=-2, argc=1, argv=0xbeb44584, env=0xb6a0a240) at dl-open.c:837 #8 0xb6eeed18 in dlopen_doit (a=0xbeb42344) at dlopen.c:66 #9 0xb6cec4f4 in __GI__dl_catch_exception (exception=exception@entry=0xbeb42300, operate=0xb6eeecc1 , args=args@entry=0xbeb42344) at dl-error-skeleton.c:208 #10 0xb6cec588 in __GI__dl_catch_error (objname=objname@entry=0xb6a0d2ec, errstring=errstring@entry=0xb6a0d2f0, mallocedp=mallocedp@entry=0xb6a0d2e8, operate=, args=args@entry=0xbeb42344) at dl-error-skeleton.c:227 #11 0xb6eef3de in _dlerror_run (operate=, args=args@entry=0xbeb42344) at dlerror.c:170 #12 0xb6eeed9e in __dlopen (file=0xbeb4237c "/usr/lib/firefox-esr/libxul.so", mode=) at dlopen.c:87 #13 0xb6f5df84 in _start () (gdb) layout asm 0xaf6f0ab0 vmov.i32q8, #0 ; 0x This is on a Marvell Dove system, with VFPv3-D16. From /proc/cpuinfo: Features: swp half thumb fastmult vfp edsp iwmmxt thumbee vfpv3 vfpv3d16 tls The referenced (NEON?) register Q8 is not available here, nor even the VFPv3-D32 registers D16-D17 that it maps to. -- Package-specific info: -- Addons package information -- System Information: Debian Release: bullseye/sid APT prefers testing APT policy: (500, 'testing') Architecture: armhf (armv7l) Kernel: Linux 3.5.7-14-ARCH (PREEMPT) Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) Versions of packages firefox-esr depends on: ii debianutils 4.11.2 ii fontconfig 2.13.1-4.2 ii libatk1.0-0 2.36.0-2 ii libc62.31-9 ii libcairo-gobject21.16.0-5 ii libcairo21.16.0-5 ii libdbus-1-3 1.12.20-1 ii libdbus-glib-1-2 0.110-6 ii libevent-2.1-7 2.1.12-stable-1 ii libffi7 3.3-5 ii libfontconfig1 2.13.1-4.2 ii libfreetype6 2.10.4+dfsg-1 ii libgcc-s110.2.1-6 ii libgdk-pixbuf-2.0-0 2.42.2+dfsg-1 ii libglib2.0-0 2.66.6-2 ii libgtk-3-0 3.24.24-1 ii libnspr4 2:4.29-1 ii libnss3 2:3.60-1 ii libpango-1.0-0 1.46.2-3 ii libstdc++6 10.2.1-6 ii libvpx6 1.9.0-1 ii libx11-6 2:1.7.0-2 ii libx11-xcb1 2:1.7.0-2 ii libxcb-shm0 1.14-3 ii libxcb1 1.14-3 ii libxcomposite1 1:0.4.5-1 ii libxdamage1 1:1.1.5-2 ii libxext6 2:1.3.3-1.1 ii libxfixes3 1:5.0.3-2 ii libxrender1 1:0.9.10-1 ii procps 2:3.3.16-5 ii zlib1g 1:1.2.11.dfsg-2 Versions of packages firefox-esr recommends: ii libavcodec58 7:4.3.1-8 Versions of packages firefox-esr suggests: pn fonts-lmodern pn fonts-stix | otf-stix pn libcanberra0 ii libgssapi-krb5-2 1.18.3-4 pn libgtk2.0-0 pn pulseaudio -- no debconf information
