Re: Regression in backport MEMREAD ioctl ? [Was: Re: mt7622: belkin-rt3200: r22602-42eeb22450: Kernel panic: kernel stack overflow]
On 4/21/23 15:17, Michał Kępień wrote: Hi Petr, Since the crash happens right after snand driver initialization, I think the most likely candidate is this one: fa4dc86e9808 kernel: backport MEMREAD ioctl Maybe there are still some stack declarations of struct mtd_oob_ops left that aren't fully initialized. thanks for looking into that Felix, Michał any idea what might be wrong here? I remember looking for uninitialized fields in all existing instances of struct mtd_oob_ops in version 5.15.98 of the Linux kernel source tree while preparing the MEMREAD backports. However, it did not occur to me to check OpenWRT-specific patches in the same way (sorry!) - and a naïve search uncovers these two locations: $ git grep -E 'struct mtd_oob_ops [^=*{}]+;' -- ':!target/linux/generic/backport-5.15/' package/boot/uboot-mediatek/patches/100-07-mtd-nmbm-add-support-for-mtd.patch:+ struct mtd_oob_ops ops; package/boot/uboot-mediatek/patches/100-07-mtd-nmbm-add-support-for-mtd.patch:+ struct mtd_oob_ops ops; package/boot/uboot-mediatek/patches/100-11-env-add-support-for-NMBM-upper-MTD-layer.patch:+ struct mtd_oob_ops ops; These patches are applied to U-Boot and not the kernel. The "fa4dc86e9808 kernel: backport MEMREAD ioctl" change only changes he kernel. Since the panic message includes mentions of a stack overflow, another idea would be to backport this upstream patch as well: https://lore.kernel.org/linux-mtd/20230417205654.1982368-1-a...@kernel.org/ This patch has been reviewed, but it has not yet been merged anywhere. Please send a patch to the openwrt mailing list or create a pull request on github. hauke ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Regression in backport MEMREAD ioctl ? [Was: Re: mt7622: belkin-rt3200: r22602-42eeb22450: Kernel panic: kernel stack overflow]
The sender domain has a DMARC Reject/Quarantine policy which disallows sending mailing list messages using the original "From" header. To mitigate this problem, the original message has been wrapped automatically by the mailing list software.--- Begin Message --- Hi Petr, > > Since the crash happens right after snand driver initialization, I think the > > most likely candidate is this one: > > fa4dc86e9808 kernel: backport MEMREAD ioctl > > > > Maybe there are still some stack declarations of struct mtd_oob_ops left > > that aren't fully initialized. > > thanks for looking into that Felix, Michał any idea what might be wrong here? I remember looking for uninitialized fields in all existing instances of struct mtd_oob_ops in version 5.15.98 of the Linux kernel source tree while preparing the MEMREAD backports. However, it did not occur to me to check OpenWRT-specific patches in the same way (sorry!) - and a naïve search uncovers these two locations: $ git grep -E 'struct mtd_oob_ops [^=*{}]+;' -- ':!target/linux/generic/backport-5.15/' package/boot/uboot-mediatek/patches/100-07-mtd-nmbm-add-support-for-mtd.patch:+ struct mtd_oob_ops ops; package/boot/uboot-mediatek/patches/100-07-mtd-nmbm-add-support-for-mtd.patch:+ struct mtd_oob_ops ops; package/boot/uboot-mediatek/patches/100-11-env-add-support-for-NMBM-upper-MTD-layer.patch:+ struct mtd_oob_ops ops; Both structures in the first patch are zeroed out using memset() after they are declared, so that's fine, but the one in the second patch isn't. Given that MediaTek hardware is involved here, this sounds like a solid lead. Updating 100-11-env-add-support-for-NMBM-upper-MTD-layer.patch so that the line quoted above says this instead: struct mtd_oob_ops ops = {}; would be my first suggestion. Since the panic message includes mentions of a stack overflow, another idea would be to backport this upstream patch as well: https://lore.kernel.org/linux-mtd/20230417205654.1982368-1-a...@kernel.org/ This patch has been reviewed, but it has not yet been merged anywhere. Hope this helps, -- Best regards, Michał Kępień --- End Message --- ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Regression in backport MEMREAD ioctl ? [Was: Re: mt7622: belkin-rt3200: r22602-42eeb22450: Kernel panic: kernel stack overflow]
Felix Fietkau [2023-04-21 12:03:23]: [ adding Michał and Christian to the mail loop] > On 21.04.23 09:11, Petr Štetiar wrote: > > Hi, > > > > I've just noticed, that daily CI runtime testing job on belkin-rt3200 > > failed[1] due to following: > > > > Insufficient stack space to handle exception! > > ESR: 0x9647 -- DABT (current EL) > > FAR: 0xffc008c47fe0 > > Task stack: [0xffc008c48000..0xffc008c4c000] > > IRQ stack: [0xffc008008000..0xffc00800c000] > > Overflow stack: [0xff801feb00a0..0xff801feb10a0] > > CPU: 1 PID: 1 Comm: swapper/0 Tainted: G S5.15.107 #0 > > Hardware name: Linksys E8450 (DT) > > pstate: 80c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) > > pc : dequeue_entity+0x0/0x250 > > lr : dequeue_task_fair+0x98/0x290 > > sp : ffc008c48030 > > x29: ffc008c48030 x28: 0001 x27: ff801feb6380 > > x26: 0001 x25: ff801feb6300 x24: ff868000 > > x23: 0001 x22: 0009 x21: > > x20: ff801feb6380 x19: ff868080 x18: 17a740a6 > > x17: ffc008bae748 x16: ffc008bae6d8 x15: > > x14: x13: x12: 000f0101 > > x11: 0449 x10: 0127 x9 : > > x8 : 0125 x7 : 00116da1 x6 : 00116da1 > > x5 : 001165a1 x4 : ff801feb6e00 x3 : > > x2 : 0009 x1 : ff868080 x0 : ff801feb6380 > > Kernel panic - not syncing: kernel stack overflow > > SMP: stopping secondary CPUs > > SMP: failed to stop secondary CPUs 0-1 > > Kernel Offset: disabled > > CPU features: 0x3000,0802 > > Memory Limit: none > > > > Last working version was r22580-e11d00d44c[2], and first failing version was > > yesterday 1416b9bbe9, so possibly the regression was introduced in one of > > the > > following commits: > > > > 1416b9bbe9d3 tools/dwarves: update to 1.25 > > 9931188edcbc kernel: fix up qrtr packaging after 5.15.107 bump > > f4989239cc91 kernel: bump 5.15 to 5.15.107 > > 89f6ac5fd1ad tools/cmake: update to 3.26.3 > > ab3f151aa874 mwlwifi: update to version 10.3.9.0-20230311 > > 5ec781c4448b bmips: pci-bcm6348: load IO resource from DT ranges > > 16b0cbbde057 bmips: drop unneeded ath9k fixup > > db4f158c0330 bmips: hg556a: switch to kmod-owl-loader > > 36150ff6ffb2 tools/bzip2: add `bzip2` binaries > > b691362d1dbe Revert "tools/bzip2: add `bzip2` binaries" > > f7f47b136991 mac80211: ath11k: replace 160MHz fix with upstream pending > > one > > 4ab4b9ea818d build: fix incorrect initramfs gzip compression > > 69bc620180d2 build: fix incorrect initramfs bzip2 compression > > 394d7134ec42 tools/bzip2: add `bzip2` binaries > > 5264296ce480 ath79: mikrotik: update kernel on NAND using Yafut > > 27acf2413e91 yafut: add a kernel update tool for MikroTik NAND > > fa4dc86e9808 kernel: backport MEMREAD ioctl > > e722b667c5a5 mac80211: update to v6.1.24 > > Since the crash happens right after snand driver initialization, I think the > most likely candidate is this one: > fa4dc86e9808 kernel: backport MEMREAD ioctl > > Maybe there are still some stack declarations of struct mtd_oob_ops left > that aren't fully initialized. thanks for looking into that Felix, Michał any idea what might be wrong here? Cheers, Petr ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: mt7622: belkin-rt3200: r22602-42eeb22450: Kernel panic: kernel stack overflow
On 21.04.23 09:11, Petr Štetiar wrote: Hi, I've just noticed, that daily CI runtime testing job on belkin-rt3200 failed[1] due to following: Insufficient stack space to handle exception! ESR: 0x9647 -- DABT (current EL) FAR: 0xffc008c47fe0 Task stack: [0xffc008c48000..0xffc008c4c000] IRQ stack: [0xffc008008000..0xffc00800c000] Overflow stack: [0xff801feb00a0..0xff801feb10a0] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G S5.15.107 #0 Hardware name: Linksys E8450 (DT) pstate: 80c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : dequeue_entity+0x0/0x250 lr : dequeue_task_fair+0x98/0x290 sp : ffc008c48030 x29: ffc008c48030 x28: 0001 x27: ff801feb6380 x26: 0001 x25: ff801feb6300 x24: ff868000 x23: 0001 x22: 0009 x21: x20: ff801feb6380 x19: ff868080 x18: 17a740a6 x17: ffc008bae748 x16: ffc008bae6d8 x15: x14: x13: x12: 000f0101 x11: 0449 x10: 0127 x9 : x8 : 0125 x7 : 00116da1 x6 : 00116da1 x5 : 001165a1 x4 : ff801feb6e00 x3 : x2 : 0009 x1 : ff868080 x0 : ff801feb6380 Kernel panic - not syncing: kernel stack overflow SMP: stopping secondary CPUs SMP: failed to stop secondary CPUs 0-1 Kernel Offset: disabled CPU features: 0x3000,0802 Memory Limit: none Last working version was r22580-e11d00d44c[2], and first failing version was yesterday 1416b9bbe9, so possibly the regression was introduced in one of the following commits: 1416b9bbe9d3 tools/dwarves: update to 1.25 9931188edcbc kernel: fix up qrtr packaging after 5.15.107 bump f4989239cc91 kernel: bump 5.15 to 5.15.107 89f6ac5fd1ad tools/cmake: update to 3.26.3 ab3f151aa874 mwlwifi: update to version 10.3.9.0-20230311 5ec781c4448b bmips: pci-bcm6348: load IO resource from DT ranges 16b0cbbde057 bmips: drop unneeded ath9k fixup db4f158c0330 bmips: hg556a: switch to kmod-owl-loader 36150ff6ffb2 tools/bzip2: add `bzip2` binaries b691362d1dbe Revert "tools/bzip2: add `bzip2` binaries" f7f47b136991 mac80211: ath11k: replace 160MHz fix with upstream pending one 4ab4b9ea818d build: fix incorrect initramfs gzip compression 69bc620180d2 build: fix incorrect initramfs bzip2 compression 394d7134ec42 tools/bzip2: add `bzip2` binaries 5264296ce480 ath79: mikrotik: update kernel on NAND using Yafut 27acf2413e91 yafut: add a kernel update tool for MikroTik NAND fa4dc86e9808 kernel: backport MEMREAD ioctl e722b667c5a5 mac80211: update to v6.1.24 Since the crash happens right after snand driver initialization, I think the most likely candidate is this one: fa4dc86e9808 kernel: backport MEMREAD ioctl Maybe there are still some stack declarations of struct mtd_oob_ops left that aren't fully initialized. - Felix ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
[PATCH v2 5/5] tfa-layerscape: Change to github and use the latest tag
The sender domain has a DMARC Reject/Quarantine policy which disallows sending mailing list messages using the original "From" header. To mitigate this problem, the original message has been wrapped automatically by the mailing list software.--- Begin Message --- The dafault location of tfa-layerscape has been changed from codeuaurora to github. Also use the latest tag for Layerscape Linux Development POC from NXP. v2: * restored ls1012afrdm board * added platfrom defines to fiptool so ls-ddr-phy can be built Signed-off-by: Wojciech Dubowik --- package/boot/tfa-layerscape/Makefile | 15 +- .../patches/001-fiptool-hostbuild-fixes.patch | 37 ++- ...ix-create_pbl-and-byte_swap-host-bui.patch | 16 +- ...plat-nxp-restore-ls1012afrdm-support.patch | 232 ++ 4 files changed, 264 insertions(+), 36 deletions(-) create mode 100644 package/boot/tfa-layerscape/patches/004-plat-nxp-restore-ls1012afrdm-support.patch diff --git a/package/boot/tfa-layerscape/Makefile b/package/boot/tfa-layerscape/Makefile index 072036da2f..d9d9242941 100644 --- a/package/boot/tfa-layerscape/Makefile +++ b/package/boot/tfa-layerscape/Makefile @@ -8,13 +8,13 @@ include $(TOPDIR)/rules.mk PKG_NAME:=tfa-layerscape -PKG_VERSION:=21.08 +PKG_VERSION:=lf-6.1.1-1.0.0 PKG_RELEASE:=$(AUTORELEASE) PKG_SOURCE_PROTO:=git -PKG_SOURCE_URL:=https://source.codeaurora.org/external/qoriq/qoriq-components/atf -PKG_SOURCE_VERSION:=LSDK-21.08 -PKG_MIRROR_HASH:=893f2d28a77dcc9d4413a619b4719ca5f1f4dc78dd824a8488e7d543e66bcf95 +PKG_SOURCE_URL:=https://github.com/nxp-qoriq/atf +PKG_SOURCE_VERSION:=lf-6.1.1-1.0.0 +PKG_MIRROR_HASH:=c0e3c7d0cc9ee5f9ae68cd705ce7f07f714c4fe3fd4a5a79de5699e7865b759f PKG_BUILD_DEPENDS:=tfa-layerscape/host include $(INCLUDE_DIR)/host-build.mk @@ -25,18 +25,19 @@ HOST_CFLAGS += -Wall -Werror -pedantic -std=c99 define Host/Compile $(MAKE) -C \ $(HOST_BUILD_DIR)/tools/fiptool \ + PLAT_FIPTOOL_HELPER_MK="$(HOST_BUILD_DIR)/tools/nxp/plat_fiptool/plat_fiptool.mk" CFLAGS="$(HOST_CFLAGS)" \ LDFLAGS="$(HOST_LDFLAGS)" \ HOSTCCFLAGS="$(HOST_CFLAGS)" $(MAKE) -C \ - $(HOST_BUILD_DIR)/tools/nxp \ + $(HOST_BUILD_DIR)/tools/nxp/create_pbl \ CFLAGS="$(HOST_CFLAGS)" endef define Host/Install $(INSTALL_BIN) $(HOST_BUILD_DIR)/tools/fiptool/fiptool $(STAGING_DIR_HOST)/bin/fiptool-layerscape - $(INSTALL_BIN) $(HOST_BUILD_DIR)/tools/nxp/create_pbl $(STAGING_DIR_HOST)/bin/tfa-create-pbl - $(INSTALL_BIN) $(HOST_BUILD_DIR)/tools/nxp/byte_swap $(STAGING_DIR_HOST)/bin/tfa-byte-swap + $(INSTALL_BIN) $(HOST_BUILD_DIR)/tools/nxp/create_pbl/create_pbl $(STAGING_DIR_HOST)/bin/tfa-create-pbl + $(INSTALL_BIN) $(HOST_BUILD_DIR)/tools/nxp/create_pbl/byte_swap $(STAGING_DIR_HOST)/bin/tfa-byte-swap endef define Trusted-Firmware-A/Default diff --git a/package/boot/tfa-layerscape/patches/001-fiptool-hostbuild-fixes.patch b/package/boot/tfa-layerscape/patches/001-fiptool-hostbuild-fixes.patch index 4b74048754..8e7375cfa2 100644 --- a/package/boot/tfa-layerscape/patches/001-fiptool-hostbuild-fixes.patch +++ b/package/boot/tfa-layerscape/patches/001-fiptool-hostbuild-fixes.patch @@ -1,6 +1,6 @@ --- a/Makefile +++ b/Makefile -@@ -801,10 +801,6 @@ CRTTOOL ?= ${CRTTOOLPATH}/cert_create$ +@@ -914,10 +914,6 @@ CRTTOOL ?= ${CRTTOOLPATH}/cert_create$ ENCTOOLPATH ?= tools/encrypt_fw ENCTOOL ?= ${ENCTOOLPATH}/encrypt_fw${BIN_EXT} @@ -11,7 +11,7 @@ # Variables for use with sptool SPTOOLPATH?= tools/sptool SPTOOL?= ${SPTOOLPATH}/sptool${BIN_EXT} -@@ -1160,13 +1156,6 @@ endif +@@ -1322,13 +1318,6 @@ endif clean: @echo " CLEAN" $(call SHELL_REMOVE_DIR,${BUILD_PLAT}) @@ -25,7 +25,7 @@ ${Q}${MAKE} PLAT=${PLAT} --no-print-directory -C ${CRTTOOLPATH} clean ${Q}${MAKE} PLAT=${PLAT} --no-print-directory -C ${ENCTOOLPATH} clean ${Q}${MAKE} --no-print-directory -C ${ROMLIBPATH} clean -@@ -1175,13 +1164,6 @@ realclean distclean: +@@ -1337,13 +1326,6 @@ realclean distclean: @echo " REALCLEAN" $(call SHELL_REMOVE_DIR,${BUILD_BASE}) $(call SHELL_DELETE_ALL, ${CURDIR}/cscope.*) @@ -39,37 +39,33 @@ ${Q}${MAKE} --no-print-directory -C ${SPTOOLPATH} clean ${Q}${MAKE} PLAT=${PLAT} --no-print-directory -C ${CRTTOOLPATH} clean ${Q}${MAKE} PLAT=${PLAT} --no-print-directory -C ${ENCTOOLPATH} realclean -@@ -1238,7 +1220,7 @@ certificates: ${CRT_DEPS} ${CRTTOOL} +@@ -1400,7 +1382,7 @@ certificates: ${CRT_DEPS} ${CRTTOOL} @${ECHO_BLANK_LINE} endif -${BUILD_PLAT}/${FIP_NAME}: ${FIP_DEPS} ${FIPTOOL} +${BUILD_PLAT}/${FIP_NAME}: ${FIP_DEPS} - ${Q}${FIPTOOL} create ${FIP_ARGS} $@ - ${Q}${FIPTOOL} info $@ - @${ECHO_BLANK_LINE}
Re: mt7622: belkin-rt3200: r22602-42eeb22450: Kernel panic: kernel stack overflow
On Fri, Apr 21, 2023 at 09:11:37AM +0200, Petr Štetiar wrote: > Hi, > > I've just noticed, that daily CI runtime testing job on belkin-rt3200 > failed[1] due to following: > > Insufficient stack space to handle exception! > ESR: 0x9647 -- DABT (current EL) > FAR: 0xffc008c47fe0 > Task stack: [0xffc008c48000..0xffc008c4c000] > IRQ stack: [0xffc008008000..0xffc00800c000] > Overflow stack: [0xff801feb00a0..0xff801feb10a0] > CPU: 1 PID: 1 Comm: swapper/0 Tainted: G S5.15.107 #0 > Hardware name: Linksys E8450 (DT) > pstate: 80c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) > pc : dequeue_entity+0x0/0x250 > lr : dequeue_task_fair+0x98/0x290 > sp : ffc008c48030 > x29: ffc008c48030 x28: 0001 x27: ff801feb6380 > x26: 0001 x25: ff801feb6300 x24: ff868000 > x23: 0001 x22: 0009 x21: > x20: ff801feb6380 x19: ff868080 x18: 17a740a6 > x17: ffc008bae748 x16: ffc008bae6d8 x15: > x14: x13: x12: 000f0101 > x11: 0449 x10: 0127 x9 : > x8 : 0125 x7 : 00116da1 x6 : 00116da1 > x5 : 001165a1 x4 : ff801feb6e00 x3 : > x2 : 0009 x1 : ff868080 x0 : ff801feb6380 > Kernel panic - not syncing: kernel stack overflow > SMP: stopping secondary CPUs > SMP: failed to stop secondary CPUs 0-1 > Kernel Offset: disabled > CPU features: 0x3000,0802 > Memory Limit: none > > Last working version was r22580-e11d00d44c[2], and first failing version was > yesterday 1416b9bbe9, so possibly the regression was introduced in one of the > following commits: > > 1416b9bbe9d3 tools/dwarves: update to 1.25 > 9931188edcbc kernel: fix up qrtr packaging after 5.15.107 bump > f4989239cc91 kernel: bump 5.15 to 5.15.107 Hi!! Thank you a lot for your effort on CI testing the RT3200 and devices in general. I would of course choose this one (the kernel bump), also due to the fact this happens straight at boot. I am not able to do more inference here - from commit message we know the problem doesn't happen on filogic/xiaomi_redmi-router-ax6000-ubootmod: this may help to narrow little bit down the problem. Thanks, Enrico > 89f6ac5fd1ad tools/cmake: update to 3.26.3 > ab3f151aa874 mwlwifi: update to version 10.3.9.0-20230311 > 5ec781c4448b bmips: pci-bcm6348: load IO resource from DT ranges > 16b0cbbde057 bmips: drop unneeded ath9k fixup > db4f158c0330 bmips: hg556a: switch to kmod-owl-loader > 36150ff6ffb2 tools/bzip2: add `bzip2` binaries > b691362d1dbe Revert "tools/bzip2: add `bzip2` binaries" > f7f47b136991 mac80211: ath11k: replace 160MHz fix with upstream pending one > 4ab4b9ea818d build: fix incorrect initramfs gzip compression > 69bc620180d2 build: fix incorrect initramfs bzip2 compression > 394d7134ec42 tools/bzip2: add `bzip2` binaries > 5264296ce480 ath79: mikrotik: update kernel on NAND using Yafut > 27acf2413e91 yafut: add a kernel update tool for MikroTik NAND > fa4dc86e9808 kernel: backport MEMREAD ioctl > e722b667c5a5 mac80211: update to v6.1.24 > > 1. > https://ynezz.gitlab.io/-/openwrt-device-runtime-testing/-/jobs/4153023579/artifacts/console_belkin-rt3200-initramfs.txt > 2. https://gitlab.com/ynezz/openwrt-device-runtime-testing/-/jobs/4137718871 > > Cheers, > > Petr > > ___ > openwrt-devel mailing list > openwrt-devel@lists.openwrt.org > https://lists.openwrt.org/mailman/listinfo/openwrt-devel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
mt7622: belkin-rt3200: r22602-42eeb22450: Kernel panic: kernel stack overflow
Hi, I've just noticed, that daily CI runtime testing job on belkin-rt3200 failed[1] due to following: Insufficient stack space to handle exception! ESR: 0x9647 -- DABT (current EL) FAR: 0xffc008c47fe0 Task stack: [0xffc008c48000..0xffc008c4c000] IRQ stack: [0xffc008008000..0xffc00800c000] Overflow stack: [0xff801feb00a0..0xff801feb10a0] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G S5.15.107 #0 Hardware name: Linksys E8450 (DT) pstate: 80c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : dequeue_entity+0x0/0x250 lr : dequeue_task_fair+0x98/0x290 sp : ffc008c48030 x29: ffc008c48030 x28: 0001 x27: ff801feb6380 x26: 0001 x25: ff801feb6300 x24: ff868000 x23: 0001 x22: 0009 x21: x20: ff801feb6380 x19: ff868080 x18: 17a740a6 x17: ffc008bae748 x16: ffc008bae6d8 x15: x14: x13: x12: 000f0101 x11: 0449 x10: 0127 x9 : x8 : 0125 x7 : 00116da1 x6 : 00116da1 x5 : 001165a1 x4 : ff801feb6e00 x3 : x2 : 0009 x1 : ff868080 x0 : ff801feb6380 Kernel panic - not syncing: kernel stack overflow SMP: stopping secondary CPUs SMP: failed to stop secondary CPUs 0-1 Kernel Offset: disabled CPU features: 0x3000,0802 Memory Limit: none Last working version was r22580-e11d00d44c[2], and first failing version was yesterday 1416b9bbe9, so possibly the regression was introduced in one of the following commits: 1416b9bbe9d3 tools/dwarves: update to 1.25 9931188edcbc kernel: fix up qrtr packaging after 5.15.107 bump f4989239cc91 kernel: bump 5.15 to 5.15.107 89f6ac5fd1ad tools/cmake: update to 3.26.3 ab3f151aa874 mwlwifi: update to version 10.3.9.0-20230311 5ec781c4448b bmips: pci-bcm6348: load IO resource from DT ranges 16b0cbbde057 bmips: drop unneeded ath9k fixup db4f158c0330 bmips: hg556a: switch to kmod-owl-loader 36150ff6ffb2 tools/bzip2: add `bzip2` binaries b691362d1dbe Revert "tools/bzip2: add `bzip2` binaries" f7f47b136991 mac80211: ath11k: replace 160MHz fix with upstream pending one 4ab4b9ea818d build: fix incorrect initramfs gzip compression 69bc620180d2 build: fix incorrect initramfs bzip2 compression 394d7134ec42 tools/bzip2: add `bzip2` binaries 5264296ce480 ath79: mikrotik: update kernel on NAND using Yafut 27acf2413e91 yafut: add a kernel update tool for MikroTik NAND fa4dc86e9808 kernel: backport MEMREAD ioctl e722b667c5a5 mac80211: update to v6.1.24 1. https://ynezz.gitlab.io/-/openwrt-device-runtime-testing/-/jobs/4153023579/artifacts/console_belkin-rt3200-initramfs.txt 2. https://gitlab.com/ynezz/openwrt-device-runtime-testing/-/jobs/4137718871 Cheers, Petr ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel