Re: [U-Boot][bug report] clearfog: EMMC boot broken on clearfog pro
Martin Rowe [2023-01-14 14:21:52]:
[adding Stefan and Pali to the Cc: loop to make them aware]
Hi,
> I did try to bisect the eMMC issue, but there are a few related
> breakages for different reasons over the last 2 years, so it's not one
> commit that has done it. For example, just the 4 changes I listed for
> kwbimage.c are introduced by the following 5 commits:
> 5c61710c9880290d54db72878c4435cdaee07d78
> 501a54a29cc20ce7df70f290fa274b8e2ea9d6f4
> e0c243c398a771df22fd3caf309b04eef3ba2683
> 700ea98b2e364a8107a9af962ba39f2eeadfc678
> aa6943ca3122db7ea0063684f94b5dbdf9c5cf51
Oli seems to have bisected[1] his SD card based boot breakage as well. Commit
e23ad5d55cbc ("tools: kwbimage: Do not put final image padding to the image
data size") boots just fine:
U-Boot SPL 2022.01-rc1-00092-ge23ad5d55c (Jan 17 2023 - 15:54:41 +0100)
whereas commit 5cad2e6ccee1 ("tools: kwbimage: Align kwbimage header to proper
size") leads to:
BootROM - 1.73
Booting from MMC
BootROM: Bad header at offset 0200
and stays in the broken state since then.
1. https://github.com/openwrt/openwrt/issues/11661#issuecomment-1385589140
Cheers,
Petr
Re: [U-Boot][bug report] clearfog: EMMC boot broken on clearfog pro
Hello again > So in order to move forward, the current plan is to find out/bisect first > commit after v2021.01 which has introduced this BootROM SD/eMMC/SATA boot > regression and inform involved folks (Author and Commiter) accordingly with a > proper bug report. I did try to bisect the eMMC issue, but there are a few related breakages for different reasons over the last 2 years, so it's not one commit that has done it. For example, just the 4 changes I listed for kwbimage.c are introduced by the following 5 commits: 5c61710c9880290d54db72878c4435cdaee07d78 501a54a29cc20ce7df70f290fa274b8e2ea9d6f4 e0c243c398a771df22fd3caf309b04eef3ba2683 700ea98b2e364a8107a9af962ba39f2eeadfc678 aa6943ca3122db7ea0063684f94b5dbdf9c5cf51 > Since you're able to boot from SATA, it would help to know if it's broken as > well. Having BootROM output with failing header offsets would be nice as well. SATA boot works for me up until around 2022.10-rc4. The following configs were required on top of the clearfog_defconfig: CONFIG_MVEBU_SPL_BOOT_DEVICE_SATA=y CONFIG_SPL_SATA=y CONFIG_SPL_FS_EXT4=y UART boot and SPI boot work, too. After the commit I tested (1977d72a69f3c8d97bd25a86a6be4da27cde3724) some mvebu timer changes got merged and __udelay gets stuck in an infinite loop while probing hardware. I'm still trying to pin that one down, but it seems entirely unrelated to BootROM finding invalid headers. In case you still need it, here's the BootROM output for SATA with no u-boot installed: BootROM - 1.73 Booting from AHCI Probing HBA- 0 port 0 SATA device found BootROM: Bad header at offset 0001 BootROM: Bad header at offset 0022 BootROM: Bad header at offset 1000 BootROM: Bad header at offset 2000 BootROM: Bad header at offset 3000 BootROM: Bad header at offset 4000 BootROM: Bad header at offset 5000 BootROM: Bad header at offset 6000 BootROM: Bad header at offset 7000 Also, I found a non-eMMC clearfog base to test SD card boot. I had assumed that SD card boot was working, therefore my patches would have broken it. It appears the inverse is true. I can boot from a SD card with the same changes I posted that fix eMMC. That means my changes address two different issues: 1. eMMC not being detected, which could be fixed with a CONFIG_CLEARFOG_EMMC kconfig and ifdef for the dts file, similar to the existing clearfog kconfigs 2. MMC boot (both eMMC and SD card) not working, which is fixed by removing the special handling that converts to/from sectors in the SDIO case and allowing a 1 sector offset for CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR How 2 gets implemented depends on what subset of devices MMC boot is broken for. If all devices are broken, my changes would fix it cleanly, otherwise the changes would need to be wrapped in some kind of ifdef which would mean the kwboot tool is no longer universal but must be compiled for specific boards. Let me know if there is any other data that would be useful. Regards Martin
Re: [U-Boot][bug report] clearfog: EMMC boot broken on clearfog pro
Martin Rowe [2023-01-10 06:01:08]: Hi, > > In other words, it would help to know, which commit after v2021.01 breaks > > booting of SD/SATA/eMMC for you. Could you help find it? > > The commit from the git bisect is one of dozens that are associated > with a major refactoring of kwboot and kwbimage; you probably want to > go back to the previous merge commit to get a working build. I don't own any such device. Florian has reported this regression 3 months ago and as you can see nothing has happened, it seems to be still broken. So in order to move forward, the current plan is to find out/bisect first commit after v2021.01 which has introduced this BootROM SD/eMMC/SATA boot regression and inform involved folks (Author and Commiter) accordingly with a proper bug report. > I didn't test SATA; I used a known working image to help isolate the > problem and have a stable base to test from. Since you're able to boot from SATA, it would help to know if it's broken as well. Having BootROM output with failing header offsets would be nice as well. Cheers, Petr
Re: [U-Boot][bug report] clearfog: EMMC boot broken on clearfog pro
On Mon, 9 Jan 2023 at 21:34, Petr Štetiar wrote:
>
> Martin Rowe [2023-01-03 03:20:49]:
>
> [Adding Florian, Tomasz and Oli to the Cc: loop]
>
> Hi,
>
> > I'm following up on this bug report from September (
> > https://lists.denx.de/pipermail/u-boot/2022-September/494811.html )
> > I also hit this issue on my Clearfog Pro when booting with eMMC.
>
> FYI we've recently got similar bug report in OpenWrt as well
> https://github.com/openwrt/openwrt/issues/11661
>
> There Oli is having problem to boot v2022.07 from SD card on his Clearfog Pro,
> version v2021.01 was working fine for him.
>
> > Florian's bug report was incredibly helpful in isolating the issue.
>
> Indeed, that eMMC handling looks quite different, as it seems that BootROM is
> looking at offset 0x0:
>
> -> BootROM - 1.73
> -> Booting from MMC
> -> BootROM: Bad header at offset
> -> BootROM: Bad header at offset 0020
> -> Switching BootPartitions.
> -> BootROM: Invalid HDR source addr
>
> but in Oli's SD card boot log we can see, that it looks at 0x200 offset first:
>
> BootROM - 1.73
> Booting from MMC
> BootROM: Bad header at offset 0200
> BootROM: Bad header at offset 4400
> BootROM: Bad header at offset 0020
>
> So perhaps SD and eMMC booting needs to be treated differently?
Much of the work that introduced these issues appears to have been
making a single kwboot version that works for all cases, rather than
having a board specific tool. When I tried creating a patch, that's
where I got stuck because there doesn't seem to be enough information
in the headers to decide between SD and eMMC. They both use the same
final header address, so maybe there's a possible solution, though it
still doesn't help with the sector size calculation.
> > > After binary patching the the srcaddr (srcaddr = srcaddr * 512) in the
> > > kwbimage v1 header, I got the BootROM to execute the SPL again.
>
> So this means following commit 501a54a29cc2 ("tools: kwbimage: Fix
> generation of SATA, SDIO and PCIe images") ? As reverting that commit yields
> following kwb image diff related to srcaddr changes:
>
> --- v2021.10-rc1-108-g501a54a29cc2.xxd 2023-01-09 20:37:08.250562742 +0100
> +++ v2021.10-rc1-108-g501a54a29cc2-revert.xxd 2023-01-09
> 20:37:17.902718024 +0100
> @@ -1,5 +1,5 @@
> -: ae00 0098 0600 0102 0080 4001 @...
> -0010: c0ff 7f00 8000 01c2
> +: ae00 4496 0600 0102 0080 0080 0200 D...
> +0010: c0ff 7f00 8000 0145 ...E
> 0020: 0201 60df 0200 5b00 6800 ..`.[...h...
> 0030:
> 0040: 0f00 00ea 14f0 9fe5 14f0 9fe5 14f0 9fe5
>
> Which in human readable form is following:
>
> --- v2021.10-rc1-108-g501a54a29cc2.header 2023-01-09 20:51:56.116867268
> +0100
> +++ v2021.10-rc1-108-g501a54a29cc2-revert.header 2023-01-09
> 20:52:55.987832692 +0100
> @@ -1,9 +1,9 @@
> blockid = 0xae,
> nandeccmode = 0x0,
> nandpagesize = 0x0,
> -blocksize = 0x69800,
> +blocksize = 0x69644,
> rsvd1 = 0x8201,
> -srcaddr = 0x140,
> +srcaddr = 0x28000,
> destaddr = 0x7fffc0,
> execaddr = 0x80,
> satapiomode = 0x0,
> @@ -11,4 +11,4 @@
> ddrinitdelay = 0x0,
> rsvd2 = 0x0,
> ext = 0x1,
> -checksum = 0xd8
> +checksum = 0x51
>
> Does it mean, that going back to the commit before this change, thus commit
> bd487ce08135 ("tools: kwbimage: Add constant for SDIO bootfrom") makes your
> device bootable again from SD, SATA and eMMC?
>
> In other words, it would help to know, which commit after v2021.01 breaks
> booting of SD/SATA/eMMC for you. Could you help find it?
The commit from the git bisect is one of dozens that are associated
with a major refactoring of kwboot and kwbimage; you probably want to
go back to the previous merge commit to get a working build.
> > I've gone a bit further and gotten eMMC boot working again, but I'm
> > not a C programmer so this is just a proof of concept/hack, not a
> > patch, and it would break SD card boot. Happy to test any proper
> > patches, though!
> >
> > These steps require a working u-boot from another boot medium. I used
> > SolidRun's u-boot-clearfog-pro-sata.kwb from
> > https://images.solid-run.com/A38X/U-Boot
>
> So am I parsing it properly, that you're not even able to boot v2022.10 from
> SATA?
I didn't test SATA; I used a known working image to help isolate the
problem and have a stable base to test from.
> > Steps to fix eMMC boot:
> > # In u-boot source
> > 1. Apply the patches at the bottom of the email to v2022.10 release
> > 2. make clearfog_defconfig
> > 3. make menuconfig
> > - Set CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR=0x1
> > - I also selected
> > CONFIG_CLEARFOG_CON3_SATA=y
> > CONFIG_CLEARFOG_CON2_SATA=y
> > CONFIG_CLEARFOG_SFP_25GB=y
> > CONFIG_DDR_
Re: [U-Boot][bug report] clearfog: EMMC boot broken on clearfog pro
Martin Rowe [2023-01-03 03:20:49]:
[Adding Florian, Tomasz and Oli to the Cc: loop]
Hi,
> I'm following up on this bug report from September (
> https://lists.denx.de/pipermail/u-boot/2022-September/494811.html )
> I also hit this issue on my Clearfog Pro when booting with eMMC.
FYI we've recently got similar bug report in OpenWrt as well
https://github.com/openwrt/openwrt/issues/11661
There Oli is having problem to boot v2022.07 from SD card on his Clearfog Pro,
version v2021.01 was working fine for him.
> Florian's bug report was incredibly helpful in isolating the issue.
Indeed, that eMMC handling looks quite different, as it seems that BootROM is
looking at offset 0x0:
-> BootROM - 1.73
-> Booting from MMC
-> BootROM: Bad header at offset
-> BootROM: Bad header at offset 0020
-> Switching BootPartitions.
-> BootROM: Invalid HDR source addr
but in Oli's SD card boot log we can see, that it looks at 0x200 offset first:
BootROM - 1.73
Booting from MMC
BootROM: Bad header at offset 0200
BootROM: Bad header at offset 4400
BootROM: Bad header at offset 0020
So perhaps SD and eMMC booting needs to be treated differently?
> > After binary patching the the srcaddr (srcaddr = srcaddr * 512) in the
> > kwbimage v1 header, I got the BootROM to execute the SPL again.
So this means following commit 501a54a29cc2 ("tools: kwbimage: Fix
generation of SATA, SDIO and PCIe images") ? As reverting that commit yields
following kwb image diff related to srcaddr changes:
--- v2021.10-rc1-108-g501a54a29cc2.xxd 2023-01-09 20:37:08.250562742 +0100
+++ v2021.10-rc1-108-g501a54a29cc2-revert.xxd 2023-01-09 20:37:17.902718024
+0100
@@ -1,5 +1,5 @@
-: ae00 0098 0600 0102 0080 4001 @...
-0010: c0ff 7f00 8000 01c2
+: ae00 4496 0600 0102 0080 0080 0200 D...
+0010: c0ff 7f00 8000 0145 ...E
0020: 0201 60df 0200 5b00 6800 ..`.[...h...
0030:
0040: 0f00 00ea 14f0 9fe5 14f0 9fe5 14f0 9fe5
Which in human readable form is following:
--- v2021.10-rc1-108-g501a54a29cc2.header 2023-01-09 20:51:56.116867268
+0100
+++ v2021.10-rc1-108-g501a54a29cc2-revert.header 2023-01-09
20:52:55.987832692 +0100
@@ -1,9 +1,9 @@
blockid = 0xae,
nandeccmode = 0x0,
nandpagesize = 0x0,
-blocksize = 0x69800,
+blocksize = 0x69644,
rsvd1 = 0x8201,
-srcaddr = 0x140,
+srcaddr = 0x28000,
destaddr = 0x7fffc0,
execaddr = 0x80,
satapiomode = 0x0,
@@ -11,4 +11,4 @@
ddrinitdelay = 0x0,
rsvd2 = 0x0,
ext = 0x1,
-checksum = 0xd8
+checksum = 0x51
Does it mean, that going back to the commit before this change, thus commit
bd487ce08135 ("tools: kwbimage: Add constant for SDIO bootfrom") makes your
device bootable again from SD, SATA and eMMC?
In other words, it would help to know, which commit after v2021.01 breaks
booting of SD/SATA/eMMC for you. Could you help find it?
> I've gone a bit further and gotten eMMC boot working again, but I'm
> not a C programmer so this is just a proof of concept/hack, not a
> patch, and it would break SD card boot. Happy to test any proper
> patches, though!
>
> These steps require a working u-boot from another boot medium. I used
> SolidRun's u-boot-clearfog-pro-sata.kwb from
> https://images.solid-run.com/A38X/U-Boot
So am I parsing it properly, that you're not even able to boot v2022.10 from
SATA?
> Steps to fix eMMC boot:
> # In u-boot source
> 1. Apply the patches at the bottom of the email to v2022.10 release
> 2. make clearfog_defconfig
> 3. make menuconfig
> - Set CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR=0x1
> - I also selected
> CONFIG_CLEARFOG_CON3_SATA=y
> CONFIG_CLEARFOG_CON2_SATA=y
> CONFIG_CLEARFOG_SFP_25GB=y
> CONFIG_DDR_RESET_ON_TRAINING_FAILURE=y
> but they should not affect eMMC booting
> 4. make
> 5. Copy u-boot-spl.kwb to booted linux on Clearfog Pro
> # In linux on Clearfog Pro
> 6. echo 0 > /sys/block/mmcblk0boot0/force_ro
> 7. dd if=/dev/zero of=/dev/mmcblk0boot0 conv=sync [probably not required]
> 8. dd if=u-boot-spl.kwb of=/dev/mmcblk0boot0 conv=sync
> 9. dd if=u-boot-spl.kwb of=/dev/mmcblk0 bs=512 seek=1
> 10. reboot
> # In working u-boot on Clearfog Pro
> 11. mmc partconf 0 1 1 0 [doesn't seem to be needed, but recommended
> by board/solidrun/clearfog/README]
> # On board
> 12. Set boot select DIP to eMMC (00111)
> 13. Power on and boot succeeds
>
> The instructions also work with
> CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR=0x0 and step 9 changed to `dd
> if=u-boot-spl.kwb of=/dev/mmcblk0 conv=sync`, but this will clobber
> any partition table that exists on eMMC making it mostly useless other
> than for booting.
>
> Weirdly both the boot and the data parts of the eMMC need to have
> u-boot. It looks like SPL comes from
Re: [U-Boot][bug report] clearfog: EMMC boot broken on clearfog pro
Hi,
I'm following up on this bug report from September (
https://lists.denx.de/pipermail/u-boot/2022-September/494811.html )
I also hit this issue on my Clearfog Pro when booting with eMMC.
Florian's bug report was incredibly helpful in isolating the issue.
I've gone a bit further and gotten eMMC boot working again, but I'm
not a C programmer so this is just a proof of concept/hack, not a
patch, and it would break SD card boot. Happy to test any proper
patches, though!
These steps require a working u-boot from another boot medium. I used
SolidRun's u-boot-clearfog-pro-sata.kwb from
https://images.solid-run.com/A38X/U-Boot
Steps to fix eMMC boot:
# In u-boot source
1. Apply the patches at the bottom of the email to v2022.10 release
2. make clearfog_defconfig
3. make menuconfig
- Set CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR=0x1
- I also selected
CONFIG_CLEARFOG_CON3_SATA=y
CONFIG_CLEARFOG_CON2_SATA=y
CONFIG_CLEARFOG_SFP_25GB=y
CONFIG_DDR_RESET_ON_TRAINING_FAILURE=y
but they should not affect eMMC booting
4. make
5. Copy u-boot-spl.kwb to booted linux on Clearfog Pro
# In linux on Clearfog Pro
6. echo 0 > /sys/block/mmcblk0boot0/force_ro
7. dd if=/dev/zero of=/dev/mmcblk0boot0 conv=sync [probably not required]
8. dd if=u-boot-spl.kwb of=/dev/mmcblk0boot0 conv=sync
9. dd if=u-boot-spl.kwb of=/dev/mmcblk0 bs=512 seek=1
10. reboot
# In working u-boot on Clearfog Pro
11. mmc partconf 0 1 1 0 [doesn't seem to be needed, but recommended
by board/solidrun/clearfog/README]
# On board
12. Set boot select DIP to eMMC (00111)
13. Power on and boot succeeds
The instructions also work with
CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR=0x0 and step 9 changed to `dd
if=u-boot-spl.kwb of=/dev/mmcblk0 conv=sync`, but this will clobber
any partition table that exists on eMMC making it mostly useless other
than for booting.
Weirdly both the boot and the data parts of the eMMC need to have
u-boot. It looks like SPL comes from a boot partition, but then tries
to load u-boot proper from the data partition. This is also noted
here: https://wiki.debian.org/InstallingDebianOn/ClearFog
Pali Rohar appears to have made some significant improvements to mvebu
booting around July 2021, but these changes seem to only account for
an SD card for MMC boot, not eMMC which doesn't appear to need the
sector (512) multiplier. Most of the patches below remove special
handling added for the IBR_HDR_SDIO_ID case which almost certainly
breaks SD card booting. The removal of the error for
CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR != 0 appears to have no side
effects for other boot modes. The dts change is a long known issue and
functionally matches a patch carried by armbian and discussed here:
https://forum.armbian.com/topic/3072-clearfog-pro-emmc-requires-sd-card-to-detect-device/
Let me know if I can help get a permanent fix merged.
Martin
diff --git a/arch/arm/dts/armada-388-clearfog.dts
b/arch/arm/dts/armada-388-clearfog.dts
index e4164f49b2..29a608abcf 100644
--- a/arch/arm/dts/armada-388-clearfog.dts
+++ b/arch/arm/dts/armada-388-clearfog.dts
@@ -101,7 +101,7 @@
sdhci@d8000 {
bus-width = <4>;
- cd-gpios = <&gpio0 20 GPIO_ACTIVE_LOW>;
+ non-removable;
no-1-8-v;
pinctrl-0 = <µsom_sdhci_pins
&clearfog_sdhci_cd_pins>;
diff --git a/arch/arm/mach-mvebu/spl.c b/arch/arm/mach-mvebu/spl.c
index ca2d5a59d7..00334562cf 100644
--- a/arch/arm/mach-mvebu/spl.c
+++ b/arch/arm/mach-mvebu/spl.c
@@ -44,9 +44,6 @@
#ifdef CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_USE_PARTITION
#error CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_USE_PARTITION is unsupported
#endif
-#if defined(CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR) &&
CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR != 0
-#error CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_SECTOR must be set to 0
-#endif
#if defined(CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_DATA_PART_OFFSET) && \
CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_DATA_PART_OFFSET != 0
#error CONFIG_SYS_MMCSD_RAW_MODE_U_BOOT_DATA_PART_OFFSET must be set to 0
@@ -196,14 +193,6 @@ int spl_parse_board_header(struct spl_image_info
*spl_image,
spl_image->offset *= 512;
}
- /*
-* For SDIO (eMMC) srcaddr is specified in number of sectors.
-* This expects that sector size is 512 bytes and recalculates
-* data offset to bytes.
-*/
- if (IS_ENABLED(CONFIG_SPL_MMC) && mhdr->blockid == IBR_HDR_SDIO_ID)
- spl_image->offset *= 512;
-
if (spl_image->offset % 4 != 0) {
printf("ERROR: Wrong srcaddr (0x%08x) in kwbimage\n",
spl_image->offset);
diff --git a/tools/kwbimage.c b/tools/kwbimage.c
index 94b7685392..43b51ffd1a 100644
--- a/tools/kwbimage.c
+++ b/tools/kwbimage.c
@@ -1021,15 +1021,6 @@ static void *image_create_v0(size_
[U-Boot][bug report] clearfog: EMMC boot broken on clearfog pro
Hi, I am using a clearfog pro with emmc as router. while upgrading from openwrt 21.02.3 to 22.03.0 I discovered, that the provided u-boot binary gets rejected by the BootROM: -> BootROM - 1.73 -> Booting from MMC -> BootROM: Bad header at offset -> BootROM: Bad header at offset 0020 -> Switching BootPartitions. -> BootROM: Invalid HDR source addr While comparing the different u-boot-spl.kwb between openwrt 21.02.3 (u-boot 2021.01) and openwrt 22.03.0 (u-boot 2022.07) I saw that the image has changed from a gpimage to the kwbimage v1. After reading some code, my gut feeling told me, that maybe the BootROM does not like the HDR source addr because it does not expect it to be in sectors but in Bytes. After binary patching the the srcaddr (srcaddr = srcaddr * 512) in the kwbimage v1 header, I got the BootROM to execute the SPL again. -> Booting from MMC -> U-Boot SPL 2022.07-OpenWrt-r19685-512e76967f (Sep 03 2022 - 02:55:34 +) -> High speed PHY - Version: 2.0 -> EEPROM TLV detection failed: Using static config for Clearfog Pro. -> Detected Device ID 6828 -> board SerDes lanes topology details: -> | Lane # | Speed | Type | -> -> | 0| 3 | SATA0 | -> | 1| 0 | SGMII1 | -> | 2| 5 | PCIe1 | -> | 3| 5 | USB3 HOST1 | -> | 4| 5 | PCIe2 | -> | 5| 0 | SGMII2 | -> -> High speed PHY - Ended Successfully -> mv_ddr: 14.0.0 -> DDR3 Training Sequence - Switching XBAR Window to FastPath Window -> mv_ddr: completed successfully -> Trying to boot from MMC1 -> ERROR: Invalid kwbimage v1 -> ERROR: Invalid data checksum in kwbimage -> SPL: failed to boot from all boot devices -> ### ERROR ### Please RESET the board ### I think I do understand the problem here, but I am not quite sure whats the best way to move forward. Unfortunately I am not that familiar with the u-boot sourcecode... Could you give me some pointers, where to start for an integrate-able solution? Thanks!
