Re: OrangePI Zero3 memory timing testing
Hello, Please try to insert udelay() calls in addition to dsb() in the mctl_mem_matches() and see if this helps. I suspect that there's just no way to do perfectly reliable synchronization all the way from the CPU to DRAM and back in this particular scenario (which involves wacky things, such as accessing aliased physical memory addresses). So just having a generous delay may do the job and solve all the timing issues. Another comment is about the use of memtester program. It's actually not very sensitive to memory reliability problems as-is. Back in A10/A20 and H3 days, it was necessary to run memtester simultaneously with a Mali GPU rendered spinning textured cube animation for best results: https://github.com/ssvb/lima-memtester For example, a misconfigured H3 DRAM was showing no problems at all in the memtester test at 624MHz, but couldn't survive memtester together with the spinning textured cube until downclocked to 384MHz. Only aggressive simultaneous access from both CPU and GPU really exposes reliability problems. The other types of DMA transfers couldn't replace the GPU. It was also possible to use the G2D accelerator instead of the Mali GPU, but only when performing 90 degrees rotations. Doing simple non-rotated blits with G2D was useless and couldn't expose reliability problems. I would suggest to install 3D drivers (whatever is available for the modern kernels nowadays) and run some textured spinning cube demo together with memtester before deciding that your DRAM configuration is alright. On Thu, Nov 30, 2023 at 2:11 AM Andre Przywara wrote: > > Hi Stephen, > > On 29/11/2023 18:45, Stephen Graf wrote: > > Some testing results: > > > > With the default DRAM timing of 30x24=720, most often when my system > > boots it comes up with DRAM 2G, but I have a 1G system. Once in a while > > the boot shows 1G. When it shows 2G the linux OS also shows 2G and > > continuing does not make much sense. > > > > On one boot where u-boot reported 1G the OS agreed and I ran memtester > > successfully. > > > > If I build u-boot with CONFIG_DRAM_CLK=792 (33*24=792) I have never seen > > my system boot with anything other than DRAM 1G showing in u-boot. I ran > > Interesting, many thanks for the elaborate experiments! > To be fair, the timing parameters were from Xunlong's setup, so with 792 > MHz, but I remember that in my early testing this failed more often than > not, so I reverted back to 720MHz. But I had different parameters back > then, I believe. I will try to arrange some boot loop, to see how it > fares with the 792 MHz, though I believe you that it's more stable. > If that works, I will send a v2 with the USB and the DRAM clock fixed. > > Thanks again for testing and the report! > > Cheers, > Andre > > > memtester successfully and this system has been stable running linux 6.6.2. > > > > Andre, if you can put together a few patches I can test them to see > > which works. > > > > If anyone else with different variations of the Zero3 would test with > > the two DRAM timings and report results. > > > > I also use the Zunlong image on occasion and it consistently shows 1G > > and from my poking around I think it runs at a 792 clk. > > > > > > Console output with DRAM default: > > > > U-Boot 2024.01-rc3-00028-g43f2873fa9-dirty (Nov 28 2023 - 22:46:39 > > -0800) Allwinner Technology > > > > CPU: Allwinner H616 (SUN50I) > > Model: OrangePi Zero3 > > DRAM: 2 GiB > > > > Last login: Wed Nov 29 09:35:15 PST 2023 on ttyS0 > > root@orangepizero3:~# free -h > > totalusedfree shared buff/cache > > available > > Mem: 1.9Gi 139Mi 1.7Gi 360Ki82Mi > > 1.7Gi > > Swap: 0B 0B 0B > > > > > > U-Boot 2024.01-rc3-00028-g43f2873fa9-dirty (Nov 28 2023 - 22:46:39 > > -0800) Allwinner Technology > > > > CPU: Allwinner H616 (SUN50I) > > Model: OrangePi Zero3 > > DRAM: 1 GiB > > > > root@orangepizero3:~# free -h > > totalusedfree shared buff/cache > > available > > Mem: 917Mi 137Mi 766Mi 360Ki79Mi > > 780Mi > > Swap: 0B 0B 0B > > > > > > root@orangepizero3:~# memtester 700M 1 > > memtester version 4.6.0 (64-bit) > > Copyright (C) 2001-2020 Charles Cazabon. > > Licensed under the GNU General Public License version 2 (only). > > > > pagesize is 4096 > > pagesizemask is 0xf000 > > want 700MB (734003200 bytes) > > got 700MB (734003200 bytes), trying mlock ...locked. > > Loop 1/1: > >Stuck Address : ok > >Random Value: ok > >Compare XOR : ok > >Compare SUB : ok > >Compare MUL : ok > >Compare DIV : ok > >Compare OR : ok > >Compare AND : ok > >Sequential Increment: ok > >Solid Bits : ok > >Block Sequential: ok > >Checkerboard: ok > >Bit Spread : ok > >Bit Fl
Re: OrangePI Zero3 memory timing testing
Hi Stephen, On 29/11/2023 18:45, Stephen Graf wrote: Some testing results: With the default DRAM timing of 30x24=720, most often when my system boots it comes up with DRAM 2G, but I have a 1G system. Once in a while the boot shows 1G. When it shows 2G the linux OS also shows 2G and continuing does not make much sense. On one boot where u-boot reported 1G the OS agreed and I ran memtester successfully. If I build u-boot with CONFIG_DRAM_CLK=792 (33*24=792) I have never seen my system boot with anything other than DRAM 1G showing in u-boot. I ran Interesting, many thanks for the elaborate experiments! To be fair, the timing parameters were from Xunlong's setup, so with 792 MHz, but I remember that in my early testing this failed more often than not, so I reverted back to 720MHz. But I had different parameters back then, I believe. I will try to arrange some boot loop, to see how it fares with the 792 MHz, though I believe you that it's more stable. If that works, I will send a v2 with the USB and the DRAM clock fixed. Thanks again for testing and the report! Cheers, Andre memtester successfully and this system has been stable running linux 6.6.2. Andre, if you can put together a few patches I can test them to see which works. If anyone else with different variations of the Zero3 would test with the two DRAM timings and report results. I also use the Zunlong image on occasion and it consistently shows 1G and from my poking around I think it runs at a 792 clk. Console output with DRAM default: U-Boot 2024.01-rc3-00028-g43f2873fa9-dirty (Nov 28 2023 - 22:46:39 -0800) Allwinner Technology CPU: Allwinner H616 (SUN50I) Model: OrangePi Zero3 DRAM: 2 GiB Last login: Wed Nov 29 09:35:15 PST 2023 on ttyS0 root@orangepizero3:~# free -h total used free shared buff/cache available Mem: 1.9Gi 139Mi 1.7Gi 360Ki 82Mi 1.7Gi Swap: 0B 0B 0B U-Boot 2024.01-rc3-00028-g43f2873fa9-dirty (Nov 28 2023 - 22:46:39 -0800) Allwinner Technology CPU: Allwinner H616 (SUN50I) Model: OrangePi Zero3 DRAM: 1 GiB root@orangepizero3:~# free -h total used free shared buff/cache available Mem: 917Mi 137Mi 766Mi 360Ki 79Mi 780Mi Swap: 0B 0B 0B root@orangepizero3:~# memtester 700M 1 memtester version 4.6.0 (64-bit) Copyright (C) 2001-2020 Charles Cazabon. Licensed under the GNU General Public License version 2 (only). pagesize is 4096 pagesizemask is 0xf000 want 700MB (734003200 bytes) got 700MB (734003200 bytes), trying mlock ...locked. Loop 1/1: Stuck Address : ok Random Value : ok Compare XOR : ok Compare SUB : ok Compare MUL : ok Compare DIV : ok Compare OR : ok Compare AND : ok Sequential Increment: ok Solid Bits : ok Block Sequential : ok Checkerboard : ok Bit Spread : ok Bit Flip : ok Walking Ones : ok Walking Zeroes : ok 8-bit Writes : ok 16-bit Writes : ok Done. Console output with DRAM clk 792: U-Boot 2024.01-rc3-00028-g43f2873fa9-dirty (Nov 28 2023 - 21:34:46 -0800) Allwinner Technology CPU: Allwinner H616 (SUN50I) Model: OrangePi Zero3 DRAM: 1 GiB root@orangepizero3:~# free -h total used free shared buff/cache available Mem: 917Mi 134Mi 768Mi 360Ki 79Mi 782Mi Swap: 0B 0B 0B root@orangepizero3:~# memtester 700M 1 memtester version 4.6.0 (64-bit) Copyright (C) 2001-2020 Charles Cazabon. Licensed under the GNU General Public License version 2 (only). pagesize is 4096 pagesizemask is 0xf000 want 700MB (734003200 bytes) got 700MB (734003200 bytes), trying mlock ...locked. Loop 1/1: Stuck Address : ok Random Value : ok Compare XOR : ok Compare SUB : ok Compare MUL : ok Compare DIV : ok Compare OR : ok Compare AND : ok Sequential Increment: ok Solid Bits : ok Block Sequential : ok Checkerboard : ok Bit Spread : ok Bit Flip : ok Walking Ones : ok Walking Zeroes : ok 8-bit Writes : ok 16-bit Writes : ok Done. On 2023-11-27 5:37 p.m., Andre Przywara wrote: This symptom is pretty surely not directly related to the DRAM frequency, it's caused by a missing barrier or delay *somewhere*. People report that this is fixed by adding another "dsb();" at the beginning of the mctl_mem_matches() function, but I don't have a good explanation why exactly there, and would like to avoid submitting patches that "just seem to work (TM)". If I find some time, I will try to setup some automated envir
OrangePI Zero3 memory timing testing
Some testing results: With the default DRAM timing of 30x24=720, most often when my system boots it comes up with DRAM 2G, but I have a 1G system. Once in a while the boot shows 1G. When it shows 2G the linux OS also shows 2G and continuing does not make much sense. On one boot where u-boot reported 1G the OS agreed and I ran memtester successfully. If I build u-boot with CONFIG_DRAM_CLK=792 (33*24=792) I have never seen my system boot with anything other than DRAM 1G showing in u-boot. I ran memtester successfully and this system has been stable running linux 6.6.2. Andre, if you can put together a few patches I can test them to see which works. If anyone else with different variations of the Zero3 would test with the two DRAM timings and report results. I also use the Zunlong image on occasion and it consistently shows 1G and from my poking around I think it runs at a 792 clk. Console output with DRAM default: U-Boot 2024.01-rc3-00028-g43f2873fa9-dirty (Nov 28 2023 - 22:46:39 -0800) Allwinner Technology CPU: Allwinner H616 (SUN50I) Model: OrangePi Zero3 DRAM: 2 GiB Last login: Wed Nov 29 09:35:15 PST 2023 on ttyS0 root@orangepizero3:~# free -h totalusedfree shared buff/cache available Mem: 1.9Gi 139Mi 1.7Gi 360Ki82Mi 1.7Gi Swap: 0B 0B 0B U-Boot 2024.01-rc3-00028-g43f2873fa9-dirty (Nov 28 2023 - 22:46:39 -0800) Allwinner Technology CPU: Allwinner H616 (SUN50I) Model: OrangePi Zero3 DRAM: 1 GiB root@orangepizero3:~# free -h totalusedfree shared buff/cache available Mem: 917Mi 137Mi 766Mi 360Ki79Mi 780Mi Swap: 0B 0B 0B root@orangepizero3:~# memtester 700M 1 memtester version 4.6.0 (64-bit) Copyright (C) 2001-2020 Charles Cazabon. Licensed under the GNU General Public License version 2 (only). pagesize is 4096 pagesizemask is 0xf000 want 700MB (734003200 bytes) got 700MB (734003200 bytes), trying mlock ...locked. Loop 1/1: Stuck Address : ok Random Value: ok Compare XOR : ok Compare SUB : ok Compare MUL : ok Compare DIV : ok Compare OR : ok Compare AND : ok Sequential Increment: ok Solid Bits : ok Block Sequential: ok Checkerboard: ok Bit Spread : ok Bit Flip: ok Walking Ones: ok Walking Zeroes : ok 8-bit Writes: ok 16-bit Writes : ok Done. Console output with DRAM clk 792: U-Boot 2024.01-rc3-00028-g43f2873fa9-dirty (Nov 28 2023 - 21:34:46 -0800) Allwinner Technology CPU: Allwinner H616 (SUN50I) Model: OrangePi Zero3 DRAM: 1 GiB root@orangepizero3:~# free -h totalusedfree shared buff/cache available Mem: 917Mi 134Mi 768Mi 360Ki79Mi 782Mi Swap: 0B 0B 0B root@orangepizero3:~# memtester 700M 1 memtester version 4.6.0 (64-bit) Copyright (C) 2001-2020 Charles Cazabon. Licensed under the GNU General Public License version 2 (only). pagesize is 4096 pagesizemask is 0xf000 want 700MB (734003200 bytes) got 700MB (734003200 bytes), trying mlock ...locked. Loop 1/1: Stuck Address : ok Random Value: ok Compare XOR : ok Compare SUB : ok Compare MUL : ok Compare DIV : ok Compare OR : ok Compare AND : ok Sequential Increment: ok Solid Bits : ok Block Sequential: ok Checkerboard: ok Bit Spread : ok Bit Flip: ok Walking Ones: ok Walking Zeroes : ok 8-bit Writes: ok 16-bit Writes : ok Done. On 2023-11-27 5:37 p.m., Andre Przywara wrote: This symptom is pretty surely not directly related to the DRAM frequency, it's caused by a missing barrier or delay *somewhere*. People report that this is fixed by adding another "dsb();" at the beginning of the mctl_mem_matches() function, but I don't have a good explanation why exactly there, and would like to avoid submitting patches that "just seem to work (TM)". If I find some time, I will try to setup some automated environment where I can run some experiments with different barrier locations. I can offer some rationale for putting it at the end of the function(s) that call mctl_mem_matches(), so hope that this fixes it. Cheers, Andre
