Hello Grant, sorry if I just hijack your conversation, but I'm curious, because we are using the same SoC. Adding linux-arm-kernel to Cc for the general performance issues and linux-mtd for the ECC questions. O:-)
Am Donnerstag, 3. Dezember 2020, 23:20:38 CET schrieb Grant Edwards: > On 2020-12-03, Andrew Lunn <and...@lunn.ch> wrote: > >> I don't think there's any way I could justify using a kernel that > >> doesn't have long-term support. > > > > 5.10 is LTS. Well, it will be, once it is actually released! > > Convincing people to ship an unreleased kernel would be a whole > 'nother bucket of worms. +1 Judging just from the release dates of the last LTS kernels, I would have guessed v5.11 will get LTS. But there has been no announcement yet and I suppose there will be none before release? For ordinary users it's just like staring in a crystal ball, so we aim at v5.4 for our more recent hardware platforms. ¯\_(ツ)_/¯ > > It's all moot now. The decision was just made to shelve the 5.4 kernel > "upgrade" and stick with 2.6.33 for now. > > >> [It looks like we're going to have to abandon the effort to use > >> 5.4. The performance is so bad compared to 2.6.33.7 that our product > >> just plain won't work. We've already had remove features to the get > >> 5.4 kernel down to a usable size, but we were prepared to live with > >> that.] > > > > Ah. Small caches? > > Yep. It's An old Atmel ARM926T part (at91sam9g20) with 32KB I-cache > and 32KB D-cache. > > A simple user-space multi-threaded TCP echo server benchmark showed a > 30-50% (depending on number of parallel connections) drop in max > throughput. Our typical applications also show a 15-25% increase in > CPU usage for an equivalent workload. Another problem is high > latencies with 5.4. A thread that is supposed to wake up every 1ms > works reliably on 2.6.33, but is a long ways from working on 5.4. We use the exact same SoC with kernel 4.9.220-rt143 (PREEMPT RT) currently, after being stuck on 2.6.36.4 for quite a while. I did not notice significant performance issues, but I have to admit, we never did extensive benchmarks on network or CPU performance, because the workload also changed for that target. However what gave us a lot less dropped packages was using the internal SRAM as DMA buffer for RX packages received by macb. That did not make it in mainline however, I did not put enough effort in polishing that patch back when we migrated from 2.6 to 4.x. If you're curious, it's on GitHub: https://github.com/LeSpocky/linux/tree/net-macb-sram-rx > I asked on the arm kernel mailing list if this was typical/expected, > but the post never made it past the moderator. > > > The OpenWRT guys make valid complaints that the code > > hot path of the network stack is getting too big to fit in the cache > > of small systems. So there is a lot of cache misses and performance is > > not good. If i remember correctly, just having netfilter in the build > > is bad, even if it is not used. > > We've already disabled absoltely everything we can and still have a > working system. With the same features enabled, the 5.4 kernel was > about 75% larger than a 2.6.33 kernel, so we had to trim quite a bit > of meat to get it to boot on existing units. Same here. v4.9 kernel image still fits, v4.14 is already too big for some devices we delivered in the early days. > We also can't get on-die ECC support for Micron NAND flash working > with 5.4. Even it did work, we'd still have to add the ability to > fall-back to SW ECC on read operations for the sake of backwards > compatibility on units that were initially shipped without on-die ECC > support enabled. IIRC the SoC itself has issues with its ECC engine? See mainline at91sam9g20ek_common.dtsi for example which sets nand-ecc-mode to "soft". The SAM9G20 Errata chapter in the complete datasheet from 2015 (Atmel-6384F) says two times in Section 44.1.3: "Perform the ECC computation by software." Greets Alex
