On Wed, 2022-10-26 at 13:21 -0400, Sean Anderson wrote: > On 10/26/22 13:00, Richard Purdie wrote: > > On Wed, 2022-10-26 at 12:49 -0400, Sean Anderson wrote: > > > On 10/26/22 10:08, Richard Purdie wrote: > > > > This looks like a pretty nice improvement, I've been worried about how > > > > entangled all this was becoming. Is there anything better we could do > > > > with testing of this code? We rely increasingly on the tests to spot > > > > regressions and I doubt the testcases we have cover the code > > > > complexity. > > > > > > FWIW, I found the test suite to be fairly helpful when debugging this > > > whole process, if agonizingly slow. I found many bugs in my initial > > > implementation which were uncovered by the test suites, > > > > > > The best way to extend the test suite would probably be to have QEMU try > > > to boot using the Linux and U-Boot. This would help cases where all the > > > artifacts are created correctly but e.g. the wrong U-Boot is packaged. > > > However, I am rather loathe to add more test cases like this, because it > > > already takes around 10 minutes (with sstate!) to run the tests. This > > > basically kills the speed of iterative development. > > > > Out of interest how were you running the tests? Did you look at the > > parallelism options? Were you just running specific tests or specific > > suites of tests? > > As noted in the cover letter, I ran > > oe-selftest -r fitimage.FitImageTests
Ok, good. That at least means you were only running one class of tests. I was worried you were running all of them! > I also tried using -j$(nproc), but I saw no increase in parallelism > outside of the usual for bitbake. This was especially noticable for > do_rootfs, which is single-threaded. Sadly the parallelism works on a per test class basis so it wouldn't help in this case. There are only small marginal gains from running tests in individual build directories so we don't do that. > This is ommitted above, but I *had* to use -j1 in order to avoid > manually wiping out my existing build directory each time (and instead > ending up with dozens of pid-named directories). This is documented > nowhere, and I found it in some old IRC logs. Parallelism using differently named build directories is an implementation detail, not something which the -j option implies. I guess you were also using --keep-builddir and could have used the -B option with $PID or similar for the same effect. We could document that? > > I do think we need something like you describe and adding a linux+uboot > > approach is something I've wanted to do for a long time. Usually the > > time is taken on our automated CI rather than impacting users, unless > > they're working in that area at which point the tests are hopefully > > helpful. > > Which of course provides no incentive to reduce runtime. Most of the tests do use our standard tasks so there is as much incentive to optimise do_rootfs as there would be otherwise :/. > > We haven't really had anyone try and optimise the tests either, I'm > > sure there will be things in there which can help. Please don't let the > > speed put you off trying to improve things and extend our coverage! > > The poor speed of these self tests (and of everything related to the > yocto project in general) makes this project frustrating to contribute > to. It took me around 2 days to go from my prototype to this series, > most of which was spent waiting for tests to compile and losing whatever > train of thought I had. I probably went through perhaps 20 revisions. If > I was working on e.g. U-Boot, I could have made 20 revisions in 2 hours, > as it takes around 15 seconds to recompile it and run the full unit test > suite. > > On the topic of these specific tests, part of the problem is that > do_rootfs is a bottleneck which takes around 45-60s on my system. Every > test which modifies something in the rootfs incurs this overhead. For better or worse we've 'a few' more moving pieces than U-Boot. Building a root filesystem from packages is a non-trivial task, taking under a minute is in some ways pretty good already. The only other thing we could do is incremental rootfs construction where it would add/remove changed packages. I'd worry that the result may not always be equal to a build from scratch and it might cause weird and interesting reproducibility problems (particularly when you consider things like postinsts). I would love to improve our development "iteration" time but I'm struggling to see where we could get the speed gains from :(. Open to other ideas... Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#172257): https://lists.openembedded.org/g/openembedded-core/message/172257 Mute This Topic: https://lists.openembedded.org/mt/94487631/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
