On 10/28/22 11:09, Richard Purdie wrote:
On Wed, 2022-10-26 at 13:21 -0400, Sean Anderson wrote:
On 10/26/22 13:00, Richard Purdie wrote:
On Wed, 2022-10-26 at 12:49 -0400, Sean Anderson wrote:
On 10/26/22 10:08, Richard Purdie wrote:
This looks like a pretty nice improvement, I've been worried about how
entangled all this was becoming. Is there anything better we could do
with testing of this code? We rely increasingly on the tests to spot
regressions and I doubt the testcases we have cover the code
complexity.
FWIW, I found the test suite to be fairly helpful when debugging this
whole process, if agonizingly slow. I found many bugs in my initial
implementation which were uncovered by the test suites,
The best way to extend the test suite would probably be to have QEMU try
to boot using the Linux and U-Boot. This would help cases where all the
artifacts are created correctly but e.g. the wrong U-Boot is packaged.
However, I am rather loathe to add more test cases like this, because it
already takes around 10 minutes (with sstate!) to run the tests. This
basically kills the speed of iterative development.
Out of interest how were you running the tests? Did you look at the
parallelism options? Were you just running specific tests or specific
suites of tests?
As noted in the cover letter, I ran
oe-selftest -r fitimage.FitImageTests
Ok, good. That at least means you were only running one class of tests.
I was worried you were running all of them!
I also tried using -j$(nproc), but I saw no increase in parallelism
outside of the usual for bitbake. This was especially noticable for
do_rootfs, which is single-threaded.
Sadly the parallelism works on a per test class basis so it wouldn't
help in this case. There are only small marginal gains from running
tests in individual build directories so we don't do that.
I estimate it could have saved me 2-3 minutes every build, since it could
have parallelized the root filesystem stuff.
This is ommitted above, but I *had* to use -j1 in order to avoid
manually wiping out my existing build directory each time (and instead
ending up with dozens of pid-named directories). This is documented
nowhere, and I found it in some old IRC logs.
Parallelism using differently named build directories is an
implementation detail, not something which the -j option implies.I
guess you were also using --keep-builddir
Failing builds don't remove the test directory so you can inspect the build
output. As you might imagine, I had a lot of failing builds.
and could have used the -B
option with $PID or similar for the same effect.
Probably.
We could document
that?
Please do.
I do think we need something like you describe and adding a linux+uboot
approach is something I've wanted to do for a long time. Usually the
time is taken on our automated CI rather than impacting users, unless
they're working in that area at which point the tests are hopefully
helpful.
Which of course provides no incentive to reduce runtime.
Most of the tests do use our standard tasks so there is as much
incentive to optimise do_rootfs as there would be otherwise :/.
We haven't really had anyone try and optimise the tests either, I'm
sure there will be things in there which can help. Please don't let the
speed put you off trying to improve things and extend our coverage!
The poor speed of these self tests (and of everything related to the
yocto project in general) makes this project frustrating to contribute
to. It took me around 2 days to go from my prototype to this series,
most of which was spent waiting for tests to compile and losing whatever
train of thought I had. I probably went through perhaps 20 revisions. If
I was working on e.g. U-Boot, I could have made 20 revisions in 2 hours,
as it takes around 15 seconds to recompile it and run the full unit test
suite.
On the topic of these specific tests, part of the problem is that
do_rootfs is a bottleneck which takes around 45-60s on my system. Every
test which modifies something in the rootfs incurs this overhead.
For better or worse we've 'a few' more moving pieces than U-Boot.
Building a root filesystem from packages is a non-trivial task, taking
under a minute is in some ways pretty good already. The only other
thing we could do is incremental rootfs construction where it would
add/remove changed packages. I'd worry that the result may not always
be equal to a build from scratch and it might cause weird and
interesting reproducibility problems (particularly when you consider
things like postinsts).
I would love to improve our development "iteration" time but I'm
struggling to see where we could get the speed gains from :(. Open to
other ideas...
We don't have to build a full root filesystem. All of these tests just want
e.g. an initramfs. An empty (or one file) filesystem would work just as well.
If you still want to boot, you can make a busybox filesystem.
--Sean
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#172259):
https://lists.openembedded.org/g/openembedded-core/message/172259
Mute This Topic: https://lists.openembedded.org/mt/94487631/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
