Package: debci tl;dr:
Please can you increase (double, say) the timeout for the srd:dgit autopkgtests on riscv64. Problem src:dgit has over 100 tests, organised by test-dependencies into about about 30 stanzas. This has generally been fine and is a mode of use I anticipated when I designed autopkgtest. A formal src:dgit test run under autopkgtest takes under an hour on my amd64 laptop. The large number of stanzas does mean that the test runner needs to reset the testbed and install the test-dependencies dozens of times. On some slow runners, this was being a problem. I filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1063533 about this (in February 2024). To work around the problem with Debian's very slow riscv64 runners I added `Architecture: !riscv64` to the least useful and most long-running stanzas. This made things reliable. However, recently, autopkgtest was changed to always reset the testbed and reinstall the test dependencies, for every test case: https://tracker.debian.org/news/1775760/accepted-autopkgtest-60-source-into-unstable/ For src:dgit's DEP-8 tests, that represents about a 4x performance regression. Test runs that used to take 2-5h on riscv64 now usually time out after ~8h. Also, it has almost entirely negated the previous workaround I had for riscv64. I had focused on reducing the number of stanzas, since that would reduce the total number of resets and reinstalls. This problem doesn't just inconvenience the src:dgit maintainers and users. dgit lives at the top of a very large dependency stack. Right now it seems like the perl transition may get hung up on this problem. Possible solutions 1. We could revert the change to autopkgtest. I think this would probably be controversial, despite this change effectively being a release critical regression for src:dgit. (TBH I am disappointed that this change was made despite the knowledge, provided in #1063533, that it would probably cause serious regressions for at least some use cases, and without any kind of coordination such as an MBF for packages with many test cases, a proper transition plan, etc.) More realistically it might be possible to put something in the test metadata. "Features: ok-to-reuse-testbed-for-multiple-tests" or something. Having the src:dgit test suite declare a new feature is an easy low-risk change which would be eminently backportable. 2. It would be possible to re-engineer the src:dgit test suite so that all the test cases which share a common set of test dependencies are represented in a single stanza in debian/tests/control. However, it's not clear that this would actually solve the problem because wouldn't want to backport such a change to stable branches. So at least some CI runs would probably still be afllicted. It would entail significant regression to reporting of failures. Right now when a test fails, you know which test case failed because that's captured in autopkgtest's structured output. It's hard to make portmanteau test cases have good names in the autopkgtest data, and when they fail you'd have to find and eyeball the right bit of the output to find out which actual test(s) failed. Also, that would be quite an intrusive change. Right now I am juggling an experimental branch and also trying to land an important bugfix in testing. I think trying to do this test suite reorganisation at the same time is very risky. I could do it later. But I think we need a short term fix. 3. We could have riscv64 runners which provide better (and more consistent) performance. 4. We could increase the timeout. We'd still end up wasting computing resources but at least the tests would pass and not need to be repeated. 5. I could mark the vast majority of th cases tests !riscv64, leaving only a handful. The dgit test suite is not designed with this in mind, but in practice it has I think never detected any arch-specific bugs so that's probably OK? I think only options 1 (revert in autopkgtest), 4 (increased timeout) and 5 (do almost no testing on riscv64) are technically feasible in the short term. I think the autopkgtest maintainers probably don't want to revert. So in the short term, I think the answer is an increased timeout. I'm hoping you (the debci team) can do that for me. Thanks, Ian. -- Ian Jackson <[email protected]> These opinions are my own. Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter.
