[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-05-04 Thread Balint Reczey
I've split the upstream test in 247.3-5ubuntu1 and also forwarded the change: https://salsa.debian.org/systemd-team/systemd/-/merge_requests/128 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1915126

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-03-08 Thread Mathew Hodson
This bug was fixed in the package systemd - 247.3-1ubuntu4 --- systemd (247.3-1ubuntu4) hirsute; urgency=medium [ Dimitri John Ledkov ] * d/p/debian/UBUNTU-resolved-Mitigate-DVE-2018-0001-by-retrying-NXDOMAIN-with.patch: Patch updated to reduce log level to debug

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-24 Thread Iain Lane
Sorry for not replying to comment #11, although we chatted on IRC. The main difference is the passing of the test name I think, but the full autopkgtest command that I used is at the top of the log. I hope using this exact command and adding --shell-fail is enough for anyone to reproduce. --

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-24 Thread Balint Reczey
@laney Thank you for looking into the problem. I will mark systemd-fsckd as flaky and look into the root cause later. It used to be flaky before as well. I have checked the last 10 failed runs from http://autopkgtest.ubuntu.com/packages/systemd/hirsute/amd64 , with the last one being from

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-24 Thread Balint Reczey
@paelzer I haven't made changes related to the failure rate. I guess the infra got lighter load for some time. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1915126 Title: autopkgtest times out (or

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-16 Thread Christian Ehrhardt 
Also @rbalint - it seems that the success rate with 247.3-1ubuntu2 got a bit better. Were there changes in the upload or was it actually converted to be a big_test already ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-16 Thread Christian Ehrhardt 
Thanks Laney for chiming in here (as one of the affected/blocked I appreciate every help). I was last week already trying that myself on canonistack and for me it didn't reproduce. This is the command I used: autopkgtest --no-built-binaries --apt-upgrade --testname=systemd-fsckd

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-12 Thread Iain Lane
Ok, here you go. I reproduced the (or a) failure in Canonistack. This was the first run FWIW, so it wasn't too hard to get it to fail: autopkgtest [14:05:12]: test systemd-fsckd: [--- test_fsck_failure (__main__.FsckdTest) Ensure that a failing fsck doesn't prevent fsckd to

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-11 Thread Iain Lane
I'm running a test on Canonistack. I'm hoping that this will reproduce the failures we're seeing, and then they can be investigated. That is closer to what is running in production than the QEMU runner. I don't think we have evidence that this is an 'infra issue', so I'm marking the

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-11 Thread Balint Reczey
** Tags added: update-excuse -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1915126 Title: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally To manage

Re: [Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-11 Thread Iain Lane
On Thu, Feb 11, 2021 at 01:49:45PM -, Balint Reczey wrote: > There is not much to do about the failures on systemd's side. The tests > are reliably passing in qemu on my i5-4670 testbed from 2013, also > passing on @paelzer's presumably newer machine. > > This is an infra issue. The Release

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-11 Thread Balint Reczey
There is not much to do about the failures on systemd's side. The tests are reliably passing in qemu on my i5-4670 testbed from 2013, also passing on @paelzer's presumably newer machine. This is an infra issue. The Release Team can decide if they want to hint the test and lose gating coverage,

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-09 Thread Balint Reczey
** Also affects: auto-package-testing Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1915126 Title: autopkgtest times out (or fails before that) in

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-09 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~rbalint/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/397740 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1915126 Title: autopkgtest

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-09 Thread Balint Reczey
** Merge proposal unlinked: https://code.launchpad.net/~rbalint/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/383953 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1915126 Title: autopkgtest

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute on test infra, passes locally

2021-02-09 Thread Balint Reczey
@paelzer I seems infra issues got worse, but there is a pending review to bump systemd to big which I linked. Even in the absence of the infra issues it would have made sense to make system run big test instances, but I expect the big instances would also improve the current situation. Instead

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute

2021-02-09 Thread Balint Reczey
** Merge proposal linked: https://code.launchpad.net/~rbalint/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/383953 ** Summary changed: - autopkgtest times out (or fails before that) in hirsute + autopkgtest times out (or fails before that) in hirsute on test infra, passes locally -- You

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute

2021-02-09 Thread Christian Ehrhardt 
Running the same on a Canonistack instance creates "different" issues and won't help much debugging it. Locally it works fine even on multiple re-runs. @rbalint / others that feel systemd'ish: What shall we do about this short term to resolve all blocking packages? -- You received this bug

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute

2021-02-09 Thread Christian Ehrhardt 
>From the log: 402209 Checking in progress on 1 disk (92.7% complete)fsckd:1:92.8:Checking in progress on 1 disk (92.8% complete) 402210 Checking in progress on 1 disk (92.8% complete)fsckd:1:92.9:Checking in progress on 1 disk (92.9% complete) 402211 Checking in progress on 1 disk (92.9%

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute

2021-02-09 Thread Christian Ehrhardt 
On average the upstream tests loose ~55 minutes compared to former runs (~90 -> ~145 minutes). Also that test times out: autopkgtest [20:14:12]: test upstream: ---] upstream FAIL timed out Also the later test for systemd-fsckd hangs until

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute

2021-02-09 Thread Christian Ehrhardt 
I've compared older good and newer runs that timeout. $ for i in good.hirsute-amd64.*; do printf "\n\n$i\n"; grep '^autopkgtest .* test .*: - - - - - - - - - - results' $i; done We see that good cases need ~110-130 minutes to complete "test boot- smoke" and then another 4-12 minutes to

[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute

2021-02-09 Thread Christian Ehrhardt 
In our worker config we see https://git.launchpad.net/autopkgtest-cloud: long_tests timeout => ~11h big_pkg timeout => ~5.5h default timeout => ~3h Currently systemd is only marked as long_test on ppc64el, so ~3h is what applies to us here. The systemd runs that failed with timeouts are all in