Bug#898216: unattended-upgrades: flaky autopkgtest

[Paul Gevers]

Hi

On 05-09-18 22:13, Paul Gevers wrote:
> On Fri, 6 Jul 2018 14:06:32 +0200 =?UTF-8?B?QsOhbGludCBSw6ljemV5?=
>  wrote:
>> I monitored the runs for some time and the test does not seem to be
>> flaky in unstable. Since I added a new test with debootstrap I keep
>> monitoring it and it shows to be flaky I may add retries somewhere.

And now involved in delaying a package due to a regression:
https://ci.debian.net/data/autopkgtest/testing/amd64/u/unattended-upgrades/1061759/log.gz

Paul



signature.asc
Description: OpenPGP digital signature

Bug#898216: unattended-upgrades: flaky autopkgtest

[Réczey Bálint]

On Fri, Sep 21, 2018, 21:00 Paul Gevers  wrote:

> Hi Balint,
>
> On 05-09-18 22:13, Paul Gevers wrote:
> > On Fri, 6 Jul 2018 14:06:32 +0200 =?UTF-8?B?QsOhbGludCBSw6ljemV5?=
> >  wrote:
> >> I monitored the runs for some time and the test does not seem to be
> >> flaky in unstable. Since I added a new test with debootstrap I keep
> >> monitoring it and it shows to be flaky I may add retries somewhere.
> >
> > September 2, 2018:
> >
> https://ci.debian.net/data/autopkgtest/testing/amd64/u/unattended-upgrades/926140/log.gz
> >
> > In unstable August 23, 2018:
> >
> https://ci.debian.net/data/autopkgtest/unstable/amd64/u/unattended-upgrades/871383/log.gz
>
> unattended-upgrades is failing now in testing with python-apt/1.7.0~rc1
> twice in a row. Can you check that this is a real regression or again
> just unattended-upgrades being flaky?
>

It is, I have the fix waiting for review on GitHub.

Cheers,
Balint


> Paul
>
>

Bug#898216: unattended-upgrades: flaky autopkgtest

[Paul Gevers]

Hi Balint,

On 05-09-18 22:13, Paul Gevers wrote:
> On Fri, 6 Jul 2018 14:06:32 +0200 =?UTF-8?B?QsOhbGludCBSw6ljemV5?=
>  wrote:
>> I monitored the runs for some time and the test does not seem to be
>> flaky in unstable. Since I added a new test with debootstrap I keep
>> monitoring it and it shows to be flaky I may add retries somewhere.
> 
> September 2, 2018:
> https://ci.debian.net/data/autopkgtest/testing/amd64/u/unattended-upgrades/926140/log.gz
> 
> In unstable August 23, 2018:
> https://ci.debian.net/data/autopkgtest/unstable/amd64/u/unattended-upgrades/871383/log.gz

unattended-upgrades is failing now in testing with python-apt/1.7.0~rc1
twice in a row. Can you check that this is a real regression or again
just unattended-upgrades being flaky?

Paul



signature.asc
Description: OpenPGP digital signature

Bug#898216: unattended-upgrades: flaky autopkgtest

[Paul Gevers]

On Fri, 6 Jul 2018 14:06:32 +0200 =?UTF-8?B?QsOhbGludCBSw6ljemV5?=
 wrote:
> I monitored the runs for some time and the test does not seem to be
> flaky in unstable. Since I added a new test with debootstrap I keep
> monitoring it and it shows to be flaky I may add retries somewhere.

September 2, 2018:
https://ci.debian.net/data/autopkgtest/testing/amd64/u/unattended-upgrades/926140/log.gz

In unstable August 23, 2018:
https://ci.debian.net/data/autopkgtest/unstable/amd64/u/unattended-upgrades/871383/log.gz

Paul



signature.asc
Description: OpenPGP digital signature

Bug#898216: unattended-upgrades: flaky autopkgtest

[Bálint Réczey]

Hi Antonio,

2018-05-11 19:45 GMT+02:00 Antonio Terceiro :
> On Fri, May 11, 2018 at 04:41:11PM +0200, Bálint Réczey wrote:
>> Control: tags -1 wontfix
>>
>> Hi Paul,
>>
>> 2018-05-08 21:48 GMT+02:00 Paul Gevers :
>> > Source: unattended-upgrades
>> > Version: 1.0
>> > Severity: important
>> > User: debian...@lists.debian.org
>> > Usertags: flaky
>> >
>> > While inspecting regressions in autopkgtest results¹, I noticed that
>> > your package unattended-upgrades fails regularly, without obvious
>> > changes. There are multiple failure modes (some copied below).
>>
>> Thank you for working on autopkgtest gating migration!
>>
>> > Could you please investigate and make your autopkgtest more robust?
>> > Please contact me if you need help and you think I can provide that (I
>> > am not subscribed to this bug).
>>
>> Checking the logs the failures seem to be infrastructure issues, most
>> often temporary problems with downloading files from
>> snapshot.debian.org (via local proxy when it is present). IMO those
>> should be fixed by monitoring the logs and retrying the test
>> automatically rather than working around the issues in u-u/test
>> itself.  An even better fix would be fixing the proxy/snapshot.d.o,
>> but this may be out of scope for the autopkgtest infrastructure.
>
> autopkgtest will retry on failures when it is accessing the network
> (such as installing test dependencies), but when your test script hits
> the network (in this case by calling deboootstrap), you must make sure
> it can survive temporary network access problems. This type of failure
> will happen occasionaly on ci.debian.net, but also on developers
> machines, build daemons, etc, so it's not realistic to expect each of
> those to monitor logs and retry arbitrary/random network access
> failures.
>
> Also, it would not be feasible for autopkgtest or debci to automatically
> detect arbitrary network failures in 10k+ source packages.
>
> Please reconsider, and make the tests robust against networking issues.

I monitored the runs for some time and the test does not seem to be
flaky in unstable. Since I added a new test with debootstrap I keep
monitoring it and it shows to be flaky I may add retries somewhere.

Cheers,
Balint

Bug#898216: unattended-upgrades: flaky autopkgtest

[Antonio Terceiro]

On Fri, May 11, 2018 at 04:41:11PM +0200, Bálint Réczey wrote:
> Control: tags -1 wontfix
> 
> Hi Paul,
> 
> 2018-05-08 21:48 GMT+02:00 Paul Gevers :
> > Source: unattended-upgrades
> > Version: 1.0
> > Severity: important
> > User: debian...@lists.debian.org
> > Usertags: flaky
> >
> > While inspecting regressions in autopkgtest results¹, I noticed that
> > your package unattended-upgrades fails regularly, without obvious
> > changes. There are multiple failure modes (some copied below).
> 
> Thank you for working on autopkgtest gating migration!
> 
> > Could you please investigate and make your autopkgtest more robust?
> > Please contact me if you need help and you think I can provide that (I
> > am not subscribed to this bug).
> 
> Checking the logs the failures seem to be infrastructure issues, most
> often temporary problems with downloading files from
> snapshot.debian.org (via local proxy when it is present). IMO those
> should be fixed by monitoring the logs and retrying the test
> automatically rather than working around the issues in u-u/test
> itself.  An even better fix would be fixing the proxy/snapshot.d.o,
> but this may be out of scope for the autopkgtest infrastructure.

autopkgtest will retry on failures when it is accessing the network
(such as installing test dependencies), but when your test script hits
the network (in this case by calling deboootstrap), you must make sure
it can survive temporary network access problems. This type of failure
will happen occasionaly on ci.debian.net, but also on developers
machines, build daemons, etc, so it's not realistic to expect each of
those to monitor logs and retry arbitrary/random network access
failures.

Also, it would not be feasible for autopkgtest or debci to automatically
detect arbitrary network failures in 10k+ source packages.

Please reconsider, and make the tests robust against networking issues.


signature.asc
Description: PGP signature

Bug#898216: unattended-upgrades: flaky autopkgtest

[Bálint Réczey]

Control: tags -1 wontfix

Hi Paul,

2018-05-08 21:48 GMT+02:00 Paul Gevers :
> Source: unattended-upgrades
> Version: 1.0
> Severity: important
> User: debian...@lists.debian.org
> Usertags: flaky
>
> While inspecting regressions in autopkgtest results¹, I noticed that
> your package unattended-upgrades fails regularly, without obvious
> changes. There are multiple failure modes (some copied below).

Thank you for working on autopkgtest gating migration!

> Could you please investigate and make your autopkgtest more robust?
> Please contact me if you need help and you think I can provide that (I
> am not subscribed to this bug).

Checking the logs the failures seem to be infrastructure issues, most
often temporary problems with downloading files from
snapshot.debian.org (via local proxy when it is present). IMO those
should be fixed by monitoring the logs and retrying the test
automatically rather than working around the issues in u-u/test
itself. An even better fix would be fixing the proxy/snapshot.d.o, but
this may be out of scope for the autopkgtest infrastructure.

Please note that a major reliability bug got fixed in 1.1 which caused
a lot of flakiness and crashes in the field, too.

> Recent discussion of gating migration by autopkgtests on debian-devel²
> noted that if this is going to work, and in particular if we are going
> to *block* migration when it causes autopkgtest regressions rather than
> merely delaying it, intermittent autopkgtest failures are likely to have
> to be considered RC due to their impact on the tested package's
> dependencies; for now I've filed it as important.

I support making the delay very long or even blocking migration, but
based on my experience flaky tests can easily be caused by
infrastructure issues, and in those cases important would still be a
more reasonable severity as infrastructure issues tend to persist for
long time (unfortunately).

>
> Simon McVittie has previously proposed³ a way to mark autopkgtests as
> unsuitable for gating CI, while still having them available for
> convenient manual testing (and optionally run, but with their results
> disregarded, on the real CI infrastructure so that their reliability can
> be assessed), but this isn't currently possible. If the autopkgtest
> can't be made reliable then it would make sense to disable it for the
> moment.

I believe the current reliability level would not warrant disabling
this test since u-u does not have a lot of dependencies and
re-triggering the test is easy.
I also find the Britney overrides a good way of tackling with
flaky/failing tests following Martin's recommendation in ³.

Cheers,
Balint

>
> Paul
>
> ¹ https://ci.debian.net/packages/u/unattended-upgrades
> ² https://lists.debian.org/debian-devel/2018/05/msg00061.html
> ³ https://bugs.debian.org/851558
>
> https://ci.debian.net/data/autopkgtest/unstable/amd64/u/unattended-upgrades/250960/log.gz
>
> I: Retrieving InRelease
> I: Retrieving Release
> I: Retrieving Release.gpg
> E: Failed getting release signature file
> http://snapshot.debian.org/archive/debian/20171001T00Z//dists/sid/Release.gpg
> autopkgtest [05:43:53]: test upgrade-between-snapshots:
> ---]
>
>
> https://ci.debian.net/data/autopkgtest/unstable/amd64/u/unattended-upgrades/220435/log.gz
>
> Get:391 http://snapshot.debian.org/archive/debian/20171001T00Z
> sid/main amd64 xserver-xorg-legacy amd64 2:1.19.3-2 [2069 kB]
> E: Failed to fetch
> http://snapshot.debian.org/archive/debian/20171001T00Z/pool/main/x/xorg-server/xserver-xorg-core_1.19.3-2_amd64.deb
>  503  Backend fetch failed [IP: 193.62.202.30 80]
> E: Unable to fetch some archives, maybe run apt-get update or try with
> --fix-missing?
> Fetched 189 MB in 16min 5s (195 kB/s)
> autopkgtest [06:15:55]: test upgrade-between-snapshots:
> ---]
>
>
> https://ci.debian.net/data/autopkgtest/testing/amd64/u/unattended-upgrades/226860/log.gz
>
> ls: cannot access
> '/tmp/autopkgtest-lxc.eglx_gdx/downtmp/autopkgtest_tmp/chroot': No such
> file or directory
> I: Retrieving InRelease
> I: Checking Release signature
> I: Valid Release signature (key id 126C0D24BD8A2942CC7DF8AC7638D0442B90D010)
> I: Retrieving Packages
> I: Retrieving Packages
> I: Retrieving Packages
> I: Retrieving Packages
> E: unknown location unstable/dists/sid/main/binary-amd64/Packages.xz
> autopkgtest [08:24:49]: test upgrade-between-snapshots:
> ---]
>
>
> https://ci.debian.net/data/autopkgtest/testing/amd64/u/unattended-upgrades/225434/log.gz
>
> autopkgtest [02:09:26]: test upgrade-between-snapshots:  - - - - - - - -
> - - results - - - - - - - - - -
> upgrade-between-snapshots FAIL stderr: ls: cannot access
> '/tmp/autopkgtest-lxc.4kr__7dt/downtmp/autopkgtest_tmp/chroot': No such
> file or directory
> autopkgtest [02:09:26]: test upgrade-between-snapshots:  - - - - - - - -
> - - stderr - - - - - - - - - -
> ls: cannot access
> 

Bug#898216: unattended-upgrades: flaky autopkgtest

[Paul Gevers]

Source: unattended-upgrades
Version: 1.0
Severity: important
User: debian...@lists.debian.org
Usertags: flaky

While inspecting regressions in autopkgtest results¹, I noticed that
your package unattended-upgrades fails regularly, without obvious
changes. There are multiple failure modes (some copied below).

Could you please investigate and make your autopkgtest more robust?
Please contact me if you need help and you think I can provide that (I
am not subscribed to this bug).

Recent discussion of gating migration by autopkgtests on debian-devel²
noted that if this is going to work, and in particular if we are going
to *block* migration when it causes autopkgtest regressions rather than
merely delaying it, intermittent autopkgtest failures are likely to have
to be considered RC due to their impact on the tested package's
dependencies; for now I've filed it as important.

Simon McVittie has previously proposed³ a way to mark autopkgtests as
unsuitable for gating CI, while still having them available for
convenient manual testing (and optionally run, but with their results
disregarded, on the real CI infrastructure so that their reliability can
be assessed), but this isn't currently possible. If the autopkgtest
can't be made reliable then it would make sense to disable it for the
moment.

Paul

¹ https://ci.debian.net/packages/u/unattended-upgrades
² https://lists.debian.org/debian-devel/2018/05/msg00061.html
³ https://bugs.debian.org/851558

https://ci.debian.net/data/autopkgtest/unstable/amd64/u/unattended-upgrades/250960/log.gz

I: Retrieving InRelease
I: Retrieving Release
I: Retrieving Release.gpg
E: Failed getting release signature file
http://snapshot.debian.org/archive/debian/20171001T00Z//dists/sid/Release.gpg
autopkgtest [05:43:53]: test upgrade-between-snapshots:
---]


https://ci.debian.net/data/autopkgtest/unstable/amd64/u/unattended-upgrades/220435/log.gz

Get:391 http://snapshot.debian.org/archive/debian/20171001T00Z
sid/main amd64 xserver-xorg-legacy amd64 2:1.19.3-2 [2069 kB]
E: Failed to fetch
http://snapshot.debian.org/archive/debian/20171001T00Z/pool/main/x/xorg-server/xserver-xorg-core_1.19.3-2_amd64.deb
 503  Backend fetch failed [IP: 193.62.202.30 80]
E: Unable to fetch some archives, maybe run apt-get update or try with
--fix-missing?
Fetched 189 MB in 16min 5s (195 kB/s)
autopkgtest [06:15:55]: test upgrade-between-snapshots:
---]


https://ci.debian.net/data/autopkgtest/testing/amd64/u/unattended-upgrades/226860/log.gz

ls: cannot access
'/tmp/autopkgtest-lxc.eglx_gdx/downtmp/autopkgtest_tmp/chroot': No such
file or directory
I: Retrieving InRelease
I: Checking Release signature
I: Valid Release signature (key id 126C0D24BD8A2942CC7DF8AC7638D0442B90D010)
I: Retrieving Packages
I: Retrieving Packages
I: Retrieving Packages
I: Retrieving Packages
E: unknown location unstable/dists/sid/main/binary-amd64/Packages.xz
autopkgtest [08:24:49]: test upgrade-between-snapshots:
---]


https://ci.debian.net/data/autopkgtest/testing/amd64/u/unattended-upgrades/225434/log.gz

autopkgtest [02:09:26]: test upgrade-between-snapshots:  - - - - - - - -
- - results - - - - - - - - - -
upgrade-between-snapshots FAIL stderr: ls: cannot access
'/tmp/autopkgtest-lxc.4kr__7dt/downtmp/autopkgtest_tmp/chroot': No such
file or directory
autopkgtest [02:09:26]: test upgrade-between-snapshots:  - - - - - - - -
- - stderr - - - - - - - - - -
ls: cannot access
'/tmp/autopkgtest-lxc.4kr__7dt/downtmp/autopkgtest_tmp/chroot': No such
file or directory



signature.asc
Description: OpenPGP digital signature