Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Paul Gevers]

Hi Sudip

On 22-09-2020 20:57, Sudip Mukherjee wrote:
> And, the final version (unless someone suggests some change):

[...]

> Executing that command is considered to be a trivial test, that
> which does not provide significant coverage for a package as a whole.

I'm not 100% sure as I'm not a native speaker, but s/that which/which/
or s/that which/that/ sounds much better to me.

[...]

> be increased to serious in future.
 ^ the ?

[...]

> If the above sounds good, I will continue with this.

Thanks.

Paul



signature.asc
Description: OpenPGP digital signature

Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Sudip Mukherjee]

On Tue, Sep 22, 2020 at 9:02 AM Holger Levsen  wrote:
>
> On Sat, Sep 19, 2020 at 08:39:44PM +0100, Sudip Mukherjee wrote:
> > After discussing with few people, I now intend to file them with
> > "severity: important" and I will also reduce the severity of the
> > previously open similar bugs to 'important'.
>
> thank you, for all your work on this! (which includes these discussions! :)

Thanks to everyone for helping me with the bug text.
And, the final version (unless someone suggests some change):

*
Subject: : autopkgtest must be marked superficial

Severity: important
User: sudipm.mukher...@gmail.com
Usertags: superficialtest

It has been noticed that the autopkgtest in  is running a
trivial command that does not provide significant test coverage:

 - 

Executing that command is considered to be a trivial test, that
which does not provide significant coverage for a package as a whole.
But these tests are a useful way to detect regressions in dependencies
and prevent them from breaking your package.

However, it is important that we are realistic about the level of
test coverage provided by these commands: most regressions cannot be
detected in this way. So it is not appropriate for packages with only
superficial tests to have the reduced migration time to migrate from
unstable to testing as that means less
opportunity for testing by users compared to the package with no tests.

To support this, the keyword "Restrictions: superficial" has been
defined [1]. Packages where all tests are marked with this keyword are not
considered for the reduced migration age from unstable to testing, and
will not be allowed to migrate automatically in later stages of the
freeze [2].

Its always better to have more extensive testing than having
superficial testing, which again is better than having no test.

Please consider i) Adding a non-trivial test, and/or ii) Mark the
trivial test with "Restrictions: superficial", similar to
[3] or [4].

The Release Team has listed this issue in the list of Release Critical
Issues for bullseye [5] and has mentioned that the test must be marked
superficial if it is not testing one of its own installed binary
packages in some way. As a result, the severity of this bug report might
be increased to serious in future.

[1] 
https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst#defined-restrictions
[2] https://release.debian.org/bullseye/freeze_policy.html
[3] 
https://salsa.debian.org/utopia-team/dbus/-/commit/a80908df7d119b181eec5eb0542634a30c2ad468
[4] 
https://salsa.debian.org/apparmor-team/apparmor/-/commit/580667513a097088ebe579884b38ac8d8666d3b3
[5] https://release.debian.org/bullseye/rc_policy.txt
*

If the above sounds good, I will continue with this.


-- 
Regards
Sudip

Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Holger Levsen]

On Sat, Sep 19, 2020 at 08:39:44PM +0100, Sudip Mukherjee wrote:
> After discussing with few people, I now intend to file them with
> "severity: important" and I will also reduce the severity of the
> previously open similar bugs to 'important'.

thank you, for all your work on this! (which includes these discussions! :)


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

"Climate change" is an euphenism. "Global warming" as well.


signature.asc
Description: PGP signature

Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Sudip Mukherjee]

On Mon, Sep 21, 2020 at 2:53 PM Simon McVittie  wrote:
>
> On Mon, 21 Sep 2020 at 09:09:51 -0300, Antonio Terceiro wrote:
> > Maybe you could include something like this (the wording can be improved):
> >
> >   Note, however, that such superficial tests are still somewhat useful,
> >   as they will be considered, for example, to block dependencies from
> >   breaking your package. In other words, please do not react to this bug
> >   report by dropping tests from your package completely. More extensive
> >   testing is of course better, but even superficial tests are better for
> >   the overal quality of Debian than no tests at all.
>
> Perhaps a more positive way to phrase the bug report would be to lead with
> this, and then go on to say why these tests should be marked superficial?
> That will hopefully reduce the tendency for maintainers to remove tests in
> response.
>
> It's also really useful for the template that is discussed on -devel to
> mention the usertags that are going to be used (if any), so that it's
> easy to search for them in a machine-readable way. It looks as though
> the bugs that have already been filed are usertagged to appear in
> 
> so let's go with that.

Those were from the first list and there was another MBF mail to debian-devel.

>
> In cases like this where the change is so trivial to make and so similar
> across multiple packages, it's probably also useful to link to examples
> of a maintainer making this change correctly, like [3] and [4] below.
>
> Maybe something like this (I'm making some assumptions here about what the
> release team does and doesn't encourage, so don't actually use this wording
> until someone from the RT has acknowledged it):

I was drafting something like the following:
*

Subject: : autopkgtest must be marked superficial

Dear Maintainer,

Your package has an autopkgtest and its executing the following command:
-  

Since executing that command is considered to be a trivial test, that
which does not provide significant coverage for a package as a whole.
All trivial tests must be marked with "Restrictions: superficial" as
defined in [1].

A package with a non-trivial autopkgtest will enjoy a reduced
migration age from unstable to testing, and also it will be allowed to
migrate in a later stage of the freeze [2].

A package having a trivial (superficial) test will not get the
migration advantage but it will have the advantage of blocking
dependencies from breaking your package.

Its always better to have more extensive testing than having
superficial testing, which again is better than having no test.

Please consider i) Adding a non-trivial test, and/or ii) Mark the
trivial test with "Restrictions: superficial".


[1]. 
https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
[2]. https://release.debian.org/bullseye/freeze_policy.html


*

But yours is much better than what I drafted.


-- 
Regards
Sudip

Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Simon McVittie]

On Mon, 21 Sep 2020 at 09:09:51 -0300, Antonio Terceiro wrote:
> Maybe you could include something like this (the wording can be improved):
> 
>   Note, however, that such superficial tests are still somewhat useful,
>   as they will be considered, for example, to block dependencies from
>   breaking your package. In other words, please do not react to this bug
>   report by dropping tests from your package completely. More extensive
>   testing is of course better, but even superficial tests are better for
>   the overal quality of Debian than no tests at all.

Perhaps a more positive way to phrase the bug report would be to lead with
this, and then go on to say why these tests should be marked superficial?
That will hopefully reduce the tendency for maintainers to remove tests in
response.

It's also really useful for the template that is discussed on -devel to
mention the usertags that are going to be used (if any), so that it's
easy to search for them in a machine-readable way. It looks as though
the bugs that have already been filed are usertagged to appear in

so let's go with that.

In cases like this where the change is so trivial to make and so similar
across multiple packages, it's probably also useful to link to examples
of a maintainer making this change correctly, like [3] and [4] below.

Maybe something like this (I'm making some assumptions here about what the
release team does and doesn't encourage, so don't actually use this wording
until someone from the RT has acknowledged it):

8<
Subject: : autopkgtest must be marked superficial

Severity: important
User: sudipm.mukher...@gmail.com
Usertags: superficialtest

It has been noticed that the autopkgtest in  is running a
trivial command that does not provide significant test coverage:

 - 

These superficial tests are a useful way to detect regressions in
dependencies and prevent them from breaking your package, and the
Release Team encourages their inclusion in packages that do not have
a more thorough test suite available.

However, it is important that we are realistic about the level of
test coverage provided by these commands: most regressions cannot be
detected in this way, so it is not appropriate for packages that only
have superficial tests to migrate to the testing distribution with less
opportunity for testing by users than if these tests had not been present.

To support this, the keyword "Restrictions: superficial" has been
defined [1]. Packages where all tests are marked with this keyword are not
considered for the reduced migration age from unstable to testing, and
will not be allowed to migrate automatically in later stages of the
freeze [2].

Please mark all superficial autopkgtests with this keyword, similar to
[3] or [4].

The Release Team has listed this issue in the list of Release Critical
Issues for bullseye [5] and has mentioned that the test must be marked
superficial if it is not testing one of its own installed binary
packages in some way. As a result, the severity of this bug report might
be increased to serious in future.

[1] 
https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst#defined-restrictions
[2] https://release.debian.org/bullseye/freeze_policy.html
[3] 
https://salsa.debian.org/utopia-team/dbus/-/commit/a80908df7d119b181eec5eb0542634a30c2ad468
[4] 
https://salsa.debian.org/apparmor-team/apparmor/-/commit/580667513a097088ebe579884b38ac8d8666d3b3
[5] https://release.debian.org/bullseye/rc_policy.txt
8<

Regards,
smcv

Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Antonio Terceiro]

On Sun, Sep 20, 2020 at 12:31:13AM +0100, Sudip Mukherjee wrote:
> HI Mattia,
> 
> On Sat, Sep 19, 2020 at 8:58 PM Mattia Rizzolo  wrote:
> >
> > On Sat, Sep 19, 2020 at 08:39:44PM +0100, Sudip Mukherjee wrote:
> > > After discussing with few people, I now intend to file them with
> > > "severity: important" and I will also reduce the severity of the
> > > previously open similar bugs to 'important'.
> >
> > That's good.
> >
> > But please also share your proposed text with this list (as the MBF
> > rules asks for).  Your past filings where IMHO written with a tone that
> > could be improved.  Also I would like to make sure that you include
> > stuff like your plans with the severity, references to the release team
> > decisions, etc.
> 
> Apologies for the tone. It was my first MBF and I was struggling to
> make the text more generalised so that it will work for all the
> packages in the list.
> For this new list of packages, how is the following text:
> 
> **
> 
> Subject: : autopkgtest must be marked superficial
> severity: important
> 
> Dear maintainer,
> 
> It has been noticed that the autpkgtest in  is running a
> trivial command.
>  - 
> 
> Those kind of tests are considered to not provide significant coverage
> for a package as a whole, and as such the keyword "Restrictions:
> superficial" has been defined [1]. Packages with all tests marked as
> 'superficial' are not considered for the reduced migration age from
> unstable to testing and also they will not be allowed to migrate in a
> later stage of the freeze [2].
>
> The Release Team has listed this issue in the list of Release Critical
> Issues for bullseye [3] and has mentioned that the test must be marked
> superficial if it is not testing one of its own installed binary
> packages in some way.
> 
> 
> [1]. 
> https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
> [2]. https://release.debian.org/bullseye/freeze_policy.html
> [3]. https://release.debian.org/bullseye/rc_policy.txt

Maybe you could include something like this (the wording can be improved):

  Note, however, that such superficial tests are still somewhat useful,
  as they will be considered, for example, to block dependencies from
  breaking your package. In other words, please do not react to this bug
  report by dropping tests from your package completely. More extensive
  testing is of course better, but even superficial tests are better for
  the overal quality of Debian than no tests at all.

We want to avoid maintainers panicking and just dropping the tests as a
response do this MBF.


signature.asc
Description: PGP signature

Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Sudip Mukherjee]

HI Mattia,

On Sat, Sep 19, 2020 at 8:58 PM Mattia Rizzolo  wrote:
>
> On Sat, Sep 19, 2020 at 08:39:44PM +0100, Sudip Mukherjee wrote:
> > After discussing with few people, I now intend to file them with
> > "severity: important" and I will also reduce the severity of the
> > previously open similar bugs to 'important'.
>
> That's good.
>
> But please also share your proposed text with this list (as the MBF
> rules asks for).  Your past filings where IMHO written with a tone that
> could be improved.  Also I would like to make sure that you include
> stuff like your plans with the severity, references to the release team
> decisions, etc.

Apologies for the tone. It was my first MBF and I was struggling to
make the text more generalised so that it will work for all the
packages in the list.
For this new list of packages, how is the following text:

**

Subject: : autopkgtest must be marked superficial
severity: important

Dear maintainer,

It has been noticed that the autpkgtest in  is running a
trivial command.
 - 

Those kind of tests are considered to not provide significant coverage
for a package as a whole, and as such the keyword "Restrictions:
superficial" has been defined [1]. Packages with all tests marked as
'superficial' are not considered for the reduced migration age from
unstable to testing and also they will not be allowed to migrate in a
later stage of the freeze [2].

The Release Team has listed this issue in the list of Release Critical
Issues for bullseye [3] and has mentioned that the test must be marked
superficial if it is not testing one of its own installed binary
packages in some way.


[1]. 
https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
[2]. https://release.debian.org/bullseye/freeze_policy.html
[3]. https://release.debian.org/bullseye/rc_policy.txt

**



-- 
Regards
Sudip

Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Mattia Rizzolo]

On Sat, Sep 19, 2020 at 08:39:44PM +0100, Sudip Mukherjee wrote:
> After discussing with few people, I now intend to file them with
> "severity: important" and I will also reduce the severity of the
> previously open similar bugs to 'important'.

That's good.

But please also share your proposed text with this list (as the MBF
rules asks for).  Your past filings where IMHO written with a tone that
could be improved.  Also I would like to make sure that you include
stuff like your plans with the severity, references to the release team
decisions, etc.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Sudip Mukherjee]

Hi All,

On Fri, Sep 18, 2020 at 9:21 PM Sudip Mukherjee
 wrote:
>
> Hi All,
>
> If the test done in the autopkgtest does not provide significant test
> coverage then it should be marked with "Restrictions: superficial".
> Ref: 
> https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
>

>
> I intend to file them with "severity: serious"

After discussing with few people, I now intend to file them with
"severity: important" and I will also reduce the severity of the
previously open similar bugs to 'important'.


-- 
Regards
Sudip

Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Sudip Mukherjee]

HI Holger,

On Fri, Sep 18, 2020 at 11:02 PM Holger Levsen  wrote:
>
> On Fri, Sep 18, 2020 at 09:21:52PM +0100, Sudip Mukherjee wrote:
> > If the test done in the autopkgtest does not provide significant test
> > coverage then it should be marked with "Restrictions: superficial".
> > Ref: 
> > https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
> [...]
> > I intend to file them with "severity: serious" [...]
>
> sigh. I too would appreciate if these bugs were filed with severity:important
> in the first place, not the least to avoid discussions like these. seriously,
> these packages are only cosmetically seriously buggy and important (and 
> normal)
> bugs should be fixed as well, anyhow. so why inflate the severity and 
> discussions
> if the same can be achieved cheaper and much nicer by just using lower 
> severity?
>
> please use important and announce to bump the severity (to serious) in 1-3 
> months
> from now, as these bugs really must not make it into bullseye! but give people
> some slack to support your cause. an important bug and being listed on d-devel
> is serious enough.

I discussed exactly this with Paul before sending the mail. So, if I
raise the bug as 'important' on 22/09 and then bump it to 'serious' on
22/11 then the autoremoval date will be set to 22/12 which will be
less than 2 months before the soft freeze. If the Release Team is ok
with having a bunch of serious bug related to autopkgtest 50 days
before the soft freeze then I can make them important now.
The world will be more on fire at that time. :)

But, just to confirm, have you also seen the last part of my mail,
that the severity can be reduced after it has been fixed in git so
that it will definitely be fixed in the archive with the next upload?


--
Regards
Sudip

Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Holger Levsen]

On Fri, Sep 18, 2020 at 09:21:52PM +0100, Sudip Mukherjee wrote:
[...]

sigh. I forgot to thank you for all the work you put into this! I *very*
much appreciate good tests and your work to improve the quality of
autopkgtests! I'm sorry this severity detail distracted me from expressing
this.

So: whatever severity you choose, thank you for all these bugs! :)


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: Mass bugs filing: autopkgtest should be marked superficial (new list)

[Holger Levsen]

On Fri, Sep 18, 2020 at 09:21:52PM +0100, Sudip Mukherjee wrote:
> If the test done in the autopkgtest does not provide significant test
> coverage then it should be marked with "Restrictions: superficial".
> Ref: 
> https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
[...]
> I intend to file them with "severity: serious" [...]

sigh. I too would appreciate if these bugs were filed with severity:important
in the first place, not the least to avoid discussions like these. seriously,
these packages are only cosmetically seriously buggy and important (and normal)
bugs should be fixed as well, anyhow. so why inflate the severity and 
discussions
if the same can be achieved cheaper and much nicer by just using lower severity?

please use important and announce to bump the severity (to serious) in 1-3 
months
from now, as these bugs really must not make it into bullseye! but give people
some slack to support your cause. an important bug and being listed on d-devel
is serious enough.

the world is on fire and these are just superficial tests.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: Mass bugs filing: autopkgtest should be marked superficial

[Thomas Goirand]

On 9/17/20 10:54 AM, Adam D. Barratt wrote:
> That's not the only possible reason for a bug to have a severity of
> "serious".
> 
> These issues do violate the RC Policy for bullseye, which means that
> each "in the ... release manager's opinion, makes the package
> unsuitable for release".

If that's the release team opinion, then I respectfully disagree with
the release team.

Cheers,

Thomas Goirand (zigo)

Re: Mass bugs filing: autopkgtest should be marked superficial

[Thomas Goirand]

On 9/4/20 8:52 PM, Sudip Mukherjee wrote:
> Hi All,
> 
> If the test done in the autopkgtest does not provide significant test
> coverage then it should be marked with "Restrictions: superficial".
> Ref: https://people.debian.org/~eriberto/README.package-tests.html
> 
> Examples of tests which are not significant includes (its not a complete 
> list):
> 
> 1) Executing the binary to check version
> Test-Command: foo -v
> Test-Command: foo -V
> Test-Command: foo --version
> 
> 2) Executing the binary to check help (foo -h)
> Test-Command: foo -h
> Test-Command: foo --help
> 
> 3) checking for files installed with 'ls'.
> Test-Command: ls -l /usr/lib/*/foo.so
> 
> 4) A Python or Perl library runs import foo or require Foo; but does
> not attempt to use the library beyond that.
>  Test-Command: python3 -c "import foo"
> 
> I am still trying to figure out a generalized method to find them but
> an initial script has found 83 packages. Attached is the dd-list.
> I intend to open the bug reports on them next week.
> 
> --
> Regards
> Sudip

While I found your work useful, it is my opinion that these bugs didn't
deserve an RC severity. Please read the definition of the severities,
what we're talking about in this thread doesn't match serious:

"is a severe violation of Debian policy"

Also, such a "problem" (if it really is one, and I'm in the opinion that
it isn't, but YMMV) doesn't invalidate the package.

In some cases, I end up maintaining autopkgtest that were contributed.
This is counter-productive, and makes me think I probably should just
remove them if this becomes so annoying to maintain... :(

Cheers,

Thomas Goirand (zigo)

Re: Mass bugs filing: autopkgtest should be marked superficial

[Shengjing Zhu]

On Thu, Sep 17, 2020 at 7:47 PM Paul Gevers  wrote:
>
> Hi all,
>
> On 17-09-2020 13:38, Raphael Hertzog wrote:
> > And consider the case where the bug has been fixed in git but the package
> > has not been uploaded because that small change didn't warrant an upload
> > of its own. When the FTBFS bug pops up, the fix for the autopkgtest will
> > be uploaded.
>
> For all maintainers that have done so, or something similar, by all
> means, you can lower the severity to avoid the autoremoval while marking
> the bug as pending.
>
> > And if it's really something that release managers wants to enforce for
> > buster, the severities can be raised say 2 months before the freeze.
>
> s/buster/bullseye/ ;) Ideally we (the release team) would have a way of
> saying "this is severity important as long as testing and unstable are
> in sync, but it's not OK to upload a new version without fixing this".
> Unfortunately we don't have such an option at this moment. Please adjust
> your bugs in accordance with this idea.
>

It could be one, if someone from RT has time to implement.
For example, tag all these bugs with an usertag, then britney to check that.

-- 
Shengjing Zhu

Re: Mass bugs filing: autopkgtest should be marked superficial

[Matthias Klose]

On 9/17/20 11:12 AM, Ole Streicher wrote:
> "Adam D. Barratt"  writes:
>> On Thu, 2020-09-17 at 09:55 +0200, Ole Streicher wrote:
>>> Graham Inggs  writes:
 On Thu, 17 Sep 2020 at 09:18, Raphael Hertzog 
 wrote:
> Please reduce the severity of all the bugs that you opened to
> "normal" or "minor".

 Why?
>>>
>>> It does not violate the Debian Policy,
>>
>> That's not the only possible reason for a bug to have a severity of
>> "serious".
>>
>> These issues do violate the RC Policy for bullseye, which means that
>> each "in the ... release manager's opinion, makes the package
>> unsuitable for release".
> 
> While the rationale given ("circumvents the testing migration delay")
> may be relevant furing the freeze time, it is irrelevant for a package
> that is now already in testing since months: even if the test would have
> been flagged as "superficial", it would have been migrated since long.

does this mean, we should file RC issues for each package that circumvents any
autopkg test, e.g. all uploads done as binNMUs?  It strikes me that we are
discussing a minor issue, but accept the majority of packages without running
*any* autopkg tests.

Matthias

Re: Mass bugs filing: autopkgtest should be marked superficial

[Raphael Hertzog]

On Thu, 17 Sep 2020, Paul Gevers wrote:
> > And consider the case where the bug has been fixed in git but the package
> > has not been uploaded because that small change didn't warrant an upload
> > of its own. When the FTBFS bug pops up, the fix for the autopkgtest will
> > be uploaded.
> 
> For all maintainers that have done so, or something similar, by all
> means, you can lower the severity to avoid the autoremoval while marking
> the bug as pending.

Thanks, in fact I had done so for pkg-security before replying here
because I really didn't expect that this bug would be covered by
the release-team RC policy.

I just double checked that all the bugs that I lowered were marked as
pending.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS


signature.asc
Description: PGP signature

Re: Mass bugs filing: autopkgtest should be marked superficial

[Raphael Hertzog]

Hi,

On Thu, 17 Sep 2020, Paul Gevers wrote:
> This. I have written it done in response to bug [#969819]:
> 
> Notwithstanding the wording, the Release Team is happy with the bugs
> that Sudip is filing. Because of the way that autopkgtests are used in
> the Debian infrastructure to influence migration from unstable to
> testing [1], it is very important that autopkgtests are recognized for
> what they are. If an autopkgtest isn't really testing the installed
> binaries (and yes, the boundary is unfortunately not well defined) it's
> crucial that the test is marked as superficial, conform our rc_policy
> [2]. The Release Team has decided that the examples that Sudip tagged,
> i.e. --version, --help, checking for some installed file and the Python
> import check, are superficial.

That's very nice, but "superficial" has not been there from the start
so those situations are not necessarily maintainers trying to cheat
with the migration delay, and while I agree with the goal, your release
policy also says "Package are encouraged to implement autopkgtests." and
here you have package maintainers that followed early your suggestion and
that are forced to do busy-work for no gain just to add a superficial
flag.

This is demotivating. For pkg-security, I got 9 "marked for autoremoval"
mails...

Somehow we lack a way to mark a bug as "not urgent but must be fixed in
next upload".

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS


signature.asc
Description: PGP signature

Re: Mass bugs filing: autopkgtest should be marked superficial

[Paul Gevers]

Hi all,

On 17-09-2020 13:38, Raphael Hertzog wrote:
> And consider the case where the bug has been fixed in git but the package
> has not been uploaded because that small change didn't warrant an upload
> of its own. When the FTBFS bug pops up, the fix for the autopkgtest will
> be uploaded.

For all maintainers that have done so, or something similar, by all
means, you can lower the severity to avoid the autoremoval while marking
the bug as pending.

> And if it's really something that release managers wants to enforce for
> buster, the severities can be raised say 2 months before the freeze.

s/buster/bullseye/ ;) Ideally we (the release team) would have a way of
saying "this is severity important as long as testing and unstable are
in sync, but it's not OK to upload a new version without fixing this".
Unfortunately we don't have such an option at this moment. Please adjust
your bugs in accordance with this idea.

Paul



signature.asc
Description: OpenPGP digital signature

Re: Mass bugs filing: autopkgtest should be marked superficial

[Raphael Hertzog]

On Thu, 17 Sep 2020, Sudip Mukherjee wrote:
> i think I will leave it for the Release Team to decide. But just
> consider the scenario when the severity of this bug for a package 'X'
> is reduced and then another FTBFS bug is raised on that same package.
> The FTBFS bug will be fixed and it will have an early migration as the
> test is not (yet) marked "superficial".

And consider the case where the bug has been fixed in git but the package
has not been uploaded because that small change didn't warrant an upload
of its own. When the FTBFS bug pops up, the fix for the autopkgtest will
be uploaded.

And if it's really something that release managers wants to enforce for
buster, the severities can be raised say 2 months before the freeze.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: Mass bugs filing: autopkgtest should be marked superficial

[Paul Gevers]

Hi David,

On 17-09-2020 12:50, David Bremner wrote:
> Paul Gevers  writes:
> OK, that's all very well, I understand the release team needs to do
> things for its own needs. However
> 
> 1) Such an autopkgtest would have prevented an actual RC (as in makes
> the package unusable) bug in a recent upload of ledger

Then, I think you are misunderstanding how it works. A failing
superficial autopkgtest *will* block migration. It's just that you'll
not get the reduced age for a passing one and during the upcoming
bullseye release, your package will not be eligible for the migration
during the hard freeze without a manual unblock by the release team.

> 2) I'm now even less motivated to add autopkgtests.
> 
> So, there can, and will be unintended consequences.  Maybe that's an
> acceptable tradeoff, I don't know.

I think there's consequences. This one I did foresee, albeit I think,
following my answer to 1, it's for the wrong reason.

Paul



signature.asc
Description: OpenPGP digital signature

Re: Mass bugs filing: autopkgtest should be marked superficial

[David Bremner]

Paul Gevers  writes:

>
> Notwithstanding the wording, the Release Team is happy with the bugs
> that Sudip is filing. Because of the way that autopkgtests are used in
> the Debian infrastructure to influence migration from unstable to
> testing [1], it is very important that autopkgtests are recognized for
> what they are. If an autopkgtest isn't really testing the installed
> binaries (and yes, the boundary is unfortunately not well defined) it's
> crucial that the test is marked as superficial, conform our rc_policy
> [2]. The Release Team has decided that the examples that Sudip tagged,
> i.e. --version, --help, checking for some installed file and the Python
> import check, are superficial.

OK, that's all very well, I understand the release team needs to do
things for its own needs. However

1) Such an autopkgtest would have prevented an actual RC (as in makes
the package unusable) bug in a recent upload of ledger

2) I'm now even less motivated to add autopkgtests.

So, there can, and will be unintended consequences.  Maybe that's an
acceptable tradeoff, I don't know.

Don't, as they say in the twit-sphere, @-me.

d


signature.asc
Description: PGP signature

Re: Mass bugs filing: autopkgtest should be marked superficial

[Sudip Mukherjee]

On Thu, Sep 17, 2020 at 10:30 AM Ole Streicher  wrote:
>
> "Adam D. Barratt"  writes:
> > On Thu, 2020-09-17 at 09:55 +0200, Ole Streicher wrote:
> >> Graham Inggs  writes:
> >> > On Thu, 17 Sep 2020 at 09:18, Raphael Hertzog 
> >> > wrote:
> >> > > Please reduce the severity of all the bugs that you opened to
> >> > > "normal" or "minor".
> >> >
> >> > Why?
> >>
> >> It does not violate the Debian Policy,
> >

>
> IMO the RC flag is understandable only for packages that are not
> migrated yet (or migrated just no longer than three days ago).
>
> Otherwise, I don't see the need to a re-upload to fix just this. Can you
> enlighten me?

i think I will leave it for the Release Team to decide. But just
consider the scenario when the severity of this bug for a package 'X'
is reduced and then another FTBFS bug is raised on that same package.
The FTBFS bug will be fixed and it will have an early migration as the
test is not (yet) marked "superficial".


-- 
Regards
Sudip

Re: Mass bugs filing: autopkgtest should be marked superficial

[Ole Streicher]

"Adam D. Barratt"  writes:
> On Thu, 2020-09-17 at 09:55 +0200, Ole Streicher wrote:
>> Graham Inggs  writes:
>> > On Thu, 17 Sep 2020 at 09:18, Raphael Hertzog 
>> > wrote:
>> > > Please reduce the severity of all the bugs that you opened to
>> > > "normal" or "minor".
>> > 
>> > Why?
>> 
>> It does not violate the Debian Policy,
>
> That's not the only possible reason for a bug to have a severity of
> "serious".
>
> These issues do violate the RC Policy for bullseye, which means that
> each "in the ... release manager's opinion, makes the package
> unsuitable for release".

While the rationale given ("circumvents the testing migration delay")
may be relevant furing the freeze time, it is irrelevant for a package
that is now already in testing since months: even if the test would have
been flagged as "superficial", it would have been migrated since long.

IMO the RC flag is understandable only for packages that are not
migrated yet (or migrated just no longer than three days ago).

Otherwise, I don't see the need to a re-upload to fix just this. Can you
enlighten me?

Cheers

Ole

Re: Mass bugs filing: autopkgtest should be marked superficial

[Adam D. Barratt]

On Thu, 2020-09-17 at 09:55 +0200, Ole Streicher wrote:
> Graham Inggs  writes:
> > On Thu, 17 Sep 2020 at 09:18, Raphael Hertzog 
> > wrote:
> > > Please reduce the severity of all the bugs that you opened to
> > > "normal" or "minor".
> > 
> > Why?
> 
> It does not violate the Debian Policy,

That's not the only possible reason for a bug to have a severity of
"serious".

These issues do violate the RC Policy for bullseye, which means that
each "in the ... release manager's opinion, makes the package
unsuitable for release".

Regards,

Adam

Re: Mass bugs filing: autopkgtest should be marked superficial

[Ole Streicher]

Graham Inggs  writes:
> On Thu, 17 Sep 2020 at 09:18, Raphael Hertzog  wrote:
>> Please reduce the severity of all the bugs that you opened to "normal"
>> or "minor".
>
> Why?

It does not violate the Debian Policy, and it does not make the package
somehow unusable. The only practical difference is that a package
without "superficial" migrates quicker than without if the test passes.

For a package that is already in "testing" (and passes the tests), it
does not matter whether the "superficial" flag is set.

Therefore, I also don't see why a package in "testing" would get this
bug set to RC and requires a new upload. It should just be changed with
the next upload.

Cheers

Ole

Re: Mass bugs filing: autopkgtest should be marked superficial

[Paul Gevers]

Hi all,

On 17-09-2020 10:03, Paul Wise wrote:
> On Thu, Sep 17, 2020 at 7:18 AM Raphael Hertzog wrote:
> 
>> I agreed about those bugs being filed but I strongly disagree about the
>> "serious" severity that you used for those bugs. You should have mentioned
>> your intent to use a RC-level severity and I would have reacted.
> 
> If I were part of the release team, I would consider these issues as
> RC because these packages are basically circumventing the testing
> migration delay for untested packages by adding trivial tests that
> don't fully test the functionality of the package.

This. I have written it done in response to bug [#969819]:

Notwithstanding the wording, the Release Team is happy with the bugs
that Sudip is filing. Because of the way that autopkgtests are used in
the Debian infrastructure to influence migration from unstable to
testing [1], it is very important that autopkgtests are recognized for
what they are. If an autopkgtest isn't really testing the installed
binaries (and yes, the boundary is unfortunately not well defined) it's
crucial that the test is marked as superficial, conform our rc_policy
[2]. The Release Team has decided that the examples that Sudip tagged,
i.e. --version, --help, checking for some installed file and the Python
import check, are superficial.

Paul

[#969819] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=969819#24
[1] reduced age for packages with successful non-trivial autopkgtests
and (since bullseye) packages with non-trivial autopkgtests are allowed
to migrate in a later stage of the freeze than other packages:
https://release.debian.org/bullseye/freeze_policy.html
[2] https://release.debian.org/bullseye/rc_policy.txt



signature.asc
Description: OpenPGP digital signature

Re: Mass bugs filing: autopkgtest should be marked superficial

[Jonas Smedegaard]

Quoting Graham Inggs (2020-09-17 09:28:05)
> Hi Raphael
> 
> On Thu, 17 Sep 2020 at 09:18, Raphael Hertzog  wrote:
> > Please reduce the severity of all the bugs that you opened to "normal"
> > or "minor".
> 
> Why?

RC severities imply "the package should be kicked if this is not solved" 
which is hardly appropriate for an issue of too weak regresion testing.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: Mass bugs filing: autopkgtest should be marked superficial

[Paul Wise]

On Thu, Sep 17, 2020 at 7:18 AM Raphael Hertzog wrote:

> I agreed about those bugs being filed but I strongly disagree about the
> "serious" severity that you used for those bugs. You should have mentioned
> your intent to use a RC-level severity and I would have reacted.

If I were part of the release team, I would consider these issues as
RC because these packages are basically circumventing the testing
migration delay for untested packages by adding trivial tests that
don't fully test the functionality of the package.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Mass bugs filing: autopkgtest should be marked superficial

[Raphael Hertzog]

Hi Graham,

On Thu, 17 Sep 2020, Graham Inggs wrote:
> On Thu, 17 Sep 2020 at 09:18, Raphael Hertzog  wrote:
> > Please reduce the severity of all the bugs that you opened to "normal"
> > or "minor".
> 
> Why?

Because the packages are not broken and do not deserve to be threatened by
a testing removal.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: Mass bugs filing: autopkgtest should be marked superficial

[Graham Inggs]

Hi Raphael

On Thu, 17 Sep 2020 at 09:18, Raphael Hertzog  wrote:
> Please reduce the severity of all the bugs that you opened to "normal"
> or "minor".

Why?

Regards
Graham

Re: Mass bugs filing: autopkgtest should be marked superficial

[Raphael Hertzog]

On Fri, 04 Sep 2020, Sudip Mukherjee wrote:
> If the test done in the autopkgtest does not provide significant test
> coverage then it should be marked with "Restrictions: superficial".
> Ref: https://people.debian.org/~eriberto/README.package-tests.html

I agreed about those bugs being filed but I strongly disagree about the
"serious" severity that you used for those bugs. You should have mentioned
your intent to use a RC-level severity and I would have reacted.

Please reduce the severity of all the bugs that you opened to "normal"
or "minor".

Thank you.
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Re: Mass bugs filing: autopkgtest should be marked superficial

[Antonio Terceiro]

On Sun, Sep 06, 2020 at 12:31:22AM +0100, Sudip Mukherjee wrote:
> Hi Paul,
> 
> On Sat, Sep 5, 2020 at 1:56 AM Paul Wise  wrote:
> >
> > On Fri, Sep 4, 2020 at 6:53 PM Sudip Mukherjee wrote:
> >
> > > If the test done in the autopkgtest does not provide significant test
> > > coverage then it should be marked with "Restrictions: superficial".
> > ...
> > > I am still trying to figure out a generalized method to find them but
> > > an initial script has found 83 packages. Attached is the dd-list.
> >
> > This sort of thing seems like something that will be an ongoing
> > problem so a more efficient way to solve it would be a lintian
> > warning, which should hopefully help prevent new occurrences. OTOH it
> > would be pretty hard to automatically check for these without a robust
> > shell parser. Perhaps morbig from Project CoLiS could be used for the
> > shell parsing and then a script could process the morbig output.
> > ShellCheck might be another option but it doesn't yet output parse
> > trees.
> 
> We were hoping that this check can be added in lintian, but looking at
> #932862 it seems you have already requested that. :)
> I will have a look at morbig and see if I can use that in my script.
> Thanks for the idea.

FWIW I don't think that opening bugs about the packages we already know
are using silly tests without "Restrictions: superficial" needs to wait
for that.

Thanks for your work on this.


signature.asc
Description: PGP signature

Re: Mass bugs filing: autopkgtest should be marked superficial

[Sudip Mukherjee]

Hi Paul,

On Sat, Sep 5, 2020 at 1:56 AM Paul Wise  wrote:
>
> On Fri, Sep 4, 2020 at 6:53 PM Sudip Mukherjee wrote:
>
> > If the test done in the autopkgtest does not provide significant test
> > coverage then it should be marked with "Restrictions: superficial".
> ...
> > I am still trying to figure out a generalized method to find them but
> > an initial script has found 83 packages. Attached is the dd-list.
>
> This sort of thing seems like something that will be an ongoing
> problem so a more efficient way to solve it would be a lintian
> warning, which should hopefully help prevent new occurrences. OTOH it
> would be pretty hard to automatically check for these without a robust
> shell parser. Perhaps morbig from Project CoLiS could be used for the
> shell parsing and then a script could process the morbig output.
> ShellCheck might be another option but it doesn't yet output parse
> trees.

We were hoping that this check can be added in lintian, but looking at
#932862 it seems you have already requested that. :)
I will have a look at morbig and see if I can use that in my script.
Thanks for the idea.


-- 
Regards
Sudip

Re: Mass bugs filing: autopkgtest should be marked superficial

[Paul Wise]

On Fri, Sep 4, 2020 at 6:53 PM Sudip Mukherjee wrote:

> If the test done in the autopkgtest does not provide significant test
> coverage then it should be marked with "Restrictions: superficial".
...
> I am still trying to figure out a generalized method to find them but
> an initial script has found 83 packages. Attached is the dd-list.

This sort of thing seems like something that will be an ongoing
problem so a more efficient way to solve it would be a lintian
warning, which should hopefully help prevent new occurrences. OTOH it
would be pretty hard to automatically check for these without a robust
shell parser. Perhaps morbig from Project CoLiS could be used for the
shell parsing and then a script could process the morbig output.
ShellCheck might be another option but it doesn't yet output parse
trees.

https://bugs.debian.org/932862
https://bugs.debian.org/629247
https://www.irif.fr/~treinen/colis/
https://github.com/colis-anr/morbig
https://www.shellcheck.net/

--
bye,
pabs

https://wiki.debian.org/PaulWise