Re: Severity bump script

[Sandro Tosi]

there seems to be disagreement on how to proceed, so for the time
being i suspended the severity bump part of the py2removal tracking
script. let me know when everybody agrees on a solution, and what that
solution is, and i'll code it and re-enable.

regards,
Sandro

On Thu, Dec 5, 2019 at 6:07 AM Paul Gevers  wrote:
>
> Hi,
>
> On 03-12-2019 13:19, Matthias Klose wrote:
> > It's unfortunate that issues for some packages only get attention when the
> > severity of an issue is raised.  Following your proposal means that the 
> > issue is
> > probably ignored forever, and you don't propose a better way going forward, 
> > just
> > saying we should stop earlier.  I don't think that's the correct choice.  
> > For
> > now these seem to be single packages, so please could you name those, and 
> > we can
> > look at those with a priority?  That's at least a path that is forward 
> > looking,
> > or feel free to propose another approach better than your current proposal 
> > for
> > not getting the attention of maintainers.
>
> I guess what I'm asking for could be fulfilled by an announcement to
> d-d-a with a package list (including via which package(s) they are
> affected) attached including the standard grouping by DD. And then some
> time to react.
>
> Paul
>
> PS, my original typed reply below. After having writing it, the idea
> above emerged.
>
> My problem with the current approach is that the way that developers
> learn that they (potentially transitively) (build) depend on a Python 2
> package is by the autoremoval message. I don't think that is acceptable
> socially in the project. My proposal is to give the maintainers about
> those packages a heads up. Via a bug report may be a bit weird in cases,
> but in some cases may be the appropriate thing as the package could work
> around the (build) dependency. At least adding "affects" helps a tiny
> bit and direct e-mails to the maintainers (e.g. via the
> @packages.debian.org address will at least give them a heads
> up. Even if the RM bug is coming eventually.
>
> Again, I don't have a problem with Python 2 removal, even at the price
> of packages that don't care that their dependency is written in Python.
> I do care about proper communication. Using RC severity to trigger
> autoremoval for non-leaf packages just yet is not appropriate in the
> opinion of the release team. Even though I am well aware of the Python 2
> removal effort I can tell from personal experience (cacti) that
> receiving an autoremoval e-mail as the first sign of such a dependency
> isn't nice, that's the problem I'm having and that needs solving in my
> opinion.
>


-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

Re: Severity bump script

[Paul Gevers]

Hi,

On 03-12-2019 13:19, Matthias Klose wrote:
> It's unfortunate that issues for some packages only get attention when the
> severity of an issue is raised.  Following your proposal means that the issue 
> is
> probably ignored forever, and you don't propose a better way going forward, 
> just
> saying we should stop earlier.  I don't think that's the correct choice.  For
> now these seem to be single packages, so please could you name those, and we 
> can
> look at those with a priority?  That's at least a path that is forward 
> looking,
> or feel free to propose another approach better than your current proposal for
> not getting the attention of maintainers.

I guess what I'm asking for could be fulfilled by an announcement to
d-d-a with a package list (including via which package(s) they are
affected) attached including the standard grouping by DD. And then some
time to react.

Paul

PS, my original typed reply below. After having writing it, the idea
above emerged.

My problem with the current approach is that the way that developers
learn that they (potentially transitively) (build) depend on a Python 2
package is by the autoremoval message. I don't think that is acceptable
socially in the project. My proposal is to give the maintainers about
those packages a heads up. Via a bug report may be a bit weird in cases,
but in some cases may be the appropriate thing as the package could work
around the (build) dependency. At least adding "affects" helps a tiny
bit and direct e-mails to the maintainers (e.g. via the
@packages.debian.org address will at least give them a heads
up. Even if the RM bug is coming eventually.

Again, I don't have a problem with Python 2 removal, even at the price
of packages that don't care that their dependency is written in Python.
I do care about proper communication. Using RC severity to trigger
autoremoval for non-leaf packages just yet is not appropriate in the
opinion of the release team. Even though I am well aware of the Python 2
removal effort I can tell from personal experience (cacti) that
receiving an autoremoval e-mail as the first sign of such a dependency
isn't nice, that's the problem I'm having and that needs solving in my
opinion.



signature.asc
Description: OpenPGP digital signature

Re: Severity bump script

[Matthias Klose]

On 02.12.19 20:28, Paul Gevers wrote:
> Hi all,
> 
> On 01-12-2019 22:45, Sandro Tosi wrote:
>> Paul, this is the thread i was talking about.
>>
>> you were copied in the original email:
>> https://lists.debian.org/debian-python/2019/10/msg00098.html
>>
>> if there is something the RT wants to discuss about this effort,
>> please do so here, not directly to me (i may be the mail address
>> sending those control commands, but the decision was taken here).
> 
> I understand the drive to push for Python 2 removal and sympathize with
> it. The issue I had yesterday with the process is that "leaf" was
> wrongly defined, it was looking at Depends, instead of also including
> Build-Depends.

that should be fixed.

> I don't want to stand in the way of Python 2 removal, but as I said
> before, pulling packages out from underneath maintainers isn't pretty so
> needs to be done with proper notifications and care. An RC bug to ones
> own package is acceptable in my opinion as it is a clear discussion
> forum, and can be (temporarily) down-graded while the discussion is
> ongoing. Being notified about some other package that I not even need
> directly but is going to pull "my" package out of testing isn't a nice
> experience and should be avoided. Yes, that slows down the process, but
> there are people, emotions and all those irrational things involved.

It's unfortunate that issues for some packages only get attention when the
severity of an issue is raised.  Following your proposal means that the issue is
probably ignored forever, and you don't propose a better way going forward, just
saying we should stop earlier.  I don't think that's the correct choice.  For
now these seem to be single packages, so please could you name those, and we can
look at those with a priority?  That's at least a path that is forward looking,
or feel free to propose another approach better than your current proposal for
not getting the attention of maintainers.

Matthias

PS: There's a RC issue for creduce now, not caused by the package itself, should
I downgrade it?

Re: Severity bump script

[Ondrej Novy]

Hi,

po 2. 12. 2019 v 20:36 odesílatel Paul Gevers  napsal:

> #942999 and #936537
>

ah right. I think this is correct. There is nothing else "in Python 2
removal process" to do on "someone else" packages. Work needs to be done in
these packages so raise severity here to unblock other bugs and continue
Python 2 removing effort.

-- 
Best regards
 Ondřej Nový

Re: Severity bump script

[Mattia Rizzolo]

On Mon, 2 Dec 2019, 10:25 pm Paul Gevers,  wrote:

> Hi,
>
> On 02-12-2019 22:15, Sandro Tosi wrote:
> > the blocks are only between py2removal packages, so if a package
> > un-related to the py2removal effort
> > depend/recomments/b-deps/autotest-triggers a py2removal *application*,
> that
> > is still considered a leaf package
>
> You'll fix that, right? Because why would the tree stop at Python? A
> leaf package is a package without Depends/Build-Depends in Debian.


Because the python2 removal is about python2.  If you depend on a python2
package then the dependant application needs to likewise be dropped or
updated, but it also is at the same time somewhat out of scope from "us".

So it's either file py2removal bugs also against packages that don't depend
on python, but just use an application that happens to use python2, or the
current status quo.
That's my take at least.

Also, I am one of those that think we should be much more forceful, and the
current situation looks just fine for me, so I might be biased.

Re: Severity bump script

[Sandro Tosi]

> You'll fix that, right? Because why would the tree stop at Python? A
> leaf package is a package without Depends/Build-Depends in Debian. (I
> appreciate it very much that you consider Recommends and
> autopkgtest-triggers as well).

so the revised logic should be:

* it's an app (package name doesnt start with 'python-' or end with -'doc')
* it has low popcorn (< 300)
* (NEW) it has "real" rdep = 0 (so the number of dependencies from
packages not from the same source pkg are 0)

is that correct or should it be something else?


-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

Re: Severity bump script

[Paul Gevers]

Hi,

On 02-12-2019 22:15, Sandro Tosi wrote:
> the blocks are only between py2removal packages, so if a package
> un-related to the py2removal effort
> depend/recomments/b-deps/autotest-triggers a py2removal *application*, that
> is still considered a leaf package

You'll fix that, right? Because why would the tree stop at Python? A
leaf package is a package without Depends/Build-Depends in Debian. (I
appreciate it very much that you consider Recommends and
autopkgtest-triggers as well).

Paul



signature.asc
Description: OpenPGP digital signature

Re: Severity bump script

[Sandro Tosi]

> > huh? We are not bumping any "blocked" bugs.
> > Depends/Build-Depends/Recommends/autopkgtests usage marks bug as
> > blocked. Any example of "wrong definition" please?
>
> #942999 and #936537

the blocks are only between py2removal packages, so if a package
un-related to the py2removal effort
depend/recomments/b-deps/autotest-triggers a py2removal *application*, that
is still considered a leaf package and will get its severity
raised (cfr 
https://github.com/sandrotosi/debian-tools/blob/master/py2rm_progress.py#L544-L546
); it doesnt happen with modules are we check the "real" rdeps for
those


--
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

Re: Severity bump script

[Ondrej Novy]

Hi,

po 2. 12. 2019 v 20:28 odesílatel Paul Gevers  napsal:

> I understand the drive to push for Python 2 removal and sympathize with
> it. The issue I had yesterday with the process is that "leaf" was
> wrongly defined, it was looking at Depends, instead of also including
> Build-Depends.
>

huh? We are not bumping any "blocked" bugs.
Depends/Build-Depends/Recommends/autopkgtests usage marks bug as blocked.
Any example of "wrong definition" please?

-- 
Best regards
 Ondřej Nový

Re: Severity bump script

[Paul Gevers]

Hi Ondrej,

On 02-12-2019 20:33, Ondrej Novy wrote:
> Hi,
> 
> po 2. 12. 2019 v 20:28 odesílatel Paul Gevers  > napsal:
> 
> I understand the drive to push for Python 2 removal and sympathize with
> it. The issue I had yesterday with the process is that "leaf" was
> wrongly defined, it was looking at Depends, instead of also including
> Build-Depends.
> 
> 
> huh? We are not bumping any "blocked" bugs.
> Depends/Build-Depends/Recommends/autopkgtests usage marks bug as
> blocked. Any example of "wrong definition" please?

#942999 and #936537

Paul



signature.asc
Description: OpenPGP digital signature

Re: Severity bump script

[Paul Gevers]

Hi all,

On 01-12-2019 22:45, Sandro Tosi wrote:
> Paul, this is the thread i was talking about.
> 
> you were copied in the original email:
> https://lists.debian.org/debian-python/2019/10/msg00098.html
> 
> if there is something the RT wants to discuss about this effort,
> please do so here, not directly to me (i may be the mail address
> sending those control commands, but the decision was taken here).

I understand the drive to push for Python 2 removal and sympathize with
it. The issue I had yesterday with the process is that "leaf" was
wrongly defined, it was looking at Depends, instead of also including
Build-Depends.

I don't want to stand in the way of Python 2 removal, but as I said
before, pulling packages out from underneath maintainers isn't pretty so
needs to be done with proper notifications and care. An RC bug to ones
own package is acceptable in my opinion as it is a clear discussion
forum, and can be (temporarily) down-graded while the discussion is
ongoing. Being notified about some other package that I not even need
directly but is going to pull "my" package out of testing isn't a nice
experience and should be avoided. Yes, that slows down the process, but
there are people, emotions and all those irrational things involved.

Paul



signature.asc
Description: OpenPGP digital signature

Re: Severity bump script

[Sandro Tosi]

Paul, this is the thread i was talking about.

you were copied in the original email:
https://lists.debian.org/debian-python/2019/10/msg00098.html

if there is something the RT wants to discuss about this effort,
please do so here, not directly to me (i may be the mail address
sending those control commands, but the decision was taken here).

Thanks,
Sandro

On Mon, Nov 11, 2019 at 1:29 AM Ondrej Novy  wrote:
>
> Hi Sandro,
>
> -- Forwarded message -
> > We are going to raise the severity of the py2removal bugs to "serious" in 
> > several steps.  In the
> > first phase we are going to raise severity of the py2removal bugs for
> > all leaf module packages and low popcon (< 300) application packages.
> > Bugs marked with the "py2keep" user tag will not have their severity
> > raised.
>
> is there any progress with script from your site please? First we need list 
> to check. Thanks a lot.
>
> --
> Best regards
>  Ondřej Nový
>


-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

Re: Severity bump script

[Ondrej Novy]

Hi,

čt 28. 11. 2019 v 19:07 odesílatel Sandro Tosi  napsal:

> this is live now, with the following header:
>

 cool, thanks again!

-- 
Best regards
 Ondřej Nový

Re: Severity bump script

[Sandro Tosi]

> I think we should use something simple/short like this:
>
> --cut--
> Raising severity of Python 2 removal bugs, for details see: 
> https://lists.debian.org/debian-devel-announce/2019/11/msg0.html
> --cut--

this is live now, with the following header:

# Part of the effort for the removal of python from bullseye
#  * https://wiki.debian.org/Python/2Removal
#  * http://sandrotosi.me/debian/py2removal/index.html
# See https://lists.debian.org/debian-devel-announce/2019/11/msg0.html
# for more details on this severity bump

# python-apns-client is a module and has 0 external rdeps
severity 935212 serious
.


-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

Re: Severity bump script

[Ondrej Novy]

Hi,

čt 28. 11. 2019 v 4:12 odesílatel Sandro Tosi  napsal:

> i'm not sure we can: in
> https://lists.debian.org/debian-python/2019/10/msg00096.html you say
> "We need to prepare draft of RC-bumper email", is this draft ready?
>

ah, you are right.


> add the description to the header (as comment) of a mail to control@?
>

this is only way to go imho.

I think we should use something simple/short like this:

--cut--
Raising severity of Python 2 removal bugs, for details see:
https://lists.debian.org/debian-devel-announce/2019/11/msg0.html
--cut--

Thanks Sandro for working on this.

-- 
Best regards
 Ondřej Nový

Re: Severity bump script

[Sandro Tosi]

On Wed, Nov 27, 2019 at 2:16 PM Ondrej Novy  wrote:
>
> Hi,
>
> pá 22. 11. 2019 v 22:22 odesílatel Sandro Tosi  napsal:
>>
>> I've removed bugs that are marked as blocked by; we're now down to 487
>> unique severity raies
>
>
> cool. Checked and imho it works fine now. So I guess we can run it.

i'm not sure we can: in
https://lists.debian.org/debian-python/2019/10/msg00096.html you say
"We need to prepare draft of RC-bumper email", is this draft ready?
how do we wanna use it: should we email each bug individually or just
add the description to the header (as comment) of a mail to control@?

it's probably not a good idea to mail each bug individually as it will
produce a lot of emails (at least in the first batch) that will result
in my laptop being blacklisted by bugs.d.o MTA.

-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

Re: Severity bump script

[Ondrej Novy]

Hi,

pá 22. 11. 2019 v 22:22 odesílatel Sandro Tosi  napsal:

> I've removed bugs that are marked as blocked by; we're now down to 487
> unique severity raies
>

cool. Checked and imho it works fine now. So I guess we can run it.

Thanks.

-- 
Best regards
 Ondřej Nový

Re: Severity bump script

[Sandro Tosi]

On Fri, Nov 22, 2019 at 3:23 AM Ondrej Novy  wrote:
>
> Hi Sandro,
>
> thanks for first version. I randomly checked few lines and found this:
>
> # libufo0 is an application and has low popcon (25 < 300)
> severity 938743 serious
> # libufo-bin is an application and has low popcon (4 < 300)
> severity 938743 serious
> # libufo-dev is an application and has low popcon (5 < 300)
> severity 938743 serious
> # python-ufw is a module and has 0 external rdeps
>
> But in bug:
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=938743
>
> is
>
> Fix blocked by 938744: ufo-filters: Python2 removal in sid/bullseye
>
> So because 938743 is blocked by 938744, we should not raise severity of 
> 938743, right?

i've removed bugs that are marked as blocked by; we're now down to 487
unique severity raies

-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

Re: Severity bump script

[Ondrej Novy]

Hi Sandro,

thanks for first version. I randomly checked few lines and found this:

# libufo0 is an application and has low popcon (25 < 300)
severity 938743 serious
# libufo-bin is an application and has low popcon (4 < 300)
severity 938743 serious
# libufo-dev is an application and has low popcon (5 < 300)
severity 938743 serious
# python-ufw is a module and has 0 external rdeps

But in bug:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=938743

is

Fix blocked by 938744: ufo-filters: Python2 removal in sid/bullseye

So because 938743 is blocked by 938744, we should not raise severity
of 938743, right?

Thanks.

--
Best regards
 Ondřej Nový

Re: Severity bump script

[Sandro Tosi]

> I see that the script considers some -doc packages as modules.

good point; fixed, but that only brings down the unique severity bumps to 688

> I can’t tell if we should remove them, but maybe the description could
> be updated? Like “python-foo-doc is a documentation package has 0 external
> rdeps”.

naah we shouldnt remote them

> Also, in other lines I would also change “have” to “has”.

fixed that too

-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

Re: Severity bump script

[Dmitry Shachnev]

Hi Sandro!

On Thu, Nov 14, 2019 at 07:13:49PM -0500, Sandro Tosi wrote:
> where is the first draft:
> http://sandrotosi.me/debian/py2removal/would_raise_to_rc.txt (it will
> be updated every time the script runs)
>
> there are 703 unique severity bumps roughly half of the pending bugs.
> please note there are duplicates in the list as data is per binary
> package, while bugs are on a source package basis.
>
> you can read the heuristic to determine if it's a module
> (https://github.com/sandrotosi/debian-tools/blob/master/py2rm_progress.py#L472)
> or and app 
> (https://github.com/sandrotosi/debian-tools/blob/master/py2rm_progress.py#L475)
> in the links, they are pretty weak, is there a better way?

I see that the script considers some -doc packages as modules.

I can’t tell if we should remove them, but maybe the description could
be updated? Like “python-foo-doc is a documentation package has 0 external
rdeps”.

Also, in other lines I would also change “have” to “has”.

--
Dmitry Shachnev


signature.asc
Description: PGP signature

Re: Severity bump script

[Sandro Tosi]

On Mon, Nov 11, 2019 at 1:29 AM Ondrej Novy  wrote:
>
> Hi Sandro,
>
> -- Forwarded message -
> > We are going to raise the severity of the py2removal bugs to "serious" in 
> > several steps.  In the
> > first phase we are going to raise severity of the py2removal bugs for
> > all leaf module packages and low popcon (< 300) application packages.
> > Bugs marked with the "py2keep" user tag will not have their severity
> > raised.
>
> is there any progress with script from your site please? First we need list 
> to check. Thanks a lot.

where is the first draft:
http://sandrotosi.me/debian/py2removal/would_raise_to_rc.txt (it will
be updated every time the script runs)

there are 703 unique severity bumps roughly half of the pending bugs.
please note there are duplicates in the list as data is per binary
package, while bugs are on a source package basis.

you can read the heuristic to determine if it's a module
(https://github.com/sandrotosi/debian-tools/blob/master/py2rm_progress.py#L472)
or and app 
(https://github.com/sandrotosi/debian-tools/blob/master/py2rm_progress.py#L475)
in the links, they are pretty weak, is there a better way?

all in all i think this is not the effect people were expecting: there
are too many packages that would become RC-buggy; also the severity
bump is done at the source package level, even if only a single binary
in the pkg set matches the criteria (which, if you read the request
above, it is what's wanted).

Let me know what you think,
-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

Re: Severity bump script

[Sandro Tosi]

On Mon, Nov 11, 2019 at 1:29 AM Ondrej Novy  wrote:
>
> Hi Sandro,
>
> -- Forwarded message -
> > We are going to raise the severity of the py2removal bugs to "serious" in 
> > several steps.  In the
> > first phase we are going to raise severity of the py2removal bugs for
> > all leaf module packages and low popcon (< 300) application packages.
> > Bugs marked with the "py2keep" user tag will not have their severity
> > raised.
>
> is there any progress with script from your site please? First we need list 
> to check. Thanks a lot.

no progress on this yet.

-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

Severity bump script

[Ondrej Novy]

Hi Sandro,

-- Forwarded message -
> We are going to raise the severity of the py2removal bugs to "serious" in
several steps.  In the
> first phase we are going to raise severity of the py2removal bugs for
> all leaf module packages and low popcon (< 300) application packages.
> Bugs marked with the "py2keep" user tag will not have their severity
> raised.

is there any progress with script from your site please? First we need list
to check. Thanks a lot.

-- 
Best regards
 Ondřej Nový