Re: Updated proposal for improving the FTP NEW process

[Adrian Bunk]

On Sat, Apr 14, 2018 at 01:00:08PM +0100, Ian Jackson wrote:
> Adrian Bunk writes ("Re: Updated  proposal for improving the FTP NEW 
> process"):
>...
> > What happens outside of our archive (e.g. in Ubuntu or .debian.net)
> > is nothing we officially provide to our users.
> 
> I don't agree, but that's just chopping semantics over "officially
> provide".  That a colliding source version was not "officially
> provided" (whatever that means) does not mean that it is good practice
> to generate this kind of confusion.

I think we are at the point where we both have to agree to disagree
on what is good practice here, and accept that different people have
different established practices regarding what they want from their
sponsorees here.

> You'll see the other thread on -devel at the moment where a package
> was mistakely left languishing in NEW; one significant causal factor
> in this mistake was IMO-inadvisable reuse of the version number.

For NEW I would agree with you.

Uploading to NEW can only be done by a DD.

That's an official upload and not from a random person,
and the quality is at least what is at mentors after
10 updates from a first-time contributor.

> Ian.

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: Updated proposal for improving the FTP NEW process

[Ian Jackson]

Adrian Bunk writes ("Re: Updated  proposal for improving the FTP NEW process"):
> Note that there are also many other situations where you already end up 
> with different contents under the same version.

Not different source code.

> An obvious example would be if you put both Debian unstable and
> Ubuntu bionic into your sources.list right now:
> It might be the majority of packages where you will then see different
> binary packages with exactly the same version.

As I say, this is neither interesting nor troublesome.

> What happens outside of our archive (e.g. in Ubuntu or .debian.net)
> is nothing we officially provide to our users.

I don't agree, but that's just chopping semantics over "officially
provide".  That a colliding source version was not "officially
provided" (whatever that means) does not mean that it is good practice
to generate this kind of confusion.

You'll see the other thread on -devel at the moment where a package
was mistakely left languishing in NEW; one significant causal factor
in this mistake was IMO-inadvisable reuse of the version number.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Updated proposal for improving the FTP NEW process

[Adrian Bunk]

On Fri, Apr 13, 2018 at 03:54:29PM +0100, Ian Jackson wrote:
>...
> Now you seem to be saying
> 
>  1 having the same version version number referring to
>multiple different source versions is completely fine
>   because
>  2 reusing version version numbers should not be forbidden
>...
> I think 1 is false, and does not follow from 2.  I agree with 2, but I
> think reusing version numbers is usually bad and should only be done
> deliberately.
>...

I am not saying that 1 is completely fine in all cases.

But mentors.d.n is a case where deliberately reusing version numbers
is better than having weird changelogs in our archive.

Note that there are also many other situations where you already end up 
with different contents under the same version.

An obvious example would be if you put both Debian unstable and
Ubuntu bionic into your sources.list right now:
It might be the majority of packages where you will then see different
binary packages with exactly the same version.

Reusing versions is always bad and must be forbidden if it would result 
in two different (source or binary) packages with the same version ever 
entering our archive.

What happens outside of our archive (e.g. in Ubuntu or .debian.net)
is nothing we officially provide to our users.

> Ian.

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: Updated proposal for improving the FTP NEW process

[Ian Jackson]

Adrian Bunk writes ("Re: Updated  proposal for improving the FTP NEW process"):
> Debian version numbers are usually not globally unique.
> 
> The binary packages of dgit 4.3 in Debian and Ubuntu are different 
> builds from the same sources, and for binary-any packages such
> different builds usually have different contents.[1]

This is not particularly interesting, and certainly not an
counter-argument against Russ and my position that reusing version
numbers for different source packages is bad practice.

> And more common is actually the reverse problem of someone publishing
> a different 5.0 after a 5.0 is already in our archive (usually by
> making modifications without changing the version number).

Our tooling makes this too easy to do; and it makes it too hard to do
the right thing.  Observe here the horrible dch stuff that a user
has to type to avoid this confusion:
   https://manpages.debian.org/unstable/dgit/dgit-user.7.en.html

But I'm glad that you recognise that, in this case at least, reuse of
the version number is undesirable.

> > Version numbers are composed of integers.  Getting another integer is
> > free, and there is not a limited supply.  We won't run out, and missing
> > sequence numbers cause no problems in the world.
> 
> Giving every person on the internet the power to steal version numbers
> for random packages would be dangerous.
...
> Apart from obvious (scripted) DoS for taking all reasonable numbers for 
> a package, it would also e.g. encourage derivates to steal Debian 
> version numbers instead of using a proper namespace.[2]

Now you seem to be saying

 1 having the same version version number referring to
   multiple different source versions is completely fine
  because
 2 reusing version version numbers should not be forbidden
  because
 3 forbidding reuse of version numbers by Debian might encourage
   preemptive reuse of *spaces* of version numbers by derivatives
  and
 4 reuse of version number spaces by different origins is bad practice

You have almost (but not quite) contradicted your conclusion!
If we ask why the final thing in my list above, which you are treating
as an axiom, is true, we could continue your pseudosyllogism with

  because
 5 reuse of version number spaces by different origins will
   probably lead to reuse of version numbers for different source
  and
 6 having the same version version number referring to
   multiple different source versions is best avoided

I think 1 is false, and does not follow from 2.  I agree with 2, but I
think reusing version numbers is usually bad and should only be done
deliberately.

> Versions of packages that are accepted into our archive must be unique, 
> but random people from the internet should not have the power to 
> restrict what a maintainer can do in Debian.

I think that random idiots on the internet should (in principle) have
the power to require a maintainer to explicitly override their idiocy.
In practice I doubt anyone is going to implement that check soon.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Updated proposal for improving the FTP NEW process

[Adrian Bunk]

On Wed, Apr 11, 2018 at 02:05:03PM -0700, Russ Allbery wrote:
> Adrian Bunk  writes:
> 
> > Imagine tomorrow a random person from the internet noone has ever heard
> > of uploads a package dgit 5.0 to mentors.d.n.
> 
> > It is clear that this would not be sponsored.
> 
> > "detected by tooling" would mean that this would result in dak
> > autorejecting any future uploads of a dgit package version 5.0 to
> > Debian.
> 
> This sounds like a feature?  I think that would be exactly the right
> outcome.  It avoids any possibility of confusion with this rogue 5.0
> version, if it should turn up somewhere else.

Debian version numbers are usually not globally unique.

The binary packages of dgit 4.3 in Debian and Ubuntu are different 
builds from the same sources, and for binary-any packages such
different builds usually have different contents.[1]

And more common is actually the reverse problem of someone publishing
a different 5.0 after a 5.0 is already in our archive (usually by
making modifications without changing the version number).

> Version numbers are composed of integers.  Getting another integer is
> free, and there is not a limited supply.  We won't run out, and missing
> sequence numbers cause no problems in the world.

Giving every person on the internet the power to steal version numbers
for random packages would be dangerous.

There's implicit meaining behind version numbers, like debhelper 12
being the first version where compat 12 will be stable.

Apart from obvious (scripted) DoS for taking all reasonable numbers for 
a package, it would also e.g. encourage derivates to steal Debian 
version numbers instead of using a proper namespace.[2]

Versions of packages that are accepted into our archive must be unique, 
but random people from the internet should not have the power to 
restrict what a maintainer can do in Debian.

cu
Adrian

[1] e.g. different Debian revisions of gcc usually generate slightly
different code
[2] e.g. using 1.0-2 instead of 1.0-1devuan, resulting in an
improved dak autorejecting a maintainer upload of 1.0-2

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: Updated proposal for improving the FTP NEW process

[Ian Jackson]

Adrian Bunk writes ("Re: Updated  proposal for improving the FTP NEW process"):
> On Mon, Apr 09, 2018 at 02:24:23PM +0100, Ian Jackson wrote:
> > apt-listchanges will present the right section of the changelog
> > anyway.
> 
> Assuming your "skip 10 versions and use one changelog stanza"
> suggestion is done.

I could be wrong, but I'm pretty sure apt-listchanges as run by
aptwill show multiple changelog stanzas when that is appropriate.
This is quite straightforward to do since all of the information is
available.

If it oesn't do that then that is obviously a bug, because it will
DTWT if for any reason the system misses an update recorded in the
changelog.  For example, on a system tracking when testing updates
straight from version N to N+3 (because N+1 and N+2 didn't migrate for
whatever reason).

> > Furthermore, of it really does get to 10 versions, containing
> > absurdities, then the most-recent-version's changelog stanza can
> > contain a summary of the differences from the previously-accepted
> > upload.
> 
> I've had cases where the only thing I criticized in a sponsorship 
> request was "this change is not mentioned in the changelog".
> 
> What I got was an updated package with an added line in the changelog,
> and that was perfectly fine.
> 
> As sponsor, being nitpicky about good packaging becomes less desirable 
> if this would result in skipped versions in the uploaded package.

I would have asked the sponsee to bump the version too.  I might not
have insisted on a new changelog stanza.  (In practice I have very few
sponsees and I feel guilty enough about the poor service I offer to
them that I avoid being picky about this kind of stuff.)

> > > Or a more funny issue:
> > > How would you notice a version reuse in all cases?
> > > A package uploaded to mentors.d.n. adopting a package with
> > > "New maintainer" as only change is usually a reject. If some DD does
> > > the same years later, there is no record anywhere that this version
> > > was already taken by some random person from the internet who once
> > > upon a time uploaded it to mentors.d.n.
> > 
> > That a bad practice cannot always be detected by tooling does not make
> > it a good practice.
> 
> Imagine tomorrow a random person from the internet noone has ever heard 
> of uploads a package dgit 5.0 to mentors.d.n.
> 
> It is clear that this would not be sponsored.
> 
> "detected by tooling" would mean that this would result in dak 
> autorejecting any future uploads of a dgit package version 5.0
> to Debian.

This is a very strange response.  I said

  That a bad practice cannot always be detected by tooling does not
  make it a good practice.

You concluded that I think it desirable for every bad practice to be
completely prevented by tooling.  I find it difficult to see how you
think that would follow.

For the avoidance of doubt, I think there are many bad practices
which: (i) cannot always (or ever) be detected by tooling
(ii) if they could be detected, ought not to be completely prevented
(because a human might have a good reason).

Certainly in the scenario you describe I would like something to warn
me, to give me the option to pick a different version number.  I would
probably do so the reasons Russ explains.

The reason why I shouldn't be completely _prevented_ is a matter of
proper process and security engineering.  Mistakes by lower-privileged
people should not be able to completely block work by
higher-privileged people.  That doesn't mean that the
higher-privileged person shouldn't be made aware of the situation so
that they can explicitly choose what the appropriate response is.

.oO{ dgit push --deliberately-collide-version-mentors }

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Updated proposal for improving the FTP NEW process

[Russ Allbery]

Adrian Bunk  writes:

> Imagine tomorrow a random person from the internet noone has ever heard
> of uploads a package dgit 5.0 to mentors.d.n.

> It is clear that this would not be sponsored.

> "detected by tooling" would mean that this would result in dak
> autorejecting any future uploads of a dgit package version 5.0 to
> Debian.

This sounds like a feature?  I think that would be exactly the right
outcome.  It avoids any possibility of confusion with this rogue 5.0
version, if it should turn up somewhere else.

Version numbers are composed of integers.  Getting another integer is
free, and there is not a limited supply.  We won't run out, and missing
sequence numbers cause no problems in the world.

-- 
Russ Allbery (r...@debian.org)   

Re: Updated proposal for improving the FTP NEW process

[Adrian Bunk]

On Mon, Apr 09, 2018 at 02:24:23PM +0100, Ian Jackson wrote:
> Adrian Bunk writes ("Re: Updated  proposal for improving the FTP NEW 
> process"):
> > A version is published to our users when it gets accepted into
> > the archive.
> > 
> > Readable information in apt-listchanges is IMHO more important
> > than theoretical discussions around whether something submitted
> > to mentors.d.n is public.
> 
> apt-listchanges will present the right section of the changelog
> anyway.

Assuming your "skip 10 versions and use one changelog stanza"
suggestion is done.

> > A changelog is also permanent, and people might read it decades later 
> > for understanding packaging decisions - already today it is not uncommon 
> > to check 20 year old changelog entries for that.
> > 
> > For either of the above a weird version history or 10 Debian revisions 
> > until a new maintainer got her first packaging attempt correct are
> > not optimal.
> 
> I disagree completely.
> 
> Furthermore, of it really does get to 10 versions, containing
> absurdities, then the most-recent-version's changelog stanza can
> contain a summary of the differences from the previously-accepted
> upload.

I've had cases where the only thing I criticized in a sponsorship 
request was "this change is not mentioned in the changelog".

What I got was an updated package with an added line in the changelog,
and that was perfectly fine.

As sponsor, being nitpicky about good packaging becomes less desirable 
if this would result in skipped versions in the uploaded package.

> > Or a more funny issue:
> > How would you notice a version reuse in all cases?
> > A package uploaded to mentors.d.n. adopting a package with
> > "New maintainer" as only change is usually a reject. If some DD does
> > the same years later, there is no record anywhere that this version
> > was already taken by some random person from the internet who once
> > upon a time uploaded it to mentors.d.n.
> 
> That a bad practice cannot always be detected by tooling does not make
> it a good practice.

Imagine tomorrow a random person from the internet noone has ever heard 
of uploads a package dgit 5.0 to mentors.d.n.

It is clear that this would not be sponsored.

"detected by tooling" would mean that this would result in dak 
autorejecting any future uploads of a dgit package version 5.0
to Debian.

> Ian.

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: Updated proposal for improving the FTP NEW process

[Ian Jackson]

Adrian Bunk writes ("Re: Updated  proposal for improving the FTP NEW process"):
> A version is published to our users when it gets accepted into
> the archive.
> 
> Readable information in apt-listchanges is IMHO more important
> than theoretical discussions around whether something submitted
> to mentors.d.n is public.

apt-listchanges will present the right section of the changelog
anyway.

> A changelog is also permanent, and people might read it decades later 
> for understanding packaging decisions - already today it is not uncommon 
> to check 20 year old changelog entries for that.
> 
> For either of the above a weird version history or 10 Debian revisions 
> until a new maintainer got her first packaging attempt correct are
> not optimal.

I disagree completely.

Furthermore, of it really does get to 10 versions, containing
absurdities, then the most-recent-version's changelog stanza can
contain a summary of the differences from the previously-accepted
upload.

> Or a more funny issue:
> How would you notice a version reuse in all cases?
> A package uploaded to mentors.d.n. adopting a package with
> "New maintainer" as only change is usually a reject. If some DD does
> the same years later, there is no record anywhere that this version
> was already taken by some random person from the internet who once
> upon a time uploaded it to mentors.d.n.

That a bad practice cannot always be detected by tooling does not make
it a good practice.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Updated proposal for improving the FTP NEW process

[Adrian Bunk]

On Tue, Mar 06, 2018 at 11:40:52PM +, Holger Levsen wrote:
> On Tue, Mar 06, 2018 at 05:54:55PM +0100, Adam Borowski wrote:
> > With my one of most active sponsors hat on: the current policy is that a
> > version that has never hit the archive must not have a separate changelog
> > entry, unless there are non-negligible users (such as a derivative, upstream
> > repository or at least the package being deployed to multiple users at a
> > workplace).  A past history is more acceptable than repeated attempts for an
> > upload.
> > 
> > This is what I was taught, and what I not only recommend but also require of
> > sponsorees.  There seems to be a concensus on -mentors that this is the
> > right way.
> 
> with my sponsoring hat on, I will be unhappy if someone reuses version
> numbers and I will ask to never do this again. I very much agree with
> Ian's position that this is bad.
> 
> As a sponsor, I'm a non-negligible user and I want to sensible be able
> to not having to again review stuff I already have reviewed.
> 
> If you have put it on mentors.d.n, it's out in the public.

As a strict policy this would create more problems than it solves.

A version is published to our users when it gets accepted into
the archive.

Readable information in apt-listchanges is IMHO more important
than theoretical discussions around whether something submitted
to mentors.d.n is public.

A changelog is also permanent, and people might read it decades later 
for understanding packaging decisions - already today it is not uncommon 
to check 20 year old changelog entries for that.

For either of the above a weird version history or 10 Debian revisions 
until a new maintainer got her first packaging attempt correct are
not optimal.

Or a more funny issue:
How would you notice a version reuse in all cases?
A package uploaded to mentors.d.n. adopting a package with
"New maintainer" as only change is usually a reject. If some DD does
the same years later, there is no record anywhere that this version
was already taken by some random person from the internet who once
upon a time uploaded it to mentors.d.n.

> cheers,
>   Holger

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: Updated proposal for improving the FTP NEW process

[Lars Wirzenius]

On Sat, 2018-03-10 at 23:06 +0100, Andreas Tille wrote:
> In this longish thread I have read one contribution where a developer
> expressed that he was happy about checking his SONAME bumped package
> that was erroneous and luckily ftpmaster found the problem. 

I'm happy that the ftp masters are doing their job in the best way
they can. This includes any re-reviews of packages already in the
archive, at whatever points they choose, be that when there's a new
binary package from an old source package, randomly chosen packages,
or targeting packages whose SHA3 ends in 42.

signature.asc
Description: This is a digitally signed message part

Re: Updated proposal for improving the FTP NEW process

[Adam Borowski]

On Sat, Mar 10, 2018 at 11:06:49PM +0100, Andreas Tille wrote:
> In this longish thread I have read one contribution where a developer
> expressed that he was happy about checking his SONAME bumped package
> that was erroneous and luckily ftpmaster found the problem.  (Sorry, I'm
> to lazy to reread the archive for the actual link.)  My point is that
> this was a *single* voice pro-ftpmaster-check-SONAME-changes.  I confirm
> its nice to fix the described error before the package hits the archive
> but the problem would have been spotted most probably afterwards by
> other QA means and the issue could have also be reported by a user via
> BTS.
> 
> All other voices of developers in this thread I have read would have
> prefered a faster processing.

This is not a vote.

But if it somehow is, here's my strong +1 to keeping _technical_ checks
for binNEW.  This includes SONAMEs.

> Several others here gave good reasons why the biased selection is a quite
> bad idea for refreshing license checks.

But here I agree.

License changes are completely unrelated to packaging changes.  Any new
upstream version can include a different license than what was checked.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄ A master species delegates.

Re: Updated proposal for improving the FTP NEW process

[Scott Kitterman]

On March 10, 2018 10:18:58 PM UTC, Andreas Tille  wrote:
>On Sat, Mar 10, 2018 at 05:37:28PM +, Scott Kitterman wrote:
>> I'm afraid you have a rather narrower view of the FTP Master role
>than the project [1], in particular, "... oversee and maintain the
>well-being of Debian's official package repositories ...".  The
>specifics listed later in the delegation message are examples, not an
>all inclusive list.
>> 
>> Checks during binary New are very much part of that task.  I know
>most people who aren't part of the team mostly interact with the FTP
>Team because of Source New, but in many respects it's one of the least
>important things the team does to keep the archive healthy.
>
>Can you please point me to the doc where it is defined to apply
>specific
>and repeated license checks on a biased package selection?

Sure, right after you point out the spot in the constitution where project 
members get micromanaging privileges over DPL delegates.

I get everyone would like New to be faster.  I'd like it if people stopped 
uploading crap packages so things would be easier for everyone.  I'm probably 
not going to get that either.

Scott K

Re: Updated proposal for improving the FTP NEW process

[Andreas Tille]

On Sat, Mar 10, 2018 at 05:37:28PM +, Scott Kitterman wrote:
> I'm afraid you have a rather narrower view of the FTP Master role than the 
> project [1], in particular, "... oversee and maintain the well-being of 
> Debian's official package repositories ...".  The specifics listed later in 
> the delegation message are examples, not an all inclusive list.
> 
> Checks during binary New are very much part of that task.  I know most people 
> who aren't part of the team mostly interact with the FTP Team because of 
> Source New, but in many respects it's one of the least important things the 
> team does to keep the archive healthy.

Can you please point me to the doc where it is defined to apply specific
and repeated license checks on a biased package selection?

Thank you

 Andreas.
 
> [1] https://lists.debian.org/debian-devel-announce/2017/11/msg1.html

-- 
http://fam-tille.de

Re: Updated proposal for improving the FTP NEW process

[Andreas Tille]

Hi Chris,

On Sat, Mar 10, 2018 at 05:25:46PM +, Chris Lamb wrote:
> > a single example in this thread that a developer was happy about the
> > check since a mistake was avoided.  But this would have happened by a
> > random user via BTS as well.
> 
> (I don't quite follow this, sorry.. can you rephrase?)

In this longish thread I have read one contribution where a developer
expressed that he was happy about checking his SONAME bumped package
that was erroneous and luckily ftpmaster found the problem.  (Sorry, I'm
to lazy to reread the archive for the actual link.)  My point is that
this was a *single* voice pro-ftpmaster-check-SONAME-changes.  I confirm
its nice to fix the described error before the package hits the archive
but the problem would have been spotted most probably afterwards by
other QA means and the issue could have also be reported by a user via
BTS.

All other voices of developers in this thread I have read would have
prefered a faster processing.

I'm not sure about the quote ftpmaster is spotting the kind of
*technical* issues (as described by the author of the mail I'm refereing
above) and I'd be fine if the check would be restricted to technical
issues of a SONAME change which might be considered potentially
dangerous.  I fail to see any sense in this kind of checks for
additional doc or python3 packages or something like this.  Several
others here gave good reasons why the biased selection is a quite bad
idea for refreshing license checks.  My bet is that if you find your
targets for license re-checking by sorting packages in UDD according to
the longest not uploaded packages you will find more interesting
targets.

Kind regards

  Andreas.

-- 
http://fam-tille.de

Re: Updated proposal for improving the FTP NEW process

[Scott Kitterman]

On March 6, 2018 7:27:40 PM UTC, Ian Campbell  wrote:
>On Tue, 2018-03-06 at 19:18 +0100, Mattia Rizzolo wrote:
>> On Tue, Mar 06, 2018 at 06:34:28PM +0100, Adam Borowski wrote:
>> > What might be reasonable is making the separation between binNEW
>and srcNEW
>> > more obvious, especially for us on the other side.  This would also
>> > encourage the ftpmasters to skip license checks for binNEW -- but
>again,
>> > without knowing their workflow I'm not in a position to improve it.
>> 
>> I know for a fact that quite regularly licence checks on binNEW
>packages
>> causes RC bugs to pop up.  I acknowledge it may be a burder for the
>ftp
>> team, but that reason alone probably deserves to keep binNEW as it
>is.
>
>That would seem to justify some sort of randomized spot checks on the
>archive, not arbitrarily focussing on the subset of packages which
>happen to need a new binary package for some reason.
>
>Ian.

Anyone is free to do that at anytime.  It doesn't take any special powers.

Scott K

Re: Updated proposal for improving the FTP NEW process

[Scott Kitterman]

On March 10, 2018 11:13:46 AM UTC, Andreas Tille  wrote:
>Hi Chris,
>
>On Wed, Mar 07, 2018 at 02:44:32PM +, Chris Lamb wrote:
>> > I think the suggestion of randomized spot checking is meant to
>replace - 
>> > not add - to the present system of checking that penalizes uploads
>of 
>> > existing source but new binaries.  So human resources should not be
>the 
>> > issue.
>> 
>> In my experience, time/energy/focus is not as fungible or easily
>> transferable as you imply.
>
>I share your assumption that if we try to get a real random set of
>packages checked instead of checking those who are ending up by random
>reasons in new we will end up with less re-checked packages.  However,
>this does not give any good reason for keeping the habit to re-check
>packages where a resulting binary package is not inside the package
>pool.  It somehow reminds me to those people who were asked: "Why are
>you doing this?" and gave the answer: "Since we did so all the time."
>
>We all know that ftpmasters have a lot of work and this thread is about
>convincing ftpmaster to stop some work that does not belong to their
>initial task which is *checking new source packages*.
...

I'm afraid you have a rather narrower view of the FTP Master role than the 
project [1], in particular, "... oversee and maintain the well-being of 
Debian's official package repositories ...".  The specifics listed later in the 
delegation message are examples, not an all inclusive list.

Checks during binary New are very much part of that task.  I know most people 
who aren't part of the team mostly interact with the FTP Team because of Source 
New, but in many respects it's one of the least important things the team does 
to keep the archive healthy.

Scott K


[1] https://lists.debian.org/debian-devel-announce/2017/11/msg1.html

Re: Updated proposal for improving the FTP NEW process

[Chris Lamb]

Hi Andreas,

> a single example in this thread that a developer was happy about the
> check since a mistake was avoided.  But this would have happened by a
> random user via BTS as well.

(I don't quite follow this, sorry.. can you rephrase?)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: Updated proposal for improving the FTP NEW process

[Simon McVittie]

On Sat, 10 Mar 2018 at 12:13:46 +0100, Andreas Tille wrote:
> I share your assumption that if we try to get a real random set of
> packages checked instead of checking those who are ending up by random
> reasons in new we will end up with less re-checked packages.  However,
> this does not give any good reason for keeping the habit to re-check
> packages where a resulting binary package is not inside the package
> pool.

There is one check that does certainly make sense for new binary packages,
which is: are the new binary package names namespace-polluting? Part
of the purpose of the NEW queue is to stop misleading or inappropriate
names from being used.

I think the reason for the copyright re-check being done at this point
might simply be that the ftp team are looking at the package anyway,
and the tools they're using were primarily designed for source-NEW.

I agree that both the copyright check and the namespace-pollution check
need to be done for source-NEW, and the namespace-pollution check needs
to be done for binary-NEW. I'm less sure about the value of a copyright
check at binary-NEW time.

smcv

Re: Updated proposal for improving the FTP NEW process

[Samuel Thibault]

Andreas Tille, on sam. 10 mars 2018 12:32:50 +0100, wrote:
> On Tue, Mar 06, 2018 at 04:44:43PM -0700, Sean Whitton wrote:
> > > On Mon, Mar 05, 2018 at 02:43:34PM -0700, Sean Whitton wrote:
> > >> If a package is maintained in git, then re-using a version number
> > >> means force-pushing a git tag
> > > Just don't tag uploads until they are accepted.
> 
> +1
> 
> > This means that uploading to new is no longer fire-and-forget.  That's
> > bad.
> 
> Why?  You get a reminder e-mail saying "ACCEPTED" in case you really
> forgot your package in new.

Actually I tag on upload, but push tags on ACCEPTED reception.  That
way at worse if I forget once, the tag will be pushed on next accepted
upload.

Samuel

Re: Updated proposal for improving the FTP NEW process

[Andreas Tille]

Hi Sean,

On Tue, Mar 06, 2018 at 04:44:43PM -0700, Sean Whitton wrote:
> > On Mon, Mar 05, 2018 at 02:43:34PM -0700, Sean Whitton wrote:
> >> If a package is maintained in git, then re-using a version number
> >> means force-pushing a git tag
> > Just don't tag uploads until they are accepted.

+1

> This means that uploading to new is no longer fire-and-forget.  That's
> bad.

Why?  You get a reminder e-mail saying "ACCEPTED" in case you really
forgot your package in new.  For me that's always a good point in time
to recheck for new versions etc.  So why should a maintainer forget
about a package that's in new?

Kind regards

  Andreas.

-- 
http://fam-tille.de

Re: Updated proposal for improving the FTP NEW process

[Andreas Tille]

On Tue, Mar 06, 2018 at 11:40:52PM +, Holger Levsen wrote:
> On Tue, Mar 06, 2018 at 05:54:55PM +0100, Adam Borowski wrote:
> > With my one of most active sponsors hat on: the current policy is that a
> > version that has never hit the archive must not have a separate changelog
> > entry, unless there are non-negligible users (such as a derivative, upstream
> > repository or at least the package being deployed to multiple users at a
> > workplace).  A past history is more acceptable than repeated attempts for an
> > upload.
> > 
> > This is what I was taught, and what I not only recommend but also require of
> > sponsorees.  There seems to be a concensus on -mentors that this is the
> > right way.

I always understand that what Adam states here is some consensus and I
follow the same policy to not have any gaps in version history (for
exactly the reasons Adam has given).
 
> with my sponsoring hat on, I will be unhappy if someone reuses version
> numbers and I will ask to never do this again. I very much agree with
> Ian's position that this is bad.
> 
> As a sponsor, I'm a non-negligible user and I want to sensible be able
> to not having to again review stuff I already have reviewed.
> 
> If you have put it on mentors.d.n, it's out in the public.

Disclaimer: I do not sponsor from mentors.d.n but I require my sponsees
to use an apropriate team Vcs and the tag will be only set if the
package has hit unstable.  I also set tags for my own packages only
after beeing accepted in unstable.

I'm afraid there is no right or wrong in this question.

Kind regards

  Andreas.

-- 
http://fam-tille.de

Re: Updated proposal for improving the FTP NEW process

[Andreas Tille]

Hi Chris,

On Wed, Mar 07, 2018 at 02:44:32PM +, Chris Lamb wrote:
> > I think the suggestion of randomized spot checking is meant to replace - 
> > not add - to the present system of checking that penalizes uploads of 
> > existing source but new binaries.  So human resources should not be the 
> > issue.
> 
> In my experience, time/energy/focus is not as fungible or easily
> transferable as you imply.

I share your assumption that if we try to get a real random set of
packages checked instead of checking those who are ending up by random
reasons in new we will end up with less re-checked packages.  However,
this does not give any good reason for keeping the habit to re-check
packages where a resulting binary package is not inside the package
pool.  It somehow reminds me to those people who were asked: "Why are
you doing this?" and gave the answer: "Since we did so all the time."

We all know that ftpmasters have a lot of work and this thread is about
convincing ftpmaster to stop some work that does not belong to their
initial task which is *checking new source packages*.  So far I've read
a single example in this thread that a developer was happy about the
check since a mistake was avoided.  But this would have happened by a
random user via BTS as well.

Kind regards

  Andreas.

-- 
http://fam-tille.de

Re: Updated proposal for improving the FTP NEW process

[Joerg Jaspert]

On 14969 March 1977, Gert Wollny wrote:

> Sure thing, I'll give it a try. Since I'm not familiar with the dak
> code, would you be so kind to point me to the functions and variables
> (if available) that are there to 
>   - extract or hold the bugs listed in the last changelog entry,
>   - query the BTS (to be able to get the header and see whether 
> it's a ITP) (if this is not available I can get that probably 
> from bugreport)
>   - where you compose the final email (to add the bug in the CC).

Your starting point is process-new.py

-- 
bye, Joerg

Re: Updated proposal for improving the FTP NEW process

[Adam Borowski]

On Wed, Mar 07, 2018 at 08:35:08PM +1100, Ben Finney wrote:
> Gert Wollny  writes:
> 
> > […] simply asking the peers doesn't make the process very public.
> 
> That is, IIUC, by design and for good reason.
> 
> Before a review of the copyright status of the work is done, we don't
> have confidence the Debian Project has permission to redistribute it
> publicly.
> 
> If it turns out the work is not redistributable by the Debian Project,
> it is then too late; and the case could be made that we should not allow
> that situation to arise (i.e. we should not redistribute before we have
> confidence the work can legally be permitted by us).
> 
> So, putting it up on some public Debian infrastructure before it passes
> review is, IIUC, problematic for that reason at least.

This argument doesn't pass laugh test.

There's plenty of places where anyone -- not a DD, not a DM -- can
distribute arbitrary files using Debian infrastructure.  Beside the obvious
(Alioth, Salsa), there's the BTS.  Mailing list archives.  Wiki.  Etc, etc.

On the other hand, to upload to NEW, you need to be a full DD.  Ie, the file
being _temporarily_ distributed passed judgement of someone who passed a
lengthy process that includes copyright training.  Not even a DM is
considered good enough.  Such a person may still make a mistake but it'd
be something minor (you can look at REJECTs you got yourself).  And then,
NEW is a temporary staging area where files get removed after review.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄ A master species delegates.

Re: Updated proposal for improving the FTP NEW process

[Ian Campbell]

On Wed, 2018-03-07 at 08:05 -0600, Steve Robbins wrote:
> I think the suggestion of randomized spot checking is meant to
> replace - not add - to the present system of checking that penalizes
> uploads of existing source but new binaries. So human resources
> should not be the issue. 

Yes, that's exactly what I meant, sorry for not making that clear.

> I would imagine that the packages currently being selected are not
> arbitrary - they are weighted towards library code. Is that fair to
> say? 

I'd reckon so (because I imagine that SONAME bumps probably account for
a largish fraction of binNEW packages).

Another big one I know of is the kernel which hits it for each ABI bump
(at least).

Ian.

Re: Updated proposal for improving the FTP NEW process

[Chris Lamb]

Steve,

> I think the suggestion of randomized spot checking is meant to replace - 
> not add - to the present system of checking that penalizes uploads of 
> existing source but new binaries.  So human resources should not be the 
> issue.

In my experience, time/energy/focus is not as fungible or easily
transferable as you imply.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: Updated proposal for improving the FTP NEW process

[Steve Robbins]

I think the suggestion of randomized spot checking is meant to replace - not 
add - to the present system of checking that penalizes uploads of existing 
source but new binaries.  So human resources should not be the issue. 

I would imagine that the packages currently being selected are not arbitrary -  
they are weighted towards library code.  Is that fair to say? 



On March 7, 2018 12:02:10 AM CST, Chris Lamb  wrote:
>Andrey Rahmatullin wrote:
>
>> > > I know for a fact that quite regularly licence checks on binNEW
>packages
>> > > causes RC bugs to pop up.  I acknowledge it may be a burder for
>the ftp
>> > > team, but that reason alone probably deserves to keep binNEW as
>it is.
>> > 
>> > That would seem to justify some sort of randomized spot checks [..]
>>
>> Exactly.
>
>Whilst it does seem a little odd, there is some merit the current
>system
>where packages get essentially-arbitrary chosen for a cursory glance by
>a
>member the FTP team.
>
>The team is already rather time-limited so an expectation of
>DFSG-checks
>of random packages already in the archive seems a little optimistic.
>
>(Identifying various types of NEWness might still be marginally useful
>for
>categorising new.html and similar interfaces, mind you.)
>
>
>Regards,
>
>-- 
>  ,''`.
> : :'  : Chris Lamb
> `. `'`  la...@debian.org / chris-lamb.co.uk
>   `-

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Updated proposal for improving the FTP NEW process

[Steve Robbins]

My preferred algorithm is dead simple : if the source package is the same, it 
is not NEW.

Sorry for the top -post.  My mobile device client is deficient.

On March 6, 2018 11:34:29 AM CST, Bastian Blank  wrote:
>Hi Steve
>
>Please don't top-post and fix the length of your lines.
>
>On Tue, Mar 06, 2018 at 10:22:22AM -0600, Steve Robbins wrote:
>> Or: change the mechanism to avoid a trip through NEW for simple cases
>that Chris outlined: new binary or soname bump.  Reserve NEW for truly
>new things.
>
>Can you describe an algorithm that can destinguish between an soname
>change and some other change?
>
>Bastian
>
>-- 
>We do not colonize.  We conquer.  We rule.  There is no other way for
>us.
>   -- Rojan, "By Any Other Name", stardate 4657.5

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Updated proposal for improving the FTP NEW process

[Gert Wollny]

Am Mittwoch, den 07.03.2018, 20:35 +1100 schrieb Ben Finney:
> Gert Wollny  writes:
> 
> > […] simply asking the peers doesn't make the process very public.
> 
> That is, IIUC, by design and for good reason.

> 
> Before a review of the copyright status of the work is done, we don't
> have confidence the Debian Project has permission to redistribute it
> publicly.

Besides of what I wrote below, even if the package in question would
not be in a public accessable place, handling the comments of such a
review in a public discussion (or BTS) should be not a problem.

> 
> If it turns out the work is not redistributable by the Debian
> Project, it is then too late; and the case could be made that we
> should not allow that situation to arise (i.e. we should not
> redistribute before we have confidence the work can legally be
> permitted by us).
> 
> So, putting it up on some public Debian infrastructure before it
> passes review is, IIUC, problematic for that reason at least.

As far as I can see, salsa.d.o is open to the public, and so is
mentors.d.n, both places where people would typically prepare or upload
 packages also before passing NEW for the first time. In the case of
mentors they actually have to upload their packages there.

So if there was an intention like this, it is not handled well. 

Best, 
Gert 

Re: Updated proposal for improving the FTP NEW process

[Jonathan McDowell]

On Wed, Mar 07, 2018 at 06:02:10AM +, Chris Lamb wrote:
> Andrey Rahmatullin wrote:
> > > > I know for a fact that quite regularly licence checks on binNEW
> > > > packages causes RC bugs to pop up.  I acknowledge it may be a
> > > > burder for the ftp team, but that reason alone probably deserves
> > > > to keep binNEW as it is.
> > > 
> > > That would seem to justify some sort of randomized spot checks
> > > [..]
> >
> > Exactly.
> 
> Whilst it does seem a little odd, there is some merit the current
> system where packages get essentially-arbitrary chosen for a cursory
> glance by a member the FTP team.

Speaking as someone who has taken over a package, been negligent in
ensuring debian/copyright was up to date and hit NEW as the result of a
soname update I am grateful for the time that the FTP team (Chris, as it
happens) invested to do a quick sanity check and tell me I was an idiot.
As a result I discovered licensecheck and have become much better
(though I doubt perfect) at ensuring such things stay up to date.

If we had enough spare people power then I've no doubt a concerted sweep
across the archive would find lots of packages that could do with some
TLC, whether copyrights or elsewhere, but realistically that's just not
going to happen.

J.

-- 
] https://www.earth.li/~noodles/ []  Minorities are the foundation of  [
]  PGP/GPG Key @ the.earth.li[]  society.  [
] via keyserver, web or email.   [][
] RSA: 4096/0x94FA372B2DA8B985   [][

Re: Updated proposal for improving the FTP NEW process

[Ben Finney]

Gert Wollny  writes:

> […] simply asking the peers doesn't make the process very public.

That is, IIUC, by design and for good reason.

Before a review of the copyright status of the work is done, we don't
have confidence the Debian Project has permission to redistribute it
publicly.

If it turns out the work is not redistributable by the Debian Project,
it is then too late; and the case could be made that we should not allow
that situation to arise (i.e. we should not redistribute before we have
confidence the work can legally be permitted by us).

So, putting it up on some public Debian infrastructure before it passes
review is, IIUC, problematic for that reason at least.

-- 
 \ “Please leave your values at the front desk.” —hotel, Paris |
  `\   |
_o__)  |
Ben Finney

Re: Updated proposal for improving the FTP NEW process

[Gert Wollny]

Am Mittwoch, den 07.03.2018, 08:09 +0100 schrieb Joerg Jaspert:
> 
> If someone comes up with a patch to process-new which does this in a
> halfway reliable way, it doesn't need a long time wasting thread on
> devel to get it.

Sure thing, I'll give it a try. Since I'm not familiar with the dak
code, would you be so kind to point me to the functions and variables
(if available) that are there to 
  - extract or hold the bugs listed in the last changelog entry,
  - query the BTS (to be able to get the header and see whether 
it's a ITP) (if this is not available I can get that probably 
from bugreport)
  - where you compose the final email (to add the bug in the CC).

Apart from that, AFAICS most of the discussion was around handling non-
ITP uploads that have to go through NEW, it's kind of orthogonal to
this proposal.

> 
> > (2) To improve the initial quality of uploads to NEW I also propose
> > the introduction a (voluntary) review step: Someone who is
> > interested in getting additional reviews to a package before
> > uploading it to NEW could file a "Request for review" (RFR) bug
> > against wnpp. [...]
> 
> And that is thankfully something everyone can just do (ask your peers
> for review). And is something that got proposed tons of times. Never
> see anything come from it.
I've not seen all the other proposals, so I can't comment, but simply
asking the peers doesn't make the process very public. In the worst
case the exchange is private. Handling such a review within the BTS
would help making the process visible to all, and adding the "Reviewed-
By" to the changelog and posssibly to the QA page would give
recognizable reputation of the reviewers.

In any case, you are right that everything apart from adding the R-b to
the QA page can simply be done, a formalization would be nice but is
initially not needed. 

best, 
Gert 

Re: Updated proposal for improving the FTP NEW process

[Joerg Jaspert]

On 14967 March 1977, Gert Wollny wrote:

I so like proposals from people who have never ever done any of the work
they propose something one. BUt hey...

> (1) Given that all new source package come with an ITP bug, when a
> package must be rejected, the FTP team could CC this bug in the
> rejection message. This would have the advantage that for everyone
> interested in the package the information why the package was rejected
> can easily be found. In addition, For large packages, where a review
> takes more than one day, the reviewer could send messages to the ITP
> about problems the moment they are found, so maintainers could start to
> work on correcting the errors earlier.

If someone comes up with a patch to process-new which does this in a
halfway reliable way, it doesn't need a long time wasting thread on
devel to get it.

> (2) To improve the initial quality of uploads to NEW I also propose the
> introduction a (voluntary) review step: Someone who is interested in
> getting additional reviews to a package before uploading it to NEW
> could file a "Request for review" (RFR) bug against wnpp. Then those
> who are willing to review packages can step in and also have a common
> place to comment on problems of the package that need fixing. When
> someone satisfied with the package they add a comment to the bug
> "Reviewed-By: name ", and when doing the actual upload, the
> maintainer replicates these R-B messages in the changelog closing the
> RFR bug. For large packages one might also add a comment "subir/module
> X Reviewed-By: ..." to indicate only a partial review.
> This R-B- information could also be added to that persons QA page under
> a new section "Reviewed Uploads".

> In a way this replicates what sponsors do for uploads of non-DDs, but
> especially for large packages a second pair of eyes is always helpful.

And that is thankfully something everyone can just do (ask your peers
for review). And is something that got proposed tons of times. Never see
anything come from it.

> Implementing the first point is essentially up the the FTP team.

No its not, its up to the ones that want it to patch dak process-new.
That is, parsing the bug-close list (if any), finding out if thats
really an ITP, and if so, CC it on rejection/prod.
Note that this has to be automatic, if it requires us to enter anything,
it is doomed to fail.

-- 
bye, Joerg

Re: Updated proposal for improving the FTP NEW process

[Andrey Rahmatullin]

On Wed, Mar 07, 2018 at 06:02:10AM +, Chris Lamb wrote:
> Andrey Rahmatullin wrote:
> 
> > > > I know for a fact that quite regularly licence checks on binNEW packages
> > > > causes RC bugs to pop up.  I acknowledge it may be a burder for the ftp
> > > > team, but that reason alone probably deserves to keep binNEW as it is.
> > > 
> > > That would seem to justify some sort of randomized spot checks [..]
> >
> > Exactly.
> 
> Whilst it does seem a little odd, there is some merit the current system
> where packages get essentially-arbitrary chosen for a cursory glance by a
> member the FTP team.
> 
> The team is already rather time-limited so an expectation of DFSG-checks
> of random packages already in the archive seems a little optimistic.
"Let's abolish binary-NEW checks"
"They are useful to catch copyright problems as a byproduct"
"Maybe we should replace them with actual copyright checks on more
randomly selected packages"
"We don't have manpower"
"LET'S ABOLISH BINARY-NEW CHECKS"


-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Updated proposal for improving the FTP NEW process

[Chris Lamb]

Andrey Rahmatullin wrote:

> > > I know for a fact that quite regularly licence checks on binNEW packages
> > > causes RC bugs to pop up.  I acknowledge it may be a burder for the ftp
> > > team, but that reason alone probably deserves to keep binNEW as it is.
> > 
> > That would seem to justify some sort of randomized spot checks [..]
>
> Exactly.

Whilst it does seem a little odd, there is some merit the current system
where packages get essentially-arbitrary chosen for a cursory glance by a
member the FTP team.

The team is already rather time-limited so an expectation of DFSG-checks
of random packages already in the archive seems a little optimistic.

(Identifying various types of NEWness might still be marginally useful for
categorising new.html and similar interfaces, mind you.)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: Updated proposal for improving the FTP NEW process

[Adam Borowski]

On Tue, Mar 06, 2018 at 06:15:10PM +, Ian Jackson wrote:
> Adam Borowski writes ("Re: Updated  proposal for improving the FTP NEW 
> process"):
> > On Tue, Mar 06, 2018 at 03:17:11PM +, Ian Jackson wrote:
> > > IMO Debian's rules should require that the revision should be
> > > incremented (at least) when you have shared the previous revision with
> > > other people as part of your Debian work.  That includes people who
> > > are processing NEW, sponsors, etc.
> > 
> > With my one of most active sponsors hat on: the current policy is that a
> > version that has never hit the archive must not have a separate changelog
> > entry, unless there are non-negligible users (such as a derivative, upstream
> > repository or at least the package being deployed to multiple users at a
> > workplace).  A past history is more acceptable than repeated attempts for an
> > upload.
> 
> Certainly in the exceptions you list, a separate changelog entry is a
> good idea.

Yeah, but my point is that, except those cases, current consensus seems to
be that there should be no separate changelog entries, no versions and no
tags.

(I say "seems" because there was no formal vote or formal thread, but when a
sponsoree produces such a changelog, no one among usual sponsors ever said
anything to the contrary.)

> I don't see where this policy you are implementing is written down.
> It doesn't seem to exist in policy or even the dev ref.

Alas, same applies to most of conventions.  It would be good to have such
things documented rather than learned by osmosis, but describing processes
is something no one volunteers for.

> > A changelog bloated with every replaced attempt is hard to read; gaps in
> > version numbering that come without an explanation also raise an eyebrow
> > (thus such a gap needs a comment in the changelog).
> 
> How many replaced attempts are we typically talking about ?

Ten is a quite extreme case, but 3-4 are common.  (Obviously most uploads
get in on the first go.)

> When an attempt is replaced, the reviewer would no doubt like not only
> the whole new package, but a human-readable summary of what the
> submitter has intentionally changed.  How do you think that
> human-readable summary ought to be communicated to the reviewer ?

That's what we have RFS bugs.  Every publicly reviewed upload has a
permanent trail.  The majority of uploads go this way.

There's no requirement to ask for review publicly, but private channels also
allow adequate communication with the reviewer.  Heck, as someone who
sponsors ~10× as many uploads as I do myself, I tend to use the same routine
I do for sponsoring for my own uploads.  In this case, the communication is
entirely within my brain!

And, the summary is not what's being checked.  I for one tend to skim over
RFS mails with so little attention I sometimes miss remarks from the
sponsoree -- looking only at debian/changelog.  This is what I read
carefully, comparing what's written there with what's in debdiff vs the last
version in the archive.

Comparing debdiffs is the main manual part of review.  And why would I care
that the shed got repainted red then blue, if it was black the last time it
was in the archive and is pink now?  This kind of details is ok for untagged
commits in git, but neither the reviewer nor the end user are interested in
changes that have been superseded between tags/uploads.  The process that
led you to decide on pink might be interesting but it's not what is being
reviewed.

The less _irrelevant_ information, the less work for us.  If the talk is
archived on the RFS bug, there's no information loss either.

> I disagree that gaps in the version numbering require an explanation,
> but even if they do then that is not a problem.

Gaps can be useful, but most of the time they're unintentional, that's why
there's a need for a comment.
 
> (Of course that the Debian revision does not need to be a single
> integer.  If you want to distinguish various attempts from the
> accepted submissions one could write -1.1 or something.)

Uhm, that's reserved for NMUs.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄ A master species delegates.

Re: Updated proposal for improving the FTP NEW process

[Sean Whitton]

Hello,

On Tue, Mar 06 2018, Andrey Rahmatullin wrote:

> On Mon, Mar 05, 2018 at 02:43:34PM -0700, Sean Whitton wrote:
>> If a package is maintained in git, then re-using a version number
>> means force-pushing a git tag
> Just don't tag uploads until they are accepted.

This means that uploading to new is no longer fire-and-forget.  That's
bad.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Updated proposal for improving the FTP NEW process

[Holger Levsen]

On Tue, Mar 06, 2018 at 05:54:55PM +0100, Adam Borowski wrote:
> With my one of most active sponsors hat on: the current policy is that a
> version that has never hit the archive must not have a separate changelog
> entry, unless there are non-negligible users (such as a derivative, upstream
> repository or at least the package being deployed to multiple users at a
> workplace).  A past history is more acceptable than repeated attempts for an
> upload.
> 
> This is what I was taught, and what I not only recommend but also require of
> sponsorees.  There seems to be a concensus on -mentors that this is the
> right way.

with my sponsoring hat on, I will be unhappy if someone reuses version
numbers and I will ask to never do this again. I very much agree with
Ian's position that this is bad.

As a sponsor, I'm a non-negligible user and I want to sensible be able
to not having to again review stuff I already have reviewed.

If you have put it on mentors.d.n, it's out in the public.


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: Updated proposal for improving the FTP NEW process

[Sean Whitton]

Hello,

On Tue, Mar 06 2018, Adam Borowski wrote:

> gaps in version numbering that come without an explanation also raise
> an eyebrow (thus such a gap needs a comment in the changelog).

I think this is the problem.

Ian's concern about non-identical but identically versioned source
packages flying around could be satisfied by bumping the version number
but retaining a single changelog entry.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Updated proposal for improving the FTP NEW process

[Lars Wirzenius]

On Wed, 2018-03-07 at 00:30 +0500, Andrey Rahmatullin wrote:
> On Tue, Mar 06, 2018 at 07:27:40PM +, Ian Campbell wrote:
> > > I know for a fact that quite regularly licence checks on binNEW packages
> > > causes RC bugs to pop up.  I acknowledge it may be a burder for the ftp
> > > team, but that reason alone probably deserves to keep binNEW as it is.
> > 
> > That would seem to justify some sort of randomized spot checks on the
> > archive, not arbitrarily focussing on the subset of packages which
> > happen to need a new binary package for some reason.
> 
> Exactly.

It's almost spring in northern Europe and with the lengthening day I
start getting many crazy ideas. Here's one: it would be truly awesome
if we could review each source package at least once per Debian
release cycle. I don't think that's possible, it would be awesome if
it were.

There is, in unstable, about 28000 source packages right now, if I'm
counting correctly. A release cycle is about two years. That's about
40 source packages per day, every day. That would require either a
very large number of extra volunteer reviewers, or automation.

If most upstreams were systemtically tagging (perhaps using SPDX)
their sources with licence information, or we had a mostly reliable
tool for deducing that information automatically, this might be
feasible.


signature.asc
Description: This is a digitally signed message part

Re: Updated proposal for improving the FTP NEW process

[Andrey Rahmatullin]

On Tue, Mar 06, 2018 at 07:27:40PM +, Ian Campbell wrote:
> > I know for a fact that quite regularly licence checks on binNEW packages
> > causes RC bugs to pop up.  I acknowledge it may be a burder for the ftp
> > team, but that reason alone probably deserves to keep binNEW as it is.
> 
> That would seem to justify some sort of randomized spot checks on the
> archive, not arbitrarily focussing on the subset of packages which
> happen to need a new binary package for some reason.
Exactly.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Updated proposal for improving the FTP NEW process

[Ian Campbell]

On Tue, 2018-03-06 at 19:18 +0100, Mattia Rizzolo wrote:
> On Tue, Mar 06, 2018 at 06:34:28PM +0100, Adam Borowski wrote:
> > What might be reasonable is making the separation between binNEW and srcNEW
> > more obvious, especially for us on the other side.  This would also
> > encourage the ftpmasters to skip license checks for binNEW -- but again,
> > without knowing their workflow I'm not in a position to improve it.
> 
> I know for a fact that quite regularly licence checks on binNEW packages
> causes RC bugs to pop up.  I acknowledge it may be a burder for the ftp
> team, but that reason alone probably deserves to keep binNEW as it is.

That would seem to justify some sort of randomized spot checks on the
archive, not arbitrarily focussing on the subset of packages which
happen to need a new binary package for some reason.

Ian.

Re: Updated proposal for improving the FTP NEW process

[Andrey Rahmatullin]

On Tue, Mar 06, 2018 at 06:15:10PM +, Ian Jackson wrote:
> > A changelog bloated with every replaced attempt is hard to read; gaps in
> > version numbering that come without an explanation also raise an eyebrow
> > (thus such a gap needs a comment in the changelog).
> 
> How many replaced attempts are we typically talking about ?
A lot?

> When an attempt is replaced, the reviewer would no doubt like not only
> the whole new package, but a human-readable summary of what the
> submitter has intentionally changed.  How do you think that
> human-readable summary ought to be communicated to the reviewer ?
As a reply to the email containing the review?

Please look at debian-mentors@ if you have additional questions. Thank
you.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Updated proposal for improving the FTP NEW process

[Mattia Rizzolo]

On Tue, Mar 06, 2018 at 06:34:28PM +0100, Adam Borowski wrote:
> What might be reasonable is making the separation between binNEW and srcNEW
> more obvious, especially for us on the other side.  This would also
> encourage the ftpmasters to skip license checks for binNEW -- but again,
> without knowing their workflow I'm not in a position to improve it.

I know for a fact that quite regularly licence checks on binNEW packages
causes RC bugs to pop up.  I acknowledge it may be a burder for the ftp
team, but that reason alone probably deserves to keep binNEW as it is.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: Updated proposal for improving the FTP NEW process

[Ian Jackson]

Adam Borowski writes ("Re: Updated  proposal for improving the FTP NEW 
process"):
> On Tue, Mar 06, 2018 at 03:17:11PM +, Ian Jackson wrote:
> > IMO Debian's rules should require that the revision should be
> > incremented (at least) when you have shared the previous revision with
> > other people as part of your Debian work.  That includes people who
> > are processing NEW, sponsors, etc.
> 
> With my one of most active sponsors hat on: the current policy is that a
> version that has never hit the archive must not have a separate changelog
> entry, unless there are non-negligible users (such as a derivative, upstream
> repository or at least the package being deployed to multiple users at a
> workplace).  A past history is more acceptable than repeated attempts for an
> upload.

Certainly in the exceptions you list, a separate changelog entry is a
good idea.

I don't see where this policy you are implementing is written down.
It doesn't seem to exist in policy or even the dev ref.

> A changelog bloated with every replaced attempt is hard to read; gaps in
> version numbering that come without an explanation also raise an eyebrow
> (thus such a gap needs a comment in the changelog).

How many replaced attempts are we typically talking about ?

When an attempt is replaced, the reviewer would no doubt like not only
the whole new package, but a human-readable summary of what the
submitter has intentionally changed.  How do you think that
human-readable summary ought to be communicated to the reviewer ?

I disagree that gaps in the version numbering require an explanation,
but even if they do then that is not a problem.

(Of course that the Debian revision does not need to be a single
integer.  If you want to distinguish various attempts from the
accepted submissions one could write -1.1 or something.)

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Updated proposal for improving the FTP NEW process

[Bastian Blank]

Hi Steve

Please don't top-post and fix the length of your lines.

On Tue, Mar 06, 2018 at 10:22:22AM -0600, Steve Robbins wrote:
> Or: change the mechanism to avoid a trip through NEW for simple cases that 
> Chris outlined: new binary or soname bump.  Reserve NEW for truly new things.

Can you describe an algorithm that can destinguish between an soname
change and some other change?

Bastian

-- 
We do not colonize.  We conquer.  We rule.  There is no other way for us.
-- Rojan, "By Any Other Name", stardate 4657.5

Re: Updated proposal for improving the FTP NEW process

[Adam Borowski]

On Tue, Mar 06, 2018 at 10:22:22AM -0600, Steve Robbins wrote:
> Or: change the mechanism to avoid a trip through NEW for simple cases that
> Chris outlined: new binary or soname bump.  Reserve NEW for truly new
> things.

Let's not try to be wiser than ftpmasters: if, despite the increased
workload, they haven't expressed such an obvious wish yet, they obviously
value sanity checks here.

What might be reasonable is making the separation between binNEW and srcNEW
more obvious, especially for us on the other side.  This would also
encourage the ftpmasters to skip license checks for binNEW -- but again,
without knowing their workflow I'm not in a position to improve it.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄ A master species delegates.

Re: Updated proposal for improving the FTP NEW process

[Steve Robbins]

Or: change the mechanism to avoid a trip through NEW for simple cases that 
Chris outlined: new binary or soname bump.  Reserve NEW for truly new things.


On March 5, 2018 9:00:06 AM CST, "W. Martin Borgert"  wrote:
>Quoting Chris Lamb :
>>> In many cases, there is an issue open about the new binary package
>>
>> (In my experience, there is not.)
>>
>>> When there is no bug report open at all, well, bad luck.
>>
>> Well, possbibly, but if one is investing time and effort in changing
>a
>> process it seems a shame not to cover these cases IMHO. :)
>
>True. Proposal: Maintainer should make sure they have a bug open about
>any new binary packages and close them with the upload. If they forget
>this "goto badluck;".

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Updated proposal for improving the FTP NEW process

[Adam Borowski]

On Tue, Mar 06, 2018 at 03:17:11PM +, Ian Jackson wrote:
> Scott Kitterman writes ("Re: Updated  proposal for improving the FTP NEW 
> process"):
> > If you consider it absurd to not increment the revision due to
> > changes that never made it in the archive, then I don't know where
> > it stops.
> 
> IMO Debian's rules should require that the revision should be
> incremented (at least) when you have shared the previous revision with
> other people as part of your Debian work.  That includes people who
> are processing NEW, sponsors, etc.

With my one of most active sponsors hat on: the current policy is that a
version that has never hit the archive must not have a separate changelog
entry, unless there are non-negligible users (such as a derivative, upstream
repository or at least the package being deployed to multiple users at a
workplace).  A past history is more acceptable than repeated attempts for an
upload.

This is what I was taught, and what I not only recommend but also require of
sponsorees.  There seems to be a concensus on -mentors that this is the
right way.

I'm not arguing that a change here is unacceptable, merely describing
currently agreed upon convention.

A changelog bloated with every replaced attempt is hard to read; gaps in
version numbering that come without an explanation also raise an eyebrow
(thus such a gap needs a comment in the changelog).

You can have private history published publicly in git, but it's best to
refrain from pushing a tag until the version has been accepted.  That's why
git doesn't push tags by default when you push the branch they apply to.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄ A master species delegates.

Re: Updated proposal for improving the FTP NEW process

[Ian Jackson]

Scott Kitterman writes ("Re: Updated  proposal for improving the FTP NEW 
process"):
> If you consider it absurd to not increment the revision due to
> changes that never made it in the archive, then I don't know where
> it stops.

IMO Debian's rules should require that the revision should be
incremented (at least) when you have shared the previous revision with
other people as part of your Debian work.  That includes people who
are processing NEW, sponsors, etc.

I hope the reasons why this is a sensible place to stop are obvious.

If you want to increment the revision for every git commit then as far
as I'm concerned that's between you and your computer :-).  (There are
sometimes good reasons for doing so when you are building binaries...)

>  I admit, this was hyperbole, but Ian's extremism annoys
> me.  I should do a better job of ignoring it.

I'm sorry to post messages that you feel like you should be ignoring.
Thanks for arguing back anyway.

> I'm not sure you actually read what I wrote since I wrote that I
> thought REQUIRING the revision to be bumped was a bad idea and you
> gave me a case where it made sense to do so.  Nowhere in this thread
> have I ever said bumping the revision is inherently a bad idea.

I am indeed suggesting that there should be a requirement.

Ultimately the purpose of the version number is so that we can
distinguish packages with the same name and different contents.  These
distinctions need to be made on users' systems and also in the proper
archive suites, but they also need to be made in NEW and in the
sponsorship queue, and when sharing informally.

Once source packages are being thrown about (in NEW, in sponsorship,
etc.), having different packages which are apparently identical (same
filenames, same metadata) is unreasonably confusing.  It requires the
people who deal with them to invent ad-hoc overlay versioning schemes.

This is all very silly[1] IMO, when we could use the existing version
number field to identify the version of the package.

AFAICT the only reasons people don't bump the revision when they
re-upload to NEW are:

 * Some of our upload tools DTWT by default if you bump the version
   number and/or add changelog stanzas, for versions that didn't make
   it to the archive suite in question.  This is IMO a tooling
   problem.  (You could use my tool and not suffer this problem; or
   you could improve the other tools.)

 * Some people have a misplaced sense that not using -1 or whatever is
   "untidy".  It's true that it does leave visible traces, but those
   visible traces are the record of what really happened.  Depending
   on exactly what happened, they can be useful in the future.  They
   are certainly not harmful.  This relates to my general refrain that
   integers are cheap and we should not be afraid to "waste" a few.

I hope you find this message more to your taste.  [1] despite my use
of the word "silly".

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Updated proposal for improving the FTP NEW process

[Andrey Rahmatullin]

On Mon, Mar 05, 2018 at 02:43:34PM -0700, Sean Whitton wrote:
> If a package is maintained in git, then re-using a version number means
> force-pushing a git tag
Just don't tag uploads until they are accepted.  

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Updated proposal for improving the FTP NEW process

[Scott Kitterman]

On Monday, March 05, 2018 04:46:27 PM Sean Whitton wrote:
> Hello,
> 
> On Mon, Mar 05 2018, Scott Kitterman wrote:
> > I'm not sure you actually read what I wrote since I wrote that I
> > thought REQUIRING the revision to be bumped was a bad idea and you
> > gave me a case where it made sense to do so.  Nowhere in this thread
> > have I ever said bumping the revision is inherently a bad idea.
> 
> Thanks.  I did read this, and wasn't trying to argue against the
> position that bumping the revision is bad.  But my e-mail might have
> made it seem that I didn't pay enough attention to what you wrote, so
> please accept my apologies for writing with insufficient care.

No problem.  Thanks for following up.

Scott K

signature.asc
Description: This is a digitally signed message part.

Re: Updated proposal for improving the FTP NEW process

[Sean Whitton]

Hello,

On Mon, Mar 05 2018, Scott Kitterman wrote:

> I'm not sure you actually read what I wrote since I wrote that I
> thought REQUIRING the revision to be bumped was a bad idea and you
> gave me a case where it made sense to do so.  Nowhere in this thread
> have I ever said bumping the revision is inherently a bad idea.

Thanks.  I did read this, and wasn't trying to argue against the
position that bumping the revision is bad.  But my e-mail might have
made it seem that I didn't pay enough attention to what you wrote, so
please accept my apologies for writing with insufficient care.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Updated proposal for improving the FTP NEW process

[Scott Kitterman]

On Monday, March 05, 2018 02:43:34 PM Sean Whitton wrote:
> Hello,
> 
> On Mon, Mar 05 2018, Scott Kitterman wrote:
> > Taken to it's logical end, then every VCS commit should have it's own
> > revision.
> 
> Could you explain how this follows?  I don't see it.

If you consider it absurd to not increment the revision due to changes that 
never made it in the archive, then I don't know where it stops.  I admit, this 
was hyperbole, but Ian's extremism annoys me.  I should do a better job of 
ignoring it.

> > I think requiring a maintainer to increment the Debian revision of a
> > package based on things that happen outside the Debian archive is "not
> > a good idea'[1].
> 
> If a package is maintained in git, then re-using a version number means
> force-pushing a git tag, which can get quite confusing quite fast (it's
> worse than force-pushing a branch, IME).

Sure.  I completely agree that there are cases where it's reasonable to do so.  
All I'm arguing is that it's not absurd not to.  There are quite reasonable 
cases where one need not do so and it leaves, IMO, a cleaner history.

Honestly, I think Ian's use of 'absurd' was absurd.  I think there are 
reasonable cases for doing it either way and there's no need to be absolutist 
about it.

I'm not sure you actually read what I wrote since I wrote that I thought 
REQUIRING the revision to be bumped was a bad idea and you gave me a case 
where it made sense to do so.  Nowhere in this thread have I ever said bumping 
the revision is inherently a bad idea.

These threads spin off into endless tangents easily enough without people 
writing responses to what they imagined other people wrote.

Scott K

signature.asc
Description: This is a digitally signed message part.

Re: Updated proposal for improving the FTP NEW process

[Sean Whitton]

Hello,

On Mon, Mar 05 2018, Scott Kitterman wrote:

> Taken to it's logical end, then every VCS commit should have it's own
> revision.

Could you explain how this follows?  I don't see it.

> I think requiring a maintainer to increment the Debian revision of a
> package based on things that happen outside the Debian archive is "not
> a good idea'[1].

If a package is maintained in git, then re-using a version number means
force-pushing a git tag, which can get quite confusing quite fast (it's
worse than force-pushing a branch, IME).

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Updated proposal for improving the FTP NEW process

[Sean Whitton]

Hello,

On Mon, Mar 05 2018, Philipp Kern wrote:

> The most common counterpoint to bumping the version is that it's
> harder to get right because for some reason our tools rely on the
> whole delta to be present in the .changes file rather than inferring
> it from the in-package changelog. So bug closure information gets lost
> unless you're careful in how you build the package, which makes it
> more likely to get wrong.

Just a note that dgit always gets this right, including exactly what's
needed in the .changes without user intervention.  So if you're hitting
this often, consider switching to `dgit push-source` and `dgit push`.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: Updated proposal for improving the FTP NEW process

[Philipp Kern]

On 03/05/2018 08:08 PM, Scott Kitterman wrote:
> On March 5, 2018 7:01:14 PM UTC, Don Armstrong  wrote:
>> On Mon, 05 Mar 2018, Scott Kitterman wrote:
>>> I think requiring a maintainer to increment the Debian revision of a
>>> package based on things that happen outside the Debian archive is
>>> "not" a good idea'[1].
>> I don't want to make it a requirement, just recommended good practice
>> when an upload has been made to Debian, and subsequent upload requires
>> changing the source package.
> I don't mind if people increment.  I understand that there are reasonable 
> reasons for doing so.  I was responding to the suggestion that not 
> incrementing the revision is absurd.  I think it's fine either way.

The most common counterpoint to bumping the version is that it's harder
to get right because for some reason our tools rely on the whole delta
to be present in the .changes file rather than inferring it from the
in-package changelog. So bug closure information gets lost unless you're
careful in how you build the package, which makes it more likely to get
wrong.

Kind regards
Philipp Kern

Re: Updated proposal for improving the FTP NEW process

[Scott Kitterman]

On March 5, 2018 7:01:14 PM UTC, Don Armstrong  wrote:
>On Mon, 05 Mar 2018, Scott Kitterman wrote:
>> I think requiring a maintainer to increment the Debian revision of a
>> package based on things that happen outside the Debian archive is
>"not
>> a good idea'[1].
>
>I don't want to make it a requirement, just recommended good practice
>when an upload has been made to Debian, and subsequent upload requires
>changing the source package.

I don't mind if people increment.  I understand that there are reasonable 
reasons for doing so.  I was responding to the suggestion that not incrementing 
the revision is absurd.  I think it's fine either way.

Scott K

Re: Updated proposal for improving the FTP NEW process

[Don Armstrong]

On Mon, 05 Mar 2018, Scott Kitterman wrote:
> I think requiring a maintainer to increment the Debian revision of a
> package based on things that happen outside the Debian archive is "not
> a good idea'[1].

I don't want to make it a requirement, just recommended good practice
when an upload has been made to Debian, and subsequent upload requires
changing the source package.

-- 
Don Armstrong  https://www.donarmstrong.com

Something the junk advertisers don't seem to understand: we live in an
information super-saturated world. If I don't want to buy something,
no amount of shouting or propagandizing will budge me; all it will do
is get me annoyed. On the other hand, if I have a need for your
product, I can seek it out in an eyeblink.
 -- Charles Stross "Toast: A Con Report" in _Toast_ p136

Re: Updated proposal for improving the FTP NEW process

[Scott Kitterman]

On Monday, March 05, 2018 05:54:59 PM Ian Jackson wrote:
> Gert Wollny writes ("Re: Updated  proposal for improving the FTP NEW 
process"):
> > The only option I see for doing this in the BTS would be to ask the ftp
> > team to file the reject messages as a new bug against the source
> > package. I refrained from proposing this because this would mean filing
> > a bug against a package version that is not yet available in Debian.
> > Since the re-upload to NEW would have the same version like the version
> > the bug is filed against, the BTS might get a hiccup. For that reason I
> > originally proposed doing this with the salsa issue tracker.
> 
> Personally I think this Debian practice of reusing version numbers for
> different packages is absurd.  If a package is rejected by ftpmaster
> (or by a sponsor, for that matter) the resubmission should have a new
> version number.

Taken to it's logical end, then every VCS commit should have it's own 
revision.

I think requiring a maintainer to increment the Debian revision of a package 
based on things that happen outside the Debian archive is "not a good 
idea'[1].

Scott K
[1] Imagine your own substantially stronger, but non-insulting phrase here.

Re: Updated proposal for improving the FTP NEW process

[Ian Jackson]

Gert Wollny writes ("Re: Updated  proposal for improving the FTP NEW process"):
> The only option I see for doing this in the BTS would be to ask the ftp
> team to file the reject messages as a new bug against the source
> package. I refrained from proposing this because this would mean filing
> a bug against a package version that is not yet available in Debian. 
> Since the re-upload to NEW would have the same version like the version
> the bug is filed against, the BTS might get a hiccup. For that reason I
> originally proposed doing this with the salsa issue tracker.

Personally I think this Debian practice of reusing version numbers for
different packages is absurd.  If a package is rejected by ftpmaster
(or by a sponsor, for that matter) the resubmission should have a new
version number.

Ian.

Re: Updated proposal for improving the FTP NEW process

[Don Armstrong]

On Mon, 05 Mar 2018, Gert Wollny wrote:
> Since the re-upload to NEW would have the same version like the
> version the bug is filed against

You don't have to upload with the same version (I usually don't, because
I've already tagged the previous upload in git). 

> the BTS might get a hiccup. For that reason I originally proposed
> doing this with the salsa issue tracker.

Even if you use the same version, so long as you mark the bug as fixed
in the version where it was uploaded everything should work properly.

-- 
Don Armstrong  https://www.donarmstrong.com

Who is thinking this?
I am.
 -- Greg Egan _Diaspora_ p38

Re: Updated proposal for improving the FTP NEW process

[Scott Kitterman]

On Monday, March 05, 2018 04:00:06 PM W. Martin Borgert wrote:
> Quoting Chris Lamb :
> >> In many cases, there is an issue open about the new binary package
> > 
> > (In my experience, there is not.)
> > 
> >> When there is no bug report open at all, well, bad luck.
> > 
> > Well, possbibly, but if one is investing time and effort in changing a
> > process it seems a shame not to cover these cases IMHO. :)
> 
> True. Proposal: Maintainer should make sure they have a bug open about
> any new binary packages and close them with the upload. If they forget
> this "goto badluck;".

Here's a thought:

How about documenting the reason for the change in the "changelog".  It can be 
frustrating reviewing a package with a new binary and zero information is 
provided about why it's there.

We already have debian/changelog that's designed explicitly for this kind of 
information, if only more maintainers would actually use it.

This has the added advantage of making the information available to system 
administrators who would like to understand why they have a new package on 
upgrade.

Scott K

Re: Updated proposal for improving the FTP NEW process

[Chris Lamb]

Simon McVittie wrote:

> Are binary-NEW packages more likely to violate the DFSG than randomly
> chosen packages? 

I can't speak to a randomly-selected package, but IME entirely new source
packages are less likely have problems IME than older ones that Just
Visiting, thus ...

> If the goal is to trigger periodic copyright/licensing
> review of already-accepted packages […]

... there is some merit the current system where (for example) a SONAME
bump of some package gets a cursory glance by a member the FTP team.

The team is already — err — arther stretched so an expectation of them
performing DFSG-checks of random packages seems a little optimistic.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: Updated proposal for improving the FTP NEW process

[Gert Wollny]

Am Montag, den 05.03.2018, 14:27 + schrieb Chris Lamb:
> Hi Gert,
> 
> > (1) Given that all new source package come with an ITP bug, when a
> > package must be rejected, the FTP team could CC this bug in the
> > rejection message.
> 
> Do you have any concrete suggestions for packages that are "Just
> Visiting" NEW, such as ones adding, say, a python3-foo, a -doc, a
> SONAME bump, first upload to experimental, etc. etc.? They do not
> have ITP bugs.

The only option I see for doing this in the BTS would be to ask the ftp
team to file the reject messages as a new bug against the source
package. I refrained from proposing this because this would mean filing
a bug against a package version that is not yet available in Debian. 
Since the re-upload to NEW would have the same version like the version
the bug is filed against, the BTS might get a hiccup. For that reason I
originally proposed doing this with the salsa issue tracker.

Apart from that it is my experience that packages that do not introduce
new source packages usually pass NEW a lot faster, and are less prone
to errors that result in a reject by the FTP team.

> I have no real idea about the stats on this but my gut feel is that
> this applies to at _least_ half of packages that pass through NEW.
> (The current items in the NEW queue will be rather misleading on
> this point.)

Specifically targeted at the new SO-version case, Steve Robbins
proposed to reword the policy that *only* uploads that introduce new
SOURCE packages go through NEW. 

I have to admid though, that I'm not sure whether this should apply to
all library packages. For instance another big package I co-maintain,
insighttoolkit4, introduces a new SO version at least every year, but
which each such version bump upstream also adds many files that may
have varying copyright that require an update of d/copyright. Giving
packages like this a free pass might not be such a good idea.

Best,
Gert

Re: Updated proposal for improving the FTP NEW process

[Simon McVittie]

On Mon, 05 Mar 2018 at 14:27:31 +, Chris Lamb wrote:
> > (1) Given that all new source package come with an ITP bug, when a
> > package must be rejected, the FTP team could CC this bug in the
> > rejection message.
> 
> Do you have any concrete suggestions for packages that are "Just
> Visiting" NEW, such as ones adding, say, a python3-foo, a -doc, a
> SONAME bump, first upload to experimental, etc. etc.? They do not
> have ITP bugs.

In principle there'd be nothing to stop us from having a policy that
NEW packages must close a bug that justifies the split/rename, even
if that bug is Severity: wishlist and was opened by the maintainer?
(e.g. "should have Python 3 version", "should split out documentation"
or "new upstream version" - although in some cases it would be annoying
for maintainers to have to wait for the BTS to send back a bug number
before they could upload.)

Another possibility would be for these packages to not get a
copyright/licensing check, just a sanity-check that their package names
are not polluting the package namespace any more than they were already.

Are binary-NEW packages more likely to violate the DFSG than randomly
chosen packages? I'm not a ftp team member, so I don't have the overview
you do; but in my experience, undermaintained packages rotting in the
archive seem to be *more* likely to have copyright/licensing issues than
actively-maintained packages that get uploaded and might have to pass
through binary-NEW, if only because ftp team review has got more careful
over time, so undermaintained packages were probably accepted under a less
careful regime. If the goal is to trigger periodic copyright/licensing
review of already-accepted packages, perhaps that should be orthogonal
to protecting the package namespace, and should act on packages in the
archive regardless of whether they are binary-NEW or not?

(In general, as I've said before, I think we are probably creating
more copyright/licensing work for ourselves than we really need to,
and I'd be pleased to see the burden placed on both the ftp team and
maintainers reduce.)

> first upload to experimental

Is this a thing? I wasn't aware that the first upload to experimental
for a package that was previously only in unstable triggered a visit to
NEW. (Although when I've uploaded to experimental for the first time, it
has often been to break out a separate binary package without blocking
unstable, so it would have gone to NEW anyway.)

smcv

Re: Updated proposal for improving the FTP NEW process

[Lars Wirzenius]

On Mon, 2018-03-05 at 14:51 +, Chris Lamb wrote:
> Hi Martin,
> 
> > In many cases, there is an issue open about the new binary package
> 
> (In my experience, there is not.)
> 
> > When there is no bug report open at all, well, bad luck.
> 
> Well, possbibly, but if one is investing time and effort in changing a
> process it seems a shame not to cover these cases IMHO. :)

I think that in cases there a package lands in NEW because it's added
a new binary package, or changed the name of one of its binary
packages, any package could be filed against the *source* package.


signature.asc
Description: This is a digitally signed message part

Re: Updated proposal for improving the FTP NEW process

[W. Martin Borgert]

Quoting Chris Lamb :

In many cases, there is an issue open about the new binary package


(In my experience, there is not.)


When there is no bug report open at all, well, bad luck.


Well, possbibly, but if one is investing time and effort in changing a
process it seems a shame not to cover these cases IMHO. :)


True. Proposal: Maintainer should make sure they have a bug open about
any new binary packages and close them with the upload. If they forget
this "goto badluck;".

Re: Updated proposal for improving the FTP NEW process

[Chris Lamb]

Hi Martin,

> In many cases, there is an issue open about the new binary package

(In my experience, there is not.)

> When there is no bug report open at all, well, bad luck.

Well, possbibly, but if one is investing time and effort in changing a
process it seems a shame not to cover these cases IMHO. :)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: Updated proposal for improving the FTP NEW process

[Debian/GNU]

On 2018-03-05 12:18, Gert Wollny wrote:
> (1) Given that all new source package come with an ITP bug, when a
> package must be rejected, the FTP team could CC this bug in the
> rejection message.

i would really like to see this.
sometimes i miss rejection emails - or at least wonder whether i missed
them. having a second place to look for such information would boost my
confidence.

fgasmdr
IOhannes

Re: Updated proposal for improving the FTP NEW process

[W. Martin Borgert]

Quoting Chris Lamb :

Do you have any concrete suggestions for packages that are "Just
Visiting" NEW, such as ones adding, say, a python3-foo, a -doc, a
SONAME bump, first upload to experimental, etc. etc.? They do not
have ITP bugs.


In many cases, there is an issue open about the new binary package.
E.g. "provide three.module.js as an es module" or "please add a
Python 3 package" etc. Maybe this can be used instead?

When there is no bug report open at all, well, bad luck.

Re: Updated proposal for improving the FTP NEW process

[Chris Lamb]

Hi Gert,

> (1) Given that all new source package come with an ITP bug, when a
> package must be rejected, the FTP team could CC this bug in the
> rejection message.

Do you have any concrete suggestions for packages that are "Just
Visiting" NEW, such as ones adding, say, a python3-foo, a -doc, a
SONAME bump, first upload to experimental, etc. etc.? They do not
have ITP bugs.

I have no real idea about the stats on this but my gut feel is that
this applies to at _least_ half of packages that pass through NEW.
(The current items in the NEW queue will be rather misleading on
this point.)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Re: Updated proposal for improving the FTP NEW process

[Paul Wise]

On Mon, Mar 5, 2018 at 7:18 PM, Gert Wollny wrote:

> (2) To improve the initial quality of uploads to NEW I also propose the
> introduction a (voluntary) review step:

These sort of things have been proposed multiple times before but
never materialised into policy, convention or common activity. Here is
an example:

https://wiki.debian.org/CopyrightReview

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Updated proposal for improving the FTP NEW process

[Gert Wollny]

Dear all, 

thanks for all the feedback, based on this I'd like to modify the
proposal like follows: 

(1) Given that all new source package come with an ITP bug, when a
package must be rejected, the FTP team could CC this bug in the
rejection message. This would have the advantage that for everyone
interested in the package the information why the package was rejected
can easily be found. In addition, For large packages, where a review
takes more than one day, the reviewer could send messages to the ITP
about problems the moment they are found, so maintainers could start to
work on correcting the errors earlier.

(2) To improve the initial quality of uploads to NEW I also propose the
introduction a (voluntary) review step: Someone who is interested in
getting additional reviews to a package before uploading it to NEW
could file a "Request for review" (RFR) bug against wnpp. Then those
who are willing to review packages can step in and also have a common
place to comment on problems of the package that need fixing. When
someone satisfied with the package they add a comment to the bug
"Reviewed-By: name ", and when doing the actual upload, the
maintainer replicates these R-B messages in the changelog closing the
RFR bug. For large packages one might also add a comment "subir/module
X Reviewed-By: ..." to indicate only a partial review.
This R-B- information could also be added to that persons QA page under
a new section "Reviewed Uploads".

In a way this replicates what sponsors do for uploads of non-DDs, but
especially for large packages a second pair of eyes is always helpful.

--

Implementing the first point is essentially up the the FTP team. 

For the second point some formalization would be required and made
public, for this I'd volunteer. Adding reviewed-by information to the
developers QA page would require that someone steps in who is knowledge
able in how these pages are created. In any case, this is only a "nice
to have" thingy.

any comments are welcome,
Gert