Re: General Resolution to deploy tag2upload

[Ansgar 🙀]

Hi Aigars,

On Thu, 2024-06-27 at 13:07 +0300, Aigars Mahinovs wrote:
> Refusing to make a decision is a decision. Ansgar has explicitly set
> a requirement for including the checksums of the end result Debian
> source package in the tag.

No, I have never made that a requirement. (Though an implementation
doing that would be fine too.)

Ansgar

> 

Re: [RFC] General Resolution to deploy tag2upload

[Ansgar 🙀]

Hi,

On Fri, 2024-06-28 at 13:00 +0200, Didier 'OdyX' Raboud wrote:
> Le vendredi, 28 juin 2024, 08.32:43 h CEST Ansgar 🙀 a écrit :
> > I'll expand on the here slightly for your benefit:
> > 
> > $ git clone https://salsa.debian.org/rra/tf5.git
> > [...]
> > $ apt-get source tf5
> > [...]
> > $ rm -rf tf5/.git tf5-5.0beta8/.pc
> > $ diff -Nur tf5 tf5-5.0beta8; echo $?
> > 0
> > 
> > If one is really bored:
> > $ (cd tf5; sha256sum $(find . -type f | sort) | sha256sum -)
> > 8d7820471fb44382a0c752319906064a1276ff18873fb4730dec1319aaf7b459  -
> > $ (cd tf5-5.0beta8; sha256sum $(find . -type f | sort) | sha256sum -)
> > 8d7820471fb44382a0c752319906064a1276ff18873fb4730dec1319aaf7b459  -
> > 
> > I will leave it as an exercise to you to compare the output and to
> > reason about results of different ways to compare both trees.
> 
> It looks to me that you have taken (by choice, or by chance) an example that 
> too conveniently fits what you want to demonstrate: in which the git 
> repository and the .dsc are treesame.

I was given this very specific example by Russ as an example where it
would not work and Sean continued to insist I rejected this specific
example as invalid by just asserting it doesn't demonstrate the problem
without any evidence.  I've tried to address Sean's concern by
expanding on my earlier reasoning.

> If I understand your position correctly (please correct me if needed): you 
> (with a ftpmaster hat) would like all uploads to come with a signed artefact 
> of hashes corresponding to the set of files as represented by the current 
> Debian source package format, as accepted by the archive today. And you would 
> like this artefact's signature be a signature by the human uploader. Did I 
> get 
> this right?

Ideally yes, though this might not work for all cases.

The next-best thing would be some reasonable representation of the
source package contents that could be signed by the submitter and
verified by the archive independently. (A signed artifact covering the
exact set of files/metadata would obviously satisfy this, but it can be
less than that.)

> If I understand dgit and tag2upload correctly, in the cases where the git 
> repository is treesame to the source package (patches-applied, with debian/
> patches file stored in git, as pointed by a tag), this artefact has the exact 
> same cryptographic value as the git tag, pointing to the git tree, pointing 
> to 
> the git objects (modulo the SHA-1 vs SHA-256 hash functions choice, which was 
> clarified elsewhere). One such example is the tf5 source that you used as 
> example.

That Russ used as an example; it doesn't come from me. I claim it is
not an interesting example.

> In that case, would you still want a outside-of-git hash, signed by 
> the human uploader?

That would be desirable for the archive (or third parties using data
from the archive) to be able to verify the contents are what the human
uploader signed as the archive does not see the Git tree.

> In the cases where the git repository is _not_ treesame to the source package 
> (patches-applied, but debian/patches not stored in git), uploads are already 
> possible via dgit push-source (and the human upload signature covers the 
> source package as it goes in the archive, not the git tree). In that other 
> case, would you still want a signed artifact of hashes, signed by the human 
> uploader?

Ideally yes, and this is where the "reasonable representation of the
source package contents" part gets relevant.

> And do we both understand that this means that some git repository 
> layouts would hence not be possible to be uploaded via tag2upload (because it 
> needs a much heavier git tag client, that builds the final source package, 
> hashes its contents, and creates the git tag)?

That is not obvious and probably not true. It is false for a trivial
reasonable representation of the source package contents, but not
necessarily for the set of all possible reasonable representations (and
there is no requirement to use a single fixed one either). (It
certainly isn't true if one leaves out the "reasonable" part.)

To understand whether a representation[1] exists that is also
reasonably easy to implement, one would need to look at it with an open
mind and, in particular, not with specific implementation details in
mind.

  [1]: Or small set of representations the upload mechanism/human
uploader can choose from.

Finding such a representation might be easier if one also changes some
possible outputs that tag2upload generates. For example, if one wanted
a reasonable representation to include patch names and contents in
d/patches, these would need to be known at the time the content hash
gets computed on the uploader's system for which limitations on
generated build artifacts might be useful. (But it could also be valid
to not include all details about d/patches in the reasonable
representation.)

It is probably hard to find such a representation if the source package
generati

Re: General Resolution to deploy tag2upload

[Ian Jackson]

Gunnar Wolf writes ("Re: General Resolution to deploy tag2upload"):
> I have two comments on your GR; I think you could accept them as
> amendments and keep the seconds it has already got:

Thanks for the contribution.  Speaking personaly, I'm certainly open
to wording tweaks

> I think you should rather write:
> 
>1. tag2upload, based in the form designed and implemented by Sean
>   Whitton and Ian Jackson, (...)
> 
> Many points have been raised during the current discussion, and you
> have accepted some observations as valid and worth including in the
> implementation. But not only that: If something good (design
> improvement) or bad (vulnerability nobody thought of before) comes
> along in the future, this GR could be seen as tying the hands of the
> maintainers to a specific implementation. By changing to "based in" or
> "derived from" a specific design idea that you and Ian implemented,
> then later Jonathan and Russ studied and commented upon, and then the
> bunch of us gave some thought and discussion to, it allows for it to
> be modified in the future.

I can see why you're suggesting this.  Getting the wording right, for
this point, is not entirely straightforward.

To my mind the problem with your wording is that the opoosing
ftpmasters contend that the change they want us to make is but a minor
technicality.  If GR with your wording passed, they might reasonably
argue that the modification they are demanding is consistent with the
GR text.  We'd be back where we started.

> Not only that: It also allows you and ftp-masters to come to specific
> compromises _leading to an implemented service_.

I am sure that there will be improvements, that we and ftpmaster can
both agree on.  I don't expect that to be a problem.

The question comes if ftpmaster *demand* changes of us, that we don't
like.  These might include the change they are currently requesting;
conceivably, it might also include other changes, which they might
realise later that they want us to make.

My hope is that the GR clearly empower us to deploy the current
design, subject to any changes which are *agreed*.

> > 2. Under Constitution §4.1(3), we overrule the ftpmaster delegate's
> >decision: the Debian Archive should be configured to accept and trust
> >uploads from the tag2upload service.
> > 
> > 3. Future changes to tag2upload should follow normal Debian processes.
> 
> Of course, that paragraph could be included by your #3. But #3 is so
> vague that it leads to ambiguity. There is too much ground covered by
> it.

Yes, that is what we intended paragraph 3 to cover.  I agree it's
quite vague.

Making some random texts up, how about

  3. This resolution does not prevent mutually agreed changes to
 tag2upload's design or implementation, or future changes made
 according to normal Debian processes.

I don't like this because it's longer.

> > 4. Nothing in this resolution should be taken as requiring maintainers
> >to use any particular git or salsa workflows.
> 
> This is a good clarification, but IMO it should be seen as a side
> note, and not part of the GR itself.

I have had multiple conversations with people who fear the imposition
of changes to their workflows.  Given the history of some other recent
changes within Debian, and indeed some of the rhetoric that one
sometimes encounters, that is a very reasonable fear.

So I think this sentence is essential.

Ian.

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Re: [RFC] General Resolution to deploy tag2upload

[Russ Allbery]

Russ Allbery  writes:

> As soon as I start using tag2upload, I am no longer running dgit locally
> and that patch generation will be represented in the Git tree that I
> sign to trigger tag2upload.

That patch generation will *not* be represented in the Git tree that I
sign to trigger tag2upload.  Among at least one other typo in that
message.  Good lord, I clearly still need more caffeine.

-- 
Russ Allbery (r...@debian.org)  

Re: [RFC] General Resolution to deploy tag2upload

[Russ Allbery]

Didier 'OdyX' Raboud  writes:
> Le vendredi, 28 juin 2024, 08.32:43 h CEST Ansgar 🙀 a écrit :
>> I'll expand on the here slightly for your benefit:
>> 
>> $ git clone https://salsa.debian.org/rra/tf5.git
>> [...]
>> $ apt-get source tf5
>> [...]
>> $ rm -rf tf5/.git tf5-5.0beta8/.pc
>> $ diff -Nur tf5 tf5-5.0beta8; echo $?
>> 0
>> 
>> If one is really bored:
>> $ (cd tf5; sha256sum $(find . -type f | sort) | sha256sum -)
>> 8d7820471fb44382a0c752319906064a1276ff18873fb4730dec1319aaf7b459  -
>> $ (cd tf5-5.0beta8; sha256sum $(find . -type f | sort) | sha256sum -)
>> 8d7820471fb44382a0c752319906064a1276ff18873fb4730dec1319aaf7b459  -
>> 
>> I will leave it as an exercise to you to compare the output and to
>> reason about results of different ways to compare both trees.

> It looks to me that you have taken (by choice, or by chance) an example
> that too conveniently fits what you want to demonstrate: in which the
> git repository and the .dsc are treesame. They are often the case, but
> not always, as documented in the various git workflows' documentation
> provided by dgit.  The salsa repo can be patches-applied and not have
> the debian/patches files, they'd be created at dgit push-source time.

The initial confusion about this is my fault.  I was the one who offered
tf5 as an example; Ansgar didn't choose it.

tf5 uses a git-debrebase workflow, which means that debian/patches has to
be synthesized from Git commits.  What I missed, much to my embarrassment,
is that since tag2upload doesn't exist, I am using dgit locally, and the
default dgit behavior with git-debrebase workflows is to generate all of
debian/patches locally and commit them to Git before doing the package
upload.  So it *looks* from the current Git state like I am maintaining a
debian/patches directory in Git that would be available to tag2upload.

This isn't actually the behavior that I wanted and I had not realized that
it was doing this.  It's was a configuration error on my part.  I've now
fixed that configuration and pushed a new version so that people can see
what I had intended the behavior to be.  Note the difference between the
debian/5.0beta8-13 tag (what I would tag with tag2upload) and the
archive/debian/5.0beta8-13 tag (what the tree looks like after the
processing that tag2upload would perform).

However, more relevant to this discussion, tf5 even with the previous
configuration only looks like it's a trivial case because I can't use
tag2upload and therefore I am still using a thick client (dgit) locally to
upload.  That means that dgit can manipulate my local Git tree before
doing the upload to do a bunch of Debian-specific work.  As soon as I
start using tag2upload, I am no longer running dgit locally and that patch
generation will be represented in the Git tree that I sign to trigger
tag2upload.

Ian explained all of this in an excellent and comprehensive message that
he sent in reply to mine, correcting my misunderstandings about how this
workflow worked.  (Hopefully I have it right now!)

-- 
Russ Allbery (r...@debian.org)  

Re: General Resolution to deploy tag2upload

[Gunnar Wolf]

Hello Sean,

I have two comments on your GR; I think you could accept them as
amendments and keep the seconds it has already got:

Sean Whitton dijo [Thu, Jun 27, 2024 at 03:15:42PM +0800]:
> =
> BEGIN FORMAL RESOLUTION TEXT
> 
> tag2upload allows DDs and DMs to upload simply by using the
> git-debpush(1) script to push a signed git tag.
> 
> 1. tag2upload, in the form designed and implemented by Sean Whitton and
>Ian Jackson, and design reviewed by Jonathan McDowell and Russ
>Allbery, should be deployed to official Debian infrastructure.

I think you should rather write:

   1. tag2upload, based in the form designed and implemented by Sean
  Whitton and Ian Jackson, (...)

Many points have been raised during the current discussion, and you
have accepted some observations as valid and worth including in the
implementation. But not only that: If something good (design
improvement) or bad (vulnerability nobody thought of before) comes
along in the future, this GR could be seen as tying the hands of the
maintainers to a specific implementation. By changing to "based in" or
"derived from" a specific design idea that you and Ian implemented,
then later Jonathan and Russ studied and commented upon, and then the
bunch of us gave some thought and discussion to, it allows for it to
be modified in the future.

Not only that: It also allows you and ftp-masters to come to specific
compromises _leading to an implemented service_.

> 2. Under Constitution §4.1(3), we overrule the ftpmaster delegate's
>decision: the Debian Archive should be configured to accept and trust
>uploads from the tag2upload service.
> 
> 3. Future changes to tag2upload should follow normal Debian processes.

Of course, that paragraph could be included by your #3. But #3 is so
vague that it leads to ambiguity. There is too much ground covered by
it.

> 4. Nothing in this resolution should be taken as requiring maintainers
>to use any particular git or salsa workflows.

This is a good clarification, but IMO it should be seen as a side
note, and not part of the GR itself.


signature.asc
Description: PGP signature

Re: General Resolution to deploy tag2upload

[Gerardo Ballabio]

I believe that an option of "raise this issue with the Technical
Committee and let them decide" should be on the ballot. While the TC
normally hasn't the power to overrule delegates, it is my
understanding that a GR can give them that power (think of it as "we
overrule the FTP team's decision with whatever the TC will decide").

I am not a DD, so I can't propose that option myself, so if any DD
agrees with me, please do it. If you wish me to help with the wording
of the option, feel free to ask.

Gerardo

Re: [RFC] General Resolution to deploy tag2upload

[Didier 'OdyX' Raboud]

(dropping all CC, focusing on the content)

Le vendredi, 28 juin 2024, 08.32:43 h CEST Ansgar 🙀 a écrit :
> I'll expand on the here slightly for your benefit:
> 
> $ git clone https://salsa.debian.org/rra/tf5.git
> [...]
> $ apt-get source tf5
> [...]
> $ rm -rf tf5/.git tf5-5.0beta8/.pc
> $ diff -Nur tf5 tf5-5.0beta8; echo $?
> 0
> 
> If one is really bored:
> $ (cd tf5; sha256sum $(find . -type f | sort) | sha256sum -)
> 8d7820471fb44382a0c752319906064a1276ff18873fb4730dec1319aaf7b459  -
> $ (cd tf5-5.0beta8; sha256sum $(find . -type f | sort) | sha256sum -)
> 8d7820471fb44382a0c752319906064a1276ff18873fb4730dec1319aaf7b459  -
> 
> I will leave it as an exercise to you to compare the output and to
> reason about results of different ways to compare both trees.

It looks to me that you have taken (by choice, or by chance) an example that 
too conveniently fits what you want to demonstrate: in which the git 
repository and the .dsc are treesame. They are often the case, but not always, 
as documented in the various git workflows' documentation provided by dgit. 
The salsa repo can be patches-applied and not have the debian/patches files, 
they'd be created at dgit push-source time.

If I understand your position correctly (please correct me if needed): you 
(with a ftpmaster hat) would like all uploads to come with a signed artefact 
of hashes corresponding to the set of files as represented by the current 
Debian source package format, as accepted by the archive today. And you would 
like this artefact's signature be a signature by the human uploader. Did I get 
this right?

If I understand dgit and tag2upload correctly, in the cases where the git 
repository is treesame to the source package (patches-applied, with debian/
patches file stored in git, as pointed by a tag), this artefact has the exact 
same cryptographic value as the git tag, pointing to the git tree, pointing to 
the git objects (modulo the SHA-1 vs SHA-256 hash functions choice, which was 
clarified elsewhere). One such example is the tf5 source that you used as 
example. In that case, would you still want a outside-of-git hash, signed by 
the human uploader?

In the cases where the git repository is _not_ treesame to the source package 
(patches-applied, but debian/patches not stored in git), uploads are already 
possible via dgit push-source (and the human upload signature covers the 
source package as it goes in the archive, not the git tree). In that other 
case, would you still want a signed artifact of hashes, signed by the human 
uploader? And do we both understand that this means that some git repository 
layouts would hence not be possible to be uploaded via tag2upload (because it 
needs a much heavier git tag client, that builds the final source package, 
hashes its contents, and creates the git tag)?

(I have refrained from formulating the questions as explicit to the ftpmaster 
delegates "as a team", I merely want to understand Ansgar's position first ).

-- 
OdyX

Re: [RFC] General Resolution to deploy tag2upload

[Matthias Urlichs]

Clarification:

of the d/$D and a/d/$V tags


"debian/$V" and "archive/debian/$V" tags of course. Not $D. Sigh.

--
-- mit freundlichen Grüßen
--
-- Matthias Urlichs



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: [RFC] General Resolution to deploy tag2upload

[Matthias Urlichs]

On 28.06.24 08:32, Ansgar 🙀 wrote:

I will leave it as an exercise to you to compare the output and to
reason about results of different ways to compare both trees.


The problem is that this particular package is the result of one of 
several possible workflows, where previous work has already resulted in 
a dgit run that assembled various entries in d/patches for us; see e.g. 
commit 556ef01 which is simply dgit's handling of 3a8b795.


A git-centered tag2upload-based workflow *would not do that*, leaving 
the job of generating commits like 556ef01 to the t2u server and thus 
preventing you from doing this comparison.


This fact has been stated multiple times since the RFC has been posted. 
I can't believe you have not read / understood any of these emails, or 
that you forgot the discussion about the significance of identity (or 
not) of the d/$D and a/d/$V tags on browse.dgit.d.o.


(tf5 is an example of identity, which should not surprise anybody at 
this point; reminder: less than half of the 50 most-recent sources on 
browse.dgit.d.o look like this.)



IMHO this message amply demonstrates that any further delay to a GR, as 
premature as the call for it might have seemed to some (including me) 
yesterday morning, is not warranted.


--
-- mit freundlichen Grüßen
--
-- Matthias Urlichs



OpenPGP_signature.asc
Description: OpenPGP digital signature