On 17263 March 1977, Russ Allbery wrote:
So far, from this thread, it looks like the decision from 2019 may
still
stand, but I think there are still places to explore.
The requests from there are still valid, yes.
There ought to be one point that is doing this step, not many, yes.
Includes that it is the delegated work and task description of
FTPMaster
to do this, though that can be addressed by either us ending up
running
it, or adjusting delegations. Not sure the latter ends up with happy
people, but is one existing way.
Elsewhere in this thread, Jessica Clarke made the excellent suggestion
that perhaps the authentication check concern could be resolved by dak
providing an API for performing the authentication and authorization
check. I am embarrassed that I didn't think of that; thank you very
much
to Jessica for that suggestion.
That gives me some hope that this point has a relatively neat
solution, so
I'm going to focus on exactly what dak needs the uploader signature to
cover in order to accept the package.
Yes, Ansgar also replied and doesn't appear to hat it, and it sounds
good to me too, so multiple in favor already.
While I still answer to some of the rest of your mail, this api thing
sounds like a good way forward to me.
I mean, Ansgar already hashed out some things that t2u should send - all
bits that it should have/know anyways, and exact details don't need to
be found right in this thread anyways - and then dak does whatever dak
does and replies to t2u with a OK or NOK. Which can then either ditch or
continue with the upload.
It will mean that the property of "everything to trace uploaded files
back to the uploader is on the mirror" is going away, but that appears
not a muched loved one anyways. Also, maybe, depending on what t2u
sends, we can store that for others to look at.
Funny, this API would also enable one, if they are inclined to, adjust
queued (or maybe better yet, rewrite) and put early-acceptance checks in
that thing.
Also, currently we have the nicety that we store all signatures
directly
besides the source package, available for everyone to go and check.
Linking back to the actual Uploader, not to a random service key. You
can take that, run a gpgv on it and via the checksums of the files
then
see that, sure, this is the code that the maintainer took and
uploaded.
You do *not* need to trust any other random key on that. Not that of
tag2upload. *AND* not that of FTPMaster.
The dgit-repos server similarly archives the signed Git tag with the
Git
tree over which it is a signature, ensuring that this is independent
of
Salsa where the tag could potentially be deleted by someone. This is
not
in the archive, of course, but I don't see any technical reason why
some
version of that data couldn't also be uploaded to the archive if one
wanted to use the archive as a highly distributed backup of the
dgit-repos
server. There is, however, the long-standing concern about any
variation
on the 3.0 (git) source package format that the Git tree the
maintainer
signed may contain non-free code somewhere in its history.
So here too, I'm not sure that this is inherently a blocker, although
in
the past the FTP team has been reluctant to include in the archive the
data that is required to preseve a complete record of what is signed
by a
Git tag. (One obvious potential solution is to only put a shallow
clone
in the archive, so you can verify the signature but some of the
content-addressable store references are unresolved.)
The thing with a complete git tree is just that it is not exactly
supported to get undistributable things out of that trees history,
without rewriting that history. And rewriting only our view of history
does
feel kind of silly.
Now, making a source format that's based on a subset of the git history
(shallow clone, git archive, whatever) and not the full git history
would avoid this.
Unsure those are the right words. We want to have the uploader create
a
signature over the content they want to have appear in the archive.
In a
way, that this signature can be taken and placed beside the source,
and
then independently verified. *Currently* this is done using .dsc
files.
Okay, so again I think it's easier to talk about specifics, so let me
make
this concrete by using myself as the use case.
I use the git-debrebase workflow for maintaining most of my Debian
packages. What this means, for those who aren't familiar with it, is
that
my workflow looks like this (this is idealized; I'm still migrating my
packages fully to this workflow so the specifics currently vary
somewhat):
[... Description of a git workflow ...]
Now, I would like to use tag2upload rather than using dgit locally to
make
the upload. I want to move my testing into Salsa CI so that my
overall
workflow more closely matches the way that I do all of my development
in
my day job. Salsa CI is great about not getting lazy and skipping
test
steps just because I am in a hurry to get a package uploaded, and I
can
capture every test that was useful and not have to remember to re-run
it.
(This is the part that I haven't done yet; I know I want to do it and
have
not yet found the time.)
What signed artifact do I need to provide so that the FTP team will be
comfortable accepting my tag2upload-built source package?
See below.
Note, importantly, that the source package contains things that are
not in
the files present in the working tree of a local Git checkout of my
source
package. The patch descriptions and committer information and related
metadata are where they are supposed to be in Git: in the metadata for
the
corresponding Git commit, not in a file in my working tree. The
transformation that puts that data into a 3.0 (quilt) source package
is
not rocket science, but it's not trivial either.
The signed artifact that I'm naturally providing is a signature across
the
entire Git tree, which includes all of the history and thus all of the
data that goes into the source package. So everything that goes into
the
source package *is signed*, by me, when I trigger a tag2upload upload.
The problem comes when dak wants to verify the correspondence between
that
data structure and the source package. It certainly can verify that
my
Git tag is valid and it can verify that the tag specifies the correct
source package, version, and so forth. But if it wants to verify that
the
construction of the debian/patches/* directory is correct, I think it
would have to perform the same transformation on my Git history that
dgit
and tag2upload perform.
I do like that workflow. The one magic part seems to be the thing that
creates debian/patches files out of your git tree. As soon as that step
is done, I think everything should be there that makes up the thing that
gets uploaded?
But I bet there are other ways to work that make it less trivial magic
than what you seem to have.
To answer your question from above: At the point the magic is done,
anything that lists the files and checksums of (ignore metadata like
file timestamps) that will appear when one unpacks the source package.
I basically assume that the uploader *does* need to have their source
locally, no matter what. (Their git cloned).
Yes, I agree. I don't think there's any way to avoid this: the source
has
to be in the same place that the key is in, or close to in the case of
secure key storage, in order for the uploader to sign it and know what
they are signing.
I also do assume that the uploader will build things, to see if the
stuff they are going to "push to the archive" (and our users)
actually
does what they intended it to do - and to test it.
This is the assumption that I think is no longer valid given Salsa CI.
It
used to be that this was the only way to test a package; now we can do
equally well and often better by letting Salsa CI do the hard work.
I'm not a believer of that, but I accept that people do so.
--
bye, Joerg
