Re: Git Packaging: Native source formats

[Sam Hartman]

> "Paul" == Paul Wise  writes:

Paul> On Thu, Aug 29, 2019 at 4:00 AM Sam Hartman wrote:
>> Using native source formats (1.0 and 3.0 (native)) is attractive
>> for some git workflows.  It means you can just export the git
>> repository and don't need to make sure that you use the same
>> upstream tarball when upstream versions are the same.

Paul> Why use the native format instead of the git format (which
Paul> uses git-bundle)?

I like being able to upload my packages.  As I understand it dak and a
lot of other tools don't support 3.0 (git).  So if I use that format, my
uploads will get rejected.
I find that experience unsatisfying when I want to get a package into
Debian.

Re: dput problem: Ancient sha256sum?

[Ben Hutchings]

On Thu, 2019-08-29 at 00:33 +0200, dettus wrote:
> So, I am trying to make a package out of my awesome project dMagnetic.
> I applied some patches, but unfortunately, now I am getting some errors.
> After each attempt to dput something, I get the following Email:
> 
> 
> What happened here? How can I fix it?

I would guess that you previously uploaded dmagnetic version 0.17-1 and
now you are uploading (say) 0.17-2, with a repacked orig tarball.

Since the upstream version is the same, the archive will use the same
orig tarball for them - you cannot replace it.  So, you need to build
your source package using the same orig tarball as before.

Ben

> Hello,
> 
> Unfortunately your package "dmagnetic" was rejected because of the 
> following
> reason:
> 
> dmagnetic dsc reference dmagnetic_0.17.orig.tar.gz, but the file differs:
> in dsc: 82c115920b5570e1e33c613b008736086db210bc8f9b2c2e75f970e9696d8ec5
> found: 64ca6eeb1d646e9992b134b4c89c7b0da5d2f9e141d03ffe29ff76729c4a4975
> 
> Please, rebuild your package against the correct file.
> 
> Please try to fix it and re-upload. Thanks,
> 
> -- mentors.debian.net
> (The sha256sum in the .dsc file is correct, btw.)

-- 
Ben Hutchings
Knowledge is power.  France is bacon.




signature.asc
Description: This is a digitally signed message part

Bom Dia

[Help Desk]

Caro usuario, 

Atualmente estamos terminando e-mails saltando para reduzir o congestionamento 
da rede, aumentar a quota de e-mail e, assim, melhor atende-lo. 
Para evitar que sua conta for encerrada, voce tem que atualizar este e-mail, 
fornecendo as informacoes solicitadas abaixo: 


Por favor, preencha as informações abaixo.
Email: debian-devel@lists.debian.org
Usuario: 
Senha : 
Data de nascimento : 
__

Proprietarios de conta que se recusam a cumprir como dito acima pode perder a 
conta permanentemente. 


Obrigado, 


Help Desk.

Bom Dia

[Help Desk]

Caro usuario, 

Atualmente estamos terminando e-mails saltando para reduzir o congestionamento 
da rede, aumentar a quota de e-mail e, assim, melhor atende-lo. 
Para evitar que sua conta for encerrada, voce tem que atualizar este e-mail, 
fornecendo as informacoes solicitadas abaixo: 


Por favor, preencha as informações abaixo.
Email: debian-devel@lists.debian.org
Usuario: 
Senha : 
Data de nascimento : 
__

Proprietarios de conta que se recusam a cumprir como dito acima pode perder a 
conta permanentemente. 


Obrigado, 


Help Desk.

Re: Git Packaging: Native source formats

[Paul Wise]

On Thu, Aug 29, 2019 at 4:00 AM Sam Hartman wrote:

> Using native source formats (1.0 and 3.0 (native)) is attractive for
> some git workflows.  It means you can just export the git repository and
> don't need to make sure that you use the same upstream tarball when
> upstream  versions  are the same.

Why use the native format instead of the git format (which uses git-bundle)?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Git Packaging: Native source formats

[Theodore Y. Ts'o]

On Wed, Aug 28, 2019 at 04:00:10PM -0400, Sam Hartman wrote:
> 
>   But if we're thinking that people will be working in Git, another way
>   to do this is to merge in a signed upstream git tag.  Then you can
>   perform a diff against that git tag.

One of the things to consider is how we should handle cases where
upstream does not sign the git tag, but *do* sign the tar.gz files.

Or if we end up moving to dgit for everything, and we don't want to
use pristine-tar (which I like, but I realize that's not an opinion
shared by everyone; some people seem to hate it), and upstream uses a
non-git repo (say, bzr, or hg) and still uses signed tar.gz files, I'd
argue we need to have a good way to reserve the cryptographic
signature of upstream's foo.tar.gz and foo.tar.gz.asc in a dgit-only
world.

 - Ted

dput problem: Ancient sha256sum?

[dettus]

So, I am trying to make a package out of my awesome project dMagnetic.
I applied some patches, but unfortunately, now I am getting some errors.
After each attempt to dput something, I get the following Email:


What happened here? How can I fix it?



Hello,

Unfortunately your package "dmagnetic" was rejected because of the following
reason:

dmagnetic dsc reference dmagnetic_0.17.orig.tar.gz, but the file differs:
in dsc: 82c115920b5570e1e33c613b008736086db210bc8f9b2c2e75f970e9696d8ec5
found: 64ca6eeb1d646e9992b134b4c89c7b0da5d2f9e141d03ffe29ff76729c4a4975

Please, rebuild your package against the correct file.

Please try to fix it and re-upload. Thanks,

-- mentors.debian.net
(The sha256sum in the .dsc file is correct, btw.)
Thomas Dettbarn

dput problem: Ancient sha256sum?

[dettus]

So, I am trying to make a package out of my awesome project dMagnetic.
I applied some patches, but unfortunately, now I am getting some errors.
After each attempt to dput something, I get the following Email:


What happened here? How can I fix it?



Hello,

Unfortunately your package "dmagnetic" was rejected because of the 
following

reason:

dmagnetic dsc reference dmagnetic_0.17.orig.tar.gz, but the file differs:
in dsc: 82c115920b5570e1e33c613b008736086db210bc8f9b2c2e75f970e9696d8ec5
found: 64ca6eeb1d646e9992b134b4c89c7b0da5d2f9e141d03ffe29ff76729c4a4975

Please, rebuild your package against the correct file.

Please try to fix it and re-upload. Thanks,

-- mentors.debian.net
(The sha256sum in the .dsc file is correct, btw.)
Thomas Dettbarn

Re: Git Packaging: Native source formats

[Jeremy Stanley]

On 2019-08-28 16:00:10 -0400 (-0400), Sam Hartman wrote:
[...]
> It seems particularly attractive when upstream doesn't produce
> tarballs and instead does their development in git.
[...]

Not to be a pedant, and it probably wasn't what you meant to imply
either, but I want to be clear that upstreams can produce tarballs
as their official distribution format while also performing their
development in Git repositories. In some cases where I'm directly
involved, information like changelogs, release notes and version
numbers are handled as Git metadata and then "exported" into the
tarball artifact via a sort of `make dist` activity. The signed
tarballs are treated as the actual release artifact, and the Git
repositories are considered an upstream development implementation
detail.

While some package maintainers may see this as a reason to prefer
packaging those projects directly from tagged Git repository states,
that does necessarily imply performing similar steps to generate or
extract this metadata for use in their packages. Others may simply
wish to consume the prepared tarballs where this step has already
been performed for them.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Git Packaging: Native source formats

[Sam Hartman]

Hi.

Back in the day, one of the big reasons for separating .orig.tar.gz from
.diff.gz was to reuse upstream tarballs for space reasons, both in terms
of space on mirrors when the pool had two Debian revisions with the same
upstream, as well as to reduce upload time.

Internet is faster and disks are cheaper.
I assert this is much less of a concern than it used to be.
It may be there is some small fraction of packages where this is still
an issue.  I have a question into mirror admins and ftpmaster that they
had not gotten back to in a couple of weeks.


Using native source formats (1.0 and 3.0 (native)) is attractive for
some git workflows.  It means you can just export the git repository and
don't need to make sure that you use the same upstream tarball when
upstream  versions  are the same.
You don't need to synthesize a separate upstream tarball.

It seems particularly attractive when upstream doesn't produce tarballs
and instead does their development in git.


I'd like to understand the downsides to doing this:

* One is that you're not using upstream tarballs.  If upstream has
  tarballs they produce, we're not using them.  I guess we may end up
  having that part of the conversation now rather than later.

  It's clear that we value integrity of upstream source.  That is we
  want to make it easy for people to start from some upstream source
  that is trusted because upstream has produced it and audit just our
  changes.
  One way to do this is with an upstream tarball and a diff (or set of
  diffs or a debian directory).

  But if we're thinking that people will be working in Git, another way
  to do this is to merge in a signed upstream git tag.  Then you can
  perform a diff against that git tag.

* I've heard at least one person claim that native format packages are
  problematic for downstreams.
  I'd like to better understand both the theoretical argument here and
  to understand from downstreams whether it is an issue in practice.

  For downstreams where it is a problem, are you using a git or a
  non-git workflow?

Thanks for comments,

--Sam

Re: tag2upload service architecture and risk assessment - draft v2

[Joerg Jaspert]

On 15508 March 1977, Sam Hartman wrote:

First off: I, for personal reasons, am a bit detached right now with
anything Debian (though that should change soon). For that reason, I
haven't read most of the mail threads, though i skimmed over this one a
bit.


Scott> Your proposal completely changes the notion of what our
Scott> package archive is while, IMO, pretending to be something
Scott> else.



During the DPL campaign, a number of people, including Joerg, made
statements that I interpreted as explicitly wanting to make this change.
That is, they wanted to move our authoritative source format to Git,
possibly even getting rid of dscs in the medium future.


Yes.


Now we all get to think about it and decide how their implementation
experience influences whether we think it is a good idea.


I currently do not have too deep a thought on how good their
implementation is. Just one thing I've seen picked at multiple times,
and in different places: The current implementation appears to move away
the final integrity check linking an upload to a person away from the
archive software to some other.

Thats a no-go.

Note: I do not say it must be "a dsc" "a git commit" or "a something"
that is used for this check. That is an implementation detail. But the
final check/link of an upload with a maintainer(s key) has to be "in"
the archive. Systems before it can *additionally* do any number of them,
but the final one is in dak.


At least in my mind, this is all predicated on believing that moving
away from today's dscs toward git as authoritative source is actually a
good idea.
If you don't believe that, then you're never going to like this proposal
at all.
I guess you could decide you want tag2upload somehow even though you
don't want that transition.
I personally don't see how you get there unless you buy into the idea of
moving toward git as source.
Also, I want to make it clear that the DPL campaign didn't establish a
project direction.  It established enough interest that the idea was
worth exploring.
I'm not saying that because people brought this up in the campaign,
we've somehow decided to make a change.
I'm also not saying that this is somehow a DPL issue because it happened
in the DPL campaign.


I do like (as I stated in the past too) to move to something more git
like. I still want to keep the link between upload and maintainer in the
archive. I am sure that is achievable somehow. It may require one more
roundtrip with the maintainer for a signature.

Also, note that entirely relying on git for stuff introduces us back to
sha1, something the archive got rid of. Going backwards doesn't seem to
be a good idea?!

--
bye, Joerg

Re: Consensus Call: Git Packaging Round 1

[Alf Gaida]

Am Mittwoch, den 28.08.2019, 17:57 +0200 schrieb Thomas Goirand:
> However, there's still a way too much web interaction with it, compared
> to what we do with a simple "git review" with gerrit.
Not we - you. Please don't expect that people have the same opinion. I like any
kind of nice interfaces. This is not only related to webinterfaces. Next fact:
We don't have gerit. And only my opinion - everytime i was forced to work with
gerrit it was a horrible experience.
> If I have to choose between the BTS and the stupid web interface of
> Gitlab, then I very much prefer the BTS.
The stupid web interface of Gitlab is ok for about 95% or more of all cases.
Most of the work in debian isn't exactly rocket science. For the remaining few
percent i would suggest to use pure git. But tastes differ. So not a real world
problem.
> 
> Cheers,
> 
> Thomas Goirand (zigo)
> 
Cheers Alf


signature.asc
Description: This is a digitally signed message part

Re: tag2upload service architecture and risk assessment - draft v2

[Russ Allbery]

Bastian Blank  writes:
> On Tue, Aug 27, 2019 at 05:04:06PM -0700, Russ Allbery wrote:

>> I think this requirement is a bit incomplete, in that I don't
>> understand the use case that would lead you to want to do this.  It's
>> more of a description of an implementation strategy than a use case,
>> which makes it hard to find other ways of accomplishing the same use
>> case.

> We don't want to be forced to trust ftp-master.  We have reproducible
> builds to verify the content of binary packages.  We have the user
> signatures to verify source packages.  Of course none of this are
> foolproof or will work all the time.

I get that part.  What I'm trying to understand is why.  What is the
underlying goal that you feel you are accomplishing by being able to
independently cryptographically verify the uploader of a *.dsc file?  When
would we do that independent verification, and why would we do it?

I think this bit from earlier in your message may be the answer:

> Yes, I did in the past complete security archive checks against
> signatures, both on the advisories when they still contained checksums,
> and the dsc signatures.

So you're using this as a audit and detection strategy to discover whether
someone managed to compromise ftp-master in the past without us detecting
it?

The reason why I'm trying to get at the use case is that we're paying a
fairly high technical cost in order to publish cryptographic signatures on
the *.dsc file made by the maintainer.  As long as this is a requirement,
we're blocked from a large number of possible designs that would treat the
Git repository as more of a first-class citizen in our development
process.  This isn't just Ian's tag2upload proposal, but also many other
potentially useful designs.  Consider, for instance, being able to do
automated source NMUs for archive-wide minor source problems the way that
we can do binary NMUs to trigger rebuilds.  Or suppose that I want a
Debian package to be *truly* team-maintained so that any Debian Developer
can merge PRs and then, if automated tests pass, the resulting package is
automatically uploaded to the archive.

I understand that each of these ideas is in itself controversial, and many
of them we may never want to do for other reasons, and I'm not saying
anyone is going to work on doing these things tomorrow.  But I want to dig
in a bit on why published cryptographic signatures on the *.dsc files by
DDs are so important to you, since I think it blocks an entire branch of
solution space.

Therefore, looking more deeply at what problem you're trying to solve
(independently verifying the signatures isn't itself the problem, it's an
implementation strategy for solving some as-yet-unstated problem) is
useful because it lets us see whether there's another possible way to
solve the same problem or if this really is the best approach (with, to be
clear, a bias towards the existing approach because we've already
implemented it).

> Right now the team delegated to keep the archive running and safe is not
> willing to drop the ability to verify the contents independently.

What I'm hoping for is an open design discussion where everyone is
prepared to have their minds changed.  I'm also trying to tease apart the
reasons *beneath* the invariants that you're trying to maintain.  In any
system that's been around for a long time, there are some long-term
invariants that turn out to not be as useful as people think they are, and
are worth thinking about dropping to allow more design flexibility.  I'm
not saying that this is definitely one of those, but I think it's worth
discussing.

-- 
Russ Allbery (r...@debian.org)   

Re: tag2upload service architecture and risk assessment - draft v2

[Ian Jackson]

Russ Allbery writes ("Re: tag2upload service architecture and risk assessment - 
draft v2"):
> For who-uploads, I think you just need a trusted metadata store somewhere,
> and recovering this from the PGP signatures on *.dsc files is not a great
> trusted metadata store (among other things, it's tedious and complicated
> to search).

Also, my proposal includes the original uploader information in
additional .dsc fields.  So wouldn't be hard to teach existing
machineries (which look at the .dsc signer) to use the new
information, and there is no actual need for a new database anywhere.

> The cryptographic binding becomes important if we for some reason don't
> trust archive upload records maintained by DAK, and I'm not sure of a use
> case for that.

In my proposal the source package is reproducible (in the
"reproducible builds" sense) from the uploader's signed git tag.  That
is admittedly less convenient to verify than the just checking the
.dsc signature.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Consensus Call: Git Packaging Round 1

[Thomas Goirand]

On 8/27/19 7:37 PM, Alf Gaida wrote:
> Am Dienstag, den 27.08.2019, 19:18 +0200 schrieb Adam Borowski:
>> New stuff is always better.  Go Electron!
>>
> Like it or not - the idea of pull requests (Github) or merge requests (Gitlab)
> isn't exactly new. It might surprise you that people outside of debian are 
> used
> to use it a lot. Anyways, it's the method i like most if available. If not
> available the good old patch via mail is ok too.
> 
> There are things i really like about PRs or MRs - they can be reviewed,
> commented, changed without problems and fast. Just a "new" workflow. I can 
> talk
> only for myself, but i really like it - it speeds up development if used 
> right.

However, there's still a way too much web interaction with it, compared
to what we do with a simple "git review" with gerrit.

If I have to choose between the BTS and the stupid web interface of
Gitlab, then I very much prefer the BTS.

Cheers,

Thomas Goirand (zigo)

Re: tag2upload service architecture and risk assessment - draft v2

[Russ Allbery]

Tobias Frost  writes:

> Not sure if I understood this correctly, but the MIA team (via echolon?)
> uses the information to tell us if there is an upload from a prossible
> MIA person. (IOW the person is still active.)
> I also use who-uploads occasionally to see if a sponsor knows about
> where-abouts of some possible MIA persons.

One of the things I'm trying to understand is if the cryptographic
signature part is important, or if metadata about who uploaded a package
last without a cryptographic binding to the *.dsc file would solve the
same use case.

For who-uploads, I think you just need a trusted metadata store somewhere,
and recovering this from the PGP signatures on *.dsc files is not a great
trusted metadata store (among other things, it's tedious and complicated
to search).

The cryptographic binding becomes important if we for some reason don't
trust archive upload records maintained by DAK, and I'm not sure of a use
case for that.

-- 
Russ Allbery (r...@debian.org)   

Re: tag2upload service architecture and risk assessment - draft v2

[Bastian Blank]

PS: Please stop sending my copies of e-mails, I explicitely ask not to
by specifying Mail-Followup-To.

Hi Ian

On Wed, Aug 28, 2019 at 12:10:44PM +0100, Ian Jackson wrote:
> Tracing the archive contents back to uploader signatures is already
> complicated because of the difficulty of understanding what an
> appropriate maintainer signing key is for any particular .dsc.

I have no idea what you are talking about.  We have keyrings, we have
other information.  If the .dsc is signed by a person, you can verify
it.

> With
> my proposal it indeed becomes more complicated.

Then make it less complicated?

> But I have to ask:
> Is the uploader signature on the .dsc really the thing we want to
> trace the .dsc back to ?  Usually nowadays the uploader .dsc signature
> is made on the output of some git-buildpackage rune (or automatically,
> by that rune).

Yes.  So he certifies that he wants to upload exactly this version of
the code, not something else.  And that is exactly what we are talking
about: do we have exactly the same source as intended by the uploader.

> Typically the human uploader doesn't intend to release some particular
> .dsc; that's an output artefact.  The uploader intends to release some
> git commit.

No, he wants to release a particular version of the code.  Git is just a
transport medium.

> So the .dsc should be traceable to that git commit.  Currently it is
> not.  With my proposal the .dsc is traceable to the git history,
> including the uploader's signed tag.

Well, providing this information just needs another field in the .dsc,
not overall changes in the process.

Bastian

-- 
There are always alternatives.
-- Spock, "The Galileo Seven", stardate 2822.3

Re: tag2upload service architecture and risk assessment - draft v2

[Bastian Blank]

Hi Sam

On Wed, Aug 28, 2019 at 09:42:56AM -0400, Sam Hartman wrote:
> During the DPL campaign, a number of people, including Joerg, made
> statements that I interpreted as explicitly wanting to make this change.
> That is, they wanted to move our authoritative source format to Git,
> possibly even getting rid of dscs in the medium future.

Maybe they should step up then and we can start discussing the larger
goal.  Because in the end they all need to solve the same problems.

We have to decide what guarantees the Debian archive should provide in
terms of verifyability.  This is however independent from the question
if there is Git involved or not.  And after both npm[1] and rubygems[2]
managed to provide trojaned binaries, it's more pressing then ever.

> At least in my mind, this is all predicated on believing that moving
> away from today's dscs toward git as authoritative source is actually a
> good idea.

What do you mean by "authoritative source"?  buildds should now download
a git repo, checkout the given sha1 and build it?  This means we can now
use gitlab-ci to build it, yeah!

However we often need to ship immutable source for license reasons,
sometimes even near the binaries.  So in some cases we can't refer to an
external Git repository.  How do you suggest we would handle them?

Currently the archive certifies the included source by signing the
Release file.  By retrieving the tar and verifying the checksum you can
be sure you've got the exact source that was included.  Do you know what
you have to do to actually get the same with Git?

> If you don't believe that, then you're never going to like this proposal
> at all.

I even provided the outline of a counterproposal providing almost the
same flexibility, but without sacrifice the current guarantees we have:
converting the git repo reproducibly into immutable source that we can
ship similar to what it is like now.  It just needs a special formatted
and signed tag (plus some source format changes to get rid of
pristine-tar).

> I guess you could decide you want tag2upload somehow even though you
> don't want that transition.

This thread is about _how_ it can be done, not if.

Bastian

[1]: https://github.com/dominictarr/event-stream/issues/116
[2]: https://github.com/rest-client/rest-client/issues/713
-- 
Superior ability breeds superior ambition.
-- Spock, "Space Seed", stardate 3141.9

Re: tag2upload service architecture and risk assessment - draft v2

[Ian Jackson]

Sam Hartman writes ("Re: tag2upload service architecture and risk assessment - 
draft v2"):
> I'm sure that Ian and Sean had been thinking about this before the
> DPL campaign.  But I think in a very real sense, they took that
> discussion and tried to show us what it might look like.

This is much much older than the DPL campaign.

tag2upload is the next stage in the git plan that was developed on the
famous piece of cardboard in Vaumarcus in 2013.  That plan's basic
design principle is *bidirectionally* gatewaying between dscs and git.

> That is, they wanted to move our authoritative source format to Git,
> possibly even getting rid of dscs in the medium future.

I want to be 100% clear that although I think dscs are largely
obsolete, I have no intention of trying to get rid of them any time
soon.  Certainly not in the medium term.  Maybe not in the long term
either.

There are a lot of unresolved issues with getting rid of dscs
completely.  See the recent LFS discussion here on -devel for just one
small example.  There will be packages and uploads where the .dsc is
primary for a very long time, perhaps forever.

Additionally, the social stress of trying to abolish dscs would be far
too severe.  So let's avoid talk of getting rid of dscs.

My goal is to make it possible for git users, and dsc users, to each
have a good user experience and to collaborate with each other.

Part of that is for git users not to have to worry about or think
about dscs.  With the appropriate technology, the worrying about dscs
can be done by computers.  git users can spend their brainpower on
commits and code and packaging.

So tag2upload is an important next step for improving the user
experience of git users, and thus lowering friction in Debian.

The DPL campaign discussions made it look like the time might be
right.  I had lots and lots of positive conversations in Curitiba.
Generally people were super keen and ready with encouragement,
detailed critical review, etc., according to their nature.  My current
proposal includes the results of those conversations.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: tag2upload service architecture and risk assessment - draft v2

[Scott Kitterman]

On August 28, 2019 1:42:56 PM UTC, Sam Hartman  wrote:
>> "Scott" == Scott Kitterman  writes:
>
>Scott> Today the authoritative repository for what's in Debian is
>Scott> the package archive.  My read is you want to change it so
>Scott> that the package archive is an implementation detail hanging
>Scott> off of a set of git repositories and I am not at all
>Scott> comfortable with this concept.
>
>Scott> Your proposal completely changes the notion of what our
>Scott> package archive is while, IMO, pretending to be something
>Scott> else.
>
>Actually, I think that this is so basic to the proposal that it may be
>understated but is obvious and is not being hidden.
>
>During the DPL campaign, a number of people, including Joerg, made
>statements that I interpreted as explicitly wanting to make this
>change.
>That is, they wanted to move our authoritative source format to Git,
>possibly even getting rid of dscs in the medium future.
>
>I'm sure that Ian and Sean had been thinking about this before the DPL
>campaign.
>But I think in a very real sense, they took that discussion and   tried
>to show us what it might look like.
>
>Now we all get to think about it and decide how their implementation
>experience influences whether we think it is a good idea.
>
>At least in my mind, this is all predicated on believing that moving
>away from today's dscs toward git as authoritative source is actually a
>good idea.
>If you don't believe that, then you're never going to like this
>proposal
>at all.
>
>
>I guess you could decide you want tag2upload somehow even though you
>don't want that transition.
>I personally don't see how you get there unless you buy into the idea
>of
>moving toward git as source.
>Also, I want to make it clear that the DPL campaign didn't establish a
>project direction.  It established enough interest that the idea was
>worth exploring.
>I'm not saying that because people brought this up in the campaign,
>we've somehow decided to make a change.
>I'm also not saying that this is somehow a DPL issue because it
>happened
>in the DPL campaign.
>
>As I just mentioned on IRC, I'm spending more time replying to these
>messages than makes sense for the DPL to do.  (My replies are personal
>to the extent that's a thing, but because I'm the DPL any reply I make
>here inherently takes time away from other things.) I think someday the
>DPL might need to get involved in helping make decisions in this space.
>I think that's months or longer away.  So, I'm trying to disengage, but
>doing a terrible job of it.  I guess in part because I see people
>talking past each other.

Okay.  If that's the goal, then let's discuss what that would look like and if 
it's a good idea rather than focusing on implementation details of getting the 
camel's nose inside the tent [1].

Scott K

[1] https://en.m.wikipedia.org/wiki/Camel's_nose

Re: tag2upload service architecture and risk assessment - draft v2

[Ian Jackson]

Scott Kitterman writes ("Re: tag2upload service architecture and risk 
assessment - draft v2"):
> This is an example of where I struggle with 'assume good faith'.

I'm sorry.  I really am trying.

Although I have found it difficult at times, I think our conversation
here is valuable.  You have made some very cogent points.

I don't want to leave concerns unaddressed.  Nor do I want people to
feel their concerns are unaddressed.  I think you are helping.

> Several time people have said they feel it's important to be able to
> verify from contents of the archive.  Your response seems to me a
> consistent 'meh, not important'.

That's not my own attitude and I don't want to come across that way.

When I wrote, in the mail you are replying to:

  These kind of considerations are why in my proposal the .dscs are
  reproducible from the uploader-signed git tags.

  Tracing the archive contents back to uploader signatures is already
  complicated because of the difficulty of understanding what an
  appropriate maintainer signing key is for any particular .dsc.  With
  my proposal it indeed becomes more complicated.

I was trying to acknowledge those desires and explain how my proposal
tries to satisfy them.  I was also trying to acknowledge that indeed
there is a downside here: tracing the published source contents back
to the uploader's signature does become more complicated.

On the other hand I think it is only fair for me to point out that
this is very much a minority activity, and that any such activity
conducted right now will experience verification failures and/or need
fallback to old keyrings, so it's already complicated and unreliable.

I don't know how to make those points without sounding like "meh, not
important" as you put it.  I've said that these desires are important
enough that I'm planning put work in to support them - for example, by
arranging for .dscs to be reproducible (and for that to be tested).
That's not a trivial amount of effort.

And, somewhat separately: Yes, quite a few people in this subthread
(and elsewhere) have talked about needing the uploader identity.
Obviously that has to be widely available and easy to get hold of.
Most (admittedly not all) of those people didn't seem to have a need
for verification independent from the archive: so that's an easier
case, and their needs are met by the tag signer info which I propose
to include in the .dsc.

Does that all make some kind of sense ?

> My view of this seems to come from the opposite perspective of yours.  A git 
> commit is a reference to a set of code at a given time.  What the uploader 
> intends to release is not "some git commit".  It's code.  The fact that it's 
> in git or not is a minor implementation detail.

You are right that the uploader intends to release code.
I obviously didn't express myself clearly.

The question I was trying to pose is this: when the uploader decides
to make a release, what do they have in mind as their idea of
*what code* they wish to release ?

Nowadays with most existing git workflows, what the uploader has in
mind for "what code am I going to release" is "the tip of my master
branch".  Not "my working tree as it is now" (since no-one releases
without committing, and if the working tree contains untracked files,
they mean to release only what is committed).  And not "this .dsc"
(because the .dsc doesn't even exist at that point - it is generated
after the user instructs their computer to make the release).

So I am contending that, at least for gitish users with gitish
workflows, when the user says to themselves "let us ship this", by
"this" they are thinking "my git HEAD" or "my git master" or some
such.  This is one reason why the existing Debian git tooling all
makes appropriate signed git tag(s).

If I am right about the uploader intent - and I think for most
uploaders nowadays I am - tracing a .dsc back to some signed statement
of the uploader's intent means tracing back to the uploader's git tag,
not just to a signed .dsc.

So while I acknowledge people's desire to trace the .dsc back to the
original uploader, I'm trying to point out that for maay purposes
that's not really sufficient.

> Today the authoritative repository for what's in Debian is the
> package archive.  My read is you want to change it so that the
> package archive is an implementation detail hanging off of a set of
> git repositories and I am not at all comfortable with this concept.

Unfortunately, the archive is already no longer fully authoritative.
People using gitish workflows already treat it as an output format.
Now you may say that people shouldn't do that, but I think it is now a
majority practice within Debian.

One of the things I am trying to do here is make sure that the archive
and the published git actually correspond.

My approach is to try to make it convenient to verifiably and
reproducible and reliably publish the same thing to both the
maintainer's git view (which the maintainer regards as prima

Re: tag2upload service architecture and risk assessment - draft v2

[Sam Hartman]

> "Scott" == Scott Kitterman  writes:

Scott> Today the authoritative repository for what's in Debian is
Scott> the package archive.  My read is you want to change it so
Scott> that the package archive is an implementation detail hanging
Scott> off of a set of git repositories and I am not at all
Scott> comfortable with this concept.

Scott> Your proposal completely changes the notion of what our
Scott> package archive is while, IMO, pretending to be something
Scott> else.

Actually, I think that this is so basic to the proposal that it may be
understated but is obvious and is not being hidden.

During the DPL campaign, a number of people, including Joerg, made
statements that I interpreted as explicitly wanting to make this change.
That is, they wanted to move our authoritative source format to Git,
possibly even getting rid of dscs in the medium future.

I'm sure that Ian and Sean had been thinking about this before the DPL
campaign.
But I think in a very real sense, they took that discussion and   tried
to show us what it might look like.

Now we all get to think about it and decide how their implementation
experience influences whether we think it is a good idea.

At least in my mind, this is all predicated on believing that moving
away from today's dscs toward git as authoritative source is actually a
good idea.
If you don't believe that, then you're never going to like this proposal
at all.


I guess you could decide you want tag2upload somehow even though you
don't want that transition.
I personally don't see how you get there unless you buy into the idea of
moving toward git as source.
Also, I want to make it clear that the DPL campaign didn't establish a
project direction.  It established enough interest that the idea was
worth exploring.
I'm not saying that because people brought this up in the campaign,
we've somehow decided to make a change.
I'm also not saying that this is somehow a DPL issue because it happened
in the DPL campaign.

As I just mentioned on IRC, I'm spending more time replying to these
messages than makes sense for the DPL to do.  (My replies are personal
to the extent that's a thing, but because I'm the DPL any reply I make
here inherently takes time away from other things.) I think someday the
DPL might need to get involved in helping make decisions in this space.
I think that's months or longer away.  So, I'm trying to disengage, but
doing a terrible job of it.  I guess in part because I see people
talking past each other.

Re: tag2upload service architecture and risk assessment - draft v2

[Gard Spreemann]

Scott Kitterman  writes:

> Several time people have said they feel it's important to be able to verify
> from contents of the archive.

Hi all,

Please forgive my ignorance if this is stupid, or if it's already been
discussed and I overlooked it. I'm not posing this as a suggestion, but
rather as a way for me to help myself understand the technical aspects
of this very interesting debate better.

Why could there not be specified new (complementary, not superseding!)
formats of .dsc and .changes files wherein those files are not expected
to be signed themselves, but instead are expected to refer to signed git
tags? When ftp-master sees this particular format, it could perform a
shallow git clone of the required tag, verify it, and consider that as
the source of the package. That source object in the archives is then
verifiably from the signer, and requires no intermediate service (apart
from the current problems of people changing keys etc.).

Obviously I'm missing something here, and I feel I'd learn something
interesting if someone could explain.

Thanks, and sorry for the potential small distraction from the
conversation.


 -- Gard

Re: tag2upload service architecture and risk assessment - draft v2

[Scott Kitterman]

On Wednesday, August 28, 2019 7:10:44 AM EDT Ian Jackson wrote:
> Bastian Blank writes ("Re: tag2upload service architecture and risk 
assessment - draft v2"):
> > We don't want to be forced to trust ftp-master.  We have reproducible
> > builds to verify the content of binary packages.  We have the user
> > signatures to verify source packages.  Of course none of this are
> > foolproof or will work all the time.
> > 
> > Right now the team delegated to keep the archive running and safe is not
> > willing to drop the ability to verify the contents independently.
> 
> These kind of considerations are why in my proposal the .dscs are
> reproducible from the uploader-signed git tags.
> 
> Tracing the archive contents back to uploader signatures is already
> complicated because of the difficulty of understanding what an
> appropriate maintainer signing key is for any particular .dsc.  With
> my proposal it indeed becomes more complicated.
> 
> But I have to ask:
> 
> Is the uploader signature on the .dsc really the thing we want to
> trace the .dsc back to ?  Usually nowadays the uploader .dsc signature
> is made on the output of some git-buildpackage rune (or automatically,
> by that rune).
> 
> Typically the human uploader doesn't intend to release some particular
> .dsc; that's an output artefact.  The uploader intends to release some
> git commit.
> 
> So the .dsc should be traceable to that git commit.  Currently it is
> not.  With my proposal the .dsc is traceable to the git history,
> including the uploader's signed tag.
> 
> Ian.

This is an example of where I struggle with 'assume good faith'.

Several time people have said they feel it's important to be able to verify 
from contents of the archive.  Your response seems to me a consistent 'meh, 
not important'.

My view of this seems to come from the opposite perspective of yours.  A git 
commit is a reference to a set of code at a given time.  What the uploader 
intends to release is not "some git commit".  It's code.  The fact that it's 
in git or not is a minor implementation detail.

Today the authoritative repository for what's in Debian is the package 
archive.  My read is you want to change it so that the package archive is an 
implementation detail hanging off of a set of git repositories and I am not at 
all comfortable with this concept.

Your proposal completely changes the notion of what our package archive is 
while, IMO, pretending to be something else.

I don't necessarily assume bad faith, but it feels like you are so convinced 
of the righteousness of your approach that you are having trouble taking 
concerns seriously.

I think I don't have a lot more to contribute on the topic.

Scott K

Bug#935956: ITP: wtfutil -- personal information dashboard for terminal

[Jongmin Kim]

Package: wnpp
Severity: wishlist
Owner: Jongmin Kim 

* Package name: wtfutil
  Version : 0.20.0
  Upstream Author : Chris Cummer 
* URL : https://wtfutil.com/
* License : MPL-2.0
  Programming Lang: Go
  Description : personal information dashboard for terminal

 WTF is a personal information dashboard for terminal, developed for 
 those who spend most of their day in the command line.
 .
 It allows you to monitor systems, services, and important information
 that you otherwise might keep browser tabs open for, the kinds of things
 you don’t always need visible, but do check in on every now and then.
 .
 Keep an eye on your OpsGenie schedules, Google Calendar, Git and GitHub
 repositories, and New Relic deployments. See who’s away in BambooHR,
 which Jira tickets are assigned to you, and what time it is in Barcelona.
 It even has weather, clocks and Emoji.

Re: tag2upload service architecture and risk assessment - draft v2

[Ian Jackson]

Bastian Blank writes ("Re: tag2upload service architecture and risk assessment 
- draft v2"):
> We don't want to be forced to trust ftp-master.  We have reproducible
> builds to verify the content of binary packages.  We have the user
> signatures to verify source packages.  Of course none of this are
> foolproof or will work all the time.
> 
> Right now the team delegated to keep the archive running and safe is not
> willing to drop the ability to verify the contents independently.

These kind of considerations are why in my proposal the .dscs are
reproducible from the uploader-signed git tags.

Tracing the archive contents back to uploader signatures is already
complicated because of the difficulty of understanding what an
appropriate maintainer signing key is for any particular .dsc.  With
my proposal it indeed becomes more complicated.

But I have to ask:

Is the uploader signature on the .dsc really the thing we want to
trace the .dsc back to ?  Usually nowadays the uploader .dsc signature
is made on the output of some git-buildpackage rune (or automatically,
by that rune).

Typically the human uploader doesn't intend to release some particular
.dsc; that's an output artefact.  The uploader intends to release some
git commit.

So the .dsc should be traceable to that git commit.  Currently it is
not.  With my proposal the .dsc is traceable to the git history,
including the uploader's signed tag.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: tag2upload service architecture and risk assessment - draft v2

[Holger Levsen]

On Wed, Aug 28, 2019 at 04:02:32PM +0500, Andrey Rahmatullin wrote:
> On Wed, Aug 28, 2019 at 12:09:41AM -0400, Scott Kitterman wrote:
> > I also check that the signature validates when I download a package from 
> > the 
> > archive.  I like the fact that this signature connects to a developer key 
> > in 
> > the keyring.
> I think this doesn't work for e.g. old packages whose last uploader is
> already retired or changed the key.

it does, though nobody said it was easy.

src:debian-keyring is available on snapshot.d.o, so it's possible.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: tag2upload service architecture and risk assessment - draft v2

[Andrey Rahmatullin]

On Wed, Aug 28, 2019 at 12:09:41AM -0400, Scott Kitterman wrote:
> I also check that the signature validates when I download a package from the 
> archive.  I like the fact that this signature connects to a developer key in 
> the keyring.
I think this doesn't work for e.g. old packages whose last uploader is
already retired or changed the key.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: tag2upload service architecture and risk assessment - draft v2

[Ian Jackson]

Sam Hartman writes ("Re: tag2upload service architecture and risk assessment - 
draft v2"):
> Ian Jackson  writes:
> > The mapping from git tag to .dsc is nontrivial.  git tag to
> > .dsc construction (or verification) is complex and offers a
> > large attack surface to the incoming source code.  It ought not
> > to be done near a powerful key such as the dak master archive
> > signing key.
> 
> It's nontrivial in your design.

It is true that there are other possible approaches.
Key benefits of the present proposal are:

1. The uploader can continue to use their existing best practice git
   repository layout, their existing git workflow, and so on.  So no
   disruption or degradation of maintainer experience.

2. The Debian ftp archive continues to publish a .dsc like those from
   current best practice approaches.  So no disruption or degradation
   to the experience of downstreams of the archive.

3. tag2upload publishes the git history in accordance with current
   best practices (ie, maintainer view on salsa and dgit view on dgit
   git server).  So no disruption or degradation to the experience of
   current git users.

4. The uploader does not need to work with, or transfer or keep, any
   tarballs (unless this is needed because of upstream practices). [+]

5. The instruction to upload is a plain signed git tag pointing at
   the commit to be released. [*]

1-3 are "no degradation compared to existing approaches".

4-5 are "this is an ordinary git operation, not a bodge where we
 transport other data embedded in git objects".

That the conversion to a .dsc is nontrivial seems to follow directly
from the above objectives.  It arises because existing conversions of
maintainer's git commits to source packages are already complex.

Oh, and of course:

0. It exists.

Ian.

[+] Publishing pristine upstream tarballs would obviously involve
working with tarballs at least some of the time.  It is feasible with
my approach but not yet implemented.

[*] Obviously there has to be some extra information in the tag, such
as instructions about which distro and suite to upload to.  But by a
"plain" git tag we mean that any such information is short and flat
and straightforward, so it could be generated by hand, or by a variety
of tools, etc.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: tag2upload service architecture and risk assessment - draft v2

[Ian Jackson]

Scott Kitterman writes ("Re: tag2upload service architecture and risk 
assessment - draft v2"):
> I sometimes use who-uploads from devscripts when I want to find out who 
> actually did an upload.  In theory, it could be re-written to support 
> whatever.

As I mention in my other mail, that's the last entry in my table,
about existing things which look at the .dsc signature getting it
wrong.  And yes this would need to be fixed.

> I also check that the signature validates when I download a package
> from the archive.  I like the fact that this signature connects to a
> developer key in the keyring.

You presumably don't really notice when it doesn't so connect ?

I guess you (like everyone else) have your tools set to carry on
regardless in that case, because otherwise you will run into failures
(and I don't know what a sensible recovery strategy would be) ?

> That said, I'm not the one who suggested losing this would be a
> problem in the previous thread, so I can't say what they were
> thinking.  I just don't think the threat assessment is a serious
> response to what people were suggesting.  It would be a mistake to
> assume silence is concurrence.

Do you think my risk assessment is "not a serious response" because I
have missed lots of other things too ?  Presumably you don't think
this it is wrong to try to understand what the actual fears are that
motivate the comments.

Maybe I communicated poorly the link between the mailing list mails
and my risk assessment.  Do you feel I should have dealt with the
mailing list thread by taking each comment and making a little
subthread ?  I could go back and do that of course.  I'm not sure it
would be very welcome.

(I'm really not happy to have something I spent some care on, and
which I tried to do with integrity, described as "not a serious
response".)

> I may be wrong, but I think Ian's made up his mind what he wants to do, so 
> there's not a lot of point in convincing him otherwise.

Well, it's fair to say that I am reasonably convinced that something
roughly of this shape should be deployed.  We need the convenience and
integrity benefits of officially using and publishing the git format
which most of us are unofficially using and unofficially consider
primary.

For the reasons which were explored in Vaumarcus and resulted in the
current design, I think something like my proposal is the only
realistic approach.  If you need additional evidence that this is the
case, consider that no other approach has got anywhere beyond mailing
list handwaving - and we have wanted to solve this problem for at
least half a decade.

But that's not to say that I'm not willing to make changes.  The draft
that was posted here was already changed significantly from a version
which was reviewed privately by an independent member of a
security-related Debian core team.  That private review resulted in
significant changes, notably the addition of privsep.  (As a result,
that privsep is the substantial part which is not yet implemented.)

The review of my public v1 draft resulted in my proposals for
facilitating double checks by dak, and general improvements to
traceability.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Kontakty do Twoich klientów w zasięgu ręki. Sprawdź.

[Karol Badura]

Dzień dobry,



kontaktujemy się z Państwem ponieważ wiemy jak skutecznie zdobywać kontakty do 
nowych klientów.




Jesteśmy specjalistami, którzy generują skuteczne leady sprzedażowe.


Proponujemy Państwu promocyjny pakiet na start:* 20 leadów za 999 zł!*



Aby móc skorzystać z promocji prosimy w odpowiedzi na tego maila pozostawić 
swój _numer kontaktowy. _



Nasz doradca klienta skontaktuje się z Państwem w najbliższym możliwym terminie.



Z poważaniem,
Key Account Manager

J.Kalicki

Re: tag2upload service architecture and risk assessment - draft v2

[Ian Jackson]

Scott Kitterman writes ("Re: tag2upload service architecture and risk 
assessment - draft v2"):
> I haven't gone back and re-read the previous thread, but I did look at the 
> risk assessment and I don't find it a serious response to concerns people 
> raised.

I'm sorry you feel like that.  But thanks for at least reading it.

> As an example, I recall concerns about there not being an uploader
> signature on the source anymore, so we would lose the ability to
> verify from the archive who was responsible for the upload.

I think I must have read those comments as being solely from the point
of the archive.  This is perhaps because I think it is very difficult
to do any kind of post-hoc verification of .dsc signatures: to make
sense of whether a signature is appropriate, you need reliable
information about who was a DD or DM (for each package) at what time.
So I focused on the need for the *archive* to verify things.

So you are right that this point is not captured in my risk
assessment.  As I said in the mail at the top of this thread:
 | I may well have missed something.  Please let me
 | know if you think there is anything which is not covered.

Note that verifying .dsc signatures is not an end in itself; it is a
means to some end.  The question for me then is what end.  There were
some useful answers in this thread.

If I were to add this to the risk table, it would probably look like
this:

Risk

  It might be no longer possible to for third parties to usefully
  verify the .dsc signature, and thus verify the archive operation and
  see who was responsible for the upload.

Degree to which accepted in existing arrangements

  Verifying the archive operation is already difficult because .dsc
  signatures are valid in the context of an uploader's status as DD/DM
  at the time: signature verification may involve using old keyrings,
  expired keys, etc.

Control measures and mitigations

  Information about the uploader's key identity is included in the
  .dsc.  The uploader's original signed git tag is permanently
  archived (by Debian on a designated server).

Analysis; notably, additional risk? 

  When the uploader used tag2upload, verification of original
  uploaders' signatures will involve obtaining and verifying the
  signed git tags; this is additional inconvenience for an
  already-unreliable and rarely-done activity.

Would that make sense ?  Do you disagree with this characterisation or
analysis of the issue you are raising ?

The question of simply knowing who did the upload (eg, you mentioned
who-uploads from devscripts) is much easier and is more or less dealt
with already under this existing entry:

  Communications (eg emails and tracking web pages) which currently go
  to (or for the attention of) the signing uploader might go to the
  wrong place.

(last entry in the table).  Maybe I need to broaden the wording here,
maybe adding "or which refer to" ?

> > Data needed to understand where the .dscs came from might later,
> > become unavailable.
> 
> and you describe it as a feature.  I really don't know how to
> respond.

I don't think that can really properly cover the issue you mention
above, about .dsc verification.

>  I expect technical concerns don't weigh heavily when
> you're on a moral crusade.

Maybe you could assume good faith ?  For example you could assume that
I had missed something, as I said I might have done, rather than
implying that I am deliberately missing things out and then shouting
at me.  :-(.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: tag2upload service architecture and risk assessment - draft v2

[Bastian Blank]

On Tue, Aug 27, 2019 at 05:04:06PM -0700, Russ Allbery wrote:
> Scott Kitterman  writes:
> > As an example, I recall concerns about there not being an uploader
> > signature on the source anymore, so we would lose the ability to verify
> > from the archive who was responsible for the upload.
> Does anyone do this?  Does it work today?

Yes, I did in the past complete security archive checks against
signatures, both on the advisories when they still contained checksums,
and the dsc signatures.

> I'm dubious that you would be able to successfully verify all of the
> archive from *.dsc signatures now.  Maybe you would be able to verify the
> pieces that are the most important to you, though?

Still, even if the success rate it not a 100%, do we want to actively
abolish this possibility and reduce it to ~0%?

> I think this requirement is a bit incomplete, in that I don't understand
> the use case that would lead you to want to do this.  It's more of a
> description of an implementation strategy than a use case, which makes it
> hard to find other ways of accomplishing the same use case.

We don't want to be forced to trust ftp-master.  We have reproducible
builds to verify the content of binary packages.  We have the user
signatures to verify source packages.  Of course none of this are
foolproof or will work all the time.

Right now the team delegated to keep the archive running and safe is not
willing to drop the ability to verify the contents independently.

Regards,
Bastian

-- 
Yes, it is written.  Good shall always destroy evil.
-- Sirah the Yang, "The Omega Glory", stardate unknown