Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Andrej Shadura]

Hi,

On Fri, 12 Feb 2021, at 08:59, Raphael Hertzog wrote:
> On Fri, 12 Feb 2021, Guillem Jover wrote:
> > > If we assume that the archive is meant to store immutable content
> > > under a given filename (and to me that requirement seems to be a good
> > > idea), then we should question ourselves whether we really want to store
> > > those signatures in a filename that's associated to the upstream version.
> > > They should either be tied to the Debian revision (so that they can change
> > > over time without any new upstream release) or be incorporated in the
> > > Debian tarball.

> > The upstream signatures are important to determine the provenance of
> > the source at the time of packaging, just like the signatures on .dsc,
> > both lose relevance once they hit an archive.

> I agree with this. Why do we want to upload them and store them forever
> then?

> > This seems mostly a tooling problem TBH.

> Yeah, it would go a long way if pristine-tar would store the associated
> signature and restore it as well. It's easy to forget to include it
> when the uploads are not done by the same person.

By the way, that’s what pristine-lfs always does.

-- 
Cheers,
  Andrej

Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Julien Cristau]

On Thu, Feb 11, 2021 at 09:59:42PM +0100, Raphaël Hertzog wrote:
> Those files are not really meant to be immutable:
> - signing keys can expire and be revoked, upstream might want to update
>   signatures of already released tarballs
> - the set of "upstream release managers" might evolve over time and the
>   official signature to use might change...
> 
As far as we're concerned they are immutable, they are the signature of
the tarball at the time that tarball was uploaded to debian.  There's no
reason for that to change without the tarball itself changing, at which
point both filenames change.

Cheers,
Julien

Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[gregor herrmann]

On Fri, 12 Feb 2021 15:41:09 +0100, Raphael Hertzog wrote:

> On Fri, 12 Feb 2021, Peter Pentchev wrote:
> > > Yeah, it would go a long way if pristine-tar would store the associated
> > > signature and restore it as well. It's easy to forget to include it
> > > when the uploads are not done by the same person.
> > 
> > It can, since version 1.41:
> > 
> > debcheckout confget
> > cd confget
> > git checkout pristine-tar
> > git checkout master
> > git checkout debian/master
> > pristine-tar checkout -s ../confget_2.3.4.orig.tar.xz.asc 
> > ../confget_2.3.4.orig.tar.xz
> 
> Well, then I assume that the git-buildpackage integration doesn't do
> this automatically. Honestly, you should not have to specify that you
> want to check out the associated signature at the same time or maybe with
> a generic option --include-associated-files that would not fail if
> there's no associated file.

From the changelog and the manpage of gbp-buildpackage, there's

   --git-upstream-signatures=[auto|on|off]
  Whether to export the upstream tarball with signatures.

which defaults to 'auto' … and after checking out pristine-tar it
does what it says on the tin:

gbp:info: Tarballs 'confget_2.3.4.orig.tar.xz' not found at '../tarballs/'
gbp:info: Creating /home/gregoa/tmp/build-area/confget_2.3.4.orig.tar.xz
[no message about the *.asc here]
…
[but it's there:]
dpkg-source: info: building confget using existing ./confget_2.3.4.orig.tar.xz
dpkg-source: info: building confget using existing 
./confget_2.3.4.orig.tar.xz.asc

and also in the output directory:

% ll ../build-area/*orig*
-rw-rw-r-- 1 gregoa gregoa 34724 Feb 12 18:36 
../build-area/confget_2.3.4.orig.tar.xz
-rw-rw-r-- 1 gregoa gregoa   833 Feb 12 18:36 
../build-area/confget_2.3.4.orig.tar.xz.asc


Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   NP: Neil Young: My My, Hey Hey


signature.asc
Description: Digital Signature

Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Raphael Hertzog]

On Fri, 12 Feb 2021, Peter Pentchev wrote:
> > Yeah, it would go a long way if pristine-tar would store the associated
> > signature and restore it as well. It's easy to forget to include it
> > when the uploads are not done by the same person.
> 
> It can, since version 1.41:
> 
> debcheckout confget
> cd confget
> git checkout pristine-tar
> git checkout master
> git checkout debian/master
> pristine-tar checkout -s ../confget_2.3.4.orig.tar.xz.asc 
> ../confget_2.3.4.orig.tar.xz

Well, then I assume that the git-buildpackage integration doesn't do
this automatically. Honestly, you should not have to specify that you
want to check out the associated signature at the same time or maybe with
a generic option --include-associated-files that would not fail if
there's no associated file.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS


signature.asc
Description: PGP signature

Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Mattia Rizzolo]

On Fri, 12 Feb 2021, 10:25 am Rene Engelhard,  wrote:

> Hi,
>
> Am 11.02.21 um 21:59 schrieb Raphaël Hertzog:
>
> > [1] For details it happened in dbus-glib:
> > https://snapshot.debian.org/package/dbus-glib/0.110-2/ -> it has .asc
> file
> > https://snapshot.debian.org/package/dbus-glib/0.110-3/ -> no .asc
> > https://snapshot.debian.org/package/dbus-glib/0.110-4/ -> no .asc
> > https://snapshot.debian.org/package/dbus-glib/0.110-5/ -> it has a
> > different .asc file
> >
> Why should anything else than -1 have a .asc file anyways in the upload?
>
> That's .orig.tar.xz (or whatever compression) and the accompanying
> .orig.tar.xz.asc.
>
> Since -2 etc don't upload the .orig again there's no need to upload the
> signature of the .orig again.


You are likely confusing the .dsc and the .changes.
The .dsc *always* refer to all the source files, even if not uploaded.
That clearly also includes the .asc.


>

Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Peter Pentchev]

On Fri, Feb 12, 2021 at 08:59:12AM +0100, Raphael Hertzog wrote:
> Control: block -1 by 876643
> 
> Hi,
> 
> thanks for your quick answer!
> 
> On Fri, 12 Feb 2021, Guillem Jover wrote:
> > > If we assume that the archive is meant to store immutable content
> > > under a given filename (and to me that requirement seems to be a good
> > > idea), then we should question ourselves whether we really want to store
> > > those signatures in a filename that's associated to the upstream version.
> > > They should either be tied to the Debian revision (so that they can change
> > > over time without any new upstream release) or be incorporated in the
> > > Debian tarball.
> > 
> > The upstream signatures are important to determine the provenance of
> > the source at the time of packaging, just like the signatures on .dsc,
> > both lose relevance once they hit an archive.
> 
> I agree with this. Why do we want to upload them and store them forever
> then?
> 
> > This seems mostly a tooling problem TBH.
> 
> Yeah, it would go a long way if pristine-tar would store the associated
> signature and restore it as well. It's easy to forget to include it
> when the uploads are not done by the same person.

It can, since version 1.41:

debcheckout confget
cd confget
git checkout pristine-tar
git checkout master
git checkout debian/master
pristine-tar checkout -s ../confget_2.3.4.orig.tar.xz.asc 
../confget_2.3.4.orig.tar.xz
gpg --verify ../confget_2.3.4.orig.tar.xz{.asc,}

G'luck,
Peter

-- 
Peter Pentchev  r...@ringlet.net r...@debian.org p...@storpool.com
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13


signature.asc
Description: PGP signature

Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Rene Engelhard]

Hi,

Am 11.02.21 um 21:59 schrieb Raphaël Hertzog:

> [1] For details it happened in dbus-glib:
> https://snapshot.debian.org/package/dbus-glib/0.110-2/ -> it has .asc file
> https://snapshot.debian.org/package/dbus-glib/0.110-3/ -> no .asc
> https://snapshot.debian.org/package/dbus-glib/0.110-4/ -> no .asc
> https://snapshot.debian.org/package/dbus-glib/0.110-5/ -> it has a
> different .asc file
>
Why should anything else than -1 have a .asc file anyways in the upload?

That's .orig.tar.xz (or whatever compression) and the accompanying
.orig.tar.xz.asc.

Since -2 etc don't upload the .orig again there's no need to upload the
signature of the .orig again.


(And then there ia slso the problem that subcomponent tarballs can't get
their .ascs when they need to be repa ckaged to fill dpkgs needs. So
stuff like LO only gets the .asc for the "core" tarball but not for
helpcontent2 and translations, which would have .asc files otherwise)

Regards,


Rene

Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Raphael Hertzog]

Control: block -1 by 876643

Hi,

thanks for your quick answer!

On Fri, 12 Feb 2021, Guillem Jover wrote:
> > If we assume that the archive is meant to store immutable content
> > under a given filename (and to me that requirement seems to be a good
> > idea), then we should question ourselves whether we really want to store
> > those signatures in a filename that's associated to the upstream version.
> > They should either be tied to the Debian revision (so that they can change
> > over time without any new upstream release) or be incorporated in the
> > Debian tarball.
> 
> The upstream signatures are important to determine the provenance of
> the source at the time of packaging, just like the signatures on .dsc,
> both lose relevance once they hit an archive.

I agree with this. Why do we want to upload them and store them forever
then?

> This seems mostly a tooling problem TBH.

Yeah, it would go a long way if pristine-tar would store the associated
signature and restore it as well. It's easy to forget to include it
when the uploads are not done by the same person.

Because I already saw the warning saying that I lack the signature file
(based on the idea that if we have the uptsream key we want to upload the
signature, but I don't buy this, I believe it's a help for the maintainer
to verify the tarball it downloads during uscan but I don't see the point
to upload it for eternity) and I saw it as a nuisance more than a help...
I would usually not check if there really was a signature file before or
not.

On Fri, 12 Feb 2021, Guillem Jover wrote:
> > This usually appears in the way of people uploading a package with the same
> > name and version of something that was removed long long ago and since then
> > archived and forgotten by dak.
> 
> Ah, sorry, right, that dak forgetfulness problem which seems
> contagious. :)
>
> Ok, so then these seems like two bugs in dak. dak sees the .asc, then
> they disappear and it forgets them, and then files with different
> content can then be uploaded.

FTR this is this bug that has been ignored for years and that also
affected Kali more than once:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876643

Kali would be happy to sponsor anyone who can tackle this bug in dak...

> While ideally dak would never forget, the problem here is that dak
> allows uploads that drop the .asc files, no?

Possibly... but then it means that you need to treat signature file
differently from any other extra file that you want to attach to the .dsc.
Because for extra .orig tarballs, we want the current behaviour where you
can add and drop them freely between Debian revisions.

So I'm not sure it's worth the extra logic.

Cheers,
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog 
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄   Debian Long Term Support: https://deb.li/LTS

Processed: Re: Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Debian Bug Tracking System]

Processing control commands:

> block -1 by 876643
Bug #982562 [general] general: Storing upstream signatures next to upstream 
tarballs is problematic
982562 was not blocked by any bugs.
982562 was not blocking any bugs.
Added blocking bug(s) of 982562: 876643

-- 
982562: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982562
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Guillem Jover]

On Fri, 2021-02-12 at 01:05:21 +0100, Mattia Rizzolo wrote:
> On Fri, 12 Feb 2021, 12:52 am Guillem Jover,  wrote:
> > Then there's the problem with changing contents for already seen
> > files, which seems like a dak bug. It does not allow to change a
> > tarball once it has been seen, so I don't see why it should allow a
> > changed .asc either?
> 
> That's not true.
> 
> Call it a dak bug or a feature, depending on where you stand.  Dak forgets
> everything concerning a file as soon as it's not present in any suite it
> manages.
> This usually appears in the way of people uploading a package with the same
> name and version of something that was removed long long ago and since then
> archived and forgotten by dak.

Ah, sorry, right, that dak forgetfulness problem which seems
contagious. :)

Ok, so then these seems like two bugs in dak. dak sees the .asc, then
they disappear and it forgets them, and then files with different
content can then be uploaded. While ideally dak would never forget,
the problem here is that dak allows uploads that drop the .asc files,
no?

Thanks,
Guillem

Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Mattia Rizzolo]

On Fri, 12 Feb 2021, 12:52 am Guillem Jover,  wrote:

> Then there's the problem with changing contents for already seen
> files, which seems like a dak bug. It does not allow to change a
> tarball once it has been seen, so I don't see why it should allow a
> changed .asc either?
>

That's not true.

Call it a dak bug or a feature, depending on where you stand.  Dak forgets
everything concerning a file as soon as it's not present in any suite it
manages.
This usually appears in the way of people uploading a package with the same
name and version of something that was removed long long ago and since then
archived and forgotten by dak.


It's totally possible to overwrite a tarball with the same filename too
that way, you just need to wait the appropriate amount of time and upload
things in a way that you replace the upstream tarball.
(Honestly I haven't tried this myself, but I have a package where if you'd
like I can actually go and try to prove my point).


Back to the original bug report: I personally believe that the signatures
there are fine, and I don't believe in the "upstream the re-sign an already
released tarball" story.  But I consider the current forgetfulness of dak
as a bug.

Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Guillem Jover]

Hi!

On Thu, 2021-02-11 at 21:59:42 +0100, Raphaël Hertzog wrote:
> Package: general
> Severity: normal
> User: de...@kali.org
> Usertags: origin-kali
> X-Debbugs-Cc: hert...@debian.org, debian-d...@lists.debian.org
> Control: affects -1 ftp.debian.org dpkg-dev

> After having been bitten (in Kali) by failures to import Debian packages
> because a PGP signature file has been modified [1], this lead me to think
> about this problem space and I concluded that the way we are storing
> such signatures is not appropriate.
> 
> Those files are not really meant to be immutable:
> - signing keys can expire and be revoked, upstream might want to update
>   signatures of already released tarballs

These files have similar properties and problems as our own .dsc, so
I don't see a huge difference here.

> - the set of "upstream release managers" might evolve over time and the
>   official signature to use might change...

If this changes I'd expect in most cases the signing-key.asc to get
out of date too, anyway.

> If we assume that the archive is meant to store immutable content
> under a given filename (and to me that requirement seems to be a good
> idea), then we should question ourselves whether we really want to store
> those signatures in a filename that's associated to the upstream version.
> They should either be tied to the Debian revision (so that they can change
> over time without any new upstream release) or be incorporated in the
> Debian tarball.

The upstream signatures are important to determine the provenance of
the source at the time of packaging, just like the signatures on .dsc,
both lose relevance once they hit an archive.

> After all the key to verify those signatures is already stored in the
> Debian tarball (when you use the uscan feature to verify those
> signatures), so why not store the signature there as well?

This looks messy, taking into account multi orig support for example.

> I originally filed this in https://bugs.debian.org/949962 against
> ftp.debian.org but the bug got closed because it's not really the
> responsibility of ftpmasters to change this. So I'm starting a wider
> discussion to gather feedback of all interested parties (at least
> Guillem as dpkg maintainer). I won't drive this much further but
> I wanted to have it properly recorded and considered.

This seems mostly a tooling problem TBH.

There's the accidental omissions part, which I also got bitten by at
the beginning, because the error mode was silent. This has since then
been incrementally improved, and now we have lintian warnings and
dpkg-source also warns (when upstream sig is missing but there's
upstream signing keys, and on the reverse too). I attempted to make
the former an error, but stuff was breaking so that had to be reverted
(see #963821). Ideally that would eventually be turned back into an
error, with an option to make it a warning.

Then there's the problem with changing contents for already seen
files, which seems like a dak bug. It does not allow to change a
tarball once it has been seen, so I don't see why it should allow a
changed .asc either?

Thanks,
Guillem

Bug#982562: general: Storing upstream signatures next to upstream tarballs is problematic

[Raphaël Hertzog]

Package: general
Severity: normal
User: de...@kali.org
Usertags: origin-kali
X-Debbugs-Cc: hert...@debian.org, debian-d...@lists.debian.org
Control: affects -1 ftp.debian.org dpkg-dev

Hi people,

After having been bitten (in Kali) by failures to import Debian packages
because a PGP signature file has been modified [1], this lead me to think
about this problem space and I concluded that the way we are storing
such signatures is not appropriate.

Those files are not really meant to be immutable:
- signing keys can expire and be revoked, upstream might want to update
  signatures of already released tarballs
- the set of "upstream release managers" might evolve over time and the
  official signature to use might change...

If we assume that the archive is meant to store immutable content
under a given filename (and to me that requirement seems to be a good
idea), then we should question ourselves whether we really want to store
those signatures in a filename that's associated to the upstream version.
They should either be tied to the Debian revision (so that they can change
over time without any new upstream release) or be incorporated in the
Debian tarball.

After all the key to verify those signatures is already stored in the
Debian tarball (when you use the uscan feature to verify those
signatures), so why not store the signature there as well?

I originally filed this in https://bugs.debian.org/949962 against
ftp.debian.org but the bug got closed because it's not really the
responsibility of ftpmasters to change this. So I'm starting a wider
discussion to gather feedback of all interested parties (at least
Guillem as dpkg maintainer). I won't drive this much further but
I wanted to have it properly recorded and considered.

Cheers,

[1] For details it happened in dbus-glib:
https://snapshot.debian.org/package/dbus-glib/0.110-2/ -> it has .asc file
https://snapshot.debian.org/package/dbus-glib/0.110-3/ -> no .asc
https://snapshot.debian.org/package/dbus-glib/0.110-4/ -> no .asc
https://snapshot.debian.org/package/dbus-glib/0.110-5/ -> it has a
different .asc file