Bug#932943: Missing SHA512 and gpg signature
On 2019-08-05 11:41:34 +0200 (+0200), Bastian Blank wrote: > On Sun, Aug 04, 2019 at 10:05:32PM +0100, Chris Boot wrote: > > On 04/08/2019 17:29, Bastian Blank wrote: [...] > > > No, don't. Use base64 like everyone else. > > > > I strongly disagree with this. Practically everyone else uses > > hexadecimal for plain checksums. A GPG signature is its own > > thing but is (generally) plaintext (I'm assuming clearsign). > > This is what we (as in the project) use for the archive and for > > ISOs. > > Everything current switches to base64. It's shorter and easier to > see changes. Hex only survives where people tend to read it. You mentioned Kubernetes (which I haven't really used so have yet to notice), but who else's "current" software encodes checksums in base64 besides the Kubernetes ecosystem? I'm honestly curious as I still only ever see checksums in hexidecimal notation. The sha512sum(1) manpage makes no mention of having support for verifying base64-encoded checksums, for example. There's something to be said for sticking with traditional standards; newer is not always better. -- Jeremy Stanley
Bug#932943: Missing SHA512 and gpg signature
On Sun, Aug 04, 2019 at 10:05:32PM +0100, Chris Boot wrote: > On 04/08/2019 17:29, Bastian Blank wrote: > > Only one of them. And I would go directly to SHA3 for new stuff. > > Buster doesn't have any SHA3 tools in coreutils. While I don't have > anything against calculating such checksums in addition to the usual > suspects, I want to make sure people with current distros can easily > check them. Okay, let's stay with sha-512 for now. Turns out it is also faster. > >> 3. Add a new mapping within the "data" mapping called "checksums" with > >> keys for each algorithm, e.g. "data.checksums.sha256". > > > > The usual way would be something like this: > > > > | data: > > | verify: > > | - name: sha3_256 > > | content: ABC= > > | - name: gpg > > | content: AAA= > > That kind of structure works for me. That way we *can*[1] have checksums > for multiple image formats and multiple algorithms, e.g. the raw image, > qcow2, compressed tarball, or whatever. We have one upload manifest per image file. I'm not sure yet why we would want to have different checksums. > > No, don't. Use base64 like everyone else. > I strongly disagree with this. Practically everyone else uses > hexadecimal for plain checksums. A GPG signature is its own thing but is > (generally) plaintext (I'm assuming clearsign). This is what we (as in > the project) use for the archive and for ISOs. Everything current switches to base64. It's shorter and easier to see changes. Hex only survives where people tend to read it. Regards, Bastian -- You can't evaluate a man by logic alone. -- McCoy, "I, Mudd", stardate 4513.3
Bug#932943: Missing SHA512 and gpg signature
On 8/4/19 6:29 PM, Bastian Blank wrote: > On Sat, Aug 03, 2019 at 03:06:39PM +0100, Chris Boot wrote: >> - Which checksums should we include? Our Apt repos use MD5 and SHA-256, >> and our ISOs use MD5, SHA-1, SHA-256 and SHA-512. I'd be inclined to >> suggest SHA-256 and SHA-512 only, personally. > > Only one of them. And I would go directly to SHA3 for new stuff. Unfortunately, OpenStack Glance only shows SHA512, not SHA3, so even if we do both, we must keep SHA2 512. Cheers, Thomas Goirand (zigo)
Bug#932943: Missing SHA512 and gpg signature
On 04/08/2019 17:29, Bastian Blank wrote:
> On Sat, Aug 03, 2019 at 03:06:39PM +0100, Chris Boot wrote:
>> - Which checksums should we include? Our Apt repos use MD5 and SHA-256,
>> and our ISOs use MD5, SHA-1, SHA-256 and SHA-512. I'd be inclined to
>> suggest SHA-256 and SHA-512 only, personally.
>
> Only one of them. And I would go directly to SHA3 for new stuff.
Buster doesn't have any SHA3 tools in coreutils. While I don't have
anything against calculating such checksums in addition to the usual
suspects, I want to make sure people with current distros can easily
check them.
>> 1. Add labels of the form "checksum.cloud.debian.org/${ALGO}" under
>> metadata.labels, for example "checksum.cloud.debian.org/sha256".
>
> Labels are to search for stuff, not describe the content.
That seems like a good enough reason.
>> 3. Add a new mapping within the "data" mapping called "checksums" with
>> keys for each algorithm, e.g. "data.checksums.sha256".
>
> The usual way would be something like this:
>
> | data:
> | verify:
> | - name: sha3_256
> | content: ABC=
> | - name: gpg
> | content: AAA=
That kind of structure works for me. That way we *can*[1] have checksums
for multiple image formats and multiple algorithms, e.g. the raw image,
qcow2, compressed tarball, or whatever.
1. I carefully chose that word. I don't wish to imply that we should,
necessarily, I would just like to keep the options open.
>> In each case I expect the values to be hex strings, effectively the same
>> as the first column of the output from sha1sum, sha256sum, sha512sum,
>> etc... from coreutils.
>
> No, don't. Use base64 like everyone else.
I strongly disagree with this. Practically everyone else uses
hexadecimal for plain checksums. A GPG signature is its own thing but is
(generally) plaintext (I'm assuming clearsign). This is what we (as in
the project) use for the archive and for ISOs.
Kubernetes does usually Base64 encode data in Secret resources (though
stringData seems to be a well-kept, um, secret) and I don't really
understand where that came from. I'm not convinced that by itself is a
good example to follow.
Cheers,
Chris
--
Chris Boot
bo...@debian.org
signature.asc
Description: OpenPGP digital signature
Bug#932943: Missing SHA512 and gpg signature
On Sun, Aug 04, 2019 at 08:38:38PM +0200, Thomas Lange wrote: >> On Sun, 4 Aug 2019 18:29:30 +0200, Bastian Blank said: > >>> In each case I expect the values to be hex strings, effectively the same >>> as the first column of the output from sha1sum, sha256sum, sha512sum, >>> etc... from coreutils. > >> No, don't. Use base64 like everyone else. >Can you explain why we should use base64? On cdimages.d.o we also use >the plain output of the shaXXXsums commands. +1. It's what we have all over the place. Ditto for having sha256 and sha512. -- Steve McIntyre, Cambridge, UK.st...@einval.com Google-bait: https://www.debian.org/CD/free-linux-cd Debian does NOT ship free CDs. Please do NOT contact the mailing lists asking us to send them to you.
Bug#932943: Missing SHA512 and gpg signature
> On Sun, 4 Aug 2019 18:29:30 +0200, Bastian Blank said: >> In each case I expect the values to be hex strings, effectively the same >> as the first column of the output from sha1sum, sha256sum, sha512sum, >> etc... from coreutils. > No, don't. Use base64 like everyone else. Can you explain why we should use base64? On cdimages.d.o we also use the plain output of the shaXXXsums commands. -- regards Thomas
Bug#932943: Missing SHA512 and gpg signature
On Sat, Aug 03, 2019 at 09:23:58PM +0100, Chris Boot wrote: > Currently the build tools generate the raw image using FAI, then wrap it > into a tarball. The tarball is then xz compressed outside > debian-cloud-images in the GitLab CI job. > Is there any reason to keep it that way? Why don't we move the > compression into debian-cloud-images? The output of the build process is uncompressed. This is by design, as compression is inherent expensive. The stuff is only compressed with xz because the artifacts would be too large otherwise, as zip does not feature really useful compression. > Without this I don't see how debian-cloud-images can calculate a useful > set of checksums for the cloud images. We either checksum the raw image > itself or the tarball but neither is what ends up on patterson. The checksum of the stuff after build is not that useful. Only the converted and compressed output on the storage is relevant. Bastian -- Suffocating together ... would create heroic camaraderie. -- Khan Noonian Singh, "Space Seed", stardate 3142.8
Bug#932943: Missing SHA512 and gpg signature
On Sat, Aug 03, 2019 at 03:06:39PM +0100, Chris Boot wrote:
> - Which checksums should we include? Our Apt repos use MD5 and SHA-256,
> and our ISOs use MD5, SHA-1, SHA-256 and SHA-512. I'd be inclined to
> suggest SHA-256 and SHA-512 only, personally.
Only one of them. And I would go directly to SHA3 for new stuff.
> 1. Add labels of the form "checksum.cloud.debian.org/${ALGO}" under
> metadata.labels, for example "checksum.cloud.debian.org/sha256".
Labels are to search for stuff, not describe the content.
> 3. Add a new mapping within the "data" mapping called "checksums" with
> keys for each algorithm, e.g. "data.checksums.sha256".
The usual way would be something like this:
| data:
| verify:
| - name: sha3_256
| content: ABC=
| - name: gpg
| content: AAA=
> In each case I expect the values to be hex strings, effectively the same
> as the first column of the output from sha1sum, sha256sum, sha512sum,
> etc... from coreutils.
No, don't. Use base64 like everyone else.
Bastian
--
A father doesn't destroy his children.
-- Lt. Carolyn Palamas, "Who Mourns for Adonais?",
stardate 3468.1.
Bug#932943: Missing SHA512 and gpg signature
On 03/08/2019 11:06, Chris Boot wrote: > On 25/07/2019 00:00, Thomas Goirand wrote: >> On the secret files found here: >> http://cloud.debian.org/cdimage/cloud/ >> >> there's a json file attached. Unfortunately, there's no SHA512, and of >> course, no GPG signature of this file, so it's impossible check the validity >> of the images. Please at least add a SHA512, then we can see later how we >> can sign the json file. [snip] > Clearly it makes sense to include the checksum(s) in the build.json > file, but: [snip] Currently the build tools generate the raw image using FAI, then wrap it into a tarball. The tarball is then xz compressed outside debian-cloud-images in the GitLab CI job. Is there any reason to keep it that way? Why don't we move the compression into debian-cloud-images? Without this I don't see how debian-cloud-images can calculate a useful set of checksums for the cloud images. We either checksum the raw image itself or the tarball but neither is what ends up on patterson. We can generate checksums of the compressed tarball after the fact but it doesn't feel right injecting that into the manifest after the fact. It wouldn't be so bad just having detached checksums in that case but it feels like it could be genuinely useful to hold them in the manifest so that the upload step could validate them, for example. What are others' views on this? Cheers, Chris -- Chris Boot bo...@debian.org signature.asc Description: OpenPGP digital signature
Bug#932943: Missing SHA512 and gpg signature
On 25/07/2019 00:00, Thomas Goirand wrote:
> On the secret files found here:
> http://cloud.debian.org/cdimage/cloud/
>
> there's a json file attached. Unfortunately, there's no SHA512, and of
> course, no GPG signature of this file, so it's impossible check the validity
> of the images. Please at least add a SHA512, then we can see later how we
> can sign the json file.
What do we need in order to actually make this happen? I presume this
needs a code change in
https://salsa.debian.org/cloud-team/debian-cloud-images? If so, I'd like
to take that on.
Clearly it makes sense to include the checksum(s) in the build.json
file, but:
- Which checksums should we include? Our Apt repos use MD5 and SHA-256,
and our ISOs use MD5, SHA-1, SHA-256 and SHA-512. I'd be inclined to
suggest SHA-256 and SHA-512 only, personally.
- I know the manifests are inspired by Kubernetes, but the checksums
don't feel like they have a natural place in the current data structure.
I can see three options:
1. Add labels of the form "checksum.cloud.debian.org/${ALGO}" under
metadata.labels, for example "checksum.cloud.debian.org/sha256".
2. Add keys under data.info of the form "${ALGO}sum", for example
"sha256sum".
3. Add a new mapping within the "data" mapping called "checksums" with
keys for each algorithm, e.g. "data.checksums.sha256".
In each case I expect the values to be hex strings, effectively the same
as the first column of the output from sha1sum, sha256sum, sha512sum,
etc... from coreutils.
- Should we also generate the relevant SHA1SUM / SHA256SUM / SHA512SUM /
etc... files as might be consumed by the coreutils tools?
- Should we GPG-sign the manifests, logs, and/or checksum files? How
might we go about this?
Cheers,
Chris
--
Chris Boot
bo...@debian.org
signature.asc
Description: OpenPGP digital signature
Bug#932943: Missing SHA512 and gpg signature
Package: cloud.debian.org Severity: important On the secret files found here: http://cloud.debian.org/cdimage/cloud/ there's a json file attached. Unfortunately, there's no SHA512, and of course, no GPG signature of this file, so it's impossible check the validity of the images. Please at least add a SHA512, then we can see later how we can sign the json file. Cheers, Thomas Goirand (zigo)
