Re: Archive changes

[Thomas Schmitt]

Hi,

Joerg Jaspert wrote:
> > > As of now, InRelease/Release files, Packages and Sources no longer
> > > provide MD5Sum and SHA1sums, only SHA256.

Steve McIntyre wrote:
> > We can't fix this easily in a short
> > timescale - current released jigdo clients (both in Debian and
> > externally) use md5 internally to reference files in the archive.

Joerg Jaspert wrote:
> Im not sure we *want* to support that, at least for sure not for more than
> stretch, but we could.

> > We need more time than that to fix things up.

> How much?

Currently Jigdo is produced by xorriso, libisofs, and libjte.
libjte upstream is Steve, with George Danchev and me as co-authors.
libisofs and xorriso upstream is me.

I only know of consumer jigit. Are there others (externals) ?

--
xorriso:

- Calls API function libjte_set_md5_path().
  Either there needs to be a new call for the SHA256 list file path,
  or we keep this function as half misnomer and let the MD5 file reader
  of libjte recognize SHA256 automatically.

libisofs:

- Seems not to refer to the checksum type.
  (Let's hope my assessment is true.)

libjte:

- Offers API call  libjte_set_md5_path().
  As mentioned above, one could let it recognize MD5 and SHA256 list
  content so that the callers do not have to care for the change.
  Only the runner of xorriso (debian-cd, i assume) would have to
  submit a suitable SHA256 list file instead of the old MD5 list.

- Internal function parse_md5_list() and its subordinates read
  the given list file and create a list of md5_list_entry_t.
typedef struct _md5_list_entry
{
struct _md5_list_entry *next;
unsigned char   MD5[16];
uint64_t size;
char   *filename;
} md5_list_entry_t;
  This struct and the reader would have to be adapted to the new
  checksum list format.

- The equipment for SHA256 computation seems already present.
  At least there is a file sha256.c.

- Internal call list_file_in_jigdo() makes the decision, whether
  a data file in the future output matches a listed file item by
  basename and by MD5.
  This function needs to learn to process SHA256.

- Some message texts talk of "MD5" hardcodedly. There we'd need a
  variable text snippet.

- Internal function write_template_desc_entries() in one of its
  cases writes a jigdo_file_entry_t to the .template file.
typedef struct _jigdo_file_entry
{
unsigned char type;
unsigned char fileLen[6];
unsigned char fileRsync[8];
unsigned char fileMD5[16];
} jigdo_file_entry_t;
  This will cause a layout change in the .template file.
  (Jigdo consumer jigit has Steve as upstrem.)

- Internal function add_file_entry() records a file_entry_t for
  each matching file:
typedef struct _file_entry
{
unsigned char   md5[16];
off_t   file_length;
uint64_trsyncsum;
char   *filename;
} file_entry_t;
  Internal function write_jigdo_file() writes the MD5 of the .template
  file and the MD5s of the file_entry_t (as union member of entry_t).
  This will cause a layout change in the .jigdo file.

- The most probably unused "Traditional Data File API" deals with MD5
  sums computed from data files:
int libjte_decide_file_jigdo(struct libjte_env *jte_handle,
 char *filename, off_t size, char **mirror_name,
 unsigned char md5[16]);
int libjte_write_match_record(struct libjte_env *jte_handle,
char *filename, char *mirror_name, int sector_size,
off_t size, unsigned char md5[16]);
  One should consider to give up this API in favor of the
  "Simplified Data File API" as used by libisofs.

There are many more occurences of [Mm][Dd]5 in jte.c and libjte.c.
They seem to be consequences of the gestures described above.
All of them have to be checked and adapted, of course.

--
Summary:

Program jigit and library libjte would need a thorough overhaul.
The higher levels inside xorriso could stay as they are.

The impact on external Jidgo tools has to be judged by their
maintainers.

The effort in libjte would be at least one week of dense work and
testing. A substantial part of this is the plight to analyze the
old code which none of us touched in the last 5 years.


Have a nice day :)

Thomas

About archived squeeze

[Mathieu Parent]

Hello Joerg,

squeeze-lts on archive [1] has a Valid-Until header which makes apt complain.

Also, squeeze is still on security [2] and LTS is still on primary miror [3].

Regards


[1]: http://archive.debian.org/debian/dists/squeeze-lts/Release
[2]: http://security.debian.org/dists/squeeze/updates/Release
[3]: http://ftp.debian.org/debian/dists/squeeze-lts/Release

-- 
Mathieu Parent

Re: Archive changes

[Guillem Jover]

Hi!

On Tue, 2016-03-15 at 15:32:40 -0700, Josh Triplett wrote:
> On Tue, Mar 15, 2016 at 11:15:16PM +0100, Joerg Jaspert wrote:
> > I've just activated a few changes to the archive we talk(ed) about for a
> > long time. And while it is not exactly the start of this release cycle,
> > it should still work out nicely (so one hopes).
> > 
> > As of now, InRelease/Release files, Packages and Sources no longer
> > provide MD5Sum and SHA1sums, only SHA256.
> > 
> > Additionally I turned off generating gzip compressed versions of those
> > files, xz is there.
> 
> In addition to the security improvement,

The only way this might possibly improve security is by perhaps flushing
out things that rely exclusively on weak hashes, once these start to fail.
Otherwise reducing the amount of hashes is not a security improvement.

Increased security is what apt is doing now, which will validate all the
hashes but consider weak ones not sufficient to consider the repo secure.

> a quick analysis on
> binary-amd64 shows that this will reduce the size of Packages for
> binary-amd64 from 39MB to 35MB (uncompressed), and the size of the
> xz-compressed version from 7.9MB to 5.9MB.  Very nice!

While the space reduction is nice…

> That also helps reduce the impact and overhead of adding additional
> binary packages.

…I get the feeling you seem to be fixated on the metadata as the only
problem with an explosion of additional binary packages (tiny or not).
As I've commented on before, metadata size is just a tiny part of the
overhead for a package introduced into the system:

  

Thanks,
Guillem

Re: Archive changes

[Joerg Jaspert]

Am 2016-03-16 01:20, schrieb Steve McIntyre:

I've just activated a few changes to the archive we talk(ed) about for 
a
long time. And while it is not exactly the start of this release 
cycle,

it should still work out nicely (so one hopes).
As of now, InRelease/Release files, Packages and Sources no longer
provide MD5Sum and SHA1sums, only SHA256.

That (Packages and Sources) will break jigdo generation for debian-cd
(and hence all CD/DVD/BD builds). We can't fix this easily in a short
timescale - current released jigdo clients (both in Debian and
externally) use md5 internally to reference files in the archive. Not
as a *security* feature; this is the core design of jigdo.


B.

If it really turns out that this is unchangeable for now - the code is
flexible enough to allow to freely select checksum types by suite, so
md5 could be turned on for a suite too. Without getting sha1 back.
(Its written so that it can simply support any checksum apt_pkg 
supports).


Im not sure we *want* to support that, at least for sure not for more 
than stretch, but we could.



Additionally I turned off generating gzip compressed versions of those
files, xz is there.

And that will break various other parts of debian-cd.


Question is how hard a change of a compression tool is there.

To test it, this is limited to experimental. We hope nothing breaks on 
it,

but lets try for a few days. If that works out, we should adjust
unstable, and another short time later coordinate with the release 
team

to adjust testing, so it ends up in the next release.

Please, no. We need more time than that to fix things up.


Its not like its an entirely new idea to do this.
How much?


Also, from reading the current replies, noone has a problem with 
removing sha1, so that one seems a set thing. md5 and gz files removals 
make people more happy.


--
bye Joerg

Re: Archive changes

[Joerg Jaspert]

Am 2016-03-16 08:32, schrieb Stefano Zacchiroli:


>To test it, this is limited to experimental. We hope nothing breaks on it,
>but lets try for a few days. If that works out, we should adjust
>unstable, and another short time later coordinate with the release team
>to adjust testing, so it ends up in the next release.

I've noticed an upcoming breakage in sources.debian.net too, and
reported it as #818324. Luckily that is trivial to fix for us.

But it might be a good idea to track all upcoming breakages using the
BTS. Fell free to tag and/or mark as blocking the above bug report as
needed. And to report more similar bugs :)


Yes please.

--
bye Joerg

Re: Archive changes

[Paul Wise]

On Wed, Mar 16, 2016 at 6:15 AM, Joerg Jaspert wrote:

> Additionally I turned off generating gzip compressed versions of those
> files, xz is there.

This has broken some things, at least buildd status pages, DDPO, qadb
ftp-update, sources.d.n.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Archive changes

[Stefano Zacchiroli]

On Wed, Mar 16, 2016 at 12:20:43AM +, Steve McIntyre wrote:
> >To test it, this is limited to experimental. We hope nothing breaks on it,
> >but lets try for a few days. If that works out, we should adjust
> >unstable, and another short time later coordinate with the release team
> >to adjust testing, so it ends up in the next release.
> 
> Please, no. We need more time than that to fix things up.

I've noticed an upcoming breakage in sources.debian.net too, and
reported it as #818324. Luckily that is trivial to fix for us.

But it might be a good idea to track all upcoming breakages using the
BTS. Fell free to tag and/or mark as blocking the above bug report as
needed. And to report more similar bugs :)

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader . . . . . @zacchiro . . . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


signature.asc
Description: PGP signature