Re: jigdo vs binary diff

[George Danchev]

On Thursday 10 June 2004 01:35, Richard Atterer wrote:
> On Wed, Jun 09, 2004 at 08:45:05PM +0300, George Danchev wrote:
> > what advantages I'll get when using jigdo to update my old image instead
> > of patching the old iso with rdiff.
>
> Well, Debian doesn't offer CD image upgrades in rdiff/xdelta format.
> Generally, IMHO jigdo has the following advantages over binary diffs:
>
> - jigdo allows you to update to your image from /any/ previous version of
>   the same image. That's most useful with weekly snapshots - imagine having
>   to download and apply 6 "patches" because you last fetched a weekly
>   snapshot 6 weeks ago.
>
> - you can download all the changed/new packages from any Debian mirror, not
>   just designated "CD patch mirrors". With patches, IMHO many regular
>   Debian mirrors would not be too excited about mirroring large patch
>   files, so the few mirrors who offered them would be fairly busy.
>
> - related to the point above: Debian does not need to duplicate the data
>   for the changed/new packages on the servers in a binary diff, which saves
>   space. (jigdo can download these packages from the "normal archive" of
>   packages on a Debian server.)
>
> - jigdo allows completely different kinds of "updates", e.g. you can update
>   from a set of CD images to one DVD image, without downloading all the
>   package data again.
>
> - jigdo updates still work fine in case your old CD has read errors.

Thanks for pointing these out. Your explanation is more that sufficient. I 
would add one more advantage of jigdo over rdiff:
- The new image might be piped instead of generated and stored on the server 
from debian-cd (or whatever) to jigdo-file to create the .jigdo/.template 
files. With rdiff one need to have the old and new image files to create the 
diff.

I think this comparing information is very important to clear the things up 
for the users and must be added to the jigdo's site and docs.

-- 
pub 4096R/0E4BD0AB  2003-03-18  
fingerprint 1AE7 7C66 0A26 5BFF DF22 5D55 1C57 0C89 0E4B D0AB 


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: JTE (Jigdo Template Export) v1.0

[Steve McIntyre]

On Thu, Jun 10, 2004 at 12:16:23AM +0200, Richard Atterer wrote:
>On Wed, Jun 09, 2004 at 05:34:03PM +0100, Steve McIntyre wrote:
>> Unfortunately, I don't see us (quite) getting that far. To generate the
>> md5 of the full image file (which is kind of useful), we need to read all
>> of the data through anyway. You can't simply lump together multiple md5
>> chunks.
>
>Ah - indeed, you're right. :-/
>
>Hmmm. If the ability to create images this fast turns out to be a
>"must-have" feature one day, the template format could be changed: Either
>the image md5sum field could be left at zero to indicate "no md5sum
>available", or we could switch over to using what I call a "64 bit rsync
>sum" - a (cryptographically weak) checksum which allows "lumping together", 
>already used internally by jigdo.

Hmm. Md5sums are nice, and people are comfortable with them. We'll
need to be generating md5sums of the full ISO images anyway for that
reason.

In my original discussion with Phil, I was hoping that at some point
we could completely bypass a lot of the CD-building code and write
templates by *just* parsing the Packages and Sources files on a
mirror. That had the same problem. I think the quickest place to do a
lot of this is in mkisofs, which we're already tied to to actually
create the image. At least we can then optimise it so we stream
through the set of files just once.

>> That's a bug to do with large file support in libstdc++ IIRC? I've
>> noticed the problem myself with large jigdo images. I'm just testing JTE
>> v1.1 right now to make sure I don't have similar issues.
>
>There *is* currently a problem with big files for gcc 3.x, x<4. However,
>the issue Manty was referring to is something else:
>
>Currently, the algorithm which searches for matching files inside the image
>sometimes has to discard prospective matches in order to avoid becoming too
>slow (ie not O(n²) instead of O(n) time). I have an idea for a way to
>improve the accuracy, which should result in a smaller template size
>because more files are found in the image.

Ah, OK.

>> Absolutely. It'd be great to be able to get _lots_ of jigdo files created
>> for all the different options including multiple variants of CD and
>> different DVDs. There are still some more optimisations that should be
>> possible in the image-building stages; at the moment we end up md5summing
>> the entirety of the data on each disk several times, and that's a little
>> bit wasteful.
>
>If you're talking about the md5sums.txt files on the CDs, note that you can 
>take advantage of jigdo-file's cache when creating them:
>
>  jigdo-file md5sum --cache=x.db --hex FILES...

At the moment, I'm not actually using the cache file at all - I don't
have a need for it. Maybe we can optimise still further and add
md5-checking (and caching) into my JTE code. There are several places
where we read in all the files and/or generate checksums when doing a
full CD run at the moment:

mirror check
creating the md5sums.txt on each disk
create the iso image
jigdo template creation

It would be nice to be able to lose a couple of these passes, and it
should be possible.

-- 
Steve McIntyre, Cambridge, UK.[EMAIL PROTECTED]
Welcome my son, welcome to the machine.


signature.asc
Description: Digital signature

jigdo vs binary diff

[Richard Atterer]

On Wed, Jun 09, 2004 at 08:45:05PM +0300, George Danchev wrote:
> what advantages I'll get when using jigdo to update my old image instead of 
> patching the old iso with rdiff.

Well, Debian doesn't offer CD image upgrades in rdiff/xdelta format. 
Generally, IMHO jigdo has the following advantages over binary diffs:

- jigdo allows you to update to your image from /any/ previous version of 
  the same image. That's most useful with weekly snapshots - imagine having 
  to download and apply 6 "patches" because you last fetched a weekly 
  snapshot 6 weeks ago.

- you can download all the changed/new packages from any Debian mirror, not 
  just designated "CD patch mirrors". With patches, IMHO many regular
  Debian mirrors would not be too excited about mirroring large patch
  files, so the few mirrors who offered them would be fairly busy.
  
- related to the point above: Debian does not need to duplicate the data 
  for the changed/new packages on the servers in a binary diff, which saves
  space. (jigdo can download these packages from the "normal archive" of
  packages on a Debian server.)

- jigdo allows completely different kinds of "updates", e.g. you can update 
  from a set of CD images to one DVD image, without downloading all the
  package data again.

- jigdo updates still work fine in case your old CD has read errors.

Cheers,

  Richard

-- 
  __   _
  |_) /|  Richard Atterer |  GnuPG key:
  | \/¯|  http://atterer.net  |  0x888354F7
  ¯ '` ¯


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: JTE (Jigdo Template Export) v1.0

[Richard Atterer]

On Wed, Jun 09, 2004 at 05:34:03PM +0100, Steve McIntyre wrote:
> Unfortunately, I don't see us (quite) getting that far. To generate the
> md5 of the full image file (which is kind of useful), we need to read all
> of the data through anyway. You can't simply lump together multiple md5
> chunks.

Ah - indeed, you're right. :-/

Hmmm. If the ability to create images this fast turns out to be a
"must-have" feature one day, the template format could be changed: Either
the image md5sum field could be left at zero to indicate "no md5sum
available", or we could switch over to using what I call a "64 bit rsync
sum" - a (cryptographically weak) checksum which allows "lumping together", 
already used internally by jigdo.

> >Richard: I remember that you were about to put out a new version of
> >jigdo that could help in this, how is this going?
> 
> That's a bug to do with large file support in libstdc++ IIRC? I've
> noticed the problem myself with large jigdo images. I'm just testing JTE
> v1.1 right now to make sure I don't have similar issues.

There *is* currently a problem with big files for gcc 3.x, x<4. However,
the issue Manty was referring to is something else:

Currently, the algorithm which searches for matching files inside the image
sometimes has to discard prospective matches in order to avoid becoming too
slow (ie not O(n²) instead of O(n) time). I have an idea for a way to
improve the accuracy, which should result in a smaller template size
because more files are found in the image.

> Absolutely. It'd be great to be able to get _lots_ of jigdo files created
> for all the different options including multiple variants of CD and
> different DVDs. There are still some more optimisations that should be
> possible in the image-building stages; at the moment we end up md5summing
> the entirety of the data on each disk several times, and that's a little
> bit wasteful.

If you're talking about the md5sums.txt files on the CDs, note that you can 
take advantage of jigdo-file's cache when creating them:

  jigdo-file md5sum --cache=x.db --hex FILES...

Cheers,

  Richard

-- 
  __   _
  |_) /|  Richard Atterer |  GnuPG key:
  | \/¯|  http://atterer.net  |  0x888354F7
  ¯ '` ¯


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: TRANS.TBL on CDs

[Steve McIntyre]

On Wed, Jun 09, 2004 at 11:45:42PM +0200, Santiago Garcia Mantinan wrote:
>> >Agreed. Feel free to commit a fix.
>> Done.
>
>I have applied your changes to gluck before today's build, but I do not see
>any important changes in sizes :- at least comparing today's build with
>yesterday's and I have checked them and there is no TRANS.TBL files on them.

OK. What size difference is there? It won't be huge (a few bytes per
directory), but I'd expect it to be noticeable, especially on the
businesscard image where we're short of space.

-- 
Steve McIntyre, Cambridge, UK.[EMAIL PROTECTED]
We don't need no education.
We don't need no thought control.


signature.asc
Description: Digital signature

Re: Testing CD/DVD images "Official"?

[Santiago Garcia Mantinan]

> I'm working on getting setup to become yet another Debian-CD vendor, and 
> am wondering about the proper labeling for the testing CD/DVD images 
> available from http://cdimage.debian.org/pub/cdimage-testing/

I wouldn't even think in distributing those images, either official or
unofficial, they have not been well tested and the few test they have had
show that they are badly broken, the testing until now has been done using
the small cd images (businesscard and netinst) and not the full sized cd
images or dvd images.

> Can images formed from the .jigdo files/templates available here be 
> labeled 'official' (assuming MD5 sums match) ie:
> 
>   Debian GNU/Linux
>   Testing "Sarge"
>   Official Snapshot
>   alpha Binary-1
>   

I'd say that it is better for everyone if they are not yet distributed, but
maybe somebody with release powers can tell you something else :-?

Regards...
-- 
Manty/BestiaTester -> http://manty.net


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: TRANS.TBL on CDs

[Santiago Garcia Mantinan]

> >Agreed. Feel free to commit a fix.
> Done.

I have applied your changes to gluck before today's build, but I do not see
any important changes in sizes :- at least comparing today's build with
yesterday's and I have checked them and there is no TRANS.TBL files on them.

Regards...
-- 
Manty/BestiaTester -> http://manty.net


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: Testing CD/DVD images "Official"?

[Richard Atterer]

On Wed, Jun 09, 2004 at 12:05:11PM -0500, Charles Steinkuehler wrote:
> Can images formed from the .jigdo files/templates available here be
> labeled 'official' (assuming MD5 sums match) ie:
> 
>   Debian GNU/Linux
>   Testing "Sarge"
>   Official Snapshot
>   alpha Binary-1
>   

Labelling the CDs like that is OK IMO. See
 for my attempt at describing what
"official" really means in this context.

Unless anyone objects, I'll add to
 a paragraph on how to label 
snapshot CDs:

  In the case of official weekly snapshots, version numbers like "3.0 rev1"
  should not be used to avoid confusion with released Debian versions.
  Instead, label the image with a codename like "sarge" or a distribution
  name like "testing". Also add "snapshot" and the date of the snapshot:
  
  Debian GNU/Linux "sarge"
  Official Snapshot alpha Binary-2
  2003-12-31

> - The README file in the debian-cd package (found in a general search to
> find README.official file, which is listed in the debain-cd changelog)
> just says to never use "official" on anything I build myself.

...which is correct, but only applies if you use debian-cd to create your
own CD images. If you use jigdo to create the image, you're recreating a
bit-exact copy of the official image, so no problem.

> - The images available in the cdimage-testing directory seem to be 
> labeled "official" (at least the first one I checked, which was titled: 
> 'Debian GNU/Linux Testing "Sarge" Official Snapshot alpha Binary-1' in 
> the README.txt file ).

Yes, they're "official".

Cheers,

  Richard

-- 
  __   _
  |_) /|  Richard Atterer |  GnuPG key:
  | \/¯|  http://atterer.net  |  0x888354F7
  ¯ '` ¯


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: TRANS.TBL on CDs

[Steve McIntyre]

On Wed, Jun 09, 2004 at 08:17:23AM +0200, Raphael Hertzog wrote:
>Quoting Steve McIntyre:
>> Guys,
>> 
>> We're still putting TRANS.TBL files in every directory on the
>> CDs. Surely by now we can just lose them? They're going to be wasting
>> space on all the images (especially important on netinst and business
>> card), and I'd bet that we're not bothered about supporting people
>> renaming files by hand on DOS machines any more...?
>
>Agreed. Feel free to commit a fix.

Done.

-- 
Steve McIntyre, Cambridge, UK.[EMAIL PROTECTED]
Can't keep my eyes from the circling sky,
Tongue-tied & twisted, Just an earth-bound misfit, I...


signature.asc
Description: Digital signature

Re: Bug#253033: installation-reports

[Steve McIntyre]

On Tue, Jun 08, 2004 at 11:11:39PM -0400, Joey Hess wrote:
>Steve McIntyre wrote:
>> Hmmm. That wasn't very successful. Netcfg loaded fine off the CD, but
>> only after I loaded it by hand. The 3c59x driver for the network card
>> in the test machine just didn't get loaded.
>> 
>> Looking at the *nic* .udebs on the image, they seem quite old (9th
>> May); is that normal?
>
>So I had a look at this CD. It seems to have a rather old d-i initrd on
>it, with a 2.4.25 kernel. This is a problem, since all the kernel udebs
>on the CD are for the 2.4.26 kernel. I noticed when partman didn't have
>support for the ext3 filesystem, since ext3-modules was not loaded as
>there was no version for 2.4.25 on the CD. Your nic modules problem is
>probably the same.

Ah, yes. That would explain why I couldn't load the nic drivers.

>The initrd on the CD is version 20040429, which is the one for d-i
>beta4. I don't know why the full CDs are being built against the beta4
>initrds, but this is a problem.
>
>I told jigdo to use
>http://cdimage.debian.org/pub/cdimage-testing/cd/jigdo-area/i386/sarge-i386-1.jigdo,
>and my iso has a md5sum of f0b66636c58c4d2e455f26db5519bff5.

Yup, same image as me.

-- 
Steve McIntyre, Cambridge, UK.[EMAIL PROTECTED]
"This dress doesn't reverse." -- Alden Spiess


signature.asc
Description: Digital signature

Re: JTE (Jigdo Template Export) v1.0

[George Danchev]

On Wednesday 09 June 2004 19:21, Richard Atterer wrote:
> On Wed, Jun 09, 2004 at 05:35:10PM +0200, Santiago Garcia Mantinan wrote:
> > Richard: I remember that you were about to put out a new version of jigdo
> > that could help in this, how is this going?
>
> It's on my TODO list, but I'm afraid I haven't found the time yet to work
> on it. :-/

Hello, 
what advantages I'll get when using jigdo to update my old image instead of 
patching the old iso with rdiff.

-- 
pub 4096R/0E4BD0AB  2003-03-18  
fingerprint 1AE7 7C66 0A26 5BFF DF22 5D55 1C57 0C89 0E4B D0AB 


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Debian_sarge task missing debian-installer+kernel?

[Santiago Garcia Mantinan]

Hi!

Joeyh Hess has told me today that he had noticed that some files included in
the debian-installer+kernel were not included in some cds (debian-educ) and
that after seing this he had found that debian-installer+kernel was not
included in the Debian_sarge task.

I believe that debian-installer+kernel should be part of Debian_sarge,
unless there can be a problem if any package part of debian-installer+kernel
is already in Debian_sarge, I have included debian-installer+kernel from
Debian_sarge at gluck, for testing purposes, but I'd like to know opinions
on this from other people, is this wrong?

Regards...
-- 
Manty/BestiaTester -> http://manty.net


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Testing CD/DVD images "Official"?

[Charles Steinkuehler]

I'm working on getting setup to become yet another Debian-CD vendor, and 
am wondering about the proper labeling for the testing CD/DVD images 
available from http://cdimage.debian.org/pub/cdimage-testing/

Can images formed from the .jigdo files/templates available here be 
labeled 'official' (assuming MD5 sums match) ie:

  Debian GNU/Linux
  Testing "Sarge"
  Official Snapshot
  alpha Binary-1
  
...or do they need an 'unofficial' moniker?
NOTES:
- I have been unable to find the README.official file referenced by 
README.CD-manufacture in the debian repository.  Perhaps this needs to 
be updated?

- The README file in the debian-cd package (found in a general search to 
find README.official file, which is listed in the debain-cd changelog) 
just says to never use "official" on anything I build myself.

- The images available in the cdimage-testing directory seem to be 
labeled "official" (at least the first one I checked, which was titled: 
'Debian GNU/Linux Testing "Sarge" Official Snapshot alpha Binary-1' in 
the README.txt file ).

I just want to be sure, before I mistakenly label something official 
that's not.

--
Charles Steinkuehler
[EMAIL PROTECTED]
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: JTE (Jigdo Template Export) v1.0

[Steve McIntyre]

On Wed, Jun 09, 2004 at 05:35:10PM +0200, Santiago Garcia Mantinan wrote:
>On Jun 07 2004, Richard Atterer wrote:
>> Yes, but once you can use the cache, mkisofs needn't even read the file
>> data anymore. It could just skip over the file, output the filename to the
>> .jte and leave the rest to jigdo-file. I can imagine that image creation
>> would be a matter of 30 seconds that way...
>
>Wow, this would really speed things up a lot.

Unfortunately, I don't see us (quite) getting that far. To generate
the md5 of the full image file (which is kind of useful), we need to
read all of the data through anyway. You can't simply lump together
multiple md5 chunks.

>BTW... now that you are at this, I remember talking to Richard about a
>warning in the dvd creation logs, because of files over the 4 gigas, and
>indeed I think that we are having some kind of trouble with jigdo creation
>on dvds, as the templates for i386 dvds sum around 265 megs where the
>templates for the cds sum under 15 megs.
>
>Richard: I remember that you were about to put out a new version of jigdo
>that could help in this, how is this going?

That's a bug to do with large file support in libstdc++ IIRC? I've
noticed the problem myself with large jigdo images. I'm just testing
JTE v1.1 right now to make sure I don't have similar issues.

>Well, I'm commenting all these because DVDs are really going to be important
>for our distro in the near future, whether it is in 4.7 gigs dvd format or
>even bigger to allow us to distribute sarge on a single double layer media
>dvd.

Absolutely. It'd be great to be able to get _lots_ of jigdo files
created for all the different options including multiple variants of
CD and different DVDs. There are still some more optimisations that
should be possible in the image-building stages; at the moment we end
up md5summing the entirety of the data on each disk several times, and
that's a little bit wasteful.

-- 
Steve McIntyre, Cambridge, UK.[EMAIL PROTECTED]
You raise the blade, you make the change... You re-arrange me 'til I'm sane...


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: JTE (Jigdo Template Export) v1.0

[Richard Atterer]

On Wed, Jun 09, 2004 at 05:35:10PM +0200, Santiago Garcia Mantinan wrote:
> Richard: I remember that you were about to put out a new version of jigdo
> that could help in this, how is this going?

It's on my TODO list, but I'm afraid I haven't found the time yet to work 
on it. :-/

  Richard

-- 
  __   _
  |_) /|  Richard Atterer |  GnuPG key:
  | \/¯|  http://atterer.net  |  0x888354F7
  ¯ '` ¯


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: JTE (Jigdo Template Export) v1.0

[Santiago Garcia Mantinan]

On Jun 07 2004, Richard Atterer wrote:
> Yes, but once you can use the cache, mkisofs needn't even read the file
> data anymore. It could just skip over the file, output the filename to the
> .jte and leave the rest to jigdo-file. I can imagine that image creation
> would be a matter of 30 seconds that way...

Wow, this would really speed things up a lot.

BTW... now that you are at this, I remember talking to Richard about a
warning in the dvd creation logs, because of files over the 4 gigas, and
indeed I think that we are having some kind of trouble with jigdo creation
on dvds, as the templates for i386 dvds sum around 265 megs where the
templates for the cds sum under 15 megs.

Richard: I remember that you were about to put out a new version of jigdo
that could help in this, how is this going?

Well, I'm commenting all these because DVDs are really going to be important
for our distro in the near future, whether it is in 4.7 gigs dvd format or
even bigger to allow us to distribute sarge on a single double layer media
dvd.

Regards...
-- 
Manty/BestiaTester -> http://manty.net


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Wed, 09 Jun 2004 09:19:35 -0600

[Terrie Dunn]

Here is a casino giving away $25 Free when you sign up an account.
No credit card required
http://secret.rxt2.org/iwin.html


Jamaal

Re: USB memory stick

[Jacques]

Maarten Weijman wrote:
Debian team,
 
I am having a slight problem with the live debian cd, because my 
computer does not have a cd drive.
Would it be possible to write the cd images to a usb memory stick and 
let it work?
Your help would be very much apreciated.
 
yours sincerely,
 
Maarten Weijman
Look at debian site, http://www.debian.org/devel/debian-installer/, 
there might be something to boot off a usb device. You should need an 
internet access though in order to install the rest of the distribution.

Hope this helps
Jacques
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]