Re: debian-cd CVS back online - new jigdo fallback generation feature
At Thu, 18 Dec 2003 11:37:33 +0100, Richard Atterer <[EMAIL PROTECTED]> wrote: > > [1 ] > On Thu, Dec 18, 2003 at 10:45:15AM +0100, Raphael Hertzog wrote: > > the debian-cd CVS repository is back online (after a check of its > > integrity). > > Thanks, Raphaïl! > > I've just committed some code which makes the generation of fallback > mirrors much easier. Furthermore, it produces relative template URLs in > jigdo files by default. > > However, Phil, this /might/ break your publish_cds script - it is intended > to replace publish_cds. > > With this feature, you can make debian-cd generate a directory with > fallback links not after, but *during* template generation. This is done > with jigdo-file's --match-exec switch, which executes a user-defined > command whenever a file is found in the image. Fine, publish_cds only replaces what's in there already, so I can either remove that code, or make it so that the replacement is also relative. I still need publish CDs to move the images from the staging area to the published area, and generate the MD5SUMS files, but it would become pretty trivial if it didn't have to do the snapshot. > Advantages over publish_cds: Better integrated into debian-cd, less of a > hack. ;) The nice thing about setting it at publish time is that you can run publish_cds with a version number of "pre-release-test" say, and if it works, re-run it with "3.X_rY" on the same images, with no oportunity to screw up. ;-) Not that I do that sort of test, now that publish_cds works. Oh the other thing that you may not have noticed is that I have a hacked up version of publish_cds, called build_snapshot, that I run on raff to build the matching snapshot for us.cdimage.d.o. Since the bulk of fallback requests go to raff, I don't see the nasty hack going away too soon I'm afraid :-/ I should tidy build_snapshot up, take the funcionality out of publish_cds, and put build_snapshot into CVS really, so others can build snapshots (although I'm not sure how useful that is, given that there are only two places that the jigdos point at) Cheers, Phil.
Re: debian-cd CVS back online - new jigdo fallback generation feature
At Thu, 18 Dec 2003 11:37:33 +0100, Richard Atterer <[EMAIL PROTECTED]> wrote: > > [1 ] > On Thu, Dec 18, 2003 at 10:45:15AM +0100, Raphael Hertzog wrote: > > the debian-cd CVS repository is back online (after a check of its > > integrity). > > Thanks, Raphaïl! > > I've just committed some code which makes the generation of fallback > mirrors much easier. Furthermore, it produces relative template URLs in > jigdo files by default. > > However, Phil, this /might/ break your publish_cds script - it is intended > to replace publish_cds. > > With this feature, you can make debian-cd generate a directory with > fallback links not after, but *during* template generation. This is done > with jigdo-file's --match-exec switch, which executes a user-defined > command whenever a file is found in the image. Fine, publish_cds only replaces what's in there already, so I can either remove that code, or make it so that the replacement is also relative. I still need publish CDs to move the images from the staging area to the published area, and generate the MD5SUMS files, but it would become pretty trivial if it didn't have to do the snapshot. > Advantages over publish_cds: Better integrated into debian-cd, less of a > hack. ;) The nice thing about setting it at publish time is that you can run publish_cds with a version number of "pre-release-test" say, and if it works, re-run it with "3.X_rY" on the same images, with no oportunity to screw up. ;-) Not that I do that sort of test, now that publish_cds works. Oh the other thing that you may not have noticed is that I have a hacked up version of publish_cds, called build_snapshot, that I run on raff to build the matching snapshot for us.cdimage.d.o. Since the bulk of fallback requests go to raff, I don't see the nasty hack going away too soon I'm afraid :-/ I should tidy build_snapshot up, take the funcionality out of publish_cds, and put build_snapshot into CVS really, so others can build snapshots (although I'm not sure how useful that is, given that there are only two places that the jigdos point at) Cheers, Phil.
Re: debian-cd CVS back online - new jigdo fallback generation feature
At Thu, 18 Dec 2003 11:37:33 +0100, Richard Atterer <[EMAIL PROTECTED]> wrote: > > [1 ] > On Thu, Dec 18, 2003 at 10:45:15AM +0100, Raphael Hertzog wrote: > > the debian-cd CVS repository is back online (after a check of its > > integrity). > > Thanks, Raphaïl! > > I've just committed some code which makes the generation of fallback > mirrors much easier. Furthermore, it produces relative template URLs in > jigdo files by default. > > However, Phil, this /might/ break your publish_cds script - it is intended > to replace publish_cds. > > With this feature, you can make debian-cd generate a directory with > fallback links not after, but *during* template generation. This is done > with jigdo-file's --match-exec switch, which executes a user-defined > command whenever a file is found in the image. Fine, publish_cds only replaces what's in there already, so I can either remove that code, or make it so that the replacement is also relative. I still need publish CDs to move the images from the staging area to the published area, and generate the MD5SUMS files, but it would become pretty trivial if it didn't have to do the snapshot. > Advantages over publish_cds: Better integrated into debian-cd, less of a > hack. ;) The nice thing about setting it at publish time is that you can run publish_cds with a version number of "pre-release-test" say, and if it works, re-run it with "3.X_rY" on the same images, with no oportunity to screw up. ;-) Not that I do that sort of test, now that publish_cds works. Oh the other thing that you may not have noticed is that I have a hacked up version of publish_cds, called build_snapshot, that I run on raff to build the matching snapshot for us.cdimage.d.o. Since the bulk of fallback requests go to raff, I don't see the nasty hack going away too soon I'm afraid :-/ I should tidy build_snapshot up, take the funcionality out of publish_cds, and put build_snapshot into CVS really, so others can build snapshots (although I'm not sure how useful that is, given that there are only two places that the jigdos point at) Cheers, Phil. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: debian-cd CVS back online - new jigdo fallback generation feature
At Thu, 18 Dec 2003 11:37:33 +0100, Richard Atterer <[EMAIL PROTECTED]> wrote: > > [1 ] > On Thu, Dec 18, 2003 at 10:45:15AM +0100, Raphael Hertzog wrote: > > the debian-cd CVS repository is back online (after a check of its > > integrity). > > Thanks, Raphaïl! > > I've just committed some code which makes the generation of fallback > mirrors much easier. Furthermore, it produces relative template URLs in > jigdo files by default. > > However, Phil, this /might/ break your publish_cds script - it is intended > to replace publish_cds. > > With this feature, you can make debian-cd generate a directory with > fallback links not after, but *during* template generation. This is done > with jigdo-file's --match-exec switch, which executes a user-defined > command whenever a file is found in the image. Fine, publish_cds only replaces what's in there already, so I can either remove that code, or make it so that the replacement is also relative. I still need publish CDs to move the images from the staging area to the published area, and generate the MD5SUMS files, but it would become pretty trivial if it didn't have to do the snapshot. > Advantages over publish_cds: Better integrated into debian-cd, less of a > hack. ;) The nice thing about setting it at publish time is that you can run publish_cds with a version number of "pre-release-test" say, and if it works, re-run it with "3.X_rY" on the same images, with no oportunity to screw up. ;-) Not that I do that sort of test, now that publish_cds works. Oh the other thing that you may not have noticed is that I have a hacked up version of publish_cds, called build_snapshot, that I run on raff to build the matching snapshot for us.cdimage.d.o. Since the bulk of fallback requests go to raff, I don't see the nasty hack going away too soon I'm afraid :-/ I should tidy build_snapshot up, take the funcionality out of publish_cds, and put build_snapshot into CVS really, so others can build snapshots (although I'm not sure how useful that is, given that there are only two places that the jigdos point at) Cheers, Phil. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: debian-cd CVS back online - new jigdo fallback generation feature
On Thu, Dec 18, 2003 at 10:45:15AM +0100, Raphael Hertzog wrote: > the debian-cd CVS repository is back online (after a check of its > integrity). Thanks, Raphaël! I've just committed some code which makes the generation of fallback mirrors much easier. Furthermore, it produces relative template URLs in jigdo files by default. However, Phil, this /might/ break your publish_cds script - it is intended to replace publish_cds. With this feature, you can make debian-cd generate a directory with fallback links not after, but *during* template generation. This is done with jigdo-file's --match-exec switch, which executes a user-defined command whenever a file is found in the image. Advantages over publish_cds: Better integrated into debian-cd, less of a hack. ;) The following things in the default CONF.sh have changed: # HTTP/FTP URL for directory where you intend to make the templates # available. You should not need to change this; the default value "" # means "template in same dir as the .jigdo file", which is usually # correct. If it is non-empty, it needs a trailing slash. "%ARCH%" # will be substituted by the current architecture. #export JIGDOTEMPLATEURL="" # # Name of a directory on disc to create data for a fallback server in. # Should later be made available by you at the URL given in # JIGDOFALLBACKURLS. In the directory, two subdirs named "Debian" and # "Non-US" will be created, and filled with hard links to the actual # files in your FTP archive. Because of the hard links, the dir must # be on the same partition as the FTP archive! If unset, no fallback # data is created, which may cause problems - see README. #export JIGDOFALLBACKPATH="$(OUT)/snapshot/" # # Space-separated list of label->URL mappings for "jigdo fallback # server(s)" to add to .jigdo file. If unset, no fallback URL is # added, which may cause problems - see README. #export JIGDOFALLBACKURLS="Debian=http://myserver/snapshot/Debian/ Non-US=http://myserver/snapshot/Non-US/"; # # Space-separated list of "include URLs" to add to the .jigdo file. # The included files are used to provide an up-to-date list of Debian # mirrors to the jigdo _GUI_application_ (_jigdo-lite_ doesn't support # "[Include ...]"). export JIGDOINCLUDEURLS="http://cdimage.debian.org/debian-cd/debian-servers.jigdo"; # # $JIGDOTEMPLATEURL and $JIGDOINCLUDEURLS are passed to # "tools/jigdo_header", which is used by default to generate the # [Image] and [Servers] sections of the .jigdo file. You can provide # your own script if you need the .jigdo file to contain different # data. #export JIGDOSCRIPT="myscript" Additional info in the README: About jigdo "fallback servers": jigdo works by downloading individual packages and other files from a normal Debian mirror, and using them to regenerate a CD/DVD image. However, the content of Debian mirrors changes over time, files are added and removed. But jigdo must have access to all files needed for the image it has to regenerate, even those that have been removed from the normal Debian mirrors. A fallback server contains a backup of the Debian FTP space for the moment the .jigdo files were generated. This backup is made available under a certain URL which is written to the .jigdo files. jigdo will *only* revert to the fallback server after an unsuccessful attempt to retrieve a file from the normal user-selected Debian mirror, so the bandwidth requirements are modest. A fallback is even necessary for .jigdo files of the stable release, because some files (typically documentation or boot floppies) can change at any time. debian-cd allows you to automatically create a directory on disc which is suitable for use as a fallback mirror. It is populated with hard links to the archive contents. In CONF.sh, simply supply as JIGDOFALLBACKPATH the name of the directory, and as JIGDOFALLBACKURLS the URLs under which it will be made available (two separate URLs are necessary, one for "Debian" and one for "Non-US"). Hopefully I haven't broken debian-cd by committing this, but it seems to work fine judging from my tests on farbror. Cheers, Richard -- __ _ |_) /| Richard Atterer | GnuPG key: | \/¯| http://atterer.net | 0x888354F7 ¯ '` ¯ pgpqA2nyKliSX.pgp Description: PGP signature
debian-cd CVS back online
Hello everyone, the debian-cd CVS repository is back online (after a check of its integrity). Please commit any change that you postponed until now. Cheers, -- Raphaël Hertzog -+- http://www.ouaza.com Formation Linux et logiciel libre : http://www.logidee.com Earn money with free software: http://www.geniustrader.org
Re: debian-cd CVS back online - new jigdo fallback generation feature
On Thu, Dec 18, 2003 at 10:45:15AM +0100, Raphael Hertzog wrote: > the debian-cd CVS repository is back online (after a check of its > integrity). Thanks, Raphaël! I've just committed some code which makes the generation of fallback mirrors much easier. Furthermore, it produces relative template URLs in jigdo files by default. However, Phil, this /might/ break your publish_cds script - it is intended to replace publish_cds. With this feature, you can make debian-cd generate a directory with fallback links not after, but *during* template generation. This is done with jigdo-file's --match-exec switch, which executes a user-defined command whenever a file is found in the image. Advantages over publish_cds: Better integrated into debian-cd, less of a hack. ;) The following things in the default CONF.sh have changed: # HTTP/FTP URL for directory where you intend to make the templates # available. You should not need to change this; the default value "" # means "template in same dir as the .jigdo file", which is usually # correct. If it is non-empty, it needs a trailing slash. "%ARCH%" # will be substituted by the current architecture. #export JIGDOTEMPLATEURL="" # # Name of a directory on disc to create data for a fallback server in. # Should later be made available by you at the URL given in # JIGDOFALLBACKURLS. In the directory, two subdirs named "Debian" and # "Non-US" will be created, and filled with hard links to the actual # files in your FTP archive. Because of the hard links, the dir must # be on the same partition as the FTP archive! If unset, no fallback # data is created, which may cause problems - see README. #export JIGDOFALLBACKPATH="$(OUT)/snapshot/" # # Space-separated list of label->URL mappings for "jigdo fallback # server(s)" to add to .jigdo file. If unset, no fallback URL is # added, which may cause problems - see README. #export JIGDOFALLBACKURLS="Debian=http://myserver/snapshot/Debian/ Non-US=http://myserver/snapshot/Non-US/"; # # Space-separated list of "include URLs" to add to the .jigdo file. # The included files are used to provide an up-to-date list of Debian # mirrors to the jigdo _GUI_application_ (_jigdo-lite_ doesn't support # "[Include ...]"). export JIGDOINCLUDEURLS="http://cdimage.debian.org/debian-cd/debian-servers.jigdo"; # # $JIGDOTEMPLATEURL and $JIGDOINCLUDEURLS are passed to # "tools/jigdo_header", which is used by default to generate the # [Image] and [Servers] sections of the .jigdo file. You can provide # your own script if you need the .jigdo file to contain different # data. #export JIGDOSCRIPT="myscript" Additional info in the README: About jigdo "fallback servers": jigdo works by downloading individual packages and other files from a normal Debian mirror, and using them to regenerate a CD/DVD image. However, the content of Debian mirrors changes over time, files are added and removed. But jigdo must have access to all files needed for the image it has to regenerate, even those that have been removed from the normal Debian mirrors. A fallback server contains a backup of the Debian FTP space for the moment the .jigdo files were generated. This backup is made available under a certain URL which is written to the .jigdo files. jigdo will *only* revert to the fallback server after an unsuccessful attempt to retrieve a file from the normal user-selected Debian mirror, so the bandwidth requirements are modest. A fallback is even necessary for .jigdo files of the stable release, because some files (typically documentation or boot floppies) can change at any time. debian-cd allows you to automatically create a directory on disc which is suitable for use as a fallback mirror. It is populated with hard links to the archive contents. In CONF.sh, simply supply as JIGDOFALLBACKPATH the name of the directory, and as JIGDOFALLBACKURLS the URLs under which it will be made available (two separate URLs are necessary, one for "Debian" and one for "Non-US"). Hopefully I haven't broken debian-cd by committing this, but it seems to work fine judging from my tests on farbror. Cheers, Richard -- __ _ |_) /| Richard Atterer | GnuPG key: | \/¯| http://atterer.net | 0x888354F7 ¯ '` ¯ pgp0.pgp Description: PGP signature
debian-cd CVS back online
Hello everyone, the debian-cd CVS repository is back online (after a check of its integrity). Please commit any change that you postponed until now. Cheers, -- Raphaël Hertzog -+- http://www.ouaza.com Formation Linux et logiciel libre : http://www.logidee.com Earn money with free software: http://www.geniustrader.org -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
