Re: Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
On Fri, Feb 06, 2009 at 11:15:43AM +0100, Frans Pop wrote: On Thursday 05 February 2009, Frans Pop wrote: So the following really has to be implemented in update_tasks itself. On Thursday 05 February 2009, Frans Pop wrote: So IMO: * ideally we should run update_tasks for every architecture separately, using the Packages file for that arch; for source-only CDs we should use i386 with fallback to amd64, and fail otherwise * but as long as we do not do that - for binary or binary/source CDs: prefer i386, with fallback to: 1) amd64 2) arches listed in $ARCHES - for source-only CDs: use i386, with fallback to amd64, and fail otherwise But, given the reasons I gave in [1], we could also do this a bit differently and as a bonus improve the task expansion: * if ARCHES contains a single arch OR a single arch + source, then use that arch * if ARCHES contains multiple arches or is source-only, then use i386 with a fallback to the arches listed in ARCHES In fact, I'd say the simplest answer would be to pick the first named arch. If we're doing source-only then don't run update_tasks at all. How does that sound? -- Steve McIntyre, Cambridge, UK.st...@einval.com Because heaters aren't purple! -- Catherine Pitt -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
On Wednesday 11 February 2009, Steve McIntyre wrote: In fact, I'd say the simplest answer would be to pick the first named arch. If we're doing source-only then don't run update_tasks at all. How does that sound? Bad IMO. It means: - m-a image may be different when built with i386 amd64 than when built with amd64 i386; this is even more true for i386 powerpc versus powerpc i386; IMO the order in which arches are listed should not change the resulting image - source only set will have packages in in a completely different order from the corresponding binary set -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
Moot points here, as I'm looking into going with Jonathan's latest patchset anyway. On Wed, Feb 11, 2009 at 11:29:13PM +0100, Frans Pop wrote: On Wednesday 11 February 2009, Steve McIntyre wrote: In fact, I'd say the simplest answer would be to pick the first named arch. If we're doing source-only then don't run update_tasks at all. How does that sound? Bad IMO. It means: - m-a image may be different when built with i386 amd64 than when built with amd64 i386; this is even more true for i386 powerpc versus powerpc i386; IMO the order in which arches are listed should not change the resulting image It's always likely to, though: imagine if we don't have the space for the two different-arch versions of the last package in the image. The order that we add things is likely going to affect which one is missed out. - source only set will have packages in in a completely different order from the corresponding binary set Personally, I don't care in the slightest. In fact, I'd be surprised if the tasks affect the source builds at all right now... For source-only sets, I normally expect them to just be in alphabetical order. Testing that now. -- Steve McIntyre, Cambridge, UK.st...@einval.com You can't barbecue lettuce! -- Ellie Crane -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
On Wed, Feb 11, 2009 at 10:49:06PM +, Steve McIntyre wrote: On Wed, Feb 11, 2009 at 11:29:13PM +0100, Frans Pop wrote: - source only set will have packages in in a completely different order from the corresponding binary set Personally, I don't care in the slightest. In fact, I'd be surprised if the tasks affect the source builds at all right now... For source-only sets, I normally expect them to just be in alphabetical order. Testing that now. In fact, simply looking at the lenny RC2 images you can see that sources are in alphabetical order. We may as well simply not do anything in the $(TASKDIR): target for a source-only build and save the processing time. Agreed? -- Steve McIntyre, Cambridge, UK.st...@einval.com The problem with defending the purity of the English language is that English is about as pure as a cribhouse whore. We don't just borrow words; on occasion, English has pursued other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary. -- James D. Nicoll -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
On Wednesday 11 February 2009, Steve McIntyre wrote: Bad IMO. It means: - m-a image may be different when built with i386 amd64 than when built with amd64 i386; this is even more true for i386 powerpc versus powerpc i386; IMO the order in which arches are listed should not change the resulting image It's always likely to, though: imagine if we don't have the space for the two different-arch versions of the last package in the image. The order that we add things is likely going to affect which one is missed out. That's an edge case. I'm talking about packages going missing completely because they are e.g. available for i386, but not for powerpc. Which means that if you run update_tasks based on powerpc the packages just won't be there on the early CDs. As I've mentioned before in this thread the only correct solution is to somehow run update_tasks for each arch and merge them. But that will only result in a really stable list if the merge is effectively done line-by-line (a package that is listed 5th for a task for the second arch should not end up below a package that is listed 200th for the same task for the first arch, or even worse after the packages for all tasks for the first arch). The problem is not trivially solvable and because of that I'd prefer to use a logic that is at least predictable and gives the best result for our primary architectures. -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
On Wednesday 11 February 2009, Steve McIntyre wrote: In fact, simply looking at the lenny RC2 images you can see that sources are in alphabetical order. We may as well simply not do anything in the $(TASKDIR): target for a source-only build and save the processing time. Agreed? I don't know. I'd have to check in detail why tasks get ignored (for which I'm not motivated) and IMO this is the wrong time to be making such changes anyway. You're not going to get any real testing or feedback anymore before the release. -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
On Thu, Feb 12, 2009 at 12:10:56AM +0100, Frans Pop wrote: On Wednesday 11 February 2009, Steve McIntyre wrote: Bad IMO. It means: - m-a image may be different when built with i386 amd64 than when built with amd64 i386; this is even more true for i386 powerpc versus powerpc i386; IMO the order in which arches are listed should not change the resulting image It's always likely to, though: imagine if we don't have the space for the two different-arch versions of the last package in the image. The order that we add things is likely going to affect which one is missed out. That's an edge case. I'm talking about packages going missing completely because they are e.g. available for i386, but not for powerpc. Which means that if you run update_tasks based on powerpc the packages just won't be there on the early CDs. Oh, sure. As I've mentioned before in this thread the only correct solution is to somehow run update_tasks for each arch and merge them. But that will only result in a really stable list if the merge is effectively done line-by-line (a package that is listed 5th for a task for the second arch should not end up below a package that is listed 200th for the same task for the first arch, or even worse after the packages for all tasks for the first arch). I think I already cope with that, actually: I saw this problem coming when I started doing multi-arch CDs. If you look at the code in tools/merge_package_lists you'll see how that works. -- Steve McIntyre, Cambridge, UK.st...@einval.com sladen I actually stayed in a hotel and arrived to find a post-it note stuck to the mini-bar saying Paul: This fridge and fittings are the correct way around and do not need altering -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
On Thu, Feb 12, 2009 at 12:14:02AM +0100, Frans Pop wrote: On Wednesday 11 February 2009, Steve McIntyre wrote: In fact, simply looking at the lenny RC2 images you can see that sources are in alphabetical order. We may as well simply not do anything in the $(TASKDIR): target for a source-only build and save the processing time. Agreed? I don't know. I'd have to check in detail why tasks get ignored (for which I'm not motivated) Because source-only discs do nothing with the tasks at all. and IMO this is the wrong time to be making such changes anyway. You're not going to get any real testing or feedback anymore before the release. Understood. I'm looking into this now to help fix the bugs Enrico raised, which I'm told are causing real problems for CDD users. From review of the code, I'm quite certain it's not going to affect us for our builds, but it should make a big difference for others. -- Steve McIntyre, Cambridge, UK.st...@einval.com liw everything I know about UK hotels I learned from Fawlty Towers -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
On Thursday 05 February 2009, Frans Pop wrote: On Thursday 05 February 2009, Jonathan Hall wrote: --- debian-cd/tools/update_tasks(revision 7407) +++ debian-cd/tools/update_tasks(revision 7487) Wouldn't it be much simpler to use tools/which_deb instead? That would also avoid having arch lists (which will need to be updated!) all over the place. This is not going to work as we don't only need the tasksel package, but also the Packages file for a specific arch as that is our basis for the task expansion. So, although which_deb could be used to get tasksel, it would not solve the whole problem. So the following really has to be implemented in update_tasks itself. On Thursday 05 February 2009, Frans Pop wrote: So IMO: * ideally we should run update_tasks for every architecture separately, using the Packages file for that arch; for source-only CDs we should use i386 with fallback to amd64, and fail otherwise * but as long as we do not do that - for binary or binary/source CDs: prefer i386, with fallback to: 1) amd64 2) arches listed in $ARCHES - for source-only CDs: use i386, with fallback to amd64, and fail otherwise But, given the reasons I gave in [1], we could also do this a bit differently and as a bonus improve the task expansion: * if ARCHES contains a single arch OR a single arch + source, then use that arch * if ARCHES contains multiple arches or is source-only, then use i386 with a fallback to the arches listed in ARCHES This means in most cases we'll do exactly the right thing. Except for multi-arch images, and for those we use the most generic default we have, which at least ensures consistency (amd64 i386 powerpc will give the same result as powerpc amd64 i386). [1] http://bugs.debian.org/514237#40 -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
Package: debian-cd
Version: 3.1.0-nymgy1
Severity: wishlist
Tags: patch
This patch adds two features to debian-cd, filed as a single bug/patch, per
Steve McIntyre's suggestion on the mailing list:
1) It no longer requires the existence of the i386 architecture in the
mirror to build non-i386 CD images.
2) It no longer requires that all D-I modules (udebs) exist in the 'main'
component
I have tested this patch to work when building the following combinations of
CDs from a mirror with binary-i386, binary-amd64, and source archs:
binary-i386
binary-i386 + source
binary-amd64
binary-amd64 + source
binary-i386 + binary-amd64
binary-i386 + binary-amd64 + source
source
I have also tested it to build the following CDs from a mirror with only
binary-amd64 and source archs:
binary-amd64
binary-amd64 + source
source
I have not tested with any other architectures, or CD combinations but I
beleive this patch should work with any architecture(s).
*** /home/jonhall/non-i386.patch
Index: debian-cd/tools/update_tasks
===
--- debian-cd/tools/update_tasks(revision 7407)
+++ debian-cd/tools/update_tasks(revision 7487)
@@ -1,5 +1,6 @@
#!/bin/sh
set -e
+ARCHES=alpha arm armel hppa hurd-i386 i386 ia64 mips mipsel powerpc s390
sparc amd64
if [ -z $CODENAME ]; then
echo update_tasks: codename not specified 2
@@ -149,10 +150,14 @@
}' | sort -s -n -k1 | cut -d: -f2 $file
}
-# We need to gunzip a copy of the appropriate Packages.gz file
-# Assume i386, use the $CODENAME main Packages file
+# We need to gunzip a copy of the appropriate Packages.gz file(s)
+# Find an arch that exists in our mirror...
+for arch in $ARCHES; do
+if [ -e $MIRROR/dists/$CODENAME/main/binary-$arch ]; then break; fi
+done
TMP_PKG=$TDIR/Packages
-zcat $MIRROR/dists/$CODENAME/main/binary-i386/Packages.gz $TMP_PKG
+zcat $MIRROR/dists/$CODENAME/main/binary-$arch/Packages.gz $TMP_PKG
+[ -n $LOCAL ] zcat $MIRROR/dists/$CODENAME/local/binary-$arch/Packages.gz
$TMP_PKG
# Now grab the appropriate tasksel package
TASKSEL_DEB=$MIRROR/`mawk '
Index: debian-cd/tools/make_disc_trees.pl
===
--- debian-cd/tools/make_disc_trees.pl (revision 7407)
+++ debian-cd/tools/make_disc_trees.pl (revision 7487)
@@ -735,7 +735,7 @@
}
$pdir = $dir/dists/$codename/$dist;
-if ($section eq debian-installer) {
+if ($section and $section eq debian-installer) {
$pdir = $dir/dists/$codename/$dist/debian-installer;
}
return $pdir;
Index: debian-cd/tools/generate_di_list
===
--- debian-cd/tools/generate_di_list(revision 7407)
+++ debian-cd/tools/generate_di_list(revision 7487)
@@ -24,32 +24,38 @@
EOF
my @common_excludes = read_exclude(exclude-udebs);
-
+my $mirror_path = $ENV{MIRROR}/dists/$ENV{DI_CODENAME};
+opendir COMP, $mirror_path;
+my @components = grep { -d $mirror_path/$_ and $_ !~ /^\./ } readdir COMP;
+close COMP;
foreach my $arch (@ARCHES) {
- my
$packagefile=$ENV{MIRROR}/dists/$ENV{DI_CODENAME}/main/debian-installer/binary-$arch/Packages.gz;
- unless (-f $packagefile) {
- print Missing package file for arch $arch.\n;
- next;
- }
(my $cpparch = $arch) =~ s/-/_/g;
- print OUT #ifdef ARCH_$cpparch\n;
- my @exclude = @common_excludes;
- push @exclude, read_exclude(exclude-udebs-$arch)
- if -e exclude_path(exclude-udebs-$arch);
-UDEB: foreach my $udeb (map { chomp; $_ } `zcat $packagefile | awk
'/^Package:/ {print \$2}'`) {
- foreach my $pattern (@exclude) {
- if ($udeb =~ /^$pattern$/) {
- next UDEB;
- }
+ my $output = '';
+ for my $component ( @components ) {
+ my
$packagefile=$mirror_path/$component/debian-installer/binary-$arch/Packages.gz;
+ unless ( -f $packagefile ) {
+ print Missing package file for $component/$arch.\n;
+ next;
}
- print OUT $udeb\n;
+ my @exclude = @common_excludes;
+ push @exclude, read_exclude(exclude-udebs-$arch)
+ if -e exclude_path(exclude-udebs-$arch);
+ foreach my $udeb (map { chomp; $_ } `zcat $packagefile | awk
'/^Package:/ {print \$2}'`) {
+ $output .= $udeb\n unless grep { $udeb =~ /^${_}$/ }
@exclude;
+ }
}
+ next unless $output;
+ print OUT #ifdef ARCH_$cpparch\n;
+ print OUT $output;
print OUT #endif /* ARCH_$cpparch */\n;
}
sub read_exclude {
my $file=exclude_path(shift);
- open (IN, $file) || warn failed to read exclude file $file;
+ unless ( open (IN, $file) ) {
+ warn failed to read
Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
On Thursday 05 February 2009, Jonathan Hall wrote: Index: debian-cd/tools/update_tasks === --- debian-cd/tools/update_tasks (revision 7407) +++ debian-cd/tools/update_tasks (revision 7487) @@ -1,5 +1,6 @@ #!/bin/sh set -e +ARCHES=alpha arm armel hppa hurd-i386 i386 ia64 mips mipsel powerpc s390 sparc amd64 [...] -# We need to gunzip a copy of the appropriate Packages.gz file -# Assume i386, use the $CODENAME main Packages file +# We need to gunzip a copy of the appropriate Packages.gz file(s) +# Find an arch that exists in our mirror... +for arch in $ARCHES; do +if [ -e $MIRROR/dists/$CODENAME/main/binary-$arch ]; then break; fi +done Wouldn't it be much simpler to use tools/which_deb instead? That would also avoid having arch lists (which will need to be updated!) all over the place. --- debian-cd/tools/which_deb (revision 7407) +++ debian-cd/tools/which_deb (revision 7487) Wouldn't it make much more sense for which_deb to just try the current arch(es) (the one(s) for which we're building the image) instead of trying arches randomly? Or at least to try relevant arches first? This will limit the use of the script a bit, but as it is currently only called from boot-* scripts it isn't a problem. If it is also called from update_tasks it should still not be a problem, though in that case ARCH will not yet be set and you'd have to take ARCHES. Maybe ARCH/ARCHES should be passed as a parameter instead of being hardcoded. signature.asc Description: This is a digitally signed message part.
Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
On Thursday 05 February 2009, Jonathan Hall wrote: Frans Pop wrote: Wouldn't it make much more sense for which_deb to just try the current arch(es) (the one(s) for which we're building the image) instead of trying arches randomly? Or at least to try relevant arches first? This works fine, except when building a source-only CD Then I'm back to looking through all possible arches, unless someone can think of a more reasonable thing to do in that case..? For the calls from boot-* it does not matter as those scripts won't be called for source only CDs. So we're left with the call from update_tasks. There we really have another, more fundamental issue that has to be considered. We currently use the Packages file from i386 to expand a task to a package list. Essentially this is broken because different arches can have different dependencies! In practice this problem is quite limited, but it _is_ important that we don't use one of the more obscure arches to build the list as that is *a lot* more likely to result in a broken expanded package list than when we use i386 or amd64. So IMO: * ideally we should run update_tasks for every architecture separately, using the Packages file for that arch; for source-only CDs we should use i386 with fallback to amd64, and fail otherwise * but as long as we do not do that - for binary or binary/source CDs: prefer i386, with fallback to: 1) amd64 2) arches listed in $ARCHES - for source-only CDs: use i386, with fallback to amd64, and fail otherwise IMO it's reasonable to fail if someone wants to build source only but does not have a full mirror, or at least has the most common arches. Other cases will be extremely rare, and if someone really wants to he can just modify the script. -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#514237: debian-cd: Support non-i386 mirrors, and support D-I modules in non-main
On Thursday 05 February 2009, Frans Pop wrote: We currently use the Packages file from i386 to expand a task to a s/currently use/currently always use/ package list. Essentially this is broken because different arches can have different dependencies! And also - probably more importantly - because some packages that belong to a task may not be available for all architectures. This means that for example if a task lists a powerpc- or sparc-specific package, it will currently not get included on the CD (unless it gets included for other reasons of course). This won't affect installs where a mirror is used, but can affect installs from CD/DVD only. -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
