Re: debootstrap and debian-ports
Hi,
your finding that packages from both unstable and unreleased are needed
is correct (along with the complication that some may not be availabe
at any given time).
For m68k, Thosten Glaser is perhaps the most experienced in dealing
with bootstrapping an install.
Cheers,
Michael
Looking on the web for a recommended way of using debootstrap
with debian-ports, I couldn't find a definitive answer.
Here are my findings - may it help ?
Perhaps someone wiser could comment and shed some light
on the matter ?
debootstrap and debian ports
Recently (or not) the question arose on the mailing list :
http://lists.debian.org/debian-alpha/2012/06/msg7.html
http://lists.debian.org/debian-m68k/2013/08/msg00108.html
The install from debian-ports comes with two hurdles :
1/ Packages from both suite unstable and unreleased are needed
(by contrast, a normal install pulls packages from one sole suite
chosen among oldstable, stable or unstable)
2/ There are often missing, yet to build dependencies absent of the
repositories.
Let's concentrate on 1/
debian-ports hosts packages for unofficial architectures until they
are integrated in the main Debian archive. There are 3 suites :
unstable, unreleased and experimental, and in each one component : main
(one have to download non-free firmwares from the main Debian archive).
From the description on http://www.debian-ports.org :
The unreleased suite : this suite is used to upload packages
which need hacks to build until the changes are integrated into
Debian.
In an ideal world, the changes would be included in the next upstream
version
of the package. And this new version entering into unstable would
replace
the hacked one in unreleased.
In the real world, it is often necessary to keep different versions in
the
archive (think of the toolchain). This results in several versions of
one package spread among unstable and unreleased. libstdc++6 is the
most
prominent example : one for each gcc-${version} source package.
From these versions, debootstrap should install the newest one.
debootstrap and multiple versions
-
debootstrap is designed to download and parse the Packages file from
one
sole suite (the Packages file contains stanzas describing each a
version of
a package in the suite).
The names of all packages of priority required, of
priority important (unless told otherwise) and their dependencies
(unless told otherwise) are collected.
(function work_out_debs, line 18 of
/usr/share/debootstrap/scripts/unstable,
in debootstrap version 1.0.55)
For each name, the version selected for installation is the one in the
last
stanza of the first consecutive sequence of stanzas of this package
name.
(function download_debs, line 623 of /usr/share/debootstrap/functions,
function pkgdetails_perl, lines 1095, 1180 and 1186 of
/usr/share/debootstrap/functions,
in debootstrap version 1.0.55)
This amounts to choose the newest version iff the Packages file is
sorted,
i.e. all stanzas of a given package name are grouped together, and
this group is
sorted by ascending version number.
Some order in Packages files
The program mini-dak at work at debian-ports currently assembles a
Packages file with Architecture: all packages first,
and then the ones of the given architecture.
apt-sortpkgs (of package apt-utils) sorts separately each
of these groups. However, it is a mere lexical sort on package name
and then on version string to break tie. Epochs and version comparison
rules are not taken into account.
(line 50 of cmdline/apt-sortpkgs.cc
line 604 of apt-pkg/contrib/strutl.cc
in apt version 0.9.14.1)
(line 39 of archive-reindex-meta in mini-dak)
Some packages (e.g. sensible-utils 0.0.0 and 0.0.9 on m68k, iproute
20120521-3 and
1:3.12.0-1 everywhere) have moved from one architecture to
Architecture: all
in newer versions (or conversely) - both versions are present in the
archive...
(debootstrap --no-resolve-deps actually ends up in a fatal error :
sensible-utils is counted twice when collecting the names, once when
selecting the version to install. Is it a bug in debootstrap ?
This isn't a common use case though)
A solution (?)
--
I would suggest to modify debootstrap to download Packages files from
both unstable
and unreleased, to concatenate them and to sort by package name and by
ascending version number.
Some previous (unsuccesful) thoughts
-
See also http://bugs.debian.org/690210
- Give precedence to packages in unreleased (or in unstable either) :
this is
wrong if a newer version exists in the other suite. It depends on the
history
of package changes (e.g. libstdc++6 on alpha and m68k).
- Replace selected versions from unstable by newer versions from
unreleased
if available : it doesn't address the architecture - all problem.
- Concatenate Packages files from unstable and unreleased
Re: debootstrap and debian-ports
Michael Schmitz dixit: your finding that packages from both unstable and unreleased are needed is correct (along with the complication that some may not be availabe at any given time). There’s another problem: even in the main Debian archive, “unstable” is *not* guaranteed to be debootstrap’able, and has regularily been broken. Good news for m68k though: eglibc, gcc-4.8 and linux are no longer in “unreleased”. In fact: tg@freewrt:~ $ u=/var/lib/apt/lists/ftp.de.debian.org_debian-ports_dists_unreleased_main_binary-m68k_Packages tg@freewrt:~ $ # test idempotency tg@freewrt:~ $ grep-dctrl -r -P . $u | diff -u - $u | wc 0 0 0 tg@freewrt:~ $ # get me all source packages that have packages in unreleased/m68k tg@freewrt:~ $ grep-dctrl -r -P . -n -s Source:Package $u | sort -u atari-bootstrap atari-fdisk gcc-4.6 gcj-4.6 glib-networking gnat-4.6 google-gadgets libbluray m68k-vme-tftplilo m68kboot mesa mysql-5.1 vmelilo webkit We can group them by: • architecture-specific packages atari-bootstrap atari-fdisk m68k-vme-tftplilo m68kboot vmelilo • architecture-specific patches, packages going away in sid soon anyway gcc-4.6 gcj-4.6 gnat-4.6 mysql-5.1 (actually already gone) • maintainer refuses integrating our patches libbluray (maybe ping again?) mesa(refusal also upstream) • patches need to be updated against current versions of the packages google-gadgets (waits on webkit/gtk) webkit • “Build without libproxy, for bootstrapping.” glib-networking None of them is, however, strictly needed for debootstrap (although the architecture-specific packages may be needed when d-i’ing a system). I read somewhere that Aurélien regularily creates snapshots of debian-ports – which means that we can install m68k from these, Right Now™. deb http://ftp.debian-ports.org/debian-snapshot/2013-12-12/ unstable main This should work. Maybe Aurélien can “freeze” one of these, if needed? -- Back to debootstrap. Yes, it needs support for multiple versions (already has some, atm) and the unreleased distribution right now. I guess APT’s ordering (from a given package, always use the dpkg-numerically largest version, ignoring all dpkg-numerically smaller versions, period) would work for now, as we don’t have the arch:all/arch:any mix in the minbase, base or buildd set much (except libsemanage-common). Everything else needs a very complicated solver (such as, use an older libsemanage-common that works with the libsemanage1 version in the archive) and is out of scope for the sh-based debootstrap. bye, //mirabilos (short, caught the flu) -- mirabilos│ untested Natureshadow │ tut natürlich Natureshadow │ was auch sonst ... mirabilos│ fijn ☺ -- To UNSUBSCRIBE, email to debian-alpha-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/pine.bsm.4.64l.1312181709050.19...@herc.mirbsd.org
Re: debootstrap and debian-ports
jhcha54008 dixit: Custom mini-repositories for installation - One may download the missing packages from http://snapshot.debian.org/archive/debian-ports. Indeed, but – as I said – the regular debian-ports archive is also weekly snapshotted, and Aurélien can persist them TTBOMK (like etch-m68k was). I’ve got a manually created mini-repository for m68k, but experience shows acceptance of these is questionable even if done by a DD, *and* you need custom archive keys, so I think it’s best to stick to “more official” ways if these contain all needed packages in unstable (or debootstrap’s patched to handle unreleased). bye, //mirabilos -- Using Lynx is like wearing a really good pair of shades: cuts out the glare and harmful UV (ultra-vanity), and you feel so-o-o COOL. -- Henry Nelson, March 1999 -- To UNSUBSCRIBE, email to debian-alpha-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/pine.bsm.4.64l.1312182157360.19...@herc.mirbsd.org
Re: debootstrap and debian-ports
Thank you for your answer Le mercredi 18 décembre à 17h 26mn 40s (+), Thorsten Glaser a écrit : Michael Schmitz dixit: your finding that packages from both unstable and unreleased are needed is correct (along with the complication that some may not be availabe at any given time). There’s another problem: even in the main Debian archive, “unstable” is *not* guaranteed to be debootstrap’able, and has regularily been broken. Good news for m68k though: eglibc, gcc-4.8 and linux are no longer in “unreleased”. In fact: tg@freewrt:~ $ u=/var/lib/apt/lists/ftp.de.debian.org_debian-ports_dists_unreleased_main_binary-m68k_Packages tg@freewrt:~ $ # test idempotency tg@freewrt:~ $ grep-dctrl -r -P . $u | diff -u - $u | wc 0 0 0 tg@freewrt:~ $ # get me all source packages that have packages in unreleased/m68k tg@freewrt:~ $ grep-dctrl -r -P . -n -s Source:Package $u | sort -u atari-bootstrap atari-fdisk gcc-4.6 gcj-4.6 glib-networking gnat-4.6 google-gadgets libbluray m68k-vme-tftplilo m68kboot mesa mysql-5.1 vmelilo webkit I am reading after my last post, and I realize that I tryied hard to implement something already available in grep-dctrl... We can group them by: • architecture-specific packages atari-bootstrap atari-fdisk m68k-vme-tftplilo m68kboot vmelilo • architecture-specific patches, packages going away in sid soon anyway gcc-4.6 gcj-4.6 gnat-4.6 mysql-5.1 (actually already gone) • maintainer refuses integrating our patches libbluray (maybe ping again?) mesa(refusal also upstream) • patches need to be updated against current versions of the packages google-gadgets (waits on webkit/gtk) webkit • “Build without libproxy, for bootstrapping.” glib-networking Nice work indeed (same thing for debian-alpha). It is now possible (checked mid-november) to install from m68k on debian-ports a lxc container with qemu user mode emulation thanks to Laurent Vivier's qemu-m68k (https://gitorious.org/qemu-m68k/qemu-m68k) None of them is, however, strictly needed for debootstrap (although the architecture-specific packages may be needed when d-i’ing a system). I read somewhere that Aurélien regularily creates snapshots of debian-ports – which means that we can install m68k from these, Right Now™. deb http://ftp.debian-ports.org/debian-snapshot/2013-12-12/ unstable main This should work. Maybe Aurélien can “freeze” one of these, if needed? I am not sure : are these snapshots apt-gettable ? Due to the way they are taken (cp -al, lines 37, 38, 39 of archive-snapshot in mini-dak, and archive-reindex-meta line 87 ), every package or metadata is kept, but the Release files which seem to be hard links towards the current ones (Is it correct ?) -- Back to debootstrap. Yes, it needs support for multiple versions (already has some, atm) and the unreleased distribution right now. I guess APT’s ordering (from a given package, always use the dpkg-numerically largest version, ignoring all dpkg-numerically smaller versions, period) would work for now, as we don’t have the arch:all/arch:any mix in the minbase, base or buildd set much (except libsemanage-common). Everything else needs a very complicated solver (such as, use an older libsemanage-common that works with the libsemanage1 version in the archive) and is out of scope for the sh-based debootstrap. bye, //mirabilos (short, caught the flu) -- mirabilos│ untested Natureshadow │ tut natürlich Natureshadow │ was auch sonst ... mirabilos│ fijn ☺ Gute Besserung ! J.-H. Chatenet -- To UNSUBSCRIBE, email to debian-alpha-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131219005316.GA1928@debirf-dominicain
typo (was Re: debootstrap and debian-ports)
Sorry, debian-ports archive key is http://ftp.debian-ports.org/archive/archive_2013.key and debian snapshots are http://snapshot.debian.org/archive/debian-ports/ (note the final /) J.-H. Chatenet -- To UNSUBSCRIBE, email to debian-alpha-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131219010207.GA1958@debirf-dominicain
