Bug#652946: busybox tar fails to copy filesystem.squashfs
On Tue, Jul 10, 2012 at 06:38:57AM +0100, Rui Bernardo wrote: I was going to prepare another patch but I hold myself this time... I've searched what was creating those directories in the /target/. Turns out to be /lib/partman/finish.d/20mount_partitions from partman-target [1]. That code isn't touched since 2007 and creates the var/run and var/lock directories in /target. I meant that the relevant file was not changed since 2007. I'm not sure if /var/run and /var/lock still need to be created in /target since debian policy changed for those directories. As Michael pointed out, they are links now. Reading the debian wiki about /var/run transition [2] and debian policy 9.1.1.7 [3] I think debian-installer partman-target could only create /run now. It shouldn't be a problem if initscripts and initramfs-tools are ready to use /run instead of /var/run. [1] http://anonscm.debian.org/gitweb/?p=d-i/partman-target.git [2] http://wiki.debian.org/ReleaseGoals/RunDirectory [3] http://www.debian.org/doc/debian-policy/ch-opersys.html -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#652946: busybox tar fails to copy filesystem.squashfs
Today I've gave another try and this is still happening with today's sid
build.
Following your hint that this is busybox related I've tried to find how
far back this issue was happening using a patched live-build to use
snapshots.debian.org. From the available daily installers this bug is
present in all installers in http://d-i.debian.org/daily-images/i386.
The only date I didn't confirm this bug was 20110509 because the
installer stopped in the keyboard step. So I can assume this bug is
present at least since 20110930.
I've tried to enable/disable config options in busybox udeb that were
enabled since 20110930 (the last build I could reproduce this).
Looking at
http://anonscm.debian.org/gitweb/?p=d-i/busybox.git;a=history;f=debian/config/pkg;hb=HEAD
I've tried to disable in busybox udeb CONFIG_MKTEMP and
CONFIG_SWITCH_ROOT, just because their names are suspicious. Didn't
work. The installer stopped in the keyboard step without any of them.
I can't check all the busybox udeb changes to trace this, it takes too
much time to do it without knowing why the changes were made.
Another factor is busybox upstream version change since squeeze. It went
from 1.17 to actual 1.20. So, this can also be an upstream problem or
something that changed upstream. I don't know busybox enough to trace
this in upstream versions in debian-installer.
But, while I was messing with busybox udeb, I've tried to enable
CONFIG_FEATURE_TAR_FROM and tried to exclude var/lock and var/run in
busybox tar (not the target's tar as in my previous patches) and all
worked ok.
In busybox:
@@ -152,7 +152,7 @@ CONFIG_GZIP_FAST=0
CONFIG_TAR=y
CONFIG_FEATURE_TAR_CREATE=y
# CONFIG_FEATURE_TAR_AUTODETECT is not set
-# CONFIG_FEATURE_TAR_FROM is not set
+CONFIG_FEATURE_TAR_FROM=y
# CONFIG_FEATURE_TAR_OLDGNU_COMPATIBILITY is not set
# CONFIG_FEATURE_TAR_OLDSUN_COMPATIBILITY is not set
CONFIG_FEATURE_TAR_GNU_EXTENSIONS=y
In live-installer:
@@ -42,9 +42,13 @@ install_live_system () {
COUNT=0
OLD_IFS=$IFS
+
+ echo 'var/lock' /tmp/excludes
+ echo 'var/run' /tmp/excludes
+
mkdir -p /target
exec 40
- tar c . | \
+ tar c . -X /tmp/excludes | \
(chdir /target tar xv) | \
(
while read line; do
@@ -60,6 +64,8 @@ install_live_system () {
exec 04
IFS=$OLD_IFS
+ rm /tmp/excludes
+
chdir /
eval ${SUPPORT}_teardown
done
It worked, without using target tar and without chroot.
busybox udeb must be changed or the tar command will fail during the
install.
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#652946: busybox tar fails to copy filesystem.squashfs
Um. This is just insane. The bug is filed 22-Dec 2011, today is 09-Jul 2012, you have the actual cause, and it is still open, and the maintainers of suspected package (busybox) knows nothing about it... :( Okay. Rui, you found the root cause of this issue, it is there: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=652946#15 The problem is that during extraction, target directory var/lock exists, and the installer tries to replace it with something else (I guess a symlink). Busybox tar does not try to remove directories in this case, to be replaced with files. So it stops there, treating it as fatal error, and does not extract remaining files, creating issues for the system running later. Whenever it is a bug in busybox tar or not is an open question. POSIX does not specify whenever it should do that or not. GNU tar tries to rmdir() the existing file if unlink()ing it returned EISDIR, and will fail only if the directory is non-empty. Ofcourse this rmdir can be added to busybox tar too, it is rather trivial. But this is not the point. The point is the presence of var/lock directory in the target system. It should not be there to start with. We're extracting a whole set of directories, and any existing directory in the target which clashes with a file in the archive we're extracting may break the extraction with any tar implementation, not just busybox one. Ie, always extract to empty target directory. Now, I've no idea how this live system works, so I can't say more. Besides, I verified - busybox tar will correctly return failure (and write accutate reason to stderr) in this case. So something else in the live installer should be broken too -- the part which loses error messages and non-zero status codes. So, to sum it all: it is a bug in the live installer, it a) calls tar incorrectly (losing exit code and stderr) and b) does not verify that the target directory is empty or at least has sane content. Thanks, /mjt -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#652946: busybox tar fails to copy filesystem.squashfs
Hi Michael,
On Mon, Jul 09, 2012 at 11:11:24PM +0400, Michael Tokarev wrote:
Um. This is just insane. The bug is filed 22-Dec 2011,
today is 09-Jul 2012, you have the actual cause, and it
is still open, and the maintainers of suspected package
(busybox) knows nothing about it... :(
I think we didn't know for sure that busybox was the to blame. Daniel
said the error is in busybox, not a busybox error.
Because this bug is hard to trace (got to build and rebuild several live
images), lack of time, etc, time passed... Anyway, as you pointed out,
it's not busybox. Thank you.
Okay.
Rui, you found the root cause of this issue, it is there:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=652946#15
The problem is that during extraction, target directory
var/lock exists, and the installer tries to replace it
with something else (I guess a symlink).
Busybox tar does not try to remove directories in this
case, to be replaced with files. So it stops there,
treating it as fatal error, and does not extract remaining
files, creating issues for the system running later.
Whenever it is a bug in busybox tar or not is an open
question. POSIX does not specify whenever it should do
that or not. GNU tar tries to rmdir() the existing
file if unlink()ing it returned EISDIR, and will fail
only if the directory is non-empty. Ofcourse this rmdir
can be added to busybox tar too, it is rather trivial.
But this is not the point.
The point is the presence of var/lock directory in the
target system. It should not be there to start with.
We're extracting a whole set of directories, and any
existing directory in the target which clashes with
a file in the archive we're extracting may break the
extraction with any tar implementation, not just
busybox one. Ie, always extract to empty target
directory.
Thanks again for clearing the fog here :)
Following your pointers I've put a sleep 2000 in
live-installer.postinst just _before_ the line mkdir -p /target and
then I've open tty2 console in debian-installer to check what was in
/target (see below).
Now, I've no idea how this live system works, so I can't
say more.
Besides, I verified - busybox tar will correctly return
failure (and write accutate reason to stderr) in this
case. So something else in the live installer should be
broken too -- the part which loses error messages and
non-zero status codes.
I think the problem about redirecting the error messages is because
live-installer.postint extracts the filesystem in background with exec
4. The relevant part of postinst:
eval ${SUPPORT}_prepare
STEPS=$(eval ${SUPPORT}_count)
db_progress INFO live-installer/progress/copying
COUNT=0
OLD_IFS=$IFS
mkdir -p /target
exec 40
tar c . | \
(chdir /target tar xv) | \
(
while read line; do
COUNT=$(($COUNT + 1))
CURRENT=$(($COUNT * 100 / $STEPS))
[ x$CURRENT = x$LAST_UPDATE ] continue
LAST_UPDATE=$CURRENT
db_progress STEP 1 4
done
)
exec 04
I don't know if adding a 2 or a |tee -a would work. It's a question
of trying it. It would really help in the future.
So, to sum it all: it is a bug in the live installer,
it a) calls tar incorrectly (losing exit code and stderr)
and b) does not verify that the target directory is empty
or at least has sane content.
Thanks for this, I really thought it was something about links or
hardlinks in the squash filesystem all the time. Turns out that is
/target that is not empty when the extraction happens!
As I've said above, I've put a sleep 2000 before mkdir -p /target
and SURPRISE (for me at least): /target, which I thought should be empty
because it is apparently created after partman has formated the
partitions (so I assumed completely empty), is _not_ empty:
root@quartor:~# tar tf target.tar.gz
./
./etc/
./etc/fstab
./lost+found/
./media/
./media/cdrom0/
./media/cdrom
./var/
./var/lock/
./var/run/
I've attached the tar file with what I've found in /target/ before the
extraction began.
I presume this must be something that line eval ${SUPPORT}_prepare
does.
Thanks,
/mjt
Thank you very much!! Debian is just great :)
target.tar.gz
Description: Binary data
Bug#652946: busybox tar fails to copy filesystem.squashfs
I was going to prepare another patch but I hold myself this time... I've searched what was creating those directories in the /target/. Turns out to be /lib/partman/finish.d/20mount_partitions from partman-target [1]. That code isn't touched since 2007 and creates the var/run and var/lock directories in /target. I'm not sure if /var/run and /var/lock still need to be created in /target since debian policy changed for those directories. As Michael pointed out, they are links now. If the directories still need to exist for another package other than live-installer or for anything else, then we should patch live-installer to rmdir /target/var/run /target/var/lock before extracting the squashfs. If not, maybe it should be partman-target that should stop creating those directories. What do you guys think? [1] http://anonscm.debian.org/gitweb/?p=d-i/partman-target.git -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
