Re: Creating Compressed Loop FS from stdin
Never mind - people are inheretedly error prone creatures. ;-) In your case shell has been passing file descriptor of the open file, not pipe, so that seeking has been working properly. Anyway, I think that your patch is useful, since it should allow using disk devices. -Maxim Peter Pentchev wrote: On Thu, Dec 30, 2004 at 04:55:43PM +0200, Peter Pentchev wrote: On Thu, Dec 30, 2004 at 04:20:16PM +0200, Maxim Sobolev wrote: You don't check return code of the second lseek - I bet it fails. This probably leads to creation of seemingly valid loop fs (i.e. with valid header), but filled with zeroes or some random junk. I said I'd tested it before posting it the first time. It works. It creates a valid loop fs, containing exactly the files that are in the input ISO image. Errr. Oops. Sorry everyone - the patch does not really work. I keep testing it with a *file* passed on mkuzip's stdin, all the while feeling surprised that lseek() works on the pipe... when there is no pipe at all :( I just tested it with a real pipe, and of course, it failed. Again, sorry for wasting your time; I guess it'd be best if I tucked in for the holidays now :( G'luck, Peter ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Creating Compressed Loop FS from stdin
You don't check return code of the second lseek - I bet it fails. This probably leads to creation of seemingly valid loop fs (i.e. with valid header), but filled with zeroes or some random junk. -Maxim Peter Pentchev wrote: On Thu, Dec 30, 2004 at 03:32:27PM +0200, Maxim Sobolev wrote: Peter Pentchev wrote: On Thu, Dec 30, 2004 at 01:28:28PM +0100, Matteo Riondato wrote: Il giorno Gio, 30-12-2004 alle 12:34 +0200, Peter Pentchev ha scritto: This could be fixed by the following patch. I'm CC'ing Maxim Sobolev, the author of mkuzip(8); Maxim, do you have any objections to this patch? Thank you for the answer and fo the patch! I hope Maxim will commit it soon. Actually, if Maxim has no objections, I could commit it myself. However, it would be totally understandable if he doesn't answer in the next day or three, what with the calendar moving ahead and all :) It will not help, since AFAIK you can't seek stdin anyway, or even if I am wrong and you can seek it to the end you will be unable to seek it backward. I tested the patch before posting it, fully expecting to find that stdin really cannot be seeked (sought? :), and surprisingly it worked, at least on RELENG_5 as of today! I've already replied to this message, but Matteo has some very strange settings of his smtp relay so that neither my original message nor my follow-up in which I had forwarded mail delivery error message got through. -Maxim Original Message ---- Subject: Re: Creating Compressed Loop FS from stdin Date: Fri, 17 Dec 2004 17:14:48 +0200 From: Maxim Sobolev <[EMAIL PROTECTED]> Organization: Porta Software Ltd To: Matteo Riondato <[EMAIL PROTECTED]> References: <[EMAIL PROTECTED]> This is not going to work by design unfortunately. cloop format has serious design flaw: it contains variable-lengh header at the beginning of the compressed image, so that before doing compression mkuzip(1) uses stat(2) call at the original file to get its size and reserve necessary space, which doesn't work with /dev/stdin as you may guess. Original GNU utility either keeps the whole compressed image in memory or uses some form of temporary storage (I don't quite remember) to work around this problem. Well, another solution would be to make mkuzip use a temporary file (obviously, keeping the whole thing in memory would be a bad idea for big ISO filesystems ;). However, as I noted above, my lseek patch works at least for RELENG_5. Can somebody test it on -CURRENT? G'luck, Peter ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Creating Compressed Loop FS from stdin
Peter Pentchev wrote: On Thu, Dec 30, 2004 at 01:28:28PM +0100, Matteo Riondato wrote: Il giorno Gio, 30-12-2004 alle 12:34 +0200, Peter Pentchev ha scritto: This could be fixed by the following patch. I'm CC'ing Maxim Sobolev, the author of mkuzip(8); Maxim, do you have any objections to this patch? Thank you for the answer and fo the patch! I hope Maxim will commit it soon. Actually, if Maxim has no objections, I could commit it myself. However, it would be totally understandable if he doesn't answer in the next day or three, what with the calendar moving ahead and all :) It will not help, since AFAIK you can't seek stdin anyway, or even if I am wrong and you can seek it to the end you will be unable to seek it backward. I've already replied to this message, but Matteo has some very strange settings of his smtp relay so that neither my original message nor my follow-up in which I had forwarded mail delivery error message got through. -Maxim Original Message ---- Subject: Re: Creating Compressed Loop FS from stdin Date: Fri, 17 Dec 2004 17:14:48 +0200 From: Maxim Sobolev <[EMAIL PROTECTED]> Organization: Porta Software Ltd To: Matteo Riondato <[EMAIL PROTECTED]> References: <[EMAIL PROTECTED]> This is not going to work by design unfortunately. cloop format has serious design flaw: it contains variable-lengh header at the beginning of the compressed image, so that before doing compression mkuzip(1) uses stat(2) call at the original file to get its size and reserve necessary space, which doesn't work with /dev/stdin as you may guess. Original GNU utility either keeps the whole compressed image in memory or uses some form of temporary storage (I don't quite remember) to work around this problem. Regards, Maxim Original Message Subject: Returned mail: see transcript for details Date: Fri, 17 Dec 2004 16:14:58 +0100 (CET) From: Mail Delivery Subsystem To: <[EMAIL PROTECTED]> The original message was received at Fri, 17 Dec 2004 16:14:55 +0100 (CET) from [192.168.1.26] - The following addresses had permanent fatal errors - <[EMAIL PROTECTED]> (reason: 550 Error: Sorry, unallowed MIME charset (too much spam)) - Transcript of session follows - ... while talking to relay.gufi.org.: >>> DATA <<< 550 Error: Sorry, unallowed MIME charset (too much spam) 554 5.0.0 Service unavailable ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Creating Compressed Loop FS from stdin
Hi folks! I think you know what FreeSBIE is and that we use compressed loop filesystems for /usr and /var directories on our LiveCD. At the moment, to create compressed filesystems, we rely on sysutils/cloop-utils, but I know /usr/bin/mkuzip is present in HEAD and in RELENG_5 and would like to switch to that. However, it seems that I cannot create valid cloopfs with mkuzip. To create the cloop image, I use: /usr/local/bin/mkisofs -lrJL $FREESBIEBASEDIR/usr | /usr/bin/mkuzip -v \ -o $FREESBIEBASEDIR/cloop/usr.cloop -s 65536 /dev/stdin (similar line for /var clooped image) Creation of the clooped images does finish without any error, but I cannot mount it: #kldload geom_uzip #mdconfig -a -t vnode usr.cloop md0 #mount_cd9660 /dev/md0.uzip /mnt mount_cd9660: /dev/md0.uzip: Input/output error Can you give me any hint? Perhaps I'm missing something in the creation of the image. I will provide further information if needed. Thank you in advance. Best Regards -- Rionda aka Matteo Riondato GUFI Staff Member (http://www.gufi.org) FreeSBIE Developer (http://www.freesbie.org) BSD-FAQ-it Main Developer (http://utenti.gufi.org/~rionda) Sent from: kaiser.sig11.org running FreeBSD-6.0-CURRENT signature.asc Description: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata
Re: Creating Compressed Loop FS from stdin
On Thu, Dec 30, 2004 at 04:55:43PM +0200, Peter Pentchev wrote: > On Thu, Dec 30, 2004 at 04:20:16PM +0200, Maxim Sobolev wrote: > > You don't check return code of the second lseek - I bet it fails. This > > probably leads to creation of seemingly valid loop fs (i.e. with valid > > header), but filled with zeroes or some random junk. > > I said I'd tested it before posting it the first time. It works. > It creates a valid loop fs, containing exactly the files that are in > the input ISO image. Errr. Oops. Sorry everyone - the patch does not really work. I keep testing it with a *file* passed on mkuzip's stdin, all the while feeling surprised that lseek() works on the pipe... when there is no pipe at all :( I just tested it with a real pipe, and of course, it failed. Again, sorry for wasting your time; I guess it'd be best if I tucked in for the holidays now :( G'luck, Peter -- Peter Pentchev [EMAIL PROTECTED][EMAIL PROTECTED][EMAIL PROTECTED] PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 The rest of this sentence is written in Thailand, on pgp9wWVT6euCu.pgp Description: PGP signature
Re: Creating Compressed Loop FS from stdin
On Thu, Dec 30, 2004 at 04:20:16PM +0200, Maxim Sobolev wrote:
> You don't check return code of the second lseek - I bet it fails. This
> probably leads to creation of seemingly valid loop fs (i.e. with valid
> header), but filled with zeroes or some random junk.
I said I'd tested it before posting it the first time. It works.
It creates a valid loop fs, containing exactly the files that are in
the input ISO image.
However, your point is valid; here's another patch which returns to the
start only if the original lseek() did not move away (or failed), and
checks the return value, too. This version of mkuzip works on today's
RELENG_5, as can be seen from the attached mkuzip.script sequence of
commands.
G'luck,
Peter
--
Peter Pentchev [EMAIL PROTECTED][EMAIL PROTECTED][EMAIL PROTECTED]
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553
Nostalgia ain't what it used to be.
Script started on Thu Dec 30 16:50:57 2004
Starting interactive C shell
[EMAIL PROTECTED] ~/fbsd/r/src/usr.bin/mkuzip]> `make -V .OBJDIR`/`make -V
PROG` -v -s 65536 -o dns-stdin.uz /dev/stdin < ~/dns.iso
data size 5505024 bytes, number of clusters 84, index lengh 680 bytes
cluster #0, in 65536 bytes, out 2263 bytes
cluster #1, in 65536 bytes, out 11581 bytes
[snip more of the same]
cluster #82, in 65536 bytes, out 11684 bytes
cluster #83, in 65536 bytes, out 752 bytes
padding data with 161 bytes so that file size is multiple of 512
compressed data to 1361920 bytes, saved 4143104 bytes, 75.26% decrease.
[EMAIL PROTECTED] ~/fbsd/r/src/usr.bin/mkuzip]> sudo mdconfig -a -f dns-stdin.uz
otp-md5 455 st1434 ext
Password:
md0
[EMAIL PROTECTED] ~/fbsd/r/src/usr.bin/mkuzip]> sudo mount_cd9660 /dev/md0.uzip
/mnt
[EMAIL PROTECTED] ~/fbsd/r/src/usr.bin/mkuzip]> sudo find /mnt | wc -l
604
[EMAIL PROTECTED] ~/fbsd/r/src/usr.bin/mkuzip]> sudo md5
/mnt/djbdns-1.05/tinydns
MD5 (/mnt/djbdns-1.05/tinydns) = d16c2fe610c19148ff005fb35242561c
[EMAIL PROTECTED] ~/fbsd/r/src/usr.bin/mkuzip]> sudo md5
/home/roam/fbsd/r/ports/dns/djbdns/work/djbdns-1.05/tinydns
MD5 (/home/roam/fbsd/r/ports/dns/djbdns/work/djbdns-1.05/tinydns) =
d16c2fe610c19148ff005fb35242561c
[EMAIL PROTECTED] ~/fbsd/r/src/usr.bin/mkuzip]> sudo umount /mnt
[EMAIL PROTECTED] ~/fbsd/r/src/usr.bin/mkuzip]> sudo mdconfig -d -u 0
[EMAIL PROTECTED] ~/fbsd/r/src/usr.bin/mkuzip]> exit
exit
Script done on Thu Dec 30 16:52:53 2004
Index: src/usr.bin/mkuzip/mkuzip.c
===
RCS file: /home/ncvs/src/usr.bin/mkuzip/mkuzip.c,v
retrieving revision 1.2
diff -u -r1.2 mkuzip.c
--- src/usr.bin/mkuzip/mkuzip.c 10 Sep 2004 23:16:05 - 1.2
+++ src/usr.bin/mkuzip/mkuzip.c 30 Dec 2004 14:47:35 -
@@ -122,10 +122,28 @@
signal(SIGXFSZ, exit);
atexit(cleanup);
- if (stat(iname, &sb) != 0) {
+ fdr = open(iname, O_RDONLY);
+ if (fdr < 0) {
+ err(1, "%s", iname);
+ /* Not reached */
+ }
+
+ /* Try seeking to the end; if that fails, fall back to fstat() */
+ memset(&sb, 0, sizeof(sb));
+ sb.st_size = lseek(fdr, 0, SEEK_END);
+ if (sb.st_size < 1) {
+ if (fstat(fdr, &sb) != 0) {
+ err(1, "%s", iname);
+ /* Not reached */
+ }
+ } else if (lseek(fdr, 0, SEEK_SET) != 0) {
err(1, "%s", iname);
/* Not reached */
}
+ if (sb.st_size < 1) {
+ errx(1, "Could not determine the size of %s", iname);
+ /* Not reached */
+ }
hdr.nblocks = sb.st_size / hdr.blksz;
if ((sb.st_size % hdr.blksz) != 0) {
if (verbose != 0)
@@ -135,11 +153,6 @@
}
toc = safe_malloc((hdr.nblocks + 1) * sizeof(*toc));
- fdr = open(iname, O_RDONLY);
- if (fdr < 0) {
- err(1, "%s", iname);
- /* Not reached */
- }
fdw = open(oname, O_WRONLY | O_TRUNC | O_CREAT,
S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
if (fdw < 0) {
pgpuWy2LWMvux.pgp
Description: PGP signature
Re: Creating Compressed Loop FS from stdin
[EMAIL PROTECTED] wrote: > > It will not help, since AFAIK you can't seek stdin anyway, or even if I > > am wrong and you can seek it to the end you will be unable to seek it > > backward. > > I tested the patch before posting it, fully expecting to find that stdin > really cannot be seeked (sought? :), and surprisingly it worked, at least > on RELENG_5 as of today! You can always seek stdin, if stdin happens to be associated with a seekable descriptor. It isn't given any special treatment simply because it has a vector of 0. That is, if you use something along the lines of: % ./seekme http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Creating Compressed Loop FS from stdin
On Thu, Dec 30, 2004 at 03:32:27PM +0200, Maxim Sobolev wrote: > Peter Pentchev wrote: > >On Thu, Dec 30, 2004 at 01:28:28PM +0100, Matteo Riondato wrote: > > > >>Il giorno Gio, 30-12-2004 alle 12:34 +0200, Peter Pentchev ha scritto: > >> > >>>This could be fixed by the following patch. I'm CC'ing Maxim Sobolev, > >>>the author of mkuzip(8); Maxim, do you have any objections to this patch? > >> > >>Thank you for the answer and fo the patch! I hope Maxim will commit it > >>soon. > > > > > >Actually, if Maxim has no objections, I could commit it myself. > >However, it would be totally understandable if he doesn't answer in > >the next day or three, what with the calendar moving ahead and all :) > > It will not help, since AFAIK you can't seek stdin anyway, or even if I > am wrong and you can seek it to the end you will be unable to seek it > backward. I tested the patch before posting it, fully expecting to find that stdin really cannot be seeked (sought? :), and surprisingly it worked, at least on RELENG_5 as of today! > I've already replied to this message, but Matteo has some very strange > settings of his smtp relay so that neither my original message nor my > follow-up in which I had forwarded mail delivery error message got through. > > -Maxim > > Original Message > Subject: Re: Creating Compressed Loop FS from stdin > Date: Fri, 17 Dec 2004 17:14:48 +0200 > From: Maxim Sobolev <[EMAIL PROTECTED]> > Organization: Porta Software Ltd > To: Matteo Riondato <[EMAIL PROTECTED]> > References: <[EMAIL PROTECTED]> > > This is not going to work by design unfortunately. cloop format has > serious design flaw: it contains variable-lengh header at the beginning > of the compressed image, so that before doing compression mkuzip(1) uses > stat(2) call at the original file to get its size and reserve necessary > space, which doesn't work with /dev/stdin as you may guess. Original GNU > utility either keeps the whole compressed image in memory or uses some > form of temporary storage (I don't quite remember) to work around this > problem. Well, another solution would be to make mkuzip use a temporary file (obviously, keeping the whole thing in memory would be a bad idea for big ISO filesystems ;). However, as I noted above, my lseek patch works at least for RELENG_5. Can somebody test it on -CURRENT? G'luck, Peter -- Peter Pentchev [EMAIL PROTECTED][EMAIL PROTECTED][EMAIL PROTECTED] PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 This inert sentence is my body, but my soul is alive, dancing in the sparks of your brain. pgpsUZMsPR7Tz.pgp Description: PGP signature
Re: Creating Compressed Loop FS from stdin
On Thu, Dec 30, 2004 at 01:28:28PM +0100, Matteo Riondato wrote: > Il giorno Gio, 30-12-2004 alle 12:34 +0200, Peter Pentchev ha scritto: > > This could be fixed by the following patch. I'm CC'ing Maxim Sobolev, > > the author of mkuzip(8); Maxim, do you have any objections to this patch? > > Thank you for the answer and fo the patch! I hope Maxim will commit it > soon. Actually, if Maxim has no objections, I could commit it myself. However, it would be totally understandable if he doesn't answer in the next day or three, what with the calendar moving ahead and all :) G'luck, Peter -- Peter Pentchev [EMAIL PROTECTED][EMAIL PROTECTED][EMAIL PROTECTED] PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553 If this sentence didn't exist, somebody would have invented it. pgpSQ6T6MKQq1.pgp Description: PGP signature
Re: Creating Compressed Loop FS from stdin
Il giorno Gio, 30-12-2004 alle 12:34 +0200, Peter Pentchev ha scritto: > This could be fixed by the following patch. I'm CC'ing Maxim Sobolev, > the author of mkuzip(8); Maxim, do you have any objections to this patch? Thank you for the answer and fo the patch! I hope Maxim will commit it soon. Best Regards -- Rionda aka Matteo Riondato GUFI Staff Member (http://www.gufi.org) FreeSBIE Developer (http://www.freesbie.org) BSD-FAQ-it Main Developer (http://utenti.gufi.org/~rionda) Sent from: kaiser.sig11.org running FreeBSD-6.0-CURRENT signature.asc Description: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata
Re: Creating Compressed Loop FS from stdin
On Wed, Dec 29, 2004 at 11:15:40PM +0100, Matteo Riondato wrote:
> Hi folks!
>
> I think you know what FreeSBIE
> is and that we use compressed loop filesystems for /usr and /var
> directories on our LiveCD.
> At the moment, to create compressed filesystems, we rely on
> sysutils/cloop-utils, but I know /usr/bin/mkuzip is present in HEAD and
> in RELENG_5 and would like to switch to that. However, it seems that I
> cannot create valid cloopfs with mkuzip.
> To create the cloop image, I use:
>
> /usr/local/bin/mkisofs -lrJL $FREESBIEBASEDIR/usr | /usr/bin/mkuzip -v \
> -o $FREESBIEBASEDIR/cloop/usr.cloop -s 65536 /dev/stdin
Well, as you can see from the very first line output by mkuzip in that
case (if -v is indeed specified), it thinks that the input file is 0 bytes
long. This is so because the first thing mkuzip does is a stat(2) on
the input file descriptor, and when you point it at /dev/stdin, stat()
returns a length of 0 bytes since it is impossible to determine beforehand
how much data will be produced by the pipe.
This could be fixed by the following patch. I'm CC'ing Maxim Sobolev,
the author of mkuzip(8); Maxim, do you have any objections to this patch?
G'luck,
Peter
Index: src/usr.bin/mkuzip/mkuzip.c
===
RCS file: /home/ncvs/src/usr.bin/mkuzip/mkuzip.c,v
retrieving revision 1.2
diff -u -r1.2 mkuzip.c
--- src/usr.bin/mkuzip/mkuzip.c 10 Sep 2004 23:16:05 - 1.2
+++ src/usr.bin/mkuzip/mkuzip.c 30 Dec 2004 10:32:38 -
@@ -122,10 +122,23 @@
signal(SIGXFSZ, exit);
atexit(cleanup);
- if (stat(iname, &sb) != 0) {
+ fdr = open(iname, O_RDONLY);
+ if (fdr < 0) {
err(1, "%s", iname);
/* Not reached */
}
+
+ /* Try seeking to the end; if that fails, fall back to fstat() */
+ sb.st_size = lseek(fdr, 0, SEEK_END);
+ if (sb.st_size < 1 && fstat(fdr, &sb) != 0) {
+ err(1, "%s", iname);
+ /* Not reached */
+ }
+ if (sb.st_size < 1) {
+ errx(1, "Could not determine the size of %s", iname);
+ /* Not reached */
+ }
+ lseek(fdr, 0, SEEK_SET);
hdr.nblocks = sb.st_size / hdr.blksz;
if ((sb.st_size % hdr.blksz) != 0) {
if (verbose != 0)
@@ -135,11 +148,6 @@
}
toc = safe_malloc((hdr.nblocks + 1) * sizeof(*toc));
- fdr = open(iname, O_RDONLY);
- if (fdr < 0) {
- err(1, "%s", iname);
- /* Not reached */
- }
fdw = open(oname, O_WRONLY | O_TRUNC | O_CREAT,
S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
if (fdw < 0) {
--
Peter Pentchev [EMAIL PROTECTED][EMAIL PROTECTED][EMAIL PROTECTED]
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint FDBA FD79 C26F 3C51 C95E DF9E ED18 B68D 1619 4553
This sentence claims to be an Epimenides paradox, but it is lying.
pgpFfxLg3fluR.pgp
Description: PGP signature
Creating Compressed Loop FS from stdin
Hi folks! I think you know what FreeSBIE is and that we use compressed loop filesystems for /usr and /var directories on our LiveCD. At the moment, to create compressed filesystems, we rely on sysutils/cloop-utils, but I know /usr/bin/mkuzip is present in HEAD and in RELENG_5 and would like to switch to that. However, it seems that I cannot create valid cloopfs with mkuzip. To create the cloop image, I use: /usr/local/bin/mkisofs -lrJL $FREESBIEBASEDIR/usr | /usr/bin/mkuzip -v \ -o $FREESBIEBASEDIR/cloop/usr.cloop -s 65536 /dev/stdin (similar line for /var clooped image) Creation of the clooped images does finish without any error, but I cannot mount it: #kldload geom_uzip #mdconfig -a -t vnode usr.cloop md0 #mount_cd9660 /dev/md0.uzip /mnt mount_cd9660: /dev/md0.uzip: Input/output error Can you give me any hint? Perhaps I'm missing something in the creation of the image. I will provide further information if needed. Thank you in advance. Best Regards -- Rionda aka Matteo Riondato GUFI Staff Member (http://www.gufi.org) FreeSBIE Developer (http://www.freesbie.org) BSD-FAQ-it Main Developer (http://utenti.gufi.org/~rionda) Sent from: kaiser.sig11.org running FreeBSD-6.0-CURRENT signature.asc Description: Questa parte del messaggio =?ISO-8859-1?Q?=E8?= firmata
