troubles to extract bzip2 compressed tar file
Hello, I have a server archive created with tar and bzip2 (via -j tar option) on OpenBSD 5.3 GENERIC#50 i386 Now, I'm trying to extract it on a new machine, but I'm getting error: cd /server doas tar xjf /path/whole_server_archive-20151101.tar.bz2 bzip2: I/O or other error, bailing out. Possible reason follows. bzip2: Broken pipe Input file = (stdin), output file = (stdout) Abort trap (core dumped) doas find /server | wc -l 377293 tar tjf /path/whole_server_archive-20151101.tar.bz2 | wc -l 378175 tar file is 5,2G big, there is space and inodes available in /server I'm following current on amd64 Working file: src/bin/pax/tar.c head: 1.58 /usr/local/bin/bzip2 --help bzip2, a block-sorting file compressor. Version 1.0.6, 6-Sept-2010. tar xjvf ../whole_server_archive-20151101.tar.bz2 : ./var/dovecot/dovecot.conf ./var/dovecot/login ./var/dovecot/login/ssl-parameters.dat ./var/dovecot/master.pid bzip2: I/O or other error, bailing out. Possible reason follows. Abort trap (core dumped) bzip2: Broken pipe Input file = (stdin), output file = (stdout) no issue with -t parameter tar tjf ../whole_server_archive-20151101.tar.bz2 : ./var/nsd/dev ./var/nsd/run ./var/nsd/zones ./boot ./bsd ./bsd.old ./bsd.rd ./bsd.new ./obsd ./obsd.rd I'm able to restore this tar on original machine with OpenBSD 5.3 GENERIC#50 i386 without problem. Is this something worth to debug? Best regards, Jiri
Re: bzip2
On Thu, Jun 6, 2013 at 11:30 PM, Theo de Raadt dera...@cvs.openbsd.org wrote: [re Has anyone looked at zopfli] If we did add it, it would only benefit the fast architectures, since the others cannot afford the additional build time. Developers would use up the space gains quickly. Right now a few architectures are neck and neck regarding which install media are close to full. Older architectures would hit full install media issues first. A smaller contingent of developers who take care of those architectures would have to deal with the fallout, creating further friction... For comparison, on the slowest machine I currently have access to (a 500MHz ALIX) zopfli on the fastest setting (--i1) is 14.2 times slower than gzip -9 (192 seconds vs 13.5) for the same kernel and produces output that's 35702 bytes smaller. Anyway, I'm not the one who has to deal with this either way so I'll leave it there. -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69 Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement.
Re: bzip2
If it covers that tar needs for tar xjf bla.tbz2 to work then this would be a good addition. 2013/6/6 Ted Unangst t...@tedunangst.com Something that comes up from time to time is the question of whether to import bzip2 into base or not. Turns out the question is moot because already have imported it. There's a copy in perl. (I didn't know this until I happened to be watching a build closer than usual.) Since we already have the code built, why not let people use it? This is a small perl script that implements the 90% subset of functionality people expect from the regular bzip2 and bunzip2 utilities. It's not really complete or perfect, but I haven't spent all that much time on it. --- /dev/null Wed Jun 5 20:54:56 2013 +++ bzip2/Makefile Wed Jun 5 20:51:51 2013 @@ -0,0 +1,15 @@ +# $OpenBSD$ + +.include bsd.own.mk + +MAN= + +SCRIPT=bzip2.pl +LINKS=${BINDIR}/bzip2 ${BINDIR}/bunzip2 + +realinstall: + ${INSTALL} ${INSTALL_COPY} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \ + ${.CURDIR}/${SCRIPT} ${DESTDIR}${BINDIR}/bzip2 + + +.include bsd.prog.mk --- /dev/null Wed Jun 5 20:55:02 2013 +++ bzip2/bzip2.pl Wed Jun 5 20:53:23 2013 @@ -0,0 +1,113 @@ +#!/usr/bin/perl -w +# $OpenBSD$ +# Copyright (c) Ted Unangst t...@openbsd.org +# +# Permission to use, copy, modify, and distribute this software for any +# purpose with or without fee is hereby granted, provided that the above +# copyright notice and this permission notice appear in all copies. +# +# THE SOFTWARE IS PROVIDED AS IS AND THE AUTHOR DISCLAIMS ALL WARRANTIES +# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +use strict; +use Getopt::Long qw(:config posix_default bundling no_ignore_case); +use IO::Compress::Bzip2 qw(bzip2); + + +my $usestdout; +my $level = 9; +my $reqlevel; # requested level, who cares? +GetOptions( + c|stdout = \$usestdout, + 1|2|3|4|5|6|7|8|9|fast|best = \$reqlevel +); + +sub bzipfile { + my $fname = shift; + my $ofname; + my $in; + if ($fname ne -) { + if ($usestdout) { + $ofname = -; + } else { + $ofname = $fname . .bz2; + if (-e $ofname) { + warn Output file $ofname already exists.\n; + return; + } + } + $in = new IO::File $fname; + if (!$in) { + warn Can't open input file $fname.\n; + return; + } + } else { + $in = -; + $ofname = -; + } + if ($ofname eq - -t STDOUT) { + warn I won't write compressed data to a terminal.\n; + return; + } + if (not bzip2 $in = $ofname, BlockSize100K = $level) { + warn failz; + return; + } + if ($ofname ne -) { + unlink($fname); + } +} + +sub bunzipfile { +print bunzipping\n; + my $fname = shift; + my $ofname; + my $in; + if ($fname ne -) { + if ($usestdout) { + $ofname = -; + } else { + $ofname = $fname; + $ofname =~ s/\.bz2$//; + if (-e $ofname) { + warn Output file $ofname already exists.\n; + return; + } + } + $in = new IO::File $fname; + if (!$in) { + warn Can't open input file $fname.\n; + return; + } + } else { + $in = -; + $ofname = -; + } + if (not bunzip2 $in = $ofname) { + warn failz; + return; + } + if ($ofname ne -) { + unlink($fname); + } +} + +my $main; +if ($0 =~ /bunzip2$/) { + $main = \bunzipfile; +} else { + $main = \bzipfile; +} + +if (@ARGV == 0) { + $main(-); +} else { + foreach my $f (@ARGV) { + $main($f); + } +} -- May the most significant bit of your life be positive.
Re: bzip2
On 2013/06/06 08:04, Janne Johansson wrote: If it covers that tar needs for tar xjf bla.tbz2 to work then this would be a good addition. That needs support for -d (and -c but we have that already).
Re: bzip2
Date: Wed, 05 Jun 2013 20:59:39 -0400 From: Ted Unangst t...@tedunangst.com Something that comes up from time to time is the question of whether to import bzip2 into base or not. Turns out the question is moot because already have imported it. There's a copy in perl. (I didn't know this until I happened to be watching a build closer than usual.) Since we already have the code built, why not let people use it? This is a small perl script that implements the 90% subset of functionality people expect from the regular bzip2 and bunzip2 utilities. It's not really complete or perfect, but I haven't spent all that much time on it. full disclosure I'm a Perl hater /full disclosure I've ranted before about implementing standard tools in Perl. The user experience just isn't the same as with C code. But even more so than with nl(1), why would we want to use something that's different from what everybody else uses? If we want bzip2 in base (and I think there are good reasons for having it) we should simply use the standard bzip2 code. --- /dev/null Wed Jun 5 20:54:56 2013 +++ bzip2/MakefileWed Jun 5 20:51:51 2013 @@ -0,0 +1,15 @@ +# $OpenBSD$ + +.include bsd.own.mk + +MAN= + +SCRIPT=bzip2.pl +LINKS=${BINDIR}/bzip2 ${BINDIR}/bunzip2 + +realinstall: + ${INSTALL} ${INSTALL_COPY} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \ + ${.CURDIR}/${SCRIPT} ${DESTDIR}${BINDIR}/bzip2 + + +.include bsd.prog.mk --- /dev/null Wed Jun 5 20:55:02 2013 +++ bzip2/bzip2.plWed Jun 5 20:53:23 2013 @@ -0,0 +1,113 @@ +#!/usr/bin/perl -w +# $OpenBSD$ +# Copyright (c) Ted Unangst t...@openbsd.org +# +# Permission to use, copy, modify, and distribute this software for any +# purpose with or without fee is hereby granted, provided that the above +# copyright notice and this permission notice appear in all copies. +# +# THE SOFTWARE IS PROVIDED AS IS AND THE AUTHOR DISCLAIMS ALL WARRANTIES +# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +use strict; +use Getopt::Long qw(:config posix_default bundling no_ignore_case); +use IO::Compress::Bzip2 qw(bzip2); + + +my $usestdout; +my $level = 9; +my $reqlevel; # requested level, who cares? +GetOptions( + c|stdout = \$usestdout, + 1|2|3|4|5|6|7|8|9|fast|best = \$reqlevel +); + +sub bzipfile { + my $fname = shift; + my $ofname; + my $in; + if ($fname ne -) { + if ($usestdout) { + $ofname = -; + } else { + $ofname = $fname . .bz2; + if (-e $ofname) { + warn Output file $ofname already exists.\n; + return; + } + } + $in = new IO::File $fname; + if (!$in) { + warn Can't open input file $fname.\n; + return; + } + } else { + $in = -; + $ofname = -; + } + if ($ofname eq - -t STDOUT) { + warn I won't write compressed data to a terminal.\n; + return; + } + if (not bzip2 $in = $ofname, BlockSize100K = $level) { + warn failz; + return; + } + if ($ofname ne -) { + unlink($fname); + } +} + +sub bunzipfile { +print bunzipping\n; + my $fname = shift; + my $ofname; + my $in; + if ($fname ne -) { + if ($usestdout) { + $ofname = -; + } else { + $ofname = $fname; + $ofname =~ s/\.bz2$//; + if (-e $ofname) { + warn Output file $ofname already exists.\n; + return; + } + } + $in = new IO::File $fname; + if (!$in) { + warn Can't open input file $fname.\n; + return; + } + } else { + $in = -; + $ofname = -; + } + if (not bunzip2 $in = $ofname) { + warn failz; + return; + } + if ($ofname ne -) { + unlink($fname); + } +} + +my $main; +if ($0 =~ /bunzip2$/) { + $main = \bunzipfile; +} else { + $main = \bzipfile; +} + +if (@ARGV == 0) { + $main(-); +} else { + foreach my $f (@ARGV) { + $main($f); + } +}
Re: bzip2
On Thu, Jun 6, 2013 at 2:20 PM, Mark Kettenis mark.kette...@xs4all.nl wrote: full disclosure I'm a Perl hater /full disclosure I've ranted before about implementing standard tools in Perl. The user experience just isn't the same as with C code. But even more so than with nl(1), why would we want to use something that's different from what everybody else uses? If we want bzip2 in base (and I think there are good reasons for having it) we should simply use the standard bzip2 code. Seconded. ciao, David
Re: bzip2
On Thu, 6 Jun 2013, David Coppa wrote: But even more so than with nl(1), why would we want to use something that's different from what everybody else uses? If we want bzip2 in base (and I think there are good reasons for having it) we should simply use the standard bzip2 code. Seconded. Thirded
Re: bzip2
On Thu, Jun 06, 2013 at 14:20, Mark Kettenis wrote: I've ranted before about implementing standard tools in Perl. The user experience just isn't the same as with C code. But even more so than with nl(1), why would we want to use something that's different from what everybody else uses? If we want bzip2 in base (and I think there are good reasons for having it) we should simply use the standard bzip2 code. I don't have a problem with importing bzip2, per se. But iirc previous discussions basically ended with it adds more code and will slow down builds. But we've already been slowing down builds for the past two years. Adding another copy of the C version returns us to the bloat discussion. I'm trying to dance around that objection by using code that already has been imported and built. It's not so much that I really need bzip2 in base. But having paid the cost to build, I'd like some return on that investment. At least that's my rationale.
Re: bzip2
On Thu, Jun 06, 2013 at 14:20, Mark Kettenis wrote: I've ranted before about implementing standard tools in Perl. The user experience just isn't the same as with C code. But even more so than with nl(1), why would we want to use something that's different from what everybody else uses? If we want bzip2 in base (and I think there are good reasons for having it) we should simply use the standard bzip2 code. I don't have a problem with importing bzip2, per se. But iirc previous discussions basically ended with it adds more code and will slow down builds. If I recall, previous discussions were not about but then we can use it as a complete replacement for gzip or use it for the install media or ..., as if this is a general replacement algorithm. But we've already been slowing down builds for the past two years. Adding another copy of the C version returns us to the bloat discussion. I'm trying to dance around that objection by using code that already has been imported and built. If it is now very common, we might as well put it in base properly. It's not so much that I really need bzip2 in base. But having paid the cost to build, I'd like some return on that investment. At least that's my rationale. If the code built in base will just be picked up by perl, fine. It is primarily used by ports. If anyone thinks using this for the install or boot media is going to help, don't say a word until you can prove it on all platforms.
Re: bzip2
Theo de Raadt dera...@cvs.openbsd.org wrote: It is primarily used by ports. Before t2k13, there were 739 .tar.bz2|.tbz2 distfiles--compared to 268 .tar.xz|.txz ones. I don't know how fast that balance is shifting. There are some 40 ports that directly depend on libbz2, and some 10 more that pull it in indirectly by way of libarchive or such. -- Christian naddy Weisgerber na...@mips.inka.de
Re: bzip2
Date: Thu, 6 Jun 2013 09:48:24 -0600 (MDT) From: Theo de Raadt dera...@cvs.openbsd.org On Thu, Jun 06, 2013 at 14:20, Mark Kettenis wrote: I've ranted before about implementing standard tools in Perl. The user experience just isn't the same as with C code. But even more so than with nl(1), why would we want to use something that's different from what everybody else uses? If we want bzip2 in base (and I think there are good reasons for having it) we should simply use the standard bzip2 code. I don't have a problem with importing bzip2, per se. But iirc previous discussions basically ended with it adds more code and will slow down builds. If I recall, previous discussions were not about but then we can use it as a complete replacement for gzip or use it for the install media or ..., as if this is a general replacement algorithm. Somebody said /usr/bin is full ;). But we've already been slowing down builds for the past two years. Adding another copy of the C version returns us to the bloat discussion. I'm trying to dance around that objection by using code that already has been imported and built. If it is now very common, we might as well put it in base properly. These days I see more .tar.bz2 than .tar.gz. However, I think .bz2 usage is declining in favour of .xz. It is primarily used by ports. Dunno about others, but I download lots of source code to take a look at it, and it usually comes as a .tar.bz2 these days. Installing bzip2 is typically one of the first things I do on a machine.
Re: bzip2
On Thu, Jun 6, 2013 at 11:48 AM, Theo de Raadt dera...@cvs.openbsd.org wrote: [...] If anyone thinks using this for the install or boot media is going to help, don't say a word until you can prove it on all platforms. Has anyone looked at zopfli[1] for the install media? It's a (apache 2 licensed) slightly better but much slower gzip compressor that still produces gzip-compatible output. For an amd64 ramdiskA it makes a bsd.gz that's 48k smaller than gzip (in ~16 seconds) and should compatible with the bootloader. I tried it once in a vm and it booted ok. $ time gzip -9cn bsd.strip bsd.gz real0m0.648s user0m0.610s sys 0m0.000s $ time zopfli -c bsd.strip bsd.zopfli.gz real0m17.409s user0m16.810s sys 0m0.580s $ ls -l bsd.gz bsd.zopfli.gz -rw-r--r-- 1 dtucker wsrc 1349508 Jun 6 15:40 bsd.gz -rw-r--r-- 1 dtucker wsrc 1310302 Jun 6 15:42 bsd.zopfli.gz [1] https://code.google.com/p/zopfli/ -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69 Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement.
Re: bzip2
On Thu, Jun 6, 2013 at 11:48 AM, Theo de Raadt dera...@cvs.openbsd.org wrote: [...] If anyone thinks using this for the install or boot media is going to help, don't say a word until you can prove it on all platforms. Has anyone looked at zopfli[1] for the install media? It's a (apache 2 licensed) slightly better but much slower gzip compressor that still produces gzip-compatible output. For an amd64 ramdiskA it makes a bsd.gz that's 48k smaller than gzip (in ~16 seconds) and should compatible with the bootloader. I tried it once in a vm and it booted ok. $ time gzip -9cn bsd.strip bsd.gz real0m0.648s user0m0.610s sys 0m0.000s $ time zopfli -c bsd.strip bsd.zopfli.gz real0m17.409s user0m16.810s sys 0m0.580s $ ls -l bsd.gz bsd.zopfli.gz -rw-r--r-- 1 dtucker wsrc 1349508 Jun 6 15:40 bsd.gz -rw-r--r-- 1 dtucker wsrc 1310302 Jun 6 15:42 bsd.zopfli.gz [1] https://code.google.com/p/zopfli/ I don't know where we'd put it in the tree. If we did add it, it would only benefit the fast architectures, since the others cannot afford the additional build time. Developers would use up the space gains quickly. Right now a few architectures are neck and neck regarding which install media are close to full. Older architectures would hit full install media issues first. A smaller contingent of developers who take care of those architectures would have to deal with the fallout, creating further friction... I understand where the suggestion comes from, but also seea more downsides than benefits.
bzip2
Something that comes up from time to time is the question of whether to import bzip2 into base or not. Turns out the question is moot because already have imported it. There's a copy in perl. (I didn't know this until I happened to be watching a build closer than usual.) Since we already have the code built, why not let people use it? This is a small perl script that implements the 90% subset of functionality people expect from the regular bzip2 and bunzip2 utilities. It's not really complete or perfect, but I haven't spent all that much time on it. --- /dev/null Wed Jun 5 20:54:56 2013 +++ bzip2/Makefile Wed Jun 5 20:51:51 2013 @@ -0,0 +1,15 @@ +# $OpenBSD$ + +.include bsd.own.mk + +MAN= + +SCRIPT=bzip2.pl +LINKS=${BINDIR}/bzip2 ${BINDIR}/bunzip2 + +realinstall: + ${INSTALL} ${INSTALL_COPY} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \ + ${.CURDIR}/${SCRIPT} ${DESTDIR}${BINDIR}/bzip2 + + +.include bsd.prog.mk --- /dev/null Wed Jun 5 20:55:02 2013 +++ bzip2/bzip2.pl Wed Jun 5 20:53:23 2013 @@ -0,0 +1,113 @@ +#!/usr/bin/perl -w +# $OpenBSD$ +# Copyright (c) Ted Unangst t...@openbsd.org +# +# Permission to use, copy, modify, and distribute this software for any +# purpose with or without fee is hereby granted, provided that the above +# copyright notice and this permission notice appear in all copies. +# +# THE SOFTWARE IS PROVIDED AS IS AND THE AUTHOR DISCLAIMS ALL WARRANTIES +# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +use strict; +use Getopt::Long qw(:config posix_default bundling no_ignore_case); +use IO::Compress::Bzip2 qw(bzip2); + + +my $usestdout; +my $level = 9; +my $reqlevel; # requested level, who cares? +GetOptions( + c|stdout = \$usestdout, + 1|2|3|4|5|6|7|8|9|fast|best = \$reqlevel +); + +sub bzipfile { + my $fname = shift; + my $ofname; + my $in; + if ($fname ne -) { + if ($usestdout) { + $ofname = -; + } else { + $ofname = $fname . .bz2; + if (-e $ofname) { + warn Output file $ofname already exists.\n; + return; + } + } + $in = new IO::File $fname; + if (!$in) { + warn Can't open input file $fname.\n; + return; + } + } else { + $in = -; + $ofname = -; + } + if ($ofname eq - -t STDOUT) { + warn I won't write compressed data to a terminal.\n; + return; + } + if (not bzip2 $in = $ofname, BlockSize100K = $level) { + warn failz; + return; + } + if ($ofname ne -) { + unlink($fname); + } +} + +sub bunzipfile { +print bunzipping\n; + my $fname = shift; + my $ofname; + my $in; + if ($fname ne -) { + if ($usestdout) { + $ofname = -; + } else { + $ofname = $fname; + $ofname =~ s/\.bz2$//; + if (-e $ofname) { + warn Output file $ofname already exists.\n; + return; + } + } + $in = new IO::File $fname; + if (!$in) { + warn Can't open input file $fname.\n; + return; + } + } else { + $in = -; + $ofname = -; + } + if (not bunzip2 $in = $ofname) { + warn failz; + return; + } + if ($ofname ne -) { + unlink($fname); + } +} + +my $main; +if ($0 =~ /bunzip2$/) { + $main = \bunzipfile; +} else { + $main = \bzipfile; +} + +if (@ARGV == 0) { + $main(-); +} else { + foreach my $f (@ARGV) { + $main($f); + } +}