Re: [gentoo-dev] [RFC] unpacker.eclass extensions
On Monday 17 June 2013 16:37:06 Rick "Zero_Chaos" Farina wrote: > On 06/17/2013 04:19 PM, Diego Elio Pettenò wrote: > > On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote: > >> I make all my files with "tar cJf" > >> > >> zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz > >> /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data > > > > cJ with _current_ tar will generate XZ > > cJ with _past_ tar could generate lzma > > xJ with _current_ tar will extract both XZ and lzma > > zJ with _past_ tar will only extract lzma > > tar for the last several years properly and automatically detects the > format of the input using just 'x'. This is true for gnu and bsd tar > which should cover all of gentoo. and that relies on the magic being in the output format which, as we pointed out, wasn't the case. it's one of the reasons why the XZ format came to be. build lzma-utils and compress some files and see how well tar & file handle it: http://tukaani.org/lzma/ -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] [RFC] unpacker.eclass extensions
On Mon, 2013-06-17 at 16:37 -0400, Rick "Zero_Chaos" Farina wrote:
> On 06/17/2013 04:19 PM, Diego Elio Pettenò wrote:
> > On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote:
> >> I make all my files with "tar cJf"
> >>
> >> zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz
> >> /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data
> >
> > cJ with _current_ tar will generate XZ
> > cJ with _past_ tar could generate lzma
> > xJ with _current_ tar will extract both XZ and lzma
> > zJ with _past_ tar will only extract lzma
> >
> >
> tar for the last several years properly and automatically detects the
> format of the input using just 'x'. This is true for gnu and bsd tar
> which should cover all of gentoo. We should be passing "tar xf" to
> extract things unless we want to hijack the decompression command like
> is possible in portage with BZIP2_COMMAND=lbzip2 in which case we want
> to pass "tar -I $BZIP2_COMMAND -xf" or (not yet in portage) "tar -I pixz
> -xf". Some good examples of how this are handled are in catalyst but
> I'll let dolson talk about his own work as he is much more qualified
> than I am.
>
> -Zero
Well, I don't code bash, but...
In catalyst, I had a request to add more types of
compression/decompression capabilities. Plus I am also trying to clean
up the messy, poorly structured code. What I have come up with is a
separate class that can be loaded with either compression or
decompression commands that can be configured, overridden, added to,...
It will provide a python interface to perform the actions requested,
keeping the detailed coding central and internal. It will provide
automatic detection of the command to use via it's extension as a
combination of a pre-determined configured preference (if it is
compatible) or a suitable command determined by that extension. There
can also be several predefined ways to do the same action, in which case
the pre-configured preference will take priority.
I hadn't gotten into reading this thread much, but did note the
similarity of what I was currently doing and what it was discussing.
What is directly relevant to the tar commands is listed in the
compression_definitions and decompression_definitions at the top of the
file attached.
This python lib can easily be made into a standalone pkg with a cli
interface to perform many common actions without the need of additional
coding in eclasses, package managers, etc.. It could provide gentoo
with a common, central, easily extended and updated method of performing
normal compression/decompression.
It is a work in progress, with only preliminary testing done inside my
catalyst rewrite. I have not yet added configuration options and
override preferences.
It is available in my catalyst git repo in my dev space. It is in the
compress branch. Note there is no master branch there, so a basic clone
will error when it tries to checkout a master working copy on completion
of the clone. Just "git checkout compress" after.
http://dev.gentoo.org/~dolsen/catalyst/
I've attached the current compress.py file so you can peruse and decide
if making this a standalone gentoo project is desired. If not, it will
get merged into catalyst code base.
--
Brian Dolbec
# Maintained in full by:
# Catalyst Team
# Release Engineering Team
'''
compress.py
Utility class to hold and handle all possible compression
and de-compression of files. Including rsync transfers.
'''
import os
from collections import namedtuple
from support import cmd
definition_fields = ["func", "cmd", "args", "id", "extension"]
defintition_types = [ str,str, list, str, str]
'''The definition entries are to follow the the definition_types
with the exception of the first entry "Type" which is a mode identifier
for use in the class as a type ID and printable string.'''
compress_definitions = {
"Type": ["Compression", "Compression definitions loaded"],
"rsync" :["rsync", "rsync", ["-a", "--delete", "%(source)s", "%(destination)s"], "RSYNC", None],
"lbzip2" :["_common", "tar", ["-I", "lbzip2", "-cf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "LBZIP2", "tbz2"],
"tbz2" :["_common", "tar", ["-I", "lbzip2", "-cf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "LBZIP2", "tbz2"],
"bz2" :["_common", "tar", ["-cpjf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "BZIP2", "tar.bz2"],
"tar" :["_common", "tar", ["-cpf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "TAR", "tar"],
"xz" :["_common", "tar", ["-cpJf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "XZ", "tar.xz"],
"pixz" :["_common", "tar", ["-I", "pixz", "-cpf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "PIXZ", "xz"],
"zip" :["_common", "tar", ["-cpzf", "%(filename)s", "-C", "%(basedir)s", "%(source)s"], "GZIP", "zip"],
}
decompress_definitions = {
"Type": ["Decompression", "Decompression definitions loaded"],
"rsync" :["rsync", "rsync", ["-a", "--delete", "%(source)s", "
Re: [gentoo-dev] [RFC] unpacker.eclass extensions
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/17/2013 04:19 PM, Diego Elio Pettenò wrote: > On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote: >> I make all my files with "tar cJf" >> >> zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz >> /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data > > cJ with _current_ tar will generate XZ > cJ with _past_ tar could generate lzma > xJ with _current_ tar will extract both XZ and lzma > zJ with _past_ tar will only extract lzma > > tar for the last several years properly and automatically detects the format of the input using just 'x'. This is true for gnu and bsd tar which should cover all of gentoo. We should be passing "tar xf" to extract things unless we want to hijack the decompression command like is possible in portage with BZIP2_COMMAND=lbzip2 in which case we want to pass "tar -I $BZIP2_COMMAND -xf" or (not yet in portage) "tar -I pixz - -xf". Some good examples of how this are handled are in catalyst but I'll let dolson talk about his own work as he is much more qualified than I am. - -Zero -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJRv3NyAAoJEKXdFCfdEflKTYIP/ifbrOuhtEW2W3NRnUNImOmn YfRuRt+50OpEeHbniAp32zOhg0l6nYCgZ4CqzoUDTUm106EyAbyQSNheBC1d6DL6 eRFbuV41+B1cPDFB5FJMHrhapvKQmciGxyfgpPLrVnIHXQhNUvM+3h7qPq2SOnog tF8gQ3J3umfcJa3NMEqRRrWNkiytTniKZ537fLT0vxVvtOXSXswe3o3TJPNLpNVO BAyHNvZz3TKPy3gsHtvedo1iz37yUxUxAPnI0R0ccLRnPEGlnnNcseaQvCLGBvZP a7xqPMnM1gRv4xH9deWvh//mCSvIBjGoaeQLEP5ztjinrWalkFDikUk8xx8YZRev YYlQbj1n3NgKk7Io/fyXjaNmZGXHkdT0kYosT2w+uzwHMQVSbwIsdR24dUIxXrcp SG37KNB/wtPzOWpUYkV13wJgkLc1mVmEj92F47dSgGt4EKtSUng3sudDs2bPz4ds 7Z0lV4xLiG36ITZidBIIpjIWAKIQbQWIW+mIC25g40WGJ+qJtSz5bivwJa2a9PAx Zqhja3pQFbYz+YhIUidM/rkcy1XINicaL1lUbpaHZqzy8z/ivWAh6iXMLmulqdfa BjfPHVj0JcJs+ySbHMP/W3DN823BCzxOdc8cvHhLOG/Vhdysr+t5ga0yAQOawJsz fWqUGz1tNiKLU3uBs+20 =2YDM -END PGP SIGNATURE-
Re: [gentoo-dev] [RFC] unpacker.eclass extensions
On 17/06/2013 17:54, Rick "Zero_Chaos" Farina wrote: > I make all my files with "tar cJf" > > zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz > /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data cJ with _current_ tar will generate XZ cJ with _past_ tar could generate lzma xJ with _current_ tar will extract both XZ and lzma zJ with _past_ tar will only extract lzma -- Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/ signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [RFC] unpacker.eclass extensions
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/17/2013 12:08 PM, Mike Frysinger wrote: > On Monday 17 June 2013 03:15:53 Diego Elio Pettenò wrote: >> On 17/06/2013 06:55, Mike Frysinger wrote: > tar -xJf "./$i" >>> >>> why doesn't the bzip2 detect as bzip2 ? >> >> Capital J is XZ not bzip2.. still, it makes me think it needs a newer >> version of file to recognize it. > > the stupid default font doesn't clearly render that. guess that's the older > LZMA format rather than the newer XZ as the latter can be detected by file. I make all my files with "tar cJf" zero@ozzie ~ % file /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz /usr/portage/distfiles/gr-osmosdr-0.0.2.tar.xz: XZ compressed data worksforme - -Zero -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJRvz9dAAoJEKXdFCfdEflKLIsQAJ+AeXP5CIrlZSefzJCFgP4G 6Fd3XJPsTKr58KBl5KLmAVPOtRwac0vXty4F9lHMHECvnfNOh0UiajvNV/Jyz1yz zPZ/91Vv0vXc5ZlNNLqmq+rtIt3wuEXCg0TNwxTMbKX0MWW8S7ssOzibiYcLLoqT IHFKLXhtU3zIxQ4kLiMZ7e1QRBnrVvA4/VlcOKAs8R6cdsp/1e3fzhgZgMx9ft03 hmHLW9kMMC06/Bz8ZKsvnfuyWhvQrlvsOUH8JWC9pbyZO58fiCgwSFWRyma04d8k ulJ2y8K4B/WkIwRZzL47ROZQ91FcEnv9Ei9GzXLK6u4gIIrepX0NPcea441F1uVa 4i/9sjyk/zm6M3dAZyGlsCxZX96pAuFuE/H2OFcsaj9PRcdD2nR0Q4uvKwUqn+BM Mhcr2wQlBsLDHiQr/3g/WepZ7VGcKrfmWymjxcCZoj9VGMhMR6O/h3uGEAdLm3YK 1gLdvyQz+koUFdkmQIQHXFjd8q+165Lbky5CYqs6xekM1aSeLpskdZpVSI7GRp+j NvbB7gtEw2iz2pS8OSQ834K7KNnWI7OydVsv4cYnCySg3md+BAL1S5LLsc7FdmHz cRLjH8Yw5mH4EfeGFnIebWYsVMeRn8M0KJxVuz7GH1gjtIvOjfIOcjdWMBwOhryP LgmjPDvPM7NGJ3mneTLM =UOVj -END PGP SIGNATURE-
Re: [gentoo-dev] [RFC] unpacker.eclass extensions
On Monday 17 June 2013 03:15:53 Diego Elio Pettenò wrote: > On 17/06/2013 06:55, Mike Frysinger wrote: > >> > tar -xJf "./$i" > > > > why doesn't the bzip2 detect as bzip2 ? > > Capital J is XZ not bzip2.. still, it makes me think it needs a newer > version of file to recognize it. the stupid default font doesn't clearly render that. guess that's the older LZMA format rather than the newer XZ as the latter can be detected by file. -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] [RFC] unpacker.eclass extensions
On 17/06/2013 06:55, Mike Frysinger wrote: >> > tar -xJf "./$i" > why doesn't the bzip2 detect as bzip2 ? > Capital J is XZ not bzip2.. still, it makes me think it needs a newer version of file to recognize it. -- Diego Elio Pettenò — Flameeyes flamee...@flameeyes.eu — http://blog.flameeyes.eu/ signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] [RFC] unpacker.eclass extensions
On Saturday 15 June 2013 04:39:10 Vadim A. Misbakh-Soloviov wrote:
> # @DESCRIPTION:
> # Unpack nixstaller generated files
needs a period at the end. content in @DESCRIPTION is normalized.
> # They're shell scripts with the blob package tagged onto
> # the end of the archive. In the blob placed tarballs with
> # actual content.
"They're shell scripts with a tarball appended to them."
> # Please note, if you need additional dependecies make sure to unpack
> subarch
> # archive as first argument.
no idea what this means
> nixstaller_unpack() {
this does not follow the API naming convention ("unpack" comes first)
> local unpack_files="$@"
this doesn't work. you normalized the input into a string. just inline the
"$@" below. or don't specify it at all ... this does the same thing (albeit,
correctly):
local src
for src ; do
> unpack_banner "$i"
> # Make sure that file exists
> [[ -f "./$i" ]] && (
> local type=$(file -b ${i})
> case ${type} in
> data)
> tar -xJf "./$i"
why doesn't the bzip2 detect as bzip2 ?
> ;;
> gzip*)
> tar -xzf "./$i"
> ;;
> esac
> ) || die "Failed to unpack $i"
the subshell should go away -- you don't even need it:
[[ $? -eq 0 ]] || die ...
i wish we could merge with the file detection in unpack_makeself somehow
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-dev] [RFC] unpacker.eclass extensions
On 15 June 2013 15:33, Markos Chandras wrote: >Also || die does not work in subshells. > > http://devmanual.gentoo.org/ebuild-writing/error-handling/index.html > Sorry scratch that. I just realized || die is not inside the subshell -- Regards, Markos Chandras - Gentoo Linux Developer http://dev.gentoo.org/~hwoarang
Re: [gentoo-dev] [RFC] unpacker.eclass extensions
Hi Vadim,
On 15 June 2013 09:39, Vadim A. Misbakh-Soloviov wrote:
> # Make sure that file exists
> [[ -f "./$i" ]] && (
> local type=$(file -b ${i})
> case ${type} in
> data)
> tar -xJf "./$i"
> ;;
> gzip*)
> tar -xzf "./$i"
> ;;
> esac
> ) || die "Failed to unpack $i"
> done
> }
>
I have a couple of questions. Why sub-shell here? My understanding is
that first you need to make sure the
file exists and then unpack it. Also || die does not work in subshells.
http://devmanual.gentoo.org/ebuild-writing/error-handling/index.html
> 2) It'd be also nice to detect mojo installers (most of the time,
> they're "*-bin" or "*Installer" ELF32 sfx-archives) and pass them to
> unpack_zip by default.
> Unfortunately, I've no free time ATM, but if it is needed, I can write
> prototype for Mojo unpacking.
>
We can only merge what you show us for review :)
--
Regards,
Markos Chandras - Gentoo Linux Developer
http://dev.gentoo.org/~hwoarang
[gentoo-dev] [RFC] unpacker.eclass extensions
As gamerlay maintainer, I'd be glad to introduce some changes to
unpacker.eclass:
1) merging unpacker-nixstaller (Makeself subspecies) from gamerlay:
# @FUNCTION: unpack_nixstaller
# @USAGE:
# @DESCRIPTION:
# Unpack nixstaller generated files
# They're shell scripts with the blob package tagged onto
# the end of the archive. In the blob placed tarballs with
# actual content.
#
# Please note, if you need additional dependecies make sure to unpack
subarch
# archive as first argument.
#
nixstaller_unpack() {
unpack_makeself
local unpack_files="$@"
for i in $unpack_files ; do
unpack_banner "$i"
# Make sure that file exists
[[ -f "./$i" ]] && (
local type=$(file -b ${i})
case ${type} in
data)
tar -xJf "./$i"
;;
gzip*)
tar -xzf "./$i"
;;
esac
) || die "Failed to unpack $i"
done
}
Original author is Azamat Hackimov aka winterheart (in CC).
2) It'd be also nice to detect mojo installers (most of the time,
they're "*-bin" or "*Installer" ELF32 sfx-archives) and pass them to
unpack_zip by default.
Unfortunately, I've no free time ATM, but if it is needed, I can write
prototype for Mojo unpacking.
signature.asc
Description: OpenPGP digital signature
