[gentoo-portage-dev] [PATCH] BinpkgExtractorAsync: xz and gzip decompression (142579)
This adds support for using a binary package's compression header to
determine the compression type, providing forward-compatibility for
xz and gzip decompression. The file name extension is disregared, so
that it will be possible use a compression-independent file naming
scheme in the future (see bug #150031 for discussion about proposed
file naming schemes).
Currently, only decompression is supported. It's useful to provide
forward-compatibility now, so that binhost clients will be prepared
to handle future binhost servers that use xz or gzip compression.
X-Gentoo-Bug: 142579
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=142579
---
pym/_emerge/BinpkgExtractorAsync.py | 28 ++--
pym/portage/util/compression_probe.py | 62 +++
2 files changed, 88 insertions(+), 2 deletions(-)
create mode 100644 pym/portage/util/compression_probe.py
diff --git a/pym/_emerge/BinpkgExtractorAsync.py
b/pym/_emerge/BinpkgExtractorAsync.py
index be74c2f..8d446f9 100644
--- a/pym/_emerge/BinpkgExtractorAsync.py
+++ b/pym/_emerge/BinpkgExtractorAsync.py
@@ -1,8 +1,12 @@
# Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
+import logging
+
from _emerge.SpawnProcess import SpawnProcess
import portage
+from portage.localization import _
+from portage.util.compression_probe import compression_probe
import signal
import subprocess
@@ -20,19 +24,39 @@ class BinpkgExtractorAsync(SpawnProcess):
if b--xattrs in output:
tar_options = --xattrs
+ comp = compression_probe(self.pkg_path)
+ if comp is None:
+ self.scheduler.output(!!! %s\n %
+ _(File compression header unrecognized: %s) %
+ self.pkg_path, log_path=self.logfile,
+ background=self.background, level=logging.ERROR)
+ self.returncode = 1
+ self._async_wait()
+ return
+
# Add -q to bzip2 opts, in order to avoid trailing garbage
after
# EOF ignored warning messages due to xpak trailer.
+ if comp == bzip2:
+ decomp_cmd =
${PORTAGE_BUNZIP2_COMMAND:-${PORTAGE_BZIP2_COMMAND} -d}
+ elif comp == xz:
+ decomp_cmd = xz -d
+ elif comp == gzip:
+ decomp_cmd = gzip -d
+ else:
+ raise AssertionError(Unexpected compression: %s %
comp)
+
# SIGPIPE handling (128 + SIGPIPE) should be compatible with
# assert_sigpipe_ok() that's used by the ebuild unpack() helper.
self.args = [self._shell_binary, -c,
- (${PORTAGE_BUNZIP2_COMMAND:-${PORTAGE_BZIP2_COMMAND}
-d} -cq -- %s | tar -xp %s -C %s -f - ; + \
+ (%s -cq -- %s | tar -xp %s -C %s -f - ; + \
p=(${PIPESTATUS[@]}) ; + \
if [[ ${p[0]} != 0 ${p[0]} != %d ]] ; then % (128
+ signal.SIGPIPE) + \
echo bzip2 failed with status ${p[0]} ; exit ${p[0]} ;
fi ; + \
if [ ${p[1]} != 0 ] ; then + \
echo tar failed with status ${p[1]} ; exit ${p[1]} ;
fi ; + \
exit 0 ;) % \
- (portage._shell_quote(self.pkg_path),
+ (decomp_cmd,
+ portage._shell_quote(self.pkg_path),
tar_options,
portage._shell_quote(self.image_dir))]
diff --git a/pym/portage/util/compression_probe.py
b/pym/portage/util/compression_probe.py
new file mode 100644
index 000..7bdd28f
--- /dev/null
+++ b/pym/portage/util/compression_probe.py
@@ -0,0 +1,62 @@
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import errno
+import re
+import sys
+
+if sys.hexversion = 0x300:
+ basestring = str
+
+from portage import _encodings, _unicode_encode
+from portage.exception import FileNotFound, PermissionDenied
+
+_compression_re = re.compile(b'^(' +
+ b'(?Pgzip\x1f\x8b)|' +
+ b'(?Pbzip2\x42\x5a\x68\x39)|' +
+ b'(?Pxz\xfd\x37\x7a\x58\x5a\x00))')
+
+def compression_probe(f):
+
+ Identify the compression type of a file. Returns one of the
+ following identifier strings:
+
+ bzip2
+ gzip
+ xz
+
+ @param f: a file path, or file-like object
+ @type f: file-like object
+ @return: a string identifying the compression type, or None if the
+ compression type is unrecognized
+ @rtype str or None
+
+
+ open_file = isinstance(f, basestring)
+ if open_file:
+ try:
+ f =
Re: [gentoo-portage-dev] [PATCH] BinpkgExtractorAsync: xz and gzip decompression (142579)
On 01/15/2015 07:00 PM, Brian Dolbec wrote:
On Thu, 15 Jan 2015 17:27:23 -0800
Zac Medico zmed...@gentoo.org wrote:
# Add -q to bzip2 opts, in order to avoid trailing
garbage after # EOF ignored warning messages due to xpak trailer.
+if comp == bzip2:
+decomp_cmd =
${PORTAGE_BUNZIP2_COMMAND:-${PORTAGE_BZIP2_COMMAND} -d}
+elif comp == xz:
+decomp_cmd = xz -d
+elif comp == gzip:
+decomp_cmd = gzip -d
+else:
+raise AssertionError(Unexpected
No offense, but yuk to the if foo else bar... else...
It's simple and it works. Why don't we just go with this simple approach
first, and add fancy stuff later on?
I already have code that does this much better, I decided I was going
to release it separately from catalyst because it is generally useful
and in many ways more future proof. I know you were interested in it,
but I hadn't gotten around to establishing it in a separate repo. I
also didn't know you were already working on this.
I've attached it to this email for you to look over. It needs a bit of
work for an independent release, but it is easily extended with
configuration changes. It can also be easily extended with custom
commands if needed.
If you are interested, it would not take long to have it release worthy
and in the tree.
I am interested in your library, but it does much more than is needed in
BinpkgExtractorAsync.The requirements are:
1) Map bzip2, xz, or gzip to an appropriate decompression command. The
command should be guaranteed to support -c and -q options, since
BinpkgExtractorAsync relies on them.
2) Respect the user's PORTAGE_BUNZIP2_COMMAND and PORTAGE_BZIP2_COMMAND
variables.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH] BinpkgExtractorAsync: xz and gzip decompression (142579)
On Thu, 15 Jan 2015 20:53:02 -0800
Zac Medico zmed...@gentoo.org wrote:
On 01/15/2015 07:00 PM, Brian Dolbec wrote:
On Thu, 15 Jan 2015 17:27:23 -0800
Zac Medico zmed...@gentoo.org wrote:
# Add -q to bzip2 opts, in order to avoid
trailing garbage after # EOF ignored warning messages due to
xpak trailer.
+ if comp == bzip2:
+ decomp_cmd =
${PORTAGE_BUNZIP2_COMMAND:-${PORTAGE_BZIP2_COMMAND} -d}
+ elif comp == xz:
+ decomp_cmd = xz -d
+ elif comp == gzip:
+ decomp_cmd = gzip -d
+ else:
+ raise AssertionError(Unexpected
No offense, but yuk to the if foo else bar... else...
It's simple and it works. Why don't we just go with this simple
approach first, and add fancy stuff later on?
PORTAGE_COMPRESSORS = {
bzip2: ${PORTAGE_BUNZIP2_COMMAND:-${PORTAGE_BZIP2_COMMAND} -d},
xz: xz -d,
gzip: gzip -d,
}
try:
decomp_cmd = PORTAGE_COMPRESSORS[comp]
except: AtributeError:
writemsg(You fool we don't support the %s (de)compression type!)
makes it simple to add new types, saves mile long if, elif, elif, else
code ;) There is also a good chance the code won't need editing to add
a new type, just add it to the dict definition. Second advantage is the
code should be slightly faster. (yes, I realize this isn't in a
multi k iteration loop, so speed-up is minuscule) but portage can't
afford to waste many cycles ;) it all adds up...
P.S. I don't expect you to use the writemsg(...) as is :D
I already have code that does this much better, I decided I was
going to release it separately from catalyst because it is
generally useful and in many ways more future proof. I know you
were interested in it, but I hadn't gotten around to establishing
it in a separate repo. I also didn't know you were already working
on this.
I've attached it to this email for you to look over. It needs a
bit of work for an independent release, but it is easily extended
with configuration changes. It can also be easily extended with
custom commands if needed.
If you are interested, it would not take long to have it release
worthy and in the tree.
I am interested in your library, but it does much more than is needed
in BinpkgExtractorAsync.The requirements are:
1) Map bzip2, xz, or gzip to an appropriate decompression command. The
command should be guaranteed to support -c and -q options, since
BinpkgExtractorAsync relies on them.
2) Respect the user's PORTAGE_BUNZIP2_COMMAND and
PORTAGE_BZIP2_COMMAND variables.
yes, I know there is more there than is needed and a few options need
adding.
--
Brian Dolbec dolsen
Re: [gentoo-dev] Moving CPU flags into USE_EXPAND
On Wed, 14 Jan 2015 21:59:37 +0100 Andreas K. Huettel dilfri...@gentoo.org wrote: That said, long time ago I was taught that instruction set use-flags should be avoided as much as possible. I don't remember the source for that anymore. Question to all, is that documented anywhere, and what are the specifics? I suspect use flags that only switch CFLAGS etc are forbidden, useflags that enable assembler code etc are allowed? I don't think this is documented besides the fact that forcing cflags is forbidden. For asm code you don't have much choice: either your package compiles all asm and has some kind of cpu detection so that it choses the best implementation at runtime (wasting a bit of resources), either if you enable something your cpu doesn't support then your program gets killed by SIGILL. Also, this is not the case for x86* but iirc on arm, if your cflags have some -march or -mcpu then gcc forwards it to as which in turns refuses any instruction not supported by your cpu: you must disable forbidden instructions in order to build your package. Alexis.
Re: [gentoo-dev] Gentoo-sources - should we stable?
03.01.2015 00:53, Mike Pagano пишет: On Saturday, January 03, 2015 12:39:39 AM Mikle Kolyada wrote: 02.01.2015 20:25, Mike Pagano пишет: This is in no way complaining about how long it takes to stabilize a kernel. As for this fact. hat type=arch teams developer The main problem is that: we only can test sources on machine we can reboot. For example me and Agostino have access to the rest hardware like alpha, ia64 and so on. But we can't reboot it for clear reason i think. Another side is that: not all hardware i have around can use gentoo-sources, so it works with custom kernels. /hat Mikle, Let me reiterate. This should be in no way interpreted as an attack on the arch teams. I'm getting more and more constrained by life and slacking like crazy, so I would never complain about the amount of time other volunteers put into this distribution. AFAIC, you definitely don't need to defend the arch teams whom I respect and whose efforts I greatly appreciate. Mike Mike, i think there was no attacking/defending here, just some misunderstanding. In my point of view, Mikle just wanted to say that proper testing of kernel requires booting in that kernel(it's obvious, i think). And, while we, as arch team members, have access for those machines, there is big problem: if tested kernel will panic - we are stuck until machine's owner will fix it. And you understand, how he would be disappointed, especially if nobody warns him. That's not related to kernel exclusively. That's the common problem for all system stuff, that can break, while you have only remote access for the machine. That's why i prefer to test kernel, Glibc, OpenRC, udev, etc. on machines, that i have physical access on. So, i like your idea to stick stable to the LTS kernel. While it can lead to potential problems with some external modules(which are, for example, marked stable now but does not support 3.4 kernel) the majority of really stable external kernel modules usually backward compatible with LTS kernels. And, as they get security fixes and breaks rarely(at least much more rare, than migrating from 3.X to 3.X+1), people, i think who likes stable will be much more happy. At least i will be for sure :-) -- Best regards, Sergey Popov Gentoo developer Gentoo Desktop Effects project lead Gentoo Quality Assurance project lead Gentoo Proxy maintainers project lead signature.asc Description: OpenPGP digital signature
[gentoo-dev] Re: Moving CPU flags into USE_EXPAND
Christopher Head ch...@chead.ca wrote: All that requires is knowing the names, though; it would be fine if no package actually uses the feature yet. ++ More precisely: When changing the names anyway, IMHO it would be a very good idea to follow the convention of the flag names in /proc/cpuinfo and add all flags supported there as possible USE-flags, no matter, whether they are currently used in some package or not. The list might still be extended when a new processor with new features comes out. If this is done properly, the user could easily write a script to set these flags appropriate for his current machine. One might even give portage a --use-native switch (or FEATURE=use-native) which sets the USE defaults corresponding to the current /proc/cpuinfo output. The latter is what typical desktop users probably would prefer: A complete analogue of CFLAGS=-march=native; just generating optimal code for his machine, without requiring the user to understand all the fancy details of his processor. To properly support packages which can choose the appropriate code at runtime, I suggest to add in addition an all flag which just compiles in all possible optimized code-paths.
Re: [gentoo-dev] Re: Moving CPU flags into USE_EXPAND
Il 15/01/2015 11:30, Alexis Ballier ha scritto:
On Thu, 15 Jan 2015 10:20:15 + (UTC)
Martin Vaeth mar...@mvath.de wrote:
Christopher Head ch...@chead.ca wrote:
All that requires is knowing the names, though; it would be
fine if no package actually uses the feature yet.
++
More precisely: When changing the names anyway,
IMHO it would be a very good idea to follow the convention of the
flag names in /proc/cpuinfo and add all flags supported
there as possible USE-flags, no matter, whether they are currently
used in some package or not.
seriously ?
$ grep flags /proc/cpuinfo | head -n 1 | wc -w
94
Actually I like the idea, there are performances problems related to
ebuilds or portage?
CPU_FLAGS=$(grep -m1 flags /proc/cpuinfo)
CPU_FLAGS=${CPU_FLAGS#*:}
echo $CPU_FLAGS | wc -w
92
Re: [gentoo-dev] Re: Moving CPU flags into USE_EXPAND
On Thu, 15 Jan 2015 11:50:25 +0100 viv...@gmail.com viv...@gmail.com wrote: Il 15/01/2015 11:30, Alexis Ballier ha scritto: On Thu, 15 Jan 2015 10:20:15 + (UTC) Martin Vaeth mar...@mvath.de wrote: Christopher Head ch...@chead.ca wrote: All that requires is knowing the names, though; it would be fine if no package actually uses the feature yet. ++ More precisely: When changing the names anyway, IMHO it would be a very good idea to follow the convention of the flag names in /proc/cpuinfo and add all flags supported there as possible USE-flags, no matter, whether they are currently used in some package or not. seriously ? $ grep flags /proc/cpuinfo | head -n 1 | wc -w 94 Actually I like the idea, there are performances problems related to ebuilds or portage? it is not about performance but rather about writing the proper .desc file; i don't think this will ever happen if nobody baking up the idea comes with such a file Alexis.
Re: [gentoo-dev] Re: Moving CPU flags into USE_EXPAND
On Thu, 15 Jan 2015 10:20:15 + (UTC) Martin Vaeth mar...@mvath.de wrote: Christopher Head ch...@chead.ca wrote: All that requires is knowing the names, though; it would be fine if no package actually uses the feature yet. ++ More precisely: When changing the names anyway, IMHO it would be a very good idea to follow the convention of the flag names in /proc/cpuinfo and add all flags supported there as possible USE-flags, no matter, whether they are currently used in some package or not. seriously ? $ grep flags /proc/cpuinfo | head -n 1 | wc -w 94
Re: [gentoo-dev] Gentoo-sources - should we stable?
On Thu, Jan 15, 2015 at 3:31 AM, Sergey Popov pinkb...@gentoo.org wrote: So, i like your idea to stick stable to the LTS kernel. While it can lead to potential problems with some external modules(which are, for example, marked stable now but does not support 3.4 kernel) the majority of really stable external kernel modules usually backward compatible with LTS kernels. By LTS I was thinking more of 3.14, and not so much 3.4 or (gasp) 2.6.32. We just want to get out of the version of the month grind. -- Rich
Re: [gentoo-dev] Gentoo-sources - should we stable?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 01/15/2015 11:01 AM, Rich Freeman wrote: On Thu, Jan 15, 2015 at 3:31 AM, Sergey Popov pinkb...@gentoo.org wrote: So, i like your idea to stick stable to the LTS kernel. While it can lead to potential problems with some external modules(which are, for example, marked stable now but does not support 3.4 kernel) the majority of really stable external kernel modules usually backward compatible with LTS kernels. By LTS I was thinking more of 3.14, and not so much 3.4 or (gasp) 2.6.32. We just want to get out of the version of the month grind. Fwiw I still have servers running Gentoo on the 3.4 series, and doubt I'm the only one. I tend to stick to the LTS from when it is set up as I don't need new functionality on it and want to retain a level of stability, in particular on servers that are not in my physical presence (at least without a great deal of travelling). Thankfully have all servers upgraded to get rid of 2.6 series though :) - -- Kristian Fiskerstrand Public PGP key 0xE3EDFAE3 at hkp://pool.sks-keyservers.net fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3 -BEGIN PGP SIGNATURE- iQIcBAEBCgAGBQJUt6vBAAoJEPw7F94F4Tag1tEP/iJuyG5bXw1Tq87xKP71ZQRd B+aIrzPlMYJjLYVLuYUAtlNivyHJIpE6+wFEp8hJoYs9Qcz7IgT6k/hHluNpY9eM rp+I+8hzOKxWIoeEXQDhXcDNTZSV3jd6uLUGwDG4rEMNVh9KnZZzdD0whVYduFaB n/vZ8pfnn8aMtoYGFT9HAHDz752RzP++ImItTqNdzMAhdUU5M9q7Vj0dS0wa5teX e2Bz8ok7X6fIIBP8r3hLK5uyfzCuzgoZsASFABwJJwH/Ffr6zFbHCqnm8TKOdHx/ CqhDg8U7zPpOO6kiaKA3FPdP6QFh/8d8FMpknhp2F7NHsb9CDSDGQnKVIJg/sCv6 eBb0lVwUwUrHK7SRVOHSxR6koIMCNObZXhJhDzhIuwG0JU43Bw12yia9IlveedTz 2Y7pPp/WOi7J4SRSbsL4KYynUsTmf751f5hnJrvHs0vr19tTGiCg27grS/e8bDih JPFjIwU74JDAp19MadnfXhoKsx5xexrsDtd73TCuvL2YoS6+COtYGil0xzBsadE/ keIwiJQqJJSFge1uf0GlzTo7CLTQCT1zAz7ojD09KUUpjNTNUPhCXNkvxQFvDCrM IAoet0VuHiL1a1QDiXWTDWI31oYOsBRvf20/wsAtznOgjUhCAL7IgTXwufIKlSn2 nOjCupHWHCBPc8Wb8LvH =/Crr -END PGP SIGNATURE-
[gentoo-dev] Re: Moving CPU flags into USE_EXPAND
Alexis Ballier aball...@gentoo.org wrote: More precisely: When changing the names anyway, IMHO it would be a very good idea to follow the convention of the flag names in /proc/cpuinfo and add all flags supported there as possible USE-flags, no matter, whether they are currently used in some package or not. seriously ? $ grep flags /proc/cpuinfo | head -n 1 | wc -w 94 Well, currently, we have at least 10-20 already: At a very quick check (probably missing some), I found: mmx sse sse2 aes-ni ssse3 sse3 sse4 sse4a sse4.1 sse4.2 3des 3dnow 3dnowext and probably there are more to be added anyway in future packages. I am not even sure whether some of these sse4* are dupes or not since they do not follow /proc/cpuinfo convention. Some others do not follow this convention at all, e.g. sse3 is called pni in /proc/cpuinfo or aes-ni is called aes in /proc/cpuinfo This is mainly what I meant: Currently, users have to look up how sse3 is called in /proc/cpuinfo to understand whether their processor has support for it. This could be automated. If not all of the 94 flags are included, it is perhaps not a drama; my main suggestion is to follow /proc/cpuinfo convention in the existing ones. but rather about writing the proper .desc file Well, currently the descriptions are: 3dnow: Use the 3DNow! instruction set mmx: Use the mmx instruction set sse: Use the SSE2 instruction set sse4_1: Enable sse4.1 acceleration ... So they are practically useless unless you know anyway what the corresponding processor flags mean. I suggest to start with a stub .desc list, e.g.: yyy: Use the yyy CPU feature, cf. /proc/cpuinfo This is enough for the user to check on which of his systems he might want it. A more verbose description can be added for some flags some day as a bonus; such a list can grow over time, and some experts might want to send some patches to fix/extend the descriptions. But why require for the start more than we have currently? If there is no list to start with, then you can also not expect that somebody will suggest patches (I mean: to a non-listed feature...)
Re: [gentoo-dev] Re: Moving CPU flags into USE_EXPAND
On Thu, 15 Jan 2015 12:03:27 + (UTC)
Martin Vaeth mar...@mvath.de wrote:
Alexis Ballier aball...@gentoo.org wrote:
More precisely: When changing the names anyway,
IMHO it would be a very good idea to follow the convention of
the flag names in /proc/cpuinfo and add all flags supported
there as possible USE-flags, no matter, whether they are
currently used in some package or not.
seriously ?
$ grep flags /proc/cpuinfo | head -n 1 | wc -w
94
Well, currently, we have at least 10-20 already:
At a very quick check (probably missing some), I found:
mmx sse sse2 aes-ni ssse3 sse3 sse4 sse4a sse4.1 sse4.2
3des 3dnow 3dnowext
and probably there are more to be added anyway
in future packages.
I am not even sure whether some of these sse4* are dupes
or not since they do not follow /proc/cpuinfo convention.
Some others do not follow this convention at all,
e.g. sse3 is called pni in /proc/cpuinfo
or aes-ni is called aes in /proc/cpuinfo
This is mainly what I meant: Currently, users have to look
up how sse3 is called in /proc/cpuinfo to understand whether
their processor has support for it. This could be automated.
/proc/cpuinfo is quite useless as a description
when i added such flags i tried to follow wikipedia naming and added
min cpu version that supported it in the description.
flags are usually extensively described in:
http://en.wikipedia.org/wiki/${uppercase flag}
algo, e.g. pni and aes pages are disambiguation ones, while SSE3 and
AES-NI are quite clear on what they are.
from wikipedia:
- SSE2, Willamette New Instructions (WNI)
- SSE3, also called Prescott New Instructions (PNI)
- SSSE3, Merom New Instructions (MNI)
- SSE4, Penryn New Instructions (PNI)
- AVX (Advanced Vector Extensions), Gesher New Instructions (GNI)
why do I have pni and avx in /proc/cpuinfo ?
this sounds quite inconsistent and is probably due to the fact that
those flags are added early in the kernel and have to stay forever,
while the naming might end up being a poor naming a few months or
years later. sys-apps/cpuid disambiguates it better.
I don't think it is a good idea to forward this inconsistency as-is:
better write a cpuinfo2useflags script that you could to make.conf
if automation is really the goal.
Alexis.
