Re: [Rpm-maint] [PATCH 07/15] Use macros to hide "int" implementation differences between Python 2 and 3
On Thu, 15 Oct 2009, David Malcolm wrote: On Python 3, use macros to alias all usage of PyInt_ to PyLong_ equivalents, as the former no longer exists. Right, simply aliasing to PyLong is probably quite ok. Likewise, the "cmpfunc" typedef no longer exists; introduce a dummy definition to enable our code to compile ...but this is not really :) We'll need to convert the tp_compare cases to tp_richcompare, AFAICT Python handles this compatibly in 2.x so it shouldn't be a big deal. - Panu - ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [PATCH 10/15] Port tagNumFromPyObject on Python 3 to use unicode objects
On Thu, 2009-10-15 at 15:15 -0400, David Malcolm wrote:
> Preserve the string-based API to headers:
>h['name']
> by expecting a PyUnicode on Python 3, and a PyString on Python 2.
> ---
> python/header-py.c |7 +++
> 1 files changed, 7 insertions(+), 0 deletions(-)
>
> diff --git a/python/header-py.c b/python/header-py.c
> index 98cd753..f3454df 100644
> --- a/python/header-py.c
> +++ b/python/header-py.c
> @@ -421,8 +421,15 @@ int tagNumFromPyObject (PyObject *item, rpmTag *tagp)
> if (PyInt_Check(item)) {
> /* XXX we should probably validate tag numbers too */
> tag = PyInt_AsLong(item);
> +#if PY_MAJOR_VERSION >= 3
> +} else if (PyUnicode_Check(item)) {
> + PyObject *utf8_bytes = PyUnicode_AsUTF8String(item);
> + tag = rpmTagGetValue(PyBytes_AsString(utf8_bytes));
> + Py_XDECREF(utf8_bytes);
> +#else
There's no reason for ifdef here, as it isn't a bug if:
ipkg.hdr[u'name']
...works in 2.* ... dito. hdr[b'name'] working in 3.*.
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
[Rpm-maint] rpm: support lzip compression for %setup (patch)
On Sat, 29 Nov 2008, Ralf Corsepius wrote: > IMO, rpm should not adopt lzip support, unless lzip has proven to be > a functional and viable tool. Many people think lzip has already proven that: http://lpar.ath0.com/2009/09/25/documentation-as-an-indicator-of-code-quality/ http://www.dragora.org/dokuwiki/doku.php?id=makepkg http://www.tuxradar.com/content/100-open-source-gems-part-2 http://packages.debian.org/sid/lzip Lzip is even used to distribute GNU packages on ftp.gnu.org. ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [RFC][PATCH] Large file support
On Wed, 2009-09-09 at 16:42 +0200, Florian Festi wrote: > Hi! > > I had a look at the 4GB per packaged file limit. The current cpio format > [1] uses 8 bytes to encode 32 bit integers as hexadecimal ASCII strings. > So there is no way of fixing this problem while staying compatible with > the cpio format (and keep rpm2cpio working). > > Having a look at the tar formats I do not belief that switching to tar > is a real option. The format is just horrible (GNU tar needs over 200 > lines to read an integer out of a header field) and full of hacks to > remain backward compatible (header in header + extentions). This would > be all not that bad if there where a nice little library we could link > against... Some of this is just GNU tar code, but it's true that the cpio format is much nicer. And personally I think it'd have to be an awesome win to move to a completely new format. > My favorite solution would be to use no payload format at all and just > rely on the meta data we ship in the header anyway. While this would > surely be possible it requires redoing the hard link handling (as hard > links are treated specially in the payload - like shipping the content > just once) and modifying the next upper layer within rpm (fsm.c) which > is probably the most horrible place in the whole code base. Volunteers > welcome! I assume rpm could then ship something simple that would do what everyone uses rpm2cpio for (unpack stuff to disk)? This is a better option than moving to tar/XAR/zip/whatever, IMO, but still seems like a lot of work for not much gain. > A much simpler alternative would be to use a slightly modified cpio > format. With a new magic for large files we could just put an binary > integer into the c_filesize field (or all integer fields). Another > solution could be to keep the hexadecimal encoding and just double the > c_filesize or even some more integer fields. > This will both render the payload incompatible with cpio if there are > large files (and only then). > > I did not yet ask cpio upstream or our cpio package maintainer about > accepting patches to at least read such archives... > > Attached patch uses a binary integer for large file sizes. Patch is > untested and assumes that everything else that deals with file sizes > already is 64 bit save. > > Comments? Ideas? Panic? I think a "new" cpio format that allows the extension of (quick look at cpiohdr.h) ino, uid, gid, mtime and filesize ... would probably be accepted everywhere fairly quickly. Speak to the Fedora guy and upstream (although I wouldn't expect a quick response from upstream, personally), as I'm less sure if they'd want to just double all the values or be clever in some way to keep the hdr compact. -- James Antill Fedora ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] odd chroot behavior inside the rpm transaction
On Tue, 2009-09-15 at 10:04 +0200, Michael Schroeder wrote:
> On Mon, Sep 14, 2009 at 04:54:11PM -0400, Seth Vidal wrote:
> > if, from the python bindings, I open a file with a name starting with '/'
> > while in a transaction then as expected, the file is opened inside the
> > installroot
> >
> > however, if I open a file with a name NOT starting with '/' then the file
> > is opened OUTSIDE of the installroot.
> >
> > Does this make any sense? B/c I admit I don't quite grok why it would be
> > this way.
>
> That's because rpm needs a way to switch back from the installroot
> to the old root. It does this by doing a
> chroot(".")
> call.
This is usually done by keeping around an fd for the old ".", and then
doing fchdir() on it (followed by chroot(".")).
--
James Antill
Fedora
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [PATCH 06/15] Update module initialization to work with both Python 2.* and Python 3.*
On Thu, 15 Oct 2009, David Malcolm wrote:
Introduce macros and conditional compilation to deal with the major
changes to the way that Python extension modules are initialized between
Python 2 and 3 (PEP 3121).
Hmm, this is starting to look a bit more painful :)
diff --git a/python/header-py.h b/python/header-py.h
index 8c9dd68..277ef96 100644
--- a/python/header-py.h
+++ b/python/header-py.h
@@ -12,6 +12,13 @@ extern PyTypeObject hdr_Type;
#define DEPRECATED_METHOD(_msg) \
PyErr_WarnEx(PyExc_PendingDeprecationWarning, (_msg), 2);
+/*
+ FIXME: this is "global" module state and for Python 3 this ought to be split
+ out into a module state struct as per PEP 3121
+
+ Unfortunately we don't have convenient module ptrs available at each point
+ of use, so I've punted it for now.
+*/
extern PyObject * pyrpmError;
FWIW, the current plan is to push much more work over to python side where
possible, ie make the C-level bindings just a thin layer over the librpm C
API and pythonize things in python instead. The rpm-specific exception(s)
is one of the things I'd like to push over to python side completely. Not
possible yet though, but among other things it would avoid having to deal
with these incompatibilities.
[...]
I didn't try out what it'd actually look like, but it'd seem to me that
the module initialization version differences could be made more obvious
(less #ifdefs) by splitting out much of the init work into a separate
function, ie something like
static int initModule(PyObject *m)
{
/* ... init the types, dicts et ... */
/* return 0/1 on error/success */
}
#if PY_MAJOR_VERSION >= 3
static struct PyModuleDef moduledef = {
...
}
...
PyObject * PyInit__rpm(void)
{
PyObject *m = PyModule_Create(&moduledef);
if (m == NULL || initModule(m) == 0) {
Py_XDECREF(m);
m = NULL;
}
return m;
}
#else
void init_rpm(void)
{
PyObject *m = Py_InitModule3("_rpm", rpmModuleMethods, rpm__doc__);
if (m) initModule(m);
return;
}
#endif
- Panu -
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [PATCH 05/15] Convert all usage of two-expression exception raising to modern syntax
On Thu, 15 Oct 2009, David Malcolm wrote:
Replace all usage of the two-expression
raise foo, 'bar'
syntax with the long-equivalent
raise foo('bar')
as the old syntax was removed in Python 3 (PEP 3109), and the new syntax
is supported in Python 2
Right, this works as far back as I can test (Python 2.3) so no problems
there. Applied.
- Panu -
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [PATCH 04/15] Convert relative imports within rpm package to absolute imports
On Thu, 15 Oct 2009, David Malcolm wrote: --- python/rpm/__init__.py|9 - python/rpm/transaction.py |2 +- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/python/rpm/__init__.py b/python/rpm/__init__.py index 49a4c77..3cf59f7 100644 --- a/python/rpm/__init__.py +++ b/python/rpm/__init__.py @@ -6,10 +6,9 @@ This module enables you to manipulate rpms and the rpm database. import warnings import os -from _rpm import * -from transaction import * - -import _rpm +from rpm._rpm import * +from rpm.transaction import * +import rpm._rpm as _rpm Well this is a welcome change in Python, I haven't been closely following Pythons development in recent years and this has gone unnoticed by me. AFAICT this is only supported in >= 2.5, and in 2.5 you need __future__ import to enable it. But ok, 2.5 is probably quite a reasonable compatibility cut-off point for the next non-maintenance rpm release. - Panu - ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [PATCH 03/15] Generalize type object initialization to work with both Python 2.* and Python 3.*
On Thu, 15 Oct 2009, David Malcolm wrote: The layout of PyVarObject changed between python 2 and python 3, and this leads to the existing code for all of the various PyTypeObject initializers failing to compile with python 3 Change the way we initialize these structs to use PyVarObject_HEAD_INIT directly, rather than merely PyObject_HEAD_INIT, so that it compiles cleanly with both major versions of Python Same as with Py_TYPE() - breaks Python < 2.6 but easily worked around so applied. - Panu - ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [PATCH 02/15] Generalize access to ob_type so that they work with both Python 2.* and Python 3.*
On Thu, 15 Oct 2009, David Malcolm wrote: Python 2's various object structs use macros to implement common fields at the top of each struct. Python 3's objects instead embed a PyObject struct as the first member within the more refined object structs. Use the Py_TYPE() macro when accessing ob_type in order to encapsulate this difference. This already breaks anything older than Python 2.6, but of course easily fixed by providing a compatibility macro. Applied. - Panu - ___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [PATCH 01/15] Generalize python configuration code to support both Python 2.* and Python 3.*
On Sat, 17 Oct 2009, David Malcolm wrote:
On Fri, 2009-10-16 at 20:05 +0300, Ville Skyttä wrote:
On Thursday 15 October 2009, David Malcolm wrote:
> +WITH_PYTHON_INCLUDE=`${PYTHON} -c 'from distutils.sysconfig import *;
import sys; sys.stdout.write("%s\n" % get_python_inc())'`
I think the last statement could be just "sys.stdout.write(get_python_inc())",
no need for the newline.
Thanks; yes, that's cleaner. I was trying to maintain the newline
implicitly-generated by "print", but it gets thrown away outside the
python code.
Yup, for these purposes the newline makes no difference. Applied now in
HEAD as per Ville's suggestion and also changed the various macros using
python's print to use sys.stdout.write() instead.
- Panu -___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [PATCH 00/15] first pass at Python 3 bindings for RPM
On Thu, 15 Oct 2009, David Malcolm wrote:
I've been investigating adding Python 3 support to librpm's python
bindings, and in a "release early" spirit, the following sequence of
patches is what I've got so far.
Nice, and nice timing too as the python bindings are in the a middle of a
serious spring-clean/facelift.
There are plenty of FIXMEs and hairy issues, so I wanted to get some
feedback before diving in further.
According to configure.ac, "rpm-python is based on python-2.5, with
legacy hacks to allow building against python >= 2.3". I'm seeking to
generalize the code to compile against both 2.* and 3.*, so that it's
"based on python-2.6 and python-3.1, with legacy hacks to allow building
against python >= 2.3"
Ultimately, I want to be able to build both python 2 and python 3
support during the same build, using the same source. But I think it's
easier to start by making sure that the source can be built against
either major-release of Python.
So these patches merely generalize the existing code so that it can be
built against both python 2.6 and python 3.1
So far, I'm only testing this by hand, against 2.6.3 and 3.1.1 (on a
Fedora 11 box). I suspect that these patches break the build on Python
earlier than 2.6, due to the uses of the "Bytes" macro.
Supporting Python 2.3 (which is pretty ancient by now) is by no means a
hard requirement, it's just that there hasn't been any strong reason to
require anything newer. I've for a while suspected that in order to
semi-sanely support both Python 2.x and 3.x, something has got to give.
All the major distros are already at Python 2.6 so I dont find it
unreasonable to bump up the requirements to Python 2.6 if that's what it
takes to support Python 3.x too.
I'm uneasy with purely manual testing of this code. Is there an
existing unittest suite for the rpm-python code?
Unfortunately no. And yes a test-suite for it is desperately needed.
With these patches I can do this:
[da...@brick rpm]$
PYTHONPATH=/home/david/coding/python3/rpm-python-bindings/install-prefix/lib/python3.1/site-packages
python3
Python 3.1.1 (r311:74480, Oct 1 2009, 12:20:21)
[GCC 4.4.1 20090725 (Red Hat 4.4.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import rpm
/home/david/coding/python3/rpm-python-bindings/install-prefix/lib/python3.1/site-packages/rpm/__init__.py:9:
DeprecationWarning: Type rpm.hdr defines tp_reserved (formerly tp_compare) but
not tp_richcompare. Comparisons may not behave as intended.
from rpm._rpm import *
# (clearly I need to fix this)
rpm.addMacro("_dbpath", "/var/lib/rpm")
ts = rpm.TransactionSet()
mi = ts.dbMatch()
for h in mi:
... print("%s-%s-%s" % (h['name'], h['version'], h['release'])) # note
that print has become a function
b'im-chooser'-b'1.2.6'-b'3.fc11'
(etc)
This highlights one of the big unknowns to me at the moment: what
guarantees (if any) are there as to the encoding of the bytes in a value
of RPM_STRING_CLASS? Currently in python/rpmtd-py.c:rpmtd_ItemAsPyobj I
punt the issue by returning them as Bytes objects, and hence you see the
bytes objects in the output. So as written, users of the API would have
to deal with decoding these back to py3k's unicode strings.
There are no guarantees whatsoever :-/ The spec file has no concept of
encoding so the headers end up containing whatever was used in the spec,
even mixtures of encodings within a package. See http://rpm.org/ticket/30,
this is a long-standing issue but so far nothing concrete has been done
about it. And even if/when rpm starts requiring UTF-8 encoding, anything
dealing with rpms will have to deal with legacy packages without such
a guarantee for a long, long time.
Further comments to the individual patches to follow...
- Panu -
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
