I've been following the guidelines for Python packaging found here:
https://fedoraproject.org/wiki/Packaging:Python
in particular the cookbook for supporting both Py2 and Py3.
I've discovered two places where things fail to work as expected with
respect to script installation. Both of these problems took an
inordinately long time to fully diagnose. The first problem I believe is
a bug in the python rpm macros and needs to be fixed (filed as bug
#1335203). The second problem can be fixed via a workaound in the spec
file, but the cookbook recipe will need to be updated to call attention
to it.
Problem 1:
----------
If the script includes an interpreter argument in it's shebang line the
installed script cannot execute and aborts. For example if the source
script has a shebang line like this:
#!/usr/bin/python -E
The installed script will have a shebang line like this (using Py2 as an
example)
#!/usr/bin/python2 -s -E
The problem is how the shebang line is parsed to build parameters for
exec. Unfortunately there is no standard. Some operating systems split
at whitespace and treat the first part as the path to the interpreter
and the rest as individual arguments. Some operating systems split at
the first whitespace and treat the front part as the path to the
interpeter and the rest as a single argument (this is what happens in
Linux).
Thus when the Python interpreter is run it sees argv[1] as "-s -E" which
it doesn't understand as a single argument and the result is:
Unknown option: -
usage: /usr/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.
The cause of the problem is in /usr/lib/rpm/macros.d/macros.python2 and
/usr/lib/rpm/macros.d/macros.python3. I'll show the Py2 version but the
Py3 version is equivalent.
%py2_shbang_opts -s
%py2_build() %{expand:\
CFLAGS="%{optflags}" %{__python2} %{py_setup} %{?py_setup_args} build
--executable="%{__python2} %{py2_shbang_opts}" %{?1}\
}
Then in distutils/command/build_scripts.py it uses the first_line_re
regular expression to split the shebang line into 2 parts, the
interpreter and the interpreter arguments. It then replaces the shebang
line with the value of --executable followed by a space and interpreter
arguments the regexp found. But in our case the interpreter is *not* the
path to an interpreter, rather it's the path to the interpreter *plus*
the %py2_shbang_opts. Thus you end up with multiple arguments to the
interpreter, but the interpreter doesn't see it as multiple arguments,
instead it sees 1 argument, a single string containing "-s -E".
I think the fundamental problem is that macros.python{2,3} is abusing
the --executable argument, it's meant to be the path to an executable
*only*. It clearly does not expect extra arguments.
If build_scripts.py had logic to merge arguments together, e.g. -s -E
would become -sE then all would be good, but it doesn't.
Problem 2:
----------
The Py2 version of the script (which forces /usr/bin/python2 in it's
shebang line) is installed in the python3 package. Thus if you install
python3-NVR and it has a script in /usr/bin you'll end up executing
python2 instead of python3.
The cookbook shows this snippet:
%install
# Must do the python2 install first because the scripts in /usr/bin are
# overwritten with every setup.py install, and in general we want the
# python3 version to be the default.
%py2_install
%py3_install
The problem is this is exactly the *opposite* of what actually happens
if builds are fast. %py2_install runs first and copies the Py2 version
of the script into $BUILDROOT/usr/bin. Next %py3_install runs and an
attempt is made to install the Py3 version into the exact same location
$BUILDROOT/usr/bin. However distutils has logic which avoids copying
over an existing installed file unless the src is newer. This occurs in
distutils/file_util.py which has this code in copy_file():
if update and not newer(src, dst):
if verbose >= 1:
log.debug("not copying %s (output up-to-date)", src)
return dst, 0
The documentation for newer() in distutils/dep_util.py acknowledges the
fundamental problem:
Note that this test is not very accurate: files created in the
same second will have the same "age".
This is exactly what was happening. %py2_build and %py3_build each
created the script with the same timestamp and the logic in copy_file()
concluded it shouldn't replace the Py2 script with the Py3 script.
The result is the python3-package ended up with a script with a Py2
interpreter shebang line, clearly this is wrong.
The workaround I came up with is to delay the execution of %py3_build by
at least 1 second by inserting a sleep in-between the %py2_build and
%py3_build macros in the spec file, like this:
%py2_build
sleep 1
%py3_build
Actually I think sleeping 2 seconds might be safer given the 1 second
resolution on the timestamps.
BTW, the fact the file was not updated is silent in the output of the
setup.py command. Note the message is contained in a log.debug, not a
log.info. I could find no way to enable debug log output using the
command line arguments. I had to resort to adding the following in my
setup.py script in order to see the message:
import distutils.log
distutils.log.set_verbosity(distutils.log.DEBUG) # Set DEBUG level
--
John
_______________________________________________
python-devel mailing list
python-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org
