Re: Problems with script installation in RPM's
On 05/13/2016 03:43 PM, Jason L Tibbitts III wrote: FYI, an updated python macros package is in Rawhide which includes the sleep. Thank you. I don't think manually deleting some random at a special place in the spec is a proper solution. +1 setuptools should really be either fixed upstream or patched in Fedora to fix this issue. It may not be able to depend on subsecond timestamps for general portability, but Fedora certainly can (and should) make it do so. Well you learn something new everyday, I didn't realize there was support for subsecond timestamps. Here are some snippets from the stat(2) man page: Since kernel 2.5.48, the stat structure supports nanosecond resolution for the three file timestamp fields. Nanosecond timestamps are supported on XFS, JFS, Btrfs, and ext4 (since Linux 2.6.23). Nanosecond timestamps are not supported in ext2, ext3, and Reiserfs. On filesystems that do not support subsecond timestamps, the nanosecond fields are returned with the value 0. Given the fact you can't guarantee nanosecond timestamps since it's dependent upon the file system I don't know how you could reliably address the issue of files created/modified sequentially having the same timestamp. Seems to me it needs some other solution. -- John ___ python-devel mailing list python-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org
Re: Problems with script installation in RPM's
FYI, an updated python macros package is in Rawhide which includes the sleep. I don't think manually deleting some random at a special place in the spec is a proper solution. setuptools should really be either fixed upstream or patched in Fedora to fix this issue. It may not be able to depend on subsecond timestamps for general portability, but Fedora certainly can (and should) make it do so. - J< ___ python-devel mailing list python-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org
Re: Problems with script installation in RPM's
On Thu, May 12, 2016 at 12:26:45PM -0400, John Dennis wrote: > On 05/12/2016 11:24 AM, Tomas Orsava wrote: > > That seems like the best argument and solution, I'll amend the Python > > RPM Porting Guide [PyRPG] accordingly. > > > > [PyRPG] http://python-rpm-porting.readthedocs.io/ > > Well, it looks like the doc needs updating in 2 places then because I > initially referenced https://fedoraproject.org/wiki/Packaging:Python. I > didn't even know of the existence of RPM Porting Guide. > > Maybe it's not a good idea to have 2 different documents that have to be > kept in sync. Maybe one should be definitive and the other reference it. The main difference is that Packaging:Python wikipage is just a list of Fedora packaging guidelines for Python, which fedora python package maintainers have to follow. It's not a howto or a guide. Moreover to edit it, one have to open a FPC ticket and get it through voting during FPC meeting. On the other hand, python-rpm-porting page looks more like a howto for a particular topic: python2 to python3 transition, which actually references the definitive document - Packaging:Python wikipage. -- Martin Bukatovic ___ python-devel mailing list python-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org
Re: Problems with script installation in RPM's
On 05/12/2016 11:24 AM, Tomas Orsava wrote: On 05/12/2016 05:16 PM, Miro Hrončok wrote: On 12.5.2016 15:24, John Dennis wrote: On 05/11/2016 11:54 AM, Petr Viktorin wrote: On 05/11/2016 05:16 PM, John Dennis wrote: The workaround I came up with is to delay the execution of %py3_build by at least 1 second by inserting a sleep in-between the %py2_build and %py3_build macros in the spec file, like this: %py2_build sleep 1 %py3_build Actually I think sleeping 2 seconds might be safer given the 1 second resolution on the timestamps. A better workaround would be 'rm /usr/bin/xyz' instead of the sleep. I understand where you're coming from Petr but let me offer a reason for why a short sleep might be preferable. In the interest of robustness it's best to avoid requiring someone to enumerate exceptions. File lists change as do other aspects of the build. It's very easy to miss enumerating every necessary exception on every update to the spec file. If a simple short sleep is sufficient to avoid manual enumeration then the minor inefficiency of the sleep seems a price well paid. Well, on the other hand, if setuptools/distutils decides that it will NEVER override a file, sleep stops working, rm works. With rm, what's happening is crystal clear. With sleep, not so much. rm is more explicit, explicit is better than implicit. That seems like the best argument and solution, I'll amend the Python RPM Porting Guide [PyRPG] accordingly. [PyRPG] http://python-rpm-porting.readthedocs.io/ Well, it looks like the doc needs updating in 2 places then because I initially referenced https://fedoraproject.org/wiki/Packaging:Python. I didn't even know of the existence of RPM Porting Guide. Maybe it's not a good idea to have 2 different documents that have to be kept in sync. Maybe one should be definitive and the other reference it. -- John ___ python-devel mailing list python-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org
Re: Problems with script installation in RPM's
On 05/12/2016 05:16 PM, Miro Hrončok wrote: On 12.5.2016 15:24, John Dennis wrote: On 05/11/2016 11:54 AM, Petr Viktorin wrote: On 05/11/2016 05:16 PM, John Dennis wrote: The workaround I came up with is to delay the execution of %py3_build by at least 1 second by inserting a sleep in-between the %py2_build and %py3_build macros in the spec file, like this: %py2_build sleep 1 %py3_build Actually I think sleeping 2 seconds might be safer given the 1 second resolution on the timestamps. A better workaround would be 'rm /usr/bin/xyz' instead of the sleep. I understand where you're coming from Petr but let me offer a reason for why a short sleep might be preferable. In the interest of robustness it's best to avoid requiring someone to enumerate exceptions. File lists change as do other aspects of the build. It's very easy to miss enumerating every necessary exception on every update to the spec file. If a simple short sleep is sufficient to avoid manual enumeration then the minor inefficiency of the sleep seems a price well paid. Well, on the other hand, if setuptools/distutils decides that it will NEVER override a file, sleep stops working, rm works. With rm, what's happening is crystal clear. With sleep, not so much. rm is more explicit, explicit is better than implicit. That seems like the best argument and solution, I'll amend the Python RPM Porting Guide [PyRPG] accordingly. [PyRPG] http://python-rpm-porting.readthedocs.io/ ___ python-devel mailing list python-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org
Re: Problems with script installation in RPM's
On 12.5.2016 15:24, John Dennis wrote: On 05/11/2016 11:54 AM, Petr Viktorin wrote: On 05/11/2016 05:16 PM, John Dennis wrote: The workaround I came up with is to delay the execution of %py3_build by at least 1 second by inserting a sleep in-between the %py2_build and %py3_build macros in the spec file, like this: %py2_build sleep 1 %py3_build Actually I think sleeping 2 seconds might be safer given the 1 second resolution on the timestamps. A better workaround would be 'rm /usr/bin/xyz' instead of the sleep. I understand where you're coming from Petr but let me offer a reason for why a short sleep might be preferable. In the interest of robustness it's best to avoid requiring someone to enumerate exceptions. File lists change as do other aspects of the build. It's very easy to miss enumerating every necessary exception on every update to the spec file. If a simple short sleep is sufficient to avoid manual enumeration then the minor inefficiency of the sleep seems a price well paid. Well, on the other hand, if setuptools/distutils decides that it will NEVER override a file, sleep stops working, rm works. With rm, what's happening is crystal clear. With sleep, not so much. rm is more explicit, explicit is better than implicit. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ python-devel mailing list python-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org
Re: Problems with script installation in RPM's
On 05/11/2016 11:54 AM, Petr Viktorin wrote: On 05/11/2016 05:16 PM, John Dennis wrote: The workaround I came up with is to delay the execution of %py3_build by at least 1 second by inserting a sleep in-between the %py2_build and %py3_build macros in the spec file, like this: %py2_build sleep 1 %py3_build Actually I think sleeping 2 seconds might be safer given the 1 second resolution on the timestamps. A better workaround would be 'rm /usr/bin/xyz' instead of the sleep. I understand where you're coming from Petr but let me offer a reason for why a short sleep might be preferable. In the interest of robustness it's best to avoid requiring someone to enumerate exceptions. File lists change as do other aspects of the build. It's very easy to miss enumerating every necessary exception on every update to the spec file. If a simple short sleep is sufficient to avoid manual enumeration then the minor inefficiency of the sleep seems a price well paid. -- John ___ python-devel mailing list python-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org
Re: Problems with script installation in RPM's
On 05/11/2016 06:48 PM, Jason L Tibbitts III wrote: >> "PV" == Petr Viktorin writes: > > PV> Unfortunately, changing the Guidelines isn't trivial – we have a > PV> ticket that's been sitting in FPC's queue for half a year [1]... > > Thanks for the snark. My apologies, reading this now this was unnecessarily hostile. > Nobody has submitted a draft and that ticket isn't particularly > urgent The original request in that ticket was the result of a > misunderstanding, so I'm not sure what you believe is currently wrong > besides the fact that some people get confused. Thanks for explaining your view of the state of the ticket; with this we can start moving again. It's mainly the fact that some people get confused, yes. Given that that particular ticket has grown to handling the apps/libraries distinction (even beyond Python), I guess it would be better for us to file a new ticket with a concrete draft. > Sadly it appears that distutils is broken, because it should really be > trivial to just call stat. I guess we'll just have to build in a sleep > somewhere; a second shouldn't be a big deal. Or someone could fix > distutils to actually work properly. I'll see what I can do :) -- Petr Viktorin ___ python-devel mailing list python-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org
Re: Problems with script installation in RPM's
> "PV" == Petr Viktorin writes: PV> Unfortunately, changing the Guidelines isn't trivial – we have a PV> ticket that's been sitting in FPC's queue for half a year [1]... Thanks for the snark. Nobody has submitted a draft and that ticket isn't particularly urgent The original request in that ticket was the result of a misunderstanding, so I'm not sure what you believe is currently wrong besides the fact that some people get confused. Sadly it appears that distutils is broken, because it should really be trivial to just call stat. I guess we'll just have to build in a sleep somewhere; a second shouldn't be a big deal. Or someone could fix distutils to actually work properly. - J< ___ python-devel mailing list python-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org
Re: Problems with script installation in RPM's
On 05/11/2016 05:16 PM, John Dennis wrote: > I've been following the guidelines for Python packaging found here: > > https://fedoraproject.org/wiki/Packaging:Python > > in particular the cookbook for supporting both Py2 and Py3. > > I've discovered two places where things fail to work as expected with > respect to script installation. Both of these problems took an > inordinately long time to fully diagnose. The first problem I believe is > a bug in the python rpm macros and needs to be fixed (filed as bug > #1335203). The second problem can be fixed via a workaound in the spec > file, but the cookbook recipe will need to be updated to call attention > to it. Could you file bugs, please? [...] > Problem 2: > -- > > The Py2 version of the script (which forces /usr/bin/python2 in it's > shebang line) is installed in the python3 package. Thus if you install > python3-NVR and it has a script in /usr/bin you'll end up executing > python2 instead of python3. > > The cookbook shows this snippet: > > %install > # Must do the python2 install first because the scripts in /usr/bin are > # overwritten with every setup.py install, and in general we want the > # python3 version to be the default. > %py2_install > %py3_install > [...] > > The result is the python3-package ended up with a script with a Py2 > interpreter shebang line, clearly this is wrong. > > The workaround I came up with is to delay the execution of %py3_build by > at least 1 second by inserting a sleep in-between the %py2_build and > %py3_build macros in the spec file, like this: > > %py2_build > sleep 1 > %py3_build > > Actually I think sleeping 2 seconds might be safer given the 1 second > resolution on the timestamps. A better workaround would be 'rm /usr/bin/xyz' instead of the sleep. Nevertheless, this should indeed be acknowledged in both the Packaging guidelines, and in the RPM porting guide [0]. Tomáš, can you take care of the guide? Unfortunately, changing the Guidelines isn't trivial – we have a ticket that's been sitting in FPC's queue for half a year [1]... [0] http://python-rpm-porting.readthedocs.io/en/latest/index.html [1] https://fedorahosted.org/fpc/ticket/558 -- Petr Viktorin ___ python-devel mailing list python-devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org
Problems with script installation in RPM's
I've been following the guidelines for Python packaging found here:
https://fedoraproject.org/wiki/Packaging:Python
in particular the cookbook for supporting both Py2 and Py3.
I've discovered two places where things fail to work as expected with
respect to script installation. Both of these problems took an
inordinately long time to fully diagnose. The first problem I believe is
a bug in the python rpm macros and needs to be fixed (filed as bug
#1335203). The second problem can be fixed via a workaound in the spec
file, but the cookbook recipe will need to be updated to call attention
to it.
Problem 1:
--
If the script includes an interpreter argument in it's shebang line the
installed script cannot execute and aborts. For example if the source
script has a shebang line like this:
#!/usr/bin/python -E
The installed script will have a shebang line like this (using Py2 as an
example)
#!/usr/bin/python2 -s -E
The problem is how the shebang line is parsed to build parameters for
exec. Unfortunately there is no standard. Some operating systems split
at whitespace and treat the first part as the path to the interpreter
and the rest as individual arguments. Some operating systems split at
the first whitespace and treat the front part as the path to the
interpeter and the rest as a single argument (this is what happens in
Linux).
Thus when the Python interpreter is run it sees argv[1] as "-s -E" which
it doesn't understand as a single argument and the result is:
Unknown option: -
usage: /usr/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.
The cause of the problem is in /usr/lib/rpm/macros.d/macros.python2 and
/usr/lib/rpm/macros.d/macros.python3. I'll show the Py2 version but the
Py3 version is equivalent.
%py2_shbang_opts -s
%py2_build() %{expand:\
CFLAGS="%{optflags}" %{__python2} %{py_setup} %{?py_setup_args} build
--executable="%{__python2} %{py2_shbang_opts}" %{?1}\
}
Then in distutils/command/build_scripts.py it uses the first_line_re
regular expression to split the shebang line into 2 parts, the
interpreter and the interpreter arguments. It then replaces the shebang
line with the value of --executable followed by a space and interpreter
arguments the regexp found. But in our case the interpreter is *not* the
path to an interpreter, rather it's the path to the interpreter *plus*
the %py2_shbang_opts. Thus you end up with multiple arguments to the
interpreter, but the interpreter doesn't see it as multiple arguments,
instead it sees 1 argument, a single string containing "-s -E".
I think the fundamental problem is that macros.python{2,3} is abusing
the --executable argument, it's meant to be the path to an executable
*only*. It clearly does not expect extra arguments.
If build_scripts.py had logic to merge arguments together, e.g. -s -E
would become -sE then all would be good, but it doesn't.
Problem 2:
--
The Py2 version of the script (which forces /usr/bin/python2 in it's
shebang line) is installed in the python3 package. Thus if you install
python3-NVR and it has a script in /usr/bin you'll end up executing
python2 instead of python3.
The cookbook shows this snippet:
%install
# Must do the python2 install first because the scripts in /usr/bin are
# overwritten with every setup.py install, and in general we want the
# python3 version to be the default.
%py2_install
%py3_install
The problem is this is exactly the *opposite* of what actually happens
if builds are fast. %py2_install runs first and copies the Py2 version
of the script into $BUILDROOT/usr/bin. Next %py3_install runs and an
attempt is made to install the Py3 version into the exact same location
$BUILDROOT/usr/bin. However distutils has logic which avoids copying
over an existing installed file unless the src is newer. This occurs in
distutils/file_util.py which has this code in copy_file():
if update and not newer(src, dst):
if verbose >= 1:
log.debug("not copying %s (output up-to-date)", src)
return dst, 0
The documentation for newer() in distutils/dep_util.py acknowledges the
fundamental problem:
Note that this test is not very accurate: files created in the
same second will have the same "age".
This is exactly what was happening. %py2_build and %py3_build each
created the script with the same timestamp and the logic in copy_file()
concluded it shouldn't replace the Py2 script with the Py3 script.
The result is the python3-package ended up with a script with a Py2
interpreter shebang line, clearly this is wrong.
The workaround I came up with is to delay the execution of %py3_build by
at least 1 second by inserting a sleep in-between the %py2_build and
%py3_build macros in the spec file, like this:
%py2_build
sleep 1
%py3_build
Actually I think sleeping 2 seconds might be safer given the 1 second
resolution on the timestamps.
BTW, the fact the f
