Re: [scl.org] Python3 (always latest) community SCL

[Nick Coghlan]

On Wed, May 3, 2017 at 9:44 PM, Tomas Orsava  wrote:
> On 05/03/2017 09:13 AM, Nick Coghlan wrote:
>>> On 04/11/2017 10:24 AM, Nick Coghlan wrote:
>> While I thought this sounded reasonable at the time, it turns out to
>> have a lot of problems in practice, as it makes it hard for developers
>> to say "I just want to run on a pre-built version of the latest
>> upstream Python". Instead, they have to choose between:
>>
>> * referring to python3.x in their own code, and having to update all
>> those references to switch to a new version
>> * keeping their "python3" references, and trusting that the originally
>> planned EPEL transitions will happen in a timely fashion
>
>
> I agree that it's not exactly seamless, you can make a single boolean macro
> in your spec file using which it will be switched to the alternative Python,
> so it shouldn't be a significant maintenance burden.

For the use cases I'm interested in, there's no application level spec
file - just a container definition or Ansible deployment script.

SCLs are really useful for those cases, since it's relatively
straightforward to ensure the SCL is enabled at the appropriate times,
while RPM macros aren't available.

>> And then regardless of the approach they choose, they have to rely on
>> either virtualenv or fiddling with the system level symlink to run
>> against an alternative Python 3 stack.
>
> If I may ask, what do you mean by fiddling with the system level symlink? I
> believe they can invoke `/usr/bin/python3.X` using the %{__python3_other}
> macro.

Choosing a stack other than the default one at runtime (rather than
when building an RPM). I'm not suggesting changing the symlink would
be a good idea, just pointing out it's basically the only option left
if you're not using either SCLs or a Python level venv and aren't
willing to qualify every Python 3 invocation with a specific minor
version.

>> That approach is then a lot closer to the traditional RPM update flow
>> from upstream->Fedora->RHEL->CentOS, just modified to be
>> upstream->SCLo->RHSCL->EPEL since there isn't a system Python 3 stack
>> in RHEL & CentOS.
>
> I do see the benefits of the added rolling SCL, though in my mind the
> benefits are lessened by the pre-existence of Python 3 in EPEL, and thus I'm
> not sure it's worth the added maintenance. However, that's only my personal
> reasoning.
>
> If we can find volunteers for the effort I'll be glad to lend a hand with
> the creation of the SCL.

I'd like to see this happen enough to volunteer to maintain it myself,
so I'll start working through the checklist that Honza posted :)

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Platform Engineering, Brisbane

___
SCLorg mailing list
SCLorg@redhat.com
https://www.redhat.com/mailman/listinfo/sclorg

Re: [scl.org] Python3 (always latest) community SCL

[Tomas Orsava]

On 05/03/2017 09:13 AM, Nick Coghlan wrote:

On Fri, Apr 14, 2017 at 12:27 AM, Tomas Orsava  wrote:

Hi!

On 04/11/2017 10:24 AM, Nick Coghlan wrote:

I've been mulling this idea over for the past couple of weeks, and I'm
wondering if it might make sense to create a rolling "sclo-python3"
SCL, that's initially forked from
https://www.softwarecollections.org/en/scls/rhscl/rh-python35/, but
explicitly promises to rebase to new Python feature releases when they
come out.

So if people were happy to always run on the leading edge (even for
X.Y.0 releases), they could use "sclo-python3", but if they wanted to
stay on a particular X.Y release for a while, they would need to
switch to the downstream rh-pythonXY SCLs.

Remi, if I wanted to do that, where would I start?
https://github.com/sclorg-distgit is useful as a reference for
submitting changes to existing community SCLs, but it doesn't provide
any guidance on how to start a new one (and that info is also missing
from the wiki).

It would be great to have a state-of-the-art Python in Enterprise Linux, but
I think it would be better using the existing (though lacking maintenance)
Python 3 in EPEL [0] mechanism.

While installing and using SCLs isn't hard, I think an RPM packaged version
is still easier for both maintenance and usage and people can have EPEL
packages built against it as well. In addition the transition mechanism is
smoother, as two Python versions are coexisting during the  transition
period, whereas the rolling SCL would just switch and everyone would have to
immediately adjust.

What would be the advantage of creating a rolling 'scl-python3' collection
over the EPEL mechanism [1]?

The main practical issue I see with the EPEL parallel installation
design is the part about updating "/usr/bin/python3" to new "stable"
versions:

==
In a situation when python3X is in EPEL and 3.X+1 is released
upstream, the following happens:

* python3X+1 package is created for EPEL ASAP and all extension
packages are built also for this new python3X+1 stack.
* When all packages are rebuilt for python3X+1, the old python3X stack
is retired after certain period. This period gives everyone enough
time to rebuild their packages while being as short as possible. There
is intentionally no hard limit here, we will approach this case by
case.
* "/usr/bin/python3" belongs to the "stable" python3X stack. Switching
/usr/bin/python3 from python3X to python3X+1 happens shortly before
the end of transitional period (== before obsoleting python3X) and it
is announced on epel-devel.
Usage of "/usr/bin/python3" is discouraged in favour of using
/usr/bin/python3.X explicitly.
==

While I thought this sounded reasonable at the time, it turns out to
have a lot of problems in practice, as it makes it hard for developers
to say "I just want to run on a pre-built version of the latest
upstream Python". Instead, they have to choose between:

* referring to python3.x in their own code, and having to update all
those references to switch to a new version
* keeping their "python3" references, and trusting that the originally
planned EPEL transitions will happen in a timely fashion


I agree that it's not exactly seamless, you can make a single boolean 
macro in your spec file using which it will be switched to the 
alternative Python, so it shouldn't be a significant maintenance burden.



And then regardless of the approach they choose, they have to rely on
either virtualenv or fiddling with the system level symlink to run
against an alternative Python 3 stack.


If I may ask, what do you mean by fiddling with the system level 
symlink? I believe they can invoke `/usr/bin/python3.X` using the 
%{__python3_other} macro.



Accordingly, the main benefit I see to the rolling community SCL is
that it makes "python3" load the SCL version when the SCL is enabled,
which *does* make it easy for developers to choose between running
against either the latest Python 3.x (by building on top of either
Fedora or the proposed community SCL), or against a specific Python
3.x version (by using the Red Hat SCLs). If the native EPEL builds are
retained, then they would be a *downstream* of the Red Hat SCLs, so
the complete maintenance flow would be:

- upstream release happens
- Fedora system Python and rolling Python 3 community SCL are updated
- Python 3.x Red Hat SCLs are updated
- EPEL Python 3.x stacks are updated

That approach is then a lot closer to the traditional RPM update flow
from upstream->Fedora->RHEL->CentOS, just modified to be
upstream->SCLo->RHSCL->EPEL since there isn't a system Python 3 stack
in RHEL & CentOS.


I do see the benefits of the added rolling SCL, though in my mind the 
benefits are lessened by the pre-existence of Python 3 in EPEL, and thus 
I'm not sure it's worth the added maintenance. However, that's only my 
personal reasoning.


If we can find volunteers for the effort I'll be glad to lend a hand 
with the creation of the 

Re: [scl.org] Python3 (always latest) community SCL

[Nick Coghlan]

On Fri, Apr 14, 2017 at 12:27 AM, Tomas Orsava  wrote:
> Hi!
>
> On 04/11/2017 10:24 AM, Nick Coghlan wrote:
>>
>> I've been mulling this idea over for the past couple of weeks, and I'm
>> wondering if it might make sense to create a rolling "sclo-python3"
>> SCL, that's initially forked from
>> https://www.softwarecollections.org/en/scls/rhscl/rh-python35/, but
>> explicitly promises to rebase to new Python feature releases when they
>> come out.
>>
>> So if people were happy to always run on the leading edge (even for
>> X.Y.0 releases), they could use "sclo-python3", but if they wanted to
>> stay on a particular X.Y release for a while, they would need to
>> switch to the downstream rh-pythonXY SCLs.
>>
>> Remi, if I wanted to do that, where would I start?
>> https://github.com/sclorg-distgit is useful as a reference for
>> submitting changes to existing community SCLs, but it doesn't provide
>> any guidance on how to start a new one (and that info is also missing
>> from the wiki).
>
> It would be great to have a state-of-the-art Python in Enterprise Linux, but
> I think it would be better using the existing (though lacking maintenance)
> Python 3 in EPEL [0] mechanism.
>
> While installing and using SCLs isn't hard, I think an RPM packaged version
> is still easier for both maintenance and usage and people can have EPEL
> packages built against it as well. In addition the transition mechanism is
> smoother, as two Python versions are coexisting during the  transition
> period, whereas the rolling SCL would just switch and everyone would have to
> immediately adjust.
>
> What would be the advantage of creating a rolling 'scl-python3' collection
> over the EPEL mechanism [1]?

The main practical issue I see with the EPEL parallel installation
design is the part about updating "/usr/bin/python3" to new "stable"
versions:

==
In a situation when python3X is in EPEL and 3.X+1 is released
upstream, the following happens:

* python3X+1 package is created for EPEL ASAP and all extension
packages are built also for this new python3X+1 stack.
* When all packages are rebuilt for python3X+1, the old python3X stack
is retired after certain period. This period gives everyone enough
time to rebuild their packages while being as short as possible. There
is intentionally no hard limit here, we will approach this case by
case.
* "/usr/bin/python3" belongs to the "stable" python3X stack. Switching
/usr/bin/python3 from python3X to python3X+1 happens shortly before
the end of transitional period (== before obsoleting python3X) and it
is announced on epel-devel.
Usage of "/usr/bin/python3" is discouraged in favour of using
/usr/bin/python3.X explicitly.
==

While I thought this sounded reasonable at the time, it turns out to
have a lot of problems in practice, as it makes it hard for developers
to say "I just want to run on a pre-built version of the latest
upstream Python". Instead, they have to choose between:

* referring to python3.x in their own code, and having to update all
those references to switch to a new version
* keeping their "python3" references, and trusting that the originally
planned EPEL transitions will happen in a timely fashion

And then regardless of the approach they choose, they have to rely on
either virtualenv or fiddling with the system level symlink to run
against an alternative Python 3 stack.

Accordingly, the main benefit I see to the rolling community SCL is
that it makes "python3" load the SCL version when the SCL is enabled,
which *does* make it easy for developers to choose between running
against either the latest Python 3.x (by building on top of either
Fedora or the proposed community SCL), or against a specific Python
3.x version (by using the Red Hat SCLs). If the native EPEL builds are
retained, then they would be a *downstream* of the Red Hat SCLs, so
the complete maintenance flow would be:

- upstream release happens
- Fedora system Python and rolling Python 3 community SCL are updated
- Python 3.x Red Hat SCLs are updated
- EPEL Python 3.x stacks are updated

That approach is then a lot closer to the traditional RPM update flow
from upstream->Fedora->RHEL->CentOS, just modified to be
upstream->SCLo->RHSCL->EPEL since there isn't a system Python 3 stack
in RHEL & CentOS.

Cheers,
Nick.




Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Platform Engineering, Brisbane

___
SCLorg mailing list
SCLorg@redhat.com
https://www.redhat.com/mailman/listinfo/sclorg