Re: State of python3 in our infrastructure

[Kevin Fenzi]

On Fri, 4 Dec 2015 11:27:22 -0800
Brendan Conoboy  wrote:

> On 12/04/2015 10:26 AM, Pierre-Yves Chibon wrote:
> > On Fri, Dec 04, 2015 at 10:04:54AM -0800, Brendan Conoboy wrote:  
> >> RHEL currently provides python3 via SCL.  Is this insufficient for
> >> infrastructure's needs?  
> >
> > Python3 itself isn't the problem, it's the whole stack of
> > dependencies that we need in addition to python.
> > Packaging things for EPEL means it's available to everyone and
> > anyone can contribute/build upon it, we know how the system and the
> > packaging work, it's easy for us to add new components and fix
> > others. Packaging things for SCL means, it's nowhere but our infra
> > repo since we can't push things to RH's SCLs and SCLs are neither
> > in Fedora nor in EPEL, it's more work as we'll need to figure out
> > how SCLs work and how packaging for SCLs work and it makes it a
> > much more closed system from a contribution point of view.  
> 
> For context, my primary interest is there being a smooth course for 
> developers and users to move from Python 2 to 3, preferably *before*
> a new RHEL major release comes out.  If EPEL is meeting your needs 
> that's great, but if you need something from RHEL that'd be good to 
> know.  Thanks,

To expand on what Pierre said: 

* SCL's aren't allowed in use in Fedora/EPEL, so we would have to not
  use koji or our usual patterns to maintaining things. Nor would it be
  as easy for us to collaborate with others who have similar goals. 

* Mixing SCLs and RPMS is a mess and complex. 

As to getting folks to move to python3 before the next RHEL major
release, I am not sure. On one hand it would be great if there was a
python3 rpm provided and maintained by RHEL folks. On the other hand if
we maintain it in EPEL we might have more flexability to change it (ie,
less guarentees than RHEL might provide). So, I guess the best world
would be some RH/RHEL python folks helping maintain the EPEL python3x
version and building up the stack somewhat would be best for us. 

kevin



pgpp5L9ebREDO.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: State of python3 in our infrastructure

[Kevin Fenzi]

On Wed, 02 Dec 2015 16:05:00 -
Toshio くらとみ  wrote:

...snip...

I'll put in my 2cents here. ;) 

> I think starting to run python3 web apps will contribute more to the
> greater good than limiting apps to python2.  It will help to
> future-proof the web-apps since python2 probably won't be default in
> RHEL8 (who knows if it will be available as an add-on, though)... so
> at some point you do have to pay this price.  So the real question is
> whether you want to pay this price now and which people within
> infrastructure are going to be doing the work:
> 
> Do it when a RHEL releases with python3:
> * sysadmins will have to manage both RHEL7 and RHEL8 web servers for
> py2 and py3.
> * Concentrated work for developers to get applications ported as
> we'll want to migrate to rhel8 and python2 web apps will be holding
> us back.
> * Packagers will have to branch (and sometimes maintain) needed
> packages from Fedora to EPEL. 

I think waiting this long is a mistake... we should get started (well,
we already are really). 
 
> Do it now with the EPEL python34 (or a new python35) stack:
> * packagers will have to create python3x packages in EPEL
> * developers can start porting their apps now which gives them a
> longer window to port and perhaps less urgency 
> * sysadmins will have to manage RHEL7 with python2 and RHEL7 with
> python3 web servers
> 
> Do it now with Fedora:
> * developers can start porting their apps now which gives them a
> longer window to port and perhaps less urgency 
> * Packagers still have to get packages running on python3 but not as
> many due to efforts from Fedora itself.
> * sysadmins will have to manage RHEL7 with python2 and Fedora with
> python3 web servers.  Fedora has a different lifecycle so there's
> additional work involved here.

I'm somewhat torn between these two. I think it's going to depend on if
we can get some time to really make the python3 stack on epel7
workable. I know the folks who worked on it were waiting to land all
the 35 stuff in Fedora before looking at epel, but we likely need to
get that done soon and get it moved to 3.5 and see what things we need
there. 

If it's not going to be workable we can do Fedora, it just means more
work for us on the sysadmin side. 

One thing that feel through the cracks recently was that when we first
started doing non builder Fedora instances, I wanted them to auto
update. So we had yum-cron setup on them. However, new instances
haven't honored this and we need to move it to dnf-automatic anyhow. 
Doing that might help reduce the burden of updates flow, but might
also increase breakage. 
 
> Waiting for RHEL is probably the least overall work but there is a
> drawback there porting applications to python3 is probably what
> will take the most time so the sooner you can get started on that the
> sooner you'll be able to stop supporting python2.

Ultimately I think it's important for us to start moving to python3. If
we use Fedora or EPEL7 python3 I guess might be more up to the coders
working on that. 

kevin



pgpv7lmK_g1fW.pgp
Description: OpenPGP digital signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: State of python3 in our infrastructure

[Brendan Conoboy]

On 12/02/2015 08:05 AM, Toshio くらとみ wrote:
[snip]

Do it when a RHEL releases with python3:
* sysadmins will have to manage both RHEL7 and RHEL8 web servers for py2 and 
py3.
* Concentrated work for developers to get applications ported as we'll want to 
migrate to rhel8 and python2 web apps will be holding us back.
* Packagers will have to branch (and sometimes maintain) needed packages from 
Fedora to EPEL.


RHEL currently provides python3 via SCL.  Is this insufficient for 
infrastructure's needs?


[snip]

--
Brendan Conoboy / RHEL Development Coordinator / Red Hat, Inc.
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: State of python3 in our infrastructure

[Pierre-Yves Chibon]

On Fri, Dec 04, 2015 at 10:04:54AM -0800, Brendan Conoboy wrote:
> On 12/02/2015 08:05 AM, Toshio くらとみ wrote:
> [snip]
> >Do it when a RHEL releases with python3:
> >* sysadmins will have to manage both RHEL7 and RHEL8 web servers for py2 and 
> >py3.
> >* Concentrated work for developers to get applications ported as we'll want 
> >to migrate to rhel8 and python2 web apps will be holding us back.
> >* Packagers will have to branch (and sometimes maintain) needed packages 
> >from Fedora to EPEL.
> 
> RHEL currently provides python3 via SCL.  Is this insufficient for
> infrastructure's needs?

Python3 itself isn't the problem, it's the whole stack of dependencies that we
need in addition to python.
Packaging things for EPEL means it's available to everyone and anyone can
contribute/build upon it, we know how the system and the packaging work, it's
easy for us to add new components and fix others.
Packaging things for SCL means, it's nowhere but our infra repo since we can't
push things to RH's SCLs and SCLs are neither in Fedora nor in EPEL, it's more
work as we'll need to figure out how SCLs work and how packaging for SCLs work
and it makes it a much more closed system from a contribution point of view.


Pierre
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: State of python3 in our infrastructure

[Brendan Conoboy]

On 12/04/2015 10:26 AM, Pierre-Yves Chibon wrote:

On Fri, Dec 04, 2015 at 10:04:54AM -0800, Brendan Conoboy wrote:

On 12/02/2015 08:05 AM, Toshio くらとみ wrote:
[snip]

Do it when a RHEL releases with python3:
* sysadmins will have to manage both RHEL7 and RHEL8 web servers for py2 and 
py3.
* Concentrated work for developers to get applications ported as we'll want to 
migrate to rhel8 and python2 web apps will be holding us back.
* Packagers will have to branch (and sometimes maintain) needed packages from 
Fedora to EPEL.


RHEL currently provides python3 via SCL.  Is this insufficient for
infrastructure's needs?


Python3 itself isn't the problem, it's the whole stack of dependencies that we
need in addition to python.
Packaging things for EPEL means it's available to everyone and anyone can
contribute/build upon it, we know how the system and the packaging work, it's
easy for us to add new components and fix others.
Packaging things for SCL means, it's nowhere but our infra repo since we can't
push things to RH's SCLs and SCLs are neither in Fedora nor in EPEL, it's more
work as we'll need to figure out how SCLs work and how packaging for SCLs work
and it makes it a much more closed system from a contribution point of view.


For context, my primary interest is there being a smooth course for 
developers and users to move from Python 2 to 3, preferably *before* a 
new RHEL major release comes out.  If EPEL is meeting your needs 
that's great, but if you need something from RHEL that'd be good to 
know.  Thanks,


--
Brendan Conoboy / RHEL Development Coordinator / Red Hat, Inc.
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: State of python3 in our infrastructure

[Miroslav Suchý]

Dne 2.12.2015 v 12:14 Pierre-Yves Chibon napsal(a):
> So what do you folks think?

Copr is already migrated to python3 (in upstream) and I'm getting ready to use 
python3 on production servers soon.
But Copr (frontend and backend) is using Fedora it is no big deal anyway.
Well we will migrate frontend to python3, backend have to be postponed since it 
is using ansible modules, which are
python2 only.
Anything new we recently wrote (e.g. keygen) is python3 only. And if possible 
run on RHEL7 (again case of keygen).

Conclusion for me - use common sense. Write code python2/3 compatible. Migrate 
if possible. If libraries are python2
only (openstack, ansible) notify they owners and wait until there is python3 
version or if those authors refuse, choose
other alternative.

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: State of python3 in our infrastructure

[Pierre-Yves Chibon]

On Wed, Dec 02, 2015 at 01:24:45PM -0700, Luke Macken wrote:
> On Wed, Dec 02, 2015 at 12:14:42PM +0100, Pierre-Yves Chibon wrote:
> > Good morning everyone,
> > 
> > I would like to start gathering our thoughts about python3 in our apps.
> > 
> > To my knowledge, we currently have two applications that are python3 (only):
> > - Mailman3 core (as in hyperkitty is still py2)
> > - mdapi
> > 
> > Both are *not* running via apache/mod_wsgi. MM3 runs on RHEL7 while mdapi 
> > runs
> > on a Fedora node for the moment.
> 
> What are they using instead of apache/mod_wsgi?

In both cases (Aurélien correct me if I'm wrong), they are just exposing an API,
REST for MM3, JSON for mdapi.
I don't know for MM3, but mdapi is async, which cannot be handled by apache
afaik, we would need nginx or something if we wanted some sort of proxy.
So at the moment mdapi runs with the web-server provided by aiohttp and is
started via `systemctl start mdapi`.

What I wanted to highlight with this section is that despite having already 2
apps running python3, neither are "classic" nor following the traditional scheme
of web-app/apache/mod_wsgi, which means when we deploy our first py3 apps under
this scheme, we'll need to figure these things out, we can't rely on existing
implementation.


Pierre


pgpkiwC84dGRp.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: State of python3 in our infrastructure

[Toshio Kuratomi]

On Dec 3, 2015 1:44 AM, "Miroslav Suchý"  wrote:
>
> Dne 2.12.2015 v 12:14 Pierre-Yves Chibon napsal(a):
> > So what do you folks think?
>
> Copr is already migrated to python3 (in upstream) and I'm getting ready
to use python3 on production servers soon.
> But Copr (frontend and backend) is using Fedora it is no big deal anyway.
> Well we will migrate frontend to python3, backend have to be postponed
since it is using ansible modules, which are
> python2 only.

That shouldn't matter.  You just need to have both python 2 and python 3
installed on the box.  Ansible doesn't run under mod_wsgi so there is no
conflict at runtime.

Unless you mean you have been using the ansible python module (rather than
the python scripts ansible runs on the remote machines)?  If so, two
things.. (1) warning: 2.0 will be out soon and the python module api will
be very different.  We don't really support a stable api upstream in that
sense just the apis for writing plugins and modules (and some plugin
apis have changed in 2.0 as well.  Ansible module api should be almost the
same though.) (2) there's been some work on getting the controller code to
be python 3 compatible but we're a ways off from that (ansible modules much
further than that even).  There's some issues about maintaining python 2.4
compat in the controller code we still know needs porting(because that code
also runs on the remote hosts). Help welcomed upstream although we're
currently in the talking phase about that code (I'm planning some
improvements for 2.1 that will make this easier to manage).

> Anything new we recently wrote (e.g. keygen) is python3 only. And if
possible run on RHEL7 (again case of keygen).
>
Sounds like you jumped the gun here since this discussion will lead to
infra deciding if they want mixed python 3 and python 2 web application
environments :-)  but eventually this is inevitable.  If not or own
upstream, eventually third party upstream will start to support python 3
only. (As is the current case for maggit).

> Conclusion for me - use common sense. Write code python2/3 compatible.
Migrate if possible. If libraries are python2
> only (openstack, ansible) notify they owners and wait until there is
python3 version or if those authors refuse, choose
> other alternative.

Here we're talking about the case of upstream that only support python 3
and whether we want to take on the maintenance burden (separate systems for
running python2 and python 3 mod_wsgi apps, packaging work) that implies.

-Toshio

-Toshio
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: State of python3 in our infrastructure

[Toshio くらとみ]

I don't contribute much time to infrastructure these days so I'll try to 
outline costs::benefits rather than saying whether something is a good idea or 
not -- I won't be spending much time doing the work so it's not right for me to 
sign other people up for it :-)

> Good morning everyone,
> 
> I would like to start gathering our thoughts about python3 in our apps.
> 
> To my knowledge, we currently have two applications that are python3 (only):
> - Mailman3 core (as in hyperkitty is still py2)
> - mdapi
> 
> Both are *not* running via apache/mod_wsgi. MM3 runs on RHEL7 while mdapi runs
> on a Fedora node for the moment.
> 
You'll probably either need to have a few load balanced Fedora webservers 
running mod_wsgi for python3 or the equivalent rhel7 webservers with mod_wsgi 
for python3 to do this.  You'll want both python2 and python3 web servers 
because it will allow your applications to port one at a time.  Doing 
everything at once will be overwhelming.

* Going with Fedora means a more frequent distro upgrade cycle but Fedora is 
pushing python3 as default so the support from packagers not in infrastructure 
is likely higher.
* Going with RHEL7 means that you can continue with your current lifecycle for 
the distro but it means that you'll have to take a more active role in the 
python3-for-epel effort.

* Note: EPEL was started in large part because mmcgrath knew we needed extra 
packages for RHEL.  Rather than doing it all alone, he pushed for us to share 
the workload with other people in the wider ecosystem that could also use the 
packages.  So there probably is a bit of "if you build it, they will come 
here"... Infrastructure might need to contribute a fair amount of packaging 
time to get python3 on RHEL7 viable but after that there will be a steady 
stream of packagers who want to contribute to making the set of packages 
bigger. 
 
> So, what do we think about python3 app in our infrastructure? Are we ok with
> it? Do we want to avoid them for the moment? Do we want to split 'backend' vs
> 'frontend' (ie web-apps)?
> 
backend-frontend may make sense because fedora is moving to python3.  So 
frontends (command line CLIs and python libraries for people to write their own 
scripts or apps) that run on python3 and backends that run on python2 would 
match the existing reality of what people want to do.  It doesn't help your 
pygit2/maggit problem on the backend, though.  It also doesn't help if some of 
your CLIs also have to run in your infrastructure.

> However, maggit is python3 only.
> But the pain of dealing with pygit2 is such that I have been considering 
> looking
> more into maggit and maybe porting pagure to it (which means porting pagure to
> python3, already checked, all of its dependencies are py3 ready).
> 
It could be less overall effort to port maggit to run on python2.7+

However, that feels like moving backwards instead of forwards.

> Before I consider such effort (which would be a 2.0 release), I want to 
> clarify
> the status of python3 web-apps.
> 
I think starting to run python3 web apps will contribute more to the greater 
good than limiting apps to python2.  It will help to future-proof the web-apps 
since python2 probably won't be default in RHEL8 (who knows if it will be 
available as an add-on, though)... so at some point you do have to pay this 
price.  So the real question is whether you want to pay this price now and 
which people within infrastructure are going to be doing the work:

Do it when a RHEL releases with python3:
* sysadmins will have to manage both RHEL7 and RHEL8 web servers for py2 and 
py3.
* Concentrated work for developers to get applications ported as we'll want to 
migrate to rhel8 and python2 web apps will be holding us back.
* Packagers will have to branch (and sometimes maintain) needed packages from 
Fedora to EPEL. 

Do it now with the EPEL python34 (or a new python35) stack:
* packagers will have to create python3x packages in EPEL
* developers can start porting their apps now which gives them a longer window 
to port and perhaps less urgency 
* sysadmins will have to manage RHEL7 with python2 and RHEL7 with python3 web 
servers

Do it now with Fedora:
* developers can start porting their apps now which gives them a longer window 
to port and perhaps less urgency 
* Packagers still have to get packages running on python3 but not as many due 
to efforts from Fedora itself.
* sysadmins will have to manage RHEL7 with python2 and Fedora with python3 web 
servers.  Fedora has a different lifecycle so there's additional work involved 
here.

Waiting for RHEL is probably the least overall work but there is a drawback 
there porting applications to python3 is probably what will take the most 
time so the sooner you can get started on that the sooner you'll be able to 
stop supporting python2.

-Toshio 
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org

Re: State of python3 in our infrastructure

[Luke Macken]

On Wed, Dec 02, 2015 at 12:14:42PM +0100, Pierre-Yves Chibon wrote:
> Good morning everyone,
> 
> I would like to start gathering our thoughts about python3 in our apps.
> 
> To my knowledge, we currently have two applications that are python3 (only):
> - Mailman3 core (as in hyperkitty is still py2)
> - mdapi
> 
> Both are *not* running via apache/mod_wsgi. MM3 runs on RHEL7 while mdapi runs
> on a Fedora node for the moment.

What are they using instead of apache/mod_wsgi?

> So, what do we think about python3 app in our infrastructure? Are we ok with
> it? Do we want to avoid them for the moment? Do we want to split 'backend' vs
> 'frontend' (ie web-apps)?

I'm definitely in favor of driving as much stuff towards py3 in our
infra as possible, but I think we should try and run them the same way
we do with all of other other apps (apache+mod_wsgi).

luke


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

State of python3 in our infrastructure

[Pierre-Yves Chibon]

Good morning everyone,

I would like to start gathering our thoughts about python3 in our apps.

To my knowledge, we currently have two applications that are python3 (only):
- Mailman3 core (as in hyperkitty is still py2)
- mdapi

Both are *not* running via apache/mod_wsgi. MM3 runs on RHEL7 while mdapi runs
on a Fedora node for the moment.

So, what do we think about python3 app in our infrastructure? Are we ok with
it? Do we want to avoid them for the moment? Do we want to split 'backend' vs
'frontend' (ie web-apps)?

What brings me to raise these questions is that I have spent my morning getting
pagure to run on pygit2 0.23.0 (the version present on F23) so that I can
upgrade my laptop still running F21 atm.
The result of this work is not pretty: https://pagure.io/pagure/pull-request/516
(and might still require some work, just the tests are passing at the moment, I
would not be surprised if I missed some things in the templates).
I am more and more thinking about replacing pygit2 and a friend of mine is
developing a potentially really interesting alternative: maggit:
https://gitlab.com/maggit/maggit
However, maggit is python3 only.
But the pain of dealing with pygit2 is such that I have been considering looking
more into maggit and maybe porting pagure to it (which means porting pagure to
python3, already checked, all of its dependencies are py3 ready).

Before I consider such effort (which would be a 2.0 release), I want to clarify
the status of python3 web-apps.

So what do you folks think? :)

Thanks,
Pierre


pgps8jOALJNnC.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: State of python3 in our infrastructure

[Patrick Uiterwijk]

Hi,

> Good morning everyone,
> 
> I would like to start gathering our thoughts about python3 in our apps.
> 
> To my knowledge, we currently have two applications that are python3 (only):
> - Mailman3 core (as in hyperkitty is still py2)
> - mdapi
> 
> Both are *not* running via apache/mod_wsgi. MM3 runs on RHEL7 while mdapi
> runs
> on a Fedora node for the moment.
> 
> So, what do we think about python3 app in our infrastructure? Are we ok with
> it? Do we want to avoid them for the moment? Do we want to split 'backend' vs
> 'frontend' (ie web-apps)?

I personally think that it doesn't make much sense to split backend vs frontend
requirements in this regard, since the distinction here differs per app, and may
not be very clear. I'd say we either allow or deny python3 in total.

I also like the idea of having the option to switch to Python 3 in the infra, 
since
more neat features will come to it. I would also prefer to keep running our 
infra
on RHEL7.

That being said, I don't really know the current state of python3 for RHEL7.
There is a package for EPEL7 (python34) that has not been touched since it was
initially imported, and is still at python 3.4.3. This admittedly has not had
a new release, but there are also several bug reports open against it that seem
to have been idle for the last few months [1], and there are no co-maintainers.

One advantage of Python 2 at this point is also support. I'm not sure about
the rest of the team, but I know that I personally am not really that well
versed in the internals of CPython, and if something were to crash there, I'd 
have
no idea what is going on exactly without a lot of debugging and digging.
With python 2, we have a support account with which we could ask internal
engineers to look into it.

All in all, I think that enabling infra applications to use python 3 would be
great, but I'd like to see the current package for CPython improved, or more
co-maintainers added, before I would feel completely comfortable. (Perhaps we
can get the Fedora python3 maintainers to help with python34 for EPEL7?)

With kind regards,
Patrick Uiterwijk

[1]: 
https://bugzilla.redhat.com/buglist.cgi?quicksearch=component%3Apython34_id=4273331
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org