Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On 04/23/2015 11:29 PM, Thomas Goirand wrote: On 04/23/2015 04:24 PM, Victor Stinner wrote: Also, on the Python 3 topic, there's still a big issue with memcached (aka: python-memcache). Oh, thanks Thomas for the reminder. I just sent a pull request to port python-memcached to Python 3: https://github.com/linsomniac/python-memcached/pull/67 Cool! I'll try it, apply the patch to the version 1.54, and upload to Debian, if this works well. Yesterday, I uploaded the latest commit of python-memcache + your patch to Debian Experimental. It's blocking me from adding Python3 support to keystoneclient, and as a consequence, to almost all of OpenStack. python-keystoneclient announces a full Python 3 support with a voting Python 3 gate. I just checked locally, tox -e py34 pass. The problem is that python-memcached miss in test dependencies and so middleware tests using memcache are never run! Victor Yeah, memcached (aka: python-memcache in Debian) is said to be optional, but in reality, we really need it for the full Python 3 compatibility, otherwise that's bullshit (unless we're giving up on using memcache completely?). Now that python-memcache is fixed for Python 3, the other issue is pysaml2 which also only supports Python 2. That's also a blocker for keystoneclient. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Fri, Apr 24 2015, Victor Stinner wrote: What should we do with the following bug? memcache tests are skipped on python 3 https://bugs.launchpad.net/python-keystoneclient/+bug/1447731 Check if it applies to keystonemiddleware and if so move it there otherwise close it? -- Julien Danjou -- Free Software hacker -- http://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Keystone client version of the middleware is deprecated and only receiving minimal security updates. This code is unlikely to see any real changes due it its deprecation and frozen state. We are evaluating how to remove it from the client lib. Sent via mobile On Apr 24, 2015, at 00:27, Victor Stinner vstin...@redhat.com wrote: Hi, The part of keystoneclient that uses the memcached client was deprecated in Juno (as it was moved to the keystonemiddleware repo), Oh, I was not aware of the keystonemiddleware project. I see that Nova uses it for example. so I think we can remove it now. Do someone know if the middleware of keystoneclient is still used? You might want to patch it out of your keystoneclient package if you know everything's using the auth_token middleware from keystonemiddleware. What should we do with the following bug? memcache tests are skipped on python 3 https://bugs.launchpad.net/python-keystoneclient/+bug/1447731 I also wrote a patch for keystoneclient: Enable test_auth_token_middleware() on Python 2 https://review.openstack.org/#/c/176778/ If the keystoneclient middleware is removed, we can simply close the bug and abandon the change. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, The part of keystoneclient that uses the memcached client was deprecated in Juno (as it was moved to the keystonemiddleware repo), Oh, I was not aware of the keystonemiddleware project. I see that Nova uses it for example. so I think we can remove it now. Do someone know if the middleware of keystoneclient is still used? You might want to patch it out of your keystoneclient package if you know everything's using the auth_token middleware from keystonemiddleware. What should we do with the following bug? memcache tests are skipped on python 3 https://bugs.launchpad.net/python-keystoneclient/+bug/1447731 I also wrote a patch for keystoneclient: Enable test_auth_token_middleware() on Python 2 https://review.openstack.org/#/c/176778/ If the keystoneclient middleware is removed, we can simply close the bug and abandon the change. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, I wrote my spec to Port Nova to Python 3: https://review.openstack.org/#/c/176868/ I squashed all my commits into a single commit of my draft port and I pushed it at: https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d I like how the sha1 starts with 'bad' Ah ah, I didn't notice that. I would prefer python prefix, but it's not possible. Overall that is a pretty small patch. Cool. My general concern, is having to manually review new code for python3 compliance. Do you prefer a single very large patch (as the one I posted above) or multiple small patches fixing similar issues? If this will take more then a week or two to make a big project python3 compat (during a virtual sprint), it would be good to have some tooling in place to make sure we don't add a lot more burden on reviewers to make sure new patches are python3 compatible by hand. I tried to write a detailed plan in my spec. Until tox -e py34 pass, I would prefer to not touch nova gates nor any add Python 3 checks. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Fri, Apr 24, 2015 at 12:36 AM, Victor Stinner vstin...@redhat.com wrote: Hi, I wrote my spec to Port Nova to Python 3: https://review.openstack.org/#/c/176868/ I squashed all my commits into a single commit of my draft port and I pushed it at: https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d I like how the sha1 starts with 'bad' Ah ah, I didn't notice that. I would prefer python prefix, but it's not possible. Overall that is a pretty small patch. Cool. My general concern, is having to manually review new code for python3 compliance. Do you prefer a single very large patch (as the one I posted above) or multiple small patches fixing similar issues? If this will take more then a week or two to make a big project python3 compat (during a virtual sprint), it would be good to have some tooling in place to make sure we don't add a lot more burden on reviewers to make sure new patches are python3 compatible by hand. I tried to write a detailed plan in my spec. Until tox -e py34 pass, I would prefer to not touch nova gates nor any add Python 3 checks. Great, I'll follow up on that spec. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Also, on the Python 3 topic, there's still a big issue with memcached (aka: python-memcache). Oh, thanks Thomas for the reminder. I just sent a pull request to port python-memcached to Python 3: https://github.com/linsomniac/python-memcached/pull/67 I don't understand. I saw a lot of Port to Python 3 fixes merged in 2014, and these changes are now part of the release 1.54. Problem: running python-memcached tests with tox fail on obvious Python 3 errors. Anyway, all tests now pass with my pull request. The good news is that python-memcached looks to be actively developed. Julien Danjou ported pymemcache to Python 3, another memcached client. He suggests to use this one instead of python-memcached client. It's blocking me from adding Python3 support to keystoneclient, and as a consequence, to almost all of OpenStack. python-keystoneclient announces a full Python 3 support with a voting Python 3 gate. I just checked locally, tox -e py34 pass. The problem is that python-memcached miss in test dependencies and so middleware tests using memcache are never run! Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Excerpts from Victor Stinner's message of 2015-04-23 07:24:06 -0700: Also, on the Python 3 topic, there's still a big issue with memcached (aka: python-memcache). Oh, thanks Thomas for the reminder. I just sent a pull request to port python-memcached to Python 3: https://github.com/linsomniac/python-memcached/pull/67 I don't understand. I saw a lot of Port to Python 3 fixes merged in 2014, and these changes are now part of the release 1.54. Problem: running python-memcached tests with tox fail on obvious Python 3 errors. Anyway, all tests now pass with my pull request. The good news is that python-memcached looks to be actively developed. Julien Danjou ported pymemcache to Python 3, another memcached client. He suggests to use this one instead of python-memcached client. It's blocking me from adding Python3 support to keystoneclient, and as a consequence, to almost all of OpenStack. python-keystoneclient announces a full Python 3 support with a voting Python 3 gate. I just checked locally, tox -e py34 pass. The problem is that python-memcached miss in test dependencies and so middleware tests using memcache are never run! This is a bug worth fixing. Those skips should be removed, as memcached is quite popular as a token store. So I opened this bug to track it: https://bugs.launchpad.net/python-keystoneclient/+bug/1447731 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Victor Stinner wrote: Also, on the Python 3 topic, there's still a big issue with memcached (aka: python-memcache). Oh, thanks Thomas for the reminder. I just sent a pull request to port python-memcached to Python 3: https://github.com/linsomniac/python-memcached/pull/67 I don't understand. I saw a lot of Port to Python 3 fixes merged in 2014, and these changes are now part of the release 1.54. Problem: running python-memcached tests with tox fail on obvious Python 3 errors. Anyway, all tests now pass with my pull request. The good news is that python-memcached looks to be actively developed. Julien Danjou ported pymemcache to Python 3, another memcached client. He suggests to use this one instead of python-memcached client. Yes please to using pymemcache (I'm also an active contributor[1] there, so feel free to ask me questions to...), although what else in openstack uses it? From my current understand the nova usage of it is suboptimal (may not even be fully working/functional) due to: https://review.openstack.org/#/c/138607/ which has exposed some interesting details... [1] https://github.com/pinterest/pymemcache/graphs/contributors It's blocking me from adding Python3 support to keystoneclient, and as a consequence, to almost all of OpenStack. python-keystoneclient announces a full Python 3 support with a voting Python 3 gate. I just checked locally, tox -e py34 pass. The problem is that python-memcached miss in test dependencies and so middleware tests using memcache are never run! Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Thu, Apr 23, 2015 at 12:10 AM, Victor Stinner vstin...@redhat.com wrote: Hi, How invasive would the port to python3 be? I squashed all my commits into a single commit of my draft port and I pushed it at: https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d I like how the sha1 starts with 'bad' Overall that is a pretty small patch. As announced, changes are boring, just obvious Python2/Python3 issues: - strip L from long integer literals: 123L = 123 - replace dict.iteritems() with six.iteritems(dict) - replace list.sort(cmp_func) with list.sort(key=key_func) - replace raise exc_info[0], exc_info[1], exc_info[2] with six.reraise(*exc_info) - moved functions / modules * get http.client, urllib.request and other stdlib modules from six.moves since they moved between Python 2 and Python 3 * get itertools.izip/zip_longest from six.moves * replace unichr() with six.unichr() - replace filter(func, data) with [item for item in data if func(item)] - replace unicode() with six.text_type - replace (int, long) with six.integer_types - replace file() with open() - replace StringIO() with six.StringIO() These changes are not enough to port nova to Python 3. But they are required to be able to find next Python 3 bugs. Reminder: Python 2 compatibility is kept! There is not reason right now to drop Python 2 support, it would be a pain to upgrade! Would it be easier to have a python3 migration sprint and rip the band aid off instead of dragging it out and having partial python3 support for a whole cycle? Do you mean a physical meeting? Or focus on Python 3 during a short period (review/merge all Python 3 patches)? Focus on Python 3 during a short period. A single week may not be enough to fix all Python 3 issues. I prefer to submit changes smoothly during the Liberty cycle. My general concern, is having to manually review new code for python3 compliance. If this will take more then a week or two to make a big project python3 compat (during a virtual sprint), it would be good to have some tooling in place to make sure we don't add a lot more burden on reviewers to make sure new patches are python3 compatible by hand. In general what is the least painful way to add python3 support for a very large python project? Send patches and make incremental changes, as any other change made in OpenStack. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On 04/23/2015 04:24 PM, Victor Stinner wrote: Also, on the Python 3 topic, there's still a big issue with memcached (aka: python-memcache). Oh, thanks Thomas for the reminder. I just sent a pull request to port python-memcached to Python 3: https://github.com/linsomniac/python-memcached/pull/67 Cool! I'll try it, apply the patch to the version 1.54, and upload to Debian, if this works well. I don't understand. I saw a lot of Port to Python 3 fixes merged in 2014, and these changes are now part of the release 1.54. Problem: running python-memcached tests with tox fail on obvious Python 3 errors. Anyway, all tests now pass with my pull request. The good news is that python-memcached looks to be actively developed. Julien Danjou ported pymemcache to Python 3, another memcached client. He suggests to use this one instead of python-memcached client. pymemcache is much better in many ways, and switching to it shouldn't be motivated only because of Python 3 compat. It's blocking me from adding Python3 support to keystoneclient, and as a consequence, to almost all of OpenStack. python-keystoneclient announces a full Python 3 support with a voting Python 3 gate. I just checked locally, tox -e py34 pass. The problem is that python-memcached miss in test dependencies and so middleware tests using memcache are never run! Victor Yeah, memcached (aka: python-memcache in Debian) is said to be optional, but in reality, we really need it for the full Python 3 compatibility, otherwise that's bullshit (unless we're giving up on using memcache completely?). Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Thu, Apr 23, 2015 at 4:41 AM, Thomas Goirand z...@debian.org wrote: On 04/10/2015 09:41 AM, Thierry Carrez wrote: Victor Stinner wrote: I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(), threading.RLock, threading.Thread). Good news: the just released eventlet 0.17.3 includes these fixes and it is now fully compatible with Python 3! For example, the Oslo Messaging test suite now pass with this eventlet version! Currently, eventlet is disabled in Oslo Messaging on Python 3 (eventlet tests are skipped). Great news ! That makes the port to Python 3 question independent of the Moving off eventlet question, which should facilitate immediate progress on the former. On the latter, do you plan to file a Concurrency models cross-project session ? That sounds like a good topic to discuss face to face... See http://lists.openstack.org/pipermail/openstack-dev/2015-April/061070.html for details on how to file there. Also, on the Python 3 topic, there's still a big issue with memcached (aka: python-memcache). It's blocking me from adding Python3 support to keystoneclient, and as a consequence, to almost all of OpenStack. BTW, the Eventlet module for 0.17.3 is available from here: http://kilo-jessie.pkgs.mirantis.com/debian/pool/jessie-kilo-backports-nochange/main/p/python-eventlet/ and I will upload this to Experimental as soon as Jessie is released (that's in 3 days now...). Cheers, Thomas Goirand (zigo) The part of keystoneclient that uses the memcached client was deprecated in Juno (as it was moved to the keystonemiddleware repo), so I think we can remove it now. You might want to patch it out of your keystoneclient package if you know everything's using the auth_token middleware from keystonemiddleware. - Brant __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, How invasive would the port to python3 be? I squashed all my commits into a single commit of my draft port and I pushed it at: https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d As announced, changes are boring, just obvious Python2/Python3 issues: - strip L from long integer literals: 123L = 123 - replace dict.iteritems() with six.iteritems(dict) - replace list.sort(cmp_func) with list.sort(key=key_func) - replace raise exc_info[0], exc_info[1], exc_info[2] with six.reraise(*exc_info) - moved functions / modules * get http.client, urllib.request and other stdlib modules from six.moves since they moved between Python 2 and Python 3 * get itertools.izip/zip_longest from six.moves * replace unichr() with six.unichr() - replace filter(func, data) with [item for item in data if func(item)] - replace unicode() with six.text_type - replace (int, long) with six.integer_types - replace file() with open() - replace StringIO() with six.StringIO() These changes are not enough to port nova to Python 3. But they are required to be able to find next Python 3 bugs. Reminder: Python 2 compatibility is kept! There is not reason right now to drop Python 2 support, it would be a pain to upgrade! Would it be easier to have a python3 migration sprint and rip the band aid off instead of dragging it out and having partial python3 support for a whole cycle? Do you mean a physical meeting? Or focus on Python 3 during a short period (review/merge all Python 3 patches)? A single week may not be enough to fix all Python 3 issues. I prefer to submit changes smoothly during the Liberty cycle. In general what is the least painful way to add python3 support for a very large python project? Send patches and make incremental changes, as any other change made in OpenStack. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
I recommend to use mysqlclient instead of MySQL-python even on Python 2. https://pypi.python.org/pypi/mysqlclient https://github.com/PyMySQL/mysqlclient-python Is it packaged in popular distributions? RHEL? Fedora? SuSe? Ubuntu? Debian? Gentoo? If this library solves real bugs and provides Python 3 compatibility, I think it's worth to replace MySQL-python with this new library. Linux distro can package it, they can probably copy the packaging of MySQL-Python since it's a fork. Note: mysqlclient-python conflicts with MySQL-Python, both libraries use the same MySQLdb Python module. The good news is that mysqlclient-python is a drop-in library for MySQL-Python: no need to change any line of code, only modify requirements. @Naoki: Did you try to contact MySQL-Python authors to take over their project instead of forking? Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Thu, 23 Apr 2015 at 17:11 Victor Stinner vstin...@redhat.com wrote: As announced, changes are boring, just obvious Python2/Python3 issues: - strip L from long integer literals: 123L = 123 - replace dict.iteritems() with six.iteritems(dict) - replace list.sort(cmp_func) with list.sort(key=key_func) - replace raise exc_info[0], exc_info[1], exc_info[2] with six.reraise(*exc_info) - moved functions / modules * get http.client, urllib.request and other stdlib modules from six.moves since they moved between Python 2 and Python 3 * get itertools.izip/zip_longest from six.moves * replace unichr() with six.unichr() - replace filter(func, data) with [item for item in data if func(item)] - replace unicode() with six.text_type - replace (int, long) with six.integer_types - replace file() with open() - replace StringIO() with six.StringIO() These changes are not enough to port nova to Python 3. But they are required to be able to find next Python 3 bugs. Is there already a static analysis tool that helps find these things? (Would a pylint check for the above be useful? Some of them would be hard to find reliably, but a bunch of the above would be trivial) - Gus __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, These changes are not enough to port nova to Python 3. But they are required to be able to find next Python 3 bugs. Is there already a static analysis tool that helps find these things? (Would a pylint check for the above be useful? Some of them would be hard to find reliably, but a bunch of the above would be trivial) I read that hacking has some checks. It's quite easy to run 2to3 to see modified lines. Personally, I prefer to just run tests and fix issues one by one. It's faster to fix failing tests without having the modify the whole project at one. Sometimes, I'm also using regular expressions (in vim or grep). For example, I used [0-9]+L to find 123L. You can also use regex to modify code inplace. 2to3 tool drops Python 2 compatibility, so it's not great. But it helps to find code that needs to be modified. I now prefer 2to6 which uses six and so keeps Python 2 compatibility. I modified it a little bit to reduce changes: https://github.com/haypo/2to6/ Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Victor Stinner vstin...@redhat.com writes: Is there already a static analysis tool that helps find these things? (Would a pylint check for the above be useful? Some of them would be hard to find reliably, but a bunch of the above would be trivial) I read that hacking has some checks. It's quite easy to run 2to3 to see modified lines. In Swift, Alex has installed this tox.ini target to detect those : https://github.com/openstack/swift/blob/master/tox.ini#L38 may be a good idea to generalize. Cheers, Chmouel __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On 04/10/2015 09:41 AM, Thierry Carrez wrote: Victor Stinner wrote: I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(), threading.RLock, threading.Thread). Good news: the just released eventlet 0.17.3 includes these fixes and it is now fully compatible with Python 3! For example, the Oslo Messaging test suite now pass with this eventlet version! Currently, eventlet is disabled in Oslo Messaging on Python 3 (eventlet tests are skipped). Great news ! That makes the port to Python 3 question independent of the Moving off eventlet question, which should facilitate immediate progress on the former. On the latter, do you plan to file a Concurrency models cross-project session ? That sounds like a good topic to discuss face to face... See http://lists.openstack.org/pipermail/openstack-dev/2015-April/061070.html for details on how to file there. Also, on the Python 3 topic, there's still a big issue with memcached (aka: python-memcache). It's blocking me from adding Python3 support to keystoneclient, and as a consequence, to almost all of OpenStack. BTW, the Eventlet module for 0.17.3 is available from here: http://kilo-jessie.pkgs.mirantis.com/debian/pool/jessie-kilo-backports-nochange/main/p/python-eventlet/ and I will upload this to Experimental as soon as Jessie is released (that's in 3 days now...). Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, All, My 2c are: - yes, oslo.db supports python 3 (unittests passes, at least :) ) - MySQL-python still default MySQL DB driver in OpenStack, but at the moment the only DB driver for MySQL in python3 environment is PyMySQL, so I think, it's ok to use it with python 3. Hi. I'm maintainer of PyMySQL and mysqlclient. mysqlclient is fork of MySQL-python. It uses libmysqlclient.so. It fixes some bugs, build issues and it support Python 3. For example: * Support MariaDB's libmysqlclient.so * Support microsecond in TIME column I recommend to use mysqlclient instead of MySQL-python even on Python 2. https://pypi.python.org/pypi/mysqlclient https://github.com/PyMySQL/mysqlclient-python -- INADA Naoki songofaca...@gmail.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 04/22/2015 05:02 PM, INADA Naoki wrote: Hi, All, My 2c are: - yes, oslo.db supports python 3 (unittests passes, at least :) ) - MySQL-python still default MySQL DB driver in OpenStack, but at the moment the only DB driver for MySQL in python3 environment is PyMySQL, so I think, it's ok to use it with python 3. ?Hi.? I'm maintainer of PyMySQL and mysqlclient. mysqlclient is fork of MySQL-python. It uses libmysqlclient.so. It fixes some bugs, build issues and it support Python 3. For example: * Support MariaDB's libmysqlclient.so * Support microsecond in TIME column I recommend to use mysqlclient instead of MySQL-python even on Python 2. https://pypi.python.org/pypi/mysqlclient https://github.com/PyMySQL/mysqlclient-python Is it packaged in popular distributions? RHEL? Fedora? SuSe? Ubuntu? Debian? Gentoo? /Ihar -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJVN7qEAAoJEC5aWaUY1u5715kH/3YWVEAKqM/KIPGVBnycchZx qGiQlkLuk989WLDvELJ4iwnYaeWfLv3O0RozHOJNdetKbmxJnSJS5BZvX7RUGWrU NomHc8LfGeKyE4M3DAuyJUBeih2/YFOuvq404iPtl7YlvQPyMsoSm6lnmm/YuQlV hlB9erx0P/UPCeeRpWGKIV31L1KMLPgl+Mr7TItsfnmlKDkwtOBijIfw3ECxknqI CzUwMLTwvIL3IRfXWke+FBqzoUIZr/tXXJAaqsfWjjX31AZp0Py8LsW8AXnkbCrN GPH+raU8gkAdpgYMM34dBoxI18Y5xCrLyWXzHRIoTp42dIvzPBE24/UHldNKG7I= =z6qY -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Wed, Apr 22, 2015 at 3:15 AM, Victor Stinner vstin...@redhat.com wrote: Hi, It's moving fast. I'm currently working on porting remaining libraries to prepare my spec for nova. Great, I did't realize how close all the dependencies were. oslo.db -- looks like it is almost there I don't know the status of oslo.db support of Python 3. I guess that it already works on Python 3, it's just a matter of running tests with MySQL (MySQL-Python blocks again here). oslo.messaging -- same Two changes of my Python 3 changes were already merged last week. Three remaining Python 3 changes are almost there, they are mostly blocked by the freeze on requirements, but changes are already approved. The blocker is the bump of eventlet to 0.17.3: https://review.openstack.org/#/c/172132/ paste -- almost there I fixed that last night (!) with the release of Paste 2.0. It's the first Paste release since 2010. Paste 2.0 has an experimental support of Python 3. It should be enough for nova, according to my quick tests in my local nova repository where I fixed most obvious Python 3 issues. Ian Bicking allowed me to publish new versions of Paste. sqlalchemy-migrate -- almost there It already supports Python 3, so it doesn't block. suds -- with the suds fork shouldn't be too hard You should just switch to the fork. I contacted the author of suds-jurko: he plans to rename its component from suds-jurko to suds. So his fork would be simple drop-in which would not require any change in OpenStack except of requirements. websockify -- unknown It announces Python 3.3 and 3.4 support and has a py34 env in tox.ini. I didn't check yet if it supports Python 3, but since the project is active (last commit 12 hours ago), it's shouldn't be hard to fix them quickly. libvirt-python -- unknown Oh, I missed this dependency. It's my next target :-) mysql-python -- alternitive looks viable. Yes, it's possible to replace it with PyMySQL. Does anyone know the status of this switch? Based on there being two unknowns, and a lot of dependencies that are just almost there, and a few we may want to migrate off of, I was assuming addressing those issues would make it hard for us to make nova python3 compatible for Liberty. My plan for Liberty is not to support fully Python 3 in nova, but only start the port (well, continue to be exact). The idea is to fix syntax errors and obvious issues like dict.iteritems() = six.iteritems(dict), then more complex changes. Maybe it's possible to finish the port in a cycle, but it really depends on the time taken to merge patches. How invasive would the port to python3 be? Would it be easier to have a python3 migration sprint and rip the band aid off instead of dragging it out and having partial python3 support for a whole cycle? In general what is the least painful way to add python3 support for a very large python project? I started to port nova in my local repository. Some unit tests already pass. nova already uses six in many places, so it's not like we really start from scratch, the port is already an ingoing process. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, It's moving fast. I'm currently working on porting remaining libraries to prepare my spec for nova. oslo.db -- looks like it is almost there I don't know the status of oslo.db support of Python 3. I guess that it already works on Python 3, it's just a matter of running tests with MySQL (MySQL-Python blocks again here). oslo.messaging -- same Two changes of my Python 3 changes were already merged last week. Three remaining Python 3 changes are almost there, they are mostly blocked by the freeze on requirements, but changes are already approved. The blocker is the bump of eventlet to 0.17.3: https://review.openstack.org/#/c/172132/ paste -- almost there I fixed that last night (!) with the release of Paste 2.0. It's the first Paste release since 2010. Paste 2.0 has an experimental support of Python 3. It should be enough for nova, according to my quick tests in my local nova repository where I fixed most obvious Python 3 issues. Ian Bicking allowed me to publish new versions of Paste. sqlalchemy-migrate -- almost there It already supports Python 3, so it doesn't block. suds -- with the suds fork shouldn't be too hard You should just switch to the fork. I contacted the author of suds-jurko: he plans to rename its component from suds-jurko to suds. So his fork would be simple drop-in which would not require any change in OpenStack except of requirements. websockify -- unknown It announces Python 3.3 and 3.4 support and has a py34 env in tox.ini. I didn't check yet if it supports Python 3, but since the project is active (last commit 12 hours ago), it's shouldn't be hard to fix them quickly. libvirt-python -- unknown Oh, I missed this dependency. It's my next target :-) mysql-python -- alternitive looks viable. Yes, it's possible to replace it with PyMySQL. Does anyone know the status of this switch? Based on there being two unknowns, and a lot of dependencies that are just almost there, and a few we may want to migrate off of, I was assuming addressing those issues would make it hard for us to make nova python3 compatible for Liberty. My plan for Liberty is not to support fully Python 3 in nova, but only start the port (well, continue to be exact). The idea is to fix syntax errors and obvious issues like dict.iteritems() = six.iteritems(dict), then more complex changes. Maybe it's possible to finish the port in a cycle, but it really depends on the time taken to merge patches. I started to port nova in my local repository. Some unit tests already pass. nova already uses six in many places, so it's not like we really start from scratch, the port is already an ingoing process. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, sqlachemy-migrate already works on Python3. I sent a patch to add missing Python 3 classifiers, so the caniusepython3 automated tool will stop reporting false alarm on this library. https://review.openstack.org/174738 Or is someone aware of issues with Python 3? Again, check the wiki page for up to date information: https://wiki.openstack.org/wiki/Python3#Dependencies There seems to have been some work on that already: https://github.com/stackforge/sqlalchemy-migrate/commit/a03b141a95 Yeah, Cyril made great work! I ported a lot of OpenStack libraries. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Tue, Apr 21, 2015 at 02:58:57PM -0700, Joe Gordon wrote: On Fri, Apr 17, 2015 at 1:22 AM, Victor Stinner vstin...@redhat.com wrote: For the full list, see the wiki page: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects Thanks for updating the wiki page that is a very useful list. From the looks of things, it seems like nova getting Python3 support in Liberty is not going to happen. I based this on the wiki, but maybe I am wrong. remaining libraries for nova: oslo.db -- looks like it is almost there oslo.messaging -- same paste -- almost there sqlalchemy-migrate -- almost there suds -- with the suds fork shouldn't be too hard websockify -- unknown libvirt-python -- unknown I notice some work from Dan Berrange in late 2013 about this: https://www.redhat.com/archives/libvir-list/2013-December/msg00171.html -- [PATCH python 00/15] Initial work porting to python3 And, I see several commits in libvirt-python git repo in related to Python 3. Maybe Dan can comment when he notices this. mysql-python -- alternitive looks viable. Based on there being two unknowns, and a lot of dependencies that are just almost there, and a few we may want to migrate off of, I was assuming addressing those issues would make it hard for us to make nova python3 compatible for Liberty. [. . .] -- /kashyap __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Wed, Apr 22, 2015 at 12:57:01PM +0200, Kashyap Chamarthy wrote: On Tue, Apr 21, 2015 at 02:58:57PM -0700, Joe Gordon wrote: On Fri, Apr 17, 2015 at 1:22 AM, Victor Stinner vstin...@redhat.com wrote: For the full list, see the wiki page: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects Thanks for updating the wiki page that is a very useful list. From the looks of things, it seems like nova getting Python3 support in Liberty is not going to happen. I based this on the wiki, but maybe I am wrong. remaining libraries for nova: oslo.db -- looks like it is almost there oslo.messaging -- same paste -- almost there sqlalchemy-migrate -- almost there suds -- with the suds fork shouldn't be too hard websockify -- unknown libvirt-python -- unknown I notice some work from Dan Berrange in late 2013 about this: https://www.redhat.com/archives/libvir-list/2013-December/msg00171.html -- [PATCH python 00/15] Initial work porting to python3 And, I see several commits in libvirt-python git repo in related to Python 3. Maybe Dan can comment when he notices this. Libvirt fully supports Python 3 ever since we split libvirt-python off from main libvirt and hosted it on pypi. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, All, My 2c are: - yes, oslo.db supports python 3 (unittests passes, at least :) ) - MySQL-python still default MySQL DB driver in OpenStack, but at the moment the only DB driver for MySQL in python3 environment is PyMySQL, so I think, it's ok to use it with python 3. On Wed, Apr 22, 2015 at 1:15 PM, Victor Stinner vstin...@redhat.com wrote: Hi, It's moving fast. I'm currently working on porting remaining libraries to prepare my spec for nova. oslo.db -- looks like it is almost there I don't know the status of oslo.db support of Python 3. I guess that it already works on Python 3, it's just a matter of running tests with MySQL (MySQL-Python blocks again here). oslo.messaging -- same Two changes of my Python 3 changes were already merged last week. Three remaining Python 3 changes are almost there, they are mostly blocked by the freeze on requirements, but changes are already approved. The blocker is the bump of eventlet to 0.17.3: https://review.openstack.org/#/c/172132/ paste -- almost there I fixed that last night (!) with the release of Paste 2.0. It's the first Paste release since 2010. Paste 2.0 has an experimental support of Python 3. It should be enough for nova, according to my quick tests in my local nova repository where I fixed most obvious Python 3 issues. Ian Bicking allowed me to publish new versions of Paste. sqlalchemy-migrate -- almost there It already supports Python 3, so it doesn't block. suds -- with the suds fork shouldn't be too hard You should just switch to the fork. I contacted the author of suds-jurko: he plans to rename its component from suds-jurko to suds. So his fork would be simple drop-in which would not require any change in OpenStack except of requirements. websockify -- unknown It announces Python 3.3 and 3.4 support and has a py34 env in tox.ini. I didn't check yet if it supports Python 3, but since the project is active (last commit 12 hours ago), it's shouldn't be hard to fix them quickly. libvirt-python -- unknown Oh, I missed this dependency. It's my next target :-) mysql-python -- alternitive looks viable. Yes, it's possible to replace it with PyMySQL. Does anyone know the status of this switch? Based on there being two unknowns, and a lot of dependencies that are just almost there, and a few we may want to migrate off of, I was assuming addressing those issues would make it hard for us to make nova python3 compatible for Liberty. My plan for Liberty is not to support fully Python 3 in nova, but only start the port (well, continue to be exact). The idea is to fix syntax errors and obvious issues like dict.iteritems() = six.iteritems(dict), then more complex changes. Maybe it's possible to finish the port in a cycle, but it really depends on the time taken to merge patches. I started to port nova in my local repository. Some unit tests already pass. nova already uses six in many places, so it's not like we really start from scratch, the port is already an ingoing process. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On 21/04/15 21:37, Ian Cordasco wrote: Also, isn’t sqlalchemy-migrate something we currently maintain (or a group of OpenStack developers do it for OpenStack. Can’t we work with them to add support for Python 3? There seems to have been some work on that already: https://github.com/stackforge/sqlalchemy-migrate/commit/a03b141a95 Pádraig. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On 4/16/15, 17:54, Clint Byrum cl...@fewbar.com wrote: Excerpts from Joe Gordon's message of 2015-04-16 15:15:01 -0700: On Fri, Apr 10, 2015 at 4:01 AM, Victor Stinner vstin...@redhat.com wrote: https://wiki.openstack.org/wiki/Python3#Dependencies appears to be fairly out of date. You're right. I updated this wiki page. In practice, much more OpenStack clients, Common Libraries and Development Tools are already Python 3 compatible. I added the link to my pull request for Oslo Messaging. It would be nice to get a better sense of what the remaining libraries to port over are before the summit so we can start planning how to do the python34 migration. I checked quickly. There are small libraries like pyEClib required by Swift, but the major blocker libraries are: MySQL-Python, suds, Paste. For oslo.db, it's already Python 3 compatible no? * MySQL-Python MySQL-Python doesn't look to be active (last commit in january 2014). There are multiple Python 3 pending pull requests: https://github.com/farcepest/MySQLdb1/pulls Mike Bayer is evaluating PyMySQL which is Python 3 compatible: https://wiki.openstack.org/wiki/PyMySQL_evaluation See also https://github.com/farcepest/moist (is it alive? is it Python 3 compatible?) * suds There is https://bitbucket.org/jurko/suds : a fork compatible with Python 3. Global requirements contain this comment: # NOTE(dims): suds is not python 3.x compatible, suds-jurko is a fork that # works with py3x. oslo.vmware would convert to suds-jurko first then nova # and cinder would follow. suds should be remove immediately once those # projects move to suds-jurko for all jobs. * Paste I already fixed Python 3 compatibility issues and my changes were merged, but there is no release including my fixes yet :-( I heard that Paste is completly outdated and should be replaced. Ok, but in practice it's still used and not Python 3 compatible. Workaround: use the development (git) version of Paste. For the full list, see the wiki page: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects Thanks for updating the wiki page that is a very useful list. From the looks of things, it seems like nova getting Python3 support in Liberty is not going to happen. But we can make good progress in dependencies sorted out. By fixing the dependencies and switching a few out for better ones. What are your thoughts on how to tackle sqlalchemy-migrate? It looks like that is a blocker for several projects. And something I think we have wanted to move off of for some time now. IMHO it is quite a bit easier to port something to python 3 than to move off of it entirely. I'd say it's worth it for forward progress to try and port sqlalchemy-migrate, even if that means the effort becomes a sunk cost in a year. Also, isn’t sqlalchemy-migrate something we currently maintain (or a group of OpenStack developers do it for OpenStack. Can’t we work with them to add support for Python 3? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Fri, Apr 17, 2015 at 1:22 AM, Victor Stinner vstin...@redhat.com wrote: For the full list, see the wiki page: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects Thanks for updating the wiki page that is a very useful list. From the looks of things, it seems like nova getting Python3 support in Liberty is not going to happen. I based this on the wiki, but maybe I am wrong. remaining libraries for nova: oslo.db -- looks like it is almost there oslo.messaging -- same paste -- almost there sqlalchemy-migrate -- almost there suds -- with the suds fork shouldn't be too hard websockify -- unknown libvirt-python -- unknown mysql-python -- alternitive looks viable. Based on there being two unknowns, and a lot of dependencies that are just almost there, and a few we may want to migrate off of, I was assuming addressing those issues would make it hard for us to make nova python3 compatible for Liberty. Why? I plan to work on porting nova to Python 3. I proposed a nova session on Python 3 at the next OpenStack Summit at Vancouver. I plan to write a spec too. I'm not aware of any real blocker for nova. What are your thoughts on how to tackle sqlalchemy-migrate? It looks like that is a blocker for several projects. And something I think we have wanted to move off of for some time now. I just checked sqlachemy-migrate. The README and the documentation are completly outdated, but the project is very active: latest commit one month ago and latest release (0.9.6) one month ago. There are py33 and py34 environments and tests pass on Python 3.3 and Python 3.4! I didn't check yet, but I guess that sqlachemy-migrate 0.9.6 already works on Python 3. Python 3 classifiers are just missing in setup.cfg. I sent patches to update the doc, to add Python 3 classifiers and to upgrade requirements. The project moved to stackforge, reviews are at review.openstack.org: https://review.openstack.org/#/q/status:open+project:stackforge/sqlalchemy-migrate,n,z The wiki page said that scripttest and ibm-db-sa were not Python 3 compatible. It's no more true: scripttest is compatible Python 3, and there is ibm-db-sa-py3 which is Python 3 compatible. I updated the wiki page for sqlachemy-migrate. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Tue, Apr 21, 2015 at 1:37 PM, Ian Cordasco ian.corda...@rackspace.com wrote: On 4/16/15, 17:54, Clint Byrum cl...@fewbar.com wrote: Excerpts from Joe Gordon's message of 2015-04-16 15:15:01 -0700: On Fri, Apr 10, 2015 at 4:01 AM, Victor Stinner vstin...@redhat.com wrote: https://wiki.openstack.org/wiki/Python3#Dependencies appears to be fairly out of date. You're right. I updated this wiki page. In practice, much more OpenStack clients, Common Libraries and Development Tools are already Python 3 compatible. I added the link to my pull request for Oslo Messaging. It would be nice to get a better sense of what the remaining libraries to port over are before the summit so we can start planning how to do the python34 migration. I checked quickly. There are small libraries like pyEClib required by Swift, but the major blocker libraries are: MySQL-Python, suds, Paste. For oslo.db, it's already Python 3 compatible no? * MySQL-Python MySQL-Python doesn't look to be active (last commit in january 2014). There are multiple Python 3 pending pull requests: https://github.com/farcepest/MySQLdb1/pulls Mike Bayer is evaluating PyMySQL which is Python 3 compatible: https://wiki.openstack.org/wiki/PyMySQL_evaluation See also https://github.com/farcepest/moist (is it alive? is it Python 3 compatible?) * suds There is https://bitbucket.org/jurko/suds : a fork compatible with Python 3. Global requirements contain this comment: # NOTE(dims): suds is not python 3.x compatible, suds-jurko is a fork that # works with py3x. oslo.vmware would convert to suds-jurko first then nova # and cinder would follow. suds should be remove immediately once those # projects move to suds-jurko for all jobs. * Paste I already fixed Python 3 compatibility issues and my changes were merged, but there is no release including my fixes yet :-( I heard that Paste is completly outdated and should be replaced. Ok, but in practice it's still used and not Python 3 compatible. Workaround: use the development (git) version of Paste. For the full list, see the wiki page: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects Thanks for updating the wiki page that is a very useful list. From the looks of things, it seems like nova getting Python3 support in Liberty is not going to happen. But we can make good progress in dependencies sorted out. By fixing the dependencies and switching a few out for better ones. What are your thoughts on how to tackle sqlalchemy-migrate? It looks like that is a blocker for several projects. And something I think we have wanted to move off of for some time now. IMHO it is quite a bit easier to port something to python 3 than to move off of it entirely. I'd say it's worth it for forward progress to try and port sqlalchemy-migrate, even if that means the effort becomes a sunk cost in a year. Also, isn’t sqlalchemy-migrate something we currently maintain (or a group of OpenStack developers do it for OpenStack. Can’t we work with them to add support for Python 3? yup https://github.com/stackforge/sqlalchemy-migrate. The better question is: 'is it worth adding support in versus moving over to alembic, since we want to do that anyway?' I don't personally have an answer for that. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
For the full list, see the wiki page: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects Thanks for updating the wiki page that is a very useful list. From the looks of things, it seems like nova getting Python3 support in Liberty is not going to happen. Why? I plan to work on porting nova to Python 3. I proposed a nova session on Python 3 at the next OpenStack Summit at Vancouver. I plan to write a spec too. I'm not aware of any real blocker for nova. What are your thoughts on how to tackle sqlalchemy-migrate? It looks like that is a blocker for several projects. And something I think we have wanted to move off of for some time now. I just checked sqlachemy-migrate. The README and the documentation are completly outdated, but the project is very active: latest commit one month ago and latest release (0.9.6) one month ago. There are py33 and py34 environments and tests pass on Python 3.3 and Python 3.4! I didn't check yet, but I guess that sqlachemy-migrate 0.9.6 already works on Python 3. Python 3 classifiers are just missing in setup.cfg. I sent patches to update the doc, to add Python 3 classifiers and to upgrade requirements. The project moved to stackforge, reviews are at review.openstack.org: https://review.openstack.org/#/q/status:open+project:stackforge/sqlalchemy-migrate,n,z The wiki page said that scripttest and ibm-db-sa were not Python 3 compatible. It's no more true: scripttest is compatible Python 3, and there is ibm-db-sa-py3 which is Python 3 compatible. I updated the wiki page for sqlachemy-migrate. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Fri, Apr 10, 2015 at 4:01 AM, Victor Stinner vstin...@redhat.com wrote: https://wiki.openstack.org/wiki/Python3#Dependencies appears to be fairly out of date. You're right. I updated this wiki page. In practice, much more OpenStack clients, Common Libraries and Development Tools are already Python 3 compatible. I added the link to my pull request for Oslo Messaging. It would be nice to get a better sense of what the remaining libraries to port over are before the summit so we can start planning how to do the python34 migration. I checked quickly. There are small libraries like pyEClib required by Swift, but the major blocker libraries are: MySQL-Python, suds, Paste. For oslo.db, it's already Python 3 compatible no? * MySQL-Python MySQL-Python doesn't look to be active (last commit in january 2014). There are multiple Python 3 pending pull requests: https://github.com/farcepest/MySQLdb1/pulls Mike Bayer is evaluating PyMySQL which is Python 3 compatible: https://wiki.openstack.org/wiki/PyMySQL_evaluation See also https://github.com/farcepest/moist (is it alive? is it Python 3 compatible?) * suds There is https://bitbucket.org/jurko/suds : a fork compatible with Python 3. Global requirements contain this comment: # NOTE(dims): suds is not python 3.x compatible, suds-jurko is a fork that # works with py3x. oslo.vmware would convert to suds-jurko first then nova # and cinder would follow. suds should be remove immediately once those # projects move to suds-jurko for all jobs. * Paste I already fixed Python 3 compatibility issues and my changes were merged, but there is no release including my fixes yet :-( I heard that Paste is completly outdated and should be replaced. Ok, but in practice it's still used and not Python 3 compatible. Workaround: use the development (git) version of Paste. For the full list, see the wiki page: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects Thanks for updating the wiki page that is a very useful list. From the looks of things, it seems like nova getting Python3 support in Liberty is not going to happen. But we can make good progress in dependencies sorted out. By fixing the dependencies and switching a few out for better ones. What are your thoughts on how to tackle sqlalchemy-migrate? It looks like that is a blocker for several projects. And something I think we have wanted to move off of for some time now. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Excerpts from Joe Gordon's message of 2015-04-16 15:15:01 -0700: On Fri, Apr 10, 2015 at 4:01 AM, Victor Stinner vstin...@redhat.com wrote: https://wiki.openstack.org/wiki/Python3#Dependencies appears to be fairly out of date. You're right. I updated this wiki page. In practice, much more OpenStack clients, Common Libraries and Development Tools are already Python 3 compatible. I added the link to my pull request for Oslo Messaging. It would be nice to get a better sense of what the remaining libraries to port over are before the summit so we can start planning how to do the python34 migration. I checked quickly. There are small libraries like pyEClib required by Swift, but the major blocker libraries are: MySQL-Python, suds, Paste. For oslo.db, it's already Python 3 compatible no? * MySQL-Python MySQL-Python doesn't look to be active (last commit in january 2014). There are multiple Python 3 pending pull requests: https://github.com/farcepest/MySQLdb1/pulls Mike Bayer is evaluating PyMySQL which is Python 3 compatible: https://wiki.openstack.org/wiki/PyMySQL_evaluation See also https://github.com/farcepest/moist (is it alive? is it Python 3 compatible?) * suds There is https://bitbucket.org/jurko/suds : a fork compatible with Python 3. Global requirements contain this comment: # NOTE(dims): suds is not python 3.x compatible, suds-jurko is a fork that # works with py3x. oslo.vmware would convert to suds-jurko first then nova # and cinder would follow. suds should be remove immediately once those # projects move to suds-jurko for all jobs. * Paste I already fixed Python 3 compatibility issues and my changes were merged, but there is no release including my fixes yet :-( I heard that Paste is completly outdated and should be replaced. Ok, but in practice it's still used and not Python 3 compatible. Workaround: use the development (git) version of Paste. For the full list, see the wiki page: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects Thanks for updating the wiki page that is a very useful list. From the looks of things, it seems like nova getting Python3 support in Liberty is not going to happen. But we can make good progress in dependencies sorted out. By fixing the dependencies and switching a few out for better ones. What are your thoughts on how to tackle sqlalchemy-migrate? It looks like that is a blocker for several projects. And something I think we have wanted to move off of for some time now. IMHO it is quite a bit easier to port something to python 3 than to move off of it entirely. I'd say it's worth it for forward progress to try and port sqlalchemy-migrate, even if that means the effort becomes a sunk cost in a year. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Worth noting we've already switched to using PyMySQL in nodepool, storyboard and some of the subunit2sql tooling. It's been working out great so far. Great. Did you notice a performance regression? Mike wrote that PyMySQL is much slower than MySQL-Python. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On 2015-04-13 04:03:49 -0400 (-0400), Victor Stinner wrote: Great. Did you notice a performance regression? Nope. Worth noting, we implemented it primarily for its lack of compiled extensions, and to a lesser because it supports Python 3.x. I suspect if we do later run into any unexpected performance issues... well, it's pure Python. We have lots of people who can help. Mike wrote that PyMySQL is much slower than MySQL-Python. I don't recall him saying that specifically. Also last I recall he admitted he hadn't actually tested under the sorts of load we would drive in a production OpenStack service--merely performed some fairly artificial benchmarks looping known-expensive operations that may not ultimately reflect places in our codebase where introducing any sort of slowdown would be noticeable compared to other operations being performed. Chances are the Project Infrastructure systems will continue incrementally switching to PyMySQL mainly because it's easier to install and works on a broader variety of platforms. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Thanks for your work on this! :) On Thu, Apr 9, 2015 at 7:25 PM, Victor Stinner vstin...@redhat.com wrote: Hi, During the last OpenStack Summit at Paris, we discussed how we can port OpenStack to Python 3, because eventlet was not compatible with Python 3. There are multiple approaches: port eventlet to Python 3, replace eventlet with asyncio, replace eventlet with threads, etc. We decided to not take a decision and instead investigate all options. I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(), threading.RLock, threading.Thread). Good news: the just released eventlet 0.17.3 includes these fixes and it is now fully compatible with Python 3! For example, the Oslo Messaging test suite now pass with this eventlet version! Currently, eventlet is disabled in Oslo Messaging on Python 3 (eventlet tests are skipped). I just sent a patch for requirements and Oslo Messaging to bump to eventlet 0.17.3, but it will have to wait until everyone has master as Liberty. https://review.openstack.org/#/c/172132/ https://review.openstack.org/#/c/172135/ It becomes possible to port more projects depending on eventlet to Python 3! Liberty cycle will be a good opportunity to port more OpenStack components to Python 3. Most OpenStack clients and Common Libraries are *already* Python 3 compatible, see the wiki page: https://wiki.openstack.org/wiki/Python3 -- To replace eventlet, I wrote a spec to replace it with asyncio: https://review.openstack.org/#/c/153298/ Joshua Harlow wrote a spec to replace eventlet with threads: https://review.openstack.org/#/c/156711/ But then he wrote a single spec Replace eventlet + monkey-patching with ?? which covers threads and asyncio: https://review.openstack.org/#/c/164035/ Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, During the last OpenStack Summit at Paris, we discussed how we can port OpenStack to Python 3, because eventlet was not compatible with Python 3. There are multiple approaches: port eventlet to Python 3, replace eventlet with asyncio, replace eventlet with threads, etc. We decided to not take a decision and instead investigate all options. I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(), threading.RLock, threading.Thread). Good news: the just released eventlet 0.17.3 includes these fixes and it is now fully compatible with Python 3! For example, the Oslo Messaging test suite now pass with this eventlet version! Currently, eventlet is disabled in Oslo Messaging on Python 3 (eventlet tests are skipped). I just sent a patch for requirements and Oslo Messaging to bump to eventlet 0.17.3, but it will have to wait until everyone has master as Liberty. https://review.openstack.org/#/c/172132/ https://review.openstack.org/#/c/172135/ It becomes possible to port more projects depending on eventlet to Python 3! Liberty cycle will be a good opportunity to port more OpenStack components to Python 3. Most OpenStack clients and Common Libraries are *already* Python 3 compatible, see the wiki page: https://wiki.openstack.org/wiki/Python3 -- To replace eventlet, I wrote a spec to replace it with asyncio: https://review.openstack.org/#/c/153298/ Joshua Harlow wrote a spec to replace eventlet with threads: https://review.openstack.org/#/c/156711/ But then he wrote a single spec Replace eventlet + monkey-patching with ?? which covers threads and asyncio: https://review.openstack.org/#/c/164035/ Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
On Thu, Apr 9, 2015 at 9:25 AM, Victor Stinner vstin...@redhat.com wrote: Hi, During the last OpenStack Summit at Paris, we discussed how we can port OpenStack to Python 3, because eventlet was not compatible with Python 3. There are multiple approaches: port eventlet to Python 3, replace eventlet with asyncio, replace eventlet with threads, etc. We decided to not take a decision and instead investigate all options. I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(), threading.RLock, threading.Thread). Good news: the just released eventlet 0.17.3 includes these fixes and it is now fully compatible with Python 3! For example, the Oslo Messaging test suite now pass with this eventlet version! Currently, eventlet is disabled in Oslo Messaging on Python 3 (eventlet tests are skipped). I just sent a patch for requirements and Oslo Messaging to bump to eventlet 0.17.3, but it will have to wait until everyone has master as Liberty. https://review.openstack.org/#/c/172132/ https://review.openstack.org/#/c/172135/ It becomes possible to port more projects depending on eventlet to Python 3! Awesome! Liberty cycle will be a good opportunity to port more OpenStack components to Python 3. Most OpenStack clients and Common Libraries are *already* Python 3 compatible, see the wiki page: https://wiki.openstack.org/wiki/Python3 https://wiki.openstack.org/wiki/Python3#Dependencies appears to be fairly out of date. For example hacking works under python34 as does oslo messaging as per this email etc. Also what is the status of all the dependencies in https://github.com/openstack/nova/blob/master/requirements.txt and more generally https://github.com/openstack/requirements/blob/master/global-requirements.txt It would be nice to get a better sense of what the remaining libraries to port over are before the summit so we can start planning how to do the python34 migration. -- To replace eventlet, I wrote a spec to replace it with asyncio: https://review.openstack.org/#/c/153298/ Joshua Harlow wrote a spec to replace eventlet with threads: https://review.openstack.org/#/c/156711/ But then he wrote a single spec Replace eventlet + monkey-patching with ?? which covers threads and asyncio: https://review.openstack.org/#/c/164035/ Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
