Re: [openstack-dev] [swift] Plan to add Python 3 support to Swift
Hi, Le 08/02/2016 14:42, Victor Stinner a écrit : > https://review.openstack.org/#/c/237019/ Nice, this one was merged! https://review.openstack.org/#/c/237027/ https://review.openstack.org/#/c/236998/ These two changes look to be blocked by encodings issues. Tim Burke, Samuel Merritt and Christian Schwede have concerns about the exact behaviour on Python 3. Let me explain how I'm working. I spend between 10 minutes and one day to port a single unit test and then I submit a patch (or patches if the change is big). I prefer to work incrementally: first fix the unit test, and then enhance the code if needed. Right now, unit tests don't pass, there are still serious bytes vs Unicode issues. I would prefer to reach a first milestone where unit tests pass and then discuss how to fix complex encoding issues. Change 236998: For the hmac patch, supporting arbitrary hash prefix or suffix requires to implement a new feature. Swift parses the configuration files using the ConfigParser which doesn't support arbitrary bytes. I understand the use case and I agree that something should be done, but I suggest to do that later. Change 237027: For the encoding of HTTP headers, it looks like Swift doesn't respect HTTP RFCs. The HTTP requires headers to be encoded to Latin1, but Swift (server or client, sorry I don't know) encode headers to UTF-8. Something should be do too, but it will require a deep analysis, prepare a transition period, etc. This problem is complex and cannot be fixed right now. Just to be clear: Swift does not support Python 3, so we are still free to change completely the code for Python 3, until Swift is fully compatible with Python 3. Swift is not my main target, so I cannot spend too much time on it. If you expect me to solve all issues at once, I'm sorry, I cannot help :-/ Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [swift] Plan to add Python 3 support to Swift
Le 12/02/2016 15:12, Ian Cordasco a écrit : Just to interject here, RFC 2616 is the one you're talking about. That encoding requirement was dropped when HTTP/1.1 was updated in the 7230-7235 RFCs. (...) Where VCHAR is any visible US ASCII character. So while UTF-8 is still a bad idea for the header value (and in fact, http.client on Python 3 will auto-encode headers to Latin 1) Latin 1 is no longer the requirement. For those interested, you can read up on headers in HTTP/1.1 here: https://tools.ietf.org/html/rfc7230#section-3.2 Oh thanks, it looks like my HTTP skills are rusty :-) For Swift, it's maybe better to always try to use UTF-8, but fallback to Latin1 if an HTTP cannot be decoded from UTF-8. Swift has many clients implemented in various programming languages, I'm not sure that all clients use UTF-8. By the way, https://review.openstack.org/#/c/237027/ was merged, cool. I fixed the third patch mentioned in my previous email to support arbitrary byte strings for hash prefix and suffix in the configuration file: https://review.openstack.org/#/c/236998/ I also updated my HTTP parser patch for Python 3. With these two patches, test_utils now pass on Python 3. https://review.openstack.org/#/c/237042/ For me, it's a nice milestone to have test_utils working on Python 3. It allows to port more interesting stuff and start the real portage work ;-) Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [swift] Plan to add Python 3 support to Swift
-Original Message- From: Victor Stinner <vstin...@redhat.com> Reply: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Date: February 12, 2016 at 10:43:42 To: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [swift] Plan to add Python 3 support to Swift > Le 12/02/2016 15:12, Ian Cordasco a écrit : > > Just to interject here, RFC 2616 is the one you're talking about. That > > encoding requirement was dropped when HTTP/1.1 was updated in the > > 7230-7235 RFCs. (...) > > Where VCHAR is any visible US ASCII character. So while UTF-8 is still > > a bad idea for the header value (and in fact, http.client on Python 3 > > will auto-encode headers to Latin 1) Latin 1 is no longer the > > requirement. > > > > For those interested, you can read up on headers in HTTP/1.1 here: > > https://tools.ietf.org/html/rfc7230#section-3.2 > > Oh thanks, it looks like my HTTP skills are rusty :-) > > For Swift, it's maybe better to always try to use UTF-8, but fallback to > Latin1 if an HTTP cannot be decoded from UTF-8. Swift has many clients > implemented in various programming languages, I'm not sure that all > clients use UTF-8. I also don't meant to imply that people actually follow the RFCs. ;-) > By the way, https://review.openstack.org/#/c/237027/ was merged, cool. Woot! > I fixed the third patch mentioned in my previous email to support > arbitrary byte strings for hash prefix and suffix in the configuration file: > https://review.openstack.org/#/c/236998/ > > I also updated my HTTP parser patch for Python 3. With these two > patches, test_utils now pass on Python 3. > https://review.openstack.org/#/c/237042/ > > For me, it's a nice milestone to have test_utils working on Python 3. It > allows to port more interesting stuff and start the real portage work ;-) I've said it before, but allow me to say it again - Thank you, Victor, for your tireless efforts to get OpenStack onto Python 3. -- Ian Cordasco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [swift] Plan to add Python 3 support to Swift
Hi, The Swift port was blocked during 6 months by a complex issue related to PyEClib. Good news: this issue was fixed 2 months ago. After the Liberty Summit, a plan was defined to port Swift to Python 3 (end of october), and I understood that the whole Swift team agreed on it. Some Python 3 changes were merged, but less than I expected. My 3 latest patches are waiting for reviews since october 19, 2015: https://review.openstack.org/#/c/237027/ https://review.openstack.org/#/c/236998/ https://review.openstack.org/#/c/237019/ I can write patches, but it takes time to rebase them and to regularly try various ways to try to get a review (like ping on IRC). Because of the general lack of interest of Swift developers for Python 3, I think that I will simply abandon my patches and let others port Swift. Victor Le 30/10/2015 06:54, John Dickinson a écrit : Thanks for the update. This seems like a reasonable way forward, but also one that will take a long time. Thank you for your work. I think this will result in larger and larger chunks of work, and so it will eventually result in large patches to move different components to py3. So you'll be able to start small, but the work will get larger as you go. You're right about needing the voting gate job. That should be the first priority for py3 work. --John On 30 Oct 2015, at 12:47, Victor Stinner wrote: Hi, We talked about Python 3 with Christian Schwede, Alistair Coles, Samuel Meritt, Jaivish Kothari and others (sorry, I don't recall all names :-/) during Swift contributor meetup. It looks like we had an agreement on how to add Python 3 support to Swift. The plan is: 1) Fix the gate-swift-python34 check job 2) Make the gate-swift-python34 check job voting 3) Port remaining code step by step (incremental development) Python 3 issues had been fixed in the past in Swift, but came back. So it's important to not reintroduce such regressions by making the gate voting. Christian said that he will explain the plan at the next Swift meeting (Wednesday). I don't think that I will be able to attend this meeting, I have another one at the same time with my team :-/ I can put this plan in a blueprint if you want. So we can refer to the blueprint in Python 3 changes. It's up to you. Plan in detail. (1) To fix the Python 3 job, the idea is to only run a subset of tests on Python 3. For example, if we fix Python 3 issues with dnspython (dnspython3) and PyEClib dependencies, we can run "nosetests test/unit/common/test_exceptions.py" on Python 3 (the test pass on Python 3). We need these two changes: * "py3: Update pbr and dnspython requirements" https://review.openstack.org/#/c/217423/ * "py3: Add py34 test environment to tox" https://review.openstack.org/#/c/199034/ (2) When the gate-swift-python34 check job will pass and we waited long enough to consider that it's stable, we can make it voting. At this point, we cannot introduced Python 3 regressions on the code tested on Python 3. Then the idea is to run more and more tests on Python 3. (3) Ok, now the interesting part. To port remaining code, following changes will enlarge the code coverage of Python 3 tests by adding new tests to tox.ini. For example, port utils.py to Python 3 and add test_utils.py to tox.ini. Misc questions. Q: "Is it possible to port Swift to Python 3 in a single patch?" A: Yes, it is technically possible. But it would be one unique giant patch simply impossible to review and that will conflict at each merged change. Some changes required by Python 3 need discussions and to make technical choices. It's more convenient to work on smaller patches. Q: "How much changes do we need to port Swift to Python ?" A: Sorry, I don't know. Since we cannot run all tests on Python 3 right now, we cannot see all issues. It's really hard to estimate the number of required changes. Anyway, the plan is to port the code step by step. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [swift] Plan to add Python 3 support to Swift
Thanks for the update. This seems like a reasonable way forward, but also one that will take a long time. Thank you for your work. I think this will result in larger and larger chunks of work, and so it will eventually result in large patches to move different components to py3. So you'll be able to start small, but the work will get larger as you go. You're right about needing the voting gate job. That should be the first priority for py3 work. --John On 30 Oct 2015, at 12:47, Victor Stinner wrote: > Hi, > > We talked about Python 3 with Christian Schwede, Alistair Coles, Samuel > Meritt, Jaivish Kothari and others (sorry, I don't recall all names :-/) > during Swift contributor meetup. It looks like we had an agreement on how to > add Python 3 support to Swift. The plan is: > > 1) Fix the gate-swift-python34 check job > > 2) Make the gate-swift-python34 check job voting > > 3) Port remaining code step by step (incremental development) > > Python 3 issues had been fixed in the past in Swift, but came back. So it's > important to not reintroduce such regressions by making the gate voting. > > Christian said that he will explain the plan at the next Swift meeting > (Wednesday). I don't think that I will be able to attend this meeting, I have > another one at the same time with my team :-/ > > I can put this plan in a blueprint if you want. So we can refer to the > blueprint in Python 3 changes. It's up to you. > > > Plan in detail. > > (1) To fix the Python 3 job, the idea is to only run a subset of tests on > Python 3. For example, if we fix Python 3 issues with dnspython (dnspython3) > and PyEClib dependencies, we can run > "nosetests test/unit/common/test_exceptions.py" on Python 3 (the test pass on > Python 3). > > We need these two changes: > > * "py3: Update pbr and dnspython requirements" > https://review.openstack.org/#/c/217423/ > > * "py3: Add py34 test environment to tox" > https://review.openstack.org/#/c/199034/ > > > (2) When the gate-swift-python34 check job will pass and we waited long > enough to consider that it's stable, we can make it voting. At this point, we > cannot introduced Python 3 regressions on the code tested on Python 3. Then > the idea is to run more and more tests on Python 3. > > > (3) Ok, now the interesting part. To port remaining code, following changes > will enlarge the code coverage of Python 3 tests by adding new tests to > tox.ini. For example, port utils.py to Python 3 and add test_utils.py to > tox.ini. > > > Misc questions. > > Q: "Is it possible to port Swift to Python 3 in a single patch?" > > A: Yes, it is technically possible. But it would be one unique giant patch > simply impossible to review and that will conflict at each merged change. > Some changes required by Python 3 need discussions and to make technical > choices. It's more convenient to work on smaller patches. > > Q: "How much changes do we need to port Swift to Python ?" > > A: Sorry, I don't know. Since we cannot run all tests on Python 3 right now, > we cannot see all issues. It's really hard to estimate the number of required > changes. Anyway, the plan is to port the code step by step. > > Victor > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
