Hi all, We've been making quite some progress these last weeks. I'll give a short status update on where we are right now.
Our current goal is to make all unittests succeed in Python 3. We are currently at: 1842 tests: (SKIP=350, errors=100, failures=9) All of these remaining errors and failures are in the io and examples packages, except for some non-blocking failing typehints tests. All other packages have been ported. We're currently working on porting the io package, and have finished all core modules (iobase, filesystemio, filebasedsource/sink, textio, ...). This allowed us to fix the wordcount end to end examples, which now run successfully on Python 3! [1] Next up are all other io sources and sinks. Most of the skipped tests are due to missing GCP components in the Python 3 test suite. We will add a separate GCP test suite for Python 3 when we start porting the first GCP io module, which should be very soon. Anyone who wants to help out, can start with porting one of these sources/sinks. Kind regards, Robbe [1] https://github.com/apache/beam/pull/7447 [image: https://ml6.eu] <https://ml6.eu/> * Robbe Sneyders* ML6 Gent <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> M: +32 474 71 31 08 On Tue, 8 Jan 2019 at 04:58, Ahmet Altay <al...@google.com> wrote: > +Matthias Feys <matth...@ml6.eu> +Valentyn Tymofieiev > <valen...@google.com> +Mark Liu <mark...@google.com> could add more > details here since they are working on Python 3 for a while now. > > Hopeful state is that we might have python 3 working with DirectRunner in > the release after this one (2.12). Mark is also working on getting python 3 > working on a cluster. He has been able to run WordCount on Dataflow service > with some hacks but it was not yet ready to run out of the box. I would > like to note that we are targeting python 3 support only for portable > runners so running on Dataflow and Flink should happen at the same time. > > Matthias and Valentyn are still working on converting the SDK to be Python > 3 compatible. They are now mostly dealing with harder to convert parts of > the SDK. (e.g. Parts were python 2/3 differences results in performance > regressions or subtle changes in the behavior). > > To the folks working on this, it would be really helpful if you could > update BEAM-1251 regularly. We have shared this issue with many people and > not all of them will read this thread. > > Ahmet > > On Mon, Jan 7, 2019 at 8:15 AM Maximilian Michels <m...@apache.org> wrote: > >> Also curious because I see Python 3 requests quite often. I always say, >> we're >> close, but how close are we? :) >> >> Thanks, >> Max >> >> On 05.01.19 00:03, Manu Zhang wrote: >> > Guys, >> > >> > Happy New Year !!! >> > I haven't got much time to contribute to Python 3 support. What is the >> progress >> > now ? It seems there are quite a few open issues under >> > https://issues.apache.org/jira/browse/BEAM-1251. People have kept >> asking about >> > Python 3 support in tf.transform >> > (https://github.com/tensorflow/transform/issues/1) which is blocked by >> BEAM-1251. >> > >> > Thanks, >> > Manu Zhang >> > >> > >> > On Fri, Oct 12, 2018 at 3:17 AM Valentyn Tymofieiev < >> valen...@google.com >> > <mailto:valen...@google.com>> wrote: >> > >> > I cc'ed a few folks who are familiar with Jenkins setup on >> > https://issues.apache.org/jira/browse/BEAM-5663, I think we can >> continue the >> > discussion there or start a separate thread. >> > >> > On Wed, Oct 10, 2018 at 8:54 PM Manu Zhang <owenzhang1...@gmail.com >> > <mailto:owenzhang1...@gmail.com>> wrote: >> > >> > Does anyone know how to set up python version on Jenkins ? It’s >> Python >> > 3.5.2 now. >> > >> > Thanks, >> > Manu Zhang >> > On Oct 5, 2018, 9:24 AM +0800, Valentyn Tymofieiev < >> valen...@google.com >> > <mailto:valen...@google.com>>, wrote: >> >> I have put together a guide [1] to help get started with >> investigating >> >> Python 3-related test failures that may be helpful for new >> folks >> >> joining the effort. >> >> >> >> Comments and improvements welcome! >> >> >> >> Thanks, >> >> Valentyn >> >> [1] >> >> >> https://docs.google.com/document/d/1s1BJVCY65LB_SYK1SU1u7NbZiFANoq-nEYaEvzRbYlA >> >> >> >> >> >> On Thu, Oct 4, 2018 at 11:26 AM Valentyn Tymofieiev >> >> <valen...@google.com <mailto:valen...@google.com>> wrote: >> >> >> >> I agree there is some overlap between JIRAs that track >> individual >> >> failures and module-level JIRAs. We originally wanted to >> do the >> >> conversion on a module-by-module basis, however we learned >> that >> >> test failures in some modules require changes in other >> modules, >> >> and it may be a little easier to slice the problem if we >> focus on >> >> classes of failures. >> >> >> >> Module-level JIRAs can still be useful for tracking the end >> >> result: tox suites cover all tests in the module in Py3 >> >> environment, and there are no disabled tests in the module >> that >> >> don't have individual JIRAs tracking them. >> >> >> >> I suggest that folks who are working on module-level JIRAs >> assign >> >> to themselves the JIRAs that track individual failures >> if/when >> >> they are actively addressing them. This way, unassigned >> >> problem-specific JIRAs can use help from the community. >> >> >> >> Thanks, >> >> Valentyn >> >> >> >> >> >> On Wed, Oct 3, 2018 at 8:14 PM Manu Zhang < >> owenzhang1...@gmail.com >> >> <mailto:owenzhang1...@gmail.com>> wrote: >> >> >> >> Thanks Valentyn. Note some test failing issues are >> covered by >> >> “Finish Python 3 porting for *** module”, e.g. >> >> https://issues.apache.org/jira/browse/BEAM-5315. >> >> >> >> Manu >> >> 在 2018年10月3日 +0800 PM4:18，Valentyn Tymofieiev >> >> <valen...@google.com <mailto:valen...@google.com>>，写道： >> >>> Hi Rakesh and Manu, >> >>> >> >>> Thanks to both of you for offering help (in different >> >>> threads). It's great to see that more and more people >> get >> >>> involved with helping to make Beam Python 3 >> compatible! >> >>> >> >>> There are a few PRs in flight, and several people in >> the >> >>> community actively work on Python 3 support now. I >> would be >> >>> happy to coordinate the work so that we don't step at >> each >> >>> others toes and avoid duplication of effort. >> >>> >> >>> I recently looked at unit tests that are still >> failing in >> >>> Python 3 environment and filed a few issues (within >> range >> >>> BEAM-5615 - BEAM-5629), to track similar classes of >> errors. >> >>> You can also find them on Kanban board [1]. >> >>> In particular, BEAM-5620 and BEAM-5627 should be easy >> issues >> >>> to get started. >> >>> >> >>> There are multiple ways you can help: >> >>> - Helping to rootcause errors. Even a comment why a >> test is >> >>> failing and a suggestion how to fix it, will be >> helpful for >> >>> others when you don't have time to do the fix. >> >>> - Helping with code reviews. >> >>> - Reporting new issues (as subtasks to BEAM-1251), >> >>> deduplicating or splitting the existing issues. We >> probably >> >>> don't want to file a Jira for each of 250+ currently >> failing >> >>> tests at this point, but it may make sense to track >> the >> >>> errors that occur repeatedly share the root cause. >> >>> - Fixing the issues. Feel free to assign an issue to >> yourself >> >>> if you have a fix in mind and plan to actively work >> on it. >> >>> Due to the nature of the problem it may occasionally >> happen >> >>> that two issues share the rootcause, or fixing one >> issue is a >> >>> prerequisite for fixing another issue, so sync to >> master >> >>> often to make sure the issue you are working on is not >> >>> already fixed. >> >>> >> >>> I'll also keep an eye on the PRs and will try to keep >> the >> >>> list of open issues up to date. >> >>> >> >>> Thanks, >> >>> Valentyn >> >>> >> >>> [1]: >> >>> >> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail >> >>> >> >>> >> >>> On Tue, Oct 2, 2018 at 9:38 AM Pablo Estrada >> >>> <pabl...@google.com <mailto:pabl...@google.com>> >> wrote: >> >>> >> >>> Very cool : ) I'm also available to review / >> merge if you >> >>> need help from my side. >> >>> Best >> >>> -P. >> >>> >> >>> On Tue, Oct 2, 2018 at 7:45 AM Rakesh Kumar >> >>> <rakeshku...@lyft.com <mailto: >> rakeshku...@lyft.com>> wrote: >> >>> >> >>> Hi Rob, >> >>> >> >>> I am, Rakesh Kumar, using Beam SDK for one of >> my >> >>> projects at Lyft. I have been working closely >> with >> >>> Thomas Weise. I have already met a couple of >> Python >> >>> SDK developers in person. >> >>> I am interested to help migrate to Python 3. >> You can >> >>> assign me PRs for review. I am also more than >> happy >> >>> to take a simple ticket to begin development >> work on >> >>> Beam. >> >>> >> >>> Thank you, >> >>> Rakesh >> >>> >> >>> On Wed, Sep 5, 2018 at 9:12 AM Robbe Sneyders >> >>> <robbe.sneyd...@ml6.eu >> >>> <mailto:robbe.sneyd...@ml6.eu>> wrote: >> >>> >> >>> Hi everyone, >> >>> >> >>> With the merging of [1], we now have >> Python 3 >> >>> tests running on Jenkins, which allows us >> to move >> >>> forward with the last step of the Python >> 3 porting. >> >>> >> >>> You can follow the progress on the Jira >> Kanban >> >>> Board [2]. If you're interested in >> helping by >> >>> porting a module, you can assign one of >> the >> >>> issues to yourself and start coding. You >> can find >> >>> the different steps outlined in the design >> >>> document [3]. >> >>> >> >>> We could also use some extra reviewers. >> If you're >> >>> interested, let us know, and we'll tag >> you in our >> >>> PRs. >> >>> >> >>> [1] >> https://github.com/apache/beam/pull/6266 >> >>> [2] >> >>> >> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245 >> >>> [3] https://s.apache.org/beam-python-3 >> >>> >> >>> kind regards, >> >>> Robbe >> >>> -- >> >>> >> >>> https://ml6.eu <https://ml6.eu/> >> >>> >> >>> >> >>> >> >>> >> >>> *Robbe Sneyders* >> >>> >> >>> ML6 Gent >> >>> < >> https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl >> > >> >>> >> >>> M: +32 474 71 31 08 >> <tel:+32%20474%2071%2031%2008> >> >>> >> >>> -- >> >>> Rakesh Kumar >> >>> Software Engineer >> >>> 510-761-1364 <tel:(510)%20761-1364> | >> >>> <https://www.lyft.com/> >> >>> >> >
