Kanban board for python 3: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245
(Thank you Davor!) Ahmet On Fri, Apr 6, 2018 at 6:32 PM, Reuven Lax <re...@google.com> wrote: > I had a similar problem. > > On Fri, Apr 6, 2018, 6:23 PM Ahmet Altay <al...@google.com> wrote: > >> I tried to create a shared kanban board but I failed. I think I am >> lacking some permission to create a shared filter. Could someone help with >> creating this? >> >> The filter I planned to use was "project = BEAM AND (parent = BEAM-2784 >> OR parent = BEAM-1251) ORDER BY Rank ASC" >> >> Ahmet >> >> On Fri, Apr 6, 2018 at 5:45 AM, Robbe Sneyders <robbe.sneyd...@ml6.eu> >> wrote: >> >>> Hi all, >>> >>> I don't seem to have the permissions to create a Kanban board or even >>> assign tasks to myself. Who could help me with this? >>> >>> I've updated the coders package pull request [1] and added the applied >>> strategy to the proposal document [2]. >>> It would be great to get some feedback on this, so we can start moving >>> forward with other subpackages. >>> >>> Kind regards, >>> Robbe >>> >>> [1] https://github.com/apache/beam/pull/4990 >>> [2] https://docs.google.com/document/d/1xDG0MWVlDKDPu_ >>> IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit?usp=sharing >>> >>> On Mon, 2 Apr 2018 at 21:07 Robbe Sneyders <robbe.sneyd...@ml6.eu> >>> wrote: >>> >>>> Hello Robert, >>>> >>>> I think a Kanban board on Jira as proposed by Ahmet can be helpful for >>>> this. I'll look into setting one up tomorrow. >>>> >>>> In the meantime, you can find the first pull request with the updated >>>> coders package here: >>>> https://github.com/apache/beam/pull/4990 >>>> >>>> Kind regards, >>>> Robbe >>>> >>>> On Fri, 30 Mar 2018 at 18:01 Robert Bradshaw <rober...@google.com> >>>> wrote: >>>> >>>>> On Fri, Mar 30, 2018 at 8:39 AM Robbe Sneyders <robbe.sneyd...@ml6.eu> >>>>> wrote: >>>>> >>>>>> Thanks Ahmet and Robert, >>>>>> >>>>>> I think we can work on different subpackages in parallel, but it's >>>>>> important to apply the same strategy everywhere. I'm currently working on >>>>>> applying step 1 (was mostly done already) and 2 of the proposal to the >>>>>> coders subpackage to create a first pull request. We can then discuss the >>>>>> applied strategy in detail before merging and applying it to the other >>>>>> subpackages. >>>>>> >>>>> >>>>> Sounds good. Again, could you document (in a more permanent/easy to >>>>> look up state than email) when packages are started/done? >>>>> >>>>> >>>>>> This strategy also includes the choice of automated tools. I'm >>>>>> focusing on writing python 3 code with python 2 compatibility, which >>>>>> means >>>>>> depending on the future package instead of the six package (which is >>>>>> already used in some places in the current code base). I have already >>>>>> noticed that this indeed requires a lot of manual work after running the >>>>>> automated script. >>>>>> The future package supports python 3.3+ compatibility, so I don't >>>>>> think there is a higher cost supporting 3.4 compared to 3.5+. >>>>>> >>>>> >>>>> Sure. It may incur a higher maintenance burden long-term though. >>>>> (Basically, if we go out the door with 3.4 it's a promise to support it >>>>> for >>>>> some time to come.) >>>>> >>>>> >>>>>> I have already added a tox environment to run pylint2 with the --py3k >>>>>> argument per updated subpackage, which should help avoid regression >>>>>> between >>>>>> step 2 and step 3 of the proposal. This update will be pushed with the >>>>>> first pull request. >>>>>> >>>>>> Kind regards, >>>>>> Robbe >>>>>> >>>>>> >>>>>> On Fri, 30 Mar 2018 at 02:22 Robert Bradshaw <rober...@google.com> >>>>>> wrote: >>>>>> >>>>>>> Thank you, Robbie, for your offer to help with contribution here. I >>>>>>> read over your doc and the one thing I'd like to add is that this work >>>>>>> is >>>>>>> very parallelizable, but if we have enough people looking at it we'll >>>>>>> want >>>>>>> some way to coordinate so as to not overlap work (or just waste time >>>>>>> discovering what's been done). Tracking individual JIRAs and PRs gets >>>>>>> unwieldy, perhaps a spreadsheet with modules/packages on one axis and >>>>>>> the >>>>>>> various automated/manual conversions along the other would be helpful? >>>>>>> >>>>>>> A note on automated tools, they're sometimes overly conservative, so >>>>>>> we should be sure to review the changes manually. (A typical example of >>>>>>> this is unnecessarily importing six.moves.xrange when there was no big >>>>>>> reason to use xrange over range in Python 2, or conversely using >>>>>>> list(range(...) in Python 3.) >>>>>>> >>>>>>> Also, +1 to targetting 3.4+ and upgrading tox to prevent >>>>>>> regressions. If there's a cost to supporting 3.4 as opposed to requiring >>>>>>> 3.5+ we should identify it and decide that before widespread >>>>>>> announcement. >>>>>>> >>>>>>> On Tue, Mar 27, 2018 at 2:27 PM Ahmet Altay <al...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Mar 27, 2018 at 7:12 AM, Holden Karau <hol...@pigscanfly.ca >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Mar 27, 2018 at 4:27 AM Robbe Sneyders < >>>>>>>>> robbe.sneyd...@ml6.eu> wrote: >>>>>>>>> >>>>>>>>>> Hi Anand, >>>>>>>>>> >>>>>>>>>> Thanks for the feedback. >>>>>>>>>> >>>>>>>>>> It should be no problem to run everything on DataflowRunner as >>>>>>>>>> well. >>>>>>>>>> Are there any performance tests in place to check for performance >>>>>>>>>> regressions? >>>>>>>>>> >>>>>>>>> >>>>>>>> Yes there is a suite (https://github.com/apache/ >>>>>>>> beam/blob/master/.test-infra/jenkins/job_beam_ >>>>>>>> PerformanceTests_Python.groovy). It may not be very comprehensive >>>>>>>> and seems to be failing for a while. I would not block python 3 work on >>>>>>>> performance for now. That is the unfortuante state of things. >>>>>>>> >>>>>>>> If anybody in the community is interested, this would be a great >>>>>>>> opportunity to help with benchmarks in general. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>> Some questions were raised in the proposal document which I want >>>>>>>>>> to add to this conversation: >>>>>>>>>> >>>>>>>>>> The first comment was about the targeted python 3 versions. We >>>>>>>>>> proposed to target 3.6 since it is the latest version available and >>>>>>>>>> added >>>>>>>>>> 3.5 because 3.6 adoption seems rather low (hard to find any relevant >>>>>>>>>> sources on this though). >>>>>>>>>> If the beam community prefers 3.4, I would propose to target 3.4 >>>>>>>>>> only during porting and add 3.5 and 3.6 later so we don't slow down >>>>>>>>>> the >>>>>>>>>> porting progress. 3.4 has the advantage of already being installed >>>>>>>>>> on the >>>>>>>>>> workers and allows pySpark pipelines to be moved over to beam more >>>>>>>>>> easily. >>>>>>>>>> It would be great to get some opinions on this. >>>>>>>>>> >>>>>>>>> >>>>>>>> My preference is to support 3.4+. I searched a bit on the web to >>>>>>>> understand the usage statistics for python 3, it seems like python 3.4 >>>>>>>> has >>>>>>>> ~20% usage and python 3.4+ has 99% (https://semaphoreci.com/blog/ >>>>>>>> 2017/10/18/python-versions-used-in-commercial-projects-in-2017.html). >>>>>>>> Based on that, I think it makes sense to support it. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>> Another comment was made on how to avoid regression during the >>>>>>>>>> porting progress. >>>>>>>>>> After applying step 1 and step 2, no python 3 compatibility lint >>>>>>>>>> warnings should remain, so it would be great if we could enforce >>>>>>>>>> this check >>>>>>>>>> for every pull request on an already updated subpackage. >>>>>>>>>> After applying step 3, all tests should run on python 3, so again >>>>>>>>>> it would be great if we can enforce these per updated subpackage. >>>>>>>>>> Any insights on how to best accomplish this? >>>>>>>>>> >>>>>>>>> So you can look at some of the recent changes to tox.ini in the >>>>>>>>> git log to see what we’ve done so far around this I suspect you can >>>>>>>>> repeat >>>>>>>>> that same pattern. >>>>>>>>> >>>>>>>> >>>>>>>> +1 updating tox.ini and adding new checks to run_mini_py3lint.sh >>>>>>>> would help a lot to prevent regressions. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Robbe >>>>>>>>>> >>>>>>>>>> On Fri, 23 Mar 2018 at 19:59 Ahmet Altay <al...@google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thank you Robbe. >>>>>>>>>>> >>>>>>>>>>> I reviewed the document it looks reasonable to me. I will touch >>>>>>>>>>> on some points that were not mentioned: >>>>>>>>>>> - Runner exercise different code paths. Doing auto conversions >>>>>>>>>>> and focusing on DirectRunner is not enough. It is worthwhile to run >>>>>>>>>>> things >>>>>>>>>>> on DataflowRunner as well. This can be triggered from Jenkins. It >>>>>>>>>>> will >>>>>>>>>>> validate that we are still compatible for python 2. >>>>>>>>>>> - Similar to above but with an eye on perf regressions. >>>>>>>>>>> >>>>>>>>>>> For project tracking on JIRA, please feel free to create any new >>>>>>>>>>> issues, close stale ones, or take ownership of any open issues. All >>>>>>>>>>> JIRAs >>>>>>>>>>> should be assigned to the people actively working on them. If you >>>>>>>>>>> wan to >>>>>>>>>>> track it in a separate way, you can also propose that. (For example >>>>>>>>>>> a >>>>>>>>>>> kanban board is used for portability effort which is fully >>>>>>>>>>> supported in >>>>>>>>>>> JIRA.) >>>>>>>>>>> >>>>>>>>>>> I will also call out to a few other people in addition to Holden >>>>>>>>>>> who helped out or showed interest in helping with Python 3. @cclaus, >>>>>>>>>>> @luke-zhu, @udim, @robertwb, @charlesccychen, @tvalentyn. You >>>>>>>>>>> can include these people (and myself) for reviews and other >>>>>>>>>>> questions that >>>>>>>>>>> you have. >>>>>>>>>>> >>>>>>>>>>> Welcome again, and looking forward to your contributions. >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> Ahmet >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Mar 23, 2018 at 9:27 AM, Robbe Sneyders < >>>>>>>>>>> robbe.sneyd...@ml6.eu> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello everyone, >>>>>>>>>>>> >>>>>>>>>>>> In the next month(s), me and my colleague Matthias will commit >>>>>>>>>>>> a lot of time and effort to python 3 support for beam and we would >>>>>>>>>>>> like to >>>>>>>>>>>> discuss the best way to go forward with this. >>>>>>>>>>>> >>>>>>>>>>>> We have drawn up a document [1] with a high level outline of >>>>>>>>>>>> the proposed approach and would like to get your feedback on this. >>>>>>>>>>>> >>>>>>>>>>>> The main Jira issue [2] for python 3 support has been mostly >>>>>>>>>>>> inactive for the past year. Other smaller issues have been opened, >>>>>>>>>>>> but it's >>>>>>>>>>>> hard to track the general progress. It would be great if anyone >>>>>>>>>>>> could offer >>>>>>>>>>>> some insights on how to best handle this project on Jira. >>>>>>>>>>>> >>>>>>>>>>>> @Holden Karau, you seem to have already put in a lot of effort >>>>>>>>>>>> to add python 3 support, so it would be great to get your insights >>>>>>>>>>>> and find >>>>>>>>>>>> a way to merge our efforts. >>>>>>>>>>>> >>>>>>>>>>>> Kind regards, >>>>>>>>>>>> Robbe >>>>>>>>>>>> >>>>>>>>>>>> [1] https://docs.google.com/document/d/1xDG0MWVlDKDPu_ >>>>>>>>>>>> IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit?usp=sharing >>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/BEAM-1251 >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> [image: https://ml6.eu] <https://ml6.eu/> >>>>>>>>>>>> >>>>>>>>>>>> * Robbe Sneyders* >>>>>>>>>>>> >>>>>>>>>>>> ML6 Gent >>>>>>>>>>>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> >>>>>>>>>>>> >>>>>>>>>>>> M: +32 474 71 31 08 <+32%20474%2071%2031%2008> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> [image: https://ml6.eu] <https://ml6.eu/> >>>>>>>>>> >>>>>>>>>> * Robbe Sneyders* >>>>>>>>>> >>>>>>>>>> ML6 Gent >>>>>>>>>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> >>>>>>>>>> >>>>>>>>>> M: +32 474 71 31 08 <+32%20474%2071%2031%2008> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>> >>>>>> [image: https://ml6.eu] <https://ml6.eu/> >>>>>> >>>>>> * Robbe Sneyders* >>>>>> >>>>>> ML6 Gent >>>>>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> >>>>>> >>>>>> M: +32 474 71 31 08 <+32%20474%2071%2031%2008> >>>>>> >>>>> -- >>>> >>>> [image: https://ml6.eu] <https://ml6.eu/> >>>> >>>> * Robbe Sneyders* >>>> >>>> ML6 Gent >>>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> >>>> >>>> M: +32 474 71 31 08 <+32%20474%2071%2031%2008> >>>> >>> -- >>> >>> [image: https://ml6.eu] <https://ml6.eu/> >>> >>> * Robbe Sneyders* >>> >>> ML6 Gent >>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl> >>> >>> M: +32 474 71 31 08 >>> >> >>
