+1 for removing Python 2.7 support sooner than later. I recently added a small feature in Beam Python and I found that having to write code that worked with Python2 was quite awkward and time consuming (needing to make sure code works for both 2 and 3 and doubling the Jenkins running time), and I can imagine that may hinder or even discourage new contributions.
On Thu, Jun 18, 2020 at 5:12 PM Ahmet Altay <al...@google.com> wrote: > Good to hear from you and good to hear the news. Release branch cut date > for 2.24 is 8/12. > > On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova <chad...@gmail.com> wrote: > >> Hi all, >> Sorry I've been AWOL. I've been pulled in a number of different >> directions. >> >> We are still increasing our use of Beam, and I have it on my todo list to >> reach out to Kenneth about how Beam could be expanded into the VFX / >> Animation industries. >> >> Our industry uses a number of specialized applications with embedded >> python interpreters. We run Beam inside these interpreters, so we're >> waiting for them to switch to python3. >> >> Here's the status report for python3 adoption in our key applications: >> >> *Maya*: In Beta >> *Houdini*: Released >> *Nuke*: In Beta >> *Katana*: Not started (Alpha?) >> >> I hate to be the one holding the project back, and I understand if you >> all ultimately decide it's untenable to wait any longer. The good news is >> 3 out of 4 applications should be ready in the next 2-3 months. I can do >> some investigation into what workarounds might look like for Katana, or >> maybe we can use the Beta version once python3 support arrives, which would >> move our schedule forward. >> >> When would 2.24 release? >> >> -chad >> >> >> >> >> >> >> >> >> >> On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote: >> >>> OK, tweeted the message. Could you share on Slack? >>> >>> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <valen...@google.com> >>> wrote: >>> >>>> Alright, let's publish the message on Twitter and echo on Slack once >>>> that's done. >>>> Thank you! >>>> >>>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote: >>>> >>>>> That sounds reasonable to me. I agree, it is better to get specific >>>>> reasons rather than a yes/no answer. >>>>> >>>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev < >>>>> valen...@google.com> wrote: >>>>> >>>>>> After thinking about it for a bit, I am not sure whether a poll would >>>>>> be actionable. For example, if 1000 users provide a positive response >>>>>> and 5 >>>>>> users provide a negative response, how do we interpret that and where do >>>>>> we draw a line? How about instead we encourage users to reach out, and >>>>>> tell >>>>>> what is not working for them? For example: >>>>>> >>>>>> Beam is considering making 2.23.0 a final release supporting Py2. If >>>>>> you are not able to switch to Python 3, please let us know why: [short >>>>>> link >>>>>> to user@ thread] [1]. >>>>>> >>>>>> Thoughts? >>>>>> >>>>>> [1] >>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E >>>>>> >>>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev < >>>>>>> valen...@google.com> wrote: >>>>>>> >>>>>>>> I have reached out to user@ for feedback on Python 3 migration[1]. >>>>>>>> >>>>>>> >>>>>>> Maybe also ask on slack? There are quite a bit of users there as >>>>>>> well. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Could somebody from PMC please help with Twitter poll? >>>>>>>> >>>>>>> >>>>>>> I can try to do this. What question would you like to ask? I do not >>>>>>> know much about twitter polls but I assume they have character limits >>>>>>> similar to regular tweets. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Technically, we can proceed with the change between 2.23.0 and >>>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for >>>>>>>> users >>>>>>>> to respond. >>>>>>>> >>>>>>>> [1] >>>>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E >>>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ieme...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yes we need to poll this outside as Robert proposed. >>>>>>>>> >>>>>>>>> The proposed last support version corresponds with the next >>>>>>>>> release that will be >>>>>>>>> cut in two weeks. Sounds a bit short if we count the time to poll >>>>>>>>> people on this >>>>>>>>> subject but still could be. >>>>>>>> >>>>>>>> Technically, we can proceed with the change between 2.23.0 and >>>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for >>>>>>>> users >>>>>>>> to respond. >>>>>>>> >>>>>>>> >>>>>>>>> I remember Chad mentioned in this thread the impossibility to get >>>>>>>>> support for >>>>>>>>> python 2 in his industry until the end of the year, Maybe things >>>>>>>>> have improved. >>>>>>>>> Have they? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw < >>>>>>>>> rober...@google.com> wrote: >>>>>>>>> > >>>>>>>>> > I like that option as a concrete proposal. I think we should >>>>>>>>> circulate it more widely (the users list, twitter poll, at least a new >>>>>>>>> thread...), maybe phrasing it as "is there any reason you couldn't >>>>>>>>> migrate >>>>>>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due >>>>>>>>> to be >>>>>>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, >>>>>>>>> we >>>>>>>>> could consider holding on for one more release. >>>>>>>>> > >>>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos < >>>>>>>>> dcava...@google.com> wrote: >>>>>>>>> >> >>>>>>>>> >> +1 >>>>>>>>> >> >>>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>> >>>>>>>>> >>> +1 >>>>>>>>> >>> >>>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>> >>>>>>>>> >>>> As a concrete proposal, could we commit to removing python 2 >>>>>>>>> support by 2.24? In other words, mark the next release 2.23 as the >>>>>>>>> last >>>>>>>>> python 2 compatible Beam version. >>>>>>>>> >>>> >>>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev < >>>>>>>>> valen...@google.com> wrote: >>>>>>>>> >>>>> >>>>>>>>> >>>>> Another input here: >>>>>>>>> >>>>> >>>>>>>>> >>>>> If you opened a Python PR in the last few days, you probably >>>>>>>>> noticed that our test suites were broken by a transitive dependency >>>>>>>>> of Beam >>>>>>>>> that dropped python 2 support, but did not declare python_requires>=3 >>>>>>>>> in >>>>>>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users >>>>>>>>> (who >>>>>>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam >>>>>>>>> development[2]. >>>>>>>>> >>>>> >>>>>>>>> >>>>> This is the second time[3] Beam is affected with an issue of >>>>>>>>> this kind, so support of Python 2 starts to slow down our >>>>>>>>> development, and >>>>>>>>> add toil for maintainers of packages we depend on (both directly and >>>>>>>>> transitively). >>>>>>>>> >>>>> >>>>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152 >>>>>>>>> >>>>> [2] >>>>>>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E >>>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131 >>>>>>>>> >>>>> >>>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of >>>>>>>>> EOLing py2 support sooner than later. The reality is that we will not >>>>>>>>> be >>>>>>>>> effectively supporting beam python 2 for a long time while the >>>>>>>>> ecosystem >>>>>>>>> already EOLed python 2. That said, a significant chunk (but no longer >>>>>>>>> a >>>>>>>>> majority) of our users are still using python 2. Upgrades are >>>>>>>>> painful, it >>>>>>>>> might be especially painful nowadays. It would be good to hear >>>>>>>>> counter view >>>>>>>>> points, user voices related to this. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev < >>>>>>>>> valen...@google.com> wrote: >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> Back at the end of February we decided to revisit this >>>>>>>>> conversation in 3 months. Do folks on this thread have any new input >>>>>>>>> or >>>>>>>>> perspective regarding us balancing "user pain/contributor pain/our >>>>>>>>> ability >>>>>>>>> to continuously test with python 2 in a shifting environment"? >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> Some new information on my end is that we have been seeing >>>>>>>>> steady adoption of Python 3 among Beam Python users in Dataflow, >>>>>>>>> particularly strong adoption among streaming users, and Dataflow is >>>>>>>>> sunsetting Python 2 support for all released Beam SDKs later this >>>>>>>>> year [1]. >>>>>>>>> We will have to remove Python 2 Beam test suites that use Dataflow >>>>>>>>> when >>>>>>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 >>>>>>>>> EOL >>>>>>>>> (when we have to remove all Py2 suites), including performance tests >>>>>>>>> that >>>>>>>>> still use Dataflow on Python 3. >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> I am curious how much motivation there is in the community >>>>>>>>> at this moment to continue Py2 support in Beam, whether any previous >>>>>>>>> Py3 >>>>>>>>> migration blockers were resolved or any new blockers discovered among >>>>>>>>> Beam >>>>>>>>> users. >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> [1] >>>>>>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev < >>>>>>>>> valen...@google.com> wrote: >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> That's good news! Thanks for sharing. >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies >>>>>>>>> that no longer release new py2 artifacts (I looked at >>>>>>>>> REQUIRED_PACKAGES + >>>>>>>>> aws, gcp, and interactive extras): >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> hdfs >>>>>>>>> >>>>>>>> numpy >>>>>>>>> >>>>>>>> pyarrow >>>>>>>>> >>>>>>>> ipython >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> There are more if we include transitive dependencies and >>>>>>>>> test-only packages. I also remember encountering one issue last month >>>>>>>>> that >>>>>>>>> was broken only on Py2, which we had to go back and fix. >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2 >>>>>>>>> support or have updates on previously mentioned Py3 migration >>>>>>>>> blockers, >>>>>>>>> feel free to post them. >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw < >>>>>>>>> rober...@google.com> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a >>>>>>>>> milestone that >>>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if >>>>>>>>> just briefly. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía < >>>>>>>>> ieme...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3 >>>>>>>>> months again. We need to balance between user pain/contributor >>>>>>>>> pain/our >>>>>>>>> ability to continuously test with python 2 in a shifting environment. >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that >>>>>>>>> point also distributions will probably be phasing out python2 by >>>>>>>>> default >>>>>>>>> which definitely help in this direction. >>>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay < >>>>>>>>> al...@google.com> wrote: >>>>>>>>> >>>>>>>>> >> >>>>>>>>> >>>>>>>>> >> >>>>>>>>> >>>>>>>>> >> >>>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía < >>>>>>>>> ieme...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> >>> >>>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it >>>>>>>>> a bit more, even if it >>>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some >>>>>>>>> workarounds as Robert suggests, >>>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing >>>>>>>>> the python 3 catchup game, >>>>>>>>> >>>>>>>>> >>> so worth to support those users. >>>>>>>>> >>>>>>>>> >>> >>>>>>>>> >>>>>>>>> >>> >>>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state >>>>>>>>> later in the year. >>>>>>>>> >>>>>>>>> >> >>>>>>>>> >>>>>>>>> >> >>>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3 >>>>>>>>> months again. We need to balance between user pain/contributor >>>>>>>>> pain/our >>>>>>>>> ability to continuously test with python 2 in a shifting environment. >>>>>>>>> >>>>>>>>> >> >>>>>>>>> >>>>>>>>> >>> >>>>>>>>> >>>>>>>>> >>> In the >>>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in >>>>>>>>> the website with this info and >>>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up >>>>>>>>> to date). >>>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/ >>>>>>>>> >>>>>>>>> >> >>>>>>>>> >>>>>>>>> >> >>>>>>>>> >>>>>>>>> >> I made a minor change to update that page ( >>>>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive >>>>>>>>> update to that page and linked ( >>>>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) >>>>>>>>> would still be welcome. >>>>>>>>> >>>>>>>>> >> >>>>>>>>> >>>>>>>>> >>> >>>>>>>>> >>>>>>>>> >>> >>>>>>>>> >>>>>>>>> >>> - Ismaël >>>>>>>>> >>>>>>>>> >>> >>>>>>>>> >>>>>>>>> >>> >>>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw < >>>>>>>>> rober...@google.com> wrote: >>>>>>>>> >>>>>>>>> >>>> >>>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova < >>>>>>>>> chad...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> >>>> >> >>>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the >>>>>>>>> type hints will have to be redone in the for 3.x. >>>>>>>>> >>>>>>>>> >>>> > >>>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically >>>>>>>>> converting type comments to annotations: >>>>>>>>> https://github.com/ilevkivskyi/com2ann >>>>>>>>> >>>>>>>>> >>>> > >>>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you. >>>>>>>>> >>>>>>>>> >>>> >>>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily >>>>>>>>> automated. >>>>>>>>> >>>>>>>>> >>>> >>>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be >>>>>>>>> using in the Beam source that you cannot now. >>>>>>>>> >>>>>>>>> >>>> >>>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into >>>>>>>>> wanting keyword-only >>>>>>>>> >>>>>>>>> >>>> arguments the other day. >>>>>>>>> >>>>>>>>> >>>> >>>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better. >>>>>>>>> >>>>>>>>> >>>> > >>>>>>>>> >>>>>>>>> >>>> > >>>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a >>>>>>>>> refresher for those who care: some of the key vendors that support my >>>>>>>>> industry will not offer python3-compatible versions of their software >>>>>>>>> until >>>>>>>>> the 4th quarter of 2020. If Beam switches to python3-only before that >>>>>>>>> point we may be forced to stop contributing features (note: I'm the >>>>>>>>> guy who >>>>>>>>> added the type hints :). Every month you can give us would be >>>>>>>>> greatly >>>>>>>>> appreciated. >>>>>>>>> >>>>>>>>> >>>> >>>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 >>>>>>>>> for downloads at >>>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a >>>>>>>>> grain of salt, but >>>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to >>>>>>>>> be way higher for >>>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty >>>>>>>>> noisy, but say it >>>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least >>>>>>>>> mid-year before we >>>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 >>>>>>>>> is probably a >>>>>>>>> >>>>>>>>> >>>> stretch. >>>>>>>>> >>>>>>>>> >>>> >>>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an >>>>>>>>> all-or-nothing thing as >>>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 >>>>>>>>> only sooner than >>>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well >>>>>>>>> justified.) Another >>>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 >>>>>>>>> and Python 3 in the >>>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a >>>>>>>>> library that you need >>>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back >>>>>>>>> your whole >>>>>>>>> >>>>>>>>> >>>> pipeline. >>>>>>>>> >>>>>>>>> >>>> >>>>>>>>> >>>>>>>>> >>>> - Robert >>>>>>>>> >>>>>>>>> >>>> >>>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , >>>>>>>>> and that 20% may just >>>>>>>>> >>>>>>>>> >>>> be a spike. >>>>>>>>> >>>>>>>>
