Good to hear from you and good to hear the news. Release branch cut date for 2.24 is 8/12.
On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova <chad...@gmail.com> wrote: > Hi all, > Sorry I've been AWOL. I've been pulled in a number of different > directions. > > We are still increasing our use of Beam, and I have it on my todo list to > reach out to Kenneth about how Beam could be expanded into the VFX / > Animation industries. > > Our industry uses a number of specialized applications with embedded > python interpreters. We run Beam inside these interpreters, so we're > waiting for them to switch to python3. > > Here's the status report for python3 adoption in our key applications: > > *Maya*: In Beta > *Houdini*: Released > *Nuke*: In Beta > *Katana*: Not started (Alpha?) > > I hate to be the one holding the project back, and I understand if you all > ultimately decide it's untenable to wait any longer. The good news is 3 > out of 4 applications should be ready in the next 2-3 months. I can do > some investigation into what workarounds might look like for Katana, or > maybe we can use the Beta version once python3 support arrives, which would > move our schedule forward. > > When would 2.24 release? > > -chad > > > > > > > > > > On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote: > >> OK, tweeted the message. Could you share on Slack? >> >> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <valen...@google.com> >> wrote: >> >>> Alright, let's publish the message on Twitter and echo on Slack once >>> that's done. >>> Thank you! >>> >>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote: >>> >>>> That sounds reasonable to me. I agree, it is better to get specific >>>> reasons rather than a yes/no answer. >>>> >>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev < >>>> valen...@google.com> wrote: >>>> >>>>> After thinking about it for a bit, I am not sure whether a poll would >>>>> be actionable. For example, if 1000 users provide a positive response and >>>>> 5 >>>>> users provide a negative response, how do we interpret that and where do >>>>> we draw a line? How about instead we encourage users to reach out, and >>>>> tell >>>>> what is not working for them? For example: >>>>> >>>>> Beam is considering making 2.23.0 a final release supporting Py2. If >>>>> you are not able to switch to Python 3, please let us know why: [short >>>>> link >>>>> to user@ thread] [1]. >>>>> >>>>> Thoughts? >>>>> >>>>> [1] >>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E >>>>> >>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev < >>>>>> valen...@google.com> wrote: >>>>>> >>>>>>> I have reached out to user@ for feedback on Python 3 migration[1]. >>>>>>> >>>>>> >>>>>> Maybe also ask on slack? There are quite a bit of users there as well. >>>>>> >>>>>> >>>>>>> >>>>>>> Could somebody from PMC please help with Twitter poll? >>>>>>> >>>>>> >>>>>> I can try to do this. What question would you like to ask? I do not >>>>>> know much about twitter polls but I assume they have character limits >>>>>> similar to regular tweets. >>>>>> >>>>>> >>>>>>> >>>>>>> Technically, we can proceed with the change between 2.23.0 and >>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for >>>>>>> users >>>>>>> to respond. >>>>>>> >>>>>>> [1] >>>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E >>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ieme...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Yes we need to poll this outside as Robert proposed. >>>>>>>> >>>>>>>> The proposed last support version corresponds with the next release >>>>>>>> that will be >>>>>>>> cut in two weeks. Sounds a bit short if we count the time to poll >>>>>>>> people on this >>>>>>>> subject but still could be. >>>>>>> >>>>>>> Technically, we can proceed with the change between 2.23.0 and >>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for >>>>>>> users >>>>>>> to respond. >>>>>>> >>>>>>> >>>>>>>> I remember Chad mentioned in this thread the impossibility to get >>>>>>>> support for >>>>>>>> python 2 in his industry until the end of the year, Maybe things >>>>>>>> have improved. >>>>>>>> Have they? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw < >>>>>>>> rober...@google.com> wrote: >>>>>>>> > >>>>>>>> > I like that option as a concrete proposal. I think we should >>>>>>>> circulate it more widely (the users list, twitter poll, at least a new >>>>>>>> thread...), maybe phrasing it as "is there any reason you couldn't >>>>>>>> migrate >>>>>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due >>>>>>>> to be >>>>>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, >>>>>>>> we >>>>>>>> could consider holding on for one more release. >>>>>>>> > >>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos < >>>>>>>> dcava...@google.com> wrote: >>>>>>>> >> >>>>>>>> >> +1 >>>>>>>> >> >>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> >>>>>>>> wrote: >>>>>>>> >>> >>>>>>>> >>> +1 >>>>>>>> >>> >>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> >>>>>>>> wrote: >>>>>>>> >>>> >>>>>>>> >>>> As a concrete proposal, could we commit to removing python 2 >>>>>>>> support by 2.24? In other words, mark the next release 2.23 as the last >>>>>>>> python 2 compatible Beam version. >>>>>>>> >>>> >>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev < >>>>>>>> valen...@google.com> wrote: >>>>>>>> >>>>> >>>>>>>> >>>>> Another input here: >>>>>>>> >>>>> >>>>>>>> >>>>> If you opened a Python PR in the last few days, you probably >>>>>>>> noticed that our test suites were broken by a transitive dependency of >>>>>>>> Beam >>>>>>>> that dropped python 2 support, but did not declare python_requires>=3 >>>>>>>> in >>>>>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users >>>>>>>> (who >>>>>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam >>>>>>>> development[2]. >>>>>>>> >>>>> >>>>>>>> >>>>> This is the second time[3] Beam is affected with an issue of >>>>>>>> this kind, so support of Python 2 starts to slow down our development, >>>>>>>> and >>>>>>>> add toil for maintainers of packages we depend on (both directly and >>>>>>>> transitively). >>>>>>>> >>>>> >>>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152 >>>>>>>> >>>>> [2] >>>>>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E >>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131 >>>>>>>> >>>>> >>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>> >>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of >>>>>>>> EOLing py2 support sooner than later. The reality is that we will not >>>>>>>> be >>>>>>>> effectively supporting beam python 2 for a long time while the >>>>>>>> ecosystem >>>>>>>> already EOLed python 2. That said, a significant chunk (but no longer a >>>>>>>> majority) of our users are still using python 2. Upgrades are painful, >>>>>>>> it >>>>>>>> might be especially painful nowadays. It would be good to hear counter >>>>>>>> view >>>>>>>> points, user voices related to this. >>>>>>>> >>>>>> >>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev < >>>>>>>> valen...@google.com> wrote: >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> Back at the end of February we decided to revisit this >>>>>>>> conversation in 3 months. Do folks on this thread have any new input or >>>>>>>> perspective regarding us balancing "user pain/contributor pain/our >>>>>>>> ability >>>>>>>> to continuously test with python 2 in a shifting environment"? >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> Some new information on my end is that we have been seeing >>>>>>>> steady adoption of Python 3 among Beam Python users in Dataflow, >>>>>>>> particularly strong adoption among streaming users, and Dataflow is >>>>>>>> sunsetting Python 2 support for all released Beam SDKs later this year >>>>>>>> [1]. >>>>>>>> We will have to remove Python 2 Beam test suites that use Dataflow >>>>>>>> when >>>>>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 >>>>>>>> EOL >>>>>>>> (when we have to remove all Py2 suites), including performance tests >>>>>>>> that >>>>>>>> still use Dataflow on Python 3. >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> I am curious how much motivation there is in the community >>>>>>>> at this moment to continue Py2 support in Beam, whether any previous >>>>>>>> Py3 >>>>>>>> migration blockers were resolved or any new blockers discovered among >>>>>>>> Beam >>>>>>>> users. >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> [1] >>>>>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev < >>>>>>>> valen...@google.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> That's good news! Thanks for sharing. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies >>>>>>>> that no longer release new py2 artifacts (I looked at >>>>>>>> REQUIRED_PACKAGES + >>>>>>>> aws, gcp, and interactive extras): >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> hdfs >>>>>>>> >>>>>>>> numpy >>>>>>>> >>>>>>>> pyarrow >>>>>>>> >>>>>>>> ipython >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> There are more if we include transitive dependencies and >>>>>>>> test-only packages. I also remember encountering one issue last month >>>>>>>> that >>>>>>>> was broken only on Py2, which we had to go back and fix. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2 >>>>>>>> support or have updates on previously mentioned Py3 migration blockers, >>>>>>>> feel free to post them. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw < >>>>>>>> rober...@google.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a >>>>>>>> milestone that >>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if >>>>>>>> just briefly. >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía < >>>>>>>> ieme...@gmail.com> wrote: >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3 >>>>>>>> months again. We need to balance between user pain/contributor pain/our >>>>>>>> ability to continuously test with python 2 in a shifting environment. >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point >>>>>>>> also distributions will probably be phasing out python2 by default >>>>>>>> which >>>>>>>> definitely help in this direction. >>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay < >>>>>>>> al...@google.com> wrote: >>>>>>>> >>>>>>>>> >> >>>>>>>> >>>>>>>>> >> >>>>>>>> >>>>>>>>> >> >>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía < >>>>>>>> ieme...@gmail.com> wrote: >>>>>>>> >>>>>>>>> >>> >>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it >>>>>>>> a bit more, even if it >>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some >>>>>>>> workarounds as Robert suggests, >>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing >>>>>>>> the python 3 catchup game, >>>>>>>> >>>>>>>>> >>> so worth to support those users. >>>>>>>> >>>>>>>>> >>> >>>>>>>> >>>>>>>>> >>> >>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state >>>>>>>> later in the year. >>>>>>>> >>>>>>>>> >> >>>>>>>> >>>>>>>>> >> >>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3 >>>>>>>> months again. We need to balance between user pain/contributor pain/our >>>>>>>> ability to continuously test with python 2 in a shifting environment. >>>>>>>> >>>>>>>>> >> >>>>>>>> >>>>>>>>> >>> >>>>>>>> >>>>>>>>> >>> In the >>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the >>>>>>>> website with this info and >>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up >>>>>>>> to date). >>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/ >>>>>>>> >>>>>>>>> >> >>>>>>>> >>>>>>>>> >> >>>>>>>> >>>>>>>>> >> I made a minor change to update that page ( >>>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive >>>>>>>> update to that page and linked ( >>>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) >>>>>>>> would still be welcome. >>>>>>>> >>>>>>>>> >> >>>>>>>> >>>>>>>>> >>> >>>>>>>> >>>>>>>>> >>> >>>>>>>> >>>>>>>>> >>> - Ismaël >>>>>>>> >>>>>>>>> >>> >>>>>>>> >>>>>>>>> >>> >>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw < >>>>>>>> rober...@google.com> wrote: >>>>>>>> >>>>>>>>> >>>> >>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova < >>>>>>>> chad...@gmail.com> wrote: >>>>>>>> >>>>>>>>> >>>> >> >>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the >>>>>>>> type hints will have to be redone in the for 3.x. >>>>>>>> >>>>>>>>> >>>> > >>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically >>>>>>>> converting type comments to annotations: >>>>>>>> https://github.com/ilevkivskyi/com2ann >>>>>>>> >>>>>>>>> >>>> > >>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you. >>>>>>>> >>>>>>>>> >>>> >>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily >>>>>>>> automated. >>>>>>>> >>>>>>>>> >>>> >>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be >>>>>>>> using in the Beam source that you cannot now. >>>>>>>> >>>>>>>>> >>>> >>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into >>>>>>>> wanting keyword-only >>>>>>>> >>>>>>>>> >>>> arguments the other day. >>>>>>>> >>>>>>>>> >>>> >>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better. >>>>>>>> >>>>>>>>> >>>> > >>>>>>>> >>>>>>>>> >>>> > >>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a >>>>>>>> refresher for those who care: some of the key vendors that support my >>>>>>>> industry will not offer python3-compatible versions of their software >>>>>>>> until >>>>>>>> the 4th quarter of 2020. If Beam switches to python3-only before that >>>>>>>> point we may be forced to stop contributing features (note: I'm the >>>>>>>> guy who >>>>>>>> added the type hints :). Every month you can give us would be greatly >>>>>>>> appreciated. >>>>>>>> >>>>>>>>> >>>> >>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 >>>>>>>> for downloads at >>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a >>>>>>>> grain of salt, but >>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to >>>>>>>> be way higher for >>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty >>>>>>>> noisy, but say it >>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least >>>>>>>> mid-year before we >>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is >>>>>>>> probably a >>>>>>>> >>>>>>>>> >>>> stretch. >>>>>>>> >>>>>>>>> >>>> >>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an >>>>>>>> all-or-nothing thing as >>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 >>>>>>>> only sooner than >>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well >>>>>>>> justified.) Another >>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 >>>>>>>> and Python 3 in the >>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a >>>>>>>> library that you need >>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back >>>>>>>> your whole >>>>>>>> >>>>>>>>> >>>> pipeline. >>>>>>>> >>>>>>>>> >>>> >>>>>>>> >>>>>>>>> >>>> - Robert >>>>>>>> >>>>>>>>> >>>> >>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , >>>>>>>> and that 20% may just >>>>>>>> >>>>>>>>> >>>> be a spike. >>>>>>>> >>>>>>>
