Thank you! Shared on Slack as well. On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote:
> OK, tweeted the message. Could you share on Slack? > > On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <valen...@google.com> > wrote: > >> Alright, let's publish the message on Twitter and echo on Slack once >> that's done. >> Thank you! >> >> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote: >> >>> That sounds reasonable to me. I agree, it is better to get specific >>> reasons rather than a yes/no answer. >>> >>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <valen...@google.com> >>> wrote: >>> >>>> After thinking about it for a bit, I am not sure whether a poll would >>>> be actionable. For example, if 1000 users provide a positive response and 5 >>>> users provide a negative response, how do we interpret that and where do >>>> we draw a line? How about instead we encourage users to reach out, and tell >>>> what is not working for them? For example: >>>> >>>> Beam is considering making 2.23.0 a final release supporting Py2. If >>>> you are not able to switch to Python 3, please let us know why: [short link >>>> to user@ thread] [1]. >>>> >>>> Thoughts? >>>> >>>> [1] >>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E >>>> >>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> wrote: >>>> >>>>> >>>>> >>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev < >>>>> valen...@google.com> wrote: >>>>> >>>>>> I have reached out to user@ for feedback on Python 3 migration[1]. >>>>>> >>>>> >>>>> Maybe also ask on slack? There are quite a bit of users there as well. >>>>> >>>>> >>>>>> >>>>>> Could somebody from PMC please help with Twitter poll? >>>>>> >>>>> >>>>> I can try to do this. What question would you like to ask? I do not >>>>> know much about twitter polls but I assume they have character limits >>>>> similar to regular tweets. >>>>> >>>>> >>>>>> >>>>>> Technically, we can proceed with the change between 2.23.0 and >>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for >>>>>> users >>>>>> to respond. >>>>>> >>>>>> [1] >>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E >>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ieme...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Yes we need to poll this outside as Robert proposed. >>>>>>> >>>>>>> The proposed last support version corresponds with the next release >>>>>>> that will be >>>>>>> cut in two weeks. Sounds a bit short if we count the time to poll >>>>>>> people on this >>>>>>> subject but still could be. >>>>>> >>>>>> Technically, we can proceed with the change between 2.23.0 and >>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for >>>>>> users >>>>>> to respond. >>>>>> >>>>>> >>>>>>> I remember Chad mentioned in this thread the impossibility to get >>>>>>> support for >>>>>>> python 2 in his industry until the end of the year, Maybe things >>>>>>> have improved. >>>>>>> Have they? >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <rober...@google.com> >>>>>>> wrote: >>>>>>> > >>>>>>> > I like that option as a concrete proposal. I think we should >>>>>>> circulate it more widely (the users list, twitter poll, at least a new >>>>>>> thread...), maybe phrasing it as "is there any reason you couldn't >>>>>>> migrate >>>>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due to >>>>>>> be >>>>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, we >>>>>>> could consider holding on for one more release. >>>>>>> > >>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dcava...@google.com> >>>>>>> wrote: >>>>>>> >> >>>>>>> >> +1 >>>>>>> >> >>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> >>>>>>> wrote: >>>>>>> >>> >>>>>>> >>> +1 >>>>>>> >>> >>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> >>>>>>> wrote: >>>>>>> >>>> >>>>>>> >>>> As a concrete proposal, could we commit to removing python 2 >>>>>>> support by 2.24? In other words, mark the next release 2.23 as the last >>>>>>> python 2 compatible Beam version. >>>>>>> >>>> >>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev < >>>>>>> valen...@google.com> wrote: >>>>>>> >>>>> >>>>>>> >>>>> Another input here: >>>>>>> >>>>> >>>>>>> >>>>> If you opened a Python PR in the last few days, you probably >>>>>>> noticed that our test suites were broken by a transitive dependency of >>>>>>> Beam >>>>>>> that dropped python 2 support, but did not declare python_requires>=3 in >>>>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who >>>>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam >>>>>>> development[2]. >>>>>>> >>>>> >>>>>>> >>>>> This is the second time[3] Beam is affected with an issue of >>>>>>> this kind, so support of Python 2 starts to slow down our development, >>>>>>> and >>>>>>> add toil for maintainers of packages we depend on (both directly and >>>>>>> transitively). >>>>>>> >>>>> >>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152 >>>>>>> >>>>> [2] >>>>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E >>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131 >>>>>>> >>>>> >>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> >>>>>>> wrote: >>>>>>> >>>>>> >>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of >>>>>>> EOLing py2 support sooner than later. The reality is that we will not be >>>>>>> effectively supporting beam python 2 for a long time while the ecosystem >>>>>>> already EOLed python 2. That said, a significant chunk (but no longer a >>>>>>> majority) of our users are still using python 2. Upgrades are painful, >>>>>>> it >>>>>>> might be especially painful nowadays. It would be good to hear counter >>>>>>> view >>>>>>> points, user voices related to this. >>>>>>> >>>>>> >>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev < >>>>>>> valen...@google.com> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Back at the end of February we decided to revisit this >>>>>>> conversation in 3 months. Do folks on this thread have any new input or >>>>>>> perspective regarding us balancing "user pain/contributor pain/our >>>>>>> ability >>>>>>> to continuously test with python 2 in a shifting environment"? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Some new information on my end is that we have been seeing >>>>>>> steady adoption of Python 3 among Beam Python users in Dataflow, >>>>>>> particularly strong adoption among streaming users, and Dataflow is >>>>>>> sunsetting Python 2 support for all released Beam SDKs later this year >>>>>>> [1]. >>>>>>> We will have to remove Python 2 Beam test suites that use Dataflow when >>>>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL >>>>>>> (when we have to remove all Py2 suites), including performance tests >>>>>>> that >>>>>>> still use Dataflow on Python 3. >>>>>>> >>>>>>> >>>>>>> >>>>>>> I am curious how much motivation there is in the community >>>>>>> at this moment to continue Py2 support in Beam, whether any previous >>>>>>> Py3 >>>>>>> migration blockers were resolved or any new blockers discovered among >>>>>>> Beam >>>>>>> users. >>>>>>> >>>>>>> >>>>>>> >>>>>>> [1] >>>>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev < >>>>>>> valen...@google.com> wrote: >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> That's good news! Thanks for sharing. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies >>>>>>> that no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES >>>>>>> + >>>>>>> aws, gcp, and interactive extras): >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> hdfs >>>>>>> >>>>>>>> numpy >>>>>>> >>>>>>>> pyarrow >>>>>>> >>>>>>>> ipython >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> There are more if we include transitive dependencies and >>>>>>> test-only packages. I also remember encountering one issue last month >>>>>>> that >>>>>>> was broken only on Py2, which we had to go back and fix. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2 >>>>>>> support or have updates on previously mentioned Py3 migration blockers, >>>>>>> feel free to post them. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw < >>>>>>> rober...@google.com> wrote: >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a >>>>>>> milestone that >>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if >>>>>>> just briefly. >>>>>>> >>>>>>>>> >>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía < >>>>>>> ieme...@gmail.com> wrote: >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3 >>>>>>> months again. We need to balance between user pain/contributor pain/our >>>>>>> ability to continuously test with python 2 in a shifting environment. >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point >>>>>>> also distributions will probably be phasing out python2 by default which >>>>>>> definitely help in this direction. >>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay < >>>>>>> al...@google.com> wrote: >>>>>>> >>>>>>>>> >> >>>>>>> >>>>>>>>> >> >>>>>>> >>>>>>>>> >> >>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía < >>>>>>> ieme...@gmail.com> wrote: >>>>>>> >>>>>>>>> >>> >>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it a >>>>>>> bit more, even if it >>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some >>>>>>> workarounds as Robert suggests, >>>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing >>>>>>> the python 3 catchup game, >>>>>>> >>>>>>>>> >>> so worth to support those users. >>>>>>> >>>>>>>>> >>> >>>>>>> >>>>>>>>> >>> >>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state >>>>>>> later in the year. >>>>>>> >>>>>>>>> >> >>>>>>> >>>>>>>>> >> >>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3 >>>>>>> months again. We need to balance between user pain/contributor pain/our >>>>>>> ability to continuously test with python 2 in a shifting environment. >>>>>>> >>>>>>>>> >> >>>>>>> >>>>>>>>> >>> >>>>>>> >>>>>>>>> >>> In the >>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the >>>>>>> website with this info and >>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up to >>>>>>> date). >>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/ >>>>>>> >>>>>>>>> >> >>>>>>> >>>>>>>>> >> >>>>>>> >>>>>>>>> >> I made a minor change to update that page ( >>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive >>>>>>> update to that page and linked ( >>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would >>>>>>> still be welcome. >>>>>>> >>>>>>>>> >> >>>>>>> >>>>>>>>> >>> >>>>>>> >>>>>>>>> >>> >>>>>>> >>>>>>>>> >>> - Ismaël >>>>>>> >>>>>>>>> >>> >>>>>>> >>>>>>>>> >>> >>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw < >>>>>>> rober...@google.com> wrote: >>>>>>> >>>>>>>>> >>>> >>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova < >>>>>>> chad...@gmail.com> wrote: >>>>>>> >>>>>>>>> >>>> >> >>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the >>>>>>> type hints will have to be redone in the for 3.x. >>>>>>> >>>>>>>>> >>>> > >>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically >>>>>>> converting type comments to annotations: >>>>>>> https://github.com/ilevkivskyi/com2ann >>>>>>> >>>>>>>>> >>>> > >>>>>>> >>>>>>>>> >>>> > So don't let that part bother you. >>>>>>> >>>>>>>>> >>>> >>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily >>>>>>> automated. >>>>>>> >>>>>>>>> >>>> >>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be >>>>>>> using in the Beam source that you cannot now. >>>>>>> >>>>>>>>> >>>> >>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into >>>>>>> wanting keyword-only >>>>>>> >>>>>>>>> >>>> arguments the other day. >>>>>>> >>>>>>>>> >>>> >>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better. >>>>>>> >>>>>>>>> >>>> > >>>>>>> >>>>>>>>> >>>> > >>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a >>>>>>> refresher for those who care: some of the key vendors that support my >>>>>>> industry will not offer python3-compatible versions of their software >>>>>>> until >>>>>>> the 4th quarter of 2020. If Beam switches to python3-only before that >>>>>>> point we may be forced to stop contributing features (note: I'm the guy >>>>>>> who >>>>>>> added the type hints :). Every month you can give us would be greatly >>>>>>> appreciated. >>>>>>> >>>>>>>>> >>>> >>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 >>>>>>> for downloads at >>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a >>>>>>> grain of salt, but >>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to >>>>>>> be way higher for >>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty >>>>>>> noisy, but say it >>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least >>>>>>> mid-year before we >>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is >>>>>>> probably a >>>>>>> >>>>>>>>> >>>> stretch. >>>>>>> >>>>>>>>> >>>> >>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an >>>>>>> all-or-nothing thing as >>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 >>>>>>> only sooner than >>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well >>>>>>> justified.) Another >>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and >>>>>>> Python 3 in the >>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a >>>>>>> library that you need >>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back >>>>>>> your whole >>>>>>> >>>>>>>>> >>>> pipeline. >>>>>>> >>>>>>>>> >>>> >>>>>>> >>>>>>>>> >>>> - Robert >>>>>>> >>>>>>>>> >>>> >>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and >>>>>>> that 20% may just >>>>>>> >>>>>>>>> >>>> be a spike. >>>>>>> >>>>>>
