I have reached out to user@ for feedback on Python 3 migration[1]. Could somebody from PMC please help with Twitter poll?
Technically, we can proceed with the change between 2.23.0 and 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users to respond. [1] https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ieme...@gmail.com> wrote: > Yes we need to poll this outside as Robert proposed. > > The proposed last support version corresponds with the next release that > will be > cut in two weeks. Sounds a bit short if we count the time to poll people > on this > subject but still could be. Technically, we can proceed with the change between 2.23.0 and 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users to respond. > I remember Chad mentioned in this thread the impossibility to get support > for > python 2 in his industry until the end of the year, Maybe things have > improved. > Have they? > > > > > On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <rober...@google.com> > wrote: > > > > I like that option as a concrete proposal. I think we should circulate > it more widely (the users list, twitter poll, at least a new thread...), > maybe phrasing it as "is there any reason you couldn't migrate to Python 3 > (or stick with an older version of Beam) after 2.23 (due to be cut here in > a couple of weeks)?" If there is strong concern/pushback, we could consider > holding on for one more release. > > > > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dcava...@google.com> > wrote: > >> > >> +1 > >> > >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> wrote: > >>> > >>> +1 > >>> > >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> wrote: > >>>> > >>>> As a concrete proposal, could we commit to removing python 2 support > by 2.24? In other words, mark the next release 2.23 as the last python 2 > compatible Beam version. > >>>> > >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev < > valen...@google.com> wrote: > >>>>> > >>>>> Another input here: > >>>>> > >>>>> If you opened a Python PR in the last few days, you probably noticed > that our test suites were broken by a transitive dependency of Beam that > dropped python 2 support, but did not declare python_requires>=3 in its > setup.py [1]. This temporarily broke a subset of Beam Py2 users (who did > not explicitly pin the 'rsa' dependency), and still affects Beam > development[2]. > >>>>> > >>>>> This is the second time[3] Beam is affected with an issue of this > kind, so support of Python 2 starts to slow down our development, and add > toil for maintainers of packages we depend on (both directly and > transitively). > >>>>> > >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152 > >>>>> [2] > https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E > >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131 > >>>>> > >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> wrote: > >>>>>> > >>>>>> Thank you for re-opening this Valentyn. I am in favor of EOLing py2 > support sooner than later. The reality is that we will not be effectively > supporting beam python 2 for a long time while the ecosystem already EOLed > python 2. That said, a significant chunk (but no longer a majority) of our > users are still using python 2. Upgrades are painful, it might be > especially painful nowadays. It would be good to hear counter view points, > user voices related to this. > >>>>>> > >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev < > valen...@google.com> wrote: > >>>>>>> > >>>>>>> Back at the end of February we decided to revisit this > conversation in 3 months. Do folks on this thread have any new input or > perspective regarding us balancing "user pain/contributor pain/our ability > to continuously test with python 2 in a shifting environment"? > >>>>>>> > >>>>>>> Some new information on my end is that we have been seeing steady > adoption of Python 3 among Beam Python users in Dataflow, particularly > strong adoption among streaming users, and Dataflow is sunsetting Python 2 > support for all released Beam SDKs later this year [1]. We will have to > remove Python 2 Beam test suites that use Dataflow when Dataflow runner > disables Py2 support if this happens before Beam Py2 EOL (when we have to > remove all Py2 suites), including performance tests that still use Dataflow > on Python 3. > >>>>>>> > >>>>>>> I am curious how much motivation there is in the community at this > moment to continue Py2 support in Beam, whether any previous Py3 migration > blockers were resolved or any new blockers discovered among Beam users. > >>>>>>> > >>>>>>> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow > >>>>>>> > >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev < > valen...@google.com> wrote: > >>>>>>>> > >>>>>>>> That's good news! Thanks for sharing. > >>>>>>>> > >>>>>>>> Another datapoint, here are a few of Beam's dependencies that no > longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws, > gcp, and interactive extras): > >>>>>>>> > >>>>>>>> hdfs > >>>>>>>> numpy > >>>>>>>> pyarrow > >>>>>>>> ipython > >>>>>>>> > >>>>>>>> There are more if we include transitive dependencies and > test-only packages. I also remember encountering one issue last month that > was broken only on Py2, which we had to go back and fix. > >>>>>>>> > >>>>>>>> If others have noticed frictions related to ongoing Py2 support > or have updates on previously mentioned Py3 migration blockers, feel free > to post them. > >>>>>>>> > >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw < > rober...@google.com> wrote: > >>>>>>>>> > >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a > milestone that > >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just > briefly. > >>>>>>>>> > >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ieme...@gmail.com> > wrote: > >>>>>>>>> > > >>>>>>>>> > > I would suggest re-evaluating this within the next 3 months > again. We need to balance between user pain/contributor pain/our ability to > continuously test with python 2 in a shifting environment. > >>>>>>>>> > > >>>>>>>>> > Good idea for the in 3 months evaluation, at that point also > distributions will probably be phasing out python2 by default which > definitely help in this direction. > >>>>>>>>> > Thanks for updating the roadmap Ahmet > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com> > wrote: > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía < > ieme...@gmail.com> wrote: > >>>>>>>>> >>> > >>>>>>>>> >>> I am with Chad on this, we should probably extend it a bit > more, even if it > >>>>>>>>> >>> makes us struggle a bit at least we have some workarounds as > Robert suggests, > >>>>>>>>> >>> and as Chad said there are still many people playing the > python 3 catchup game, > >>>>>>>>> >>> so worth to support those users. > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> But maybe it is worth to evaluate the current state later in > the year. > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> I would suggest re-evaluating this within the next 3 months > again. We need to balance between user pain/contributor pain/our ability to > continuously test with python 2 in a shifting environment. > >>>>>>>>> >> > >>>>>>>>> >>> > >>>>>>>>> >>> In the > >>>>>>>>> >>> meantime can someone please update our Roadmap in the > website with this info and > >>>>>>>>> >>> where we are with Python 3 support (it looks not up to date). > >>>>>>>>> >>> https://beam.apache.org/roadmap/ > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> I made a minor change to update that page ( > https://github.com/apache/beam/pull/10848). A more comprehensive update > to that page and linked ( > https://beam.apache.org/roadmap/python-sdk/#python-3-support) would still > be welcome. > >>>>>>>>> >> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> - Ismaël > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw < > rober...@google.com> wrote: > >>>>>>>>> >>>> > >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova < > chad...@gmail.com> wrote: > >>>>>>>>> >>>> >> > >>>>>>>>> >>>> >> Not to mention that all the nice work for the type > hints will have to be redone in the for 3.x. > >>>>>>>>> >>>> > > >>>>>>>>> >>>> > Note that there's a tool for automatically converting > type comments to annotations: https://github.com/ilevkivskyi/com2ann > >>>>>>>>> >>>> > > >>>>>>>>> >>>> > So don't let that part bother you. > >>>>>>>>> >>>> > >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily automated. > >>>>>>>>> >>>> > >>>>>>>>> >>>> > I'm curious what other features you'd like to be using in > the Beam source that you cannot now. > >>>>>>>>> >>>> > >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting > keyword-only > >>>>>>>>> >>>> arguments the other day. > >>>>>>>>> >>>> > >>>>>>>>> >>>> >> It seems the faster we drop support the better. > >>>>>>>>> >>>> > > >>>>>>>>> >>>> > > >>>>>>>>> >>>> > I've already gone over my position on this, but a > refresher for those who care: some of the key vendors that support my > industry will not offer python3-compatible versions of their software until > the 4th quarter of 2020. If Beam switches to python3-only before that > point we may be forced to stop contributing features (note: I'm the guy who > added the type hints :). Every month you can give us would be greatly > appreciated. > >>>>>>>>> >>>> > >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for > downloads at > >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain of > salt, but > >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be way > higher for > >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, > but say it > >>>>>>>>> >>>> doubles every 3 months that would put us at least mid-year > before we > >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is > probably a > >>>>>>>>> >>>> stretch. > >>>>>>>>> >>>> > >>>>>>>>> >>>> We could consider whether it needs to be an all-or-nothing > thing as > >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only > sooner than > >>>>>>>>> >>>> the whole codebase. (This would have to be well justified.) > Another > >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and > Python 3 in the > >>>>>>>>> >>>> same pipeline with portability, so if there's a library > that you need > >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your > whole > >>>>>>>>> >>>> pipeline. > >>>>>>>>> >>>> > >>>>>>>>> >>>> - Robert > >>>>>>>>> >>>> > >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that > 20% may just > >>>>>>>>> >>>> be a spike. >
