On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <valen...@google.com> wrote:
> I have reached out to user@ for feedback on Python 3 migration[1]. > Maybe also ask on slack? There are quite a bit of users there as well. > > Could somebody from PMC please help with Twitter poll? > I can try to do this. What question would you like to ask? I do not know much about twitter polls but I assume they have character limits similar to regular tweets. > > Technically, we can proceed with the change between 2.23.0 and 2.24.0, so > that's after 2.23.0 is cut and we give sufficient time for users to > respond. > > [1] > https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E > On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ieme...@gmail.com> wrote: > >> Yes we need to poll this outside as Robert proposed. >> >> The proposed last support version corresponds with the next release that >> will be >> cut in two weeks. Sounds a bit short if we count the time to poll people >> on this >> subject but still could be. > > Technically, we can proceed with the change between 2.23.0 and 2.24.0, so > that's after 2.23.0 is cut and we give sufficient time for users to > respond. > > >> I remember Chad mentioned in this thread the impossibility to get support >> for >> python 2 in his industry until the end of the year, Maybe things have >> improved. >> Have they? >> >> >> >> >> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <rober...@google.com> >> wrote: >> > >> > I like that option as a concrete proposal. I think we should circulate >> it more widely (the users list, twitter poll, at least a new thread...), >> maybe phrasing it as "is there any reason you couldn't migrate to Python 3 >> (or stick with an older version of Beam) after 2.23 (due to be cut here in >> a couple of weeks)?" If there is strong concern/pushback, we could consider >> holding on for one more release. >> > >> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dcava...@google.com> >> wrote: >> >> >> >> +1 >> >> >> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> wrote: >> >>> >> >>> +1 >> >>> >> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> wrote: >> >>>> >> >>>> As a concrete proposal, could we commit to removing python 2 support >> by 2.24? In other words, mark the next release 2.23 as the last python 2 >> compatible Beam version. >> >>>> >> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> >>>>> >> >>>>> Another input here: >> >>>>> >> >>>>> If you opened a Python PR in the last few days, you probably >> noticed that our test suites were broken by a transitive dependency of Beam >> that dropped python 2 support, but did not declare python_requires>=3 in >> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who >> did not explicitly pin the 'rsa' dependency), and still affects Beam >> development[2]. >> >>>>> >> >>>>> This is the second time[3] Beam is affected with an issue of this >> kind, so support of Python 2 starts to slow down our development, and add >> toil for maintainers of packages we depend on (both directly and >> transitively). >> >>>>> >> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152 >> >>>>> [2] >> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E >> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131 >> >>>>> >> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> >> wrote: >> >>>>>> >> >>>>>> Thank you for re-opening this Valentyn. I am in favor of EOLing >> py2 support sooner than later. The reality is that we will not be >> effectively supporting beam python 2 for a long time while the ecosystem >> already EOLed python 2. That said, a significant chunk (but no longer a >> majority) of our users are still using python 2. Upgrades are painful, it >> might be especially painful nowadays. It would be good to hear counter view >> points, user voices related to this. >> >>>>>> >> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> >>>>>>> >> >>>>>>> Back at the end of February we decided to revisit this >> conversation in 3 months. Do folks on this thread have any new input or >> perspective regarding us balancing "user pain/contributor pain/our ability >> to continuously test with python 2 in a shifting environment"? >> >>>>>>> >> >>>>>>> Some new information on my end is that we have been seeing steady >> adoption of Python 3 among Beam Python users in Dataflow, particularly >> strong adoption among streaming users, and Dataflow is sunsetting Python 2 >> support for all released Beam SDKs later this year [1]. We will have to >> remove Python 2 Beam test suites that use Dataflow when Dataflow runner >> disables Py2 support if this happens before Beam Py2 EOL (when we have to >> remove all Py2 suites), including performance tests that still use Dataflow >> on Python 3. >> >>>>>>> >> >>>>>>> I am curious how much motivation there is in the community at >> this moment to continue Py2 support in Beam, whether any previous Py3 >> migration blockers were resolved or any new blockers discovered among Beam >> users. >> >>>>>>> >> >>>>>>> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow >> >>>>>>> >> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> >>>>>>>> >> >>>>>>>> That's good news! Thanks for sharing. >> >>>>>>>> >> >>>>>>>> Another datapoint, here are a few of Beam's dependencies that no >> longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws, >> gcp, and interactive extras): >> >>>>>>>> >> >>>>>>>> hdfs >> >>>>>>>> numpy >> >>>>>>>> pyarrow >> >>>>>>>> ipython >> >>>>>>>> >> >>>>>>>> There are more if we include transitive dependencies and >> test-only packages. I also remember encountering one issue last month that >> was broken only on Py2, which we had to go back and fix. >> >>>>>>>> >> >>>>>>>> If others have noticed frictions related to ongoing Py2 support >> or have updates on previously mentioned Py3 migration blockers, feel free >> to post them. >> >>>>>>>> >> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw < >> rober...@google.com> wrote: >> >>>>>>>>> >> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a >> milestone that >> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just >> briefly. >> >>>>>>>>> >> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía < >> ieme...@gmail.com> wrote: >> >>>>>>>>> > >> >>>>>>>>> > > I would suggest re-evaluating this within the next 3 months >> again. We need to balance between user pain/contributor pain/our ability to >> continuously test with python 2 in a shifting environment. >> >>>>>>>>> > >> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point also >> distributions will probably be phasing out python2 by default which >> definitely help in this direction. >> >>>>>>>>> > Thanks for updating the roadmap Ahmet >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com> >> wrote: >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía < >> ieme...@gmail.com> wrote: >> >>>>>>>>> >>> >> >>>>>>>>> >>> I am with Chad on this, we should probably extend it a bit >> more, even if it >> >>>>>>>>> >>> makes us struggle a bit at least we have some workarounds >> as Robert suggests, >> >>>>>>>>> >>> and as Chad said there are still many people playing the >> python 3 catchup game, >> >>>>>>>>> >>> so worth to support those users. >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> But maybe it is worth to evaluate the current state later >> in the year. >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> >> I would suggest re-evaluating this within the next 3 months >> again. We need to balance between user pain/contributor pain/our ability to >> continuously test with python 2 in a shifting environment. >> >>>>>>>>> >> >> >>>>>>>>> >>> >> >>>>>>>>> >>> In the >> >>>>>>>>> >>> meantime can someone please update our Roadmap in the >> website with this info and >> >>>>>>>>> >>> where we are with Python 3 support (it looks not up to >> date). >> >>>>>>>>> >>> https://beam.apache.org/roadmap/ >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> >> I made a minor change to update that page ( >> https://github.com/apache/beam/pull/10848). A more comprehensive update >> to that page and linked ( >> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would >> still be welcome. >> >>>>>>>>> >> >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> - Ismaël >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw < >> rober...@google.com> wrote: >> >>>>>>>>> >>>> >> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova < >> chad...@gmail.com> wrote: >> >>>>>>>>> >>>> >> >> >>>>>>>>> >>>> >> Not to mention that all the nice work for the type >> hints will have to be redone in the for 3.x. >> >>>>>>>>> >>>> > >> >>>>>>>>> >>>> > Note that there's a tool for automatically converting >> type comments to annotations: https://github.com/ilevkivskyi/com2ann >> >>>>>>>>> >>>> > >> >>>>>>>>> >>>> > So don't let that part bother you. >> >>>>>>>>> >>>> >> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily automated. >> >>>>>>>>> >>>> >> >>>>>>>>> >>>> > I'm curious what other features you'd like to be using >> in the Beam source that you cannot now. >> >>>>>>>>> >>>> >> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting >> keyword-only >> >>>>>>>>> >>>> arguments the other day. >> >>>>>>>>> >>>> >> >>>>>>>>> >>>> >> It seems the faster we drop support the better. >> >>>>>>>>> >>>> > >> >>>>>>>>> >>>> > >> >>>>>>>>> >>>> > I've already gone over my position on this, but a >> refresher for those who care: some of the key vendors that support my >> industry will not offer python3-compatible versions of their software until >> the 4th quarter of 2020. If Beam switches to python3-only before that >> point we may be forced to stop contributing features (note: I'm the guy who >> added the type hints :). Every month you can give us would be greatly >> appreciated. >> >>>>>>>>> >>>> >> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for >> downloads at >> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain of >> salt, but >> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be way >> higher for >> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, >> but say it >> >>>>>>>>> >>>> doubles every 3 months that would put us at least mid-year >> before we >> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is >> probably a >> >>>>>>>>> >>>> stretch. >> >>>>>>>>> >>>> >> >>>>>>>>> >>>> We could consider whether it needs to be an all-or-nothing >> thing as >> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only >> sooner than >> >>>>>>>>> >>>> the whole codebase. (This would have to be well >> justified.) Another >> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and >> Python 3 in the >> >>>>>>>>> >>>> same pipeline with portability, so if there's a library >> that you need >> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your >> whole >> >>>>>>>>> >>>> pipeline. >> >>>>>>>>> >>>> >> >>>>>>>>> >>>> - Robert >> >>>>>>>>> >>>> >> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that >> 20% may just >> >>>>>>>>> >>>> be a spike. >> >
