I have reviewed Slack, Twitter and User@ channels [1-3] we used to communicate our consideration to make 2.23.0 a final release supporting Py2.
I did not see any objections or negative feedback there. I suggest we hold a VOTE to decide whether 2.24.0 should be the final release supporting Py2. I can send the voting email. [1] https://app.slack.com/client/T4S1WH2J3/CBDNLQZM1/thread/C9H0YNP3P-1592524218.050700 [2] https://twitter.com/ApacheBeam/status/1273760375570194432 [3] https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E On Thu, Jul 23, 2020 at 10:57 AM Robert Bradshaw <rober...@google.com> wrote: > The release cut date for 2.24 is in a couple of weeks; if this is the last > release supporting 2.7 we should make the call and announce it soon. > > On Mon, Jul 6, 2020 at 2:26 PM Robert Bradshaw <rober...@google.com> > wrote: > >> Note that just because Beam drops 2.x support in new releases doesn't >> mean that the old releases won't continue to work. One can even use an >> expansion service to run 2.x transforms (on an older version of Beam) >> within a Python 3 pipeline running the newest version of Beam until >> such a time that (possibly incrementally) the dependent libraries >> catch up, if it comes to that. Not that this will be ideal. >> >> Now that the 2.23 release branch has been cut, we *could* remove 2.7 >> support if we will actually plan to remove it in 2.24. (One concern, >> however, is how tied the release testing infrastructure is to >> mainline...) However, other than Chad's response (the VFX / Animation >> isn't quite there yet) it seems we still don't have a very strong >> signal either direction. Pypi downloads are still hovering around 50%. >> Did we hear anything back from twitter/slack? >> >> My inclination would be to publish loudly that 2.24 would be the last >> Beam to support Python 2.7 (this is pushing it back one release), so >> if you're still stuck on it make sure it has everything you need by >> the release cut date (which is still in the future). >> >> >> On Mon, Jul 6, 2020 at 9:52 AM David Yan <david...@google.com> wrote: >> > >> > +1 for removing Python 2.7 support sooner than later. >> > >> > I recently added a small feature in Beam Python and I found that having >> to write code that worked with Python2 was quite awkward and time consuming >> (needing to make sure code works for both 2 and 3 and doubling the Jenkins >> running time), and I can imagine that may hinder or even discourage new >> contributions. >> > >> > On Thu, Jun 18, 2020 at 5:12 PM Ahmet Altay <al...@google.com> wrote: >> >> >> >> Good to hear from you and good to hear the news. Release branch cut >> date for 2.24 is 8/12. >> >> >> >> On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova <chad...@gmail.com> >> wrote: >> >>> >> >>> Hi all, >> >>> Sorry I've been AWOL. I've been pulled in a number of different >> directions. >> >>> >> >>> We are still increasing our use of Beam, and I have it on my todo >> list to reach out to Kenneth about how Beam could be expanded into the VFX >> / Animation industries. >> >>> >> >>> Our industry uses a number of specialized applications with embedded >> python interpreters. We run Beam inside these interpreters, so we're >> waiting for them to switch to python3. >> >>> >> >>> Here's the status report for python3 adoption in our key applications: >> >>> >> >>> Maya: In Beta >> >>> Houdini: Released >> >>> Nuke: In Beta >> >>> Katana: Not started (Alpha?) >> >>> >> >>> I hate to be the one holding the project back, and I understand if >> you all ultimately decide it's untenable to wait any longer. The good news >> is 3 out of 4 applications should be ready in the next 2-3 months. I can >> do some investigation into what workarounds might look like for Katana, or >> maybe we can use the Beta version once python3 support arrives, which would >> move our schedule forward. >> >>> >> >>> When would 2.24 release? >> >>> >> >>> -chad >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote: >> >>>> >> >>>> OK, tweeted the message. Could you share on Slack? >> >>>> >> >>>> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> >>>>> >> >>>>> Alright, let's publish the message on Twitter and echo on Slack >> once that's done. >> >>>>> Thank you! >> >>>>> >> >>>>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> >> wrote: >> >>>>>> >> >>>>>> That sounds reasonable to me. I agree, it is better to get >> specific reasons rather than a yes/no answer. >> >>>>>> >> >>>>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> >>>>>>> >> >>>>>>> After thinking about it for a bit, I am not sure whether a poll >> would be actionable. For example, if 1000 users provide a positive response >> and 5 users provide a negative response, how do we interpret that and >> where do we draw a line? How about instead we encourage users to reach out, >> and tell what is not working for them? For example: >> >>>>>>> >> >>>>>>> Beam is considering making 2.23.0 a final release supporting Py2. >> If you are not able to switch to Python 3, please let us know why: [short >> link to user@ thread] [1]. >> >>>>>>> >> >>>>>>> Thoughts? >> >>>>>>> >> >>>>>>> [1] >> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E >> >>>>>>> >> >>>>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> >> wrote: >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> >>>>>>>>> >> >>>>>>>>> I have reached out to user@ for feedback on Python 3 >> migration[1]. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> Maybe also ask on slack? There are quite a bit of users there as >> well. >> >>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> Could somebody from PMC please help with Twitter poll? >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> I can try to do this. What question would you like to ask? I do >> not know much about twitter polls but I assume they have character limits >> similar to regular tweets. >> >>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> Technically, we can proceed with the change between 2.23.0 and >> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users >> to respond. >> >>>>>>>>> >> >>>>>>>>> [1] >> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E >> >>>>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ieme...@gmail.com> >> wrote: >> >>>>>>>>>> >> >>>>>>>>>> Yes we need to poll this outside as Robert proposed. >> >>>>>>>>>> >> >>>>>>>>>> The proposed last support version corresponds with the next >> release that will be >> >>>>>>>>>> cut in two weeks. Sounds a bit short if we count the time to >> poll people on this >> >>>>>>>>>> subject but still could be. >> >>>>>>>>> >> >>>>>>>>> Technically, we can proceed with the change between 2.23.0 and >> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users >> to respond. >> >>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> I remember Chad mentioned in this thread the impossibility to >> get support for >> >>>>>>>>>> python 2 in his industry until the end of the year, Maybe >> things have improved. >> >>>>>>>>>> Have they? >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw < >> rober...@google.com> wrote: >> >>>>>>>>>> > >> >>>>>>>>>> > I like that option as a concrete proposal. I think we should >> circulate it more widely (the users list, twitter poll, at least a new >> thread...), maybe phrasing it as "is there any reason you couldn't migrate >> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be >> cut here in a couple of weeks)?" If there is strong concern/pushback, we >> could consider holding on for one more release. >> >>>>>>>>>> > >> >>>>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos < >> dcava...@google.com> wrote: >> >>>>>>>>>> >> >> >>>>>>>>>> >> +1 >> >>>>>>>>>> >> >> >>>>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> >> wrote: >> >>>>>>>>>> >>> >> >>>>>>>>>> >>> +1 >> >>>>>>>>>> >>> >> >>>>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay < >> al...@google.com> wrote: >> >>>>>>>>>> >>>> >> >>>>>>>>>> >>>> As a concrete proposal, could we commit to removing >> python 2 support by 2.24? In other words, mark the next release 2.23 as the >> last python 2 compatible Beam version. >> >>>>>>>>>> >>>> >> >>>>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> >>>>>>>>>> >>>>> >> >>>>>>>>>> >>>>> Another input here: >> >>>>>>>>>> >>>>> >> >>>>>>>>>> >>>>> If you opened a Python PR in the last few days, you >> probably noticed that our test suites were broken by a transitive >> dependency of Beam that dropped python 2 support, but did not declare >> python_requires>=3 in its setup.py [1]. This temporarily broke a subset of >> Beam Py2 users (who did not explicitly pin the 'rsa' dependency), and still >> affects Beam development[2]. >> >>>>>>>>>> >>>>> >> >>>>>>>>>> >>>>> This is the second time[3] Beam is affected with an >> issue of this kind, so support of Python 2 starts to slow down our >> development, and add toil for maintainers of packages we depend on (both >> directly and transitively). >> >>>>>>>>>> >>>>> >> >>>>>>>>>> >>>>> [1] >> https://github.com/sybrenstuvel/python-rsa/issues/152 >> >>>>>>>>>> >>>>> [2] >> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E >> >>>>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131 >> >>>>>>>>>> >>>>> >> >>>>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay < >> al...@google.com> wrote: >> >>>>>>>>>> >>>>>> >> >>>>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor >> of EOLing py2 support sooner than later. The reality is that we will not be >> effectively supporting beam python 2 for a long time while the ecosystem >> already EOLed python 2. That said, a significant chunk (but no longer a >> majority) of our users are still using python 2. Upgrades are painful, it >> might be especially painful nowadays. It would be good to hear counter view >> points, user voices related to this. >> >>>>>>>>>> >>>>>> >> >>>>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> >>>>>>>>>> >>>>>>> >> >>>>>>>>>> >>>>>>> Back at the end of February we decided to revisit this >> conversation in 3 months. Do folks on this thread have any new input or >> perspective regarding us balancing "user pain/contributor pain/our ability >> to continuously test with python 2 in a shifting environment"? >> >>>>>>>>>> >>>>>>> >> >>>>>>>>>> >>>>>>> Some new information on my end is that we have been >> seeing steady adoption of Python 3 among Beam Python users in Dataflow, >> particularly strong adoption among streaming users, and Dataflow is >> sunsetting Python 2 support for all released Beam SDKs later this year [1]. >> We will have to remove Python 2 Beam test suites that use Dataflow when >> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL >> (when we have to remove all Py2 suites), including performance tests that >> still use Dataflow on Python 3. >> >>>>>>>>>> >>>>>>> >> >>>>>>>>>> >>>>>>> I am curious how much motivation there is in the >> community at this moment to continue Py2 support in Beam, whether any >> previous Py3 migration blockers were resolved or any new blockers >> discovered among Beam users. >> >>>>>>>>>> >>>>>>> >> >>>>>>>>>> >>>>>>> [1] >> https://cloud.google.com/python/docs/python2-sunset/#dataflow >> >>>>>>>>>> >>>>>>> >> >>>>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> >>>>>>>>>> >>>>>>>> >> >>>>>>>>>> >>>>>>>> That's good news! Thanks for sharing. >> >>>>>>>>>> >>>>>>>> >> >>>>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's >> dependencies that no longer release new py2 artifacts (I looked at >> REQUIRED_PACKAGES + aws, gcp, and interactive extras): >> >>>>>>>>>> >>>>>>>> >> >>>>>>>>>> >>>>>>>> hdfs >> >>>>>>>>>> >>>>>>>> numpy >> >>>>>>>>>> >>>>>>>> pyarrow >> >>>>>>>>>> >>>>>>>> ipython >> >>>>>>>>>> >>>>>>>> >> >>>>>>>>>> >>>>>>>> There are more if we include transitive dependencies >> and test-only packages. I also remember encountering one issue last month >> that was broken only on Py2, which we had to go back and fix. >> >>>>>>>>>> >>>>>>>> >> >>>>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing >> Py2 support or have updates on previously mentioned Py3 migration blockers, >> feel free to post them. >> >>>>>>>>>> >>>>>>>> >> >>>>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw < >> rober...@google.com> wrote: >> >>>>>>>>>> >>>>>>>>> >> >>>>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call >> out a milestone that >> >>>>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on >> pypi, if just briefly. >> >>>>>>>>>> >>>>>>>>> >> >>>>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía < >> ieme...@gmail.com> wrote: >> >>>>>>>>>> >>>>>>>>> > >> >>>>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the >> next 3 months again. We need to balance between user pain/contributor >> pain/our ability to continuously test with python 2 in a shifting >> environment. >> >>>>>>>>>> >>>>>>>>> > >> >>>>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that >> point also distributions will probably be phasing out python2 by default >> which definitely help in this direction. >> >>>>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet >> >>>>>>>>>> >>>>>>>>> > >> >>>>>>>>>> >>>>>>>>> > >> >>>>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay < >> al...@google.com> wrote: >> >>>>>>>>>> >>>>>>>>> >> >> >>>>>>>>>> >>>>>>>>> >> >> >>>>>>>>>> >>>>>>>>> >> >> >>>>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía < >> ieme...@gmail.com> wrote: >> >>>>>>>>>> >>>>>>>>> >>> >> >>>>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably >> extend it a bit more, even if it >> >>>>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some >> workarounds as Robert suggests, >> >>>>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people >> playing the python 3 catchup game, >> >>>>>>>>>> >>>>>>>>> >>> so worth to support those users. >> >>>>>>>>>> >>>>>>>>> >>> >> >>>>>>>>>> >>>>>>>>> >>> >> >>>>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current >> state later in the year. >> >>>>>>>>>> >>>>>>>>> >> >> >>>>>>>>>> >>>>>>>>> >> >> >>>>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the >> next 3 months again. We need to balance between user pain/contributor >> pain/our ability to continuously test with python 2 in a shifting >> environment. >> >>>>>>>>>> >>>>>>>>> >> >> >>>>>>>>>> >>>>>>>>> >>> >> >>>>>>>>>> >>>>>>>>> >>> In the >> >>>>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap >> in the website with this info and >> >>>>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not >> up to date). >> >>>>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/ >> >>>>>>>>>> >>>>>>>>> >> >> >>>>>>>>>> >>>>>>>>> >> >> >>>>>>>>>> >>>>>>>>> >> I made a minor change to update that page ( >> https://github.com/apache/beam/pull/10848). A more comprehensive update >> to that page and linked ( >> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would >> still be welcome. >> >>>>>>>>>> >>>>>>>>> >> >> >>>>>>>>>> >>>>>>>>> >>> >> >>>>>>>>>> >>>>>>>>> >>> >> >>>>>>>>>> >>>>>>>>> >>> - Ismaël >> >>>>>>>>>> >>>>>>>>> >>> >> >>>>>>>>>> >>>>>>>>> >>> >> >>>>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw < >> rober...@google.com> wrote: >> >>>>>>>>>> >>>>>>>>> >>>> >> >>>>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova < >> chad...@gmail.com> wrote: >> >>>>>>>>>> >>>>>>>>> >>>> >> >> >>>>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for >> the type hints will have to be redone in the for 3.x. >> >>>>>>>>>> >>>>>>>>> >>>> > >> >>>>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically >> converting type comments to annotations: >> https://github.com/ilevkivskyi/com2ann >> >>>>>>>>>> >>>>>>>>> >>>> > >> >>>>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you. >> >>>>>>>>>> >>>>>>>>> >>>> >> >>>>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily >> automated. >> >>>>>>>>>> >>>>>>>>> >>>> >> >>>>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to >> be using in the Beam source that you cannot now. >> >>>>>>>>>> >>>>>>>>> >>>> >> >>>>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into >> wanting keyword-only >> >>>>>>>>>> >>>>>>>>> >>>> arguments the other day. >> >>>>>>>>>> >>>>>>>>> >>>> >> >>>>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the >> better. >> >>>>>>>>>> >>>>>>>>> >>>> > >> >>>>>>>>>> >>>>>>>>> >>>> > >> >>>>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, >> but a refresher for those who care: some of the key vendors that support >> my industry will not offer python3-compatible versions of their software >> until the 4th quarter of 2020. If Beam switches to python3-only before >> that point we may be forced to stop contributing features (note: I'm the >> guy who added the type hints :). Every month you can give us would be >> greatly appreciated. >> >>>>>>>>>> >>>>>>>>> >>>> >> >>>>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on >> Py2/Py3 for downloads at >> >>>>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with >> a grain of salt, but >> >>>>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio >> needs to be way higher for >> >>>>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's >> pretty noisy, but say it >> >>>>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at >> least mid-year before we >> >>>>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 >> 2020 is probably a >> >>>>>>>>>> >>>>>>>>> >>>> stretch. >> >>>>>>>>>> >>>>>>>>> >>>> >> >>>>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an >> all-or-nothing thing as >> >>>>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be >> Python 3 only sooner than >> >>>>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well >> justified.) Another >> >>>>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python >> 2 and Python 3 in the >> >>>>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a >> library that you need >> >>>>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold >> back your whole >> >>>>>>>>>> >>>>>>>>> >>>> pipeline. >> >>>>>>>>>> >>>>>>>>> >>>> >> >>>>>>>>>> >>>>>>>>> >>>> - Robert >> >>>>>>>>>> >>>>>>>>> >>>> >> >>>>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam >> , and that 20% may just >> >>>>>>>>>> >>>>>>>>> >>>> be a spike. >> >
