As suggested and enthusiastically supported by several folks in this thread, I will send a vote to sign a pledge on http://python3statement.org on behalf of Apache Beam to discontinue Python 2 support in or before 2020.
The motivation for signing the pledge is: - to provide another signal to Beam users, and projects that depend on Beam that Beam Python 2 offering will soon sunset; - to facilitate adoption of Python 3 by Beam users, developers, and runner maintainers; - to facilitate adoption of Python 3 in wider Python ecosystem. See also http://python3stament.org for background behind this pledge and the list of projects which have already signed it. On Mon, Sep 23, 2019 at 4:45 PM Kyle Weaver <kcwea...@google.com> wrote: > Re feedback collection, we already print a message: > "You are using Apache Beam with Python 2. New releases of Apache Beam will > soon support Python 3 only." > When users run Python 2 pipelines. This might be a good place to provide > additional info along with a place to send feedback (probably user@). > While I'm sure not everyone out there reads their logs, I imagine this is a > sure and easy way of reaching at least some Python 2 users. > > Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com > > > On Fri, Sep 20, 2019 at 10:28 AM Valentyn Tymofieiev <valen...@google.com> > wrote: > >> Thank you, Chad, for refreshing this conversation and adding the >> perspective of Python 2 users of Beam who have not(yet) completed the >> migration. My thoughts below. >> >> - It is in the best interest of everyone to ensure a smooth migration for >> Beam users. However a migration needs to happen since Python ecosystem is >> moving off of Python 2. >> - Beam has a couple of dozen dependencies, and we cannot have an >> expectation that Python 2 versions of these dependencies will be maintained >> in 2020. >> - BEAM-1251 should be closed, since it may communicate a signal that Beam >> does not support Python 3, while it does. Beam has first announced support >> of Python 3 in Beam 2.11.0, admittedly later than many mainstream libraries >> in Python ecosystem. >> - I think Python 2 LTS release (if we continue them) may have critical >> bug fixes, but not new features, so we won't be backporting new features. >> - Beam portability allows users to customize usercode runtime >> environment, and it should be possible for users to supply a Python 2 SDK >> harness container, should they have no other option. This would require a >> backported user-supplied version of Beam SDK that works on Python 2, >> although such SDK may become difficult/impractical to maintain for most >> users. >> - There are several open issues related to Python 3, but they are >> improvements in nature, and we are steadily closing them off. I am not >> aware of any adoption blockers for Beam Python 3, specific to Beam. >> - I have not heard of users reports who attempted but were not able to >> use Beam on Python 3. >> - This does not mean that our offering is perfect, there may be errors >> and omissions that are yet to be discovered. However, it would be in the >> best interest of the Beam community to discover these issues earlier. A >> message that Beam will discontinue Python 2 support will encourage users to >> migrate, therefore I also support Beam signing >> https://python3statement.org. >> - Having more usage statistics and feedback closer to 2020 can help us be >> more confident in deciding when to stop Python 2 support. >> >> On Thu, Sep 19, 2019 at 6:05 PM Ahmet Altay <al...@google.com> wrote: >> >>> Thanks a lot for sharing your thoughts, I completely agree that we need >>> to minimize the burden on our users as much as possible. Especially in this >>> case when we are offering a robust python 3 solution just now. However I do >>> share the same concerns related to dependencies and tool chains, It will be >>> increasingly difficult for us to keep our code base compatible with python2 >>> and python3 overtime. (To be very explicit, one of those dependencies is >>> Dataflow's python pre-portability workers.) >>> >>> On Thu, Sep 19, 2019 at 5:17 PM Maximilian Michels <m...@apache.org> >>> wrote: >>> >>>> Granted that we just have finalized the Python 3 support, we should >>>> allow time for it to mature and for users to make the switch. >>>> >>>> > Oh, and one more thing, I think it'd make sense for Apache Beam to >>>> > sign https://python3statement.org/. The promise is that we'd >>>> > discontinue Python 2 support *in* 2020, which is not committing us to >>>> > January if we're not ready. Worth a vote? >>>> >>>> +1 >>>> >>> >>> +1 >>> >>> >>>> >>>> On 19.09.19 15:59, Robert Bradshaw wrote: >>>> > Oh, and one more thing, I think it'd make sense for Apache Beam to >>>> > sign https://python3statement.org/. The promise is that we'd >>>> > discontinue Python 2 support *in* 2020, which is not committing us to >>>> > January if we're not ready. Worth a vote? >>>> > >>>> > >>>> > On Thu, Sep 19, 2019 at 3:58 PM Robert Bradshaw <rober...@google.com> >>>> wrote: >>>> >> >>>> >> Exactly how long we support Python 2 depends on our users. Other than >>>> >> those that speak up (such as yourself, thanks!), it's hard to get a >>>> >> handle on how many need Python 2 and for how long. (Should we send >>>> out >>>> >> a survey? Maybe after some experience with 2.16?) >>>> >>> >>> +1, we had some success with collecting information from users using >>> Twitter surveys. >>> >>> >>>> >> >>>> >> On the one hand, the whole ecosystem is finally moving on, and even >>>> if >>>> >> Beam continues to support Python 2 our dependencies, or other >>>> projects >>>> >> that are being used in conjunction with Beam, will also be going >>>> >> Python 3 only. On the other hand, Beam is, admittedly, quite late to >>>> >> the party and could be the one holding people back, and looking at >>>> how >>>> >> long it took us, if we just barely make it by the end of the year >>>> it's >>>> >> unreasonable to say at that point "oh, and we're dropping 2.7 at the >>>> >> same time." >>>> >> >>>> >> The good news is that 2.16 is shaping up to be a release I would >>>> >> recommend everyone migrate to Python 3 on. The remaining issues are >>>> >> things like some issues with main sessions (which already has issues >>>> >> in Python 2) and not supporting keyword-only arguments (a new >>>> feature, >>>> >> not a regression). I would guess that even 2.15 is already good >>>> enough >>>> >> for most people, at least to kick the tires and running tests to >>>> start >>>> >> the effort. >>>> >>> >>> I share the same sentiment. Beam 2.16 will offer a strong python 3 >>> offering. Yes, there are known issues but this is not much different than >>> the known issues for rest of the python offering. >>> >>> >>>> >> >>>> >> (I also agree with the sentiment that once we go 3.x only, it'll be >>>> >> likely harder to maintain a 2.x LTS... but the whole LTS thing is >>>> >> being discussed in another thread.) >>> >>> >> >>>> >> On Thu, Sep 19, 2019 at 2:44 PM Chad Dombrova <chad...@gmail.com> >>>> wrote: >>>> >>> >>>> >>> Hi all, >>>> >>> I had a read through this thread in the archives. It occurred >>>> before I joined the mailing list, so I hope that this email connects up >>>> with the thread properly for everyone. >>>> >>> >>>> >>> I'd like to respond to the following points: >>>> >>> >>>> >>>> I believe we are referring to two separate things with support: >>>> >>>> - Supporting existing releases for patches - I agree that we need >>>> to give >>>> >>>> users a long enough window to upgrade. Great if it happens with an >>>> LTS >>>> >>>> release. Even if it does not, I think it will be fair to offer >>>> patches on >>>> >>>> the last python 2 supporting release during some part of 2020 if >>>> that >>>> >>>> becomes necessary. >>>> >>>> - Making new releases with python 2 support - Each new Beam >>>> release with >>>> >>>> python 2 support will implicitly extend the lifetime of beam's >>>> python 2 >>>> >>>> support. I do not think we need to extend this to beyond 2019. 2 >>>> releases >>>> >>>> (~ 3 months) after solid python 3 support will very likely put the >>>> last >>>> >>>> python 2 supporting release to last quarter of 2019 already. >>>> >>> >>>> >>> >>>> >>> With so many important features still under active development >>>> (portability, expansion, external IO transforms, schema coders) and new >>>> versions of executors tied to the Beam source, staying behind is not really >>>> an option for many of us, and with python3 support not yet fully completed, >>>> the window in which Beam is fully working for both python versions is >>>> rapidly approaching 2 months, and could ultimately be even less, depending >>>> on how long it takes to complete the dozen remaining issues in Jira, and >>>> whatever pops up thereafter. >>>> >>> >>>> >>>> The cost of maintaining Python 2.7 support is higher than 0. Some >>>> issues >>>> >>>> that come to mind: >>>> >>>> - Maintaining Py2.7 / Py 3+ compatibility of Beam codebase makes it >>>> >>>> difficult to use Python 3 syntax in Beam which may be necessary to >>>> support >>>> >>>> and test syntactic constructs introduced in Python 3. >>>> >>>> - Running additional test suites increases the load on test >>>> infrastructure >>>> >>>> and increases flakiness. >>>> >>> >>>> >>> >>>> >>> I would argue that the cost of maintaining a python2-only LTS >>>> version will be far greater than maintaining python2 support for a little >>>> while longer. Dropping support for python2 could mean a number of things >>>> from simply disabling the python2 tests, to removing 2-to-3 idioms in favor >>>> of python3-only constructs. If what you have in mind is anything like the >>>> latter then the master branch will become quite divergent from the LTS >>>> release, and backporting changes will be not be as simple as cherry-picking >>>> commits. All-in-all, I think it's a lose/lose for everyone -- users and >>>> developers, of which I am both -- to drop python2 support on such a short >>>> timeline. >>>> >>> >>>> >>> I'm an active contributor to this project and it will put me and >>>> the company that I work for in a very bad position if you force us onto an >>>> LTS release in early 2020. I understand the appeal of moving to >>>> python3-only code and I want to get there too, but I would hope that you >>>> give your users are much time to transition their own code as the Beam >>>> project itself has taken. I'm not asking for a full 12 months to >>>> transition, but more than a couple will be required. >>>> >>> >>> What would be the ideal time frame for you? >>> >>> >>>> >>> >>>> >>> thanks, >>>> >>> -chad >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>>
