Thank you, Chad, for refreshing this conversation and adding the perspective of Python 2 users of Beam who have not(yet) completed the migration. My thoughts below.
- It is in the best interest of everyone to ensure a smooth migration for Beam users. However a migration needs to happen since Python ecosystem is moving off of Python 2. - Beam has a couple of dozen dependencies, and we cannot have an expectation that Python 2 versions of these dependencies will be maintained in 2020. - BEAM-1251 should be closed, since it may communicate a signal that Beam does not support Python 3, while it does. Beam has first announced support of Python 3 in Beam 2.11.0, admittedly later than many mainstream libraries in Python ecosystem. - I think Python 2 LTS release (if we continue them) may have critical bug fixes, but not new features, so we won't be backporting new features. - Beam portability allows users to customize usercode runtime environment, and it should be possible for users to supply a Python 2 SDK harness container, should they have no other option. This would require a backported user-supplied version of Beam SDK that works on Python 2, although such SDK may become difficult/impractical to maintain for most users. - There are several open issues related to Python 3, but they are improvements in nature, and we are steadily closing them off. I am not aware of any adoption blockers for Beam Python 3, specific to Beam. - I have not heard of users reports who attempted but were not able to use Beam on Python 3. - This does not mean that our offering is perfect, there may be errors and omissions that are yet to be discovered. However, it would be in the best interest of the Beam community to discover these issues earlier. A message that Beam will discontinue Python 2 support will encourage users to migrate, therefore I also support Beam signing https://python3statement.org. - Having more usage statistics and feedback closer to 2020 can help us be more confident in deciding when to stop Python 2 support. On Thu, Sep 19, 2019 at 6:05 PM Ahmet Altay <al...@google.com> wrote: > Thanks a lot for sharing your thoughts, I completely agree that we need to > minimize the burden on our users as much as possible. Especially in this > case when we are offering a robust python 3 solution just now. However I do > share the same concerns related to dependencies and tool chains, It will be > increasingly difficult for us to keep our code base compatible with python2 > and python3 overtime. (To be very explicit, one of those dependencies is > Dataflow's python pre-portability workers.) > > On Thu, Sep 19, 2019 at 5:17 PM Maximilian Michels <m...@apache.org> wrote: > >> Granted that we just have finalized the Python 3 support, we should >> allow time for it to mature and for users to make the switch. >> >> > Oh, and one more thing, I think it'd make sense for Apache Beam to >> > sign https://python3statement.org/. The promise is that we'd >> > discontinue Python 2 support *in* 2020, which is not committing us to >> > January if we're not ready. Worth a vote? >> >> +1 >> > > +1 > > >> >> On 19.09.19 15:59, Robert Bradshaw wrote: >> > Oh, and one more thing, I think it'd make sense for Apache Beam to >> > sign https://python3statement.org/. The promise is that we'd >> > discontinue Python 2 support *in* 2020, which is not committing us to >> > January if we're not ready. Worth a vote? >> > >> > >> > On Thu, Sep 19, 2019 at 3:58 PM Robert Bradshaw <rober...@google.com> >> wrote: >> >> >> >> Exactly how long we support Python 2 depends on our users. Other than >> >> those that speak up (such as yourself, thanks!), it's hard to get a >> >> handle on how many need Python 2 and for how long. (Should we send out >> >> a survey? Maybe after some experience with 2.16?) >> > > +1, we had some success with collecting information from users using > Twitter surveys. > > >> >> >> >> On the one hand, the whole ecosystem is finally moving on, and even if >> >> Beam continues to support Python 2 our dependencies, or other projects >> >> that are being used in conjunction with Beam, will also be going >> >> Python 3 only. On the other hand, Beam is, admittedly, quite late to >> >> the party and could be the one holding people back, and looking at how >> >> long it took us, if we just barely make it by the end of the year it's >> >> unreasonable to say at that point "oh, and we're dropping 2.7 at the >> >> same time." >> >> >> >> The good news is that 2.16 is shaping up to be a release I would >> >> recommend everyone migrate to Python 3 on. The remaining issues are >> >> things like some issues with main sessions (which already has issues >> >> in Python 2) and not supporting keyword-only arguments (a new feature, >> >> not a regression). I would guess that even 2.15 is already good enough >> >> for most people, at least to kick the tires and running tests to start >> >> the effort. >> > > I share the same sentiment. Beam 2.16 will offer a strong python 3 > offering. Yes, there are known issues but this is not much different than > the known issues for rest of the python offering. > > >> >> >> >> (I also agree with the sentiment that once we go 3.x only, it'll be >> >> likely harder to maintain a 2.x LTS... but the whole LTS thing is >> >> being discussed in another thread.) > > >> >> >> On Thu, Sep 19, 2019 at 2:44 PM Chad Dombrova <chad...@gmail.com> >> wrote: >> >>> >> >>> Hi all, >> >>> I had a read through this thread in the archives. It occurred before >> I joined the mailing list, so I hope that this email connects up with the >> thread properly for everyone. >> >>> >> >>> I'd like to respond to the following points: >> >>> >> >>>> I believe we are referring to two separate things with support: >> >>>> - Supporting existing releases for patches - I agree that we need to >> give >> >>>> users a long enough window to upgrade. Great if it happens with an >> LTS >> >>>> release. Even if it does not, I think it will be fair to offer >> patches on >> >>>> the last python 2 supporting release during some part of 2020 if that >> >>>> becomes necessary. >> >>>> - Making new releases with python 2 support - Each new Beam release >> with >> >>>> python 2 support will implicitly extend the lifetime of beam's >> python 2 >> >>>> support. I do not think we need to extend this to beyond 2019. 2 >> releases >> >>>> (~ 3 months) after solid python 3 support will very likely put the >> last >> >>>> python 2 supporting release to last quarter of 2019 already. >> >>> >> >>> >> >>> With so many important features still under active development >> (portability, expansion, external IO transforms, schema coders) and new >> versions of executors tied to the Beam source, staying behind is not really >> an option for many of us, and with python3 support not yet fully completed, >> the window in which Beam is fully working for both python versions is >> rapidly approaching 2 months, and could ultimately be even less, depending >> on how long it takes to complete the dozen remaining issues in Jira, and >> whatever pops up thereafter. >> >>> >> >>>> The cost of maintaining Python 2.7 support is higher than 0. Some >> issues >> >>>> that come to mind: >> >>>> - Maintaining Py2.7 / Py 3+ compatibility of Beam codebase makes it >> >>>> difficult to use Python 3 syntax in Beam which may be necessary to >> support >> >>>> and test syntactic constructs introduced in Python 3. >> >>>> - Running additional test suites increases the load on test >> infrastructure >> >>>> and increases flakiness. >> >>> >> >>> >> >>> I would argue that the cost of maintaining a python2-only LTS version >> will be far greater than maintaining python2 support for a little while >> longer. Dropping support for python2 could mean a number of things from >> simply disabling the python2 tests, to removing 2-to-3 idioms in favor of >> python3-only constructs. If what you have in mind is anything like the >> latter then the master branch will become quite divergent from the LTS >> release, and backporting changes will be not be as simple as cherry-picking >> commits. All-in-all, I think it's a lose/lose for everyone -- users and >> developers, of which I am both -- to drop python2 support on such a short >> timeline. >> >>> >> >>> I'm an active contributor to this project and it will put me and the >> company that I work for in a very bad position if you force us onto an LTS >> release in early 2020. I understand the appeal of moving to python3-only >> code and I want to get there too, but I would hope that you give your users >> are much time to transition their own code as the Beam project itself has >> taken. I'm not asking for a full 12 months to transition, but more than a >> couple will be required. >> > > What would be the ideal time frame for you? > > >> >>> >> >>> thanks, >> >>> -chad >> >>> >> >>> >> >>> >> >>> >> >
