Re feedback collection, we already print a message: "You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only." When users run Python 2 pipelines. This might be a good place to provide additional info along with a place to send feedback (probably user@). While I'm sure not everyone out there reads their logs, I imagine this is a sure and easy way of reaching at least some Python 2 users.
Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com On Fri, Sep 20, 2019 at 10:28 AM Valentyn Tymofieiev <valen...@google.com> wrote: > Thank you, Chad, for refreshing this conversation and adding the > perspective of Python 2 users of Beam who have not(yet) completed the > migration. My thoughts below. > > - It is in the best interest of everyone to ensure a smooth migration for > Beam users. However a migration needs to happen since Python ecosystem is > moving off of Python 2. > - Beam has a couple of dozen dependencies, and we cannot have an > expectation that Python 2 versions of these dependencies will be maintained > in 2020. > - BEAM-1251 should be closed, since it may communicate a signal that Beam > does not support Python 3, while it does. Beam has first announced support > of Python 3 in Beam 2.11.0, admittedly later than many mainstream libraries > in Python ecosystem. > - I think Python 2 LTS release (if we continue them) may have critical bug > fixes, but not new features, so we won't be backporting new features. > - Beam portability allows users to customize usercode runtime environment, > and it should be possible for users to supply a Python 2 SDK harness > container, should they have no other option. This would require a > backported user-supplied version of Beam SDK that works on Python 2, > although such SDK may become difficult/impractical to maintain for most > users. > - There are several open issues related to Python 3, but they are > improvements in nature, and we are steadily closing them off. I am not > aware of any adoption blockers for Beam Python 3, specific to Beam. > - I have not heard of users reports who attempted but were not able to use > Beam on Python 3. > - This does not mean that our offering is perfect, there may be errors and > omissions that are yet to be discovered. However, it would be in the best > interest of the Beam community to discover these issues earlier. A message > that Beam will discontinue Python 2 support will encourage users to > migrate, therefore I also support Beam signing > https://python3statement.org. > - Having more usage statistics and feedback closer to 2020 can help us be > more confident in deciding when to stop Python 2 support. > > On Thu, Sep 19, 2019 at 6:05 PM Ahmet Altay <al...@google.com> wrote: > >> Thanks a lot for sharing your thoughts, I completely agree that we need >> to minimize the burden on our users as much as possible. Especially in this >> case when we are offering a robust python 3 solution just now. However I do >> share the same concerns related to dependencies and tool chains, It will be >> increasingly difficult for us to keep our code base compatible with python2 >> and python3 overtime. (To be very explicit, one of those dependencies is >> Dataflow's python pre-portability workers.) >> >> On Thu, Sep 19, 2019 at 5:17 PM Maximilian Michels <m...@apache.org> >> wrote: >> >>> Granted that we just have finalized the Python 3 support, we should >>> allow time for it to mature and for users to make the switch. >>> >>> > Oh, and one more thing, I think it'd make sense for Apache Beam to >>> > sign https://python3statement.org/. The promise is that we'd >>> > discontinue Python 2 support *in* 2020, which is not committing us to >>> > January if we're not ready. Worth a vote? >>> >>> +1 >>> >> >> +1 >> >> >>> >>> On 19.09.19 15:59, Robert Bradshaw wrote: >>> > Oh, and one more thing, I think it'd make sense for Apache Beam to >>> > sign https://python3statement.org/. The promise is that we'd >>> > discontinue Python 2 support *in* 2020, which is not committing us to >>> > January if we're not ready. Worth a vote? >>> > >>> > >>> > On Thu, Sep 19, 2019 at 3:58 PM Robert Bradshaw <rober...@google.com> >>> wrote: >>> >> >>> >> Exactly how long we support Python 2 depends on our users. Other than >>> >> those that speak up (such as yourself, thanks!), it's hard to get a >>> >> handle on how many need Python 2 and for how long. (Should we send out >>> >> a survey? Maybe after some experience with 2.16?) >>> >> >> +1, we had some success with collecting information from users using >> Twitter surveys. >> >> >>> >> >>> >> On the one hand, the whole ecosystem is finally moving on, and even if >>> >> Beam continues to support Python 2 our dependencies, or other projects >>> >> that are being used in conjunction with Beam, will also be going >>> >> Python 3 only. On the other hand, Beam is, admittedly, quite late to >>> >> the party and could be the one holding people back, and looking at how >>> >> long it took us, if we just barely make it by the end of the year it's >>> >> unreasonable to say at that point "oh, and we're dropping 2.7 at the >>> >> same time." >>> >> >>> >> The good news is that 2.16 is shaping up to be a release I would >>> >> recommend everyone migrate to Python 3 on. The remaining issues are >>> >> things like some issues with main sessions (which already has issues >>> >> in Python 2) and not supporting keyword-only arguments (a new feature, >>> >> not a regression). I would guess that even 2.15 is already good enough >>> >> for most people, at least to kick the tires and running tests to start >>> >> the effort. >>> >> >> I share the same sentiment. Beam 2.16 will offer a strong python 3 >> offering. Yes, there are known issues but this is not much different than >> the known issues for rest of the python offering. >> >> >>> >> >>> >> (I also agree with the sentiment that once we go 3.x only, it'll be >>> >> likely harder to maintain a 2.x LTS... but the whole LTS thing is >>> >> being discussed in another thread.) >> >> >> >>> >> On Thu, Sep 19, 2019 at 2:44 PM Chad Dombrova <chad...@gmail.com> >>> wrote: >>> >>> >>> >>> Hi all, >>> >>> I had a read through this thread in the archives. It occurred before >>> I joined the mailing list, so I hope that this email connects up with the >>> thread properly for everyone. >>> >>> >>> >>> I'd like to respond to the following points: >>> >>> >>> >>>> I believe we are referring to two separate things with support: >>> >>>> - Supporting existing releases for patches - I agree that we need >>> to give >>> >>>> users a long enough window to upgrade. Great if it happens with an >>> LTS >>> >>>> release. Even if it does not, I think it will be fair to offer >>> patches on >>> >>>> the last python 2 supporting release during some part of 2020 if >>> that >>> >>>> becomes necessary. >>> >>>> - Making new releases with python 2 support - Each new Beam release >>> with >>> >>>> python 2 support will implicitly extend the lifetime of beam's >>> python 2 >>> >>>> support. I do not think we need to extend this to beyond 2019. 2 >>> releases >>> >>>> (~ 3 months) after solid python 3 support will very likely put the >>> last >>> >>>> python 2 supporting release to last quarter of 2019 already. >>> >>> >>> >>> >>> >>> With so many important features still under active development >>> (portability, expansion, external IO transforms, schema coders) and new >>> versions of executors tied to the Beam source, staying behind is not really >>> an option for many of us, and with python3 support not yet fully completed, >>> the window in which Beam is fully working for both python versions is >>> rapidly approaching 2 months, and could ultimately be even less, depending >>> on how long it takes to complete the dozen remaining issues in Jira, and >>> whatever pops up thereafter. >>> >>> >>> >>>> The cost of maintaining Python 2.7 support is higher than 0. Some >>> issues >>> >>>> that come to mind: >>> >>>> - Maintaining Py2.7 / Py 3+ compatibility of Beam codebase makes it >>> >>>> difficult to use Python 3 syntax in Beam which may be necessary to >>> support >>> >>>> and test syntactic constructs introduced in Python 3. >>> >>>> - Running additional test suites increases the load on test >>> infrastructure >>> >>>> and increases flakiness. >>> >>> >>> >>> >>> >>> I would argue that the cost of maintaining a python2-only LTS >>> version will be far greater than maintaining python2 support for a little >>> while longer. Dropping support for python2 could mean a number of things >>> from simply disabling the python2 tests, to removing 2-to-3 idioms in favor >>> of python3-only constructs. If what you have in mind is anything like the >>> latter then the master branch will become quite divergent from the LTS >>> release, and backporting changes will be not be as simple as cherry-picking >>> commits. All-in-all, I think it's a lose/lose for everyone -- users and >>> developers, of which I am both -- to drop python2 support on such a short >>> timeline. >>> >>> >>> >>> I'm an active contributor to this project and it will put me and the >>> company that I work for in a very bad position if you force us onto an LTS >>> release in early 2020. I understand the appeal of moving to python3-only >>> code and I want to get there too, but I would hope that you give your users >>> are much time to transition their own code as the Beam project itself has >>> taken. I'm not asking for a full 12 months to transition, but more than a >>> couple will be required. >>> >> >> What would be the ideal time frame for you? >> >> >>> >>> >>> >>> thanks, >>> >>> -chad >>> >>> >>> >>> >>> >>> >>> >>> >>> >>
