On Tue, Jun 18, 2019 at 2:01 PM Ahmet Altay <al...@google.com> wrote:
> Thank you for the update, very helpful. It might be worthwhile to share a > version of this with user mailing list after 2.14. > I think so too, we can send an update to user list when 2.14.0 is released. > Remaining question for me is: There is no plan for an LTS release > currently. Would it make sense for us to target one after known remaining > issues are mostly fixed. What release would that be? > >From Python 3 standpoint, 2.16.0 could be a good target for an LTS release, that would give us two more releases in 2019 where we can mark Python 2 support as deprecated and remove it with the first release in 2020. > > On Tue, Jun 18, 2019 at 12:08 AM Valentyn Tymofieiev <valen...@google.com> > wrote: > >> To give a better understanding where we are w.r.t. Python 3, I'd like to >> give a quick overview of the recent work that has been happening in Beam >> community to support Python 3, and to summarize the current status of this >> effort. >> >> Current status: >> >> 1. >> >> Beam 2.11.0 was the first release that offered Python 3 support, >> specifically Python 3.5 support. Due to several limitations that have been >> fixed since 2.11.0, Beam 2.13.0 (or newer version) is recommended for >> Python 3 pipelines. >> 2. >> >> Pipelines running on Portable Flink / Spark runners may have to use >> Beam 2.14.0 once it becomes available. >> 3. >> >> Python 3.5 or newer version of the interpreter is required to install >> Beam and run Python 3 pipelines. >> >> >> Known remaining limitations of current Python 3 offering: >> >> >> 1. >> >> Several syntactic constructs introduced in Python 3 (keyword-only >> arguments, dataclasses), are not yet supported. See: BEAM-5878, BEAM-7284. >> 2. >> >> Pickling errors occasionally prevent usage of --save_main_session >> flag, but changes to the pipeline code may help to overcome this >> limitation. See: BEAM-6158, BEAM-7540 >> 3. >> >> Beam has limited type inference capabilities support in Python 3.6+, >> and type checking of Beam typehints is not always enforced, see: >> BEAM-2713, >> BEAM-7377. >> >> >> The cause of limitations 1-2 largely lies in Beam dependency 'dill' that >> supports pickling. In the immediate future we will be working on evaluating >> replacements or/and fixes to address this. We are also working on an >> improved typehints support in Python 3, see: BEAM-2713. >> >> The efforts to make Beam codebase Python3-compatible started back in >> 2017. Most of this work is visible in BEAM-1251[1] and in Kanban Board [2]. >> >> >> 2017: >> >> - >> >> BEAM-1251 is opened, and first efforts to make Beam codebase >> Python3-compatible followed shortly. >> >> >> Q3-Q4 2019: >> >> - >> >> Active work on "futurizing" Beam codebase piece-by-piece while >> preventing regressions in performance in existing Python 2 offering. >> - >> >> Building test infrastructure to incorporate Python 3 test scenarios. >> >> >> Apache Beam 2.11.0 (Q1 2019): >> >> - >> >> "Futurization" of Beam Python codebase completed. >> - >> >> Apache Beam 2.11.0 is released with Python 3 support, with >> limitations. >> - >> >> Continuous pre-commit and post-commit test suites added for Python >> 3.5. >> - >> >> Gaps in Python 3 support in Datastore IO, Avro IO, Bigquery IO >> identified and scoped. >> - >> >> Continuous testing mostly limited to Python 3.5. >> >> >> Apache Beam 2.12.0 (Q2 2019): >> >> - >> >> Pre and Post-commit test coverage expanded to Python 3.5, 3.6, 3.7. >> - >> >> Direct and Dataflow runners added support for Python 3.6 - 3.7. >> >> >> Apache Beam 2.13.0 (Q2 2019) >> >> - >> >> Avro IO support enabled on Python 3. >> - >> >> Datastore IO support enabled on Python 3. >> - >> >> Bigquery IO support for BYTES datatype enabled on Python 3. >> >> >> Apache Beam 2.14.0 (to be released in Q3 2019) >> >> - >> >> Python 3 bug fixes for Bigquery IO and Portable Runner >> - >> >> Every Python SDK commit exercises Direct, Dataflow, and Portable >> Flink runners on Python 3 in various test suites. >> - >> >> Beam 2.14.0 will declare Python 3.5, 3.6, 3.7 support in PyPi. >> >> >> Next steps: >> >> - >> >> Address known limitations and user feedback. >> - >> >> Increase Python 3 test coverage in portable runner. >> - >> >> Assist Beam users in Python 2 -> Python 3 migration. >> - >> >> Deprecate of Python 2 support in Beam, cleanup the codebase. >> >> >> I'd like to thank all Beam contributors who have been helping to push >> this effort so far. >> >> >> [1] https://issues.apache.org/jira/browse/BEAM-1251 >> >> [2] >> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail >> >> On Tue, Jun 18, 2019 at 12:03 AM Valentyn Tymofieiev <valen...@google.com> >> wrote: >> >>> I like the update Ismaël referenced [1], I think we should prepare a >>> similar update for Beam users. I would propose the following: >>> - Designate last LTS release that we will have in 2019 to be the last >>> LTS release with Python 2 support. >>> - Add a Beam-specific deprecation warning on Python 2 starting from the >>> last LTS release, or last 2 releases of Beam in 2019, whichever happens >>> earlier. >>> - Remove Python 2 support starting from the first release in 2020. >>> >>> The cost of maintaining Python 2.7 support is higher than 0. Some issues >>> that come to mind: >>> - Maintaining Py2.7 / Py 3+ compatibility of Beam codebase makes it >>> difficult to use Python 3 syntax in Beam which may be necessary to support >>> and test syntactic constructs introduced in Python 3. >>> - Running additional test suites increases the load on test >>> infrastructure and increases flakiness. >>> >>> [1] https://spark.apache.org/news/plan-for-dropping-python-2 >>> -support.html >>> >>> On Tue, Jun 11, 2019 at 7:57 AM Robert Bradshaw <rober...@google.com> >>> wrote: >>> >>>> Sounds good. >>>> >>>> On Fri, Jun 7, 2019 at 8:28 PM Ahmet Altay <al...@google.com> wrote: >>>> >>>>> I agree with you. A more recent LTS release with python 2 support will >>>>> be good. Cost of maintaining python 2 support is also fairly low (maybe >>>>> zero actually besides keeping some pre-existing compatibility code). >>>>> >>>>> I believe we are referring to two separate things with support: >>>>> - Supporting existing releases for patches - I agree that we need to >>>>> give users a long enough window to upgrade. Great if it happens with an >>>>> LTS >>>>> release. Even if it does not, I think it will be fair to offer patches on >>>>> the last python 2 supporting release during some part of 2020 if that >>>>> becomes necessary. >>>>> - Making new releases with python 2 support - Each new Beam release >>>>> with python 2 support will implicitly extend the lifetime of beam's python >>>>> 2 support. I do not think we need to extend this to beyond 2019. 2 >>>>> releases >>>>> (~ 3 months) after solid python 3 support will very likely put the last >>>>> python 2 supporting release to last quarter of 2019 already. >>>>> >>>>> On Fri, Jun 7, 2019 at 2:15 AM Robert Bradshaw <rober...@google.com> >>>>> wrote: >>>>> >>>>>> I don't think the second release with robust/recommended Python 3 >>>>>> support should be the last release with Python 2 support--that is >>>>>> simply not enough time for people to migrate. (Look at how long it >>>>>> took us...) It does make a lot of sense to at least have one LTS >>>>>> release with support for both. >>>>>> >>>>>> Regarding timeline, I think we could safely say we expect to support >>>>>> Python 2 through 2019, likely for some of 2020 (possibly only via an >>>>>> LTS release), and (very) unlikely beyond 2020. >>>>>> >>>>>> On Wed, Jun 5, 2019 at 6:34 PM Ahmet Altay <al...@google.com> wrote: >>>>>> > >>>>>> > I agree with the sentiment on this thread. Our priority needs to be >>>>>> offering good python 3 support that we can comfortably recommend users to >>>>>> switch. Progress on that so far has been promising and I do anticipate >>>>>> that >>>>>> we will reach there in the near future. >>>>>> > >>>>>> > My proposal would be, once we reach to that state, we can mark the >>>>>> first subsequent Beam release as the last Beam release that supports >>>>>> Python >>>>>> 2. (Alternatively: in line with the previous experimental/deprecated >>>>>> discussion we can make 2 more release with python 2 support rather than >>>>>> just 1 more.) With the current state, we would not give users plenty of >>>>>> time to upgrade python 3. So in addition, I would suggest we can consider >>>>>> and upgrade relief by offering something like a 6-month support on the >>>>>> last >>>>>> python 2 compatible release. We might do that in the context of an LTS >>>>>> release. >>>>>> > >>>>>> > I do not believe we have a timeline we can share with users at this >>>>>> point. However if we go with this suggestion, we will probably support >>>>>> python 2 approximately until mid-2020. >>>>>> > >>>>>> > Ahmet >>>>>> > >>>>>> > On Wed, Jun 5, 2019 at 4:53 AM Tanay Tummalapalli < >>>>>> ttanay...@gmail.com> wrote: >>>>>> >> >>>>>> >> We can support Python 2 for some time in 2020, but, we should >>>>>> target a date no later than 2020 to drop support. >>>>>> >> If we do plan to drop support for Python 2 in 2020, we should sign >>>>>> the Python 3 statement[1], declaring that we will "drop support for >>>>>> Python >>>>>> 2.7 no later than 2020". >>>>>> >> >>>>>> >> In addition to the statement, keeping a target release and date(if >>>>>> possible) or timeline to drop support would also help users to decide >>>>>> when >>>>>> they need to work on migrating to Python 3. >>>>>> >> >>>>>> >> Regards, >>>>>> >> - TT >>>>>> >> >>>>>> >> [1] https://python3statement.org/ >>>>>> >> >>>>>> >> On Wed, Jun 5, 2019 at 4:37 PM Robert Bradshaw < >>>>>> rober...@google.com> wrote: >>>>>> >>> >>>>>> >>> Until Python 3 support for Beam is officially out of beta and >>>>>> >>> recommended, I don't think we can tell people to stop using >>>>>> Python 2. >>>>>> >>> Given that 2020 is just over 6 months away, that seems a short >>>>>> >>> transition time, so I would guess we'll have to continue >>>>>> supporting >>>>>> >>> Python 2 sometime into 2020. >>>>>> >>> >>>>>> >>> A quick survey of users would be valuable here. But first >>>>>> priority is >>>>>> >>> making Python 3 rock solid so we can unconditionally recommend it >>>>>> over >>>>>> >>> Python 2. >>>>>> >>> >>>>>> >>> On Wed, Jun 5, 2019 at 12:27 PM Ismaël Mejía <ieme...@gmail.com> >>>>>> wrote: >>>>>> >>> > >>>>>> >>> > Python 2 won't be maintained after 2020 [1]. I was wondering >>>>>> what will >>>>>> >>> > be our (Beam) plan for this. Other projects [2] have started to >>>>>> alert >>>>>> >>> > users that support will be removed so maybe we should decide or >>>>>> policy >>>>>> >>> > for this too. >>>>>> >>> > >>>>>> >>> > [1] https://pythonclock.org/ >>>>>> >>> > [2] >>>>>> https://spark.apache.org/news/plan-for-dropping-python-2-support.html >>>>>> >>>>>
