gemini-code-assist[bot] commented on code in PR #36811: URL: https://github.com/apache/beam/pull/36811#discussion_r2523952761
########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. Review Comment:  There are a couple of typos in this sentence. `The means` should be `This means`, and `priorty` should be `priority`. ```suggestion Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. This means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priority than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. ``` ########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. + +1. Add a Beam Python container for the new python version. + * https://github.com/apache/beam/tree/master/sdks/python/container + +1. Add a new Python version to different test suites: + * [Tox test suties](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) Review Comment:  There is a typo here. `suties` should be `suites`. ```suggestion * [Tox test suites](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) ``` ########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. + +1. Add a Beam Python container for the new python version. + * https://github.com/apache/beam/tree/master/sdks/python/container + +1. Add a new Python version to different test suites: + * [Tox test suties](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + * Gradle tasks such as pre-commits, post-commits etc. + * Runner-specific versioning checks + * Fix any tests that fail on the new Python version. + * Typically, a new Python version requires updating Beam Type Inference code. See https://github.com/apache/beam/issues/31047 + +1. Add the GitHub actions workflows for the new Python version. + * Example: https://github.com/apache/beam/blob/master/.github/workflows/python_tests.yml + * The minimum and maximum Python versions are defined in a number of workflows and the [test-properties.json](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/actions/setup-default-test-properties/test-properties.json) file, there will be potentially hundreds of changes for this step. + +1. Add support for building wheels for the new Python version. + * https://github.com/apache/beam/blob/master/.github/workflows/build_wheels.yml + +1. Update the upper limit in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78) with the next major Python version. Review Comment:  This link is pinned to a specific commit hash. It would be better to point to the `master` branch to prevent the link from becoming stale. ```suggestion 1. Update the upper limit in [__init__.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/__init__.py#L78) with the next major Python version. ``` ########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. + +1. Add a Beam Python container for the new python version. + * https://github.com/apache/beam/tree/master/sdks/python/container + +1. Add a new Python version to different test suites: + * [Tox test suties](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + * Gradle tasks such as pre-commits, post-commits etc. + * Runner-specific versioning checks + * Fix any tests that fail on the new Python version. + * Typically, a new Python version requires updating Beam Type Inference code. See https://github.com/apache/beam/issues/31047 + +1. Add the GitHub actions workflows for the new Python version. + * Example: https://github.com/apache/beam/blob/master/.github/workflows/python_tests.yml + * The minimum and maximum Python versions are defined in a number of workflows and the [test-properties.json](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/actions/setup-default-test-properties/test-properties.json) file, there will be potentially hundreds of changes for this step. + +1. Add support for building wheels for the new Python version. + * https://github.com/apache/beam/blob/master/.github/workflows/build_wheels.yml + +1. Update the upper limit in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78) with the next major Python version. + +1. Add the new Python version in release validation scripts: https://github.com/apache/beam/pull/31415 + +* If there is a new feature update or there is a regression when adding a new Python version, please file an [issue](https://github.com/apache/beam/issues). + * **All the unit tests and Integration tests must pass before merging the new version.** + * If you are a non-committer, please ask the committers to run a seed job on your PR to test all the new changes. + +For an example, see PRs associated with https://github.com/apache/beam/issues/29149, and commits on https://github.com/apache/beam/pull/30828 which add Python 3.12 support. + +## Removing a Python Version + +1. Bump the Python version in [setup.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/setup.py#L152) and update the Python version warning in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78). Review Comment:  These links are pinned to specific commit hashes. It would be better to point to the `master` branch to prevent the links from becoming stale. Also, there is a trailing whitespace at the end of the line that should be removed. ```suggestion 1. Bump the Python version in [setup.py](https://github.com/apache/beam/blob/master/sdks/python/setup.py#L152) and update the Python version warning in [__init__.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/__init__.py#L78). ``` ########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. + +1. Add a Beam Python container for the new python version. + * https://github.com/apache/beam/tree/master/sdks/python/container + +1. Add a new Python version to different test suites: + * [Tox test suties](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + * Gradle tasks such as pre-commits, post-commits etc. + * Runner-specific versioning checks + * Fix any tests that fail on the new Python version. + * Typically, a new Python version requires updating Beam Type Inference code. See https://github.com/apache/beam/issues/31047 + +1. Add the GitHub actions workflows for the new Python version. + * Example: https://github.com/apache/beam/blob/master/.github/workflows/python_tests.yml + * The minimum and maximum Python versions are defined in a number of workflows and the [test-properties.json](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/actions/setup-default-test-properties/test-properties.json) file, there will be potentially hundreds of changes for this step. + +1. Add support for building wheels for the new Python version. + * https://github.com/apache/beam/blob/master/.github/workflows/build_wheels.yml + +1. Update the upper limit in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78) with the next major Python version. + +1. Add the new Python version in release validation scripts: https://github.com/apache/beam/pull/31415 + +* If there is a new feature update or there is a regression when adding a new Python version, please file an [issue](https://github.com/apache/beam/issues). + * **All the unit tests and Integration tests must pass before merging the new version.** + * If you are a non-committer, please ask the committers to run a seed job on your PR to test all the new changes. + +For an example, see PRs associated with https://github.com/apache/beam/issues/29149, and commits on https://github.com/apache/beam/pull/30828 which add Python 3.12 support. + +## Removing a Python Version + +1. Bump the Python version in [setup.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/setup.py#L152) and update the Python version warning in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78). + +1. Remove test suites for the unsupported Python version: + * Migrate GitHub actions workflows from the deprecated Python version to the next one + * Example PR: https://github.com/apache/beam/pull/32429 + * Make these changes on a branch in the main Beam repository if possible so you can execute the new workflows directly for testing. + * Some workflows only run on the minimum supported Python version (like the linting and coverage precommits.) These may utilize libraries that need updates to run on the next Python version. + * Remove the unsuppported Python version from the following files/directories: + * sdks/python/test-suites/gradle.properties + * apache_beam/testing/tox + Move any workflows that exist only for the minimum Python version from tox/py3X to the next minimum Python version's folder + * apache_beam/testing/dataflow + * apache_beam/testing/direct + * apache_beam/testing/portable + * Remove the unsupported python version gradle tasks from Review Comment:  For consistency with the rest of the document, `python` should be capitalized to `Python`. ```suggestion * Remove the unsupported Python version gradle tasks from ``` ########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. + +1. Add a Beam Python container for the new python version. + * https://github.com/apache/beam/tree/master/sdks/python/container + +1. Add a new Python version to different test suites: + * [Tox test suties](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + * Gradle tasks such as pre-commits, post-commits etc. + * Runner-specific versioning checks + * Fix any tests that fail on the new Python version. + * Typically, a new Python version requires updating Beam Type Inference code. See https://github.com/apache/beam/issues/31047 + +1. Add the GitHub actions workflows for the new Python version. + * Example: https://github.com/apache/beam/blob/master/.github/workflows/python_tests.yml + * The minimum and maximum Python versions are defined in a number of workflows and the [test-properties.json](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/actions/setup-default-test-properties/test-properties.json) file, there will be potentially hundreds of changes for this step. + +1. Add support for building wheels for the new Python version. + * https://github.com/apache/beam/blob/master/.github/workflows/build_wheels.yml + +1. Update the upper limit in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78) with the next major Python version. + +1. Add the new Python version in release validation scripts: https://github.com/apache/beam/pull/31415 + +* If there is a new feature update or there is a regression when adding a new Python version, please file an [issue](https://github.com/apache/beam/issues). + * **All the unit tests and Integration tests must pass before merging the new version.** + * If you are a non-committer, please ask the committers to run a seed job on your PR to test all the new changes. + +For an example, see PRs associated with https://github.com/apache/beam/issues/29149, and commits on https://github.com/apache/beam/pull/30828 which add Python 3.12 support. + +## Removing a Python Version + +1. Bump the Python version in [setup.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/setup.py#L152) and update the Python version warning in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78). + +1. Remove test suites for the unsupported Python version: + * Migrate GitHub actions workflows from the deprecated Python version to the next one + * Example PR: https://github.com/apache/beam/pull/32429 + * Make these changes on a branch in the main Beam repository if possible so you can execute the new workflows directly for testing. + * Some workflows only run on the minimum supported Python version (like the linting and coverage precommits.) These may utilize libraries that need updates to run on the next Python version. + * Remove the unsuppported Python version from the following files/directories: + * sdks/python/test-suites/gradle.properties + * apache_beam/testing/tox + Move any workflows that exist only for the minimum Python version from tox/py3X to the next minimum Python version's folder + * apache_beam/testing/dataflow + * apache_beam/testing/direct + * apache_beam/testing/portable + * Remove the unsupported python version gradle tasks from + * build.gradle.kts + * settings.gradle.kts + * buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy + * Remove the support for building wheels and source distributions for the unsupported Python version from [.github/workflows/build_wheels.yml](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/workflows/build_wheels.yml) Review Comment:  This link is pinned to a specific commit hash. It would be better to point to the `master` branch to prevent the link from becoming stale. ```suggestion * Remove the support for building wheels and source distributions for the unsupported Python version from [.github/workflows/build_wheels.yml](https://github.com/apache/beam/blob/master/.github/workflows/build_wheels.yml) ``` ########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. Review Comment:  There is a typo here. `dependenciess` should be `dependencies`. ```suggestion * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependencies. ``` ########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. + +1. Add a Beam Python container for the new python version. Review Comment:  For consistency with the rest of the document, `python` should be capitalized to `Python`. ```suggestion 1. Add a Beam Python container for the new Python version. ``` ########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. + +1. Add a Beam Python container for the new python version. + * https://github.com/apache/beam/tree/master/sdks/python/container + +1. Add a new Python version to different test suites: + * [Tox test suties](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + * Gradle tasks such as pre-commits, post-commits etc. + * Runner-specific versioning checks + * Fix any tests that fail on the new Python version. Review Comment:  Please remove the trailing whitespace at the end of this line. ```suggestion * Fix any tests that fail on the new Python version. ``` ########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. + +1. Add a Beam Python container for the new python version. + * https://github.com/apache/beam/tree/master/sdks/python/container + +1. Add a new Python version to different test suites: + * [Tox test suties](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + * Gradle tasks such as pre-commits, post-commits etc. + * Runner-specific versioning checks + * Fix any tests that fail on the new Python version. + * Typically, a new Python version requires updating Beam Type Inference code. See https://github.com/apache/beam/issues/31047 + +1. Add the GitHub actions workflows for the new Python version. + * Example: https://github.com/apache/beam/blob/master/.github/workflows/python_tests.yml + * The minimum and maximum Python versions are defined in a number of workflows and the [test-properties.json](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/actions/setup-default-test-properties/test-properties.json) file, there will be potentially hundreds of changes for this step. + +1. Add support for building wheels for the new Python version. + * https://github.com/apache/beam/blob/master/.github/workflows/build_wheels.yml + +1. Update the upper limit in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78) with the next major Python version. + +1. Add the new Python version in release validation scripts: https://github.com/apache/beam/pull/31415 + +* If there is a new feature update or there is a regression when adding a new Python version, please file an [issue](https://github.com/apache/beam/issues). + * **All the unit tests and Integration tests must pass before merging the new version.** + * If you are a non-committer, please ask the committers to run a seed job on your PR to test all the new changes. + +For an example, see PRs associated with https://github.com/apache/beam/issues/29149, and commits on https://github.com/apache/beam/pull/30828 which add Python 3.12 support. + +## Removing a Python Version + +1. Bump the Python version in [setup.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/setup.py#L152) and update the Python version warning in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78). + +1. Remove test suites for the unsupported Python version: + * Migrate GitHub actions workflows from the deprecated Python version to the next one + * Example PR: https://github.com/apache/beam/pull/32429 + * Make these changes on a branch in the main Beam repository if possible so you can execute the new workflows directly for testing. + * Some workflows only run on the minimum supported Python version (like the linting and coverage precommits.) These may utilize libraries that need updates to run on the next Python version. + * Remove the unsuppported Python version from the following files/directories: Review Comment:  There is a typo here. `unsuppported` should be `unsupported`. ```suggestion * Remove the unsupported Python version from the following files/directories: ``` ########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. + +1. Add a Beam Python container for the new python version. + * https://github.com/apache/beam/tree/master/sdks/python/container + +1. Add a new Python version to different test suites: + * [Tox test suties](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + * Gradle tasks such as pre-commits, post-commits etc. + * Runner-specific versioning checks + * Fix any tests that fail on the new Python version. + * Typically, a new Python version requires updating Beam Type Inference code. See https://github.com/apache/beam/issues/31047 + +1. Add the GitHub actions workflows for the new Python version. + * Example: https://github.com/apache/beam/blob/master/.github/workflows/python_tests.yml + * The minimum and maximum Python versions are defined in a number of workflows and the [test-properties.json](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/actions/setup-default-test-properties/test-properties.json) file, there will be potentially hundreds of changes for this step. Review Comment:  This link is pinned to a specific commit hash. It would be better to point to the `master` branch to prevent the link from becoming stale. ```suggestion * The minimum and maximum Python versions are defined in a number of workflows and the [test-properties.json](https://github.com/apache/beam/blob/master/.github/actions/setup-default-test-properties/test-properties.json) file, there will be potentially hundreds of changes for this step. ``` ########## contributor-docs/updating-supported-python-versions.md: ########## @@ -0,0 +1,83 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. + +1. Add a Beam Python container for the new python version. + * https://github.com/apache/beam/tree/master/sdks/python/container + +1. Add a new Python version to different test suites: + * [Tox test suties](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + * Gradle tasks such as pre-commits, post-commits etc. + * Runner-specific versioning checks + * Fix any tests that fail on the new Python version. + * Typically, a new Python version requires updating Beam Type Inference code. See https://github.com/apache/beam/issues/31047 + +1. Add the GitHub actions workflows for the new Python version. + * Example: https://github.com/apache/beam/blob/master/.github/workflows/python_tests.yml + * The minimum and maximum Python versions are defined in a number of workflows and the [test-properties.json](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/actions/setup-default-test-properties/test-properties.json) file, there will be potentially hundreds of changes for this step. + +1. Add support for building wheels for the new Python version. + * https://github.com/apache/beam/blob/master/.github/workflows/build_wheels.yml + +1. Update the upper limit in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78) with the next major Python version. + +1. Add the new Python version in release validation scripts: https://github.com/apache/beam/pull/31415 + +* If there is a new feature update or there is a regression when adding a new Python version, please file an [issue](https://github.com/apache/beam/issues). + * **All the unit tests and Integration tests must pass before merging the new version.** + * If you are a non-committer, please ask the committers to run a seed job on your PR to test all the new changes. + +For an example, see PRs associated with https://github.com/apache/beam/issues/29149, and commits on https://github.com/apache/beam/pull/30828 which add Python 3.12 support. + +## Removing a Python Version + +1. Bump the Python version in [setup.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/setup.py#L152) and update the Python version warning in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78). + +1. Remove test suites for the unsupported Python version: + * Migrate GitHub actions workflows from the deprecated Python version to the next one + * Example PR: https://github.com/apache/beam/pull/32429 + * Make these changes on a branch in the main Beam repository if possible so you can execute the new workflows directly for testing. + * Some workflows only run on the minimum supported Python version (like the linting and coverage precommits.) These may utilize libraries that need updates to run on the next Python version. + * Remove the unsuppported Python version from the following files/directories: + * sdks/python/test-suites/gradle.properties + * apache_beam/testing/tox + Move any workflows that exist only for the minimum Python version from tox/py3X to the next minimum Python version's folder + * apache_beam/testing/dataflow + * apache_beam/testing/direct + * apache_beam/testing/portable + * Remove the unsupported python version gradle tasks from + * build.gradle.kts + * settings.gradle.kts + * buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy + * Remove the support for building wheels and source distributions for the unsupported Python version from [.github/workflows/build_wheels.yml](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/workflows/build_wheels.yml) + * Remove the unsupported Python version from [sdks/python/tox.ini](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + + Review Comment:  This extra blank line can be removed for better formatting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
