This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new caec93ae70aa [SPARK-45990][SPARK-45987][PYTHON][CONNECT] Upgrade `protobuf` to 4.25.1 to support `Python 3.11` caec93ae70aa is described below commit caec93ae70aa0a0f83da64f134e1d4824256be14 Author: Dongjoon Hyun <dh...@apple.com> AuthorDate: Sat Nov 18 22:53:47 2023 -0800 [SPARK-45990][SPARK-45987][PYTHON][CONNECT] Upgrade `protobuf` to 4.25.1 to support `Python 3.11` ### What changes were proposed in this pull request? This PR aims to upgrade `protobuf` from `3.20.3` to `4.25.1` to fix PySpark failures on Python 3.11 environment for Apache Spark 4.0.0. ### Why are the changes needed? Currently, Python 3.11 Daily CI is failing. - https://github.com/apache/spark/actions/workflows/build_python.yml `v4.25.0` is the first release where `protobuf` starts to support Python 3.11 officially. - https://pypi.org/project/protobuf/4.25.0/ - https://pypi.org/project/protobuf/4.25.1/ (the latest) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. I also verified this manually. ``` $ python/run-tests --modules=pyspark-connect --parallelism=1 --python-executables=python3.11 ... Finished test(python3.11): pyspark.sql.connect.window (6s) Tests passed in 780 seconds ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #43885 from dongjoon-hyun/SPARK-45990. Authored-by: Dongjoon Hyun <dh...@apple.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- .github/workflows/build_and_test.yml | 4 ++-- dev/infra/Dockerfile | 6 +++--- dev/requirements.txt | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index 96d2b5fcc231..135f43d07b13 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -256,7 +256,7 @@ jobs: - name: Install Python packages (Python 3.8) if: (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-')) || contains(matrix.modules, 'connect') run: | - python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57' 'protobuf==3.20.3' + python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57' 'protobuf==4.25.1' python3.8 -m pip list # Run the tests. - name: Run tests @@ -708,7 +708,7 @@ jobs: mkdir -p $HOME/buf tar -xvzf buf-Linux-x86_64.tar.gz -C $HOME/buf --strip-components 1 rm buf-Linux-x86_64.tar.gz - python3.9 -m pip install 'protobuf==3.20.3' 'mypy-protobuf==3.3.0' + python3.9 -m pip install 'protobuf==4.25.1' 'mypy-protobuf==3.3.0' - name: Python code generation check if: inputs.branch != 'branch-3.3' && inputs.branch != 'branch-3.4' run: if test -f ./dev/connect-check-protos.py; then PATH=$PATH:$HOME/buf/bin PYTHON_EXECUTABLE=python3.9 ./dev/connect-check-protos.py; fi diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile index a6e55200a55b..cc00289b963b 100644 --- a/dev/infra/Dockerfile +++ b/dev/infra/Dockerfile @@ -92,7 +92,7 @@ RUN pypy3 -m pip install numpy 'pandas<=2.1.3' scipy coverage matplotlib RUN python3.9 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn==1.1.*' # Add Python deps for Spark Connect. -RUN python3.9 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57' 'protobuf==3.20.3' 'googleapis-common-protos==1.56.4' +RUN python3.9 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57' 'protobuf==4.25.1' 'googleapis-common-protos==1.56.4' # Add torch as a testing dependency for TorchDistributor RUN python3.9 -m pip install 'torch<=2.0.1' torchvision --index-url https://download.pytorch.org/whl/cpu @@ -107,7 +107,7 @@ RUN apt-get update && apt-get install -y \ && rm -rf /var/lib/apt/lists/* RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10 RUN python3.10 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn==1.1.*' -RUN python3.10 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57' 'protobuf==3.20.3' 'googleapis-common-protos==1.56.4' +RUN python3.10 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57' 'protobuf==4.25.1' 'googleapis-common-protos==1.56.4' RUN python3.10 -m pip install 'torch<=2.0.1' torchvision --index-url https://download.pytorch.org/whl/cpu RUN python3.10 -m pip install torcheval RUN python3.10 -m pip install deepspeed @@ -119,7 +119,7 @@ RUN apt-get update && apt-get install -y \ && rm -rf /var/lib/apt/lists/* RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 RUN python3.11 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn==1.1.*' -RUN python3.11 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57' 'protobuf==3.20.3' 'googleapis-common-protos==1.56.4' +RUN python3.11 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57' 'protobuf==4.25.1' 'googleapis-common-protos==1.56.4' RUN python3.11 -m pip install 'torch<=2.0.1' torchvision --index-url https://download.pytorch.org/whl/cpu RUN python3.11 -m pip install torcheval RUN python3.11 -m pip install deepspeed diff --git a/dev/requirements.txt b/dev/requirements.txt index f222f99d5831..2658f8eec82d 100644 --- a/dev/requirements.txt +++ b/dev/requirements.txt @@ -53,7 +53,7 @@ py # Spark Connect (required) grpcio>=1.48,<1.57 grpcio-status>=1.48,<1.57 -protobuf==3.20.3 +protobuf==4.25.1 googleapis-common-protos==1.56.4 # Spark Connect python proto generation plugin (optional) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org