This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 151f88b53e6 [SPARK-45142][INFRA] Specify the range for Spark Connect
dependencies in pyspark base image
151f88b53e6 is described below
commit 151f88b53e67944d6ca5c635466f50958019c8b4
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Wed Sep 13 21:20:19 2023 +0900
[SPARK-45142][INFRA] Specify the range for Spark Connect dependencies in
pyspark base image
This PR proposes to pin the dependencies related to Spark Connect in its
base image according to the range we support.
See also
https://github.com/apache/spark/blob/master/python/docs/source/getting_started/install.rst#dependencies
To properly test the dependency versions we support.
No, dev-only.
In this PR, it will be tested.
No.
Closes #42898 from HyukjinKwon/SPARK-45142.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
(cherry picked from commit 61435b42fdc4071f35aba6af9248ff9ad8fc8514)
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
dev/infra/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index af8e1a980f9..d3bae836cc6 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -68,7 +68,7 @@ RUN pypy3 -m pip install numpy 'pandas<=2.0.3' scipy coverage
matplotlib
RUN python3.9 -m pip install numpy pyarrow 'pandas<=2.0.3' scipy
unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl
'memory-profiler==0.60.0' 'scikit-learn==1.1.*'
# Add Python deps for Spark Connect.
-RUN python3.9 -m pip install grpcio protobuf googleapis-common-protos
grpcio-status
+RUN python3.9 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==3.20.3' 'googleapis-common-protos==1.56.4'
# Add torch as a testing dependency for TorchDistributor
RUN python3.9 -m pip install torch torchvision torcheval
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
