This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 2dde0164aaca [SPARK-51159][INFRA] Pin `plotly<6.0.0` and `torch<2.6.0`
in release docker image
2dde0164aaca is described below
commit 2dde0164aaca18031539687e7d680a9755e365a1
Author: Ruifeng Zheng <ruife...@apache.org>
AuthorDate: Wed Feb 12 08:25:11 2025 +0800
[SPARK-51159][INFRA] Pin `plotly<6.0.0` and `torch<2.6.0` in release docker
image
### What changes were proposed in this pull request?
Pin `plotly<6.0.0` and `torch<2.6.0` in release docker image
### Why are the changes needed?
both `plotly==6.0.0` and `torch==2.6.0` introduced breaking changes and
fail pyspark
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
I have manually triggered 3.9 and 3.10 daily test including the changes:
https://github.com/apache/spark/actions/runs/13259081421
https://github.com/apache/spark/actions/runs/13259077916
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #49886 from zhengruifeng/fix_release_docker_image.
Authored-by: Ruifeng Zheng <ruife...@apache.org>
Signed-off-by: Ruifeng Zheng <ruife...@apache.org>
(cherry picked from commit b21428f6e14a6aa4dd857948dc945467936b2824)
Signed-off-by: Ruifeng Zheng <ruife...@apache.org>
---
dev/create-release/spark-rm/Dockerfile | 16 +++-------------
1 file changed, 3 insertions(+), 13 deletions(-)
diff --git a/dev/create-release/spark-rm/Dockerfile
b/dev/create-release/spark-rm/Dockerfile
index bbf10755e840..4678af06b96b 100644
--- a/dev/create-release/spark-rm/Dockerfile
+++ b/dev/create-release/spark-rm/Dockerfile
@@ -100,7 +100,7 @@ RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3
RUN pypy3 -m pip install numpy 'six==1.16.0' 'pandas==2.2.3' scipy coverage
matplotlib lxml
-ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0 pandas==2.2.3 scipy
plotly>=4.8 mlflow>=2.8.1 coverage matplotlib openpyxl memory-profiler>=0.61.0
scikit-learn>=1.3.2 twine==3.4.1"
+ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0 pandas==2.2.3 scipy
plotly<6.0.0 mlflow>=2.8.1 coverage matplotlib openpyxl memory-profiler>=0.61.0
scikit-learn>=1.3.2 twine==3.4.1"
# Python deps for Spark Connect
ARG CONNECT_PIP_PKGS="grpcio==1.67.0 grpcio-status==1.67.0 protobuf==5.29.1
googleapis-common-protos==1.65.0 graphviz==0.20.3"
@@ -109,7 +109,7 @@ RUN curl -sS https://bootstrap.pypa.io/get-pip.py |
python3.10
RUN python3.10 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
RUN python3.10 -m pip install --ignore-installed 'six==1.16.0' # Avoid
`python3-six` installation
RUN python3.10 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
- python3.10 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
+ python3.10 -m pip install 'torch<2.6.0' torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.10 -m pip install deepspeed torcheval && \
python3.10 -m pip cache purge
@@ -121,20 +121,10 @@ RUN apt-get update && apt-get install -y \
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9
RUN python3.9 -m pip install --ignore-installed blinker>=1.6.2 # mlflow needs
this
RUN python3.9 -m pip install --force $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
- python3.9 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
+ python3.9 -m pip install 'torch<2.6.0' torchvision --index-url
https://download.pytorch.org/whl/cpu && \
python3.9 -m pip install torcheval && \
python3.9 -m pip cache purge
-# Should unpin 'sphinxcontrib-*' after upgrading sphinx>5
-# See 'ipython_genutils' in SPARK-38517
-# See 'docutils<0.18.0' in SPARK-39421
-RUN python3.9 -m pip install 'sphinx==4.5.0' mkdocs
'pydata_sphinx_theme>=0.13' sphinx-copybutton nbsphinx numpydoc jinja2
markupsafe 'pyzmq<24.0.0' \
-ipython ipython_genutils sphinx_plotly_directive 'numpy>=1.20.0' pyarrow
pandas 'plotly>=4.8' 'docutils<0.18.0' \
-'flake8==3.9.0' 'mypy==1.8.0' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3'
'black==23.12.1' \
-'pandas-stubs==1.2.0.53' 'grpcio==1.67.0' 'grpc-stubs==1.24.11'
'googleapis-common-protos-stubs==2.2.0' \
-'sphinxcontrib-applehelp==1.0.4' 'sphinxcontrib-devhelp==1.0.2'
'sphinxcontrib-htmlhelp==2.0.1' 'sphinxcontrib-qthelp==1.0.3'
'sphinxcontrib-serializinghtml==1.1.5'
-RUN python3.9 -m pip list
-
RUN gem install --no-document "bundler:2.4.22"
RUN ln -s "$(which python3.9)" "/usr/local/bin/python"
RUN ln -s "$(which python3.9)" "/usr/local/bin/python3"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
