HyukjinKwon commented on code in PR #46417:
URL: https://github.com/apache/spark/pull/46417#discussion_r1592148119
##########
.github/workflows/build_python_connect35.yml:
##########
@@ -57,29 +59,44 @@ jobs:
with:
distribution: zulu
java-version: 17
- - name: Install Python 3.11
- uses: actions/setup-python@v5
- with:
- python-version: '3.11'
- architecture: x64
+ - name: Free up disk space
+ shell: 'script -q -e -c "bash {0}"'
+ run: |
+ ./dev/free_disk_space_container
- name: Build Spark
run: |
./build/sbt -Phive Test/package
- name: Install Python dependencies
run: |
+ # Server enviornment
+ # See also https://github.com/conda/conda/issues/7980
+ source "$CONDA_PREFIX/etc/profile.d/conda.sh"
+ conda update -q conda
+ conda create -c conda-forge -q -n python3.11 python=3.11
+ conda activate python3.11
+ pip install -r dev/requirements.txt
+ conda deactivate
+
+ # Client enviornment
+ conda create -c conda-forge -q -n python3.9 python=3.9
+ conda activate python3.9
pip install 'numpy==1.25.1' 'pyarrow==12.0.1' 'pandas<=2.0.3' scipy
unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage 'matplotlib==3.7.2'
openpyxl 'memory-profiler==0.60.0' 'scikit-learn==1.1.*'
# Add Python deps for Spark Connect.
pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57'
'protobuf==3.20.3' 'googleapis-common-protos==1.56.4'
# Add torch as a testing dependency for TorchDistributor
- pip install 'torch==2.0.1' 'torchvision==0.15.2' torcheval
+ # pip install 'torch==2.0.1' 'torchvision==0.15.2' torcheval
Review Comment:
```suggestion
pip install 'torch==2.0.1'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
