This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b93fd9c074bd [SPARK-47116][INFRA][R] Install proper Python version in
SparkR Windows build to avoid warnings
b93fd9c074bd is described below
commit b93fd9c074bd9d0dd0d7cc06b1cccf0d8525c9ec
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Wed Feb 21 00:42:21 2024 -0800
[SPARK-47116][INFRA][R] Install proper Python version in SparkR Windows
build to avoid warnings
### What changes were proposed in this pull request?
This PR installs Python 3.11 in SparkR build on Windows.
### Why are the changes needed?
To remove unrelated warnings:
(https://github.com/HyukjinKwon/spark/actions/runs/7985005685/job/21802732830
)
```
Traceback (most recent call last):
File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\runpy.py", line
183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\runpy.py", line
109, in _get_module_details
__import__(pkg_name)
File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\__init__.py", line
[53](https://github.com/HyukjinKwon/spark/actions/runs/7985005685/job/21802732830#step:10:54),
in <module>
File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\rdd.py", line
[54](https://github.com/HyukjinKwon/spark/actions/runs/7985005685/job/21802732830#step:10:55),
in <module>
File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\java_gateway.py",
line 33, in <module>
File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\serializers.py",
line 69, in <module>
File
"D:\a\spark\spark\python\lib\pyspark.zip\pyspark\cloudpickle\__init__.py", line
1, in <module>
File
"D:\a\spark\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle.py",
line 80, in <module>
ImportError: cannot import name 'CellType' from 'types'
(C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\types.py)
```
SparkR build does not need Python. However, it shows warnings when the
Python version is too low during the attempt to look up Python Data Sources for
session initialization. The Windows 2019 runner includes Python 3.7, which
Spark does not support. Therefore, we simply install the proper Python for
simplicity.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Manually
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45196 from HyukjinKwon/python-errors.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.github/workflows/build_sparkr_window.yml | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/.github/workflows/build_sparkr_window.yml
b/.github/workflows/build_sparkr_window.yml
index fbaca36f9f87..4375b21be55e 100644
--- a/.github/workflows/build_sparkr_window.yml
+++ b/.github/workflows/build_sparkr_window.yml
@@ -58,6 +58,15 @@ jobs:
Rscript -e "install.packages(c('knitr', 'rmarkdown', 'testthat',
'e1071', 'survival', 'arrow', 'xml2'), repos='https://cloud.r-project.org/')"
Rscript -e "pkg_list <- as.data.frame(installed.packages()[,c(1,
3:4)]); pkg_list[is.na(pkg_list$Priority), 1:2, drop = FALSE]"
shell: cmd
+ # SparkR build does not need Python. However, it shows warnings when the
Python version is too low during
+ # the attempt to look up Python Data Sources for session initialization.
The Windows 2019 runner
+ # includes Python 3.7, which Spark does not support. Therefore, we simply
install the proper Python
+ # for simplicity, see SPARK-47116.
+ - name: Install Python 3.11
+ uses: actions/setup-python@v5
+ with:
+ python-version: '3.11'
+ architecture: x64
- name: Build Spark
run: |
rem 1. '-Djna.nosys=true' is required to avoid kernel32.dll load
failure.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
