This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 26c871f18030 [SPARK-48068][PYTHON] `mypy` should have `--python-executable` parameter 26c871f18030 is described below commit 26c871f180306fbf86ce65f14f8e7a71f89885ed Author: Dongjoon Hyun <dh...@apple.com> AuthorDate: Wed May 1 10:42:26 2024 -0700 [SPARK-48068][PYTHON] `mypy` should have `--python-executable` parameter ### What changes were proposed in this pull request? This PR aims to fix `mypy` failure by propagating `lint-python`'s `PYTHON_EXECUTABLE` to `mypy`'s parameter correctly. ### Why are the changes needed? We assumed that `PYTHON_EXECUTABLE` is used for `dev/lint-python` like the following. That's not always guaranteed. We need to use `mypy`'s parameter to make it sure. https://github.com/apache/spark/blob/ff401dde50343c9bbc1c49a0294272f2da7d01e2/.github/workflows/build_and_test.yml#L705 This patch is useful whose `python3` chooses one of multiple Python installation like our CI environment. ``` $ docker run -it --rm ghcr.io/apache/apache-spark-ci-image:master-8905641334 bash WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested root2ef6ce08d2c4:/# python3 --version Python 3.10.12 root2ef6ce08d2c4:/# python3.9 --version Python 3.9.19 ``` For example, the following shows that `PYTHON_EXECUTABLE` is not considered by `mypy`. ``` root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --python-executable=python3.11 --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 3428 root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 1 root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.11 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 1 ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46314 from dongjoon-hyun/SPARK-48068. Authored-by: Dongjoon Hyun <dh...@apple.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- dev/lint-python | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dev/lint-python b/dev/lint-python index 6bd843103bd7..b8703310bc4b 100755 --- a/dev/lint-python +++ b/dev/lint-python @@ -125,6 +125,7 @@ function mypy_annotation_test { echo "starting mypy annotations test..." MYPY_REPORT=$( ($MYPY_BUILD \ + --python-executable $PYTHON_EXECUTABLE \ --namespace-packages \ --config-file python/mypy.ini \ --cache-dir /tmp/.mypy_cache/ \ @@ -184,6 +185,7 @@ function mypy_examples_test { echo "starting mypy examples test..." MYPY_REPORT=$( (MYPYPATH=python $MYPY_BUILD \ + --python-executable $PYTHON_EXECUTABLE \ --namespace-packages \ --config-file python/mypy.ini \ --exclude "mllib/*" \ --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org