This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 19ec9915a771 [SPARK-54632][INFRA][FOLLOW-UP] Enable ruff on our CI and
lint-python
19ec9915a771 is described below
commit 19ec9915a77145a9e685a7dc3d15c79ed477e438
Author: Tian Gao <[email protected]>
AuthorDate: Thu Dec 11 08:58:11 2025 +0900
[SPARK-54632][INFRA][FOLLOW-UP] Enable ruff on our CI and lint-python
### What changes were proposed in this pull request?
This add `ruff` to our docker image and enables ruff check in our CI. Also
this adds ruff check in `dev/lint-python`.
We want to have both `ruff` and `flake8` run in CI for a while to confirm
the compatibility then we will deprecate `flake8`.
~~It is intentional to leave ruff version blank - so it uses the latest
version. I think the linter rule is pretty stable and the version upgrade
should not affect our workflow too much. This is an experiment about using the
latest version of devtool. Pinning the version has its advantage but upgrading
the version is painful. If we can occasionally (per month?) fix a small amount
of code for linter, I think that's acceptable.~~
After discussion with Yicong-Huang I think it's a good idea to pin the
version - so we can do all the code fix with devtool change in a single PR. We
should avoid having multiple people trying to fix the code when there is a new
rule.
### Why are the changes needed?
`flake8` is too slow and our pinned version is just too old. We should
replace it.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Local ruff test passes.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #53412 from gaogaotiantian/add-ruff-to-ci.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
dev/lint-python | 1 +
dev/spark-test-image/lint/Dockerfile | 1 +
python/pyspark/sql/tests/arrow/test_arrow_udf_typehints.py | 2 +-
python/pyspark/sql/tests/pandas/test_pandas_udf_typehints.py | 2 +-
.../tests/pandas/test_pandas_udf_typehints_with_future_annotations.py | 3 ++-
5 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/dev/lint-python b/dev/lint-python
index fab7ad9ebe26..3b64671c5cde 100755
--- a/dev/lint-python
+++ b/dev/lint-python
@@ -79,6 +79,7 @@ if [[ -z
"$COMPILE_TEST$BLACK_TEST$PYSPARK_CUSTOM_ERRORS_CHECK_TEST$FLAKE8_TEST$
BLACK_TEST=true
PYSPARK_CUSTOM_ERRORS_CHECK_TEST=true
FLAKE8_TEST=true
+ RUFF_TEST=true
MYPY_TEST=true
MYPY_EXAMPLES_TEST=true
MYPY_DATA_TEST=true
diff --git a/dev/spark-test-image/lint/Dockerfile
b/dev/spark-test-image/lint/Dockerfile
index 6ab571bf35d6..4984f56fc763 100644
--- a/dev/spark-test-image/lint/Dockerfile
+++ b/dev/spark-test-image/lint/Dockerfile
@@ -80,6 +80,7 @@ RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
RUN python3.11 -m pip install \
'black==23.12.1' \
'flake8==3.9.0' \
+ 'ruff==0.14.8' \
'googleapis-common-protos-stubs==2.2.0' \
'grpc-stubs==1.24.11' \
'grpcio-status==1.76.0' \
diff --git a/python/pyspark/sql/tests/arrow/test_arrow_udf_typehints.py
b/python/pyspark/sql/tests/arrow/test_arrow_udf_typehints.py
index a117a049952c..5136cadf7ab8 100644
--- a/python/pyspark/sql/tests/arrow/test_arrow_udf_typehints.py
+++ b/python/pyspark/sql/tests/arrow/test_arrow_udf_typehints.py
@@ -461,7 +461,7 @@ class ArrowUDFTypeHintsTests(ReusedSQLTestCase):
if __name__ == "__main__":
- from pyspark.sql.tests.arrow.test_arrow_udf_typehints import * # noqa:
#F401
+ from pyspark.sql.tests.arrow.test_arrow_udf_typehints import * # noqa:
F401
try:
import xmlrunner
diff --git a/python/pyspark/sql/tests/pandas/test_pandas_udf_typehints.py
b/python/pyspark/sql/tests/pandas/test_pandas_udf_typehints.py
index 7e9f22bb94bb..d4c1cf5d62ff 100644
--- a/python/pyspark/sql/tests/pandas/test_pandas_udf_typehints.py
+++ b/python/pyspark/sql/tests/pandas/test_pandas_udf_typehints.py
@@ -444,7 +444,7 @@ class PandasUDFTypeHintsTests(ReusedSQLTestCase):
if __name__ == "__main__":
- from pyspark.sql.tests.pandas.test_pandas_udf_typehints import * # noqa:
#F401
+ from pyspark.sql.tests.pandas.test_pandas_udf_typehints import * # noqa:
F401
try:
import xmlrunner
diff --git
a/python/pyspark/sql/tests/pandas/test_pandas_udf_typehints_with_future_annotations.py
b/python/pyspark/sql/tests/pandas/test_pandas_udf_typehints_with_future_annotations.py
index 88cc4b81778a..d2e5dd167316 100644
---
a/python/pyspark/sql/tests/pandas/test_pandas_udf_typehints_with_future_annotations.py
+++
b/python/pyspark/sql/tests/pandas/test_pandas_udf_typehints_with_future_annotations.py
@@ -367,7 +367,8 @@ class
PandasUDFTypeHintsWithFutureAnnotationsTests(ReusedSQLTestCase):
if __name__ == "__main__":
- from
pyspark.sql.tests.pandas.test_pandas_udf_typehints_with_future_annotations
import * # noqa: #F401
+ # E501: line too long
+ from
pyspark.sql.tests.pandas.test_pandas_udf_typehints_with_future_annotations
import * # noqa: F401, E501
try:
import xmlrunner
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
