[
https://issues.apache.org/jira/browse/SPARK-61421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-61421:
--------------------------------
Fix Version/s: connect-rust-4.3.0
> Bring python/tests and the hand-written pyspark shim under the ruff lint gate
> in Spark Connect Rust
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-61421
> URL: https://issues.apache.org/jira/browse/SPARK-61421
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: L. C. Hsieh
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In spark-connect-rust, ruff only lints scripts/ and the embedded Row
> implementation (crates/pyspark-rs/src/row.py). pyproject.toml excludes all of
> python/pyspark, and the lint workflow passes only those two paths, so the
> project's own Python code is never linted:
> * python/tests (about 39k lines, including the mock Connect server), and
> * the hand-written modules of the drop-in under python/pyspark, i.e. those
> that are neither vendored from Apache Spark nor generated.
> Bring both under `ruff check`:
> * Include python/tests and python/pyspark in [tool.ruff], and exclude the
> vendored paths (those of scripts/check_vendored_upstream.sh, which must stay
> byte-identical to the pinned tag) and the generated functions_generated.py.
> * Add a check that the vendored paths excluded from ruff are exactly those of
> the drift gate, so that the two lists cannot diverge.
> * Fix the findings in the code the project owns, and use scoped
> per-file-ignores, each with its reason, where a rule does not apply (test
> idioms, re-export surfaces, fork-adapted copies of upstream modules that
> should stay textually close to upstream).
> * Update the ruff steps of .github/workflows/lint.yml accordingly.
> The formatter (`ruff format`) is out of scope: reformatting python/tests and
> the shim in one sweep would conflict with every open pull request, so it can
> follow separately.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]