[ 
https://issues.apache.org/jira/browse/SPARK-61421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

L. C. Hsieh resolved SPARK-61421.
---------------------------------
    Resolution: Fixed

Issue resolved by pull request 497
[https://github.com/apache/spark-connect-rust/pull/497]

> Bring python/tests and the hand-written pyspark shim under the ruff lint gate 
> in Spark Connect Rust
> ---------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61421
>                 URL: https://issues.apache.org/jira/browse/SPARK-61421
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> In spark-connect-rust, ruff only lints scripts/ and the embedded Row 
> implementation (crates/pyspark-rs/src/row.py). pyproject.toml excludes all of 
> python/pyspark, and the lint workflow passes only those two paths, so the 
> project's own Python code is never linted:
> * python/tests (about 39k lines, including the mock Connect server), and
> * the hand-written modules of the drop-in under python/pyspark, i.e. those 
> that are neither vendored from Apache Spark nor generated.
> Bring both under `ruff check`:
> * Include python/tests and python/pyspark in [tool.ruff], and exclude the 
> vendored paths (those of scripts/check_vendored_upstream.sh, which must stay 
> byte-identical to the pinned tag) and the generated functions_generated.py.
> * Add a check that the vendored paths excluded from ruff are exactly those of 
> the drift gate, so that the two lists cannot diverge.
> * Fix the findings in the code the project owns, and use scoped 
> per-file-ignores, each with its reason, where a rule does not apply (test 
> idioms, re-export surfaces, fork-adapted copies of upstream modules that 
> should stay textually close to upstream).
> * Update the ruff steps of .github/workflows/lint.yml accordingly.
> The formatter (`ruff format`) is out of scope: reformatting python/tests and 
> the shim in one sweep would conflict with every open pull request, so it can 
> follow separately.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to