L. C. Hsieh created SPARK-61422:
-----------------------------------

             Summary: Make the generated function dispatch and wrappers of 
spark-connect-rust reproducible from their generator, and check them in CI
                 Key: SPARK-61422
                 URL: https://issues.apache.org/jira/browse/SPARK-61422
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: L. C. Hsieh


In spark-connect-rust, crates/pyspark-rs/src/dispatch_generated.rs and 
python/pyspark/sql/functions_generated.py both start with "Generated by 
scripts/gen_fn_dispatch.py - do not edit directly", but neither can be trusted 
to be what the generator writes:

* The Rust dispatch was edited by hand. The generator keeps 
{{--write-dispatch}} off by default because regenerating would regress the 
hand-tuned arms (round -> round_scale, instr -> instr_with_start, 
count_distinct, ...), and it would in fact emit arms that do not compile.
* The generator finds functions with a regex that misses every generic function 
({{pub fn array<C: Into<Column>>(...)}}) and lets a function that does not 
return a Column swallow the next one (broadcast -> cume_dist).
* The set of Python wrappers is read back from functions_generated.py itself, 
so a wrapper deleted from that file is not regenerated.
* Nothing in CI compares either file with the generator, so drift is 
undetectable.

This issue makes the generator reproduce both checked-in files byte for byte 
(every arm that does not follow from a Rust signature is declared in a table of 
the generator, and a function that fits no rule fails the generation instead of 
being dropped), adds a {{--check}} mode that exits non-zero with a diff, and 
adds a CI workflow and a unit test that run it. The generated files themselves 
do not change.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to