dbtsai commented on PR #120: URL: https://github.com/apache/spark-connect-rust/pull/120#issuecomment-6511704546
Implemented propagation in https://github.com/apache/spark-connect-rust/pull/120/commits/ed7efa6c1c427ef9b63ca94c3f8c578d34e1217e, along with the documentation and test cleanup from this review. Selected unary transformations snapshot already-resolved schemas. Set operations reuse them only when both cached schemas match exactly in the same session. `to(schema)` and stateless partition/group pandas/Arrow UDFs use declared struct outputs. Propagation does not resolve parents or wait on an in-flight analysis. The mock service now lives in `tests/_mock_connect_server.py` with neutral names. One reference detail: pinned Spark 4.2's [`createDataFrame`](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/sql/connect/session.py#L792) constructs `DataFrame(plan, self)` without seeding `_cached_schema`, including with an explicit StructType. I kept that behavior. Validation: 398 Rust library tests and 318 CI-selected offline Python tests pass. Nineteen live cases match reference PySpark on Spark 4.2 for schemas, metadata, columns, dtypes, and rows. The prior build fails all 16 selected propagation cases; a deliberately blocking snapshot fails the bounded lock-wait test. Release/WASM integration targets compile; Clippy has the same 923 diagnostics as the base. The PR description now reflects this scope and validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
