dbtsai commented on PR #109: URL: https://github.com/apache/spark-connect-rust/pull/109#issuecomment-6477294094
@viirya Thanks for checking this against the base and for pointing out the redundant coercion pass. Addressed in f26ea7e: - Removed `coerce_value` from the input path entirely, along with the unused scalar builder and wrapper. The column loop now passes original values directly to the schema-driven encoder. - Replaced the duplicate-name bug expectation with an in-range Float/Double precision regression: both fields select the original source value, so Float rounding no longer affects the Double field. Removed the out-of-range integer expectations. - Changed the core width check to return `SparkError`, retaining schema-validation-before-width and width-before-encoding ordering. Also checked widths in the Python wrapper before `Row::new`, since explicit-schema input could otherwise still panic there. Added short/long Rust cases and Python inferred/names-only/StructType cases. - Merged the IPC fixtures into the existing test module and reused `struct_field`. Removed the historical implementation comments and changed the zero-column assertion to our own error prefix. Local validation passed: 392 library tests, the offline Rust suite, integration-target compilation with and without `wasm-udf`, formatting/core Clippy, the rebuilt extension's Python width regressions, an unchanged 108-class Python API, and 11 independent Python fixture comparisons. Those fixtures use canned RPC responses, so they do not establish live Spark parity. The new CI runs, including Connect/Pandas/Pipelines parity, are currently queued. Workspace Clippy is blocked locally by a missing wasm32 target; Python-crate Clippy also reports unrelated lint errors in existing code. I updated the description to make clear that the old benchmark numbers cover the earlier implementation, and fresh measurements are needed for this revision. The Python-side `spec_names` copies remain a separate follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
