dbtsai opened a new pull request, #111: URL: https://github.com/apache/spark-connect-rust/pull/111
### What changes were proposed in this pull request? Convert exact built-in Python `int` and `float` values before generic bool/Decimal detection in local DataFrame input conversion. Bool, numeric subclasses, custom conversion protocols and overflowing integers retain the existing fallback. Add offline regression tests for numeric boundaries, float bits, subclasses, protocols and nested values. ### Why are the changes needed? Ordinary numeric inputs currently pay for generic type detection for every value. In an isolated macOS arm64/Python 3.12 benchmark, the same target effects repeated in a full screen and independent confirmation. Confirmation medians relative to `c36c846`: | Input | CPU | Wall | Held RSS | | --- | ---: | ---: | ---: | | Tuples, 10,000 × 32 | −74.39% | −74.40% | −0.039 MiB | | Tuples, 10,000 × 3 | −41.92% | −41.92% | −0.016 MiB | | Numeric arrays | −37.80% | −37.77% | 0.000 MiB | The production patch is the measured patch. Each phase used 30 fresh processes per build/case in rotating order across 33 cases, with isolated release wheels and a same-binary control. Input timing includes conversion and IPC serialization; complete expected-value checks happen outside timing. Harness: [frozen benchmark revision](https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae). **This draft has not passed the complete CPU or memory acceptance policy.** Each phase retained 60 pre-existing `Duration(Microsecond)` warmup failures across the two Rust builds ([SPARK-60607](https://issues.apache.org/jira/browse/SPARK-60607)); no samples were missing. Three screen and five confirmation timing metrics have inconclusive guard confidence bounds. Successful-operation RSS guards pass, but no repeated memory-saving criterion passes. These target observations do not establish overall acceptance. ### Does this PR introduce _any_ user-facing change? No intended behavior or API change. ### How was this patch tested? Revalidated 410 Rust unit/golden/sharing checks, all integration-test compilation in default and `wasm-udf` modes, formatting and core Clippy. Full offline Python suite: 288 passed, including 15 new numeric cases. The measured build also passed 38 numeric expectations, 11 nested Python wire comparisons and an unchanged 108-class API snapshot. Workspace Clippy is blocked by the missing `wasm32-unknown-unknown` target in the example build. Full live Spark/official-suite parity remains pending; no local Connect server is listening. No direct allocation-count claim. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
