luc-pimentel opened a new pull request, #59326: URL: https://github.com/apache/spark/pull/59326
### What changes were proposed in this pull request? Use a shared pandas-to-Arrow array constructor for DataFrame input and pandas-UDF output. Arrow-array protocol results are extracted first, then explicitly cast with the requested type and safe-conversion setting. Add tests for requested types, nullable/chunked inputs, safe and unsafe narrowing, and string/binary offset widths. Add a PySpark migration note. ### Why are the changes needed? SPARK-60009: on PyArrow 18, protocol-backed pandas columns can ignore the requested type and bypass overflow checks. With safe conversion enabled, converting an Arrow-backed value of 128 to ByteType silently returns -128. A pandas UDF declared as LongType can also fail with ARROW_TYPE_MISMATCH. On PyArrow 19, the implicit protocol cast uses safe conversion even when Spark's setting is disabled. ### Does this PR introduce _any_ user-facing change? Yes. Arrow-backed pandas columns honor requested schema types and spark.sql.execution.pandas.convertToArrowArraySafely. Safe overflow conversions are rejected, and explicitly unsafe conversions are allowed. Null handling, NumPy-backed conversion, and string/binary offset-width behavior are preserved. ### How was this patch tested? - Built Spark with Hive support using build/sbt -Phive package. - Added four tests that fail on the original code with PyArrow 18 and pass with the fix. A fifth test checks that string and binary conversions still work correctly. - All 54 conversion-module tests passed on PyArrow 18.0.0, 19.0.0, and 25.0.1. - Local JVM checks passed for DataFrame inputs, pandas-UDF output, overflow rejection, and explicitly unsafe casts on all three Arrow versions. - Spark's Python compilation, custom-error checks, Ruff checks, and formatting passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenCode 1.18.35 (openai/gpt-6.1-sol) This contribution is my original work, and I license it under the Apache License 2.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
