Spenserrrr opened a new pull request, #57996: URL: https://github.com/apache/spark/pull/57996
### What changes were proposed in this pull request? This PR adds golden-file drift tests for `pa.Array.from_pandas(series, type=..., safe=...)`, the non-default `type=` / `safe=` arguments PySpark passes when converting pandas to Arrow (`sql/conversion.py`, `sql/pandas/conversion.py`). It is a sub-task of the SPARK-54936 upstream-library-monitoring umbrella and a sibling of SPARK-58657 (which covered the `mask` argument). A new test class `PyArrowArrayFromPandasTypeTests` is added to the existing `test_pyarrow_array_from_pandas_non_default.py`, reusing the shared `_PyArrowFromPandasTestBase` introduced by the mask PR. Following the `pa.Array.cast` tests, `safe=True` and `safe=False` are two separate test methods with two golden files (`..._type_safe` and `..._type_unsafe`) rather than doubling the columns of an already-2D matrix. Each golden is a `source \ target` matrix over a curated set of source Series and the scalar Arrow target types that `to_arrow_type` can request. The source rows and target columns are deliberately curated rather than a dense product. Spark uses `type=` diagonally (the source dtype and the requested Arrow type both come from the same schema), so the tests pin the cells that make `type=`/`safe=` observable — range/precision flips, silent losses, the numpy-vs-protocol `safe=` drop, and SPARK-46776 — while leaving general source-to-target conversion to the `pa.Array.cast` tests. Off-diagonal cells such as `int -> large_binary` are pyarrow-construction behavior Spark never triggers and are intentionally excluded to keep the goldens stable across versions. This PR also adds one `large_binary[pyarrow]` source row to the shared base (`_protocol_sources`). It is the binary counterpart of the existing `string[pyarrow]` row and exercises the binary half of the SPARK-46776 followup; it appears as one extra row in the existing default and mask goldens, which are regenerated accordingly. There is no production code change — this PR adds tests and golden files only. ### Why are the changes needed? PySpark relies on `pa.Array.from_pandas(..., type=..., safe=...)` to coerce pandas data into the Arrow type dictated by a Spark schema, in `createDataFrame(pandas_df)` and in every pandas UDF return value. These tests run without a Spark session and act as a drift canary: if a PyArrow or pandas upgrade changes how a value is coerced — a `safe=` flip, a silent truncation, or the pyarrow < 19 protocol behavior tracked by SPARK-46776 — a golden cell changes and the test fails loudly, instead of PySpark silently producing wrong data downstream. ### Does this PR introduce _any_ user-facing change? No. This PR only adds tests and golden files. ### How was this patch tested? The new tests are golden-file comparisons that run without a Spark session. They were validated across a PyArrow 18-25 x pandas 2/3 version matrix (16 combinations), all passing. The only version-conditional expectation is the pyarrow < 19 SPARK-46776 followup, where the `__arrow_array__` protocol path ignores the requested `type=` and hands back the source's stored large_ type; this is encoded as a version-gated override so the same golden files serve every supported version. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
