hvph-uyen opened a new pull request, #55561:
URL: https://github.com/apache/spark/pull/55561
### What changes were proposed in this pull request?
This PR updates pandas-on-Spark `concat` so that native pandas `DataFrame`
and `Series` objects are accepted when they are passed inside the input
iterable.
It also fixes the error message for unsupported inputs so that it reports
the actual invalid element type instead of the outer container type.
The existing behavior for bare inputs such as `ps.concat(pdf)` and
`ps.concat(pser)` is preserved.
### Why are the changes needed?
Currently, `ps.concat` rejects native pandas objects even when they can be
converted in the iterable case.
Also, the current error message is misleading because it can report `list`
instead of the actual invalid input type.
### Does this PR introduce _any_ user-facing change?
Yes.
Before this change, `ps.concat([pdf, pdf])` rejected native pandas inputs,
and unsupported input errors could report `list` instead of the actual invalid
element type.
After this change, native pandas `DataFrame` and `Series` inputs are
accepted in the iterable case, and unsupported input errors report the actual
invalid element type.
### How was this patch tested?
- Added regression coverage in `pyspark.pandas.tests.test_namespace`.
- Ran:
`python/run-tests --python-executables python3 --testnames
pyspark.pandas.tests.test_namespace`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Generative AI tooling was used to help inspect the issue, understand the
relevant Spark codebase. The final patch was manually reviewed and tested
before submission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
