Spenserrrr commented on code in PR #58903:
URL: https://github.com/apache/spark/pull/58903#discussion_r4066692356
##########
python/pyspark/sql/conversion.py:
##########
@@ -180,6 +180,115 @@ def select_columns(cls, batch: "pa.RecordBatch",
column_indices: list[int]) -> "
[batch.schema.names[i] for i in column_indices],
)
+ @staticmethod
+ def concat_batches(batches: Sequence["pa.RecordBatch"]) ->
"pa.RecordBatch":
+ """Concatenate same-schema RecordBatches by row.
+
+ A single batch is returned unchanged. PyArrow before 19.0.0 has no
``concat_batches``;
+ the fallback concatenates the equivalent StructArrays and converts the
result back to a
+ RecordBatch. Element-wise iterator UDFs use this when one input
batch's flattened result
Review Comment:
Makes sense. I removed the element-wise iterator detial and kept the
docstring focused on concatenation and the fallback for older PyArrow versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
