Yicong-Huang commented on code in PR #57800:
URL: https://github.com/apache/spark/pull/57800#discussion_r3770461543
##########
python/pyspark/worker.py:
##########
@@ -1923,6 +1933,16 @@ def func(split_index: int, data:
Iterator[pa.RecordBatch]) -> Iterator[pa.Record
# invoke the UDF
output_batches = udf_func(input_batches)
+ # The declared signature is Iterator[...], so a strict iterator is
required. The
+ # legacy flag restores the pre-4.3.0 behavior of accepting any
iterable (e.g. list)
+ # by adapting it into an iterator before the shared element-type
verification.
+ if (
+ runner_conf.map_in_batch_legacy_accept_any_iterable
+ and not isinstance(output_batches, Iterator)
+ and hasattr(output_batches, "__iter__")
Review Comment:
the `__iter__` check was a legacy behavior and we should pin it to what we
have today. the goal is to remove this legacy behavior, I think we should avoid
making it more wide by accepting more types that implement `__getitem__`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
