cloud-fan commented on code in PR #57800:
URL: https://github.com/apache/spark/pull/57800#discussion_r3789672870
##########
python/pyspark/worker.py:
##########
@@ -1923,6 +1933,16 @@ def func(split_index: int, data:
Iterator[pa.RecordBatch]) -> Iterator[pa.Record
# invoke the UDF
output_batches = udf_func(input_batches)
+ # The declared signature is Iterator[...], so a strict iterator is
required. The
+ # legacy flag restores the pre-4.3.0 behavior of accepting any
iterable (e.g. list)
+ # by adapting it into an iterator before the shared element-type
verification.
+ if (
+ runner_conf.map_in_batch_legacy_accept_any_iterable
+ and not isinstance(output_batches, Iterator)
+ and hasattr(output_batches, "__iter__")
Review Comment:
Python's iterable protocol includes the sequence protocol: objects
implementing `__getitem__` can be consumed by `iter(...)` even without
`__iter__`. The current config name and migration note promise acceptance of
any iterable, so please either implement that advertised contract by attempting
`iter(...)` in both branches (with a sequence-protocol regression test), or
narrow the config name and documentation consistently to describe the
historical `__iter__`-only behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
