chaoqin-li1123 commented on code in PR #45977: URL: https://github.com/apache/spark/pull/45977#discussion_r1563370002
########## python/pyspark/sql/worker/plan_data_source_read.py: ########## @@ -51,6 +52,71 @@ ) +def records_to_arrow_batches( + output_iter: Iterator[Tuple], + max_arrow_batch_size: int, + return_type: StructType, + data_source: DataSource, +) -> Iterable[pa.RecordBatch]: + def batched(iterator: Iterator, n: int) -> Iterator: + return iter(functools.partial(lambda it: list(islice(it, n)), iterator), []) Review Comment: This part of code was refactored out of the old plan_data_source_read file, I am not sure whether this is lazy or not. But for streaming we assume that simple python data source is light weight without partitioning and can materialize all records of a microbatch at once, so it doesn't matter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
