BryanCutler commented on issue #24650: [SPARK-27778][PYTHON] Fix toPandas conversion of empty DataFrame with Arrow enabled URL: https://github.com/apache/spark/pull/24650#issuecomment-494245140 I had another thought about this, the stuff in `doAfterLastPartition` could be removed from `handlePartitionBatches` and called after `runJob` regardless if partitions are empty or not. This really wouldn't make any difference performance-wise because it's just moving it outside the callback function and Python is waiting on it anyway. It also has the benefit that the number of partitions would not have to be kept track of, so the variables `partitionCount` and `numPartitions` could be removed. It would be a lot clearer then too. What do you think @dvogelbacher and @HyukjinKwon ?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
