Yicong-Huang commented on PR #58182: URL: https://github.com/apache/spark/pull/58182#issuecomment-5417459026
The worker-side split isn't about peak memory (the group is materialized in full either way); it's keeping a single output batch under Arrow's 2GB limit so it can transfer at all (SPARK-32294, ARROW-4890). I renamed the config to `spark.sql.execution.pythonUDF.arrow.workerOutputBatchMaxBytes` so it no longer reads as a sibling of `spark.sql.execution.arrow.maxBytesPerOutputBatch`, and the doc now names that one and states the relationship: mine is a best-effort worker-side pre-split; the existing one slices JVM-side with a byte-accurate measure, but only after the batch is read back, so it can't help a batch too large to transfer. They compose, worker split first. I kept the algorithm as-is per your point about not duplicating the JVM slicer, and added a test that runs the suite with both caps small so the composition is exercised. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
