andygrove opened a new issue, #6771: URL: https://github.com/apache/datafusion-comet/issues/6771
A native fanout write keeps a data file open for every partition a task writes to, and since #6247 its buffers count against the task's memory pool. The writer cannot spill, so a task that writes to enough partitions fails with `Additional allocation failed for IcebergWriteExec`, where iceberg-java's fanout writer, buffering on the JVM heap, succeeds. Spark's retry takes the same path, so the job fails. On Iceberg 1.5+ an unsorted partitioned table uses the fanout writer by default, and #6664 makes native writes the default, so this has to be fixed first. Raised in review of #6664. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
