manuzhang commented on code in PR #6773: URL: https://github.com/apache/datafusion-comet/pull/6773#discussion_r4220981379
########## docs/source/contributor-guide/iceberg-writes.md: ########## @@ -281,9 +282,16 @@ hold in memory. It hands `ParquetWriter` each file's `OutputFile` behind a `Coun writer counts the bytes that leave memory on their way to storage, and a file reports what it has written less those. When they leave depends on the storage (`StorageWrites`). After every batch `run_write_task` resizes the task's reservation to what the open files report plus the rows each -`PartitionFeed` holds, for the dictionary choice or for pacing, and a resize the pool refuses fails -the task. What that figure covers, and what it misses, is described under -[Native writers](memory_management.md#native-writers). +`PartitionFeed` holds, for the dictionary choice or for pacing (`InnerWriter::reserve`). When the +pool refuses a fanout write, `reserve` writes out and closes the partitions holding the most, in +their open file and their feed, until the resize succeeds, and `run_write_task` counts them in the +`files_closed_early` metric. Each partition's files report to a child of the task's +`OpenFileMemory`, which is how `reserve` finds them. That is why the fanout path uses +`FanoutPartitions` rather than iceberg-rust's `FanoutWriter`, which cannot close one partition's +writer. A closed partition's next rows open a new file with the properties its first file used, +which the partition keeps. A write the pool still refuses, with nothing left to close, fails the Review Comment: Can the pool ever refuse write now? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
