unikdahal opened a new issue, #6581: URL: https://github.com/apache/datafusion-comet/issues/6581
### What is the problem the feature request solves? As part of #5649, Comet's native Iceberg write path needs failure cleanup semantics that are safe across the entire native → JVM lifecycle. #5618 added task-attempt cleanup for native Iceberg writes, but there is still a small ownership gap during the handoff. Today, native `AbortOnDrop` is disarmed once the output batch has been constructed and yielded. The JVM only takes cleanup ownership after it reads that batch, decodes the written-file locations, and registers them with `WrittenFileCleanup`. A failure between those two points can therefore leave an uncommitted data file without an active cleanup owner. This is also relevant to #6240. Native merge-on-read writes will extend the executor-side writer to produce both data files and position-delete files, so the write lifecycle needs a reliable cleanup ownership contract before the MOR path builds on top of it. ### Describe the potential solution Keep native cleanup armed until the JVM explicitly completes the handoff: 1. Native writes and tracks every generated file location. 2. Native yields the output batch while keeping `AbortOnDrop` armed. 3. The JVM reads the locations and registers them with `WrittenFileCleanup`. 4. The JVM performs the final EOF poll on the native stream. 5. That EOF poll acts as the acknowledgement that ownership has transferred, allowing native cleanup to disarm. There is intentionally a short period where native and JVM cleanup overlap. Both paths are best-effort, so attempting to delete the same uncommitted file twice is safe and preferable to an ownership gap. ### Additional context ### Related - #5649 — **[EPIC] Production-quality native Iceberg writes** - #6240 — **Support Native Iceberg MOR write** - #5646 — **Failure-injection tests for native Iceberg writes** - #5618 — **Native Iceberg write: clean up task-attempt data files on task failure** — this is a follow-up for the remaining handoff window - #5277 — **Remove orphaned files in Comet Iceberg split writers** - #5122 — **Accelerate row-level MERGE / UPDATE / DELETE plans** - #5318 — **Native `MergeRowsExec` support** For the MOR work specifically, #5122 / #5318 cover the row-level execution side while #6240 covers the native `WriteDelta` path. This issue hardens the underlying native writer lifecycle that those paths will rely on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
