gudladona opened a new issue, #20657:
URL: https://github.com/apache/hudi/issues/20657

   ### Bug Description
   
   **What happened:**
   
   With the row writer (`hoodie.datasource.write.row.writer.enable` / 
`hoodie.streamer.write.row.writer.enable`), a write task attempt that Spark 
does not use can upload Parquet files after the commit has completed. They are 
not in the commit's write stats, but a storage listing (no metadata table) 
counts them as data of the completed instant, so readers see duplicate rows and 
later table services pick them up. In our case clustering replaced one of them, 
so the duplicates moved into the clustered files.
   
   The write loop in `HoodieDatasetBulkInsertHelper` (master `c32b64e98a3a`, 
lines 213-221):
   
   ```scala
   try {
     iter.foreach(writer.write)
   } catch {
     case t: Throwable =>
       writer.abort()
       throw t
   } finally {
     writer.close()
   }
   ```
   
   `BulkInsertDataInternalWriterHelper.abort()` is empty (line 192), so on the 
failure path `finally` closes every open `HoodieRowCreateHandle`. That finishes 
the Parquet file and, on S3A, uploads it. A task Spark kills therefore still 
publishes its partial files.
   
   This is reachable without speculation. After a fetch failure Spark resubmits 
the stage, and the earlier attempt's tasks that are still running keep going. 
When the job finishes, `DAGScheduler` cancels them with reason `Stage 
finished`, without interrupting the thread unless the job sets 
`spark.job.interruptOnCancel`, so a task only notices at its next 
`InterruptibleIterator` check. By then the commit has run 
`reconcileAgainstMarkers`. Its list of paths comes from markers, so it does 
include the zombie's open file, but the file is not on storage yet: 
`deleteFile` returns false, `exists` is false, and nothing is deleted. The 
zombie then lands in `catch`, and `finally { writer.close() }` uploads the file.
   
   The row writer gives each attempt a random file ID prefix 
(`BulkInsertDataInternalWriterHelper`, `fileIdPrefix = UUID.randomUUID()`), so 
the zombie's file is a separate file group with the commit's instant time: 
**duplicate rows**, not a shadowed copy. The non-row-writer insert path assigns 
file IDs on the driver, so a zombie there writes into the winner's file group.
   
   A second gap, independent of the kill path: the row writer writes straight 
to final paths and never asks the driver's `OutputCommitCoordinator` whether 
its attempt may commit, as Spark's own writers do 
(`SparkHadoopMapRedUtil.commitTask`). An attempt that finishes after another 
attempt was used, or after its stage completed, keeps its files.
   
   **What you expected:**
   
   An attempt that is killed, fails, or is not the one the job used leaves no 
data files: `abort()` deletes what the attempt wrote, and a finished attempt 
checks with the driver before returning its write statuses.
   
   **Steps to reproduce:**
   
   1. Row-writer bulk insert into a COPY_ON_WRITE table, metadata table 
disabled, on S3.
   2. During the write stage, lose an executor holding map output (decommission 
or pod deletion), so some write tasks fail with `FetchFailed` and the stage is 
resubmitted.
   3. A write task of the first stage attempt that did not need the lost blocks 
keeps running. The job finishes with the resubmitted attempt, the commit 
completes, and the old attempt is cancelled with `TaskKilled (Stage finished)`.
   4. List the partition: the old attempt's file (write token 
`<partition>-<stage>-<its task id>`) is on storage, written after the commit's 
completion time, and is not in the commit's write stats. Reading the partition 
returns its rows twice.
   
   Driver log from one occurrence (Spark 3.5.5, Hudi 1.1.1 based):
   
   ```
   12:36:25 WARN TaskSetManager: Lost task 427.0 in stage 659.0 (TID 150911) 
...: FetchFailed(BlockManagerId(34, ...), shuffleId=164, ...)
      ... 16 write tasks of stage 659.0 fail the same way; stage 659 is 
resubmitted
   12:36:50 INFO HoodieTable: Removing duplicate files created due to task 
retries before committing. Paths=[..., 
<partition>/e02198ad-...-1_406-659-150895_<instant>.parquet, ...]
   12:36:51 commit completes
   12:38:32 WARN TaskSetManager: Lost task 406.0 in stage 659.0 (TID 150895) 
...: TaskKilled (Stage finished)
   12:38:33 the file e02198ad-...-1_406-659-150895_<instant>.parquet appears on 
S3
   ```
   
   ### Proposed fix
   
   Small, in the row writer plus one hook in `HoodieWrapperFileSystem`:
   
   1. `abort()` deletes every file the attempt wrote, closed or open; `close()` 
after `abort()` does nothing. The write loop calls `close()` only on success, 
not in `finally`.
   2. An open file is aborted, not closed, where the storage allows it. On S3A 
closing is the upload, so close-then-delete would still publish the whole file 
first and leave it visible if the executor dies, or a listing runs, between the 
two. `HoodieWrapperFileSystem` finds the open stream, unwraps it to the 
storage's own stream, and calls Hadoop's `Abortable.abort()` (3.3.1+) when the 
stream reports `fs.capability.outputstream.abortable`, so the object is never 
published; the Parquet writer is then closed only to release its buffers. 
Streams that cannot be aborted (local, HDFS) are closed and deleted.
   3. A file the abort could not delete is logged at ERROR as an orphan risk 
and attached as a suppressed exception to the failure that caused the abort. A 
metric cannot carry it: Spark drops accumulator updates from failed and killed 
tasks.
   4. After a successful write loop, ask the driver's `OutputCommitCoordinator` 
whether the attempt may commit (once per task attempt: the coordinator refuses 
a second request even from the same attempt). If refused, or if the driver 
cannot be reached, throw `CommitDeniedException` and abort.
   
   I have this with tests (an abort leaves no files, sorted and unsorted; an 
abortable stream is aborted, not closed, and a non-abortable one is closed and 
deleted; an attempt the driver does not authorize leaves no files) and can open 
a PR against master.
   
   Related, broader: #18395 (stage files in the marker directory and promote at 
finalize), #18273 (same trigger, GCS delete/write race), PR #9922 (task-side 
cleanup of partial files on abort, which changes 
`BulkInsertDataInternalWriterHelper` but not the `finally { writer.close() }` 
in `HoodieDatasetBulkInsertHelper`).
   
   ### Environment
   
   **Hudi version:** 1.1.1-based; the same code is on master (`c32b64e98a3a`)
   **Query engine:** Spark 3.5.5 on Kubernetes, S3 (S3A)
   **Relevant configs:** `hoodie.datasource.write.operation=bulk_insert`, row 
writer enabled, `hoodie.metadata.enable=false`, `spark.speculation=false` 
(default)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to