yihua opened a new pull request, #19940:
URL: https://github.com/apache/hudi/pull/19940
### Describe the issue this Pull Request addresses
With `hoodie.errortable.write.unification.enabled=true`, `StreamSync`
commits the error table and afterwards sums the error-table write statuses for
the record counts. That ordering duplicates every error record.
The error-table write-status RDD is kept alive only by a Spark cache. The
bulk insert persists it in `BaseSparkCommitActionExecutor.updateIndex` under
`HoodieDataCacheKey.of(basePath, instantTime)`, and the error table's own
`writeClient.commit` releases it through `releaseResources` ->
`SparkReleaseResources.releaseCachedData`. A real error-table writer typically
also drops its cache of the upstream error events inside `commit`. So by the
time the count runs, nothing backs the RDD, the `aggregate` re-evaluates its
lineage, and the executors run the bulk insert a second time under an instant
that has already completed. The re-run allocates fresh file ids, so the
duplicates land in new file groups stamped with that instant and a snapshot
read returns two rows per error record.
The counts themselves stay correct, which is why this is invisible from the
outside: the recomputed statuses still describe one write's worth of records.
This arrived with #18750. That PR's first cut counted before committing the
error table. A later revision hoisted the error-table commit ahead of the
validators so error records would survive a validator abort, and the count
moved behind it as a consequence. In the same round the data-table RDD was
given a cache and a driver collect while the error-table RDD was left as a lazy
`aggregate`, so the asymmetry went unnoticed.
### Summary and Changelog
- `StreamSync.writeToSinkAndDoMetaSync` counts records before committing the
error table, restoring the ordering the pre-#18750
`HoodieStreamerWriteStatusValidator` used. The error-table commit still runs
ahead of the validators, so the property the reorder was introduced for is
preserved. Steps renumbered accordingly.
- `ErrorTableCommitter` and `SuccessfulRecordCounter` are unchanged. The
error-table statuses are still summed on the executors with a single
`aggregate`; nothing is materialized on the driver.
- Tests: `RddBackedErrorTableWriter` is an error-table writer backed by a
real `SparkRDDWriteClient` over a real Hudi table, whose `commit` releases the
write statuses it is given and the upstream error events. The only
`BaseErrorTableWriter` implementation in the tree is a stub that stores the RDD
and releases nothing, so no existing test could observe this.
`TestErrorTableWriteOnce` asserts the record keys the error table actually
holds: one case on the fixed ordering, one pinning the previous ordering that
wrote every record twice.
#### Ordering, for review
The whole change is the position of the counting block.
`ErrorTableCommitter` and `SuccessfulRecordCounter` are byte-identical to the
base branch.
Before, in `writeToSinkAndDoMetaSync`:
```
writeStatusRDD.collect() data-table write runs
1 ErrorTableCommitter.commit(...) error table commits, its
write-status cache is released
2 runValidators(...)
3 SuccessfulRecordCounter.compute(...) error-table statuses read HERE,
after the release <-- bug
4 write-error gate
5 writeClient.commit(...)
```
After:
```
writeStatusRDD.collect() data-table write runs
1 SuccessfulRecordCounter.compute(...) error-table statuses read while
still cached
2 ErrorTableCommitter.commit(...) error table commits, cache
released, nothing reads it again
3 runValidators(...)
4 write-error gate
5 writeClient.commit(...)
```
This restores the order `HoodieStreamerWriteStatusValidator` used before
#18750, where the same two operations ran inside the `writeClient.commit`
callback rather than ahead of it. On `release-1.2.0` that is
`StreamSync.HoodieStreamerWriteStatusValidator.validate`: the error-table sum,
then `errorTableWriter.commit(...)`, then the write-error gate, and only then
does `SparkRDDWriteClient.commit` proceed to `commitStats`.
| | before #18750 | this PR |
| ---| ---| --- |
| error-table statuses summed with one distributed `aggregate` | yes | yes |
| summed on executors, not collected to the driver | yes | yes |
| sum happens before the error-table commit | yes | yes |
| error-table commit before the write-error gate | yes | yes |
| data-table commit last | yes | yes |
Step 2 stays ahead of the validators, so the property #18750 introduced the
reorder for (error records survive a validator-driven abort) is unchanged.
Two things that are deliberately *not* restored here, both introduced by
#18750 and out of scope: the sequence runs before `writeClient.commit` rather
than inside its callback, so failures roll the inflight instant back
explicitly; and the data-table statuses are collected in `StreamSync` as full
`WriteStatus` objects rather than inside the write client as `SlimWriteStats`.
The error table is a separate table with its own write client, instant and
`HoodieDataCacheKey`, so its commit releases only its own write statuses. The
data-table RDD is unaffected either way, which is why only error-table rows
were ever duplicated.
### Impact
Error-table writes land once under write unification. No config, API or
public behaviour change.
### Risk Level
low. The change is a move of the counting block within one method; no helper
signatures change. Red-then-green verified: with the count after the commit the
new test fails with each of the 20 expected record keys appearing twice, and
passes with the count before it.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]