ravjot28 opened a new issue, #20579:
URL: https://github.com/apache/hudi/issues/20579
### Bug Description
**What happened:**
With `hoodie.filesystem.view.incr.timeline.sync.enable=true`, a pending
compaction that moves from `REQUESTED` to
`INFLIGHT` makes the next `sync()` of the file system view throw and fall
back to a complete rebuild:
```
ERROR IncrementalTimelineSyncFileSystemView - Got exception trying to
perform incremental sync. Reverting to complete sync
java.lang.IllegalArgumentException: Duplicate FileGroupId found in pending
compaction operations. FgId :HoodieFileGroupId{partitionPath='2018/01/01',
fileId='...'}
at
org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java)
at
org.apache.hudi.common.table.view.HoodieTableFileSystemView.addPendingCompactionOperations(HoodieTableFileSystemView.java)
at
org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView.addPendingCompactionInstant(IncrementalTimelineSyncFileSystemView.java)
```
For log compaction it is worse: while a log compaction is pending (requested
or inflight), every sync, including one
triggered by an unrelated delta commit, sees it as new and rebuilds the view
("Duplicate FileGroupId found in pending
log compaction operations"). #15480 shows this in the
`TestRocksDBBasedIncrementalFSViewSync` logs.
A complete rebuild drops every loaded partition, so the next requests list
them again. On a merge-on-read table with
frequent compaction scheduling and many loaded partitions, the timeline
server keeps rebuilding its views.
**What you expected:**
A pending compaction or log compaction that only changed state keeps the
operations the view already holds, and the
sync stays incremental; the view after the sync equals a freshly built view.
**Steps to reproduce:**
1. Merge-on-read table, one partition, a delta commit `001` writing log
files for two file groups.
2. Build a `HoodieTableFileSystemView` with incremental timeline sync and
load the partition.
3. Schedule a compaction `002` over the two file slices, `view.sync()`:
incremental, two pending operations.
4. Transition `002` from requested to inflight, `view.sync()`: the sync
throws internally and the view is rebuilt.
For log compaction: schedule a log compaction, sync, then add a delta commit
to another partition and sync again.
**Root cause (master d5230b69e22):**
-
`hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:84`:
the new instants are
those of the new timeline not contained in the old one, and
`HoodieInstant.equals`
(`hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstant.java:86`)
compares the state, so the
inflight compaction is not equal to the requested one the view held.
- `TimelineDiffHelper.java:52,64`: the old side comes from
`filterCompletedAndCompactionInstants()`, which keeps pending
compactions but drops pending log compactions, while line 84 iterates the
unfiltered new timeline. A pending log
compaction is therefore new on every sync.
- `TimelineDiffHelper.java:69,86` already compute the transitions of the old
pending compactions and log compactions,
but use them only for the finished or removed lists.
-
`hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieTableFileSystemView.java:224-226`
(and the log
compaction and RocksDB counterparts) refuse a file group that is already
pending; the exception is caught in
`IncrementalTimelineSyncFileSystemView.java:110-115`, which clears and
rebuilds the view.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1
**Query engine:** any engine that serves a writer through the embedded (or a
standalone) timeline server; merge-on-read
tables with compaction or log compaction
**Relevant configs:**
`hoodie.filesystem.view.incr.timeline.sync.enable=true` (default `false`)
### Logs and Stack Trace
See "What happened" above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]