wangyinsheng opened a new issue, #19360:
URL: https://github.com/apache/hudi/issues/19360
# [BUG] RemoteHoodieTableFileSystemView triggers repeated server-side
timeline reload whenever any new instant appears after the view is constructed
## Describe the problem
When a client uses `RemoteHoodieTableFileSystemView` and any new instant is
committed on the timeline **after** the view has been constructed (for example
an async operation such as clean, compaction, clustering, or rollback running
in a different actor), every subsequent request from that view triggers a full
server-side `SyncableFileSystemView#sync()` — i.e.
`metaClient.reloadActiveTimeline() + clear() + init()`. The client's cached
timeline never updates and the server's timeline is already up-to-date, so the
mismatch never converges: the re-sync happens on **every single request** for
the entire lifetime of the view instance.
The impact is amplified when a client scans many partitions, e.g. calling
`getLatestFileSlice(partition)` across N partitions. Each partition results in
one HTTP request, and each request triggers one full timeline reload on the
server. N partitions → N wasted syncs.
## Root cause
Two pieces of code combine to produce the pathology:
**(1) The client's local timeline is a constructor-time snapshot and is
never refreshed.**
`RemoteHoodieTableFileSystemView` assigns `this.timeline` exactly once in
the constructor:
```java
if (metaClient.getTableConfig().isLSMBasedLogFormat()) {
this.timeline =
metaClient.getActiveTimeline().filterCompletedAndReplaceInstants();
} else {
this.timeline =
metaClient.getActiveTimeline().filterCompletedAndCompactionInstants();
}
```
There is no code path that refreshes `this.timeline` for the lifetime of the
view instance. `executeRequest` then sends `LAST_INSTANT_TS` and
`TIMELINE_HASH` derived from that frozen snapshot on every request. As soon as
any instant is added to the timeline out-of-band after the view is constructed,
the client's hash is permanently stale for that view instance.
**(2) The server treats any hash mismatch as "needs sync".**
`RequestHandler#isLocalViewBehind`:
```java
private boolean isLocalViewBehind(Context ctx) {
...
String localTimelineHash = localTimeline.getTimelineHash();
// refresh if timeline hash mismatches
if (!localTimelineHash.equals(timelineHashFromClient)) {
return true; // ← ANY mismatch is treated as "server behind"
}
// As a safety check, even if hash is same, ensure instant is present
return
!localTimeline.containsOrBeforeTimelineStarts(lastKnownInstantFromClient);
}
```
Judging by the method name, the check is supposed to answer "is the server's
view behind what the client saw?". But the implementation collapses two very
different situations into the same `true` return:
- **server behind client** → `sync()` is legitimate
- **client behind server** → `sync()` is unnecessary; the server is already
newer
The follow-up `containsOrBeforeTimelineStarts(lastKnownInstantFromClient)`
check would actually cover the "client behind server" case correctly (the
client's last instant is already present in the server timeline, so nothing
needs to be re-synced), but it is short-circuited by the earlier `return true`
on hash mismatch.
Combined effect: (1) makes the client's hash permanently stale after any
out-of-band instant, and (2) turns that stale hash into a per-request full
timeline reload on the server.
## Concrete evidence from production logs
In our environment the trigger was an async clean instant being committed
after the client's view was constructed, but any new instant produces the same
behavior.
Timeline files on HDFS at the time of the incident (last two instants):
```
20260721213743765.deltacommit (completed — this is what the client
saw)
20260721213743765.deltacommit.inflight
20260721213743765.deltacommit.requested
20260721213744648.clean (completed — appeared AFTER the
client opened its view)
20260721213744648.clean.inflight
20260721213744648.clean.requested
```
A Spark reader then queries many partitions in parallel. The client's
`TIMELINE_HASH` is stable (`6bbdf6f1...162bf`), the client's `LAST_INSTANT_TS`
is stable (`20260721213743765`, the deltacommit — the client has not observed
the newer instant), and the server's view already reflects the newer state.
Every single request triggers a "Syncing view..." log on the server:
```
2026-07-21 21:42:47.413 INFO [ForkJoinPool.commonP:l-worker-3]
o.a.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending request :
(http://xxx/v1/hoodie/view/slices/partition/latest/?partition=dt%3D2017-02-03&basepath=hdfs%3A%2F%2Fxxxxx&lastinstantts=20260721213743765&timelinehash=6bbdf6f1ea5484010b2275be7e8f1fe993033321bbe8cff9675044b654f162bf)
2026-07-21 21:42:47.575 INFO [qtp2089656019-535557:19-5355571]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20260721213744648__clean__COMPLETED__20260721213758517]}
2026-07-21 21:42:47.587 INFO [qtp2089656019-535557:19-5355571]
org.apache.hudi.common.table.HoodieTableMetaClient - Skip backtrack instance :
[]
2026-07-21 21:42:47.587 INFO [qtp2089656019-535557:19-5355571]
o.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 6 ms to read
0 instants, 0 replaced file groups
2026-07-21 21:42:47.612 INFO [qtp2089656019-535557:19-5355571]
org.apache.hudi.common.util.ClusteringUtils - Found 0 files in pending
clustering operations
2026-07-21 21:42:47.612 INFO [qtp2089656019-535557:19-5355571]
o.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file
system view for partition (dt=2014-06-10)
2026-07-21 21:42:47.617 INFO [qtp2089656019-535572:19-5355722]
org.apache.hudi.timeline.service.RequestHandler - Syncing view as client passed
last known instant 20260721213743765 as last known instant but server has the
following last instant on timeline
:Option{val=[20260721213744648__clean__COMPLETED__20260721213758517]}
2026-07-21 21:42:47.626 INFO [qtp2089656019-535557:19-5355571]
o.apache.hudi.common.table.view.AbstractTableFileSystemView - #files found in
partition (dt=2014-06-10) =11, Time taken =14
2026-07-21 21:42:47.627 INFO [qtp2089656019-535557:19-5355571]
o.apache.hudi.common.table.view.AbstractTableFileSystemView - addFilesToView:
NumFiles=11, NumFileGroups=10, FileGroupsCreationTime=1, StoreTimeTaken=0
2026-07-21 21:42:47.627 INFO [qtp2089656019-535557:19-5355571]
o.apache.hudi.common.table.view.AbstractTableFileSystemView - Time to load
partition (dt=2014-06-10) =15
2026-07-21 21:42:47.638 INFO [ForkJoinPool.commonP:l-worker-2]
o.a.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending request :
(http://xxx/v1/hoodie/view/slices/partition/latest/?partition=dt%3D2014-06-11&basepath=hdfs%3A%2F%2Fxxxxx&lastinstantts=20260721213743765&timelinehash=6bbdf6f1ea5484010b2275be7e8f1fe993033321bbe8cff9675044b654f162bf)
2026-07-21 21:42:47.789 INFO [qtp2089656019-535572:19-5355722]
org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants
upto : Option{val=[20260721213744648__clean__COMPLETED__20260721213758517]}
2026-07-21 21:42:47.804 INFO [qtp2089656019-535572:19-5355722]
org.apache.hudi.common.table.HoodieTableMetaClient - Skip backtrack instance :
[]
2026-07-21 21:42:47.804 INFO [qtp2089656019-535572:19-5355722]
o.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 8 ms to read
0 instants, 0 replaced file groups
2026-07-21 21:42:47.841 INFO [qtp2089656019-535572:19-5355722]
org.apache.hudi.common.util.ClusteringUtils - Found 0 files in pending
clustering operations
2026-07-21 21:42:47.841 INFO [qtp2089656019-535572:19-5355722]
o.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file
system view for partition (dt=2021-05-19)
2026-07-21 21:42:47.846 INFO [qtp2089656019-535586:19-5355867]
org.apache.hudi.timeline.service.RequestHandler - Syncing view as client passed
last known instant 20260721213743765 as last known instant but server has the
following last instant on timeline
:Option{val=[20260721213744648__clean__COMPLETED__20260721213758517]}
2026-07-21 21:42:47.855 INFO [qtp2089656019-535572:19-5355722]
o.apache.hudi.common.table.view.AbstractTableFileSystemView - #files found in
partition (dt=2021-05-19) =11, Time taken =14
2026-07-21 21:42:47.855 INFO [qtp2089656019-535572:19-5355722]
o.apache.hudi.common.table.view.AbstractTableFileSystemView - addFilesToView:
NumFiles=11, NumFileGroups=10, FileGroupsCreationTime=0, StoreTimeTaken=0
2026-07-21 21:42:47.855 INFO [qtp2089656019-535572:19-5355722]
o.apache.hudi.common.table.view.AbstractTableFileSystemView - Time to load
partition (dt=2021-05-19) =14
```
## Steps to reproduce
1. Have a Hudi 0.13.1 table (COW or MOR) served by an embedded/standalone
timeline server.
2. Open a long-lived `RemoteHoodieTableFileSystemView` from a client (Spark
reader, or a Flink component that reads via the remote view).
3. From a different actor (e.g. Flink's async cleaner, an inline
clean/compaction, `hudi-cli`, or any writer commit) let a new instant become
the newest one on the timeline **after** step 2.
4. From the client opened in step 2, call any partition-level API
repeatedly, ideally across many partitions in one query (e.g.
`getLatestFileSlice(partition, fileId)` for N partitions).
5. Enable INFO logs on `RequestHandler` and `AbstractTableFileSystemView`.
You will see:
- the client's `TIMELINE_HASH` in the request URL stays constant across
all requests, and
- the server logs a `Syncing view ...` line and a `HoodieActiveTimeline:
Loaded instants upto ...` reload on **every** request.
## Environment Description
- **Hudi version**: 0.13.1
- **Affected engines**: both Spark and Flink (both use
`RemoteHoodieTableFileSystemView`)
- **Storage**: HDFS
- **Running on Docker?**: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
