ravjot28 opened a new issue, #20580:
URL: https://github.com/apache/hudi/issues/20580
### Bug Description
**What happened:**
A table refresh on the timeline server (`REFRESH_TABLE`, sent when a client
calls `sync()` or `reset()` on its
`RemoteHoodieTableFileSystemView`, or `TimelineService#unregisterBasePath`)
removes the table's view from
`FileSystemViewManager` and closes it. A data request for the same table
that runs at that moment, depending on where
the refresh lands:
- syncs the view the refresh just closed: `NullPointerException` on the
cleared meta client, HTTP 500
(`HoodieRemoteException: status code: 500`) on the client; a client with a
secondary local view falls back to it,
one without fails its operation;
- reads the closed view in its handler: HTTP 500;
- answers from another view instance than the one it checked and synced.
`LAST_INSTANT`, `TIMELINE`, `LOAD_PARTITIONS` and `LOAD_ALL_PARTITIONS` read
the view with no coordination with a
refresh: the two load endpoints fail with "View is already closed" (HTTP
500), the two timeline endpoints answer from
the timeline the closed view kept.
**What you expected:**
The request is answered with the table's current files from one view, and no
request reads a view after it was closed.
A refresh does not close a view under a request that uses it, and later
requests use the replacement view.
**Steps to reproduce:**
1. Start a timeline server whose view manager uses an in-memory or spillable
view with incremental sync.
2. Commit to the table so that a client's timeline is ahead of the server's
view.
3. Send a data request (for example `getLatestBaseFiles(partition)` through
`RemoteHoodieTableFileSystemView`) and, while
the server works on it, send `REFRESH_TABLE` for the same base path from
another client.
The window is small; a test with latches that pause the server-side view at
a chosen read makes it deterministic.
**Root cause (master d5230b69e22):**
-
`hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java`:
`ViewHandler#handle`
resolves the table's view from the manager separately for each step: the
freshness check `isLocalViewBehind`
(`:737`, lookup at `:742`), `syncIfLocalViewBehind` (`:766`, which
synchronizes on the view it looked up at `:768` and
calls `view.sync()` at `:782`), the handler (`:686`; the handlers call
`viewManager.getFileSystemView(basePath)` once
more) and the final check (`:692`).
-
`hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewManager.java:88-91`:
`clearFileSystemView` removes the view and closes it without any
coordination with requests in flight; `close()`
clears the partitions and nulls the meta client and timeline
(`hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:313-330`).
- The four endpoints above are registered without the refresh check
(`RequestHandler.java:279`, `:533`) and their
handlers look the view up on their own.
Related: open PR #20239 changes `syncIfLocalViewBehind` as well; #19946
added the trailing-clean shortcut.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1, 0.14.x
**Query engine:** any engine using the timeline server (embedded in a write
client, or standalone) with several clients
on one table
**Relevant configs:** `hoodie.embed.timeline.server=true` (default) or a
standalone timeline server; in-memory or
spillable server-side view
### Logs and Stack Trace
Client: `HoodieRemoteException` with HTTP status code 500. Server: a
`NullPointerException` from
`IncrementalTimelineSyncFileSystemView#sync` called by
`RequestHandler#syncIfLocalViewBehind` on the closed view, or
"View is already closed" from the load endpoints.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]