ravjot28 opened a new issue, #20580:
URL: https://github.com/apache/hudi/issues/20580

   ### Bug Description
   
   **What happened:**
   
   A table refresh on the timeline server (`REFRESH_TABLE`, sent when a client 
calls `sync()` or `reset()` on its
   `RemoteHoodieTableFileSystemView`, or `TimelineService#unregisterBasePath`) 
removes the table's view from
   `FileSystemViewManager` and closes it. A data request for the same table 
that runs at that moment, depending on where
   the refresh lands:
   
   - syncs the view the refresh just closed: `NullPointerException` on the 
cleared meta client, HTTP 500
     (`HoodieRemoteException: status code: 500`) on the client; a client with a 
secondary local view falls back to it,
     one without fails its operation;
   - reads the closed view in its handler: HTTP 500;
   - answers from another view instance than the one it checked and synced.
   
   `LAST_INSTANT`, `TIMELINE`, `LOAD_PARTITIONS` and `LOAD_ALL_PARTITIONS` read 
the view with no coordination with a
   refresh: the two load endpoints fail with "View is already closed" (HTTP 
500), the two timeline endpoints answer from
   the timeline the closed view kept.
   
   **What you expected:**
   
   The request is answered with the table's current files from one view, and no 
request reads a view after it was closed.
   A refresh does not close a view under a request that uses it, and later 
requests use the replacement view.
   
   **Steps to reproduce:**
   
   1. Start a timeline server whose view manager uses an in-memory or spillable 
view with incremental sync.
   2. Commit to the table so that a client's timeline is ahead of the server's 
view.
   3. Send a data request (for example `getLatestBaseFiles(partition)` through 
`RemoteHoodieTableFileSystemView`) and, while
      the server works on it, send `REFRESH_TABLE` for the same base path from 
another client.
   
   The window is small; a test with latches that pause the server-side view at 
a chosen read makes it deterministic.
   
   **Root cause (master d5230b69e22):**
   
   - 
`hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java`:
 `ViewHandler#handle`
     resolves the table's view from the manager separately for each step: the 
freshness check `isLocalViewBehind`
     (`:737`, lookup at `:742`), `syncIfLocalViewBehind` (`:766`, which 
synchronizes on the view it looked up at `:768` and
     calls `view.sync()` at `:782`), the handler (`:686`; the handlers call 
`viewManager.getFileSystemView(basePath)` once
     more) and the final check (`:692`).
   - 
`hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewManager.java:88-91`:
     `clearFileSystemView` removes the view and closes it without any 
coordination with requests in flight; `close()`
     clears the partitions and nulls the meta client and timeline
     
(`hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:313-330`).
   - The four endpoints above are registered without the refresh check 
(`RequestHandler.java:279`, `:533`) and their
     handlers look the view up on their own.
   
   Related: open PR #20239 changes `syncIfLocalViewBehind` as well; #19946 
added the trailing-clean shortcut.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1, 0.14.x
   **Query engine:** any engine using the timeline server (embedded in a write 
client, or standalone) with several clients
   on one table
   **Relevant configs:** `hoodie.embed.timeline.server=true` (default) or a 
standalone timeline server; in-memory or
   spillable server-side view
   
   ### Logs and Stack Trace
   
   Client: `HoodieRemoteException` with HTTP status code 500. Server: a 
`NullPointerException` from
   `IncrementalTimelineSyncFileSystemView#sync` called by 
`RequestHandler#syncIfLocalViewBehind` on the closed view, or
   "View is already closed" from the load endpoints.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to