ravjot28 opened a new pull request, #20590:
URL: https://github.com/apache/hudi/pull/20590

   ### Describe the issue this Pull Request addresses
   
   Closes #20580
   
   A table refresh removes the table's view from `FileSystemViewManager` and 
closes it. A request resolved the view
   separately for its freshness check, its sync, its handler and its final 
check, so a refresh in between made it sync or
   read a closed view (`NullPointerException`, HTTP 500) or answer from another 
view than the one it checked.
   `LAST_INSTANT`, `TIMELINE`, `LOAD_PARTITIONS` and `LOAD_ALL_PARTITIONS` read 
the view with no coordination with a
   refresh.
   
   ### Summary and Changelog
   
   A request reads one view from its first check to its response, and a refresh 
never closes a view a request is using.
   
   - `FileSystemViewManager` hands out shared leases of a table's view 
(`leaseFileSystemView`, `ViewLease`). A refresh,
     `unregisterBasePath` or the manager's close removes the view and returns 
at once; the view is closed when no lease of
     it is held, otherwise by the release of its last lease. A view replaced 
before a lease was taken is left to its
     replacement.
   - `RequestHandler`: a request that reads the view takes one lease for the 
freshness check, the sync, the handler with
     its response and the final check, and binds the leased view for its 
handlers (`Handler#bindView`), which read it.
     Handlers of one view still run in parallel; the view's monitor covers the 
check and the sync only.
   - The four endpoints above and the timeline read of `CREATE_MARKER` with 
early conflict detection take the lease as
     well.
   - The leased view is bound per request thread instead of being passed 
through every handler method, to keep the
     handler signatures and the diff small.
   - New test class `TestRequestHandlerViewReplacement`: a refresh at each 
timeline read a request makes outside the
     view's own sync (the positions come from a request that is not paused), 
between the handler's view lookup and its
     read, and before the handler traverses its result stream; handlers read 
the leased view; two requests on one view
     run their handlers at the same time; the four endpoints; a refresh returns 
while a request holds the view; a lease
     holder can refresh and close the manager without waiting; a retired view 
with two leases closes with the last
     release.
   
   ### Impact
   
   No API change for clients and no configuration change. 
`FileSystemViewManager` gains `leaseFileSystemView` and
   `ViewLease`. A removed view may be closed slightly later, by the release of 
the last request that uses it.
   
   Open PR #20239 also changes `syncIfLocalViewBehind`; whichever lands second 
needs a small rebase there.
   
`TestRequestHandlerViewReplacement#refreshDuringARequestClosesNoViewTheRequestReads`
 finds the timeline reads of a
   request at run time, so it does not depend on how many reads that method 
makes.
   
   **Backport:** please include this fix in the next 1.2.x patch release; the 
race is present in 1.2.1. The commit does
   not cherry-pick cleanly onto `release-1.2.1`: `RequestHandler` and `Handler` 
conflict with changes made on master after
   1.2.1 (#19946 and the timeline UI). I can open an adapted change against 
`branch-1.2.x`.
   
   ### Risk Level
   
   medium. The change touches the request path of every timeline server view 
endpoint. Mitigation: the tests pause the
   server-side view at each point a refresh can land, check that handlers of 
one view still run concurrently, and the
   existing timeline server and remote view suites pass.
   
   Tests, run on master with JDK 11 and passing checkstyle checks for main and 
test sources (scalastyle did not execute
   in this Java-only reactor), with `mvn test -pl hudi-timeline-service -am 
-Dtest=...` and the classes listed below:
   - Without the main code change (the two tests that call the lease API left 
out): 9 of the 10 remaining cases of
     `TestRequestHandlerViewReplacement` fail, on "the view stays open while 
the request holds it" (the refresh closed
     a view in use; for `refreshDuringARequestClosesNoViewTheRequestReads` at 
its first measured read) and on HTTP 500 in
     `handlerReadsTheViewTheRequestHolds`.
   - With the change: `TestRequestHandlerViewReplacement` 12/12, 
`TestRequestHandler` 2/2, `TestTimelineService` 2/2,
     `TestRemoteHoodieTableFileSystemView` 54/54, 
`TestMarkerBasedEarlyConflictDetectionRunnable` 1/1,
     `TestHoodieTableFileSystemView` 51/51, 
`TestSpillableMapBasedFileSystemView` 51/51, `TestIncrementalFSViewSync` 5/5,
     `TestSpillableMapBasedIncrementalFSViewSync` 5/5, 
`TestPriorityBasedFileSystemView` 31/31 pass.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to