vamshipasunuru1 opened a new issue, #20650:
URL: https://github.com/apache/hudi/issues/20650

   ### Bug Description
   
   **What happened:**
   
   A Spark upsert job using the embedded timeline server (remote file system 
view) got stuck while listing files through the timeline server: first in the 
small-file lookup, then in the write stage when creating merge handles. Clean 
and archival for the table run in an async job.
   
   Every listing request made the timeline server re-read the timeline, drop 
its cached partition listings, rebuild the requested partition, and then reject 
the request with 400. This repeated for the whole run, one request at a time, 
because the sync runs inside `synchronized (view)`.
   
   Cause:
   
   - Executors send a hash of the timeline the driver loaded when the write 
started. That copy is frozen for the whole job 
(`RemoteHoodieTableFileSystemView`).
   - The timeline server builds its view lazily, on the first executor request 
(`FileSystemViewManager#getFileSystemView`), minutes later.
   - In between, the async job completed a clean and archived old commits. The 
server's timeline has an extra clean at the tail and is missing commits at the 
head.
   - The hash covers every instant (`BaseHoodieTimeline#computeTimelineHash`). 
The trailing-clean tolerance from #19946 tolerates the tail but not the head, 
so `RequestHandler#syncIfLocalViewBehind` syncs on every request and the two 
sides never converge.
   
   
   Bellow sequence diagram shows a sequence of events that can lead to above 
issue
   
   <img width="3116" height="2350" alt="diagram-3116x2350" 
src="https://github.com/user-attachments/assets/c2989e75-969e-4293-90e8-93edb5328137";
 />
   
   
   **What you expected:**
   - Archival of old instants should not make the timeline server and the 
executors permanently inconsistent. The request should be served without a 
sync, as it is for a trailing clean today.
   - A mismatch that a sync cannot resolve should not cost a timeline re-read 
and partition rebuild under the view lock on every request.
   
   
   **Steps to reproduce:**
   1. Run a writer with the embedded timeline server, and clean and archival in 
an async job (`hoodie.clean.automatic=false`, `hoodie.archive.automatic=false`).
   2. Start an upsert. Before its first executor request to the timeline 
server, let the async job complete a clean and archive at least one commit.
   3. Every listing request now logs the two lines above, and the listing 
stages stall.
   
   
   
   ### Environment
   
   **Hudi version:** 1.2
   **Query engine:** (Spark/Flink/Trino etc) Spark
   **Relevant configs:**
   
   
   ### Logs and Stack Trace
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to