Krishen Bhan created HUDI-8139:
----------------------------------

             Summary: Allow high priority writer to force-acquire lock within a 
bounded amount of time
                 Key: HUDI-8139
                 URL: https://issues.apache.org/jira/browse/HUDI-8139
             Project: Apache Hudi
          Issue Type: Wish
            Reporter: Krishen Bhan


*Issue*
When using HUDI multiwriter with optimistic concurrency mode enabled, a 
writer's operation may need to acquire a table lock before performing some 
steps. If the writer has to wait longer to acquire a lock, then the runtime of 
it's operation will increase or may fail if `hoodie.write.lock.wait_time_ms` 
breached. Although the lock wait time is expected to typically be short, there 
are cases where a writer may need to wait for 10-20+ minutes before finally 
getting the lock:
 # The writer that currently acquired the lock may have encountered a 
service/infra degradation thats causing it to get "stuck" in the middle of a 
transaction. For example, a user using Spark and YARN may encounter a delay 
when awaiting executor resources.
 # The writer the currently acquired the lock may have to process many instant 
files in the base table and metadata table, due to delayed 
clean/archival/compaction
 # If there are many concurrent writers writing to the dataset, each writer may 
have to wait a longer time before finally getting their "chance" to acquire the 
lock.

Although this issue can be partially mitigated by increasing 
`hoodie.write.lock.wait_time_ms` for a writer, the runtime of the operation 
will still not be reduced. Some users would want to prioritize the 
delay/runtime of a specific writer even at the risk of failing other writers. 
For example, a user may want to give an ingestion writer high priority over 
other writers in order to ensure that the dataset isn't stale. In order to 
satisfy this use case, HUDI can add an enhancement to reduce the amount of time 
"high priority" writers need to wait before acquriing a lock.

*Resolutions*

[Approach A]

One approach could be to add a new timeout config that determines the maximum 
amount of time a writer can acquire a lock before being forced to abort it's 
operation and release it. Special care needs to be given so that all DFS 
operations are aborted before the lock is released This can be implemented at 
the transaction/lock manager level in HUDI abstraction, and does not rely on 
the type of lock provider used. This in turn though also means that the changes 
proposed here will not be isolated to the lock provider class and changes to 
these other parts of HUDI need to be made. Another drawback with this approach 
though is that it might not scale well with scenario 3, since the user will 
have to keep track of the max # of writers on the dataset at any given time, 
and the allowed time window they will be allowed to hold the lock for (before 
aborting) will decrease.

[Approach B] 

Another approach would be to update ZookeeperLockProvider to allow giving 
giving a priority to a lock acquirer. The idea being that the high priority 
lock acquirer will "skip the line" and be the next job to take the lock. This 
implementation will only work on locks where an internal logical "queue" is 
maintained like ZookeeperLockProivder. This approach though may not be 
sufficient in solving this use case and may also require Approach A to be 
implemented, so that the current low priority lock holder will abort within a 
bounded time window. 

[Approach C]

Similar to approach B, except this time update the ZookeeperLockProvider to 
allow a high priority writer to forcibly "steal" the lock by forcing the 
current lock holder to fail until the writer finally acquires the lock. This 
impleme



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to