Krishen Bhan created HUDI-8139:
----------------------------------
Summary: Allow high priority writer to force-acquire lock within a
bounded amount of time
Key: HUDI-8139
URL: https://issues.apache.org/jira/browse/HUDI-8139
Project: Apache Hudi
Issue Type: Wish
Reporter: Krishen Bhan
*Issue*
When using HUDI multiwriter with optimistic concurrency mode enabled, a
writer's operation may need to acquire a table lock before performing some
steps. If the writer has to wait longer to acquire a lock, then the runtime of
it's operation will increase or may fail if `hoodie.write.lock.wait_time_ms`
breached. Although the lock wait time is expected to typically be short, there
are cases where a writer may need to wait for 10-20+ minutes before finally
getting the lock:
# The writer that currently acquired the lock may have encountered a
service/infra degradation thats causing it to get "stuck" in the middle of a
transaction. For example, a user using Spark and YARN may encounter a delay
when awaiting executor resources.
# The writer the currently acquired the lock may have to process many instant
files in the base table and metadata table, due to delayed
clean/archival/compaction
# If there are many concurrent writers writing to the dataset, each writer may
have to wait a longer time before finally getting their "chance" to acquire the
lock.
Although this issue can be partially mitigated by increasing
`hoodie.write.lock.wait_time_ms` for a writer, the runtime of the operation
will still not be reduced. Some users would want to prioritize the
delay/runtime of a specific writer even at the risk of failing other writers.
For example, a user may want to give an ingestion writer high priority over
other writers in order to ensure that the dataset isn't stale. In order to
satisfy this use case, HUDI can add an enhancement to reduce the amount of time
"high priority" writers need to wait before acquriing a lock.
*Resolutions*
[Approach A]
One approach could be to add a new timeout config that determines the maximum
amount of time a writer can acquire a lock before being forced to abort it's
operation and release it. Special care needs to be given so that all DFS
operations are aborted before the lock is released This can be implemented at
the transaction/lock manager level in HUDI abstraction, and does not rely on
the type of lock provider used. This in turn though also means that the changes
proposed here will not be isolated to the lock provider class and changes to
these other parts of HUDI need to be made. Another drawback with this approach
though is that it might not scale well with scenario 3, since the user will
have to keep track of the max # of writers on the dataset at any given time,
and the allowed time window they will be allowed to hold the lock for (before
aborting) will decrease.
[Approach B]
Another approach would be to update ZookeeperLockProvider to allow giving
giving a priority to a lock acquirer. The idea being that the high priority
lock acquirer will "skip the line" and be the next job to take the lock. This
implementation will only work on locks where an internal logical "queue" is
maintained like ZookeeperLockProivder. This approach though may not be
sufficient in solving this use case and may also require Approach A to be
implemented, so that the current low priority lock holder will abort within a
bounded time window.
[Approach C]
Similar to approach B, except this time update the ZookeeperLockProvider to
allow a high priority writer to forcibly "steal" the lock by forcing the
current lock holder to fail until the writer finally acquires the lock. This
impleme
--
This message was sent by Atlassian Jira
(v8.20.10#820010)