Sneha Vijayarajan created HADOOP-18012:
------------------------------------------
Summary: ABFS: Modify Rename idempotency code
Key: HADOOP-18012
URL: https://issues.apache.org/jira/browse/HADOOP-18012
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure
Affects Versions: 3.3.1
Reporter: Sneha Vijayarajan
Assignee: Sneha Vijayarajan
ABFS driver has a handling for rename idempotency which relies on LMT of the
destination file to conclude if the rename was successful or not when source
file is absent and if the rename request had entered retry loop.
This handling is incorrect as LMT of the destination does not change on rename.
This Jira will track the change to undo the current implementation and add a
new one where for an incoming rename operation, source file eTag is fetched
first and then rename is done only if eTag matches for the source file.
As this is going to be a costly operation given an extra HEAD request is added
to each rename, this implementation will be guarded over a config and can
enabled by customers who have workloads that do multiple renames.
Long term plan to handle rename idempotency without HEAD request is being
discussed.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
