[
https://issues.apache.org/jira/browse/HADOOP-19497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17940536#comment-17940536
]
ASF GitHub Bot commented on HADOOP-19497:
-----------------------------------------
anujmodi2021 commented on code in PR #7509:
URL: https://github.com/apache/hadoop/pull/7509#discussion_r2026158411
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java:
##########
@@ -1691,4 +1555,297 @@ public String
addClientTransactionIdToHeader(List<AbfsHttpHeader> requestHeaders
}
return clientTransactionId;
}
+
+ /**
+ * Attempts to rename a path with client transaction ID (CTId) recovery
mechanism.
+ * If the initial rename attempt fails, it tries to recover using CTId or
ETag
+ * and retries the operation.
+ *
+ * @param source the source path to be renamed
+ * @param destination the destination path for the rename
+ * @param continuation the continuation token for the operation
+ * @param tracingContext the context for tracing the operation
+ * @param sourceEtag the ETag of the source path for conditional requests
+ * @param isMetadataIncompleteState flag indicating if the metadata state is
incomplete
+ * @return an {@link AbfsClientRenameResult} containing the result of the
rename operation
+ * @throws IOException if an error occurs during the rename operation
+ */
+ private AbfsClientRenameResult renameWithCTIdRecovery(String source,
+ String destination, String continuation, TracingContext tracingContext,
+ String sourceEtag, boolean isMetadataIncompleteState) throws IOException
{
+ // Get request headers for rename operation
+ List<AbfsHttpHeader> requestHeaders = getHeadersForRename(source);
+ // Add client transaction ID to the request headers
+ String clientTransactionId =
addClientTransactionIdToHeader(requestHeaders);
+
+ // Create the URL for the rename operation
+ final URL url = createRequestUrl(destination,
+ getRenameQueryBuilder(destination, continuation).toString());
+
+ // Create the rename operation
+ AbfsRestOperation op = createRenameRestOperation(url, requestHeaders);
+ try {
+ incrementAbfsRenamePath();
+ op.execute(tracingContext);
+ // If we successfully rename a path and isMetadataIncompleteState is
true,
+ // then the rename was recovered; otherwise, it wasn’t.
+ // This is why isMetadataIncompleteState is used for renameRecovery (as
the second parameter).
+ return new AbfsClientRenameResult(op, isMetadataIncompleteState,
+ isMetadataIncompleteState);
+ } catch (AzureBlobFileSystemException e) {
+ // Handle rename exceptions and retry if applicable
+ handleRenameException(source, destination, continuation,
Review Comment:
Makes sense. Please add some code comments to make all this more readable
and easily understandable.
In the main Javadoc for both types(Etag and CT) of rename it would be godd
to explain the whole flow of recovery
> [ABFS] Enable rename and create recovery from client transaction id over DFS
> endpoint
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-19497
> URL: https://issues.apache.org/jira/browse/HADOOP-19497
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.5.0
> Reporter: Manish Bhatt
> Assignee: Manish Bhatt
> Priority: Major
> Labels: pull-request-available
>
> We have implemented create and rename recovery using client transaction IDs
> over the DFS endpoint ([HADOOP-19450] [ABFS] Rename/Create path idempotency
> client-level resolution - ASF JIRA). Since the backend changes were not fully
> rolled out, we initially implemented the changes with the flag disabled. With
> this update, we aim to enable the flag, which will start sending client
> transaction IDs. In case of a failure, we will attempt to recover from the
> failed state if possible. Here are the detailed steps and considerations for
> this process:
> 1. **Implementation Overview**: We introduced a mechanism for create and
> rename recovery via client transaction IDs to enhance reliability and data
> integrity over the DFS endpoint. This change was initially flagged as
> disabled due to incomplete backend rollouts.
> 2. **Current Update**: With the backend changes now in place, we are ready to
> enable the flag. This will activate the sending of client transaction IDs,
> allowing us to track and manage transactions more effectively.
> 3. **Failure Recovery**: The primary advantage of enabling this flag is the
> potential for recovery from failed states. If a transaction fails, we can use
> the client transaction ID to attempt a recovery, minimizing data loss and
> ensuring continuity.
> 4. **Next Steps**: We will proceed with enabling the flag and closely monitor
> the system's performance. Any issues or failures will be documented and
> addressed promptly to ensure a smooth transition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
