Pavan Pothnis created HDFS-17898:
------------------------------------
Summary: ABFS migration fails on FNS blob storage with Soft Delete
enabled
Key: HDFS-17898
URL: https://issues.apache.org/jira/browse/HDFS-17898
Project: Hadoop HDFS
Issue Type: Bug
Components: fs/azure
Affects Versions: 3.4.3, 3.4.2
Environment: *Environment*
Hadoop version: _*3.4.2 and 3.4.3*_
Storage SDK - Tried with both *6.x* and *12.X*
Storage type: Azure Blob Storage (FNS - {*}FlatNameSpace{*})
Storage account features enabled: FNS and Soft Delete
Migration path: *WASB → ABFS (using abfss:// scheme)*
Reporter: Pavan Pothnis
When migrating from WASB to ABFS on a FlatNameSpace (FNS) Azure Blob Storage
account that has both FNS and Soft Delete features enabled, the migration fails
with an incompatibility error.
*Environment*
Hadoop version: _*3.4.2 and 3.4.3*_
Storage SDK - Tried with both *6.x* and *12.X*
Storage type: Azure Blob Storage (FNS - {*}FlatNameSpace{*})
Storage account features enabled: FNS and Soft Delete
Migration path: *WASB → ABFS (using abfss:// scheme)*
*Issue Description*
Following the WASB to ABFS migration guide (as documented in wasb.md), users
are instructed to simply change URIs from WASB scheme to ABFS scheme. However,
when the underlying Azure Blob Storage account has Soft Delete enabled, the
ABFS driver encounters a critical error that prevents data access.
*Steps to Reproduce*
Create an Azure Blob Storage account (GPv2) with FlatNameSpace enabled
Enable Soft Delete feature on the blob storage account
Upgrade Hadoop from WASB to ABFS (3.4.2 )
Change URIs from wasb:// to abfss://
Attempt to read/parse Avro files or any data from the storage
*Error Message*
java.lang.RuntimeException: java.lang.RuntimeException: The file or partition
directory
[{*}abfss{*}://{*}azureblobv3-cctf{*}@{*}adapterqablob{*}.{*}dfs{*}.core.windows.net/alldatatype.avro]
is not valid. The parser encountered the following error while parsing the
content: [org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed:
"This endpoint does not support BlobStorageEvents or SoftDelete. Please disable
these account features if you would like to use this endpoint.", 409, HEAD,
https://adapterqablob.{*}dfs{*}.core.windows.net/azureblobv3-cctf/alldatatype.avro?upn=false&action=getStatus&timeout=90s,
rId: 0b404e0b-301f-00a7-4c18-bc227f000000]. Select a valid [Avro] file or
partition directory.
Changes done as part of WABS to ABFS Migration
*From -*
String fileName = "wasbs://" +
connection.getContainerClient().getBlobContainerName() + "@" +
connection.getAccountName() + ".blob." +connection.getEndpointSuffix()+ "/";
*To -*
String fileName = "abfss://" +
connection.getContainerClient().getBlobContainerName() + "@" +
connection.getAccountName() + ".dfs." +connection.getEndpointSuffix()+ "/";
*From -*
String wasbFileSystemUrl = "wasbs://" + containerName + "@" +
connection.getAccountName() +".blob."+connection.getEndpointSuffix()+"/";
*To -*
String wasbFileSystemUrl = "abfss://" + containerName + "@" +
connection.getAccountName() +".dfs."+connection.getEndpointSuffix()+"/";
*Configuration - Shared Key Authentication*
if(authenticationType.equals(AzureStorageAuthorization.SHARED_KEY_AUTHORIZATION.getValue()))
{
*From -*
String accountKey = "fs.azure.account.key." + connection.getAccountName() +
".blob." + endpointSuffix;
*To -*
**
String accountKey = "fs.azure.account.key." + connection.getAccountName() +
".dfs." + endpointSuffix;
configMap.put(accountKey, connection.getAccountKey());
}
*Configuration - Shared Access Signature*
if(authenticationType.equals(AzureStorageAuthorization.SHARED_ACCESS_SIGNATURES.getValue()))
{
*From -*
String containerName = StringUtils.isNotBlank(containerNameOverride) ?
containerNameOverride : connection.getContainer().getName();
String sharedaccesssignature = "fs.azure.sas." + containerName +"." +
connection.getAccountName() + ".blob."+ endpointSuffix;
configMap.put(sharedaccesssignature, connection.getSharedAccessSignature());
*To -*
configMap.put("fs.azure.account.auth.type." + connection.getAccountName() +
".dfs." + endpointSuffix, "SAS");
configMap.put("fs.azure.sas.fixed.token." + connection.getAccountName() +
".dfs." + endpointSuffix, connection.getSharedAccessSignature());
}
All the changes made in To section works fine as long as the Blob account is
not configured with Soft Delete / Event and fails when enabled, causing out of
box failure for all account with those settings and these are extensively used
settings.
*Current Workaround*
The issue only resolves after manually disabling Soft Delete on the Azure Blob
Storage account. This workaround defeats the purpose of having this protective
feature enabled and is not viable for production environments.
*Expected Behavior*
ABFS driver should support Soft Delete feature on FNS blob storage accounts, or
the migration documentation should clearly document this incompatibility as a
prerequisite check before migration.
*Impact*
Out-of-box failure for users migrating from WASB to ABFS on FNS storage with
Soft Delete enabled
Requires manual intervention and disabling of security features
Migration path documented in wasb.md does not account for this limitation
*Potential Solutions*
Add support for Soft Delete in ABFS driver when using FNS blob storage
Update WASB to ABFS migration guide to document this incompatibility and
require Soft Delete to be disabled before migration
Provide configuration option in ABFS to work with Soft Delete enabled accounts.
Mention minimum required Hadoop Version for FNS Blob Storage with Soft delete
migration from WASBS to ABFSS.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
