[
https://issues.apache.org/jira/browse/HADOOP-19854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18091442#comment-18091442
]
Pavan Pothnis commented on HADOOP-19854:
----------------------------------------
[~anujmodi] , [~manika137] - - We are currently working on removing the
*azure-storage-8.x (EOL)* dependency from our classpath as part of upgrading to
{*}azure-storage-blob 12.x (V2 SDK){*}. However, we've hit a blocker tied to
the *WASB* protocol behavior in Hadoop for Azure Synapse.
Azure Synapse currently doesn't have support for *FNS over ABFS Protocol* when
used in Polybase Queries with {*}Blob as Staging location{*}, Meaning we will
have to stay with *WASB* for a time being, which also mean we cannot remove
*azure-storage-8.x (EOL) from our classpath and it needs to be removed for
security compliance.*
*Issue observed:*
When the V1 SDK ({{{}azure-storage-8.x{}}}) is removed from the classpath and
*WASB* protocol is still in use, Hadoop throws:
*java.lang.NoClassDefFoundError: com/microsoft/azure/storage/StorageException*
This confirms a tight coupling between the WASB protocol and the V1 SDK - i.e.,
Hadoop looks for the V1 SDK classes when it encounters a {{wasb://}} URI,
whereas ABFS uses the V2 SDK path.
*Context:*
* We have a significant number of existing customer configurations using WASB
(blob storage with FNS as staging in Azure Synapse polybase queries)
* We are evaluating whether there is any way to continue supporting WASB while
removing the V1 SDK from the classpath
*Questions:* # Does Hadoop library specifically look for Azure-storage.6.x or
8.x V1 version of SDK when WASB protocol is used ?
# Is there any plan or mechanism to decouple the WASB protocol handler from
the hard dependency on {{azure-storage-8.x}} /
{{com.microsoft.azure.storage.*}} classes?
# Are there any known workarounds or shim layers that allow WASB URIs to be
resolved via the V2 SDK?
Any inputs or direction from the Hadoop/HDFS team would be greatly appreciated.
Thank you!
> ABFS: Forced DFS Endpoint Usage Causes Failure on FNS-Blob Accounts with Soft
> Delete Enabled
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-19854
> URL: https://issues.apache.org/jira/browse/HADOOP-19854
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Affects Versions: 3.5.0, 3.4.2, 3.4.3
> Reporter: Manika Joshi
> Assignee: Manika Joshi
> Priority: Major
> Labels: pull-request-available
>
> If _fs.azure.account.hns.enabled_ is not configured, ABFS driver issues a
> getAcl call +using the DFS endpoint+ to detect the account type (HNS or FNS)
> For FNS Blob accounts with Soft Delete enabled, this behavior leads to
> failures because the DFS endpoint used does not support Soft Delete semantics.
> Temporary fix is to explicitly set the configuration:
> _fs.azure.account.hns.enabled_ so the redundant DFS-based detection is
> bypassed and the issue does not occur.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
