[
https://issues.apache.org/jira/browse/FLINK-30745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679120#comment-17679120
]
Dheeraj Panangat edited comment on FLINK-30745 at 1/20/23 12:48 PM:
--------------------------------------------------------------------
Hi [~surendralilhore] ,
Post below changes. I don't see any errors in the logs and I can see the
checkpointing folders created on Azure too.
Used the *core-default-shaded.xml* configuration
Provided the shaded classes instead of Hadoop classes :
in core-site.xml
{code:java}
<property>
<name>fs.hdfs.impl</name>
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
</property>
<property>
<name>fs.abfs.impl</name>
<value>org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem</value>
</property>
<property>
<name>fs.file.impl</name>
<value>org.apache.hadoop.fs.LocalFileSystem</value>
</property>
<property>
<name>fs.azure.account.oauth.provider.type</name>
<value>org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider</value>
</property>{code}
in core-default-shaded.xml
{code:java}
<property>
<name>fs.hdfs.impl</name>
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
</property>
<property>
<name>fs.abfs.impl</name>
<value>org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem</value>
</property>
<property>
<name>fs.file.impl</name>
<value>org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.LocalFileSystem</value>
</property>
<property>
<name>fs.azure.account.oauth.provider.type</name>
<value>org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider</value>
</property>{code}
Thanks for checking this.
Let me know your thoughts if any on the above approach.
was (Author: JIRAUSER297631):
Hi [~surendralilhore] ,
Post below changes. I don't see any errors in the logs and I can see the
checkpointing folders created on Azure too.
Still testing this out if checkpointing is working correctly.
Used the *core-default-shaded.xml* configuration
Provided the shaded classes instead of Hadoop classes :
in core-site.xml
{code:java}
<property>
<name>fs.hdfs.impl</name>
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
</property>
<property>
<name>fs.abfs.impl</name>
<value>org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem</value>
</property>
<property>
<name>fs.file.impl</name>
<value>org.apache.hadoop.fs.LocalFileSystem</value>
</property>
<property>
<name>fs.azure.account.oauth.provider.type</name>
<value>org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider</value>
</property>{code}
in core-default-shaded.xml
{code:java}
<property>
<name>fs.hdfs.impl</name>
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
</property>
<property>
<name>fs.abfs.impl</name>
<value>org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem</value>
</property>
<property>
<name>fs.file.impl</name>
<value>org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.LocalFileSystem</value>
</property>
<property>
<name>fs.azure.account.oauth.provider.type</name>
<value>org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider</value>
</property>{code}
> Check-pointing with Azure Data Lake Storage
> -------------------------------------------
>
> Key: FLINK-30745
> URL: https://issues.apache.org/jira/browse/FLINK-30745
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.15.2, 1.14.6
> Reporter: Dheeraj Panangat
> Priority: Major
>
> Hi,
> While checkpointing to Azure Blob Storage using Flink, we get the following
> error :
> {code:java}
> Caused by: Configuration property <accoutnname>.dfs.core.windows.net not
> found.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
> We have given the configurations in core-site.xml too for following
> {code:java}
> fs.hdfs.impl
> fs.abfs.impl -> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem
> fs.file.impl
> fs.azure.account.auth.type
> fs.azure.account.oauth.provider.type
> fs.azure.account.oauth2.client.id
> fs.azure.account.oauth2.client.secret
> fs.azure.account.oauth2.client.endpoint
> fs.azure.createRemoteFileSystemDuringInitialization -> true {code}
> On debugging found that flink reads from core-default-shaded.xml, but even if
> the properties are specified there, the default configs are not loaded and we
> get a different exception as :
> {code:java}
> Caused by: Unable to load key provider class.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:540)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1136)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
