Hi DEROCCO, I think you can check the startup command of the job on k8s to see if the jar file is in the classpath.
If your job is DataStream, you need to add hadoop azure dependency in your project, and if it is an SQL job, you need to include this jar file in your Flink release package. Or you can also add this package in your cluster environment. Best, Shammon FY On Fri, May 5, 2023 at 10:21 PM DEROCCO, CHRISTOPHER <cd9...@att.com> wrote: > How can I add the package to the flink job or check if it is there? > > > > *From:* Shammon FY <zjur...@gmail.com> > *Sent:* Thursday, May 4, 2023 9:59 PM > *To:* DEROCCO, CHRISTOPHER <cd9...@att.com> > *Cc:* user@flink.apache.org > *Subject:* Re: MSI Auth to Azure Storage Account with Flink Apache > Operator not working > > > > Hi DEROCCO, > > > > I think you need to check whether there is a hadoop-azure jar file in the > classpath of your flink job. From an error message '*Caused by: > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider not found.*', your > flink job may be missing this package. > > > > Best, > > Shammon FY > > > > > > On Fri, May 5, 2023 at 4:40 AM DEROCCO, CHRISTOPHER <cd9...@att.com> > wrote: > > > > I receive the error: *Caused by: java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider not found.* > > I’m using flink 1.16 running in Azure Kubernetes using the Flink Apache > Kubernetes Operator. > > I have the following specified in the spec.flinkConfiguration: as per the > Apache Kubernetes operator documentation. > > > > fs.azure.createRemoteFileSystemDuringInitialization: "true" > > fs.azure.account.auth.type.storageaccountname.dfs.core.windows.net > <https://urldefense.com/v3/__http:/fs.azure.account.auth.type.storageaccountname.dfs.core.windows.net__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4HpyjvOZFkA$>: > OAuth > > fs.azure.account.oauth.provider.type.<storageaccountname>. > dfs.core.windows.net > <https://urldefense.com/v3/__http:/dfs.core.windows.net__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4Hpycm9yrUw$>: > org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider > > fs.azure.account.oauth2.msi.tenant. <storageaccountname>. > dfs.core.windows.net > <https://urldefense.com/v3/__http:/dfs.core.windows.net__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4Hpycm9yrUw$>: > <MY TENANT ID> > > fs.azure.account.oauth2.client.id > <https://urldefense.com/v3/__http:/fs.azure.account.oauth2.client.id__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4HpwRB0LkWg$>. > <storageaccountname>.dfs.core.windows.net > <https://urldefense.com/v3/__http:/dfs.core.windows.net__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4Hpycm9yrUw$>: > <MY CLIENT ID of VM> > > fs.azure.account.oauth2.client.endpoint. <storageaccountname>. > dfs.core.windows.net > <https://urldefense.com/v3/__http:/dfs.core.windows.net__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4Hpycm9yrUw$>: > https://login.microsoftonline.com/ > <https://urldefense.com/v3/__https:/login.microsoftonline.com/__;!!BhdT!nslIUVS9K-rzMRvjMFpWqBpcsAIiVPAfG6uroDOiSSQfmARHAQCYweWSe-TmKGHGzKD4HpzeWh7XLg$><MY > TENANT ID>/oauth2/token > > > > I also have this specified in the container environment variables. > > - name: ENABLE_BUILT_IN_PLUGINS > > value: flink-azure-fs-hadoop-1.16.1.jar > > > > I think I’m missing a configuration step because the MsiTokenProvider > class is not found based on the logs. Any help would be appreciated. > > > > > > *Chris deRocco* > > Senior – Cybersecurity > > Chief Security Office | STORM Threat Analytics > > > > *AT&T* > > Middletown, NJ > > Phone: 732-639-9342 > > Email: cd9...@att.com > > > >
