here you go. Please update the values for your specific bucket to be used. spark-defaults.com - to make sure event logs go to ADLS
spark.eventLog.enabled true spark.eventLog.dir <bucket url with folder path> - abfss:// containen...@storageaccount.dfs.core.windows.net/tenant/spark/eventlogs spark.history.fs.logDirectory abfss:// containen...@storageaccount.dfs.core.windows.net/tenant/spark/eventlogs core-site.xml <property> <name>fs.abfss.impl</name> <value>org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem</value> </property> <property> <name>fs.adl.oauth2.access.token.provider.type</name> <value>ClientCredential</value> </property> <property> <name>fs.adl.oauth2.refresh.url</name> <value>https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/authorize </value> </property> <property> <name>fs.adl.oauth2.client.id</name> <value>client_id</value> </property> <property> <name>fs.adl.oauth2.credential</name> <value>client secret</value> </property> <property> <name>fs.azure.account.key.incortapocstorage.dfs.core.windows.net</name> <value>access key</value> </property> You need to have the ADLS jars in the spark class path. I just copy these from hadoop/share/tools/lib to spark/jars. Regards Jay On Tue, Feb 14, 2023 at 9:50 PM Kondala Ponnaboina (US) <kondala.r.ponnabo...@pwc.com.invalid> wrote: > Hello, > > I need help/sample adfs(*Active Directory Federation Services) (ADLS > GEN2) *on how to configure ADLS GEN2(adfs) configurations in yaml file > with spark history server ?? > > I would like to see running jobs from JupiterLab notebook with > SparkOnK8sV3.0.2 Kernel shell. > > > > Any help is much appreciated .. > > > > Thanks, > > Kondal > > > > > > > > ------------------------------ > > The information transmitted, including any attachments, is intended only > for the person or entity to which it is addressed and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination or other use of, or taking of any action in reliance upon, > this information by persons or entities other than the intended recipient > is prohibited, and all liability arising therefrom is disclaimed. If you > received this in error, please contact the sender and delete the material > from any computer. > > In the event the content of this email includes Tax advice, the content of > this email is limited to the matters specifically addressed herein and is > not intended to address other potential tax consequences or the potential > application of tax penalties to this or any other matter. > > PricewaterhouseCoopers LLP is a Delaware limited liability partnership. > This communication may come from PricewaterhouseCoopers LLP or one of its > subsidiaries. > > ------------------------------ >
