https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management <https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management> Try to add the azure dependency via `—packages org.apache.hadoop:hadoop-azure:2.7.7` assuming you use hadoop 2.7.7.
Best Regards Roland Johann Software Developer/Data Engineer phenetic GmbH Lütticher Straße 10, 50674 Köln, Germany Mobil: +49 172 365 26 46 Mail: roland.joh...@phenetic.io Web: phenetic.io Handelsregister: Amtsgericht Köln (HRB 92595) Geschäftsführer: Roland Johann, Uwe Reimann > Am 23.08.2019 um 10:08 schrieb Krishna Chandran Nair > <kcn...@qatarairways.com.qa>: > > Please find the attached error > > From: Roland Johann <roland.joh...@phenetic.io> > Sent: 23 August 2019 10:51 AM > To: Krishna Chandran Nair <kcn...@qatarairways.com.qa> > Cc: user@spark.apache.org > Subject: [External]Re: error while connecting to azure blob storage > > Hi Krishna, > > there seems to be no attachment. > In addition, you should NEVER post private credentials to public forums. > Please renew the credentials of your storage account as soon as possible! > > Best Regards > > Roland Johann > Software Developer/Data Engineer > > phenetic GmbH > Lütticher Straße 10, 50674 Köln, Germany > > Mobil: +49 172 365 26 46 > Mail: roland.joh...@phenetic.io <mailto:roland.joh...@phenetic.io> > Web: phenetic.io <http://phenetic.io/> > > Handelsregister: Amtsgericht Köln (HRB 92595) > Geschäftsführer: Roland Johann, Uwe Reimann > > > > > Am 23.08.2019 um 08:33 schrieb Krishna Chandran Nair > <kcn...@qatarairways.com.qa <mailto:kcn...@qatarairways.com.qa>>: > > > > Hi Team, > > I have written a small code to connect to azure blob storage but go error. I > have attached the error log. Please help > > Calling command -- ./spark-submit stg.py --jars > /home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar > > Code > > > vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py > > from pyspark import SparkContext > from pyspark.sql import SparkSession > from pyspark.sql import DataFrameReader > from pyspark.sql import SparkSession > > session = SparkSession.builder.getOrCreate() > > > #session.conf.set("fs.azure", > "org.apache.hadoop.fs.azure.NativeAzureFileSystem") > #session.conf.set("fs.hdfs.impl", > "org.apache.hadoop.hdfs.DistributedFileSystem") > #session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem") > > > > #session.conf.set( > # "fs.azure.sas.snowflakestrg.blob.core.windows.net/test", > # > "?sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2020-01-01T16:37:05Z&st=2019-08-13T08:37:05Z&spr=https&sig=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D" > #) > > session.conf.set( > "fs.azure.account.key.snowflakestrg.blob.core.windows.net", > > "LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==") > > df=session.read.csv("wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv > <wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv>") > > df.show(5) > > > Qatar Airways - Going Places Together > > > > Disclaimer:- This message (including attachments) is intended solely for the > addressee named above. It may be confidential, privileged, subject to > copyright, trade secret, or other legal rules and may not be forwarded > without the author's permission. If you are not the addressee you must not > read, copy or disseminate this message. If you have received it in error > please notify the sender immediately and delete the message from all storage > devices. Any opinions expressed in this message do not necessarily represent > the official positions of Qatar Airways. Any agreements (including any > warranties, representations, or offers) concluded with Qatar Airways by using > electronic correspondence shall only come into existence if an authorized > representative of Qatar Airways has explicitly approved such contract > formation. To the fullest extent permissible by law, Qatar Airways disclaim > all liability for loss or damage to person or property arising from this > message being infected by computer virus or other contamination. > > <pyspark.txt>
