Hi Krishna,
there seems to be no attachment.
In addition, you should NEVER post private credentials to public forums. Please
renew the credentials of your storage account as soon as possible!
Best Regards
Roland Johann
Software Developer/Data Engineer
phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany
Mobil: +49 172 365 26 46
Mail: roland.joh...@phenetic.io
Web: phenetic.io
Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann
> Am 23.08.2019 um 08:33 schrieb Krishna Chandran Nair
> <kcn...@qatarairways.com.qa>:
>
>
>
> Hi Team,
>
> I have written a small code to connect to azure blob storage but go error. I
> have attached the error log. Please help
>
> Calling command -- ./spark-submit stg.py --jars
> /home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar
>
> Code
>
>
> vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py
>
> from pyspark import SparkContext
> from pyspark.sql import SparkSession
> from pyspark.sql import DataFrameReader
> from pyspark.sql import SparkSession
>
> session = SparkSession.builder.getOrCreate()
>
>
> #session.conf.set("fs.azure",
> "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> #session.conf.set("fs.hdfs.impl",
> "org.apache.hadoop.hdfs.DistributedFileSystem")
> #session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")
>
>
>
> #session.conf.set(
> # "fs.azure.sas.snowflakestrg.blob.core.windows.net/test",
> #
> "?sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2020-01-01T16:37:05Z&st=2019-08-13T08:37:05Z&spr=https&sig=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D"
> #)
>
> session.conf.set(
> "fs.azure.account.key.snowflakestrg.blob.core.windows.net",
>
> "LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==")
>
> df=session.read.csv("wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv
> <wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv>")
>
> df.show(5)
>
>
> Qatar Airways - Going Places Together
>
>
>
> Disclaimer:- This message (including attachments) is intended solely for the
> addressee named above. It may be confidential, privileged, subject to
> copyright, trade secret, or other legal rules and may not be forwarded
> without the author's permission. If you are not the addressee you must not
> read, copy or disseminate this message. If you have received it in error
> please notify the sender immediately and delete the message from all storage
> devices. Any opinions expressed in this message do not necessarily represent
> the official positions of Qatar Airways. Any agreements (including any
> warranties, representations, or offers) concluded with Qatar Airways by using
> electronic correspondence shall only come into existence if an authorized
> representative of Qatar Airways has explicitly approved such contract
> formation. To the fullest extent permissible by law, Qatar Airways disclaim
> all liability for loss or damage to person or property arising from this
> message being infected by computer virus or other contamination.
