Hi Team,
I have written a small code to connect to azure blob storage but go error. I
have attached the error log. Please help
Calling command -- ./spark-submit stg.py --jars
/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar
Code
vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py
from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql import DataFrameReader
from pyspark.sql import SparkSession
session = SparkSession.builder.getOrCreate()
#session.conf.set("fs.azure",
"org.apache.hadoop.fs.azure.NativeAzureFileSystem")
#session.conf.set("fs.hdfs.impl",
"org.apache.hadoop.hdfs.DistributedFileSystem")
#session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")
#session.conf.set(
# "fs.azure.sas.snowflakestrg.blob.core.windows.net/test",
#
"?sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2020-01-01T16:37:05Z&st=2019-08-13T08:37:05Z&spr=https&sig=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D"
#)
session.conf.set(
"fs.azure.account.key.snowflakestrg.blob.core.windows.net",
"LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==")
df=session.read.csv("wasbs://[email protected]/users.csv")
df.show(5)
Qatar Airways - Going Places Together
Disclaimer:- This message (including attachments) is intended solely for the
addressee named above. It may be confidential, privileged, subject to
copyright, trade secret, or other legal rules and may not be forwarded without
the author's permission. If you are not the addressee you must not read, copy
or disseminate this message. If you have received it in error please notify the
sender immediately and delete the message from all storage devices. Any
opinions expressed in this message do not necessarily represent the official
positions of Qatar Airways. Any agreements (including any warranties,
representations, or offers) concluded with Qatar Airways by using electronic
correspondence shall only come into existence if an authorized representative
of Qatar Airways has explicitly approved such contract formation. To the
fullest extent permissible by law, Qatar Airways disclaim all liability for
loss or damage to person or property arising from this message being infected
by computer virus or other contamination.
