RE: [External]Re: error while connecting to azure blob storage

[Krishna Chandran Nair]

I sent you a fake token and key

From: Julien Laurenceau 
Sent: 23 August 2019 6:06 PM
To: Krishna Chandran Nair 
Cc: user@spark.apache.org
Subject: [External]Re: error while connecting to azure blob storage

Hi
Did you just publicly disclosed a real token to tour blob storage ?
If so, please be aware that you share the data and maybe write permission...
Refresh tout token !

Regards

Le ven. 23 août 2019 à 08:33, Krishna Chandran Nair 
mailto:kcn...@qatarairways.com.qa>> a écrit :


Hi Team,

I have written a small code to connect to azure blob storage but go error. I 
have attached the error log.  Please help

Calling command -- ./spark-submit stg.py --jars 
/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar

Code


vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py

from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql import DataFrameReader
from pyspark.sql import SparkSession

session = SparkSession.builder.getOrCreate()


#session.conf.set("fs.azure", 
"org.apache.hadoop.fs.azure.NativeAzureFileSystem")
#session.conf.set("fs.hdfs.impl", 
"org.apache.hadoop.hdfs.DistributedFileSystem")
#session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")



#session.conf.set(
#   
"fs.azure.sas.snowflakestrg.blob.core.windows.net/test",
  #  
"?sv=2018-03-28=bfqt=sco=rwdlacup=2020-01-01T16:37:05Z=2019-08-13T08:37:05Z=https=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D"
#)

session.conf.set(
  
"fs.azure.account.key.snowflakestrg.blob.core.windows.net",

"LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==")

df=session.read.csv("wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv")

df.show(5)


Qatar Airways - Going Places Together

[OW LOGO]

Disclaimer:- This message (including attachments) is intended solely for the 
addressee named above. It may be confidential, privileged, subject to 
copyright, trade secret, or other legal rules and may not be forwarded without 
the author's permission. If you are not the addressee you must not read, copy 
or disseminate this message. If you have received it in error please notify the 
sender immediately and delete the message from all storage devices. Any 
opinions expressed in this message do not necessarily represent the official 
positions of Qatar Airways. Any agreements (including any warranties, 
representations, or offers) concluded with Qatar Airways by using electronic 
correspondence shall only come into existence if an authorized representative 
of Qatar Airways has explicitly approved such contract formation. To the 
fullest extent permissible by law, Qatar Airways disclaim all liability for 
loss or damage to person or property arising from this message being infected 
by computer virus or other contamination.

Re: error while connecting to azure blob storage

[Julien Laurenceau]

Hi
Did you just publicly disclosed a real token to tour blob storage ?
If so, please be aware that you share the data and maybe write permission...
Refresh tout token !

Regards

Le ven. 23 août 2019 à 08:33, Krishna Chandran Nair <
kcn...@qatarairways.com.qa> a écrit :

>
>
>
>
> Hi Team,
>
>
>
> I have written a small code to connect to azure blob storage but go error.
> I have attached the error log.  Please help
>
>
>
> Calling command -- ./spark-submit stg.py --jars
> /home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar
>
>
>
> Code
>
>
>
>
>
> vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py
>
>
>
> from pyspark import SparkContext
>
> from pyspark.sql import SparkSession
>
> from pyspark.sql import DataFrameReader
>
> from pyspark.sql import SparkSession
>
>
>
> session = SparkSession.builder.getOrCreate()
>
>
>
>
>
> #session.conf.set("fs.azure",
> "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
>
> #session.conf.set("fs.hdfs.impl",
> "org.apache.hadoop.hdfs.DistributedFileSystem")
>
> #session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")
>
>
>
>
>
>
>
> #session.conf.set(
>
> #   "fs.azure.sas.snowflakestrg.blob.core.windows.net/test",
>
>   #
> "?sv=2018-03-28=bfqt=sco=rwdlacup=2020-01-01T16:37:05Z=2019-08-13T08:37:05Z=https=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D"
>
> #)
>
>
>
> session.conf.set(
>
>   "fs.azure.account.key.snowflakestrg.blob.core.windows.net",
>
>
> "LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==")
>
>
>
> df=session.read.csv("wasbs://
> t...@snowflakestrg.blob.core.windows.net/users.csv")
>
>
>
> df.show(5)
>
>
>
>
> Qatar Airways - Going Places Together
>
> *[image: OW LOGO]*
>
> Disclaimer:- This message (including attachments) is intended solely for
> the addressee named above. It may be confidential, privileged, subject to
> copyright, trade secret, or other legal rules and may not be forwarded
> without the author's permission. If you are not the addressee you must not
> read, copy or disseminate this message. If you have received it in error
> please notify the sender immediately and delete the message from all
> storage devices. Any opinions expressed in this message do not necessarily
> represent the official positions of Qatar Airways. Any agreements
> (including any warranties, representations, or offers) concluded with Qatar
> Airways by using electronic correspondence shall only come into existence
> if an authorized representative of Qatar Airways has explicitly approved
> such contract formation. To the fullest extent permissible by law, Qatar
> Airways disclaim all liability for loss or damage to person or property
> arising from this message being infected by computer virus or other
> contamination.
>

Re: [External]Re: error while connecting to azure blob storage

[Roland Johann]

https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management
 

Try to add the azure dependency via `—packages 
org.apache.hadoop:hadoop-azure:2.7.7` assuming you use hadoop 2.7.7.

Best Regards

Roland Johann
Software Developer/Data Engineer

phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany

Mobil: +49 172 365 26 46
Mail: roland.joh...@phenetic.io
Web: phenetic.io

Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann



> Am 23.08.2019 um 10:08 schrieb Krishna Chandran Nair 
> :
> 
> Please find the attached error
>  
> From: Roland Johann  
> Sent: 23 August 2019 10:51 AM
> To: Krishna Chandran Nair 
> Cc: user@spark.apache.org
> Subject: [External]Re: error while connecting to azure blob storage
>  
> Hi Krishna,
>  
> there seems to be no attachment.
> In addition, you should NEVER post private credentials to public forums. 
> Please renew the credentials of your storage account as soon as possible!
>  
> Best Regards
> 
> Roland Johann
> Software Developer/Data Engineer
> 
> phenetic GmbH
> Lütticher Straße 10, 50674 Köln, Germany
> 
> Mobil: +49 172 365 26 46
> Mail: roland.joh...@phenetic.io 
> Web: phenetic.io 
> 
> Handelsregister: Amtsgericht Köln (HRB 92595)
> Geschäftsführer: Roland Johann, Uwe Reimann
>  
>  
> 
> 
> Am 23.08.2019 um 08:33 schrieb Krishna Chandran Nair 
> mailto:kcn...@qatarairways.com.qa>>:
>  
>  
>  
> Hi Team,
>  
> I have written a small code to connect to azure blob storage but go error. I 
> have attached the error log.  Please help
>  
> Calling command -- ./spark-submit stg.py --jars 
> /home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar
>  
> Code
>  
>  
> vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py
>  
> from pyspark import SparkContext
> from pyspark.sql import SparkSession
> from pyspark.sql import DataFrameReader
> from pyspark.sql import SparkSession
>  
> session = SparkSession.builder.getOrCreate()
>  
>  
> #session.conf.set("fs.azure", 
> "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> #session.conf.set("fs.hdfs.impl", 
> "org.apache.hadoop.hdfs.DistributedFileSystem")
> #session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")
>  
>  
>  
> #session.conf.set(
> #   "fs.azure.sas.snowflakestrg.blob.core.windows.net/test",
>   #  
> "?sv=2018-03-28=bfqt=sco=rwdlacup=2020-01-01T16:37:05Z=2019-08-13T08:37:05Z=https=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D"
> #)
>  
> session.conf.set(
>   "fs.azure.account.key.snowflakestrg.blob.core.windows.net",
> 
> "LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==")
>  
> df=session.read.csv("wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv
>  ")
>  
> df.show(5)
>  
>  
> Qatar Airways - Going Places Together 
> 
> 
> 
> Disclaimer:- This message (including attachments) is intended solely for the 
> addressee named above. It may be confidential, privileged, subject to 
> copyright, trade secret, or other legal rules and may not be forwarded 
> without the author's permission. If you are not the addressee you must not 
> read, copy or disseminate this message. If you have received it in error 
> please notify the sender immediately and delete the message from all storage 
> devices. Any opinions expressed in this message do not necessarily represent 
> the official positions of Qatar Airways. Any agreements (including any 
> warranties, representations, or offers) concluded with Qatar Airways by using 
> electronic correspondence shall only come into existence if an authorized 
> representative of Qatar Airways has explicitly approved such contract 
> formation. To the fullest extent permissible by law, Qatar Airways disclaim 
> all liability for loss or damage to person or property arising from this 
> message being infected by computer virus or other contamination.
>  
> 

RE: [External]Re: error while connecting to azure blob storage

[Krishna Chandran Nair]

Please find the attached error

From: Roland Johann 
Sent: 23 August 2019 10:51 AM
To: Krishna Chandran Nair 
Cc: user@spark.apache.org
Subject: [External]Re: error while connecting to azure blob storage

Hi Krishna,

there seems to be no attachment.
In addition, you should NEVER post private credentials to public forums. Please 
renew the credentials of your storage account as soon as possible!

Best Regards

Roland Johann
Software Developer/Data Engineer

phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany

Mobil: +49 172 365 26 46
Mail: roland.joh...@phenetic.io
Web: phenetic.io

Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann




Am 23.08.2019 um 08:33 schrieb Krishna Chandran Nair 
mailto:kcn...@qatarairways.com.qa>>:



Hi Team,

I have written a small code to connect to azure blob storage but go error. I 
have attached the error log.  Please help

Calling command -- ./spark-submit stg.py --jars 
/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar

Code


vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py

from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql import DataFrameReader
from pyspark.sql import SparkSession

session = SparkSession.builder.getOrCreate()


#session.conf.set("fs.azure", 
"org.apache.hadoop.fs.azure.NativeAzureFileSystem")
#session.conf.set("fs.hdfs.impl", 
"org.apache.hadoop.hdfs.DistributedFileSystem")
#session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")



#session.conf.set(
#   "fs.azure.sas.snowflakestrg.blob.core.windows.net/test",
  #  
"?sv=2018-03-28=bfqt=sco=rwdlacup=2020-01-01T16:37:05Z=2019-08-13T08:37:05Z=https=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D"
#)

session.conf.set(
  "fs.azure.account.key.snowflakestrg.blob.core.windows.net",

"LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==")

df=session.read.csv("wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv")

df.show(5)


Qatar Airways - Going Places Together

[OW LOGO]

Disclaimer:- This message (including attachments) is intended solely for the 
addressee named above. It may be confidential, privileged, subject to 
copyright, trade secret, or other legal rules and may not be forwarded without 
the author's permission. If you are not the addressee you must not read, copy 
or disseminate this message. If you have received it in error please notify the 
sender immediately and delete the message from all storage devices. Any 
opinions expressed in this message do not necessarily represent the official 
positions of Qatar Airways. Any agreements (including any warranties, 
representations, or offers) concluded with Qatar Airways by using electronic 
correspondence shall only come into existence if an authorized representative 
of Qatar Airways has explicitly approved such contract formation. To the 
fullest extent permissible by law, Qatar Airways disclaim all liability for 
loss or damage to person or property arising from this message being infected 
by computer virus or other contamination.

Error

--

citus@azcitusclient:~/spark/spark-2.3.3-bin-hadoop2.7/bin$ ./spark-submit 
stg.py --jars 
/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by 
org.apache.hadoop.security.authentication.util.KerberosUtil 
(file:/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-auth-2.7.3.jar) 
to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of 
org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal 
reflective access operations
WARNING: All illegal access operations will be denied in a future release
19/08/19 08:55:31 DEBUG NativeCodeLoader: Trying to load the custom-built 
native-hadoop library...
19/08/19 08:55:31 DEBUG NativeCodeLoader: Failed to load native-hadoop with 
error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path: 
[/usr/java/packages/lib, /usr/lib/x86_64-linux-gnu/jni, /lib/x86_64-linux-gnu, 
/usr/lib/x86_64-linux-gnu, /usr/lib/jni, /lib, /usr/lib]
19/08/19 08:55:31 DEBUG NativeCodeLoader: 
java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
19/08/19 08:55:31 WARN NativeCodeLoader: Unable to load native-hadoop library 
for your platform... using builtin-java classes where applicable
19/08/19 08:55:31 INFO SparkContext: Running Spark version 2.3.3
19/08/19 08:55:31 INFO SparkContext: Submitted application: stg.py
19/08/19 08:55:31 INFO 

Cogrouping in Streaming Datasets/DataFrames is not supported ?

[Kushagra Deep]

Hi ,
I have a use case where I have to cogroup two streams using cogroup in 
streaming. However when I do so I get an exception that “Cogrouping in 
streaming is not supported in DataFrame/Dataset”. Please clarify.



Regards ,
Kushagra Deep

Re: error while connecting to azure blob storage

[Roland Johann]

Hi Krishna,

there seems to be no attachment.
In addition, you should NEVER post private credentials to public forums. Please 
renew the credentials of your storage account as soon as possible!

Best Regards

Roland Johann
Software Developer/Data Engineer

phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany

Mobil: +49 172 365 26 46
Mail: roland.joh...@phenetic.io
Web: phenetic.io

Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann



> Am 23.08.2019 um 08:33 schrieb Krishna Chandran Nair 
> :
> 
>  
>  
> Hi Team,
>  
> I have written a small code to connect to azure blob storage but go error. I 
> have attached the error log.  Please help
>  
> Calling command -- ./spark-submit stg.py --jars 
> /home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar
>  
> Code
>  
>  
> vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py
>  
> from pyspark import SparkContext
> from pyspark.sql import SparkSession
> from pyspark.sql import DataFrameReader
> from pyspark.sql import SparkSession
>  
> session = SparkSession.builder.getOrCreate()
>  
>  
> #session.conf.set("fs.azure", 
> "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> #session.conf.set("fs.hdfs.impl", 
> "org.apache.hadoop.hdfs.DistributedFileSystem")
> #session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")
>  
>  
>  
> #session.conf.set(
> #   "fs.azure.sas.snowflakestrg.blob.core.windows.net/test",
>   #  
> "?sv=2018-03-28=bfqt=sco=rwdlacup=2020-01-01T16:37:05Z=2019-08-13T08:37:05Z=https=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D"
> #)
>  
> session.conf.set(
>   "fs.azure.account.key.snowflakestrg.blob.core.windows.net",
> 
> "LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==")
>  
> df=session.read.csv("wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv
>  ")
>  
> df.show(5)
>  
>  
> Qatar Airways - Going Places Together 
> 
> 
> 
> Disclaimer:- This message (including attachments) is intended solely for the 
> addressee named above. It may be confidential, privileged, subject to 
> copyright, trade secret, or other legal rules and may not be forwarded 
> without the author's permission. If you are not the addressee you must not 
> read, copy or disseminate this message. If you have received it in error 
> please notify the sender immediately and delete the message from all storage 
> devices. Any opinions expressed in this message do not necessarily represent 
> the official positions of Qatar Airways. Any agreements (including any 
> warranties, representations, or offers) concluded with Qatar Airways by using 
> electronic correspondence shall only come into existence if an authorized 
> representative of Qatar Airways has explicitly approved such contract 
> formation. To the fullest extent permissible by law, Qatar Airways disclaim 
> all liability for loss or damage to person or property arising from this 
> message being infected by computer virus or other contamination.

test

[Mayank Agarwal]

RE: error while connecting to azure blob storage

[Krishna Chandran Nair]

Hi Team,

I have written a small code to connect to azure blob storage but go error. I 
have attached the error log.  Please help

Calling command -- ./spark-submit stg.py --jars 
/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-azure-3.2.0.jar,/home/citus/spark/spark-2.3.3-bin-hadoop2.7/jars/azure-storage-8.4.0.jar

Code


vi ~/spark/spark-2.3.3-bin-hadoop2.7/bin/stg.py

from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql import DataFrameReader
from pyspark.sql import SparkSession

session = SparkSession.builder.getOrCreate()


#session.conf.set("fs.azure", 
"org.apache.hadoop.fs.azure.NativeAzureFileSystem")
#session.conf.set("fs.hdfs.impl", 
"org.apache.hadoop.hdfs.DistributedFileSystem")
#session.conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem")



#session.conf.set(
#   "fs.azure.sas.snowflakestrg.blob.core.windows.net/test",
  #  
"?sv=2018-03-28=bfqt=sco=rwdlacup=2020-01-01T16:37:05Z=2019-08-13T08:37:05Z=https=BgTl8mibE%2B%2BTTIMG4dKR17NnGinMWEVTtn888MD8PT4%3D"
#)

session.conf.set(
  "fs.azure.account.key.snowflakestrg.blob.core.windows.net",

"LIWCYzrJOS4hs0DiQH6fAzjuBnuj/F8myVmJImomEqOqlAV4pSt7KWfr24mj0saaOTVNZkGTKUn41k4e9hqKSA==")

df=session.read.csv("wasbs://t...@snowflakestrg.blob.core.windows.net/users.csv")

df.show(5)


Qatar Airways - Going Places Together



Disclaimer:- This message (including attachments) is intended solely for the 
addressee named above. It may be confidential, privileged, subject to 
copyright, trade secret, or other legal rules and may not be forwarded without 
the author's permission. If you are not the addressee you must not read, copy 
or disseminate this message. If you have received it in error please notify the 
sender immediately and delete the message from all storage devices. Any 
opinions expressed in this message do not necessarily represent the official 
positions of Qatar Airways. Any agreements (including any warranties, 
representations, or offers) concluded with Qatar Airways by using electronic 
correspondence shall only come into existence if an authorized representative 
of Qatar Airways has explicitly approved such contract formation. To the 
fullest extent permissible by law, Qatar Airways disclaim all liability for 
loss or damage to person or property arising from this message being infected 
by computer virus or other contamination.