Zeppelin Pulling Files from S3 That Are KMS Encrypted Denied

[Tseytlin, Keren]

Hi All,

I have a bucket that I’m working with and I want to pull orc files from there 
and use it in my Spark/Scala magic. The only thing is that these files are KMS 
encrypted. When I try to get a KMS file however, it shows me an AWS Access 
Denied error, although there is no possible way that could be true.

First, I can successfully pull a file from that directory (this file below is 
SSE encrypted):

val orctest8 = 
sqlContext.read.format("orc").load("s3a://kafkaTest/connect-prefix/BLUECOATCIM/2016/10/06/FILE123.orc")
orctest8.count()

res149: Long = 17

Then, I try to pull a file from that directory that is KMS encrypted:

val orctest8 = 
sqlContext.read.format("orc").load("s3a://kafkaTest/connect-prefix/BLUECOATCIM/2016/10/06/FILEXYZ.orc")
orctest8.count()

java.nio.file.AccessDeniedException: 
s3a://kafkaTest/connect-prefix/BLUECOATCIM/2016/10/06/FILEXYZ.orc: Reopen at 
position 1011102 on 
s3a://kafkaTest/connect-prefix/BLUECOATCIM/2016/10/06/FILEXYZ.orc: 
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: 
Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: )

As you can see above, both of the files are originating from the same 
directory. They are also the same exact file, except one is KMS encrypted and 
the other is SSE. Does Zeppelin currently have the capability to read in KMS 
encrypted files and use them? If so, what parameters need to be set?

Best,
Keren



The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

Zeppelin with Separate Spark Connection

[Tseytlin, Keren]

Hi All,

I’ve just set up Zeppelin, and I’ve also set up my own Spark with connection to 
Alluxio. I installed Zeppelin using the binary. When I use Zeppelin, it seems 
to be using some internal Spark, not the one that I set up. What configurations 
should I set in order to make the notebooks and Spark jobs execute on my own 
Spark?

I edited zeppelin-env.sh and added SPARK_HOME, but that caused anything I tried 
to run in my notebook just shoot back “ERROR” with no output.

Any help would be much appreciated! Thanks!!

Best,
Keren


The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.