subject:"Re\: classnotfoundexception using zeppelin with spark authentication"

Re: classnotfoundexception using zeppelin with spark authentication

2016-07-28 Thread Jonathan Esterhazy

yes, this PR fixes my classnotfoundexceptions. awesome!

any idea when it might make it into a release?

On Wed, Jul 27, 2016 at 11:25 PM, Jeff Zhang  wrote:

> I think it is fixed in https://github.com/apache/zeppelin/pull/1237,
> please try that.
>
> On Thu, Jul 28, 2016 at 1:58 AM, Jonathan Esterhazy <
> jonathan.esterh...@gmail.com> wrote:
>
>> I just tried this with using zeppelin master, and the classnotfound error
>> is still there.
>>
>>
>> On Thu, Jul 7, 2016 at 7:07 AM, Hyung Sung Shim 
>> wrote:
>>
>>> Hi. Jonathan.
>>> If it is possible, could you use the zeppelin master on the github?
>>> Zeppelin is working well with spark authentication when i tested.
>>>
>>>
>>>
>>> 2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterh...@gmail.com>:
>>>
 Hyung, thx for your help. I opened these:

 https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala
 problem)
 https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking
 python problem)

 LMK if I can provide more info or help in some way.

 On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim 
 wrote:

> Hi Jonathan.
> Unfortunately I got same error in my test bed.
> Do you mind create an jira issue for this?
>
> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <
> jonathan.esterh...@gmail.com>:
>
>> I added this and still get the same exception. The same property is
>> also set in spark-defaults.conf.
>>
>> After that didn't work, I also tried adding --conf
>> spark.authenticate=true and --conf
>> spark.authenticate.enableSaslEncryption=true, to match the other related
>> settings in spark-defaults.conf. Still get the same 
>> classnotfoundexception.
>>
>>
>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim 
>> wrote:
>>
>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and
>>> restart zeppelin and retry your code.
>>>
>>>
>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterh...@gmail.com>:
>>>
 yes it does. i only see this problem in zeppelin.

 On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim  wrote:

> Hi Jonathan.
> It's not easy to build the test environments but I am working on
> this.
> I have question for you.
> Does your code working well on spark-shell in the
> spark.authenticate mode?
>
> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
> jonathan.esterh...@gmail.com>:
>
>> Hyung, did you have any luck w/ zeppelin + spark authentication?
>> I'm quite stumped.
>>
>> thx.
>>
>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <
>> hss...@nflabs.com> wrote:
>>
>>> Thank you.
>>> Let me try.
>>>
>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterh...@gmail.com>:
>>>
 Hyung,

 Yes, here they are.

 zeppelin-env.sh:

 export ZEPPELIN_PORT=8890
 export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
 export ZEPPELIN_LOG_DIR=/var/log/zeppelin
 export ZEPPELIN_PID_DIR=/var/run/zeppelin
 export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
 export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
 export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
 export MASTER=yarn-client
 export SPARK_HOME=/usr/lib/spark
 export HADOOP_CONF_DIR=/etc/hadoop/conf
 export
 CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
 :/usr/share/aws/emr/emrfs/auxlib/*"
 export JAVA_HOME=/usr/lib/jvm/java-1.8.0
 export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
 export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
 export
 ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo

 spark-defaults.conf:

 spark.master yarn
 spark.driver.extraClassPath
  
 /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf

 :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
 spark.driver.extraLibraryPath
  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
 spark.executor.extraClassPath

Re: classnotfoundexception using zeppelin with spark authentication

2016-07-07 Thread Hyung Sung Shim

Hi. Jonathan.
If it is possible, could you use the zeppelin master on the github?
Zeppelin is working well with spark authentication when i tested.



2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy :

> Hyung, thx for your help. I opened these:
>
> https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala problem)
> https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking
> python problem)
>
> LMK if I can provide more info or help in some way.
>
> On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim  wrote:
>
>> Hi Jonathan.
>> Unfortunately I got same error in my test bed.
>> Do you mind create an jira issue for this?
>>
>> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> I added this and still get the same exception. The same property is also
>>> set in spark-defaults.conf.
>>>
>>> After that didn't work, I also tried adding --conf
>>> spark.authenticate=true and --conf
>>> spark.authenticate.enableSaslEncryption=true, to match the other related
>>> settings in spark-defaults.conf. Still get the same classnotfoundexception.
>>>
>>>
>>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim 
>>> wrote:
>>>
 Please add the *export SPARK_SUBMIT_OPTIONS="--conf
 spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
 zeppelin and retry your code.


 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
 jonathan.esterh...@gmail.com>:

> yes it does. i only see this problem in zeppelin.
>
> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim 
> wrote:
>
>> Hi Jonathan.
>> It's not easy to build the test environments but I am working on this.
>> I have question for you.
>> Does your code working well on spark-shell in the spark.authenticate
>> mode?
>>
>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>>> quite stumped.
>>>
>>> thx.
>>>
>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim 
>>> wrote:
>>>
 Thank you.
 Let me try.

 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
 jonathan.esterh...@gmail.com>:

> Hyung,
>
> Yes, here they are.
>
> zeppelin-env.sh:
>
> export ZEPPELIN_PORT=8890
> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
> export ZEPPELIN_PID_DIR=/var/run/zeppelin
> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
> export MASTER=yarn-client
> export SPARK_HOME=/usr/lib/spark
> export HADOOP_CONF_DIR=/etc/hadoop/conf
> export
> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
> :/usr/share/aws/emr/emrfs/auxlib/*"
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
> export
> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>
> spark-defaults.conf:
>
> spark.master yarn
> spark.driver.extraClassPath
>  
> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
> spark.driver.extraLibraryPath
>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
> spark.executor.extraClassPath
>  
> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
> spark.executor.extraLibraryPath
>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
> spark.eventLog.enabled   true
> spark.eventLog.dir   hdfs:///var/log/spark/apps
> spark.history.fs.logDirectoryhdfs:///var/log/spark/apps
> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
> spark.history.ui.port18080
> spark.shuffle.service.enabledtrue
> spark.driver.extraJavaOptions
>  -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
> -XX:MaxHeapFreeRatio=70
>

Re: classnotfoundexception using zeppelin with spark authentication

2016-07-01 Thread Jonathan Esterhazy

Hyung, thx for your help. I opened these:

https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala problem)
https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking python
problem)

LMK if I can provide more info or help in some way.

On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim  wrote:

> Hi Jonathan.
> Unfortunately I got same error in my test bed.
> Do you mind create an jira issue for this?
>
> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy  >:
>
>> I added this and still get the same exception. The same property is also
>> set in spark-defaults.conf.
>>
>> After that didn't work, I also tried adding --conf
>> spark.authenticate=true and --conf
>> spark.authenticate.enableSaslEncryption=true, to match the other related
>> settings in spark-defaults.conf. Still get the same classnotfoundexception.
>>
>>
>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim 
>> wrote:
>>
>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
>>> zeppelin and retry your code.
>>>
>>>
>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterh...@gmail.com>:
>>>
 yes it does. i only see this problem in zeppelin.

 On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim 
 wrote:

> Hi Jonathan.
> It's not easy to build the test environments but I am working on this.
> I have question for you.
> Does your code working well on spark-shell in the spark.authenticate
> mode?
>
> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
> jonathan.esterh...@gmail.com>:
>
>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>> quite stumped.
>>
>> thx.
>>
>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim 
>> wrote:
>>
>>> Thank you.
>>> Let me try.
>>>
>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterh...@gmail.com>:
>>>
 Hyung,

 Yes, here they are.

 zeppelin-env.sh:

 export ZEPPELIN_PORT=8890
 export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
 export ZEPPELIN_LOG_DIR=/var/log/zeppelin
 export ZEPPELIN_PID_DIR=/var/run/zeppelin
 export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
 export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
 export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
 export MASTER=yarn-client
 export SPARK_HOME=/usr/lib/spark
 export HADOOP_CONF_DIR=/etc/hadoop/conf
 export
 CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
 :/usr/share/aws/emr/emrfs/auxlib/*"
 export JAVA_HOME=/usr/lib/jvm/java-1.8.0
 export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
 export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
 export
 ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo

 spark-defaults.conf:

 spark.master yarn
 spark.driver.extraClassPath
  
 /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
 :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
 spark.driver.extraLibraryPath
  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
 spark.executor.extraClassPath
  
 /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
 :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
 spark.executor.extraLibraryPath
  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
 spark.eventLog.enabled   true
 spark.eventLog.dir   hdfs:///var/log/spark/apps
 spark.history.fs.logDirectoryhdfs:///var/log/spark/apps
 spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
 spark.history.ui.port18080
 spark.shuffle.service.enabledtrue
 spark.driver.extraJavaOptions
  -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
 -XX:MaxHeapFreeRatio=70
 -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
 -XX:OnOutOfMemoryError='kill -9 %p'
 spark.dynamicAllocation.enabled  true
 spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails
 -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
 -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM

Re: classnotfoundexception using zeppelin with spark authentication

2016-07-01 Thread Hyung Sung Shim

Hi Jonathan.
Unfortunately I got same error in my test bed.
Do you mind create an jira issue for this?

2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy :

> I added this and still get the same exception. The same property is also
> set in spark-defaults.conf.
>
> After that didn't work, I also tried adding --conf spark.authenticate=true
> and --conf spark.authenticate.enableSaslEncryption=true, to match the other
> related settings in spark-defaults.conf. Still get the same
> classnotfoundexception.
>
>
> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim 
> wrote:
>
>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
>> zeppelin and retry your code.
>>
>>
>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> yes it does. i only see this problem in zeppelin.
>>>
>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim 
>>> wrote:
>>>
 Hi Jonathan.
 It's not easy to build the test environments but I am working on this.
 I have question for you.
 Does your code working well on spark-shell in the spark.authenticate
 mode?

 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
 jonathan.esterh...@gmail.com>:

> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
> quite stumped.
>
> thx.
>
> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim 
> wrote:
>
>> Thank you.
>> Let me try.
>>
>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> Hyung,
>>>
>>> Yes, here they are.
>>>
>>> zeppelin-env.sh:
>>>
>>> export ZEPPELIN_PORT=8890
>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>> export MASTER=yarn-client
>>> export SPARK_HOME=/usr/lib/spark
>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>> export
>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>> export
>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>
>>> spark-defaults.conf:
>>>
>>> spark.master yarn
>>> spark.driver.extraClassPath
>>>  
>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>> spark.driver.extraLibraryPath
>>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>> spark.executor.extraClassPath
>>>  
>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>> spark.executor.extraLibraryPath
>>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>> spark.eventLog.enabled   true
>>> spark.eventLog.dir   hdfs:///var/log/spark/apps
>>> spark.history.fs.logDirectoryhdfs:///var/log/spark/apps
>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
>>> spark.history.ui.port18080
>>> spark.shuffle.service.enabledtrue
>>> spark.driver.extraJavaOptions
>>>  -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>> -XX:MaxHeapFreeRatio=70
>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>> spark.dynamicAllocation.enabled  true
>>> spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails
>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>> spark.executor.memory8640m
>>> spark.executor.cores 7
>>> spark.authenticate.enableSaslEncryption true
>>> spark.driver.memory  1g
>>> spark.network.sasl.serverAlwaysEncrypt true
>>> spark.driver.cores   1
>>> spark.ssl.protocol   TLSv1.2
>>> spark.ssl.keyStorePassword   password
>>>

Re: classnotfoundexception using zeppelin with spark authentication

2016-06-30 Thread Hyung Sung Shim

Please add the *export SPARK_SUBMIT_OPTIONS="--conf
spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
zeppelin and retry your code.


2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy 
:

> yes it does. i only see this problem in zeppelin.
>
> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim 
> wrote:
>
>> Hi Jonathan.
>> It's not easy to build the test environments but I am working on this.
>> I have question for you.
>> Does your code working well on spark-shell in the spark.authenticate mode?
>>
>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>>> quite stumped.
>>>
>>> thx.
>>>
>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim 
>>> wrote:
>>>
 Thank you.
 Let me try.

 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
 jonathan.esterh...@gmail.com>:

> Hyung,
>
> Yes, here they are.
>
> zeppelin-env.sh:
>
> export ZEPPELIN_PORT=8890
> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
> export ZEPPELIN_PID_DIR=/var/run/zeppelin
> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
> export MASTER=yarn-client
> export SPARK_HOME=/usr/lib/spark
> export HADOOP_CONF_DIR=/etc/hadoop/conf
> export
> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
> :/usr/share/aws/emr/emrfs/auxlib/*"
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
> export
> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>
> spark-defaults.conf:
>
> spark.master yarn
> spark.driver.extraClassPath
>  
> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
> spark.driver.extraLibraryPath
>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
> spark.executor.extraClassPath
>  
> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
> spark.executor.extraLibraryPath
>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
> spark.eventLog.enabled   true
> spark.eventLog.dir   hdfs:///var/log/spark/apps
> spark.history.fs.logDirectoryhdfs:///var/log/spark/apps
> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
> spark.history.ui.port18080
> spark.shuffle.service.enabledtrue
> spark.driver.extraJavaOptions
>  -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
> -XX:MaxHeapFreeRatio=70
> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
> -XX:OnOutOfMemoryError='kill -9 %p'
> spark.dynamicAllocation.enabled  true
> spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
> spark.executor.memory8640m
> spark.executor.cores 7
> spark.authenticate.enableSaslEncryption true
> spark.driver.memory  1g
> spark.network.sasl.serverAlwaysEncrypt true
> spark.driver.cores   1
> spark.ssl.protocol   TLSv1.2
> spark.ssl.keyStorePassword   password
> spark.yarn.maxAppAttempts1
> spark.ssl.keyStore   /etc/emr/security/keystore.jks
> spark.authenticate   true
> spark.ssl.keyPasswordpassword
> spark.ssl.enabledtrue
> spark.ssl.enabledAlgorithms  TLS_RSA_WITH_AES_256_CBC_SHA
> spark.ssl.trustStore /etc/emr/security/truststore.jks
> spark.authenticate.secretsecret
> spark.ssl.trustStorePassword password
>
>
>
> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim 
> wrote:
>
>> Hi.
>> Could you share your conf/zeppelin-env.sh and spark-defaults.conf ?
>>
>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> I am having trouble using zeppelin in a spark

Re: classnotfoundexception using zeppelin with spark authentication

2016-06-30 Thread Jonathan Esterhazy

yes it does. i only see this problem in zeppelin.

On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim  wrote:

> Hi Jonathan.
> It's not easy to build the test environments but I am working on this.
> I have question for you.
> Does your code working well on spark-shell in the spark.authenticate mode?
>
> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
> jonathan.esterh...@gmail.com>:
>
>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>> quite stumped.
>>
>> thx.
>>
>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim 
>> wrote:
>>
>>> Thank you.
>>> Let me try.
>>>
>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterh...@gmail.com>:
>>>
 Hyung,

 Yes, here they are.

 zeppelin-env.sh:

 export ZEPPELIN_PORT=8890
 export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
 export ZEPPELIN_LOG_DIR=/var/log/zeppelin
 export ZEPPELIN_PID_DIR=/var/run/zeppelin
 export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
 export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
 export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
 export MASTER=yarn-client
 export SPARK_HOME=/usr/lib/spark
 export HADOOP_CONF_DIR=/etc/hadoop/conf
 export
 CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
 :/usr/share/aws/emr/emrfs/auxlib/*"
 export JAVA_HOME=/usr/lib/jvm/java-1.8.0
 export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
 export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
 export
 ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo

 spark-defaults.conf:

 spark.master yarn
 spark.driver.extraClassPath
  
 /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
 :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
 spark.driver.extraLibraryPath
  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
 spark.executor.extraClassPath
  
 /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
 :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
 spark.executor.extraLibraryPath
  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
 spark.eventLog.enabled   true
 spark.eventLog.dir   hdfs:///var/log/spark/apps
 spark.history.fs.logDirectoryhdfs:///var/log/spark/apps
 spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
 spark.history.ui.port18080
 spark.shuffle.service.enabledtrue
 spark.driver.extraJavaOptions
  -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
 -XX:MaxHeapFreeRatio=70
 -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
 -XX:OnOutOfMemoryError='kill -9 %p'
 spark.dynamicAllocation.enabled  true
 spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails
 -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
 -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
 SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
 spark.executor.memory8640m
 spark.executor.cores 7
 spark.authenticate.enableSaslEncryption true
 spark.driver.memory  1g
 spark.network.sasl.serverAlwaysEncrypt true
 spark.driver.cores   1
 spark.ssl.protocol   TLSv1.2
 spark.ssl.keyStorePassword   password
 spark.yarn.maxAppAttempts1
 spark.ssl.keyStore   /etc/emr/security/keystore.jks
 spark.authenticate   true
 spark.ssl.keyPasswordpassword
 spark.ssl.enabledtrue
 spark.ssl.enabledAlgorithms  TLS_RSA_WITH_AES_256_CBC_SHA
 spark.ssl.trustStore /etc/emr/security/truststore.jks
 spark.authenticate.secretsecret
 spark.ssl.trustStorePassword password



 On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim 
 wrote:

> Hi.
> Could you share your conf/zeppelin-env.sh and spark-defaults.conf ?
>
> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
> jonathan.esterh...@gmail.com>:
>
>> I am having trouble using zeppelin in a spark cluster that has spark
>> node authentication turned on (e.g. with spark.authenticate=true,
>> spark.authenticate.secret=...)
>>
>> Notebook code that calls built-in spark functions (or other things on
>> executor classpath) work fine, but functions defined in the notebook
>> (anonymous or named) throw ClassNotFoundExceptions

Re: classnotfoundexception using zeppelin with spark authentication

2016-06-30 Thread Hyung Sung Shim

Hi Jonathan.
It's not easy to build the test environments but I am working on this.
I have question for you.
Does your code working well on spark-shell in the spark.authenticate mode?

2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy 
:

> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm quite
> stumped.
>
> thx.
>
> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim 
> wrote:
>
>> Thank you.
>> Let me try.
>>
>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> Hyung,
>>>
>>> Yes, here they are.
>>>
>>> zeppelin-env.sh:
>>>
>>> export ZEPPELIN_PORT=8890
>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>> export MASTER=yarn-client
>>> export SPARK_HOME=/usr/lib/spark
>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>> export
>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>> export
>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>
>>> spark-defaults.conf:
>>>
>>> spark.master yarn
>>> spark.driver.extraClassPath
>>>  
>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>> spark.driver.extraLibraryPath
>>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>> spark.executor.extraClassPath
>>>  
>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>> spark.executor.extraLibraryPath
>>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>> spark.eventLog.enabled   true
>>> spark.eventLog.dir   hdfs:///var/log/spark/apps
>>> spark.history.fs.logDirectoryhdfs:///var/log/spark/apps
>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
>>> spark.history.ui.port18080
>>> spark.shuffle.service.enabledtrue
>>> spark.driver.extraJavaOptions
>>>  -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>> -XX:MaxHeapFreeRatio=70
>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>> spark.dynamicAllocation.enabled  true
>>> spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails
>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>> spark.executor.memory8640m
>>> spark.executor.cores 7
>>> spark.authenticate.enableSaslEncryption true
>>> spark.driver.memory  1g
>>> spark.network.sasl.serverAlwaysEncrypt true
>>> spark.driver.cores   1
>>> spark.ssl.protocol   TLSv1.2
>>> spark.ssl.keyStorePassword   password
>>> spark.yarn.maxAppAttempts1
>>> spark.ssl.keyStore   /etc/emr/security/keystore.jks
>>> spark.authenticate   true
>>> spark.ssl.keyPasswordpassword
>>> spark.ssl.enabledtrue
>>> spark.ssl.enabledAlgorithms  TLS_RSA_WITH_AES_256_CBC_SHA
>>> spark.ssl.trustStore /etc/emr/security/truststore.jks
>>> spark.authenticate.secretsecret
>>> spark.ssl.trustStorePassword password
>>>
>>>
>>>
>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim 
>>> wrote:
>>>
 Hi.
 Could you share your conf/zeppelin-env.sh and spark-defaults.conf ?

 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
 jonathan.esterh...@gmail.com>:

> I am having trouble using zeppelin in a spark cluster that has spark
> node authentication turned on (e.g. with spark.authenticate=true,
> spark.authenticate.secret=...)
>
> Notebook code that calls built-in spark functions (or other things on
> executor classpath) work fine, but functions defined in the notebook
> (anonymous or named) throw ClassNotFoundExceptions when called from an
> executor.
>
> For example, this code works:
>
> val rdd = sc.textFile("hdfs://my-text-file")
> rdd.take(1).foreach(println)
>
> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>
> but

Re: classnotfoundexception using zeppelin with spark authentication

2016-06-30 Thread Jonathan Esterhazy

Hyung, did you have any luck w/ zeppelin + spark authentication? I'm quite
stumped.

thx.

On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim  wrote:

> Thank you.
> Let me try.
>
> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
> jonathan.esterh...@gmail.com>:
>
>> Hyung,
>>
>> Yes, here they are.
>>
>> zeppelin-env.sh:
>>
>> export ZEPPELIN_PORT=8890
>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>> export MASTER=yarn-client
>> export SPARK_HOME=/usr/lib/spark
>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>> export
>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>> :/usr/share/aws/emr/emrfs/auxlib/*"
>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>> export
>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>
>> spark-defaults.conf:
>>
>> spark.master yarn
>> spark.driver.extraClassPath
>>  
>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>> spark.driver.extraLibraryPath
>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>> spark.executor.extraClassPath
>>  
>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>> spark.executor.extraLibraryPath
>>  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>> spark.eventLog.enabled   true
>> spark.eventLog.dir   hdfs:///var/log/spark/apps
>> spark.history.fs.logDirectoryhdfs:///var/log/spark/apps
>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
>> spark.history.ui.port18080
>> spark.shuffle.service.enabledtrue
>> spark.driver.extraJavaOptions
>>  -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>> -XX:MaxHeapFreeRatio=70
>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>> -XX:OnOutOfMemoryError='kill -9 %p'
>> spark.dynamicAllocation.enabled  true
>> spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails
>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>> spark.executor.memory8640m
>> spark.executor.cores 7
>> spark.authenticate.enableSaslEncryption true
>> spark.driver.memory  1g
>> spark.network.sasl.serverAlwaysEncrypt true
>> spark.driver.cores   1
>> spark.ssl.protocol   TLSv1.2
>> spark.ssl.keyStorePassword   password
>> spark.yarn.maxAppAttempts1
>> spark.ssl.keyStore   /etc/emr/security/keystore.jks
>> spark.authenticate   true
>> spark.ssl.keyPasswordpassword
>> spark.ssl.enabledtrue
>> spark.ssl.enabledAlgorithms  TLS_RSA_WITH_AES_256_CBC_SHA
>> spark.ssl.trustStore /etc/emr/security/truststore.jks
>> spark.authenticate.secretsecret
>> spark.ssl.trustStorePassword password
>>
>>
>>
>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim 
>> wrote:
>>
>>> Hi.
>>> Could you share your conf/zeppelin-env.sh and spark-defaults.conf ?
>>>
>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterh...@gmail.com>:
>>>
 I am having trouble using zeppelin in a spark cluster that has spark
 node authentication turned on (e.g. with spark.authenticate=true,
 spark.authenticate.secret=...)

 Notebook code that calls built-in spark functions (or other things on
 executor classpath) work fine, but functions defined in the notebook
 (anonymous or named) throw ClassNotFoundExceptions when called from an
 executor.

 For example, this code works:

 val rdd = sc.textFile("hdfs://my-text-file")
 rdd.take(1).foreach(println)

 rdd.saveAsTextFile("hdfs:///my-other-text-file")

 but code like this...

 rdd.filter(_.contains("my data"))

 fails with

 Caused by: java.lang.ClassNotFoundException:
 $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
 at
 org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at

Re: classnotfoundexception using zeppelin with spark authentication

2016-06-28 Thread Jonathan Esterhazy

Hyung,

Yes, here they are.

zeppelin-env.sh:

export ZEPPELIN_PORT=8890
export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
export ZEPPELIN_LOG_DIR=/var/log/zeppelin
export ZEPPELIN_PID_DIR=/var/run/zeppelin
export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
export MASTER=yarn-client
export SPARK_HOME=/usr/lib/spark
export HADOOP_CONF_DIR=/etc/hadoop/conf
export
CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
:/usr/share/aws/emr/emrfs/auxlib/*"
export JAVA_HOME=/usr/lib/jvm/java-1.8.0
export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
export
ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo

spark-defaults.conf:

spark.master yarn
spark.driver.extraClassPath
 
/etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
spark.driver.extraLibraryPath
 /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
spark.executor.extraClassPath
 
/etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
spark.executor.extraLibraryPath
 /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
spark.eventLog.enabled   true
spark.eventLog.dir   hdfs:///var/log/spark/apps
spark.history.fs.logDirectoryhdfs:///var/log/spark/apps
spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
spark.history.ui.port18080
spark.shuffle.service.enabledtrue
spark.driver.extraJavaOptions
 -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
-XX:MaxHeapFreeRatio=70
-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
-XX:OnOutOfMemoryError='kill -9 %p'
spark.dynamicAllocation.enabled  true
spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
spark.executor.memory8640m
spark.executor.cores 7
spark.authenticate.enableSaslEncryption true
spark.driver.memory  1g
spark.network.sasl.serverAlwaysEncrypt true
spark.driver.cores   1
spark.ssl.protocol   TLSv1.2
spark.ssl.keyStorePassword   password
spark.yarn.maxAppAttempts1
spark.ssl.keyStore   /etc/emr/security/keystore.jks
spark.authenticate   true
spark.ssl.keyPasswordpassword
spark.ssl.enabledtrue
spark.ssl.enabledAlgorithms  TLS_RSA_WITH_AES_256_CBC_SHA
spark.ssl.trustStore /etc/emr/security/truststore.jks
spark.authenticate.secretsecret
spark.ssl.trustStorePassword password



On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim  wrote:

> Hi.
> Could you share your conf/zeppelin-env.sh and spark-defaults.conf ?
>
> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy  >:
>
>> I am having trouble using zeppelin in a spark cluster that has spark node
>> authentication turned on (e.g. with spark.authenticate=true,
>> spark.authenticate.secret=...)
>>
>> Notebook code that calls built-in spark functions (or other things on
>> executor classpath) work fine, but functions defined in the notebook
>> (anonymous or named) throw ClassNotFoundExceptions when called from an
>> executor.
>>
>> For example, this code works:
>>
>> val rdd = sc.textFile("hdfs://my-text-file")
>> rdd.take(1).foreach(println)
>>
>> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>>
>> but code like this...
>>
>> rdd.filter(_.contains("my data"))
>>
>> fails with
>>
>> Caused by: java.lang.ClassNotFoundException:
>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>> at
>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:348)
>> ...
>>
>>
>> I get the same kind of error if the filter function is defined as a named
>> function in the notebook, or as a member of singleton object defined in the
>> notebook.
>>
>> When I look at the executor's log output, I see this error:
>>
>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to check
>> existence of class
>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL
>> class server at

Re: classnotfoundexception using zeppelin with spark authentication

2016-06-27 Thread Hyung Sung Shim

Hi.
Could you share your conf/zeppelin-env.sh and spark-defaults.conf ?

2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy :

> I am having trouble using zeppelin in a spark cluster that has spark node
> authentication turned on (e.g. with spark.authenticate=true,
> spark.authenticate.secret=...)
>
> Notebook code that calls built-in spark functions (or other things on
> executor classpath) work fine, but functions defined in the notebook
> (anonymous or named) throw ClassNotFoundExceptions when called from an
> executor.
>
> For example, this code works:
>
> val rdd = sc.textFile("hdfs://my-text-file")
> rdd.take(1).foreach(println)
>
> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>
> but code like this...
>
> rdd.filter(_.contains("my data"))
>
> fails with
>
> Caused by: java.lang.ClassNotFoundException:
> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
> at
> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> ...
>
>
> I get the same kind of error if the filter function is defined as a named
> function in the notebook, or as a member of singleton object defined in the
> notebook.
>
> When I look at the executor's log output, I see this error:
>
> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to check
> existence of class
> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on REPL
> class server at https://172.30.54.30:34980
> java.lang.NullPointerException
> at
> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
> at
> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
> at
> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
>
> ...
>
>
> If I disable spark authentication, everything works as expected. I am
> running zeppelin 0.5.6 on spark 1.6.1 with yarn.
>
> Has anyone been able to get zeppelin working with spark authentication?
>
>
>

Re: classnotfoundexception using zeppelin with spark authentication

Re: classnotfoundexception using zeppelin with spark authentication

Re: classnotfoundexception using zeppelin with spark authentication

Re: classnotfoundexception using zeppelin with spark authentication

Re: classnotfoundexception using zeppelin with spark authentication

Re: classnotfoundexception using zeppelin with spark authentication

Re: classnotfoundexception using zeppelin with spark authentication

Re: classnotfoundexception using zeppelin with spark authentication

Re: classnotfoundexception using zeppelin with spark authentication

Re: classnotfoundexception using zeppelin with spark authentication

10 matches

Site Navigation

Mail list logo

Footer information