Zeppelin with Oauth2

[Kang Minwoo]

Hello

Can I use Oauth2 in Zeppelin? (or any Idea?)
If there is no way, Can I contribute that?

Best regards,
Minwoo Kang

Re: Getting 401 With secure websockets on AWS

[Khalid Huseynov]

Hi Michael,

Which version of Zeppelin are you using?

On Sat, Apr 22, 2017 at 3:10 AM, Knapp, Michael <
michael.kn...@capitalone.com> wrote:

> Hi,
>
>
>
> I am getting a 401, unauthorized, with all secure (and non-secure)
> websocket calls while running on AWS.  I have configured the server
> properly to use a signed certificate, I have tested and all HTTPS calls are
> successful.  I also have LDAP working.  Still, the websocket calls are all
> failing.
>
>
>
> I have a security group setup, it allows inbound TCP traffic over port 443
> from all sources (0.0.0.0/0).  I believe that alone should have been
> adequate.
>
>
>
> I set JAVA_OPTS=’-Djavax.net.debug=all’ and watched the standard output.
> Every time the websockets attempt to connect, I see these printed to
> standard out:
>
>
>
> qtp1622006612-39, called closeInbound()
>
> qtp1622006612-39, fatal error: 80: Inbound closed before receiving peer's
> close_notify: possible truncation attack?
>
> javax.net.ssl.SSLException: Inbound closed before receiving peer's
> close_notify: possible truncation attack?
>
> %% Invalidated:  [Session-50, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
>
> qtp1622006612-39, SEND TLSv1.2 ALERT:  fatal, description = internal_error
>
> qtp1622006612-39, WRITE: TLSv1.2 Alert, length = 2
>
>
>
> Nothing is written the the log file when these websocket requests fail.
>
>
>
> Can somebody please tell me why this is still not working?
>
>
>
> Michael Knapp
>
> --
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Re: Zeppelin with Oauth2

[Alexander Bezzubov]

Hi Minwoo,

great question, AFAIK, there is no way right now.

But please, feel free to create and Issue and post the link here (or
starting a thread in Dev mailing list [0] could be good idea) and then, as
soon as you have PoC with the code working, submit a PR with the code and
some documentation on how to use\enable it!

I assume you are talking about user authentication, so you might want to
use existing Apache Shiro [1] facilities inside Zeppelin.

Good luck!

 0. http://zeppelin.apache.org/community.html#mailing-list
 1. https://github.com/apache/zeppelin/blob/master/docs/
security/shiroauthentication.md


--
Alex

On Thu, May 4, 2017 at 9:54 AM, Kang Minwoo  wrote:

> Hello
>
> Can I use Oauth2 in Zeppelin? (or any Idea?)
> If there is no way, Can I contribute that?
>
> Best regards,
> Minwoo Kang
>

RE: Zeppelin with Oauth2

[Kang Minwoo]

Hello

Okay, Thanks for the reply.

I think if ouath2 supported, it is easy to integrate In-house system.
I create an issue [0].

Best regards,
Minwoo Kang

0. https://issues.apache.org/jira/browse/ZEPPELIN-2495

보낸 사람: Alexander Bezzubov 
보낸 날짜: 2017년 5월 4일 목요일 오후 5:48:27
받는 사람: users
제목: Re: Zeppelin with Oauth2

Hi Minwoo,

great question, AFAIK, there is no way right now.

But please, feel free to create and Issue and post the link here (or starting a 
thread in Dev mailing list [0] could be good idea) and then, as soon as you 
have PoC with the code working, submit a PR with the code and some 
documentation on how to use\enable it!

I assume you are talking about user authentication, so you might want to use 
existing Apache Shiro [1] facilities inside Zeppelin.

Good luck!

 0. http://zeppelin.apache.org/community.html#mailing-list
 1. 
https://github.com/apache/zeppelin/blob/master/docs/security/shiroauthentication.md


--
Alex

On Thu, May 4, 2017 at 9:54 AM, Kang Minwoo 
mailto:minwoo.k...@outlook.com>> wrote:
Hello

Can I use Oauth2 in Zeppelin? (or any Idea?)
If there is no way, Can I contribute that?

Best regards,
Minwoo Kang

Re: Zeppelin with Oauth2

[Jaideep Singh]

Hello,
Its possible, you can do it by pac4j library which provides oauth2 support,
read in more details about that.

Thanks,
Jaideep Singh

On Thursday, May 4, 2017, Kang Minwoo  wrote:

> Hello
>
> Okay, Thanks for the reply.
>
> I think if ouath2 supported, it is easy to integrate In-house system.
> I create an issue [0].
>
> Best regards,
> Minwoo Kang
>
> 0. https://issues.apache.org/jira/browse/ZEPPELIN-2495
> 
> 보낸 사람: Alexander Bezzubov >
> 보낸 날짜: 2017년 5월 4일 목요일 오후 5:48:27
> 받는 사람: users
> 제목: Re: Zeppelin with Oauth2
>
> Hi Minwoo,
>
> great question, AFAIK, there is no way right now.
>
> But please, feel free to create and Issue and post the link here (or
> starting a thread in Dev mailing list [0] could be good idea) and then, as
> soon as you have PoC with the code working, submit a PR with the code and
> some documentation on how to use\enable it!
>
> I assume you are talking about user authentication, so you might want to
> use existing Apache Shiro [1] facilities inside Zeppelin.
>
> Good luck!
>
>  0. http://zeppelin.apache.org/community.html#mailing-list
>  1. https://github.com/apache/zeppelin/blob/master/docs/
> security/shiroauthentication.md
>
>
> --
> Alex
>
> On Thu, May 4, 2017 at 9:54 AM, Kang Minwoo  > wrote:
> Hello
>
> Can I use Oauth2 in Zeppelin? (or any Idea?)
> If there is no way, Can I contribute that?
>
> Best regards,
> Minwoo Kang
>
>

Re: Zeppelin not loading the index page after redirection from IDP

[Jaideep Singh]

Hello Paul,

Thanks for your support, i am able to prototype it, as my shiro was not
updated, it resolved after updation to latest version.

Thanks,
Jaideep Singh

On Wednesday, May 3, 2017, Paul Brenner  wrote:

> Unfortunately I haven't seen a ton of Shiro expertise on this list. Maybe
> someone will know the answer to your problem but my guess is that you are
> going to have to troubleshoot this by stripping out all that fancy
> complexity until you get a basic shiro.ini that works and then methodically
> add pieces back in until you see what is breaking. Once you know what is
> going on we would all appreciate your help adding to the documentation for
> using shiro with zeppelin.
>
>  
>  Paul Brenner 
>  
>  
> 
> 
> DATA SCIENTIST
> *(217) 390-3033 <(217)%20390-3033> *
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> [image:
> PlaceIQ:Location Data Accuracy]
> 
>
>
> On Wed, May 03, 2017 at 8:36 AM Jaideep Singh  >
> wrote:
>
>> Also attaching the screen shot for 2 JSession id which i got after
>> redirection.
>>
>> On Wed, May 3, 2017 at 5:18 PM, Jaideep Singh > > wrote:
>>
>>> Hello,
>>>
>>> I have used saml based sso authentication on zeppelin url which is on
>>> localhost:8080. I am able to load the zeppelin page successfully if i
>>> disable the shiro.ini file. I have used sso authentication with wso2,
>>> configured in shiro.ini with metadata for idp and sp provided there.
>>> But after redirection from idp to zeppelin / url i am not able to load
>>> the page.
>>>
>>> Following are the assumption for problem occurence
>>> ***  Problem may be due to the websocket calls which are not initiating
>>> after redirection, but i can see it works if no authentication applied.
>>> ** *I am getting JSessionid after redirection from IDP. Is Zeppelin
>>> server also providing JSessionid which may cause conflicts?
>>>
>>> Plese help me to identify the problem.
>>>
>>> I am attaching the log file and shiro.ini.
>>>
>>> I have checked the log file the error i am getting is
>>>
>>>
>>> 17:01:05.402 [qtp1663619914-60 - /;JSESSIONID=804affc8-ea2c-40a
>>> d-9db8-0492c9f1f134/api/security/ticket] DEBUG
>>> o.e.jetty.servlet.ServletHandler - chain=org.apache.zeppelin.serv
>>> er.CorsFilter-5ae50ce6->ShiroFilter->org.eclipse.jetty.
>>> servlet.DefaultServlet-69b2283a@5b910f06==org.eclipse.jetty.
>>> servlet.DefaultServlet,-1,true
>>> 17:01:05.402 [qtp1663619914-60 - /;JSESSIONID=804affc8-ea2c-40a
>>> d-9db8-0492c9f1f134/api/security/ticket] DEBUG
>>> o.e.jetty.servlet.ServletHandler - call filter
>>> org.apache.zeppelin.server.CorsFilter-5ae50ce6
>>> 17:01:05.402 [qtp1663619914-60 - /;JSESSIONID=804affc8-ea2c-40a
>>> d-9db8-0492c9f1f134/api/security/ticket] DEBUG
>>> o.e.jetty.servlet.ServletHandler - call filter ShiroFilter
>>> 17:01:05.403 [qtp1663619914-60 - /;JSESSIONID=804affc8-ea2c-40a
>>> d-9db8-0492c9f1f134/api/security/ticket] DEBUG
>>> o.a.shiro.mgt.DefaultSecurityManager - Resolved SubjectContext context
>>> session is invalid.  Ignoring and creating an anonymous (session-less)
>>> Subject instance.
>>> org.apache.shiro.session.UnknownSessionException: There is no session
>>> with id [8

Unable to run Zeppelin Spark on YARN

[Yeshwanth Jagini]

Hi we are running cloudera CDH 5.9.1 .

while setting up zeppelin, i followed the documentation on website and
specified following options

export ZEPPELIN_JAVA_OPTS="-Dhadoop.version=2.6.0-cdh5.9.1"
# Additional jvm options. for example, export
ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g -Dspark.cores.max=16"

export
SPARK_HOME="/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/spark"
   # (required) When it is defined, load it instead of
Zeppelin embedded Spark libraries
export SPARK_SUBMIT_OPTIONS="--master yarn --deploy-mode client"
# (optional) extra options to pass to spark submit. eg)
"--driver-memory 512M --executor-memory 1G".
export SPARK_APP_NAME=Zeppelin # (optional) The
name of spark application.

export HADOOP_CONF_DIR=/etc/hadoop/conf #
yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.

export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER}
bash -c'   # Optional, when user want to run interpreter as end web
user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '

when running spark notebook, spark-submit is running in local mode and i
cannot see the application in yarn resource manager.
is there any other configuration i am missing?


Thanks,
Yeshwanth Jagini

what causes InterpreterException: Host key verification failed

[Yeshwanth Jagini]

Any idea why i am running into this issue,

INFO [2017-05-04 22:02:24,996] ({pool-2-thread-2}
SchedulerFactory.java[jobFinished]:137) - Job
paragraph_1423500779206_-1502780787 finished by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter2A94M5J1Z256540498
 INFO [2017-05-04 22:03:02,450] ({pool-2-thread-4}
SchedulerFactory.java[jobStarted]:131) - Job
paragraph_1423500779206_-1502780787 started by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter2A94M5J1Z256540498
 INFO [2017-05-04 22:03:02,451] ({pool-2-thread-4}
Paragraph.java[jobRun]:362) - run paragraph 20150210-015259_1403135953
using null org.apache.zeppelin.interpreter.LazyOpenInterpreter@5aa74c91
 INFO [2017-05-04 22:03:02,451] ({pool-2-thread-4}
RemoteInterpreterManagedProcess.java[start]:126) - Run interpreter process
[/opt/zeppelin-0.7.1-bin-all/bin/interpreter.sh, -d,
/opt/zeppelin-0.7.1-bin-all/interpreter/spark, -p, 37115, -u, bigdata, -l,
/opt/zeppelin-0.7.1-bin-all/local-repo/2CFWU98CR]
 INFO [2017-05-04 22:03:02,513] ({Exec Default Executor}
RemoteInterpreterManagedProcess.java[onProcessComplete]:180) - Interpreter
process exited 0
ERROR [2017-05-04 22:03:02,980] ({pool-2-thread-4} Job.java[run]:188) - Job
failed
org.apache.zeppelin.interpreter.InterpreterException: Host key verification
failed.

at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:143)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.reference(RemoteInterpreterProcess.java:73)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:258)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:423)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:106)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:387)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
ERROR [2017-05-04 22:03:02,981] ({pool-2-thread-4}
NotebookServer.java[afterStatusChange]:2050) - Error
org.apache.zeppelin.interpreter.InterpreterException: Host key verification
failed.

at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:143)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.reference(RemoteInterpreterProcess.java:73)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:258)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:423)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:106)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:387)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
 WARN [2017-05-04 22:03:02,982] ({pool-2-thread-4}
NotebookServer.java[afterStatusChange]:2058) - Job
20150210-015259_1403135953 is finished, status: ERROR, exception:
org.apache.zeppelin.interpreter.InterpreterException: Host key verification
failed.
, result: Host key verification failed.

 INFO [2017-05-04 22:03:03,042] ({pool-2-thread-4}
SchedulerFactory.java[jobFinished]:137) - Job
paragraph_1423500779206_-1502780787 finished by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter2A94M5J1Z256540498


Thanks,
Yeshwanth Jagini

Re: Unable to run Zeppelin Spark on YARN

[Jianfeng (Jeff) Zhang]

Could you try set yarn-client in interpreter setting page ?


Best Regard,
Jeff Zhang


From: Yeshwanth Jagini mailto:y...@yotabitesllc.com>>
Reply-To: "users@zeppelin.apache.org" 
mailto:users@zeppelin.apache.org>>
Date: Friday, May 5, 2017 at 3:13 AM
To: "users@zeppelin.apache.org" 
mailto:users@zeppelin.apache.org>>
Subject: Unable to run Zeppelin Spark on YARN

Hi we are running cloudera CDH 5.9.1 .

while setting up zeppelin, i followed the documentation on website and 
specified following options

export ZEPPELIN_JAVA_OPTS="-Dhadoop.version=2.6.0-cdh5.9.1" 
# Additional jvm options. for example, export 
ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g -Dspark.cores.max=16"

export SPARK_HOME="/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/spark"   
 # (required) When it is defined, load it instead of 
Zeppelin embedded Spark libraries
export SPARK_SUBMIT_OPTIONS="--master yarn --deploy-mode client"
   # (optional) extra options to pass to spark submit. eg) "--driver-memory 
512M --executor-memory 1G".
export SPARK_APP_NAME=Zeppelin # (optional) The name of 
spark application.

export HADOOP_CONF_DIR=/etc/hadoop/conf # yarn-site.xml 
is located in configuration directory in HADOOP_CONF_DIR.

export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash 
-c'   # Optional, when user want to run interpreter as end web user. eg) 
'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '

when running spark notebook, spark-submit is running in local mode and i cannot 
see the application in yarn resource manager.
is there any other configuration i am missing?


Thanks,
Yeshwanth Jagini

R dataframes printing as plain text

[Mark Miller]

I'm new to Zeppelin, Spark and Scala

According to https://zeppelin.apache.org/docs/0.6.2/interpreter/r.html,
"If you return a data.frame, Zeppelin will attempt to display it using
Zeppelin's built-in visualizations"

But I am just getting a plain text dump from "mtcars"

What am I doing wrong?

thanks,
Mark

---   ---   ---

I installed zeppelin-0.7.1-bin-all.tgz on an Ubuntu 16 system with openjdk
version "1.8.0_121"

I installed spark-2.1.1-bin-hadoop2.7.tgz in /usr/local/share/spark

i.e., I can run
$ /usr/local/share/spark/spark-2.1.1-bin-hadoop2.7/bin/run-example SparkPi
10

I edited zeppelin-env.sh to say:
export SPARK_HOME=/usr/local/share/spark/spark-2.1.1-bin-hadoop2.7

I did
$ cp zeppelin-site.xml.template zeppelin-site.xml

and
$ ./bin/zeppelin-daemon.sh restart

Re: R dataframes printing as plain text

[Jeff Zhang]

I don't think it support R's dataframe. Not sure why the document says
it works, may it works before.

https://issues.apache.org/jira/browse/ZEPPELIN-1181





Mark Miller 于2017年5月5日周五 上午9:43写道：

> I'm new to Zeppelin, Spark and Scala
>
> According to https://zeppelin.apache.org/docs/0.6.2/interpreter/r.html,
> "If you return a data.frame, Zeppelin will attempt to display it using
> Zeppelin's built-in visualizations"
>
> But I am just getting a plain text dump from "mtcars"
>
> What am I doing wrong?
>
> thanks,
> Mark
>
> ---   ---   ---
>
> I installed zeppelin-0.7.1-bin-all.tgz on an Ubuntu 16 system with openjdk
> version "1.8.0_121"
>
> I installed spark-2.1.1-bin-hadoop2.7.tgz in /usr/local/share/spark
>
> i.e., I can run
> $ /usr/local/share/spark/spark-2.1.1-bin-hadoop2.7/bin/run-example SparkPi
> 10
>
> I edited zeppelin-env.sh to say:
> export SPARK_HOME=/usr/local/share/spark/spark-2.1.1-bin-hadoop2.7
>
> I did
> $ cp zeppelin-site.xml.template zeppelin-site.xml
>
> and
> $ ./bin/zeppelin-daemon.sh restart
>
>

Re: Zeppelin with Oauth2

[Kang Minwoo]

Thanks for letting me know.
I will try that.

Best regards,
Minwoo Kang

From: Jaideep Singh 
Sent: Thursday, May 4, 2017 10:13:32 PM
To: users@zeppelin.apache.org
Subject: Re: Zeppelin with Oauth2

Hello,
Its possible, you can do it by pac4j library which provides oauth2 support, 
read in more details about that.

Thanks,
Jaideep Singh

On Thursday, May 4, 2017, Kang Minwoo 
mailto:minwoo.k...@outlook.com>> wrote:
Hello

Okay, Thanks for the reply.

I think if ouath2 supported, it is easy to integrate In-house system.
I create an issue [0].

Best regards,
Minwoo Kang

0. https://issues.apache.org/jira/browse/ZEPPELIN-2495

보낸 사람: Alexander Bezzubov 
보낸 날짜: 2017년 5월 4일 목요일 오후 5:48:27
받는 사람: users
제목: Re: Zeppelin with Oauth2

Hi Minwoo,

great question, AFAIK, there is no way right now.

But please, feel free to create and Issue and post the link here (or starting a 
thread in Dev mailing list [0] could be good idea) and then, as soon as you 
have PoC with the code working, submit a PR with the code and some 
documentation on how to use\enable it!

I assume you are talking about user authentication, so you might want to use 
existing Apache Shiro [1] facilities inside Zeppelin.

Good luck!

 0. http://zeppelin.apache.org/community.html#mailing-list
 1. 
https://github.com/apache/zeppelin/blob/master/docs/security/shiroauthentication.md


--
Alex

On Thu, May 4, 2017 at 9:54 AM, Kang Minwoo 
mailto:minwoo.k...@outlook.com>> wrote:
Hello

Can I use Oauth2 in Zeppelin? (or any Idea?)
If there is no way, Can I contribute that?

Best regards,
Minwoo Kang

Recent Improvements Apache Zeppelin Livy Integration

[Jeff Zhang]

For anyone that is using or interested in livy interpreter

https://hortonworks.com/blog/recent-improvements-apache-zeppelin-livy-integration/

Re: Recent Improvements Apache Zeppelin Livy Integration

[Ruslan Dautkhanov]

Thanks for sharing this Jeff!

Once Zeppelin supports yarn-cluster, what would be main benefits of using
Livy Spark interpreters, instead of just the Spark interpreters?



-- 
Ruslan Dautkhanov

On Thu, May 4, 2017 at 10:51 PM, Jeff Zhang  wrote:

> For anyone that is using or interested in livy interpreter
>
> https://hortonworks.com/blog/recent-improvements-apache-
> zeppelin-livy-integration/
>
>
>