date:20170509

Re: User Impersonation Configuration

2017-05-09 Thread Prabhjyot Singh

Hi Yeshwant,

Which version of Zeppelin are you on?

If you are on latest then you don't need to do any of ZEPPELIN_IMPERSONATE_CMD
or ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER. Just by enabling User
Impersonation check-box should be sufficient.

Can you confirm by `ps aux | grep spark`. This is what I see on my machine;

prabhjyotsingh@MACHINE:~/ps-zeppelin/logs$ ps aux | grep spark
prabhjyotsingh2496   0.2  3.9  5179540 657660 s000  S12:08PM
0:30.68
/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/bin/java
-cp
/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/*:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/lib/*:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/classes/:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/test-classes/:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-zengine/target/test-classes/:/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar:/Users/prabhjyotsingh/spark-2.0.0-bin-hadoop2.7/conf/:/Users/prabhjyotsingh/spark-2.0.0-bin-hadoop2.7/jars/*
-Xmx1g -Dfile.encoding=UTF-8
-Dlog4j.configuration=file:///Users/prabhjyotsingh/ps-zeppelin/conf/log4j.properties
-Dzeppelin.log.file=/Users/prabhjyotsingh/ps-zeppelin/logs/zeppelin-interpreter-spark-user1-spark-prabhjyotsingh-HW11610.local.log
org.apache.spark.deploy.SparkSubmit --conf
spark.driver.extraClassPath=:/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/*:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/lib/*::/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/classes:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/test-classes:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-zengine/target/test-classes:/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar
--conf spark.driver.extraJavaOptions= -Dfile.encoding=UTF-8
-Dlog4j.configuration=file:///Users/prabhjyotsingh/ps-zeppelin/conf/log4j.properties
-Dzeppelin.log.file=/Users/prabhjyotsingh/ps-zeppelin/logs/zeppelin-interpreter-spark-user1-spark-prabhjyotsingh-HW11610.local.log
--class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer
*--proxy-user
user1*
/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar
50911
prabhjyotsingh2508   0.0  0.0  2445100860 s000  S+   12:08PM
0:00.00 grep spark
prabhjyotsingh2495   0.0  0.0  2465144764 s000  S12:08PM
0:00.00 /bin/bash /Users/prabhjyotsingh/ps-zeppelin/bin/interpreter.sh -d
/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark -p 50911 -u user1 -l
/Users/prabhjyotsingh/ps-zeppelin/local-repo/2CEZC4JXN -g spark
prabhjyotsingh2484   0.0  0.0  2465144   1368 s000  S12:08PM
0:00.01 /bin/bash /Users/prabhjyotsingh/ps-zeppelin/bin/interpreter.sh -d
/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark -p 50911 -u user1 -l
/Users/prabhjyotsingh/ps-zeppelin/local-repo/2CEZC4JXN -g spark

Ref: https://github.com/apache/zeppelin/pull/1840


On 10 May 2017 at 06:10, Yeshwanth Jagini  wrote:

> Hi Users,
>
> I am trying to setup Zeppelin for multiple users.
> and i found there are multiple configurations in different places. started
> tinkering with them and i didn't had any luck .
> here's my setup and configuration.
>
> Zeppelin server is running as root.
>
> i edited zeppelin-env.sh  and uncommented
>
> export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER}
> bash -c'   # Optional, when user want to run interpreter as end web
> user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '
> export ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER=true  #Optional, by default
> is true; can be set to false if you don't want to use --proxy-user option
> with Spark interpreter when impersonation enabled
>
> and  the interpreter settings are as follows
>
> [image: Inline image 1]
>
>
> for different combination of configurations i am getting different types
> of errors
>
> if i do not specify impersonation configuration in zeppelin-env.sh  and
> specify impersonation in interpreter setting i am getting a
> org.apache.zeppelin.interpreter.InterpreterException: Host key
> verification failed.
> if i specify both zeppelin impersonation configuration and interpreter
> impersonation config, it's throwing error as user1 cannot impersonate user1
>
> if i do not specify any impersonation configuration at all , interpreter
> is launching spark-submit as root. that's expected.
>
> could some one please explain me how to set impersonation config and
> which configuration i am messing up here
>
>
> Thanks,
> Yeshwanth Jagini
>



-- 

Warm Regards,

Prabhjyot Singh

Re: Hive Reserve Keyword support

2017-05-09 Thread Dibyendu Bhattacharya

right. This backticks worked

On Wed, May 10, 2017 at 8:51 AM, Felix Cheung 
wrote:

> I think you can put backticks around the name date
>
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL
>
> --
> *From:* Jongyoul Lee 
> *Sent:* Tuesday, May 9, 2017 10:33:50 AM
> *To:* users@zeppelin.apache.org
> *Subject:* Re: Hive Reserve Keyword support
>
> If it's possible for you to pass that properties when you create a
> connection, you can passes it by setting it into interpreter setting
>
> On Sat, Apr 29, 2017 at 4:25 PM, Dibyendu Bhattacharya <
> dibyendu.bhattach...@gmail.com> wrote:
>
>> Hi,
>>
>> I have a Hive Table which has a column named date. When I tried to query
>> using Zeppelin %jdbc interpreter , I got bellow error.
>>
>>
>> Error while compiling statement: FAILED: ParseException line 1:312 Failed
>> to recognize predicate 'date'. Failed rule: 'identifier' in expression
>> specification
>> class org.apache.hive.service.cli.HiveSQLException
>> org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:231)
>> org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:217)
>> org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254)
>> org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(JDBCInte
>> rpreter.java:322)
>> org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(JDBCInter
>> preter.java:408)
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.
>> interpret(LazyOpenInterpreter.java:94)
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServ
>> er$InterpretJob.jobRun(RemoteInterpreterServer.java:341)
>> org.apache.zeppelin.scheduler.Job.run(Job.java:176)
>> org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.ru
>> n(ParallelScheduler.java:162)
>>
>>
>> My query looks like this :
>>
>> select x,y,z from mytable where date = '2017-04-28"
>>
>> I believe it is failing because date is reserve keyword . Is there anyway
>> I can set  hive.support.sql11.reserved.keywords=false in Zeppelin ?
>>
>> regards,
>> Dibyendu
>>
>>
>>
>>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>

Re: Hive Reserve Keyword support

2017-05-09 Thread Felix Cheung

I think you can put backticks around the name date

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL

From: Jongyoul Lee 
Sent: Tuesday, May 9, 2017 10:33:50 AM
To: users@zeppelin.apache.org
Subject: Re: Hive Reserve Keyword support

If it's possible for you to pass that properties when you create a connection, 
you can passes it by setting it into interpreter setting

On Sat, Apr 29, 2017 at 4:25 PM, Dibyendu Bhattacharya 
mailto:dibyendu.bhattach...@gmail.com>> wrote:
Hi,

I have a Hive Table which has a column named date. When I tried to query using 
Zeppelin %jdbc interpreter , I got bellow error.

Error while compiling statement: FAILED: ParseException line 1:312 Failed to 
recognize predicate 'date'. Failed rule: 'identifier' in expression 
specification
class org.apache.hive.service.cli.HiveSQLException
org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:231)
org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:217)
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254)
org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(JDBCInterpreter.java:322)
org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(JDBCInterpreter.java:408)
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341)
org.apache.zeppelin.scheduler.Job.run(Job.java:176)
org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelScheduler.java:162)

My query looks like this :

select x,y,z from mytable where date = '2017-04-28"

I believe it is failing because date is reserve keyword . Is there anyway I can 
set  hive.support.sql11.reserved.keywords=false in Zeppelin ?

regards,
Dibyendu

--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

User Impersonation Configuration

2017-05-09 Thread Yeshwanth Jagini

Hi Users,

I am trying to setup Zeppelin for multiple users.
and i found there are multiple configurations in different places. started
tinkering with them and i didn't had any luck .
here's my setup and configuration.

Zeppelin server is running as root.

i edited zeppelin-env.sh  and uncommented

export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER}
bash -c'   # Optional, when user want to run interpreter as end web
user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '
export ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER=true  #Optional, by default is
true; can be set to false if you don't want to use --proxy-user option with
Spark interpreter when impersonation enabled

and  the interpreter settings are as follows

[image: Inline image 1]


for different combination of configurations i am getting different types of
errors

if i do not specify impersonation configuration in zeppelin-env.sh  and
specify impersonation in interpreter setting i am getting a
org.apache.zeppelin.interpreter.InterpreterException:
Host key verification failed.
if i specify both zeppelin impersonation configuration and interpreter
impersonation config, it's throwing error as user1 cannot impersonate user1

if i do not specify any impersonation configuration at all , interpreter is
launching spark-submit as root. that's expected.

could some one please explain me how to set impersonation config and
which configuration i am messing up here


Thanks,
Yeshwanth Jagini

Re: ZeppelinContext textbox for passwords

2017-05-09 Thread Ruslan Dautkhanov

In pyspark with Jupyter we used to do
getpass.getpass "Prompt the user for a password without echoing"
https://docs.python.org/2/library/getpass.html
but with Zeppelin Spark Interpreter wouldn't pass interactive request
to pyspark REPL - it actually makes spark interpreter paragraph hang.

I was thinking to submit a jira for this a while back.
Not sure how hard it would be to add in Zeppelin.

Here's how it looks in Jupyter (Jupyer actually displays an interactive
prompt under the paragraph):

[image: Inline image 1]

Here's how it looks in Zeppelin (after canceling execution as it gets
stuck):

[image: Inline image 2]

I think if Zeppelin could understand that there is an interactive prompt,
this will be helpful not only with password prompts but any other cases
(including shell interpreter).

-- 
Ruslan Dautkhanov

On Tue, May 9, 2017 at 4:59 PM, Ben Vogan  wrote:

> Hi there,
>
> Is it possible to create a textbox for accepting passwords via the
> ZeppelinContext (i.e. one that masks input)?  I do not see any way to do
> so, but I hope I'm missing something.
>
> Thanks,
>
> --
> *BENJAMIN VOGAN* | Data Platform Team Lead
>
> 
>  
>  
> 
>

Re: ZeppelinContext textbox for passwords

2017-05-09 Thread Jeff Zhang

It is not possible for now, but I already made the dynamic forms
extensible, should be able to create password textbox easily. You can
create ticket for it.




Ben Vogan 于2017年5月9日周二 下午3:59写道：

> Hi there,
>
> Is it possible to create a textbox for accepting passwords via the
> ZeppelinContext (i.e. one that masks input)?  I do not see any way to do
> so, but I hope I'm missing something.
>
> Thanks,
>
>
> --
> *BENJAMIN VOGAN* | Data Platform Team Lead
>
> 
>  
>  
> 
>

ZeppelinContext textbox for passwords

2017-05-09 Thread Ben Vogan

Hi there,

Is it possible to create a textbox for accepting passwords via the
ZeppelinContext (i.e. one that masks input)?  I do not see any way to do
so, but I hope I'm missing something.

Thanks,

-- 
*BENJAMIN VOGAN* | Data Platform Team Lead

Re: what causes InterpreterException: Host key verification failed

2017-05-09 Thread Yeshwanth Jagini

i am not able to recreate this problem.

it is working now

On Tue, May 9, 2017 at 1:22 PM, Jongyoul Lee  wrote:

> It looks like a ssh problem. Can you tell me the setting how we use Spark
> Interpreter?
>
> On Fri, May 5, 2017 at 7:09 AM, Yeshwanth Jagini 
> wrote:
>
>> Any idea why i am running into this issue,
>>
>> INFO [2017-05-04 22:02:24,996] ({pool-2-thread-2}
>> SchedulerFactory.java[jobFinished]:137) - Job paragraph_1423500779206_-
>> 1502780787 finished by scheduler org.apache.zeppelin.interprete
>> r.remote.RemoteInterpreter2A94M5J1Z256540498
>>  INFO [2017-05-04 22:03:02,450] ({pool-2-thread-4}
>> SchedulerFactory.java[jobStarted]:131) - Job paragraph_1423500779206_-
>> 1502780787 started by scheduler org.apache.zeppelin.interprete
>> r.remote.RemoteInterpreter2A94M5J1Z256540498
>>  INFO [2017-05-04 22:03:02,451] ({pool-2-thread-4}
>> Paragraph.java[jobRun]:362) - run paragraph 20150210-015259_1403135953
>> using null org.apache.zeppelin.interpreter.LazyOpenInterpreter@5aa74c91
>>  INFO [2017-05-04 22:03:02,451] ({pool-2-thread-4}
>> RemoteInterpreterManagedProcess.java[start]:126) - Run interpreter
>> process [/opt/zeppelin-0.7.1-bin-all/bin/interpreter.sh, -d,
>> /opt/zeppelin-0.7.1-bin-all/interpreter/spark, -p, 37115, -u, bigdata,
>> -l, /opt/zeppelin-0.7.1-bin-all/local-repo/2CFWU98CR]
>>  INFO [2017-05-04 22:03:02,513] ({Exec Default Executor}
>> RemoteInterpreterManagedProcess.java[onProcessComplete]:180) -
>> Interpreter process exited 0
>> ERROR [2017-05-04 22:03:02,980] ({pool-2-thread-4} Job.java[run]:188) -
>> Job failed
>> org.apache.zeppelin.interpreter.InterpreterException: Host key
>> verification failed.
>>
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterMana
>> gedProcess.start(RemoteInterpreterManagedProcess.java:143)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProc
>> ess.reference(RemoteInterpreterProcess.java:73)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>> open(RemoteInterpreter.java:258)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>> getFormType(RemoteInterpreter.java:423)
>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormT
>> ype(LazyOpenInterpreter.java:106)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:387)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>> RemoteScheduler.java:329)
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:473)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> ERROR [2017-05-04 22:03:02,981] ({pool-2-thread-4}
>> NotebookServer.java[afterStatusChange]:2050) - Error
>> org.apache.zeppelin.interpreter.InterpreterException: Host key
>> verification failed.
>>
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterMana
>> gedProcess.start(RemoteInterpreterManagedProcess.java:143)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProc
>> ess.reference(RemoteInterpreterProcess.java:73)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>> open(RemoteInterpreter.java:258)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>> getFormType(RemoteInterpreter.java:423)
>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormT
>> ype(LazyOpenInterpreter.java:106)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:387)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>> RemoteScheduler.java:329)
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:473)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>  WARN [2017-05-04 22:03:02,982] ({pool-2-thread-4}
>> NotebookServer.java[afterStatusChange]:2058) - Job
>> 20150210-015259_1403135953 is finished, status: ERROR, exception:
>> org.apache.zeppelin.interpreter.InterpreterException: Host key
>> verification failed.
>> , result: Host key verification failed.
>>
>>  INFO [201

Re: Spark-CSV - Zeppelin tries to read CSV locally in Standalon mode

2017-05-09 Thread Jongyoul Lee

Could you test if it works with spark-shell?

On Sun, May 7, 2017 at 5:22 PM, Sofiane Cherchalli 
wrote:

> Hi,
>
> I have a standalone cluster, one master and one worker, running in
> separate nodes. Zeppelin is running is in a separate node too in client
> mode.
>
> When I run a notebook that reads a CSV file located in the worker
> node with Spark-CSV package, Zeppelin tries to read the CSV locally and
> fails because the CVS is in the worker node and not in Zeppelin node.
>
> Is this the expected behavior?
>
> Thanks.
>



-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Re: what causes InterpreterException: Host key verification failed

2017-05-09 Thread Jongyoul Lee

It looks like a ssh problem. Can you tell me the setting how we use Spark
Interpreter?

On Fri, May 5, 2017 at 7:09 AM, Yeshwanth Jagini 
wrote:

> Any idea why i am running into this issue,
>
> INFO [2017-05-04 22:02:24,996] ({pool-2-thread-2} 
> SchedulerFactory.java[jobFinished]:137)
> - Job paragraph_1423500779206_-1502780787 finished by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter2A94M5J1Z256540498
>  INFO [2017-05-04 22:03:02,450] ({pool-2-thread-4} 
> SchedulerFactory.java[jobStarted]:131)
> - Job paragraph_1423500779206_-1502780787 started by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter2A94M5J1Z256540498
>  INFO [2017-05-04 22:03:02,451] ({pool-2-thread-4}
> Paragraph.java[jobRun]:362) - run paragraph 20150210-015259_1403135953
> using null org.apache.zeppelin.interpreter.LazyOpenInterpreter@5aa74c91
>  INFO [2017-05-04 22:03:02,451] ({pool-2-thread-4}
> RemoteInterpreterManagedProcess.java[start]:126) - Run interpreter
> process [/opt/zeppelin-0.7.1-bin-all/bin/interpreter.sh, -d,
> /opt/zeppelin-0.7.1-bin-all/interpreter/spark, -p, 37115, -u, bigdata,
> -l, /opt/zeppelin-0.7.1-bin-all/local-repo/2CFWU98CR]
>  INFO [2017-05-04 22:03:02,513] ({Exec Default Executor}
> RemoteInterpreterManagedProcess.java[onProcessComplete]:180) -
> Interpreter process exited 0
> ERROR [2017-05-04 22:03:02,980] ({pool-2-thread-4} Job.java[run]:188) -
> Job failed
> org.apache.zeppelin.interpreter.InterpreterException: Host key
> verification failed.
>
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProces
> s.start(RemoteInterpreterManagedProcess.java:143)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
> reference(RemoteInterpreterProcess.java:73)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(
> RemoteInterpreter.java:258)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(
> RemoteInterpreter.java:423)
> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(
> LazyOpenInterpreter.java:106)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:387)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
> RemoteScheduler.java:329)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ERROR [2017-05-04 22:03:02,981] ({pool-2-thread-4} 
> NotebookServer.java[afterStatusChange]:2050)
> - Error
> org.apache.zeppelin.interpreter.InterpreterException: Host key
> verification failed.
>
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProces
> s.start(RemoteInterpreterManagedProcess.java:143)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
> reference(RemoteInterpreterProcess.java:73)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(
> RemoteInterpreter.java:258)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(
> RemoteInterpreter.java:423)
> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(
> LazyOpenInterpreter.java:106)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:387)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
> RemoteScheduler.java:329)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>  WARN [2017-05-04 22:03:02,982] ({pool-2-thread-4} 
> NotebookServer.java[afterStatusChange]:2058)
> - Job 20150210-015259_1403135953 is finished, status: ERROR, exception:
> org.apache.zeppelin.interpreter.InterpreterException: Host key
> verification failed.
> , result: Host key verification failed.
>
>  INFO [2017-05-04 22:03:03,042] ({pool-2-thread-4} 
> SchedulerFactory.java[jobFinished]:137)
> - Job paragraph_1423500779206_-1502780787 finished by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter2A94M5J1Z25

Re: spark.r interpreter becomes unresponsive after some time and R process quits silently

2017-05-09 Thread Pietro Pugni

I opened a JIRA with all the details (logs etc):
https://issues.apache.org/jira/plugins/servlet/mobile#issue/ZEPPELIN-2515

Thank you
 Pietro Pugni

Il 9 mag 2017 7:48 PM, "Jongyoul Lee"  ha scritto:

Hi, Thanks for this detail debugging.

At first, notebookserver doesn't have any clue for this symptom because
it's used between browser and zeppelin server.

I don't know why R has stoped unexpectedly. Is there any log related to R?
I'm not familiar with R actually.

BTW, I'll install R and test it in my local

On Tue, May 9, 2017 at 8:29 AM, Pietro Pugni  wrote:

> I repost this because it didn’t appear on the mailing list board.
>
> These are the step needed to reproduce the error and to track down the log
> message.
>
> 1) I started a brand new instance of zeppelin issuing:
> service zeppelin start
>
> and started a bash script that tracks down R processes activity.
> After running a simple R script from Zeppelin, the R interpreter process
> was started:
>
> Mon May  8 11:27:59 CEST 2017 >>> R started
>
> 2) I left the browser open and at 12:26:15 I closed the browser. Zeppelin
> tracked down the connection being closed:
> INFO [2017-05-08 12:26:15,879] ({qtp423031029-60}
> NotebookServer.java[onClose]:363) - Closed connection to 127.0.0.1 :
> 33798. (1001) null
>
> 3) At 13:08:00 R was closed. My script returned:
> Mon May  8 13:08:00 CEST 2017 >>> R stopped
>
> This is the output from the interpreter log file (deleted non-useful
> lines):
> INFO [2017-05-08 11:27:43,632] ({Thread-0} 
> RemoteInterpreterServer.java[run]:95)
> - Starting remote interpreter server on port 45227
> INFO [2017-05-08 11:27:44,600] ({pool-1-thread-3}
> RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate
> interpreter org.apache.zeppelin.spark.SparkInterpreter
> INFO [2017-05-08 11:27:44,624] ({pool-1-thread-3}
> RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate
> interpreter org.apache.zeppelin.spark.SparkSqlInterpreter
> INFO [2017-05-08 11:27:44,629] ({pool-1-thread-3}
> RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate
> interpreter org.apache.zeppelin.spark.DepInterpreter
> INFO [2017-05-08 11:27:44,640] ({pool-1-thread-3}
> RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate
> interpreter org.apache.zeppelin.spark.PySparkInterpreter
> INFO [2017-05-08 11:27:44,643] ({pool-1-thread-3}
> RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate
> interpreter org.apache.zeppelin.spark.SparkRInterpreter
> ...
> INFO [2017-05-08 11:28:00,188] ({pool-2-thread-2}
> SchedulerFactory.java[jobFinished]:137) - Job
> remoteInterpretJob_1494235664723 finished by scheduler
> org.apache.zeppelin.spark.SparkRInterpreter2097894179
> DEBUG [2017-05-08 11:28:00,819] ({pool-1-thread-3}
> RemoteInterpreterServer.java[resourcePoolGetAll]:911) - Request getAll
> from ZeppelinServer
> *DEBUG [2017-05-08 13:08:00,187] ({Exec Stream Pumper}
> InterpreterOutputStream.java[processLine]:72) - Interpreter output:Error in
> handleErrors(returnStatus, conn) : *
> *DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper}
> InterpreterOutputStream.java[processLine]:72) - Interpreter output:  No
> status is returned. Java SparkR backend might have failed.*
> *DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper}
> InterpreterOutputStream.java[processLine]:72) - Interpreter output:Calls:
>  -> invokeJava -> handleErrors*
> *DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper}
> InterpreterOutputStream.java[processLine]:72) - Interpreter
> output:Execution halted*
>
> This is the output from zeppelin log file (it didn't track the R
> interpreter failure):
> INFO [2017-05-08 11:28:00,221] ({pool-2-thread-2}
> NotebookServer.java[afterStatusChange]:2056) - Job
> 20170506-145151_1585482989 is finished successfully, status: FINISHED
> INFO [2017-05-08 11:28:00,675] ({pool-2-thread-2}
> SchedulerFactory.java[jobFinished]:137) - Job paragraph_1494075111996_-
> 1250116940 finished by scheduler org.apache.zeppelin.interprete
> r.remote.RemoteInterpretershared_session2130846287
> *INFO [2017-05-08 12:26:15,879] ({qtp423031029-60}
> NotebookServer.java[onClose]:363) - Closed connection to 127.0.0.1 : 33798.
> (1001) null*
> INFO [2017-05-08 12:27:12,126] ({Thread-33} 
> AbstractValidatingSessionManager.java[validateSessions]:271)
> - Validating all active sessions...
> INFO [2017-05-08 12:27:12,126] ({Thread-33} 
> AbstractValidatingSessionManager.java[validateSessions]:304)
> - Finished session validation.  No sessions were stopped.
>
> Hope this helps.
> Any hints?
>
> Il giorno 08 mag 2017, alle ore 11:08, Pietro Pugni <
> pietro.pu...@gmail.com> ha scritto:
>
> I know for sure that R process gets killed (or quits) but don't know if
> its father process (interpreter.sh) gets killed too.
>
> I noticed that I can always restart the interpreter on 0.7.1 while
> sometimes it was impossible to do on 0.7.0 (I had to manually restart
> zeppelin service). Probably that JIRA improved the situation a littl

Re: spark.r interpreter becomes unresponsive after some time and R process quits silently

2017-05-09 Thread Jongyoul Lee

Hi, Thanks for this detail debugging.

At first, notebookserver doesn't have any clue for this symptom because
it's used between browser and zeppelin server.

I don't know why R has stoped unexpectedly. Is there any log related to R?
I'm not familiar with R actually.

BTW, I'll install R and test it in my local

On Tue, May 9, 2017 at 8:29 AM, Pietro Pugni  wrote:

> I repost this because it didn’t appear on the mailing list board.
>
> These are the step needed to reproduce the error and to track down the log
> message.
>
> 1) I started a brand new instance of zeppelin issuing:
> service zeppelin start
>
> and started a bash script that tracks down R processes activity.
> After running a simple R script from Zeppelin, the R interpreter process
> was started:
>
> Mon May  8 11:27:59 CEST 2017 >>> R started
>
> 2) I left the browser open and at 12:26:15 I closed the browser. Zeppelin
> tracked down the connection being closed:
> INFO [2017-05-08 12:26:15,879] ({qtp423031029-60}
> NotebookServer.java[onClose]:363) - Closed connection to 127.0.0.1 :
> 33798. (1001) null
>
> 3) At 13:08:00 R was closed. My script returned:
> Mon May  8 13:08:00 CEST 2017 >>> R stopped
>
> This is the output from the interpreter log file (deleted non-useful
> lines):
> INFO [2017-05-08 11:27:43,632] ({Thread-0} 
> RemoteInterpreterServer.java[run]:95)
> - Starting remote interpreter server on port 45227
> INFO [2017-05-08 11:27:44,600] ({pool-1-thread-3}
> RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate
> interpreter org.apache.zeppelin.spark.SparkInterpreter
> INFO [2017-05-08 11:27:44,624] ({pool-1-thread-3}
> RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate
> interpreter org.apache.zeppelin.spark.SparkSqlInterpreter
> INFO [2017-05-08 11:27:44,629] ({pool-1-thread-3}
> RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate
> interpreter org.apache.zeppelin.spark.DepInterpreter
> INFO [2017-05-08 11:27:44,640] ({pool-1-thread-3}
> RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate
> interpreter org.apache.zeppelin.spark.PySparkInterpreter
> INFO [2017-05-08 11:27:44,643] ({pool-1-thread-3}
> RemoteInterpreterServer.java[createInterpreter]:190) - Instantiate
> interpreter org.apache.zeppelin.spark.SparkRInterpreter
> ...
> INFO [2017-05-08 11:28:00,188] ({pool-2-thread-2} 
> SchedulerFactory.java[jobFinished]:137)
> - Job remoteInterpretJob_1494235664723 finished by scheduler
> org.apache.zeppelin.spark.SparkRInterpreter2097894179
> DEBUG [2017-05-08 11:28:00,819] ({pool-1-thread-3}
> RemoteInterpreterServer.java[resourcePoolGetAll]:911) - Request getAll
> from ZeppelinServer
> *DEBUG [2017-05-08 13:08:00,187] ({Exec Stream Pumper}
> InterpreterOutputStream.java[processLine]:72) - Interpreter output:Error in
> handleErrors(returnStatus, conn) : *
> *DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper}
> InterpreterOutputStream.java[processLine]:72) - Interpreter output:  No
> status is returned. Java SparkR backend might have failed.*
> *DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper}
> InterpreterOutputStream.java[processLine]:72) - Interpreter output:Calls:
>  -> invokeJava -> handleErrors*
> *DEBUG [2017-05-08 13:08:00,188] ({Exec Stream Pumper}
> InterpreterOutputStream.java[processLine]:72) - Interpreter
> output:Execution halted*
>
> This is the output from zeppelin log file (it didn't track the R
> interpreter failure):
> INFO [2017-05-08 11:28:00,221] ({pool-2-thread-2} 
> NotebookServer.java[afterStatusChange]:2056)
> - Job 20170506-145151_1585482989 is finished successfully, status: FINISHED
> INFO [2017-05-08 11:28:00,675] ({pool-2-thread-2} 
> SchedulerFactory.java[jobFinished]:137)
> - Job paragraph_1494075111996_-1250116940 finished by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_
> session2130846287
> *INFO [2017-05-08 12:26:15,879] ({qtp423031029-60}
> NotebookServer.java[onClose]:363) - Closed connection to 127.0.0.1 : 33798.
> (1001) null*
> INFO [2017-05-08 12:27:12,126] ({Thread-33} 
> AbstractValidatingSessionManager.java[validateSessions]:271)
> - Validating all active sessions...
> INFO [2017-05-08 12:27:12,126] ({Thread-33} 
> AbstractValidatingSessionManager.java[validateSessions]:304)
> - Finished session validation.  No sessions were stopped.
>
> Hope this helps.
> Any hints?
>
> Il giorno 08 mag 2017, alle ore 11:08, Pietro Pugni <
> pietro.pu...@gmail.com> ha scritto:
>
> I know for sure that R process gets killed (or quits) but don't know if
> its father process (interpreter.sh) gets killed too.
>
> I noticed that I can always restart the interpreter on 0.7.1 while
> sometimes it was impossible to do on 0.7.0 (I had to manually restart
> zeppelin service). Probably that JIRA improved the situation a little bit.
>
> Now I'm running a bash script that tracks start and stop time of R process
> in order to shed some light on this issue. I enabled DEBUG logging in log4j
> properties file.
>
>
> Il 6 mag

Re: Hive Reserve Keyword support

2017-05-09 Thread Jongyoul Lee

If it's possible for you to pass that properties when you create a
connection, you can passes it by setting it into interpreter setting

On Sat, Apr 29, 2017 at 4:25 PM, Dibyendu Bhattacharya <
dibyendu.bhattach...@gmail.com> wrote:

> Hi,
>
> I have a Hive Table which has a column named date. When I tried to query
> using Zeppelin %jdbc interpreter , I got bellow error.
>
>
> Error while compiling statement: FAILED: ParseException line 1:312 Failed
> to recognize predicate 'date'. Failed rule: 'identifier' in expression
> specification
> class org.apache.hive.service.cli.HiveSQLException
> org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:231)
> org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:217)
> org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254)
> org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(
> JDBCInterpreter.java:322)
> org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(
> JDBCInterpreter.java:408)
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(
> LazyOpenInterpreter.java:94)
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$
> InterpretJob.jobRun(RemoteInterpreterServer.java:341)
> org.apache.zeppelin.scheduler.Job.run(Job.java:176)
> org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.
> run(ParallelScheduler.java:162)
>
>
> My query looks like this :
>
> select x,y,z from mytable where date = '2017-04-28"
>
> I believe it is failing because date is reserve keyword . Is there anyway
> I can set  hive.support.sql11.reserved.keywords=false in Zeppelin ?
>
> regards,
> Dibyendu
>
>
>
>


-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Re: How to create cluster for different Interpreters.

2017-05-09 Thread Jongyoul Lee

What is the cluster you told?

On Sun, Apr 23, 2017 at 7:32 PM, Pankaj Singh 
wrote:

> Hi All,
>
> Is there a smart way to create clusters which are needed for testing
> interpreters.
>
> Thanks,
> Pankaj
>



-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Re: java.lang.NullPointerException on adding local jar as dependency to the spark interpreter

2017-05-09 Thread Jongyoul Lee

Can you add your spark interpreter's log file?

On Sat, May 6, 2017 at 12:53 AM, shyla deshpande 
wrote:

> Also, my local jar file that I want to add as dependency is a fat jar with
> dependencies.  Nothing works after I add my local fat jar, I get 
> *java.lang.NullPointerException
> for everything. Please help*
>
> On Thu, May 4, 2017 at 10:18 PM, shyla deshpande  > wrote:
>
>> Adding the dependency by filling groupId:artifactId:version works good.
>> But when I add add a local jar file as the artifact , get
>> *ERROR java.lang.NullPointerException*. I see the local jar file being
>> added to local-repo, but I get the ERROR.
>>
>> Please help.
>>
>>
>


-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Re: Unable to run Zeppelin Spark on YARN

2017-05-09 Thread Jongyoul Lee

Hi,

"--master yarn --deploy-mode client" will be overridden when create spark
context by SparkInterpreter In zeppelin. You have to set those values in
interpreter setting page

Regards,
Jongyoul

On Fri, May 5, 2017 at 8:33 AM, Jianfeng (Jeff) Zhang <
jzh...@hortonworks.com> wrote:

>
> Could you try set yarn-client in interpreter setting page ?
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: Yeshwanth Jagini 
> Reply-To: "users@zeppelin.apache.org" 
> Date: Friday, May 5, 2017 at 3:13 AM
> To: "users@zeppelin.apache.org" 
> Subject: Unable to run Zeppelin Spark on YARN
>
> Hi we are running cloudera CDH 5.9.1 .
>
> while setting up zeppelin, i followed the documentation on website and
> specified following options
>
> export ZEPPELIN_JAVA_OPTS="-Dhadoop.version=2.6.0-cdh5.9.1"
>   # Additional jvm options. for example, export
> ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g -Dspark.cores.max=16"
>
> export SPARK_HOME="/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/spark"
># (required) When it is defined, load it instead
> of Zeppelin embedded Spark libraries
> export SPARK_SUBMIT_OPTIONS="--master yarn --deploy-mode client"
> # (optional) extra options to pass to spark submit. eg)
> "--driver-memory 512M --executor-memory 1G".
> export SPARK_APP_NAME=Zeppelin # (optional) The
> name of spark application.
>
> export HADOOP_CONF_DIR=/etc/hadoop/conf #
> yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.
>
> export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER}
> bash -c'   # Optional, when user want to run interpreter as end web
> user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '
>
> when running spark notebook, spark-submit is running in local mode and i
> cannot see the application in yarn resource manager.
> is there any other configuration i am missing?
>
>
> Thanks,
> Yeshwanth Jagini
>



-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Re: User Impersonation Configuration

Re: Hive Reserve Keyword support

Re: Hive Reserve Keyword support

User Impersonation Configuration

Re: ZeppelinContext textbox for passwords

Re: ZeppelinContext textbox for passwords

ZeppelinContext textbox for passwords

Re: what causes InterpreterException: Host key verification failed

Re: Spark-CSV - Zeppelin tries to read CSV locally in Standalon mode

Re: what causes InterpreterException: Host key verification failed

Re: spark.r interpreter becomes unresponsive after some time and R process quits silently

Re: spark.r interpreter becomes unresponsive after some time and R process quits silently

Re: Hive Reserve Keyword support

Re: How to create cluster for different Interpreters.

Re: java.lang.NullPointerException on adding local jar as dependency to the spark interpreter

Re: Unable to run Zeppelin Spark on YARN

16 matches

Site Navigation

Mail list logo

Footer information