Re: Extra SparkSubmit process in running Cassandra queries
Solved, The problem was I had set schedule for my spark note and it starts
automatically!
On Sat, Apr 28, 2018 at 1:52 PM, Soheil Pourbafrani
wrote:
> Here are the logs where after starting Zeppelin-daemon, SparkSubmit created
>
> INFO [2018-04-28 13:45:00,010] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:188) - Create interpreter
> instance Spark-Yarn for note 2DBFYYVZ2
> INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
> org.apache.zeppelin.spark.SparkInterpreter 507249349 created
> INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
> org.apache.zeppelin.spark.SparkSqlInterpreter 2147110707 created
> INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
> org.apache.zeppelin.spark.DepInterpreter 418560773 created
> INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
> org.apache.zeppelin.spark.PySparkInterpreter 904620448 created
> INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
> org.apache.zeppelin.spark.SparkRInterpreter 1262448626 created
> INFO [2018-04-28 13:45:00,019] ({DefaultQuartzScheduler_Worker-1}
> Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261
> INFO [2018-04-28 13:45:00,019] ({pool-2-thread-2}
> SchedulerFactory.java[jobStarted]:131) - Job
> paragraph_1523518438102_-1314230440 started by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541
> INFO [2018-04-28 13:45:00,021] ({pool-2-thread-2} Paragraph.java[jobRun]:362)
> - run paragraph 20180412-120358_257483588 using null
> org.apache.zeppelin.interpreter.LazyOpenInterpreter@1e3c02c5
> INFO [2018-04-28 13:45:00,027] ({pool-2-thread-2}
> RemoteInterpreterManagedProcess.java[start]:126) - Run interpreter process
> [/opt/zeppelin/bin/interpreter.sh, -d, /opt/zeppelin/interpreter/spark, -p,
> 32857, -l, /opt/zeppelin/local-repo/2DD2H1434]
> INFO [2018-04-28 13:45:01,207] ({pool-2-thread-2}
> RemoteInterpreter.java[init]:221) - Create remote interpreter
> org.apache.zeppelin.spark.SparkInterpreter
> INFO [2018-04-28 13:45:01,308] ({pool-2-thread-2}
> RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:551) - Push local
> angular object registry from ZeppelinServer to remote interpreter group
> 2DD2H1434:shared_process
> INFO [2018-04-28 13:45:01,324] ({pool-2-thread-2}
> RemoteInterpreter.java[init]:221) - Create remote interpreter
> org.apache.zeppelin.spark.SparkSqlInterpreter
> INFO [2018-04-28 13:45:01,327] ({pool-2-thread-2}
> RemoteInterpreter.java[init]:221) - Create remote interpreter
> org.apache.zeppelin.spark.DepInterpreter
> INFO [2018-04-28 13:45:01,329] ({pool-2-thread-2}
> RemoteInterpreter.java[init]:221) - Create remote interpreter
> org.apache.zeppelin.spark.PySparkInterpreter
> INFO [2018-04-28 13:45:01,340] ({pool-2-thread-2}
> RemoteInterpreter.java[init]:221) - Create remote interpreter
> org.apache.zeppelin.spark.SparkRInterpreter
> INFO [2018-04-28 13:45:01,520] ({pool-2-thread-3}
> SchedulerFactory.java[jobStarted]:131) - Job
> paragraph_1523519805660_-887954373 started by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541
> INFO [2018-04-28 13:45:01,521] ({pool-2-thread-3} Paragraph.java[jobRun]:362)
> - run paragraph 20180412-122645_1892292733 using sql
> org.apache.zeppelin.interpreter.LazyOpenInterpreter@7ffa4f33
> INFO [2018-04-28 13:46:00,002] ({DefaultQuartzScheduler_Worker-2}
> Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261
> INFO [2018-04-28 13:47:00,002] ({DefaultQuartzScheduler_Worker-3}
> Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261
> INFO [2018-04-28 13:48:00,001] ({DefaultQuartzScheduler_Worker-4}
> Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261
>
>
>
> On Sat, Apr 28, 2018 at 12:10 PM, Jeff Zhang wrote:
>
>>
>> Do you have any cron job enabled ? Please check the log to see how this
>> spark process is started.
>>
>>
>> Soheil Pourbafrani 于2018年4月28日周六 下午3:29写道：
>>
>>> Thanks for the reply, When I start Zeppelin daemon, only ZeppelinServer
>>> process will be created. when I just connect to the Zeppelin webui from the
>>> browser, (I've changed the port), a SparkSubmit process will be
>>> created!
>>>
>>> On Tue, Apr 24, 2018 at 10:14 AM, Jongyoul Lee
>>> wrote:
>>>
Hi,
AFAIK, it might be something wrong. Can you explain more about it?
JL
On Sun, Apr 22, 2018 at 6:36 PM, Soheil Pourbafrani <
soheil.i...@gmail.com> wrote:
> Hi, I use
Re: Extra SparkSubmit process in running Cassandra queries
Here are the logs where after starting Zeppelin-daemon, SparkSubmit created
INFO [2018-04-28 13:45:00,010] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:188) - Create
interpreter instance Spark-Yarn for note 2DBFYYVZ2
INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
org.apache.zeppelin.spark.SparkInterpreter 507249349 created
INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
org.apache.zeppelin.spark.SparkSqlInterpreter 2147110707 created
INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
org.apache.zeppelin.spark.DepInterpreter 418560773 created
INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
org.apache.zeppelin.spark.PySparkInterpreter 904620448 created
INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
org.apache.zeppelin.spark.SparkRInterpreter 1262448626 created
INFO [2018-04-28 13:45:00,019] ({DefaultQuartzScheduler_Worker-1}
Note.java[run]:610) - skip to run blank paragraph.
20180412-160531_1456847261
INFO [2018-04-28 13:45:00,019] ({pool-2-thread-2}
SchedulerFactory.java[jobStarted]:131) - Job
paragraph_1523518438102_-1314230440 started by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541
INFO [2018-04-28 13:45:00,021] ({pool-2-thread-2}
Paragraph.java[jobRun]:362) - run paragraph 20180412-120358_257483588
using null org.apache.zeppelin.interpreter.LazyOpenInterpreter@1e3c02c5
INFO [2018-04-28 13:45:00,027] ({pool-2-thread-2}
RemoteInterpreterManagedProcess.java[start]:126) - Run interpreter
process [/opt/zeppelin/bin/interpreter.sh, -d,
/opt/zeppelin/interpreter/spark, -p, 32857, -l,
/opt/zeppelin/local-repo/2DD2H1434]
INFO [2018-04-28 13:45:01,207] ({pool-2-thread-2}
RemoteInterpreter.java[init]:221) - Create remote interpreter
org.apache.zeppelin.spark.SparkInterpreter
INFO [2018-04-28 13:45:01,308] ({pool-2-thread-2}
RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:551) - Push
local angular object registry from ZeppelinServer to remote
interpreter group 2DD2H1434:shared_process
INFO [2018-04-28 13:45:01,324] ({pool-2-thread-2}
RemoteInterpreter.java[init]:221) - Create remote interpreter
org.apache.zeppelin.spark.SparkSqlInterpreter
INFO [2018-04-28 13:45:01,327] ({pool-2-thread-2}
RemoteInterpreter.java[init]:221) - Create remote interpreter
org.apache.zeppelin.spark.DepInterpreter
INFO [2018-04-28 13:45:01,329] ({pool-2-thread-2}
RemoteInterpreter.java[init]:221) - Create remote interpreter
org.apache.zeppelin.spark.PySparkInterpreter
INFO [2018-04-28 13:45:01,340] ({pool-2-thread-2}
RemoteInterpreter.java[init]:221) - Create remote interpreter
org.apache.zeppelin.spark.SparkRInterpreter
INFO [2018-04-28 13:45:01,520] ({pool-2-thread-3}
SchedulerFactory.java[jobStarted]:131) - Job
paragraph_1523519805660_-887954373 started by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541
INFO [2018-04-28 13:45:01,521] ({pool-2-thread-3}
Paragraph.java[jobRun]:362) - run paragraph 20180412-122645_1892292733
using sql org.apache.zeppelin.interpreter.LazyOpenInterpreter@7ffa4f33
INFO [2018-04-28 13:46:00,002] ({DefaultQuartzScheduler_Worker-2}
Note.java[run]:610) - skip to run blank paragraph.
20180412-160531_1456847261
INFO [2018-04-28 13:47:00,002] ({DefaultQuartzScheduler_Worker-3}
Note.java[run]:610) - skip to run blank paragraph.
20180412-160531_1456847261
INFO [2018-04-28 13:48:00,001] ({DefaultQuartzScheduler_Worker-4}
Note.java[run]:610) - skip to run blank paragraph.
20180412-160531_1456847261
On Sat, Apr 28, 2018 at 12:10 PM, Jeff Zhang wrote:
>
> Do you have any cron job enabled ? Please check the log to see how this
> spark process is started.
>
>
> Soheil Pourbafrani 于2018年4月28日周六 下午3:29写道：
>
>> Thanks for the reply, When I start Zeppelin daemon, only ZeppelinServer
>> process will be created. when I just connect to the Zeppelin webui from the
>> browser, (I've changed the port), a SparkSubmit process will be
>> created!
>>
>> On Tue, Apr 24, 2018 at 10:14 AM, Jongyoul Lee
>> wrote:
>>
>>> Hi,
>>>
>>> AFAIK, it might be something wrong. Can you explain more about it?
>>>
>>> JL
>>>
>>> On Sun, Apr 22, 2018 at 6:36 PM, Soheil Pourbafrani <
>>> soheil.i...@gmail.com> wrote:
>>>
Hi, I use Zeppelin 7.3
Customizing Cassandra interpreter, I configured it for my Cassandra
cluster.
When I try to get data from Cassandra using the command:
%cassandra
SELECT * FROM Key.Table ;
I expect it only creates just a RemoteInterpreterServer process to
fetch data from Cassandra, but in addition
Re: Extra SparkSubmit process in running Cassandra queries
Do you have any cron job enabled ? Please check the log to see how this spark process is started. Soheil Pourbafrani 于2018年4月28日周六 下午3:29写道： > Thanks for the reply, When I start Zeppelin daemon, only ZeppelinServer > process will be created. when I just connect to the Zeppelin webui from the > browser, (I've changed the port), a SparkSubmit process will be > created! > > On Tue, Apr 24, 2018 at 10:14 AM, Jongyoul Lee wrote: > >> Hi, >> >> AFAIK, it might be something wrong. Can you explain more about it? >> >> JL >> >> On Sun, Apr 22, 2018 at 6:36 PM, Soheil Pourbafrani < >> soheil.i...@gmail.com> wrote: >> >>> Hi, I use Zeppelin 7.3 >>> >>> Customizing Cassandra interpreter, I configured it for my Cassandra >>> cluster. >>> >>> When I try to get data from Cassandra using the command: >>> >>> %cassandra >>> >>> SELECT * FROM Key.Table ; >>> >>> I expect it only creates just a RemoteInterpreterServer process to fetch >>> data from Cassandra, but in addition to RemoteInterpreterServer, a >>> SparkSubmit process is created! >>> >>> I didn't use any spark code, just Cassandra CQL query, Why the >>> SparkSubmit process is created? >>> >> >> >> >> -- >> 이종열, Jongyoul Lee, 李宗烈 >> http://madeng.net >> > >
Re: Extra SparkSubmit process in running Cassandra queries
Thanks for the reply, When I start Zeppelin daemon, only ZeppelinServer process will be created. when I just connect to the Zeppelin webui from the browser, (I've changed the port), a SparkSubmit process will be created! On Tue, Apr 24, 2018 at 10:14 AM, Jongyoul Lee wrote: > Hi, > > AFAIK, it might be something wrong. Can you explain more about it? > > JL > > On Sun, Apr 22, 2018 at 6:36 PM, Soheil Pourbafrani > wrote: > >> Hi, I use Zeppelin 7.3 >> >> Customizing Cassandra interpreter, I configured it for my Cassandra >> cluster. >> >> When I try to get data from Cassandra using the command: >> >> %cassandra >> >> SELECT * FROM Key.Table ; >> >> I expect it only creates just a RemoteInterpreterServer process to fetch >> data from Cassandra, but in addition to RemoteInterpreterServer, a >> SparkSubmit process is created! >> >> I didn't use any spark code, just Cassandra CQL query, Why the >> SparkSubmit process is created? >> > > > > -- > 이종열, Jongyoul Lee, 李宗烈 > http://madeng.net >
Re: Extra SparkSubmit process in running Cassandra queries
Hi, AFAIK, it might be something wrong. Can you explain more about it? JL On Sun, Apr 22, 2018 at 6:36 PM, Soheil Pourbafrani wrote: > Hi, I use Zeppelin 7.3 > > Customizing Cassandra interpreter, I configured it for my Cassandra > cluster. > > When I try to get data from Cassandra using the command: > > %cassandra > > SELECT * FROM Key.Table ; > > I expect it only creates just a RemoteInterpreterServer process to fetch > data from Cassandra, but in addition to RemoteInterpreterServer, a > SparkSubmit process is created! > > I didn't use any spark code, just Cassandra CQL query, Why the SparkSubmit > process is created? > -- 이종열, Jongyoul Lee, 李宗烈 http://madeng.net
Extra SparkSubmit process in running Cassandra queries
Hi, I use Zeppelin 7.3 Customizing Cassandra interpreter, I configured it for my Cassandra cluster. When I try to get data from Cassandra using the command: %cassandra SELECT * FROM Key.Table ; I expect it only creates just a RemoteInterpreterServer process to fetch data from Cassandra, but in addition to RemoteInterpreterServer, a SparkSubmit process is created! I didn't use any spark code, just Cassandra CQL query, Why the SparkSubmit process is created?
