Re: How to compile Zeppelin with just some desired interpreter
Greate man! On Mon, Apr 30, 2018 at 8:16 PM, Ruslan Dautkhanov wrote: > You could also add a list of interpreters that you'd like to exclude from > the build: > > maven ... -pl '!cassandra,!bigquery,...' > > > > -- > Ruslan Dautkhanov > > On Mon, Apr 30, 2018 at 7:53 AM, Soheil Pourbafrani > wrote: > >> -DskipTests is in the command? >> >> On Mon, Apr 30, 2018 at 6:17 PM, Jeff Zhang wrote: >> >>> >>> Add -DskipTests to disable test >>> >>> >>> Soheil Pourbafrani 于2018年4月30日周一 下午9:11写道： >>> >>>> I downlodaed source of Zeppelin 0.8.1 from github and using the command >>>> mvn clean package -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark -Psparkr >>>> -Pscala-2.11 -DskipTests -Pbuild-distr >>>> >>>> but it errors: >>>> Failed to execute goal org.apache.maven.plugins:maven >>>> -compiler-plugin:3.1:testCompile (default-testCompile) on project >>>> zeppelin-bigquery: Compilation failure: Compilation failure: >>>> [ERROR] /home/Downloads/untitled folder/zeppelin-branch-0.8/big >>>> query/src/test/java/org/apache/zeppelin/bigquery/BigQueryInterpreterTest.java:[132,1] >>>> illegal start of type >>>> >>>> It's about big query >>>> is it possible to compile Zepplin including just Spark and Cassandra >>>> interpreter? >>>> >>> >> >
Re: How to compile Zeppelin with just some desired interpreter
-DskipTests is in the command? On Mon, Apr 30, 2018 at 6:17 PM, Jeff Zhang wrote: > > Add -DskipTests to disable test > > > Soheil Pourbafrani 于2018年4月30日周一 下午9:11写道： > >> I downlodaed source of Zeppelin 0.8.1 from github and using the command >> mvn clean package -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark -Psparkr >> -Pscala-2.11 -DskipTests -Pbuild-distr >> >> but it errors: >> Failed to execute goal org.apache.maven.plugins: >> maven-compiler-plugin:3.1:testCompile (default-testCompile) on project >> zeppelin-bigquery: Compilation failure: Compilation failure: >> [ERROR] /home/Downloads/untitled folder/zeppelin-branch-0.8/ >> bigquery/src/test/java/org/apache/zeppelin/bigquery/ >> BigQueryInterpreterTest.java:[132,1] illegal start of type >> >> It's about big query >> is it possible to compile Zepplin including just Spark and Cassandra >> interpreter? >> >
save data in a notebook to use in subsequent scripts
Is it possible to save a Cassandra query result in a variable to use in subsequent scripts?
How to compile Zeppelin with just some desired interpreter
I downlodaed source of Zeppelin 0.8.1 from github and using the command mvn clean package -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark -Psparkr -Pscala-2.11 -DskipTests -Pbuild-distr but it errors: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project zeppelin-bigquery: Compilation failure: Compilation failure: [ERROR] /home/Downloads/untitled folder/zeppelin-branch-0.8/bigquery/src/test/java/org/apache/zeppelin/bigquery/BigQueryInterpreterTest.java:[132,1] illegal start of type It's about big query is it possible to compile Zepplin including just Spark and Cassandra interpreter?
Re: Scheduler in Zeppelin 0.9
Greate! On Mon, Apr 30, 2018 at 5:18 PM, Jeff Zhang wrote: > > It is not removed, but disabled by default. > You can enable it via setting zeppelin.notebook.cron.enable to true in > zeppelin-site.xml > > > > Soheil Pourbafrani 于2018年4月30日周一 下午7:42写道： > >> I used to use Zeppelin 0.7.3 scheduler at top of each note, but in >> Zeppelin 0.9 Snapshot the scheduler is removed! How can I schedule notes in >> Zeppelin 0.9? >> >
Scheduler in Zeppelin 0.9
I used to use Zeppelin 0.7.3 scheduler at top of each note, but in Zeppelin 0.9 Snapshot the scheduler is removed! How can I schedule notes in Zeppelin 0.9?
Re: Compile Zepplin source and get output like tar file structure
Thanks On Mon, Apr 30, 2018 at 12:01 PM, Jeff Zhang wrote: > > Add this to your maven build command: > > -Pbuild-distr > > Soheil Pourbafrani 于2018年4月30日周一 下午3:26写道： > >> Thanks, How should I enable build-distr ? >> >> On Mon, Apr 30, 2018 at 11:32 AM, Jeff Zhang wrote: >> >>> >>> What do you mean the outputs are messy ? If you want to build binary >>> distribution, please enable profile build-distr ? >>> >>> >>> Soheil Pourbafrani 于2018年4月30日周一 下午2:24写道： >>> >>>> Using the command >>>> mvn clean package -DskipTests -Pspark-2.0 -Phadoop-2.4 -Pr -Pscala-2.11 >>>> >>>> I compiled Zeppelin source but the outputs are messy and unstructured. >>>> How can I compile it to get output like Zeppelin binary tar file structure? >>>> >>> >>
Re: Compile Zepplin source and get output like tar file structure
Thanks, How should I enable build-distr ? On Mon, Apr 30, 2018 at 11:32 AM, Jeff Zhang wrote: > > What do you mean the outputs are messy ? If you want to build binary > distribution, please enable profile build-distr ? > > > Soheil Pourbafrani 于2018年4月30日周一 下午2:24写道： > >> Using the command >> mvn clean package -DskipTests -Pspark-2.0 -Phadoop-2.4 -Pr -Pscala-2.11 >> >> I compiled Zeppelin source but the outputs are messy and unstructured. >> How can I compile it to get output like Zeppelin binary tar file structure? >> >
Compile Zepplin source and get output like tar file structure
Using the command mvn clean package -DskipTests -Pspark-2.0 -Phadoop-2.4 -Pr -Pscala-2.11 I compiled Zeppelin source but the outputs are messy and unstructured. How can I compile it to get output like Zeppelin binary tar file structure?
Re: Is it possible to run Zeppelin on cluster
Thanks, I meant the Zeppelin itself, not it's jobs. On Sun, Apr 29, 2018 at 11:51 PM, Michael Segel wrote: > Yes if you mean to run the spark jobs on a cluster. > > > On Apr 29, 2018, at 7:25 AM, Soheil Pourbafrani > wrote: > > I mean to configure Zeppelin in multimode. > > On Sun, Apr 29, 2018 at 4:49 PM, Soheil Pourbafrani > wrote: > >> Something like Kafka or Hadoop cluster? >> > > >
zeppelin 0.8 tar file
Is there any pre-compiled tar file of Zeppelin 0.8 to download?
Re: Is it possible to run Zeppelin on cluster
I mean to configure Zeppelin in multimode. On Sun, Apr 29, 2018 at 4:49 PM, Soheil Pourbafrani wrote: > Something like Kafka or Hadoop cluster? >
Is it possible to run Zeppelin on cluster
Something like Kafka or Hadoop cluster?
Re: Extra SparkSubmit process in running Cassandra queries
Solved, The problem was I had set schedule for my spark note and it starts
automatically!
On Sat, Apr 28, 2018 at 1:52 PM, Soheil Pourbafrani
wrote:
> Here are the logs where after starting Zeppelin-daemon, SparkSubmit created
>
> INFO [2018-04-28 13:45:00,010] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:188) - Create interpreter
> instance Spark-Yarn for note 2DBFYYVZ2
> INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
> org.apache.zeppelin.spark.SparkInterpreter 507249349 created
> INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
> org.apache.zeppelin.spark.SparkSqlInterpreter 2147110707 created
> INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
> org.apache.zeppelin.spark.DepInterpreter 418560773 created
> INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
> org.apache.zeppelin.spark.PySparkInterpreter 904620448 created
> INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
> InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
> org.apache.zeppelin.spark.SparkRInterpreter 1262448626 created
> INFO [2018-04-28 13:45:00,019] ({DefaultQuartzScheduler_Worker-1}
> Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261
> INFO [2018-04-28 13:45:00,019] ({pool-2-thread-2}
> SchedulerFactory.java[jobStarted]:131) - Job
> paragraph_1523518438102_-1314230440 started by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541
> INFO [2018-04-28 13:45:00,021] ({pool-2-thread-2} Paragraph.java[jobRun]:362)
> - run paragraph 20180412-120358_257483588 using null
> org.apache.zeppelin.interpreter.LazyOpenInterpreter@1e3c02c5
> INFO [2018-04-28 13:45:00,027] ({pool-2-thread-2}
> RemoteInterpreterManagedProcess.java[start]:126) - Run interpreter process
> [/opt/zeppelin/bin/interpreter.sh, -d, /opt/zeppelin/interpreter/spark, -p,
> 32857, -l, /opt/zeppelin/local-repo/2DD2H1434]
> INFO [2018-04-28 13:45:01,207] ({pool-2-thread-2}
> RemoteInterpreter.java[init]:221) - Create remote interpreter
> org.apache.zeppelin.spark.SparkInterpreter
> INFO [2018-04-28 13:45:01,308] ({pool-2-thread-2}
> RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:551) - Push local
> angular object registry from ZeppelinServer to remote interpreter group
> 2DD2H1434:shared_process
> INFO [2018-04-28 13:45:01,324] ({pool-2-thread-2}
> RemoteInterpreter.java[init]:221) - Create remote interpreter
> org.apache.zeppelin.spark.SparkSqlInterpreter
> INFO [2018-04-28 13:45:01,327] ({pool-2-thread-2}
> RemoteInterpreter.java[init]:221) - Create remote interpreter
> org.apache.zeppelin.spark.DepInterpreter
> INFO [2018-04-28 13:45:01,329] ({pool-2-thread-2}
> RemoteInterpreter.java[init]:221) - Create remote interpreter
> org.apache.zeppelin.spark.PySparkInterpreter
> INFO [2018-04-28 13:45:01,340] ({pool-2-thread-2}
> RemoteInterpreter.java[init]:221) - Create remote interpreter
> org.apache.zeppelin.spark.SparkRInterpreter
> INFO [2018-04-28 13:45:01,520] ({pool-2-thread-3}
> SchedulerFactory.java[jobStarted]:131) - Job
> paragraph_1523519805660_-887954373 started by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541
> INFO [2018-04-28 13:45:01,521] ({pool-2-thread-3} Paragraph.java[jobRun]:362)
> - run paragraph 20180412-122645_1892292733 using sql
> org.apache.zeppelin.interpreter.LazyOpenInterpreter@7ffa4f33
> INFO [2018-04-28 13:46:00,002] ({DefaultQuartzScheduler_Worker-2}
> Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261
> INFO [2018-04-28 13:47:00,002] ({DefaultQuartzScheduler_Worker-3}
> Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261
> INFO [2018-04-28 13:48:00,001] ({DefaultQuartzScheduler_Worker-4}
> Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261
>
>
>
> On Sat, Apr 28, 2018 at 12:10 PM, Jeff Zhang wrote:
>
>>
>> Do you have any cron job enabled ? Please check the log to see how this
>> spark process is started.
>>
>>
>> Soheil Pourbafrani 于2018年4月28日周六 下午3:29写道：
>>
>>> Thanks for the reply, When I start Zeppelin daemon, only ZeppelinServer
>>> process will be created. when I just connect to the Zeppelin webui from the
>>> browser, (I've changed the port), a SparkSubmit process will be
>>> creat
Re: Extra SparkSubmit process in running Cassandra queries
Here are the logs where after starting Zeppelin-daemon, SparkSubmit created
INFO [2018-04-28 13:45:00,010] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:188) - Create
interpreter instance Spark-Yarn for note 2DBFYYVZ2
INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
org.apache.zeppelin.spark.SparkInterpreter 507249349 created
INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
org.apache.zeppelin.spark.SparkSqlInterpreter 2147110707 created
INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
org.apache.zeppelin.spark.DepInterpreter 418560773 created
INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
org.apache.zeppelin.spark.PySparkInterpreter 904620448 created
INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1}
InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter
org.apache.zeppelin.spark.SparkRInterpreter 1262448626 created
INFO [2018-04-28 13:45:00,019] ({DefaultQuartzScheduler_Worker-1}
Note.java[run]:610) - skip to run blank paragraph.
20180412-160531_1456847261
INFO [2018-04-28 13:45:00,019] ({pool-2-thread-2}
SchedulerFactory.java[jobStarted]:131) - Job
paragraph_1523518438102_-1314230440 started by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541
INFO [2018-04-28 13:45:00,021] ({pool-2-thread-2}
Paragraph.java[jobRun]:362) - run paragraph 20180412-120358_257483588
using null org.apache.zeppelin.interpreter.LazyOpenInterpreter@1e3c02c5
INFO [2018-04-28 13:45:00,027] ({pool-2-thread-2}
RemoteInterpreterManagedProcess.java[start]:126) - Run interpreter
process [/opt/zeppelin/bin/interpreter.sh, -d,
/opt/zeppelin/interpreter/spark, -p, 32857, -l,
/opt/zeppelin/local-repo/2DD2H1434]
INFO [2018-04-28 13:45:01,207] ({pool-2-thread-2}
RemoteInterpreter.java[init]:221) - Create remote interpreter
org.apache.zeppelin.spark.SparkInterpreter
INFO [2018-04-28 13:45:01,308] ({pool-2-thread-2}
RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:551) - Push
local angular object registry from ZeppelinServer to remote
interpreter group 2DD2H1434:shared_process
INFO [2018-04-28 13:45:01,324] ({pool-2-thread-2}
RemoteInterpreter.java[init]:221) - Create remote interpreter
org.apache.zeppelin.spark.SparkSqlInterpreter
INFO [2018-04-28 13:45:01,327] ({pool-2-thread-2}
RemoteInterpreter.java[init]:221) - Create remote interpreter
org.apache.zeppelin.spark.DepInterpreter
INFO [2018-04-28 13:45:01,329] ({pool-2-thread-2}
RemoteInterpreter.java[init]:221) - Create remote interpreter
org.apache.zeppelin.spark.PySparkInterpreter
INFO [2018-04-28 13:45:01,340] ({pool-2-thread-2}
RemoteInterpreter.java[init]:221) - Create remote interpreter
org.apache.zeppelin.spark.SparkRInterpreter
INFO [2018-04-28 13:45:01,520] ({pool-2-thread-3}
SchedulerFactory.java[jobStarted]:131) - Job
paragraph_1523519805660_-887954373 started by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541
INFO [2018-04-28 13:45:01,521] ({pool-2-thread-3}
Paragraph.java[jobRun]:362) - run paragraph 20180412-122645_1892292733
using sql org.apache.zeppelin.interpreter.LazyOpenInterpreter@7ffa4f33
INFO [2018-04-28 13:46:00,002] ({DefaultQuartzScheduler_Worker-2}
Note.java[run]:610) - skip to run blank paragraph.
20180412-160531_1456847261
INFO [2018-04-28 13:47:00,002] ({DefaultQuartzScheduler_Worker-3}
Note.java[run]:610) - skip to run blank paragraph.
20180412-160531_1456847261
INFO [2018-04-28 13:48:00,001] ({DefaultQuartzScheduler_Worker-4}
Note.java[run]:610) - skip to run blank paragraph.
20180412-160531_1456847261
On Sat, Apr 28, 2018 at 12:10 PM, Jeff Zhang wrote:
>
> Do you have any cron job enabled ? Please check the log to see how this
> spark process is started.
>
>
> Soheil Pourbafrani 于2018年4月28日周六 下午3:29写道：
>
>> Thanks for the reply, When I start Zeppelin daemon, only ZeppelinServer
>> process will be created. when I just connect to the Zeppelin webui from the
>> browser, (I've changed the port), a SparkSubmit process will be
>> created!
>>
>> On Tue, Apr 24, 2018 at 10:14 AM, Jongyoul Lee
>> wrote:
>>
>>> Hi,
>>>
>>> AFAIK, it might be something wrong. Can you explain more about it?
>>>
>>> JL
>>>
>>> On Sun, Apr 22, 2018 at 6:36 PM, Soheil Pourbafrani <
>>> soheil.i...@gmail.com> wrote:
>>>
>>>> Hi, I use Zeppelin 7.3
>>>>
>>>> Customizing Cassandra interpreter, I configured it for my Cassandra
>>>> cluster.
>>>>
>>>> When
Re: Extra SparkSubmit process in running Cassandra queries
Thanks for the reply, When I start Zeppelin daemon, only ZeppelinServer process will be created. when I just connect to the Zeppelin webui from the browser, (I've changed the port), a SparkSubmit process will be created! On Tue, Apr 24, 2018 at 10:14 AM, Jongyoul Lee wrote: > Hi, > > AFAIK, it might be something wrong. Can you explain more about it? > > JL > > On Sun, Apr 22, 2018 at 6:36 PM, Soheil Pourbafrani > wrote: > >> Hi, I use Zeppelin 7.3 >> >> Customizing Cassandra interpreter, I configured it for my Cassandra >> cluster. >> >> When I try to get data from Cassandra using the command: >> >> %cassandra >> >> SELECT * FROM Key.Table ; >> >> I expect it only creates just a RemoteInterpreterServer process to fetch >> data from Cassandra, but in addition to RemoteInterpreterServer, a >> SparkSubmit process is created! >> >> I didn't use any spark code, just Cassandra CQL query, Why the >> SparkSubmit process is created? >> > > > > -- > 이종열, Jongyoul Lee, 李宗烈 > http://madeng.net >
Extra SparkSubmit process in running Cassandra queries
Hi, I use Zeppelin 7.3 Customizing Cassandra interpreter, I configured it for my Cassandra cluster. When I try to get data from Cassandra using the command: %cassandra SELECT * FROM Key.Table ; I expect it only creates just a RemoteInterpreterServer process to fetch data from Cassandra, but in addition to RemoteInterpreterServer, a SparkSubmit process is created! I didn't use any spark code, just Cassandra CQL query, Why the SparkSubmit process is created?
SPARK_SUBMIT_OPTIONS variable didin't applied!
I use SPARK_SUBMIT_OPTIONS variable in zeppeline-env.sh file to set some options for spark applications that runs for zeppelin. export SPARK_SUBMIT_OPTIONS="--executor-memory 512m --driver-memory 512m --executor-cores 2 --queue onlineQ" But running spark on the Yarn cluster, it didn't applied. When I set some options like executor-memory in interpreter setting, it will be applied, but I want to use some options like queue and I don't know if it's possible to set such option in interpreter settings?
running a Spark fat jar in Zeppelin
Hi, Is it possible, instead of codding Spark in Zeppelin environment, run a fat jar file into YARN cluster to fetch data for Zeppelin?
