Re: How to compile Zeppelin with just some desired interpreter
Greate man! On Mon, Apr 30, 2018 at 8:16 PM, Ruslan Dautkhanov wrote: > You could also add a list of interpreters that you'd like to exclude from > the build: > > maven ... -pl '!cassandra,!bigquery,...' > > > > -- > Ruslan Dautkhanov > > On Mon, Apr 30, 2018 at 7:53 AM, Soheil Pourbafrani > wrote: > >> -DskipTests is in the command? >> >> On Mon, Apr 30, 2018 at 6:17 PM, Jeff Zhang wrote: >> >>> >>> Add -DskipTests to disable test >>> >>> >>> Soheil Pourbafrani 于2018年4月30日周一 下午9:11写道: >>> >>>> I downlodaed source of Zeppelin 0.8.1 from github and using the command >>>> mvn clean package -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark -Psparkr >>>> -Pscala-2.11 -DskipTests -Pbuild-distr >>>> >>>> but it errors: >>>> Failed to execute goal org.apache.maven.plugins:maven >>>> -compiler-plugin:3.1:testCompile (default-testCompile) on project >>>> zeppelin-bigquery: Compilation failure: Compilation failure: >>>> [ERROR] /home/Downloads/untitled folder/zeppelin-branch-0.8/big >>>> query/src/test/java/org/apache/zeppelin/bigquery/BigQueryInterpreterTest.java:[132,1] >>>> illegal start of type >>>> >>>> It's about big query >>>> is it possible to compile Zepplin including just Spark and Cassandra >>>> interpreter? >>>> >>> >> >
Re: How to compile Zeppelin with just some desired interpreter
-DskipTests is in the command? On Mon, Apr 30, 2018 at 6:17 PM, Jeff Zhang wrote: > > Add -DskipTests to disable test > > > Soheil Pourbafrani 于2018年4月30日周一 下午9:11写道: > >> I downlodaed source of Zeppelin 0.8.1 from github and using the command >> mvn clean package -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark -Psparkr >> -Pscala-2.11 -DskipTests -Pbuild-distr >> >> but it errors: >> Failed to execute goal org.apache.maven.plugins: >> maven-compiler-plugin:3.1:testCompile (default-testCompile) on project >> zeppelin-bigquery: Compilation failure: Compilation failure: >> [ERROR] /home/Downloads/untitled folder/zeppelin-branch-0.8/ >> bigquery/src/test/java/org/apache/zeppelin/bigquery/ >> BigQueryInterpreterTest.java:[132,1] illegal start of type >> >> It's about big query >> is it possible to compile Zepplin including just Spark and Cassandra >> interpreter? >> >
save data in a notebook to use in subsequent scripts
Is it possible to save a Cassandra query result in a variable to use in subsequent scripts?
How to compile Zeppelin with just some desired interpreter
I downlodaed source of Zeppelin 0.8.1 from github and using the command mvn clean package -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark -Psparkr -Pscala-2.11 -DskipTests -Pbuild-distr but it errors: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project zeppelin-bigquery: Compilation failure: Compilation failure: [ERROR] /home/Downloads/untitled folder/zeppelin-branch-0.8/bigquery/src/test/java/org/apache/zeppelin/bigquery/BigQueryInterpreterTest.java:[132,1] illegal start of type It's about big query is it possible to compile Zepplin including just Spark and Cassandra interpreter?
Re: Scheduler in Zeppelin 0.9
Greate! On Mon, Apr 30, 2018 at 5:18 PM, Jeff Zhang wrote: > > It is not removed, but disabled by default. > You can enable it via setting zeppelin.notebook.cron.enable to true in > zeppelin-site.xml > > > > Soheil Pourbafrani 于2018年4月30日周一 下午7:42写道: > >> I used to use Zeppelin 0.7.3 scheduler at top of each note, but in >> Zeppelin 0.9 Snapshot the scheduler is removed! How can I schedule notes in >> Zeppelin 0.9? >> >
Scheduler in Zeppelin 0.9
I used to use Zeppelin 0.7.3 scheduler at top of each note, but in Zeppelin 0.9 Snapshot the scheduler is removed! How can I schedule notes in Zeppelin 0.9?
Re: Compile Zepplin source and get output like tar file structure
Thanks On Mon, Apr 30, 2018 at 12:01 PM, Jeff Zhang wrote: > > Add this to your maven build command: > > -Pbuild-distr > > Soheil Pourbafrani 于2018年4月30日周一 下午3:26写道: > >> Thanks, How should I enable build-distr ? >> >> On Mon, Apr 30, 2018 at 11:32 AM, Jeff Zhang wrote: >> >>> >>> What do you mean the outputs are messy ? If you want to build binary >>> distribution, please enable profile build-distr ? >>> >>> >>> Soheil Pourbafrani 于2018年4月30日周一 下午2:24写道: >>> >>>> Using the command >>>> mvn clean package -DskipTests -Pspark-2.0 -Phadoop-2.4 -Pr -Pscala-2.11 >>>> >>>> I compiled Zeppelin source but the outputs are messy and unstructured. >>>> How can I compile it to get output like Zeppelin binary tar file structure? >>>> >>> >>
Re: Compile Zepplin source and get output like tar file structure
Thanks, How should I enable build-distr ? On Mon, Apr 30, 2018 at 11:32 AM, Jeff Zhang wrote: > > What do you mean the outputs are messy ? If you want to build binary > distribution, please enable profile build-distr ? > > > Soheil Pourbafrani 于2018年4月30日周一 下午2:24写道: > >> Using the command >> mvn clean package -DskipTests -Pspark-2.0 -Phadoop-2.4 -Pr -Pscala-2.11 >> >> I compiled Zeppelin source but the outputs are messy and unstructured. >> How can I compile it to get output like Zeppelin binary tar file structure? >> >
Compile Zepplin source and get output like tar file structure
Using the command mvn clean package -DskipTests -Pspark-2.0 -Phadoop-2.4 -Pr -Pscala-2.11 I compiled Zeppelin source but the outputs are messy and unstructured. How can I compile it to get output like Zeppelin binary tar file structure?
Re: Is it possible to run Zeppelin on cluster
Thanks, I meant the Zeppelin itself, not it's jobs. On Sun, Apr 29, 2018 at 11:51 PM, Michael Segel wrote: > Yes if you mean to run the spark jobs on a cluster. > > > On Apr 29, 2018, at 7:25 AM, Soheil Pourbafrani > wrote: > > I mean to configure Zeppelin in multimode. > > On Sun, Apr 29, 2018 at 4:49 PM, Soheil Pourbafrani > wrote: > >> Something like Kafka or Hadoop cluster? >> > > >
zeppelin 0.8 tar file
Is there any pre-compiled tar file of Zeppelin 0.8 to download?
Re: Is it possible to run Zeppelin on cluster
I mean to configure Zeppelin in multimode. On Sun, Apr 29, 2018 at 4:49 PM, Soheil Pourbafrani wrote: > Something like Kafka or Hadoop cluster? >
Is it possible to run Zeppelin on cluster
Something like Kafka or Hadoop cluster?
Re: Extra SparkSubmit process in running Cassandra queries
Solved, The problem was I had set schedule for my spark note and it starts automatically! On Sat, Apr 28, 2018 at 1:52 PM, Soheil Pourbafrani wrote: > Here are the logs where after starting Zeppelin-daemon, SparkSubmit created > > INFO [2018-04-28 13:45:00,010] ({DefaultQuartzScheduler_Worker-1} > InterpreterFactory.java[createInterpretersForNote]:188) - Create interpreter > instance Spark-Yarn for note 2DBFYYVZ2 > INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1} > InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter > org.apache.zeppelin.spark.SparkInterpreter 507249349 created > INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1} > InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter > org.apache.zeppelin.spark.SparkSqlInterpreter 2147110707 created > INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1} > InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter > org.apache.zeppelin.spark.DepInterpreter 418560773 created > INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1} > InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter > org.apache.zeppelin.spark.PySparkInterpreter 904620448 created > INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1} > InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter > org.apache.zeppelin.spark.SparkRInterpreter 1262448626 created > INFO [2018-04-28 13:45:00,019] ({DefaultQuartzScheduler_Worker-1} > Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261 > INFO [2018-04-28 13:45:00,019] ({pool-2-thread-2} > SchedulerFactory.java[jobStarted]:131) - Job > paragraph_1523518438102_-1314230440 started by scheduler > org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541 > INFO [2018-04-28 13:45:00,021] ({pool-2-thread-2} Paragraph.java[jobRun]:362) > - run paragraph 20180412-120358_257483588 using null > org.apache.zeppelin.interpreter.LazyOpenInterpreter@1e3c02c5 > INFO [2018-04-28 13:45:00,027] ({pool-2-thread-2} > RemoteInterpreterManagedProcess.java[start]:126) - Run interpreter process > [/opt/zeppelin/bin/interpreter.sh, -d, /opt/zeppelin/interpreter/spark, -p, > 32857, -l, /opt/zeppelin/local-repo/2DD2H1434] > INFO [2018-04-28 13:45:01,207] ({pool-2-thread-2} > RemoteInterpreter.java[init]:221) - Create remote interpreter > org.apache.zeppelin.spark.SparkInterpreter > INFO [2018-04-28 13:45:01,308] ({pool-2-thread-2} > RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:551) - Push local > angular object registry from ZeppelinServer to remote interpreter group > 2DD2H1434:shared_process > INFO [2018-04-28 13:45:01,324] ({pool-2-thread-2} > RemoteInterpreter.java[init]:221) - Create remote interpreter > org.apache.zeppelin.spark.SparkSqlInterpreter > INFO [2018-04-28 13:45:01,327] ({pool-2-thread-2} > RemoteInterpreter.java[init]:221) - Create remote interpreter > org.apache.zeppelin.spark.DepInterpreter > INFO [2018-04-28 13:45:01,329] ({pool-2-thread-2} > RemoteInterpreter.java[init]:221) - Create remote interpreter > org.apache.zeppelin.spark.PySparkInterpreter > INFO [2018-04-28 13:45:01,340] ({pool-2-thread-2} > RemoteInterpreter.java[init]:221) - Create remote interpreter > org.apache.zeppelin.spark.SparkRInterpreter > INFO [2018-04-28 13:45:01,520] ({pool-2-thread-3} > SchedulerFactory.java[jobStarted]:131) - Job > paragraph_1523519805660_-887954373 started by scheduler > org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541 > INFO [2018-04-28 13:45:01,521] ({pool-2-thread-3} Paragraph.java[jobRun]:362) > - run paragraph 20180412-122645_1892292733 using sql > org.apache.zeppelin.interpreter.LazyOpenInterpreter@7ffa4f33 > INFO [2018-04-28 13:46:00,002] ({DefaultQuartzScheduler_Worker-2} > Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261 > INFO [2018-04-28 13:47:00,002] ({DefaultQuartzScheduler_Worker-3} > Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261 > INFO [2018-04-28 13:48:00,001] ({DefaultQuartzScheduler_Worker-4} > Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261 > > > > On Sat, Apr 28, 2018 at 12:10 PM, Jeff Zhang wrote: > >> >> Do you have any cron job enabled ? Please check the log to see how this >> spark process is started. >> >> >> Soheil Pourbafrani 于2018年4月28日周六 下午3:29写道: >> >>> Thanks for the reply, When I start Zeppelin daemon, only ZeppelinServer >>> process will be created. when I just connect to the Zeppelin webui from the >>> browser, (I've changed the port), a SparkSubmit process will be >>> creat
Re: Extra SparkSubmit process in running Cassandra queries
Here are the logs where after starting Zeppelin-daemon, SparkSubmit created INFO [2018-04-28 13:45:00,010] ({DefaultQuartzScheduler_Worker-1} InterpreterFactory.java[createInterpretersForNote]:188) - Create interpreter instance Spark-Yarn for note 2DBFYYVZ2 INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1} InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter org.apache.zeppelin.spark.SparkInterpreter 507249349 created INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1} InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter org.apache.zeppelin.spark.SparkSqlInterpreter 2147110707 created INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1} InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter org.apache.zeppelin.spark.DepInterpreter 418560773 created INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1} InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter org.apache.zeppelin.spark.PySparkInterpreter 904620448 created INFO [2018-04-28 13:45:00,012] ({DefaultQuartzScheduler_Worker-1} InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter org.apache.zeppelin.spark.SparkRInterpreter 1262448626 created INFO [2018-04-28 13:45:00,019] ({DefaultQuartzScheduler_Worker-1} Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261 INFO [2018-04-28 13:45:00,019] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:131) - Job paragraph_1523518438102_-1314230440 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541 INFO [2018-04-28 13:45:00,021] ({pool-2-thread-2} Paragraph.java[jobRun]:362) - run paragraph 20180412-120358_257483588 using null org.apache.zeppelin.interpreter.LazyOpenInterpreter@1e3c02c5 INFO [2018-04-28 13:45:00,027] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:126) - Run interpreter process [/opt/zeppelin/bin/interpreter.sh, -d, /opt/zeppelin/interpreter/spark, -p, 32857, -l, /opt/zeppelin/local-repo/2DD2H1434] INFO [2018-04-28 13:45:01,207] ({pool-2-thread-2} RemoteInterpreter.java[init]:221) - Create remote interpreter org.apache.zeppelin.spark.SparkInterpreter INFO [2018-04-28 13:45:01,308] ({pool-2-thread-2} RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:551) - Push local angular object registry from ZeppelinServer to remote interpreter group 2DD2H1434:shared_process INFO [2018-04-28 13:45:01,324] ({pool-2-thread-2} RemoteInterpreter.java[init]:221) - Create remote interpreter org.apache.zeppelin.spark.SparkSqlInterpreter INFO [2018-04-28 13:45:01,327] ({pool-2-thread-2} RemoteInterpreter.java[init]:221) - Create remote interpreter org.apache.zeppelin.spark.DepInterpreter INFO [2018-04-28 13:45:01,329] ({pool-2-thread-2} RemoteInterpreter.java[init]:221) - Create remote interpreter org.apache.zeppelin.spark.PySparkInterpreter INFO [2018-04-28 13:45:01,340] ({pool-2-thread-2} RemoteInterpreter.java[init]:221) - Create remote interpreter org.apache.zeppelin.spark.SparkRInterpreter INFO [2018-04-28 13:45:01,520] ({pool-2-thread-3} SchedulerFactory.java[jobStarted]:131) - Job paragraph_1523519805660_-887954373 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session619261541 INFO [2018-04-28 13:45:01,521] ({pool-2-thread-3} Paragraph.java[jobRun]:362) - run paragraph 20180412-122645_1892292733 using sql org.apache.zeppelin.interpreter.LazyOpenInterpreter@7ffa4f33 INFO [2018-04-28 13:46:00,002] ({DefaultQuartzScheduler_Worker-2} Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261 INFO [2018-04-28 13:47:00,002] ({DefaultQuartzScheduler_Worker-3} Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261 INFO [2018-04-28 13:48:00,001] ({DefaultQuartzScheduler_Worker-4} Note.java[run]:610) - skip to run blank paragraph. 20180412-160531_1456847261 On Sat, Apr 28, 2018 at 12:10 PM, Jeff Zhang wrote: > > Do you have any cron job enabled ? Please check the log to see how this > spark process is started. > > > Soheil Pourbafrani 于2018年4月28日周六 下午3:29写道: > >> Thanks for the reply, When I start Zeppelin daemon, only ZeppelinServer >> process will be created. when I just connect to the Zeppelin webui from the >> browser, (I've changed the port), a SparkSubmit process will be >> created! >> >> On Tue, Apr 24, 2018 at 10:14 AM, Jongyoul Lee >> wrote: >> >>> Hi, >>> >>> AFAIK, it might be something wrong. Can you explain more about it? >>> >>> JL >>> >>> On Sun, Apr 22, 2018 at 6:36 PM, Soheil Pourbafrani < >>> soheil.i...@gmail.com> wrote: >>> >>>> Hi, I use Zeppelin 7.3 >>>> >>>> Customizing Cassandra interpreter, I configured it for my Cassandra >>>> cluster. >>>> >>>> When
Re: Extra SparkSubmit process in running Cassandra queries
Thanks for the reply, When I start Zeppelin daemon, only ZeppelinServer process will be created. when I just connect to the Zeppelin webui from the browser, (I've changed the port), a SparkSubmit process will be created! On Tue, Apr 24, 2018 at 10:14 AM, Jongyoul Lee wrote: > Hi, > > AFAIK, it might be something wrong. Can you explain more about it? > > JL > > On Sun, Apr 22, 2018 at 6:36 PM, Soheil Pourbafrani > wrote: > >> Hi, I use Zeppelin 7.3 >> >> Customizing Cassandra interpreter, I configured it for my Cassandra >> cluster. >> >> When I try to get data from Cassandra using the command: >> >> %cassandra >> >> SELECT * FROM Key.Table ; >> >> I expect it only creates just a RemoteInterpreterServer process to fetch >> data from Cassandra, but in addition to RemoteInterpreterServer, a >> SparkSubmit process is created! >> >> I didn't use any spark code, just Cassandra CQL query, Why the >> SparkSubmit process is created? >> > > > > -- > 이종열, Jongyoul Lee, 李宗烈 > http://madeng.net >
Extra SparkSubmit process in running Cassandra queries
Hi, I use Zeppelin 7.3 Customizing Cassandra interpreter, I configured it for my Cassandra cluster. When I try to get data from Cassandra using the command: %cassandra SELECT * FROM Key.Table ; I expect it only creates just a RemoteInterpreterServer process to fetch data from Cassandra, but in addition to RemoteInterpreterServer, a SparkSubmit process is created! I didn't use any spark code, just Cassandra CQL query, Why the SparkSubmit process is created?
SPARK_SUBMIT_OPTIONS variable didin't applied!
I use SPARK_SUBMIT_OPTIONS variable in zeppeline-env.sh file to set some options for spark applications that runs for zeppelin. export SPARK_SUBMIT_OPTIONS="--executor-memory 512m --driver-memory 512m --executor-cores 2 --queue onlineQ" But running spark on the Yarn cluster, it didn't applied. When I set some options like executor-memory in interpreter setting, it will be applied, but I want to use some options like queue and I don't know if it's possible to set such option in interpreter settings?
running a Spark fat jar in Zeppelin
Hi, Is it possible, instead of codding Spark in Zeppelin environment, run a fat jar file into YARN cluster to fetch data for Zeppelin?