You have to edit the conf/zeppelin-env.sh
add the following line:
export SPARK_SUBMIT_OPTIONS="--packages
org.elasticsearch::elasticsearch-spark:2.2.0.BUILD-SNAPSHOT"
and then restart Zeppelin:
zeppelin-daemon.sh stop
zeppelin-daemon.sh start
>From now on every time Zeppelin starts it will load the additional packages
automatically.
Last note: I did not test it on the elastic search libs, I have done that
with other libs and it worked (Amazon Kinesis libs). But it should work as
the principle is the same :)
Hope that helps,
Josef
On 16 November 2015 at 21:27, SiS <n...@cht3.com> wrote:
> Hi
>
> thanks a lot for your help. It worked like you described it. But I have
> another question. Would it be possible to define the dependency somehow
> centrally, so that it is not necessary to insert the %dep in all notebooks
> and especially not need to restart the Interpreter all time I start the
> notebook?
>
> BR
>
>
> > Am 10.11.2015 um 23:06 schrieb Jeff Steinmetz <
> jeffrey.steinm...@gmail.com>:
> >
> > You can load the hadoop dependency directly from the elastic search
> maven repository if you like.
> > I’m using the snapshot builds since it fixes a few issues that I’ve been
> testing with costin @ elastic recently.
> >
> > In your interpreter, you will want to set a new flag of es.nodes and
> list your comma separated elastic search IP addresses or host names.
> >
> > I.e, you can do this (assuming you want to use the native
> elasticsearch-spark approach - which is preferred over hadoop
> reader/writers)
> >
> >
> > %dep
> >
> > z.addRepo("Sonatype snapshot").url("
> https://oss.sonatype.org/content/repositories/snapshots";).snapshot
> > z.load("org.elasticsearch::elasticsearch-spark:2.2.0.BUILD-SNAPSHOT")
> >
> >
> >
> > %spark
> >
> > val query = “{ some ES style query string here }”
> >
> > val RDD = sc.esJsonRDD("evo-session-reader/session", query ) // this
> returns the original json, if you omit query, it will assume match_all
> > val RDD2 = sc.esRDD("evo-session-reader/session", query ) // this
> returns a Map[String,AnyRef]
> >
> >
> >
> >
> >
> >
> >
> > Best
> > Jeff Steinmetz
> >
> >
> > On 11/10/15, 1:45 PM, "SiS" <n...@cht3.com> wrote:
> >
> >> Hi Everybody,
> >>
> >> through the effect that I’m new to Spark and Zeppelin I hope my
> question I have is here in the right place.
> >> I played around with Zeppelin and Spark and tried to load data by
> connection to an elasticsearch cluster.
> >> But to be honest I have no clue how I have to setup zeppelin or the
> notebook to use the elasticsearch/hadoop/spark
> >> library (jar) so I’m able to connect using pyspark.
> >> Do I have to copy the jar somewhere in the zeppelin folders?
> >>
> >> My plan is to transfer an index/type from elasticsearch to datafframes
> in spark.
> >>
> >> Did somebody give me a short explanation for setting this up? Or could
> point me to the right documentation.
> >>
> >> Any help would be appreciated.
> >>
> >> Thanks a lot
> >> Sven
> >
>
>
