You're close: elasticsearch-hadoop snapshot (aka dev aka master) works on spark 1.2, 1.1 and 1.0, both core and sql elasticsearch-hadoop beta3 (not snapshot) works on spark 1.1 and spark 1.0, both core and sql elasticsearch-hadoop beta2 (not snapshot) works on spark 1.0 (core and sql)
The support for spark 1.3 hasn't been committed yet, I'd like to push it out once I (hopefully) manage to keep compatibility on the sql integration with spark 1.2 and lower. Once it is, it will be available in the nightly/dev builds published (as the name implies) every 24th - each day or night, depending on your timezon [1]. > Costin - I am amazed by your ability to keep all this straight - my head would explode dealing with all the dependencies in flux. Kudos to you. Thanks. As most (if not all) things in life, practice makes perfect ;) [1] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/install.html#download-dev On Wed, Mar 18, 2015 at 12:59 AM, Jeff Steinmetz < jeffrey.steinm...@gmail.com> wrote: > Thank you for the summary - you are confirming (as a sanity check for > myself): > > elasticsearch-hadoop beta3 (not snapshot) on spark core 1.1 only > elasticsearch-hadoop-beta3-SNAPSHOT with spark core 1.1, 1.2 and 1.3 -- as > long as I don't use Spark SQL when using 1.2 and 1.3 > > Costin - I am amazed by your ability to keep all this straight - my head > would explode dealing with all the dependencies in flux. Kudos to you. > > > On Tuesday, March 17, 2015 at 2:12:06 PM UTC-7, Costin Leau wrote: >> >> es-hadoop doesn't depend on akka, only on Spark. The scala version that >> es-hadoop is compiled against matches the one used by the Spark version >> compiled against for each release - typically this shouldn't pose a problem. >> >> Unfortunately, despite the minor version increments, some of the Spark >> APIs or components (in particular Spark SQL) have changed drastically >> between each release breaking backwards compatibility. For example, Beta3 >> works until Spark 1.1 (which was the latest stable release during its >> release) but not with 1.2. This is fixed in master however the current dev >> build doesn't work with Spark SQL in the newly released 1.3 (does work with >> Spark core). >> >> This has already been fixed locally however I'm having difficulties >> trying to preserve compatibility across the Spark SQL 1.2 release and 1.3. >> >> Long story short, as long as the dependencies for Spark are in order, the >> same should apply for es-hadoop as well since it relies only on Spark (and >> Scala of course). >> >> On Tue, Mar 17, 2015 at 10:43 PM, Jeff Steinmetz <jeffrey....@gmail.com> >> wrote: >> >>> There are plenty of spark / akka / scala / elasticsearch-hadoop >>> dependencies to keep track of. >>> >>> Is it true that elasticsearch-hadoop needs to be compiled for a specific >>> spark version to run correctly on the cluster? I'm also trying to keep >>> track of the akka version and scala version. i.e, wil es-hadoop compiled >>> for spark 1.2 work with Spark 1.3 ? >>> >>> When the elasticsearch-hadoop versions are released, as v2.0 v2.1, >>> v2.1.0.Beta3, at what point do we need to keep in mind what spark version >>> it was also compiled against? >>> i.e., is it safe to assume the es-hadoop versions are tied to a specific >>> spark core version? >>> >>> I've been keeping the following chart in my notes to see what all the >>> versions are with all dependencies >>> ============================= >>> >>> Akka Version Dependencies >>> Current Akka Stable Release: 2.3.9 >>> >>> Elasticsearch-Hadoop: 2.1.0Beta3 = Spark 1.1.0 >>> Elasticsearch-Hadoop: 2.1.0Beta3-SNAPSHOT = Spark 1.2.1 >>> Elasticsearch-Hadoop: what about spark 1.3 ? >>> >>> Spark: 1.3, Akka: 2.3.4-spark >>> Spark: 1.2, Akka: 2.3.4-spark >>> Spark: 1.1, Akka: 2.2.3-shaded-protobuf >>> >>> Activator 1.2.12 comes with with Akka 2.3.4 >>> >>> Play 2.3.8, akka 2.3.4, scala 2.11.1 (will also work with 2.10.4 ) >>> Play 2.2.x, akka 2.2.0 >>> >>> Spark Job Server 0.4.1, Spark Core 1.1.0, Akka, 2.2.4 >>> Spark Job Server Master as of Feb 22, 2015, Spark Core 1.2.0, Akka >>> 2.3.4, Scala 2.10.4 >>> >>> Akka persistence latest 2.3.4 or later >>> Akka 2.3.9 is released for Scala 2.10.4 and 2.11.5 >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJogdmf9-Fs4xDH9wVoVxvOG3CMSZ_XfCZb7%2BLZDT1AvDnxrZg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
