Thank you for the summary - you are confirming (as a sanity check for myself):
elasticsearch-hadoop beta3 (not snapshot) on spark core 1.1 only elasticsearch-hadoop-beta3-SNAPSHOT with spark core 1.1, 1.2 and 1.3 -- as long as I don't use Spark SQL when using 1.2 and 1.3 Costin - I am amazed by your ability to keep all this straight - my head would explode dealing with all the dependencies in flux. Kudos to you. On Tuesday, March 17, 2015 at 2:12:06 PM UTC-7, Costin Leau wrote: > > es-hadoop doesn't depend on akka, only on Spark. The scala version that > es-hadoop is compiled against matches the one used by the Spark version > compiled against for each release - typically this shouldn't pose a problem. > > Unfortunately, despite the minor version increments, some of the Spark > APIs or components (in particular Spark SQL) have changed drastically > between each release breaking backwards compatibility. For example, Beta3 > works until Spark 1.1 (which was the latest stable release during its > release) but not with 1.2. This is fixed in master however the current dev > build doesn't work with Spark SQL in the newly released 1.3 (does work with > Spark core). > > This has already been fixed locally however I'm having difficulties trying > to preserve compatibility across the Spark SQL 1.2 release and 1.3. > > Long story short, as long as the dependencies for Spark are in order, the > same should apply for es-hadoop as well since it relies only on Spark (and > Scala of course). > > On Tue, Mar 17, 2015 at 10:43 PM, Jeff Steinmetz <jeffrey....@gmail.com > <javascript:>> wrote: > >> There are plenty of spark / akka / scala / elasticsearch-hadoop >> dependencies to keep track of. >> >> Is it true that elasticsearch-hadoop needs to be compiled for a specific >> spark version to run correctly on the cluster? I'm also trying to keep >> track of the akka version and scala version. i.e, wil es-hadoop compiled >> for spark 1.2 work with Spark 1.3 ? >> >> When the elasticsearch-hadoop versions are released, as v2.0 v2.1, >> v2.1.0.Beta3, at what point do we need to keep in mind what spark version >> it was also compiled against? >> i.e., is it safe to assume the es-hadoop versions are tied to a specific >> spark core version? >> >> I've been keeping the following chart in my notes to see what all the >> versions are with all dependencies >> ============================= >> >> Akka Version Dependencies >> Current Akka Stable Release: 2.3.9 >> >> Elasticsearch-Hadoop: 2.1.0Beta3 = Spark 1.1.0 >> Elasticsearch-Hadoop: 2.1.0Beta3-SNAPSHOT = Spark 1.2.1 >> Elasticsearch-Hadoop: what about spark 1.3 ? >> >> Spark: 1.3, Akka: 2.3.4-spark >> Spark: 1.2, Akka: 2.3.4-spark >> Spark: 1.1, Akka: 2.2.3-shaded-protobuf >> >> Activator 1.2.12 comes with with Akka 2.3.4 >> >> Play 2.3.8, akka 2.3.4, scala 2.11.1 (will also work with 2.10.4 ) >> Play 2.2.x, akka 2.2.0 >> >> Spark Job Server 0.4.1, Spark Core 1.1.0, Akka, 2.2.4 >> Spark Job Server Master as of Feb 22, 2015, Spark Core 1.2.0, Akka >> 2.3.4, Scala 2.10.4 >> >> Akka persistence latest 2.3.4 or later >> Akka 2.3.9 is released for Scala 2.10.4 and 2.11.5 >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/28ad3f78-8b3d-450a-a29d-06d3e6636cfd%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0c19d1fa-17b9-4e6e-a698-b49c7d6919d3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
