and to make things even more interesting: The CDH *5.3* version of Spark 1.2 differs from the Apache Spark 1.2 release in using Akka version 2.2.3, the version used by Spark 1.1 and CDH 5.2. Apache Spark 1.2 uses Akka version 2.3.4.
so i just compiled a program that uses akka against apache spark 1.2.0, and it indeed does not run on CDH5.3.0. i get class incompatibility errors. On Tue, Jan 6, 2015 at 10:29 AM, Koert Kuipers <ko...@tresata.com> wrote: > if the classes are in the original location than i think its safe to say > that this makes it impossible for us to build one app that can run against > spark 1.0.x, 1.1.x and spark 1.2.x. > > thats no big deal, but it does beg the question of what compatibility can > reasonably be expected for spark 1.x series. i have seen a lot of focus on > backwards compatibility of the spark 1.x api, but to me thats kind of a > moot point if i cannot run apps against all 1.x versions anyhow since > dependencies are not compatible. > > > On Mon, Jan 5, 2015 at 5:08 PM, Marcelo Vanzin <van...@cloudera.com> > wrote: > >> Spark doesn't really shade akka; it pulls a different build (kept >> under the "org.spark-project.akka" group and, I assume, with some >> build-time differences from upstream akka?), but all classes are still >> in the original location. >> >> The upgrade is a little more unfortunate than just changing akka, >> since it also changes some transitive dependencies which also have >> compatibility issues (e.g. the typesafe config library). But I believe >> it's needed to support Scala 2.11... >> >> On Mon, Jan 5, 2015 at 8:27 AM, Koert Kuipers <ko...@tresata.com> wrote: >> > since spark shaded akka i wonder if it would work, but i doubt it >> > >> > On Mon, Jan 5, 2015 at 9:56 AM, Cody Koeninger <c...@koeninger.org> >> wrote: >> >> >> >> I haven't tried it with spark specifically, but I've definitely run >> into >> >> problems trying to depend on multiple versions of akka in one project. >> >> >> >> On Sat, Jan 3, 2015 at 11:22 AM, Koert Kuipers <ko...@tresata.com> >> wrote: >> >>> >> >>> hey Ted, >> >>> i am aware of the upgrade efforts for akka. however if spark 1.2 >> forces >> >>> me to upgrade all our usage of akka to 2.3.x while spark 1.0 and 1.1 >> force >> >>> me to use akka 2.2.x then we cannot build one application that runs >> on all >> >>> spark 1.x versions, which i would consider a major incompatibility. >> >>> best, koert >> >>> >> >>> >> >>> On Sat, Jan 3, 2015 at 12:11 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>>> >> >>>> Please see http://akka.io/news/2014/05/22/akka-2.3.3-released.html >> which >> >>>> points to >> >>>> >> http://doc.akka.io/docs/akka/2.3.3/project/migration-guide-2.2.x-2.3.x.html?_ga=1.35212129.1385865413.1420220234 >> >>>> >> >>>> Cheers >> >>>> >> >>>> On Fri, Jan 2, 2015 at 9:11 AM, Koert Kuipers <ko...@tresata.com> >> wrote: >> >>>>> >> >>>>> i noticed spark 1.2.0 bumps the akka version. since spark uses it's >> own >> >>>>> akka version, does this mean it can co-exist with another akka >> version in >> >>>>> the same JVM? has anyone tried this? >> >>>>> >> >>>>> we have some spark apps that also use akka (2.2.3) and spray. if >> >>>>> different akka versions causes conflicts then spark 1.2.0 would not >> be >> >>>>> backwards compatible for us... >> >>>>> >> >>>>> thanks. koert >> >>>> >> >>>> >> >>> >> >> >> > >> >> >> >> -- >> Marcelo >> > >
