[ https://issues.apache.org/jira/browse/SPARK-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated SPARK-1789: ----------------------------------- Assignee: Sean Owen > Multiple versions of Netty dependencies cause FlumeStreamSuite failure > ---------------------------------------------------------------------- > > Key: SPARK-1789 > URL: https://issues.apache.org/jira/browse/SPARK-1789 > Project: Spark > Issue Type: Bug > Components: Build > Affects Versions: 0.9.1 > Reporter: Sean Owen > Assignee: Sean Owen > Labels: flume, netty, test > Fix For: 1.0.0 > > > TL;DR is there is a bit of JAR hell trouble with Netty, that can be mostly > resolved and will resolve a test failure. > I hit the error described at > http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-startup-time-out-td1753.html > while running FlumeStreamingSuite, and have for a short while (is it just > me?) > velvia notes: > "I have found a workaround. If you add akka 2.2.4 to your dependencies, then > everything works, probably because akka 2.2.4 brings in newer version of > Jetty." > There are at least 3 versions of Netty in play in the build: > - the new Flume 1.4.0 dependency brings in io.netty:netty:3.4.0.Final, and > that is the immediate problem > - the custom version of akka 2.2.3 depends on io.netty:netty:3.6.6. > - but, Spark Core directly uses io.netty:netty-all:4.0.17.Final > The POMs try to exclude other versions of netty, but are excluding > org.jboss.netty:netty, when in fact older versions of io.netty:netty (not > netty-all) are also an issue. > The org.jboss.netty:netty excludes are largely unnecessary. I replaced many > of them with io.netty:netty exclusions until everything agreed on > io.netty:netty-all:4.0.17.Final. > But this didn't work, since Akka 2.2.3 doesn't work with Netty 4.x. > Down-grading to 3.6.6.Final across the board made some Spark code not compile. > If the build *keeps* io.netty:netty:3.6.6.Final as well, everything seems to > work. Part of the reason seems to be that Netty 3.x used the old > `org.jboss.netty` packages. This is less than ideal, but is no worse than the > current situation. > So this PR resolves the issue and improves the JAR hell, even if it leaves > the existing theoretical Netty 3-vs-4 conflict: > - Remove org.jboss.netty excludes where possible, for clarity; they're not > needed except with Hadoop artifacts > - Add io.netty:netty excludes where needed -- except, let akka keep its > io.netty:netty > - Change a bit of test code that actually depended on Netty 3.x, to use 4.x > equivalent > - Update SBT build accordingly > A better change would be to update Akka far enough such that it agrees on > Netty 4.x, but I don't know if that's feasible. -- This message was sent by Atlassian JIRA (v6.2#6252)