[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134414#comment-15134414 ] Steve Loughran commented on SPARK-12807: mixing dependency versions in a single project is dangerous. And it still runs the risk of being brittle against whatever version of jackson Hadoop namenodes run with. While I'm not a fan of shading, until we get better classpath/process isolation in nodemanagers, shading here avoids problems and will ensure that ASF spark releases work with ASF Hadoop releases. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134412#comment-15134412 ] Steve Loughran commented on SPARK-12807: It's good to hear that things work with older versions, but they do need to be compiled in sync. The risk with downgrading jackson versions is that someone who has upgraded will find their code won't link any more. This is the same dilemma that HADOOP-10104 created: revert or tell others "sorry, time to upgrade". We went with the latter, but have added jackson to the list of dependencies whose upgrades are traumatic: Guava, protobuf (which will never be upgraded on the 2.x line) > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133946#comment-15133946 ] Sean Owen commented on SPARK-12807: --- Why vary this to use an earlier version - seems like all downside? > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133045#comment-15133045 ] Marcelo Vanzin commented on SPARK-12807: bq. I don't think we can park at 2.2.x. Spark already uses 2.5 That could be done just for the shuffle service. (Also, to be fair, Spark compiles and works fine with 2.2.x.) But either solution is fine. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132191#comment-15132191 ] Sean Owen commented on SPARK-12807: --- I don't think we can park at 2.2.x. Spark already uses 2.5 > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131015#comment-15131015 ] Steve Loughran commented on SPARK-12807: the PR I've got shades jackson; leveldb must stay in sync with hadoop as you can't shade JNI ... for that you need to rebuild leveldb or do what YARN will eventually do: fork the plugins. That helps on so many levels > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130901#comment-15130901 ] Maciej Bryński commented on SPARK-12807: I can add that I'm using 1.6.0 compiled with Jackson 2.2.3 and it's working brilliant. So solution park it at the same version as Hadoop is quite acceptable. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130884#comment-15130884 ] Marcelo Vanzin commented on SPARK-12807: - Guava is shaded in the final jar - Netty is a different version (4.x for Spark vs. 3.x for Hadoop) which lives in a different package, so fortunately no conflict - leveldb was intentionally kept at the same version as Hadoop, exactly because of the JNI bits - the javax annotations are not used at runtime so there's no need to shade them That leaves only Jackson; in hindsight it would have been better not to include it in the shuffle service, but well, that's water under the bridge now. So that leaves two solutions: - park it at the same version as Hadoop; assuming different versions of Hadoop use the same version... - relocate it Relocation is probably the safest choice given the many versions of Hadoop people may be using. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102920#comment-15102920 ] Steve Loughran commented on SPARK-12807: One thing to think about here is ramping up a notch and shading all the downstream dependencies in the YARN shuffle JAR. This is a JAR designed to be used in a specific place, the classpath. It now includes: netty, leveldb, some bits of com.google (in 1.6), some javax.annotation. What is also has for extra fun is a leveldb jni.so in native, as well as a netty one. This is going to be a problem; unless you can somehow isolate and shade that this shuffle JAR is going to force in a specific leveldb version on every bit of code picking up this JAR. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102907#comment-15102907 ] Apache Spark commented on SPARK-12807: -- User 'steveloughran' has created a pull request for this issue: https://github.com/apache/spark/pull/10782 > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102883#comment-15102883 ] Steve Loughran commented on SPARK-12807: There's a PR to shade in trunk; I'm going to do a 1.6 PR too, which should be identical (initially for ease of testing that the 1.6 branch is fixed) > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102729#comment-15102729 ] Apache Spark commented on SPARK-12807: -- User 'steveloughran' has created a pull request for this issue: https://github.com/apache/spark/pull/10780 > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102662#comment-15102662 ] Steve Loughran commented on SPARK-12807: work on YARN for isolation will address this in Hadoop 2.8+. But that does nothing for Hadoop <= 2.8. Shading will do this > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102654#comment-15102654 ] Sean Owen commented on SPARK-12807: --- I see, it's only the shuffle and only 1.6, and only happens to affect the shuffle service on YARN. Spark has otherwise been using later Jackson for a while. Shading is indeed probably the best thing for all of Spark's usages. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102647#comment-15102647 ] Steve Loughran commented on SPARK-12807: FWIW, I'm workng on shading jackson in the shuffle JAR > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102645#comment-15102645 ] Steve Loughran commented on SPARK-12807: problem is there are no guarantees that the spark versions are backwards compatible with the older version. If they come first, the NM itself may fail. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102441#comment-15102441 ] Maciej Bryński commented on SPARK-12807: I'm asking if it's possible. About running Spark shuffle. Did you miss link to: https://issues.apache.org/jira/browse/SPARK-9439 ? Problem started with Spark 1.6.0, because it's first version of Spark where Spark Shuffle has Jackson dependency > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102430#comment-15102430 ] Sean Owen commented on SPARK-12807: --- Are you asking if it's possible, a possible explanation, a workaround? I'm still not sure why it's a problem (now). For example people seem to be running Spark shuffle just fine with recent Hadoop. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102416#comment-15102416 ] Maciej Bryński commented on SPARK-12807: Sean, Maybe it's possible to compile YARN Shuffle with different version of Jackson than version using by Spark Core ? > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102169#comment-15102169 ] Sean Owen commented on SPARK-12807: --- Yes in general I'd assume Spark's classes/dependencies are supposed to come first in order to work. This certainly doesn't resolve all possible problems but yes I would expect more problems if other older versions of libs are given precdence. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102158#comment-15102158 ] Steve Loughran commented on SPARK-12807: We can replicate this intermittently. It all depends on classpath ordering in the NM. If either versions complete set of JARs are loaded first: all is well. If there's a mix: stack trace. The ordering can not only break the shuffle and so DRA, it can stop the NM coming up. This is generally considered a serious issue by ops teams. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran >Priority: Critical > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[
https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097030#comment-15097030
]
Steve Loughran commented on SPARK-12807:
...+mismatch looks to be between version of
{{com.fasterxml.jackson.databind.ObjectMapper}} and the jackson core
> Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
>
>
> Key: SPARK-12807
> URL: https://issues.apache.org/jira/browse/SPARK-12807
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, YARN
>Affects Versions: 1.6.0
> Environment: A Hadoop cluster with Jackson 2.2.3, spark running with
> dynamic allocation enabled
>Reporter: Steve Loughran
>
> When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster,
> you get to see a stack trace in the NM logs, indicating a jackson 2.x version
> mismatch.
> (reported on the spark dev list)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097028#comment-15097028 ] Maciej Bryński commented on SPARK-12807: Observed today by me :) > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[
https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097026#comment-15097026
]
Steve Loughran commented on SPARK-12807:
Stack
{code}
So connecting it gives error:
2016-01-11 16:56:51,222 INFO containermanager.AuxServices
(AuxServices.java:addService(72)) - Adding auxiliary service
spark_shuffle, "spark_shuffle"
2016-01-11 16:56:51,439 FATAL nodemanager.NodeManager
(NodeManager.java:initAndStartNodeManager(465)) - Error starting
NodeManager
java.lang.NoSuchMethodError:
com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
at
com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:457)
at
com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:379)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.(ExternalShuffleBlockResolver.java:57)
at
org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.(ExternalShuffleBlockHandler.java:56)
at
org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:128)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:237)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:253)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:462)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)
{code}
> Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
>
>
> Key: SPARK-12807
> URL: https://issues.apache.org/jira/browse/SPARK-12807
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, YARN
>Affects Versions: 1.6.0
> Environment: A Hadoop cluster with Jackson 2.2.3, spark running with
> dynamic allocation enabled
>Reporter: Steve Loughran
>
> When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster,
> you get to see a stack trace in the NM logs, indicating a jackson 2.x version
> mismatch.
> (reported on the spark dev list)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097022#comment-15097022 ] Steve Loughran commented on SPARK-12807: [~jerryshao] flagged up SPARK-9439 as the patch that made jackson2 a dependency for the shuffle; it's why this is a 1.6+ issue > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096826#comment-15096826 ] Steve Loughran commented on SPARK-12807: Jackson versioning is really a symptom of a greater problem: lack of classpath isolation in YARN aux services. Fixing CP isolation there YARN-1573 is the best option for Hadoop 2.9+; forked JVMs even better as you get failure isolation. Short term (support for Hadoop <= 2.8), I don't know. I'm now confused about what's happening here —as in "why hasn't this problem surfaced before" > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096779#comment-15096779 ] Sean Owen commented on SPARK-12807: --- Why? Spark is already not on Jackson 2.2. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096711#comment-15096711 ] Maciej Bryński commented on SPARK-12807: I think there will be a big problem with Spark 2.0.0. As far as I know Jackson 2.2.3 will be not compatible with 2.5.3. M. > Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3 > > > Key: SPARK-12807 > URL: https://issues.apache.org/jira/browse/SPARK-12807 > Project: Spark > Issue Type: Bug > Components: Shuffle, YARN >Affects Versions: 1.6.0 > Environment: A Hadoop cluster with Jackson 2.2.3, spark running with > dynamic allocation enabled >Reporter: Steve Loughran > > When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster, > you get to see a stack trace in the NM logs, indicating a jackson 2.x version > mismatch. > (reported on the spark dev list) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
[
https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096537#comment-15096537
]
Steve Loughran commented on SPARK-12807:
Needs either an upgrade to Hadoop's own dependencies (HADOOP-12705) or YARN to
support custom classpaths for aux services (YARN-1593 or YARN-4755)
Without any of those fixes, Spark will need to be rebuild with
{{-Dfasterxml.jackson.version=2.2.3}}
> Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3
>
>
> Key: SPARK-12807
> URL: https://issues.apache.org/jira/browse/SPARK-12807
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, YARN
>Affects Versions: 1.6.0
> Environment: A Hadoop cluster with Jackson 2.2.3, spark running with
> dynamic allocation enabled
>Reporter: Steve Loughran
>
> When you try to try to use dynamic allocation on a Hadoop 2.6-based cluster,
> you get to see a stack trace in the NM logs, indicating a jackson 2.x version
> mismatch.
> (reported on the spark dev list)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
