Dhruve Ashar created SPARK-16018: ------------------------------------ Summary: Shade netty for shuffle to work on YARN Key: SPARK-16018 URL: https://issues.apache.org/jira/browse/SPARK-16018 Project: Spark Issue Type: Bug Components: Shuffle Affects Versions: 2.0.0 Reporter: Dhruve Ashar Priority: Blocker Fix For: 2.0.0
We replaced LazyFileRegion with netty's DefaultFileRegion. Because of this (https://issues.apache.org/jira/browse/SPARK-15178), we have a dependency on netty 4.0.25.Final or >, but hadoop is on 4.0.23.Final, so the shuffle jar won't load in nodemanager as its trying to find a new version of the class. We need to shade netty's classes inorder to load the latest version we need and not depend on the one loaded by hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org