[ https://issues.apache.org/jira/browse/SPARK-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14162409#comment-14162409 ]
Apache Spark commented on SPARK-3166: ------------------------------------- User 'GrahamDennis' has created a pull request for this issue: https://github.com/apache/spark/pull/1890 > Custom serialisers can't be shipped in application jars > ------------------------------------------------------- > > Key: SPARK-3166 > URL: https://issues.apache.org/jira/browse/SPARK-3166 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.0.2 > Reporter: Graham Dennis > > Spark cannot currently use a custom serialiser that is shipped with the > application jar. Trying to do this causes a java.lang.ClassNotFoundException > when trying to instantiate the custom serialiser in the Executor processes. > This occurs because Spark attempts to instantiate the custom serialiser > before the application jar has been shipped to the Executor process. A > reproduction of the problem is available here: > https://github.com/GrahamDennis/spark-custom-serialiser > I've verified this problem in Spark 1.0.2, and Spark master and 1.1 branches > as of August 21, 2014. This issue is related to SPARK-2878, and my fix for > that issue (https://github.com/apache/spark/pull/1890) also solves this. My > pull request was not merged because it adds the user jar to the Executor > processes' class path at launch time. Such a significant change was thought > by [~rxin] to require more QA, and should be considered for inclusion in 1.2 > at the earliest. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org