[jira] [Commented] (SPARK-4739) spark.files.userClassPathFirst does not work in local[*] mode
[ https://issues.apache.org/jira/browse/SPARK-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288050#comment-14288050 ] Marcelo Vanzin commented on SPARK-4739: --- I'm pretty sure I tested that change in local mode, explicitly overriding a jar that Spark uses with an incompatible one, and things worked. But I can check again... (BTW my comment is about SPARK-2996, not SPARK-5358.) > spark.files.userClassPathFirst does not work in local[*] mode > - > > Key: SPARK-4739 > URL: https://issues.apache.org/jira/browse/SPARK-4739 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 1.1.0 >Reporter: Tobias Pfeiffer > > The parameter "spark.files.userClassPathFirst=true" does not work when using > spark-submit with "\-\-master local\[3\]". In particular, even though my > application jar file contains netty-3.9.4.Final, the older version from the > spark-assembly jar file is loaded (cf. SPARK-4738). When using the same jars > with "\-\-master yarn-cluster" and "spark.yarn.user.classpath.first=true" > (cf. SPARK-2996), it works correctly and my bundled classes are used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4739) spark.files.userClassPathFirst does not work in local[*] mode
[ https://issues.apache.org/jira/browse/SPARK-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288047#comment-14288047 ] Marcelo Vanzin commented on SPARK-4739: --- I'm pretty sure I tested that change in local mode, explicitly overriding a jar that Spark uses with an incompatible one, and things worked. But I can check again... > spark.files.userClassPathFirst does not work in local[*] mode > - > > Key: SPARK-4739 > URL: https://issues.apache.org/jira/browse/SPARK-4739 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 1.1.0 >Reporter: Tobias Pfeiffer > > The parameter "spark.files.userClassPathFirst=true" does not work when using > spark-submit with "\-\-master local\[3\]". In particular, even though my > application jar file contains netty-3.9.4.Final, the older version from the > spark-assembly jar file is loaded (cf. SPARK-4738). When using the same jars > with "\-\-master yarn-cluster" and "spark.yarn.user.classpath.first=true" > (cf. SPARK-2996), it works correctly and my bundled classes are used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4739) spark.files.userClassPathFirst does not work in local[*] mode
[ https://issues.apache.org/jira/browse/SPARK-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288012#comment-14288012 ] Matthew Whelan commented on SPARK-4739: --- I'm not sure that resolving 5358 will resolve this issue, but it may. It seems to me that in the local env, we just end up using the system classloader for everything, which may or may not be the design. It lets you get away with stuff in local that you can't get away with in the remote case. > spark.files.userClassPathFirst does not work in local[*] mode > - > > Key: SPARK-4739 > URL: https://issues.apache.org/jira/browse/SPARK-4739 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 1.1.0 >Reporter: Tobias Pfeiffer > > The parameter "spark.files.userClassPathFirst=true" does not work when using > spark-submit with "\-\-master local\[3\]". In particular, even though my > application jar file contains netty-3.9.4.Final, the older version from the > spark-assembly jar file is loaded (cf. SPARK-4738). When using the same jars > with "\-\-master yarn-cluster" and "spark.yarn.user.classpath.first=true" > (cf. SPARK-2996), it works correctly and my bundled classes are used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4739) spark.files.userClassPathFirst does not work in local[*] mode
[ https://issues.apache.org/jira/browse/SPARK-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234433#comment-14234433 ] Marcelo Vanzin commented on SPARK-4739: --- BTW my fix for SPARK-2996 (https://github.com/apache/spark/pull/3233) should also fix this. > spark.files.userClassPathFirst does not work in local[*] mode > - > > Key: SPARK-4739 > URL: https://issues.apache.org/jira/browse/SPARK-4739 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 1.1.0 >Reporter: Tobias Pfeiffer > > The parameter "spark.files.userClassPathFirst=true" does not work when using > spark-submit with "\-\-master local\[3\]". In particular, even though my > application jar file contains netty-3.9.4.Final, the older version from the > spark-assembly jar file is loaded (cf. SPARK-4738). When using the same jars > with "\-\-master yarn-cluster" and "spark.yarn.user.classpath.first=true" > (cf. SPARK-2996), it works correctly and my bundled classes are used. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
