Re: Spark 2.0 on YARN - Files in config archive not ending up on executor classpath

OK, JIRA created: https://issues.apache.org/jira/browse/SPARK-16080 Also, after looking at the code a bit I think I see the reason. If I'm correct, it may actually be a very easy fix. On Mon, Jun 20, 2016 at 1:21 PM Marcelo Vanzin wrote: > It doesn't hurt to have a bug

Re: Spark 2.0 on YARN - Files in config archive not ending up on executor classpath

It doesn't hurt to have a bug tracking it, in case anyone else has time to look at it before I do. On Mon, Jun 20, 2016 at 1:20 PM, Jonathan Kelly wrote: > Thanks for the confirmation! Shall I cut a JIRA issue? > > On Mon, Jun 20, 2016 at 10:42 AM Marcelo Vanzin

Re: Spark 2.0 on YARN - Files in config archive not ending up on executor classpath

Thanks for the confirmation! Shall I cut a JIRA issue? On Mon, Jun 20, 2016 at 10:42 AM Marcelo Vanzin wrote: > I just tried this locally and can see the wrong behavior you mention. > I'm running a somewhat old build of 2.0, but I'll take a look. > > On Mon, Jun 20, 2016 at

Re: Spark 2.0 on YARN - Files in config archive not ending up on executor classpath

I just tried this locally and can see the wrong behavior you mention. I'm running a somewhat old build of 2.0, but I'll take a look. On Mon, Jun 20, 2016 at 7:04 AM, Jonathan Kelly wrote: > Does anybody have any thoughts on this? > > On Fri, Jun 17, 2016 at 6:36 PM

Re: Spark 2.0 on YARN - Files in config archive not ending up on executor classpath

Does anybody have any thoughts on this? On Fri, Jun 17, 2016 at 6:36 PM Jonathan Kelly wrote: > I'm trying to debug a problem in Spark 2.0.0-SNAPSHOT > (commit bdf5fe4143e5a1a393d97d0030e76d35791ee248) where Spark's > log4j.properties is not getting picked up in the

Spark 2.0 on YARN - Files in config archive not ending up on executor classpath

I'm trying to debug a problem in Spark 2.0.0-SNAPSHOT (commit bdf5fe4143e5a1a393d97d0030e76d35791ee248) where Spark's log4j.properties is not getting picked up in the executor classpath (and driver classpath for yarn-cluster mode), so Hadoop's log4j.properties file is taking precedence in the YARN