Each application gets its own executor processes, so there should be no problem running them in parallel.
Lan > On Apr 20, 2015, at 10:25 AM, Michael Ryabtsev <michael...@gmail.com> wrote: > > Hi Lan, > > Thanks for fast response. It could be a solution if it works. I have more > than one log4 properties file, for different run modes like debug/production, > for executor and for application core. I think I would like to keep them > separate. Then, I suppose I should give all other properties files a special > names and keep the executor configuration with the default name? Can I > conclude that going this way I will not be able to run several applications > on the same cluster in parallel? > > Regarding submit, I am not using it now, I submit from the code, but I think > I should consider this option. > > Thanks. > > On Mon, Apr 20, 2015 at 5:59 PM, Lan Jiang <ljia...@gmail.com > <mailto:ljia...@gmail.com>> wrote: > Rename your log4j_special.properties file as log4j.properties and place it > under the root of your jar file, you should be fine. > > If you are using Maven to build your jar, please the log4j.properties in the > src/main/resources folder. > > However, please note that if you have other dependency jar file in the > classpath that contains another log4j.properties file this way, it might not > work since the first log4j.properties file that is loaded will be used. > > You can also do spark-submit —file log4j_special.properties … ,which should > transfer your log4j property file to the worker nodes automatically without > you copying them manually. > > Lan > > > > On Apr 20, 2015, at 9:26 AM, Michael Ryabtsev <michael...@gmail.com > > <mailto:michael...@gmail.com>> wrote: > > > > Hi all, > > > > I need to configure spark executor log4j.properties on a standalone cluster. > > It looks like placing the relevant properties file in the spark > > configuration folder and setting the spark.executor.extraJavaOptions from > > my application code: > > sparkConf.set("spark.executor.extraJavaOptions", > > "-Dlog4j.configuration=log4j_special.properties"); > > does the work, and the executor logs are written in the required place and > > level. As far as I understand, it works, because the spark configuration > > folder is on the class path, and passing parameter without path works here. > > However, I would like to avoid deploying these properties to each worker > > spark configuration folder. > > I wonder, if I put the properties in my application jar, is there any way of > > telling executor to load them? > > > > Thanks, > > > > > > > > -- > > View this message in context: > > http://apache-spark-user-list.1001560.n3.nabble.com/Configuring-logging-properties-for-executor-tp22572.html > > > > <http://apache-spark-user-list.1001560.n3.nabble.com/Configuring-logging-properties-for-executor-tp22572.html> > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > <mailto:user-unsubscr...@spark.apache.org> > > For additional commands, e-mail: user-h...@spark.apache.org > > <mailto:user-h...@spark.apache.org> > > > >
