Logging in executors

Hi guys, I'm trying to enable logging in the executors but with no luck. According to the oficial documentation and several blogs, this should be done passing the "spark.executor.extraJavaOpts=-Dlog4j.configuration=[my-file]" to the spark-submit tool. I've tried both sending a reference to a clas

Re: Logging in executors

bq. --conf "spark.executor.extraJavaOptions=-Dlog4j. configuration=env/dev/log4j-driver.properties" I think the above may have a typo : you refer to log4j-driver.properties in both arguments. FYI On Wed, Apr 13, 2016 at 8:09 AM, Carlos Rojas Matas wrote: > Hi guys, > > I'm trying to enable log

Re: Logging in executors

Thanks for your response Ted. You're right, there was a typo. I changed it, now I'm executing: bin/spark-submit --master spark://localhost:7077 --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=env/dev/log4j-driver.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration

RE: Logging in executors

sure file my_custom.properties exists in /tmp folder on ALL of your worker nodes. Yong Date: Wed, 13 Apr 2016 14:18:24 -0300 Subject: Re: Logging in executors From: cma...@despegar.com To: yuzhih...@gmail.com CC: user@spark.apache.org Thanks for your response Ted. You're right, there was a

Re: Logging in executors

erties. In this > way, you need to make sure file my_custom.properties exists in /tmp folder > on ALL of your worker nodes. > > Yong > > ---------- > Date: Wed, 13 Apr 2016 14:18:24 -0300 > Subject: Re: Logging in executors > From: cma...@despegar.com

Re: Logging in executors

file >> 2) use like log4j.configuration=file:///tmp/my_custom.properties. In this >> way, you need to make sure file my_custom.properties exists in /tmp folder >> on ALL of your worker nodes. >> >> Yong >> >> -- >> Date: Wed,

Re: Logging in executors

are matched in your jar file >>> 2) use like log4j.configuration=file:///tmp/my_custom.properties. In >>> this way, you need to make sure file my_custom.properties exists in /tmp >>> folder on ALL of your worker nodes. >>> >>> Yong >>> >>&

Re: Logging in executors

u read the log4j document here: >>>> https://logging.apache.org/log4j/1.2/manual.html >>>> >>>> When you specify the log4j.configuration=my_custom.properties, you have >>>> 2 option: >>>> >>>> 1) the my_custom.properties has to be i

Re: Logging in executors

specified as >>>>> env/dev/log4j-executor.properties? >>>>> >>>>> If you read the log4j document here: >>>>> https://logging.apache.org/log4j/1.2/manual.html >>>>> >>>>> When you specify the log4j.configuration=my_cu

RE: Logging in executors

tion code. That also worked for us, but we went the extraJavaOptions route as it was less invasive on the application side. -Ashic. Date: Mon, 18 Apr 2016 10:32:03 -0300 Subject: Re: Logging in executors From: cma...@despegar.com To: yuzhih...@gmail.com CC: user@spark.apache.org Thanks Ted, alr