Hello,
I'm building an Apache Spark Streaming application and cannot make it log
to a file on the local filesystem *when running it on YARN*. How can
achieve this?
I've set log4.properties file so that it can successfully write to a log
file in /tmp directory on the local file system (shown below
For Yarn, you need to upload your log4j.properties separately from
your app's jar, because of some internal issues that are too boring to
explain here. :-)
Basically:
spark-submit --master yarn --files log4j.properties blah blah blah
Having to keep it outside your app jar is sub-optimal, and I
Marcelo and Burak,
Thank you very much for your explanations. Now I'm able to see my logs.
On Wed, Feb 11, 2015 at 7:52 PM, Marcelo Vanzin wrote:
> For Yarn, you need to upload your log4j.properties separately from
> your app's jar, because of some internal issues that are too boring to
> expla
