Marcelo and Burak,
Thank you very much for your explanations. Now I'm able to see my logs.
On Wed, Feb 11, 2015 at 7:52 PM, Marcelo Vanzin <van...@cloudera.com> wrote:
> For Yarn, you need to upload your log4j.properties separately from
> your app's jar, because of some internal issues that are too boring to
> explain here. :-)
>
> Basically:
>
> spark-submit --master yarn --files log4j.properties blah blah blah
>
> Having to keep it outside your app jar is sub-optimal, and I think
> there's a bug filed to fix this, but so far no one has really spent
> time looking at it.
>
>
> On Wed, Feb 11, 2015 at 4:29 AM, Emre Sevinc <emre.sev...@gmail.com>
> wrote:
> > Hello,
> >
> > I'm building an Apache Spark Streaming application and cannot make it
> log to
> > a file on the local filesystem when running it on YARN. How can achieve
> > this?
> >
> > I've set log4.properties file so that it can successfully write to a log
> > file in /tmp directory on the local file system (shown below partially):
> >
> > log4j.appender.file=org.apache.log4j.FileAppender
> > log4j.appender.file.File=/tmp/application.log
> > log4j.appender.file.append=false
> > log4j.appender.file.layout=org.apache.log4j.PatternLayout
> > log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}
> %-5p
> > %c{1}:%L - %m%n
> >
> > When I run my Spark application locally by using the following command:
> >
> > spark-submit --class myModule.myClass --master local[2] --deploy-mode
> > client myApp.jar
> >
> > It runs fine and I can see that log messages are written to
> > /tmp/application.log on my local file system.
> >
> > But when I run the same application via YARN, e.g.
> >
> > spark-submit --class myModule.myClass --master yarn-client --name
> > "myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar
> >
> > or
> >
> > spark-submit --class myModule.myClass --master yarn-cluster --name
> > "myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar
> >
> > I cannot see any /tmp/application.log on the local file system of the
> > machine that runs YARN.
> >
> > What am I missing?
> >
> >
> > --
> > Emre Sevinç
>
>
>
> --
> Marcelo
>
--
Emre Sevinc
