I'm running Spark on Amazon EMR using their install bootstrap.
My Scala code had println("message here") statements - where can I find the
output of these statements?
I changed my code to use log4j - my log.info and log.error output is nowhere to
be found.
I've checked /mnt/var/log/hadoop/steps and I can see Spark output .. actually,
it looks like YARN output from the spark job - but I don't see my error or log
or println statements anywhere.
I've also tried the bootstrap script argument --log-dir and pointed it to an S3
location - it creates the directory but doesn't put any logfiles there.
Help! :)
Sent from my iPhone
