Hello all, I am trying to test JavaKafkaWordCount on Yarn, to make sure Yarn is working fine I am saving the output to hdfs. The example works fine in local mode but not on yarn mode. I cannot see any output logged when I changed the mode to yarn-client or yarn-cluster or cannot find any errors logged. For my application id I was looking for logs under /var/log/hadoop-yarn/containers (e.g /var/log/hadoop-yarn/containers/application_1439517792099_0010/container_1439517792099_0010_01_000003/stderr) but I cannot find anything useful information. Is it the only location where application logs are logged.
Also tried setting log output to spark.yarn.app.container.log.dir but got access denied error. Question: Do we need to have some special setup to run spark streaming on Yarn? How do we debug? Where to find more details to test streaming on Yarn. Thanks, Rachana
