When running long-lived job on YARN like Spark Streaming, I found that container logs gone after days on executor nodes, although the job itself is still running.
I am using cdh5.4.0 and have aggregated logs enabled. Because the local logs are gone on executor nodes, I don't see any aggregated logs on hdfs after the job is killed or failed. Is there a YARN config to keep the logs from being deleted for long-lived streaming job? -- Chen Song
