I'm building a custom YARN app where the App Master is a log running service that can start jobs in containers.
The ideal situation for us would be to be able to move all the logs produced by each container to hdfs. The aggregation options that yarn provides does this but only after the app master finishes (and in our case it never finishes). Is there any way of doing this? Thanks Emmanuel