[ https://issues.apache.org/jira/browse/YARN-1440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13831847#comment-13831847 ]
ledion bitincka commented on YARN-1440: --------------------------------------- [~zjffdu] - I stand corrected, there's currently one TFile per application per node with container_id used as the key. While this is better than creating one file per container, it still leaves the cluster exposed to the small file problem, imagine a 1000 node cluster, running 10000 apps/day - this would lead to 10M new TFiles. My hope is reduced complexity at the log file level while punting the small file problem to the FS layer - the reasoning here being that not all filesystems which can be used on Hadoop have a small file problem! > Yarn aggregated logs are difficult for external tools to understand > ------------------------------------------------------------------- > > Key: YARN-1440 > URL: https://issues.apache.org/jira/browse/YARN-1440 > Project: Hadoop YARN > Issue Type: Improvement > Reporter: ledion bitincka > Labels: log-aggregation, logs, tfile, yarn > > The log aggregation feature in Yarn is awesome! However, the file type and > format in which the log files are aggregated into (TFile) should either be > much simpler or be made pluggable. The current TFile format forces anyone who > wants to see the files to either > a) use the web UI > b) use the CLI tools (yarn logs) or > c) write custom code to read the files > My suggestion would be to simplify the log collection by collecting and > writing the raw log files into a directory structure as follows: > {noformat} > /{log-collection-dir}/{app-id}/{container-id}/{log-file-name} > {noformat} > This way the application developers can (re)use a much wider array of tools > to process the logs. > For the readers who are not familiar with logs and their format you can find > more info the following two blog posts: > http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/ > http://blogs.splunk.com/2013/11/18/hadoop-2-0-rant/ -- This message was sent by Atlassian JIRA (v6.1#6144)