[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed
[ https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-2724: Attachment: YARN-2724.3.patch fix -1 on findBug If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed - Key: YARN-2724 URL: https://issues.apache.org/jira/browse/YARN-2724 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation Affects Versions: 2.5.1 Reporter: Sumit Mohanty Assignee: Xuan Gong Attachments: YARN-2724.1.patch, YARN-2724.2.patch, YARN-2724.3.patch Look into the log output snippet. It looks like there is an issue during aggregation when an unreadable file is encountered. Likely, this results in bad encoding. {noformat} LogType: command-13.json LogLength: 13934 Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 sys=0.01, real=0.05 secs] 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs] 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, real=0.04 secs] {noformat} Specifically, look at the text after the exception text. There should be two more entries for log files but none exist. This is likely due to the fact that command-13.json is expected to be of length 13934 but its is not as the file was never read. I think, it should have been {noformat} LogType: command-13.json LogLength: Length of the exception text Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) {noformat} {noformat} LogType: errors-3.txt LogLength:0 Log Contents: {noformat} {noformat} LogType:gc.log LogLength:??? Log Contents: ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ... {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed
[ https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-2724: Attachment: YARN-2724.4.patch If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed - Key: YARN-2724 URL: https://issues.apache.org/jira/browse/YARN-2724 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation Affects Versions: 2.5.1 Reporter: Sumit Mohanty Assignee: Xuan Gong Attachments: YARN-2724.1.patch, YARN-2724.2.patch, YARN-2724.3.patch, YARN-2724.4.patch Look into the log output snippet. It looks like there is an issue during aggregation when an unreadable file is encountered. Likely, this results in bad encoding. {noformat} LogType: command-13.json LogLength: 13934 Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 sys=0.01, real=0.05 secs] 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs] 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, real=0.04 secs] {noformat} Specifically, look at the text after the exception text. There should be two more entries for log files but none exist. This is likely due to the fact that command-13.json is expected to be of length 13934 but its is not as the file was never read. I think, it should have been {noformat} LogType: command-13.json LogLength: Length of the exception text Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) {noformat} {noformat} LogType: errors-3.txt LogLength:0 Log Contents: {noformat} {noformat} LogType:gc.log LogLength:??? Log Contents: ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ... {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed
[ https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-2724: Attachment: YARN-2724.5.patch Same patch If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed - Key: YARN-2724 URL: https://issues.apache.org/jira/browse/YARN-2724 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation Affects Versions: 2.5.1 Reporter: Sumit Mohanty Assignee: Xuan Gong Attachments: YARN-2724.1.patch, YARN-2724.2.patch, YARN-2724.3.patch, YARN-2724.4.patch, YARN-2724.5.patch Look into the log output snippet. It looks like there is an issue during aggregation when an unreadable file is encountered. Likely, this results in bad encoding. {noformat} LogType: command-13.json LogLength: 13934 Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 sys=0.01, real=0.05 secs] 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs] 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, real=0.04 secs] {noformat} Specifically, look at the text after the exception text. There should be two more entries for log files but none exist. This is likely due to the fact that command-13.json is expected to be of length 13934 but its is not as the file was never read. I think, it should have been {noformat} LogType: command-13.json LogLength: Length of the exception text Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) {noformat} {noformat} LogType: errors-3.txt LogLength:0 Log Contents: {noformat} {noformat} LogType:gc.log LogLength:??? Log Contents: ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ... {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed
[ https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-2724: -- Target Version/s: (was: 2.5.1) bq. As the log aggregation is done by NM user, giving it the permissions to access the generated log file should fix this issue. Agreed. I guess the problem that YARN should address is to surface the issue with aggregation to the end-user - right now it's not clear what really happened. If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed - Key: YARN-2724 URL: https://issues.apache.org/jira/browse/YARN-2724 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation Affects Versions: 2.5.1 Reporter: Sumit Mohanty Assignee: Xuan Gong Look into the log output snippet. It looks like there is an issue during aggregation when an unreadable file is encountered. Likely, this results in bad encoding. {noformat} LogType: command-13.json LogLength: 13934 Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 sys=0.01, real=0.05 secs] 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs] 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, real=0.04 secs] {noformat} Specifically, look at the text after the exception text. There should be two more entries for log files but none exist. This is likely due to the fact that command-13.json is expected to be of length 13934 but its is not as the file was never read. I think, it should have been {noformat} LogType: command-13.json LogLength: Length of the exception text Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) {noformat} {noformat} LogType: errors-3.txt LogLength:0 Log Contents: {noformat} {noformat} LogType:gc.log LogLength:??? Log Contents: ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ... {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed
[ https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-2724: Attachment: YARN-2724.1.patch If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed - Key: YARN-2724 URL: https://issues.apache.org/jira/browse/YARN-2724 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation Affects Versions: 2.5.1 Reporter: Sumit Mohanty Assignee: Xuan Gong Attachments: YARN-2724.1.patch Look into the log output snippet. It looks like there is an issue during aggregation when an unreadable file is encountered. Likely, this results in bad encoding. {noformat} LogType: command-13.json LogLength: 13934 Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 sys=0.01, real=0.05 secs] 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs] 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, real=0.04 secs] {noformat} Specifically, look at the text after the exception text. There should be two more entries for log files but none exist. This is likely due to the fact that command-13.json is expected to be of length 13934 but its is not as the file was never read. I think, it should have been {noformat} LogType: command-13.json LogLength: Length of the exception text Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) {noformat} {noformat} LogType: errors-3.txt LogLength:0 Log Contents: {noformat} {noformat} LogType:gc.log LogLength:??? Log Contents: ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ... {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed
[ https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-2724: Attachment: YARN-2724.2.patch If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed - Key: YARN-2724 URL: https://issues.apache.org/jira/browse/YARN-2724 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation Affects Versions: 2.5.1 Reporter: Sumit Mohanty Assignee: Xuan Gong Attachments: YARN-2724.1.patch, YARN-2724.2.patch Look into the log output snippet. It looks like there is an issue during aggregation when an unreadable file is encountered. Likely, this results in bad encoding. {noformat} LogType: command-13.json LogLength: 13934 Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 sys=0.01, real=0.05 secs] 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs] 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, real=0.04 secs] {noformat} Specifically, look at the text after the exception text. There should be two more entries for log files but none exist. This is likely due to the fact that command-13.json is expected to be of length 13934 but its is not as the file was never read. I think, it should have been {noformat} LogType: command-13.json LogLength: Length of the exception text Log Contents: Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json (Permission denied)command-3.json13983Error aggregating log file. Log file : /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json (Permission denied) {noformat} {noformat} LogType: errors-3.txt LogLength:0 Log Contents: {noformat} {noformat} LogType:gc.log LogLength:??? Log Contents: ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ... {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)