[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed

2014-10-23 Thread Xuan Gong (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-2724:

Attachment: YARN-2724.3.patch

fix -1 on findBug

 If an unreadable file is encountered during log aggregation then aggregated 
 file in HDFS badly formed
 -

 Key: YARN-2724
 URL: https://issues.apache.org/jira/browse/YARN-2724
 Project: Hadoop YARN
  Issue Type: Bug
  Components: log-aggregation
Affects Versions: 2.5.1
Reporter: Sumit Mohanty
Assignee: Xuan Gong
 Attachments: YARN-2724.1.patch, YARN-2724.2.patch, YARN-2724.3.patch


 Look into the log output snippet. It looks like there is an issue during 
 aggregation when an unreadable file is encountered. Likely, this results in 
 bad encoding.
 {noformat}
 LogType: command-13.json
 LogLength: 13934
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
   
 errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 
 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 
 sys=0.01, real=0.05 secs]
 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: 
 [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 
 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs]
 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 
 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 
 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, 
 real=0.04 secs]
 {noformat}
 Specifically, look at the text after the exception text. There should be two 
 more entries for log files but none exist. This is likely due to the fact 
 that command-13.json is expected to be of length 13934 but its is not as the 
 file was never read.
 I think, it should have been
 {noformat}
 LogType: command-13.json
 LogLength: Length of the exception text
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
 {noformat}
 {noformat}
 LogType: errors-3.txt
 LogLength:0
 Log Contents:
 {noformat}
 {noformat}
 LogType:gc.log
 LogLength:???
 Log Contents:
 ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ...
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed

2014-10-23 Thread Xuan Gong (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-2724:

Attachment: YARN-2724.4.patch

 If an unreadable file is encountered during log aggregation then aggregated 
 file in HDFS badly formed
 -

 Key: YARN-2724
 URL: https://issues.apache.org/jira/browse/YARN-2724
 Project: Hadoop YARN
  Issue Type: Bug
  Components: log-aggregation
Affects Versions: 2.5.1
Reporter: Sumit Mohanty
Assignee: Xuan Gong
 Attachments: YARN-2724.1.patch, YARN-2724.2.patch, YARN-2724.3.patch, 
 YARN-2724.4.patch


 Look into the log output snippet. It looks like there is an issue during 
 aggregation when an unreadable file is encountered. Likely, this results in 
 bad encoding.
 {noformat}
 LogType: command-13.json
 LogLength: 13934
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
   
 errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 
 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 
 sys=0.01, real=0.05 secs]
 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: 
 [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 
 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs]
 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 
 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 
 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, 
 real=0.04 secs]
 {noformat}
 Specifically, look at the text after the exception text. There should be two 
 more entries for log files but none exist. This is likely due to the fact 
 that command-13.json is expected to be of length 13934 but its is not as the 
 file was never read.
 I think, it should have been
 {noformat}
 LogType: command-13.json
 LogLength: Length of the exception text
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
 {noformat}
 {noformat}
 LogType: errors-3.txt
 LogLength:0
 Log Contents:
 {noformat}
 {noformat}
 LogType:gc.log
 LogLength:???
 Log Contents:
 ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ...
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed

2014-10-23 Thread Xuan Gong (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-2724:

Attachment: YARN-2724.5.patch

Same patch

 If an unreadable file is encountered during log aggregation then aggregated 
 file in HDFS badly formed
 -

 Key: YARN-2724
 URL: https://issues.apache.org/jira/browse/YARN-2724
 Project: Hadoop YARN
  Issue Type: Bug
  Components: log-aggregation
Affects Versions: 2.5.1
Reporter: Sumit Mohanty
Assignee: Xuan Gong
 Attachments: YARN-2724.1.patch, YARN-2724.2.patch, YARN-2724.3.patch, 
 YARN-2724.4.patch, YARN-2724.5.patch


 Look into the log output snippet. It looks like there is an issue during 
 aggregation when an unreadable file is encountered. Likely, this results in 
 bad encoding.
 {noformat}
 LogType: command-13.json
 LogLength: 13934
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
   
 errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 
 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 
 sys=0.01, real=0.05 secs]
 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: 
 [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 
 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs]
 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 
 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 
 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, 
 real=0.04 secs]
 {noformat}
 Specifically, look at the text after the exception text. There should be two 
 more entries for log files but none exist. This is likely due to the fact 
 that command-13.json is expected to be of length 13934 but its is not as the 
 file was never read.
 I think, it should have been
 {noformat}
 LogType: command-13.json
 LogLength: Length of the exception text
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
 {noformat}
 {noformat}
 LogType: errors-3.txt
 LogLength:0
 Log Contents:
 {noformat}
 {noformat}
 LogType:gc.log
 LogLength:???
 Log Contents:
 ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ...
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed

2014-10-22 Thread Vinod Kumar Vavilapalli (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated YARN-2724:
--
Target Version/s:   (was: 2.5.1)

bq. As the log aggregation is done by NM user, giving it the permissions to 
access the generated log file should fix this issue.
Agreed. I guess the problem that YARN should address is to surface the issue 
with aggregation to the end-user - right now it's not clear what really 
happened.

 If an unreadable file is encountered during log aggregation then aggregated 
 file in HDFS badly formed
 -

 Key: YARN-2724
 URL: https://issues.apache.org/jira/browse/YARN-2724
 Project: Hadoop YARN
  Issue Type: Bug
  Components: log-aggregation
Affects Versions: 2.5.1
Reporter: Sumit Mohanty
Assignee: Xuan Gong

 Look into the log output snippet. It looks like there is an issue during 
 aggregation when an unreadable file is encountered. Likely, this results in 
 bad encoding.
 {noformat}
 LogType: command-13.json
 LogLength: 13934
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
   
 errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 
 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 
 sys=0.01, real=0.05 secs]
 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: 
 [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 
 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs]
 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 
 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 
 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, 
 real=0.04 secs]
 {noformat}
 Specifically, look at the text after the exception text. There should be two 
 more entries for log files but none exist. This is likely due to the fact 
 that command-13.json is expected to be of length 13934 but its is not as the 
 file was never read.
 I think, it should have been
 {noformat}
 LogType: command-13.json
 LogLength: Length of the exception text
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
 {noformat}
 {noformat}
 LogType: errors-3.txt
 LogLength:0
 Log Contents:
 {noformat}
 {noformat}
 LogType:gc.log
 LogLength:???
 Log Contents:
 ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ...
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed

2014-10-22 Thread Xuan Gong (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-2724:

Attachment: YARN-2724.1.patch

 If an unreadable file is encountered during log aggregation then aggregated 
 file in HDFS badly formed
 -

 Key: YARN-2724
 URL: https://issues.apache.org/jira/browse/YARN-2724
 Project: Hadoop YARN
  Issue Type: Bug
  Components: log-aggregation
Affects Versions: 2.5.1
Reporter: Sumit Mohanty
Assignee: Xuan Gong
 Attachments: YARN-2724.1.patch


 Look into the log output snippet. It looks like there is an issue during 
 aggregation when an unreadable file is encountered. Likely, this results in 
 bad encoding.
 {noformat}
 LogType: command-13.json
 LogLength: 13934
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
   
 errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 
 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 
 sys=0.01, real=0.05 secs]
 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: 
 [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 
 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs]
 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 
 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 
 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, 
 real=0.04 secs]
 {noformat}
 Specifically, look at the text after the exception text. There should be two 
 more entries for log files but none exist. This is likely due to the fact 
 that command-13.json is expected to be of length 13934 but its is not as the 
 file was never read.
 I think, it should have been
 {noformat}
 LogType: command-13.json
 LogLength: Length of the exception text
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
 {noformat}
 {noformat}
 LogType: errors-3.txt
 LogLength:0
 Log Contents:
 {noformat}
 {noformat}
 LogType:gc.log
 LogLength:???
 Log Contents:
 ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ...
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (YARN-2724) If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed

2014-10-22 Thread Xuan Gong (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-2724:

Attachment: YARN-2724.2.patch

 If an unreadable file is encountered during log aggregation then aggregated 
 file in HDFS badly formed
 -

 Key: YARN-2724
 URL: https://issues.apache.org/jira/browse/YARN-2724
 Project: Hadoop YARN
  Issue Type: Bug
  Components: log-aggregation
Affects Versions: 2.5.1
Reporter: Sumit Mohanty
Assignee: Xuan Gong
 Attachments: YARN-2724.1.patch, YARN-2724.2.patch


 Look into the log output snippet. It looks like there is an issue during 
 aggregation when an unreadable file is encountered. Likely, this results in 
 bad encoding.
 {noformat}
 LogType: command-13.json
 LogLength: 13934
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
   
 errors-3.txt0gc.log-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K-15575K(184320K), 
 0.0488700 secs] 163840K-15575K(1028096K), 0.0492510 secs] [Times: user=0.06 
 sys=0.01, real=0.05 secs]
 2014-10-21T04:45:14.939+: 8.027: [GC2014-10-21T04:45:14.939+: 8.027: 
 [ParNew: 179415K-11865K(184320K), 0.0941310 secs] 179415K-17228K(1028096K), 
 0.0943140 secs] [Times: user=0.13 sys=0.04, real=0.09 secs]
 2014-10-21T04:46:42.099+: 95.187: [GC2014-10-21T04:46:42.099+: 
 95.187: [ParNew: 175705K-12802K(184320K), 0.0466420 secs] 
 181068K-18164K(1028096K), 0.0468490 secs] [Times: user=0.06 sys=0.00, 
 real=0.04 secs]
 {noformat}
 Specifically, look at the text after the exception text. There should be two 
 more entries for log files but none exist. This is likely due to the fact 
 that command-13.json is expected to be of length 13934 but its is not as the 
 file was never read.
 I think, it should have been
 {noformat}
 LogType: command-13.json
 LogLength: Length of the exception text
 Log Contents:
 Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json/grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-13.json
  (Permission denied)command-3.json13983Error aggregating log file. Log file : 
 /grid/0/yarn/log/application_1413865041660_0002/container_1413865041660_0002_01_04/command-3.json/grid/0/yarn/log/application_1413865041660_0002/contaierrors-13.txt0660_0002_01_04/command-3.json
  (Permission denied)
 {noformat}
 {noformat}
 LogType: errors-3.txt
 LogLength:0
 Log Contents:
 {noformat}
 {noformat}
 LogType:gc.log
 LogLength:???
 Log Contents:
 ..-20141021044514484052014-10-21T04:45:12.046+: 5.134: 
 [GC2014-10-21T04:45:12.046+: 5.134: [ParNew: 163840K- ...
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)