[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263612#comment-13263612 ] Hudson commented on MAPREDUCE-4169: --- Integrated in Hadoop-Hdfs-0.23-Build #240 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/240/]) svn merge -c 1331012. FIXES: MAPREDUCE-4169. Container Logs appear in unsorted order (Jonathan Eagles via bobby) (Revision 1331016) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1331016 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263632#comment-13263632 ] Hudson commented on MAPREDUCE-4169: --- Integrated in Hadoop-Hdfs-trunk #1027 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1027/]) MAPREDUCE-4169. Container Logs appear in unsorted order (Jonathan Eagles via bobby) (Revision 1331012) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1331012 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263669#comment-13263669 ] Hudson commented on MAPREDUCE-4169: --- Integrated in Hadoop-Mapreduce-trunk #1062 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1062/]) MAPREDUCE-4169. Container Logs appear in unsorted order (Jonathan Eagles via bobby) (Revision 1331012) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1331012 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262894#comment-13262894 ] Jonathan Eagles commented on MAPREDUCE-4169: Awesome, thanks so much for the thorough reviews Bobby and Harsh. Jon Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262150#comment-13262150 ] Jonathan Eagles commented on MAPREDUCE-4169: Addressed Bobby's comments as part of MAPREDUCE-4194. Posting new patch. Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262225#comment-13262225 ] Hadoop QA commented on MAPREDUCE-4169: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524357/MAPREDUCE-4169.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2310//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2310//console This message is automatically generated. Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262295#comment-13262295 ] Jonathan Eagles commented on MAPREDUCE-4169: Current test failures are preexisting. Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259457#comment-13259457 ] Harsh J commented on MAPREDUCE-4169: Patch looks good to me (Did you verify the results manually too? Looks about right but I've not verified it manually yet). +1 on this - just one nit: Can you also add in a very short 'purpose' comment for each place you've done the sorting at? Having tests seems difficult for these particular classes so we can at least do with comments to help prevent removal/regress. Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259625#comment-13259625 ] Jonathan Eagles commented on MAPREDUCE-4169: Thanks, Harsh. Canceling the patch to add comments. I did verify both container logs UI and log aggregation logs UI in the history server to verify logs appear in sorted order. Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259638#comment-13259638 ] Robert Joseph Evans commented on MAPREDUCE-4169: I also like the patch for the most part, but while reviewing it I saw one disturbing preexisting problem. The LogValue constructor is called from AppLogAggregatorImpl.uploadLogsFromContainer(), which gets its value from LocalDirsHandlerService.getLogDirs() which calls DirectoryCollection.getGoodDirs(), which returns an internal ArrayList. When we sort this we are sorting an internal data structre of DirectoryCollection, that appears to be something that can be called from multiple threads and that it too modifies internally, possibly on a separate thread. I think the correct thing to do is to modify DirectoryCollection to not return internal values, as we could get concurrent modification exceptions as it is. We either need to clone the Lists before returning them, or we need to make them concurrent in some way. I am fine if we do this on a different JIRA, because it is a preexisting problem. It is just that this JIRA will potentially exacerbate the problem. Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13259676#comment-13259676 ] Hadoop QA commented on MAPREDUCE-4169: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523797/MAPREDUCE-4169.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.logaggregation.TestAggregatedLogFormat org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2295//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2295//console This message is automatically generated. Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
