[jira] [Commented] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[ https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788203#comment-13788203 ] Hadoop QA commented on YARN-1130: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12607156/YARN-1130.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.TestJobCleanup The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.v2.TestUberAM {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2136//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2136//console This message is automatically generated. Improve the log flushing for tasks when mapred.userlog.limit.kb is set -- Key: YARN-1130 URL: https://issues.apache.org/jira/browse/YARN-1130 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.0.5-alpha Reporter: Paul Han Assignee: Paul Han Fix For: 2.0.5-alpha Attachments: YARN-1130.patch, YARN-1130.patch, YARN-1130.patch When userlog limit is set with something like this: {code} property namemapred.userlog.limit.kb/name value2048/value descriptionThe maximum size of user-logs of each task in KB. 0 disables the cap. /description /property {code} the log entry will be truncated randomly for the jobs. The log size is left between 1.2MB to 1.6MB. Since the log is already limited, avoid the log truncation is crucial for user. The other issue with the current impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will not flush to file until the container shutdown and logmanager close all appenders. If user likes to see the log during task execution, it doesn't support it. Will propose a patch to add a flush mechanism and also flush the log when task is done. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[ https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788354#comment-13788354 ] Paul Han commented on YARN-1130: Thanks Arun for picking this up! I've made some changes to simplify the fix to do flush of logging with wait. That should get rid of those outofmemory errors due to earlier addition of the asynchronous flush thread. I'll look into the other unit test failures. At first glance, they don't seem to directly relate to the changes submitted. :( Improve the log flushing for tasks when mapred.userlog.limit.kb is set -- Key: YARN-1130 URL: https://issues.apache.org/jira/browse/YARN-1130 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.0.5-alpha Reporter: Paul Han Assignee: Paul Han Fix For: 2.0.5-alpha Attachments: YARN-1130.patch, YARN-1130.patch, YARN-1130.patch When userlog limit is set with something like this: {code} property namemapred.userlog.limit.kb/name value2048/value descriptionThe maximum size of user-logs of each task in KB. 0 disables the cap. /description /property {code} the log entry will be truncated randomly for the jobs. The log size is left between 1.2MB to 1.6MB. Since the log is already limited, avoid the log truncation is crucial for user. The other issue with the current impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will not flush to file until the container shutdown and logmanager close all appenders. If user likes to see the log during task execution, it doesn't support it. Will propose a patch to add a flush mechanism and also flush the log when task is done. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[ https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13787573#comment-13787573 ] Hadoop QA commented on YARN-1130: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12607066/YARN-1130.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapred.TestTaskCommit The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.v2.TestUberAM {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2131//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2131//console This message is automatically generated. Improve the log flushing for tasks when mapred.userlog.limit.kb is set -- Key: YARN-1130 URL: https://issues.apache.org/jira/browse/YARN-1130 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.0.5-alpha Reporter: Paul Han Assignee: Paul Han Fix For: 2.0.5-alpha Attachments: YARN-1130.patch, YARN-1130.patch When userlog limit is set with something like this: {code} property namemapred.userlog.limit.kb/name value2048/value descriptionThe maximum size of user-logs of each task in KB. 0 disables the cap. /description /property {code} the log entry will be truncated randomly for the jobs. The log size is left between 1.2MB to 1.6MB. Since the log is already limited, avoid the log truncation is crucial for user. The other issue with the current impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will not flush to file until the container shutdown and logmanager close all appenders. If user likes to see the log during task execution, it doesn't support it. Will propose a patch to add a flush mechanism and also flush the log when task is done. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[ https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13787370#comment-13787370 ] Arun C Murthy commented on YARN-1130: - I'm reviewing this, meanwhile, [~paulhan] can you pls look at the unit test failures? Tx. Improve the log flushing for tasks when mapred.userlog.limit.kb is set -- Key: YARN-1130 URL: https://issues.apache.org/jira/browse/YARN-1130 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.0.5-alpha Reporter: Paul Han Assignee: Paul Han Fix For: 2.0.5-alpha Attachments: YARN-1130.patch When userlog limit is set with something like this: {code} property namemapred.userlog.limit.kb/name value2048/value descriptionThe maximum size of user-logs of each task in KB. 0 disables the cap. /description /property {code} the log entry will be truncated randomly for the jobs. The log size is left between 1.2MB to 1.6MB. Since the log is already limited, avoid the log truncation is crucial for user. The other issue with the current impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will not flush to file until the container shutdown and logmanager close all appenders. If user likes to see the log during task execution, it doesn't support it. Will propose a patch to add a flush mechanism and also flush the log when task is done. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[ https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770099#comment-13770099 ] Joep Rottinghuis commented on YARN-1130: Could anybody else review this please ? Improve the log flushing for tasks when mapred.userlog.limit.kb is set -- Key: YARN-1130 URL: https://issues.apache.org/jira/browse/YARN-1130 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.0.5-alpha Reporter: Paul Han Fix For: 2.0.5-alpha Attachments: YARN-1130.patch When userlog limit is set with something like this: {code} property namemapred.userlog.limit.kb/name value2048/value descriptionThe maximum size of user-logs of each task in KB. 0 disables the cap. /description /property {code} the log entry will be truncated randomly for the jobs. The log size is left between 1.2MB to 1.6MB. Since the log is already limited, avoid the log truncation is crucial for user. The other issue with the current impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will not flush to file until the container shutdown and logmanager close all appenders. If user likes to see the log during task execution, it doesn't support it. Will propose a patch to add a flush mechanism and also flush the log when task is done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[ https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767279#comment-13767279 ] Hadoop QA commented on YARN-1130: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12603162/YARN-1130.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle org.apache.hadoop.mapred.TestNetworkedJob org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestBlockLimits org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapreduce.security.TestMRCredentials org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestMerge org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapreduce.TestMRJobClient org.apache.hadoop.mapred.TestTaskCommit org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapreduce.security.TestBinaryTokenFile org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.ipc.TestSocketFactory org.apache.hadoop.mapred.TestJobSysDirWithDFS The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.conf.TestNoDefaultsJobConf {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1929//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1929//console This message is automatically generated. Improve the log flushing for tasks when mapred.userlog.limit.kb is set -- Key: YARN-1130 URL: https://issues.apache.org/jira/browse/YARN-1130 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.0.5-alpha Reporter: Paul Han Fix For: 2.0.5-alpha Attachments: YARN-1130.patch When userlog limit is set with something
[jira] [Commented] (YARN-1130) Improve the log flushing for tasks when mapred.userlog.limit.kb is set
[ https://issues.apache.org/jira/browse/YARN-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13765870#comment-13765870 ] Paul Han commented on YARN-1130: Patch is submitted. A few notes here: # Added a LogUtils class to facilitate triggering log flush since Log4J's interface such as LogManager doesn't provide a flush() interface # Modified Task class to trigger the flushing of logs before it sends TASK_DONE event to MRMaster. In some cases, TASK_DONE may trigger the container to be killed before all log being written to disk # ContainerLogAppender supports to flush the log when a special message is received. The log flush is done in a synchronous manner with a timeout. This will ensure the invoker of flush waits until logs are written to disk or timeout happens. Improve the log flushing for tasks when mapred.userlog.limit.kb is set -- Key: YARN-1130 URL: https://issues.apache.org/jira/browse/YARN-1130 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.0.5-alpha Reporter: Paul Han Fix For: 2.0.5-alpha Attachments: YARN-1130.patch When userlog limit is set with something like this: {code} property namemapred.userlog.limit.kb/name value2048/value descriptionThe maximum size of user-logs of each task in KB. 0 disables the cap. /description /property {code} the log entry will be truncated randomly for the jobs. The log size is left between 1.2MB to 1.6MB. Since the log is already limited, avoid the log truncation is crucial for user. The other issue with the current impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will not flush to file until the container shutdown and logmanager close all appenders. If user likes to see the log during task execution, it doesn't support it. Will propose a patch to add a flush mechanism and also flush the log when task is done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira