[jira] [Commented] (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209275#comment-13209275 ] Harsh J commented on MAPREDUCE-2170: Ambud, What gain do you see in disconnecting it off of other stats (CPU/Memory/etc.) in the regular heartbeats? > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Fix For: 0.24.0 > > Attachments: mapreduce.loadaverage.r3.diff, > mapreduce.loadaverage.r4.diff, mapreduce.loadaverage.r5.diff, > mapreduce.loadaverage.r6.diff > > Original Estimate: 20m > Remaining Estimate: 20m > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209262#comment-13209262 ] Ambud Sharma commented on MAPREDUCE-2170: - Hi, I am wondering if keeping that(systemaverageload) as a separate RPC call used/made when need would make more sense? -Ambud > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Fix For: 0.24.0 > > Attachments: mapreduce.loadaverage.r3.diff, > mapreduce.loadaverage.r4.diff, mapreduce.loadaverage.r5.diff, > mapreduce.loadaverage.r6.diff > > Original Estimate: 20m > Remaining Estimate: 20m > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176042#comment-13176042 ] Philip Zeyliger commented on MAPREDUCE-2170: I will return to the office on January 17th. For urgent matters, please contact Aparna or Philip L. The week between Christmas and New Years', your best bet is Chris L. Thanks, -- Philip > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Fix For: 0.24.0 > > Attachments: mapreduce.loadaverage.r3.diff, > mapreduce.loadaverage.r4.diff, mapreduce.loadaverage.r5.diff, > mapreduce.loadaverage.r6.diff > > Original Estimate: 20m > Remaining Estimate: 20m > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176041#comment-13176041 ] Harsh J commented on MAPREDUCE-2170: I took a look at NodeManager parts, and it appears that the plugin is only used for resource view use? How does the resource plugin come into play when it comes to the resource-based scheduler? > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Fix For: 0.24.0 > > Attachments: mapreduce.loadaverage.r3.diff, > mapreduce.loadaverage.r4.diff, mapreduce.loadaverage.r5.diff, > mapreduce.loadaverage.r6.diff > > Original Estimate: 20m > Remaining Estimate: 20m > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098572#comment-13098572 ] Harsh J commented on MAPREDUCE-2170: If there's still interest for this, I can update it up for the hadoop-yarn framework (ContainersMonitor). > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Fix For: 0.23.0 > > Attachments: mapreduce.loadaverage.r3.diff, > mapreduce.loadaverage.r4.diff, mapreduce.loadaverage.r5.diff, > mapreduce.loadaverage.r6.diff > > Original Estimate: 20m > Remaining Estimate: 20m > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047940#comment-13047940 ] Harsh J commented on MAPREDUCE-2170: Well this is strange… Went through about 15 of the failed list, but they do not appear related to my TT status patch at all. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Fix For: 0.23.0 > > Attachments: mapreduce.loadaverage.r3.diff, > mapreduce.loadaverage.r4.diff, mapreduce.loadaverage.r5.diff, > mapreduce.loadaverage.r6.diff > > Original Estimate: 20m > Remaining Estimate: 20m > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047939#comment-13047939 ] Hadoop QA commented on MAPREDUCE-2170: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482150/mapreduce.loadaverage.r6.diff against trunk revision 1134180. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.examples.terasort.TestTeraSort org.apache.hadoop.fs.TestDFSIO org.apache.hadoop.fs.TestFileSystem org.apache.hadoop.ipc.TestSocketFactory org.apache.hadoop.mapred.join.TestDatamerge org.apache.hadoop.mapred.lib.TestDelegatingInputFormat org.apache.hadoop.mapred.pipes.TestPipes org.apache.hadoop.mapred.TestBadRecords org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapred.TestCommandLineJobSubmission org.apache.hadoop.mapred.TestCompressedEmptyMapOutputs org.apache.hadoop.mapred.TestControlledMapReduceJob org.apache.hadoop.mapred.TestFileInputFormat org.apache.hadoop.mapred.TestJobClient org.apache.hadoop.mapred.TestJobDirCleanup org.apache.hadoop.mapred.TestJobHistory org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapred.TestJobQueueInformation org.apache.hadoop.mapred.TestJobStatusPersistency org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapred.TestJobTrackerXmlJsp org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestMapredHeartbeat org.apache.hadoop.mapred.TestMapredSystemDir org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestMiniMRDFSCaching org.apache.hadoop.mapred.TestMiniMRDFSSort org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.TestMiniMRWithDFS org.apache.hadoop.mapred.TestMultipleLevelCaching org.apache.hadoop.mapred.TestNodeRefresh org.apache.hadoop.mapred.TestRecoveryManager org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestSetupAndCleanupFailure org.apache.hadoop.mapred.TestSeveral org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapred.TestSubmitJob org.apache.hadoop.mapred.TestTaskFail org.apache.hadoop.mapred.TestWebUIAuthorization org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat org.apache.hadoop.mapreduce.lib.input.TestDelegatingInputFormat org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge org.apache.hadoop.mapreduce.lib.join.TestJoinProperties org.apache.hadoop.mapreduce.security.TestBinaryTokenFile org.apache.hadoop.mapreduce.security.TestTokenCacheOldApi org.apache.hadoop.mapreduce.security.TestTokenCache org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapreduce.TestMRJobClient org.apache.hadoop.security.authorize.TestServiceLevelAuthorization org.apache.hadoop.security.TestMapredGroupMappingServiceRefresh org.apache.hadoop.tools.TestCopyFiles org.apache.hadoop.tools.TestDistCh org.apache.hadoop.tools.TestHadoopArchives org.apache.hadoop.tools.TestHarFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1235#comment-1235 ] Hadoop QA commented on MAPREDUCE-2170: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12466938/mapreduce.loadaverage.r5.diff against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/54//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/54//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/54//console This message is automatically generated. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Blocker > Fix For: 0.22.0 > > Attachments: mapreduce.loadaverage.r3.diff, > mapreduce.loadaverage.r4.diff, mapreduce.loadaverage.r5.diff > > Original Estimate: 20m > Remaining Estimate: 20m > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[
https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977858#action_12977858
]
Harsh J Chouraria commented on MAPREDUCE-2170:
--
Running 'ant test-patch' passes here.
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 6 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test
framework compile.
[exec]
[exec]
[exec]
[exec]
[exec]
==
[exec]
==
[exec] Finished build.
[exec]
==
[exec]
==
[exec]
[exec]
BUILD SUCCESSFUL
Total time: 31 minutes 51 seconds
{code}
> Send out last-minute load averages in TaskTrackerStatus
> ---
>
> Key: MAPREDUCE-2170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Affects Versions: 0.22.0
> Environment: GNU/Linux
>Reporter: Harsh J Chouraria
>Assignee: Harsh J Chouraria
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce.loadaverage.r3.diff,
> mapreduce.loadaverage.r4.diff, mapreduce.loadaverage.r5.diff
>
> Original Estimate: 0.33h
> Remaining Estimate: 0.33h
>
> Load averages could be useful in scheduling. This patch looks to extend the
> existing Linux resource plugin (via /proc/loadavg file) to allow transmitting
> load averages of the last one minute via the TaskTrackerStatus.
> Patch is up for review, with test cases added, at:
> https://reviews.apache.org/r/20/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935309#action_12935309 ] Hadoop QA commented on MAPREDUCE-2170: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12460091/mapreduce.loadaverage.r4.diff against trunk revision 1038357. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapred.TestControlledMapReduceJob org.apache.hadoop.mapred.TestTTResourceReporting org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken org.apache.hadoop.tools.rumen.TestRumenJobTraces org.apache.hadoop.tools.TestHarFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/7//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/7//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/7//console This message is automatically generated. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Attachments: mapreduce.loadaverage.r3.diff, > mapreduce.loadaverage.r4.diff > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933867#action_12933867 ] Nigel Daley commented on MAPREDUCE-2170: Sorry for the long precommit comment by Hadoop QA. I'm trying to get this running on MR again. Lots of tests failing. Looks like trunk wasn't properly updated after 0.22 branching. I still see references to 0.22 in the ivy/libraries.properties file on trunk. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Attachments: mapreduce.loadaverage.r3.diff > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933733#action_12933733 ] Hadoop QA commented on MAPREDUCE-2170: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459720/mapreduce.loadaverage.r3.diff against trunk revision 1036745. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.conf.TestJobConf org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.examples.pi.math.TestSummation org.apache.hadoop.examples.terasort.TestTeraSort org.apache.hadoop.examples.TestBaileyBorweinPlouffe org.apache.hadoop.fs.slive.TestSlive org.apache.hadoop.fs.TestDFSIO org.apache.hadoop.fs.TestFileSystem org.apache.hadoop.fs.TestHarFileSystem org.apache.hadoop.fs.TestJHLA org.apache.hadoop.io.TestSequenceFileMergeProgress org.apache.hadoop.ipc.TestSocketFactory org.apache.hadoop.mapred.join.TestDatamerge org.apache.hadoop.mapred.lib.TestDelegatingInputFormat org.apache.hadoop.mapred.pipes.TestPipesAsDifferentUser org.apache.hadoop.mapred.pipes.TestPipes org.apache.hadoop.mapred.TestAdminOperationsProtocolWithServiceAuthorization org.apache.hadoop.mapred.TestAuditLogger org.apache.hadoop.mapred.TestBadRecords org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestClusterStatus org.apache.hadoop.mapred.TestCompositeTaskTrackerInstrumentation org.apache.hadoop.mapred.TestCompressedEmptyMapOutputs org.apache.hadoop.mapred.TestControlledMapReduceJob org.apache.hadoop.mapred.TestDebugScriptWithLinuxTaskController org.apache.hadoop.mapred.TestDebugScript org.apache.hadoop.mapred.TestEmptyJob org.apache.hadoop.mapred.TestFieldSelection org.apache.hadoop.mapred.TestIFile org.apache.hadoop.mapred.TestInputPath org.apache.hadoop.mapred.TestIsolationRunner org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapred.TestJobClient org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapred.TestJobDirCleanup org.apache.hadoop.mapred.TestJobExecutionAsDifferentUser org.apache.hadoop.mapred.TestJobHistory org.apache.hadoop.mapred.TestJobInProgressListener org.apache.hadoop.mapred.TestJobKillAndFail org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapred.TestJobQueueClient org.apache.hadoop.mapred.TestJobQueueInformation org.apache.hadoop.mapred.TestJobRetire org.apache.hadoop.mapred.TestJobStatusPersistency org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapred.TestJobTrackerInstrumentation org.apache.hadoop.mapred.TestJobTrackerStart org.apache.hadoop.mapred.TestJobTrackerXmlJsp org.apache.hadoop.mapred.TestJvmManager org.apache.hadoop.mapred.TestJvmReuse org.apache.hadoop.mapred.TestKeyValueTextInputFormat org.apache.hadoop.mapred.TestKillSubProcessesWithLinuxTaskController org.apache.hadoop.mapred.TestKillSubProcesses org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestLimitTasksPerJobTaskScheduler org.apache.hadoop.mapred.TestLinuxTaskController org.apache.hadoop.mapred.TestLocalizationWithLinuxTaskController org.apache.hadoop.mapred.TestMapProgress org.apache.hadoop.mapred.TestMapredHeartbeat org.apache.hadoop.mapred.TestMapredSystemDir org.apac
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[
https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930227#action_12930227
]
Scott Chen commented on MAPREDUCE-2170:
---
Hey Harsh,
Why do we need to override getSystemLoadAverage()? The native one should also
work for Linux case, right?
{code}
@Override
public float getSystemLoadAverage() {
readProcLoadAverageFile();
return loadAverage;
}
{code}
> Send out last-minute load averages in TaskTrackerStatus
> ---
>
> Key: MAPREDUCE-2170
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Affects Versions: 0.22.0
> Environment: GNU/Linux
>Reporter: Harsh J Chouraria
>Assignee: Harsh J Chouraria
>Priority: Minor
> Fix For: 0.22.0
>
> Original Estimate: 0.33h
> Remaining Estimate: 0.33h
>
> Load averages could be useful in scheduling. This patch looks to extend the
> existing Linux resource plugin (via /proc/loadavg file) to allow transmitting
> load averages of the last one minute via the TaskTrackerStatus.
> Patch is up for review, with test cases added, at:
> https://reviews.apache.org/r/20/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930194#action_12930194 ] Arun C Murthy commented on MAPREDUCE-2170: -- Harsh, can you please put up the patch here on jira and grant license to ASF for inclusion? Thanks. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930068#action_12930068 ] Harsh J Chouraria commented on MAPREDUCE-2170: -- I've updated the patch on the review board. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928752#action_12928752 ] Scott Chen commented on MAPREDUCE-2170: --- Hey Harsh, I think it is OK to cast it into float. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927465#action_12927465 ] Harsh J Chouraria commented on MAPREDUCE-2170: -- I began to change this, but have run into a small issue. How do I store a 'double' inside a Configuration? There are only ways to set/get floats from it. getSystemLoadAverage() in OperatingSystemMXBean returns a double. Is it okay if we change that to a float and use it? > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927317#action_12927317 ] Scott Chen commented on MAPREDUCE-2170: --- And thank you for adding this feature, Harsh. I think this is a good thing to have :) > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927315#action_12927315 ] Scott Chen commented on MAPREDUCE-2170: --- +1 on Allen's idea of using getSystemLoadAvg() in ResourceCalculatorPlugin.getSystemLoadAverage(). So if in the future people have added a new ResourceCalculatorPlugin for other platform, they can directly use this method. I think it is better to put this in ResourceCalculatorPlugin to keep similar tools in the same class. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927301#action_12927301 ] Harsh J Chouraria commented on MAPREDUCE-2170: -- Plugins could help if there are better detailed, platform-specific ways of getting resource statistics? > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927215#action_12927215 ] Allen Wittenauer commented on MAPREDUCE-2170: - Why does it have to be in a plug-in? Why can't it just be part of the TaskTracker in general? > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927150#action_12927150 ] Harsh J Chouraria commented on MAPREDUCE-2170: -- We could add a portable plugin, sure! Are there sufficient functions available as part of the JVM itself, for this? > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927145#action_12927145 ] Allen Wittenauer commented on MAPREDUCE-2170: - That's sort of my point: why not promote this to be a Hadoop feature rather than something tied to a specific OS if we can support it in a generic way? One of the bad things about load avg is that it doesn't give you a why. In other words, a node may be totally fine running more cpu-bound tasks if the box is actually being hammered IO-wise. Since both conditions show as load But this would require a lot more smarts in the system and of the user. So while it makes me slightly uncomfortable to use this value, load should be 'good enough' given the current state of the system. > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927138#action_12927138 ] Harsh J Chouraria commented on MAPREDUCE-2170: -- Hm, I didn't know about that MXBean thingy! [Just googled that method] I could make the change, but the Linux resource plugin is only loaded when the os is detected to be "linux", that still makes it (mostly) non-portable doesn't it? Is there a portable plugin? > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2170) Send out last-minute load averages in TaskTrackerStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927129#action_12927129 ] Allen Wittenauer commented on MAPREDUCE-2170: - Why not use getSystemLoadAvg() to make it portable? > Send out last-minute load averages in TaskTrackerStatus > --- > > Key: MAPREDUCE-2170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2170 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 > Environment: GNU/Linux >Reporter: Harsh J Chouraria >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.22.0 > > Original Estimate: 0.33h > Remaining Estimate: 0.33h > > Load averages could be useful in scheduling. This patch looks to extend the > existing Linux resource plugin (via /proc/loadavg file) to allow transmitting > load averages of the last one minute via the TaskTrackerStatus. > Patch is up for review, with test cases added, at: > https://reviews.apache.org/r/20/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
