[jira] Commented: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12860180#action_12860180 ] Amareshwari Sriramadasu commented on MAPREDUCE-1219: +1 patch looks good. JobTracker Metrics causes undue load on JobTracker -- Key: MAPREDUCE-1219 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.21.0 Reporter: Jothi Padmanabhan Assignee: Sreekanth Ramakrishnan Attachments: MAPREDUCE-1219.patch, MR-1219-1.patch, MR-1219-2.patch, patch-1219-ydist.txt JobTrackerMetricsInst.doUpdates updates job-level counters of all running jobs into JobTracker's metrics causing very bad performance and hampers heartbeats. Since Job level metrics are better served by JobHistory, it may be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12860112#action_12860112 ] Amareshwari Sriramadasu commented on MAPREDUCE-1219: shall we remove the member variable jobMetrics itself? JobTracker Metrics causes undue load on JobTracker -- Key: MAPREDUCE-1219 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.21.0 Reporter: Jothi Padmanabhan Assignee: Amareshwari Sriramadasu Attachments: MAPREDUCE-1219.patch, MR-1219-1.patch, patch-1219-ydist.txt JobTrackerMetricsInst.doUpdates updates job-level counters of all running jobs into JobTracker's metrics causing very bad performance and hampers heartbeats. Since Job level metrics are better served by JobHistory, it may be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12859669#action_12859669 ] Hadoop QA commented on MAPREDUCE-1219: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442501/MR-1219-1.patch against trunk revision 936166. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/125/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/125/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/125/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/125/console This message is automatically generated. JobTracker Metrics causes undue load on JobTracker -- Key: MAPREDUCE-1219 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.21.0 Reporter: Jothi Padmanabhan Assignee: Amareshwari Sriramadasu Attachments: MAPREDUCE-1219.patch, MR-1219-1.patch, patch-1219-ydist.txt JobTrackerMetricsInst.doUpdates updates job-level counters of all running jobs into JobTracker's metrics causing very bad performance and hampers heartbeats. Since Job level metrics are better served by JobHistory, it may be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[
https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12859672#action_12859672
]
Sreekanth Ramakrishnan commented on MAPREDUCE-1219:
---
The justification is mentioned in the comment :
https://issues.apache.org/jira/browse/MAPREDUCE-1219?focusedCommentId=12779940page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12779940
I just have removed unused code from {{JobInProgress}}.
JobTracker Metrics causes undue load on JobTracker
--
Key: MAPREDUCE-1219
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.21.0
Reporter: Jothi Padmanabhan
Assignee: Amareshwari Sriramadasu
Attachments: MAPREDUCE-1219.patch, MR-1219-1.patch,
patch-1219-ydist.txt
JobTrackerMetricsInst.doUpdates updates job-level counters of all running
jobs into JobTracker's metrics causing very bad performance and hampers
heartbeats. Since Job level metrics are better served by JobHistory, it may
be a good idea to remove these from the metrics framework.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12858791#action_12858791 ] Amareshwari Sriramadasu commented on MAPREDUCE-1219: bq. shouldn't jobtracker publish its own metrics and not include job level details? Also, users can obtain the counters using different API. So can we remove the job level metrics aka counters from JobTracker metrics? +1 we should remove the Job level metrics from JobTracker metrics. Attached patch just removes the jobMetrics from doUpdates(). Shall we remove the collection part also from JobInProgress code, because we don't update anymore? JobTracker Metrics causes undue load on JobTracker -- Key: MAPREDUCE-1219 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.21.0 Reporter: Jothi Padmanabhan Assignee: Amareshwari Sriramadasu Attachments: MAPREDUCE-1219.patch, patch-1219-ydist.txt JobTrackerMetricsInst.doUpdates updates job-level counters of all running jobs into JobTracker's metrics causing very bad performance and hampers heartbeats. Since Job level metrics are better served by JobHistory, it may be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12858753#action_12858753 ] Sreekanth Ramakrishnan commented on MAPREDUCE-1219: --- In todays state, JobTracker publishes its metrics along with its running jobs metrics. The running jobs list can be pretty long and the metrics updating cycle is done every heartbeat. This causes a significant increase in heartbeat processing time. Also, the job level metrics are nothing other than counters of the running job. The counters of running job are obtained by locking up the job, which also does not help us in terms of performance. But looking at the information published, shouldn't jobtracker publish its own metrics and not include job level details? Also, users can obtain the counters using different API. So can we remove the job level metrics aka counters from JobTracker metrics? Thoughts? JobTracker Metrics causes undue load on JobTracker -- Key: MAPREDUCE-1219 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.21.0 Reporter: Jothi Padmanabhan Assignee: Amareshwari Sriramadasu Attachments: MAPREDUCE-1219.patch, patch-1219-ydist.txt JobTrackerMetricsInst.doUpdates updates job-level counters of all running jobs into JobTracker's metrics causing very bad performance and hampers heartbeats. Since Job level metrics are better served by JobHistory, it may be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12779922#action_12779922 ] V.Karthikeyan commented on MAPREDUCE-1219: -- Verified the Job metrics using FileContext property enabled. Ran jobs to verify the counters in sync with the Jobtracker UI and log file generated using FileContext enabled. JobTracker Metrics causes undue load on JobTracker -- Key: MAPREDUCE-1219 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.21.0 Reporter: Jothi Padmanabhan Assignee: Amareshwari Sriramadasu Attachments: MAPREDUCE-1219.patch, patch-1219-ydist.txt JobTrackerMetricsInst.doUpdates updates job-level counters of all running jobs into JobTracker's metrics causing very bad performance and hampers heartbeats. Since Job level metrics are better served by JobHistory, it may be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[
https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12779940#action_12779940
]
Amareshwari Sriramadasu commented on MAPREDUCE-1219:
test-patch result for Y!20 patch :
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no tests are needed for
this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
{noformat}
bq. -1 tests included.
The patch removes some existing code for which there are no unit tests.
JobTracker Metrics causes undue load on JobTracker
--
Key: MAPREDUCE-1219
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.21.0
Reporter: Jothi Padmanabhan
Assignee: Amareshwari Sriramadasu
Attachments: MAPREDUCE-1219.patch, patch-1219-ydist.txt
JobTrackerMetricsInst.doUpdates updates job-level counters of all running
jobs into JobTracker's metrics causing very bad performance and hampers
heartbeats. Since Job level metrics are better served by JobHistory, it may
be a good idea to remove these from the metrics framework.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1219) JobTracker Metrics causes undue load on JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12779941#action_12779941 ] Amareshwari Sriramadasu commented on MAPREDUCE-1219: All tests passed on my machine except TestHdfsProxy JobTracker Metrics causes undue load on JobTracker -- Key: MAPREDUCE-1219 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1219 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.21.0 Reporter: Jothi Padmanabhan Assignee: Amareshwari Sriramadasu Attachments: MAPREDUCE-1219.patch, patch-1219-ydist.txt JobTrackerMetricsInst.doUpdates updates job-level counters of all running jobs into JobTracker's metrics causing very bad performance and hampers heartbeats. Since Job level metrics are better served by JobHistory, it may be a good idea to remove these from the metrics framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
