[
https://issues.apache.org/jira/browse/MAPREDUCE-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amar Kamat updated MAPREDUCE-3787:
--
Attachment: MAPREDUCE-3787-v1.9.patch
Attaching a patch that improves the Gridmix STRESS logic. Changes are as
follows:
# StressJobFactory:
## Update and submission happens in one thread. Earlier, the job submission
and status-polling/load-calculation happened in separate thread. Slower status
polling resulted in throttling of job submission.
## No RPC calls are made in stress mode. The Stress mode relies on the
status cached by JobMonitor.
## Load calculation logic optimized to return early if the load is
considerably lower or considerably higher.
### If the total number of tasks submitted (without considering their
progress) is lower than the max permissible limit, the cluster is considered
underloaded.
### If at anytime the actual load (considering the task progress) is
higher than the permissible limit, the cluster is considered overloaded without
iterating over the remaining jobs.
## Maintains a list of blacklisted jobs that the will be ignored.
# JobMonitor:
## This is now multi-threaded. Total number of threads can be specified
using the 'gridmix.polling.threads' config key.
## The sleep interval is now made configurable. The sleep time can be set
using 'gridmix.polling.delay'.
## Invokes statistics for lost jobs. Not doing this results into missing
status-update events.
# Statistics:
## JobStats now caches job-status along with a) Running job handle, b) total
number of map tasks and c) total number of reduce tasks. Other modules should
rely on these cached status objects instead of polling for job status
individually.
## Statistics maintains a count of total number of submitted maps and
reduces.
# Execution Summarizer:
## Now reports Lost Jobs. These are the jobs for which the status is
unavailable.
# JobFactory:
## Ignores reduce only jobs. Currently, Gridmix is incapable of handling
reduce only jobs.
Modified {{TestGridmixSummary}} to test summary on lost jobs. Added
{{TestGridmixStatistics}} to test the {{Statistics.java}} changes.
With this patch, following things are timed and reported:
1. Time taken to submit a job
2. Time taken to build job splits
3. Time taken to get the job's status
test-patch and ant-tests passed.
[Gridmix] Improve STRESS mode
-
Key: MAPREDUCE-3787
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3787
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/gridmix
Affects Versions: 0.24.0
Reporter: Amar Kamat
Assignee: Amar Kamat
Labels: gridmix, stress
Fix For: 0.23.1
Attachments: MAPREDUCE-3787-v1.9.patch
Gridmix STRESS mode can be improved as follows:
1. The sleep time in JobMonitor can be reduced and/or made configurable
2. Map and reduce load calculation in StressJobFactory can be done in one loop
3. Updating the overload status from the job submitter thread (inline)
4. Optimizations to avoid un-necessary progress check (which inturn would
result into delay)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
