[jira] Updated: (MAPREDUCE-1466) FileInputFormat should save #input-files in JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-1466: Release Note: Added a private configuration variable mapreduce.input.num.files, to store number of input files being processed by M/R job. > FileInputFormat should save #input-files in JobConf > --- > > Key: MAPREDUCE-1466 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1466 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1466_yhadoop20-1.patch, > MAPREDUCE-1466_yhadoop20-2.patch, MAPREDUCE-1466_yhadoop20.patch > > > We already track the amount of data consumed by MR applications > (MAP_INPUT_BYTES), alongwith, it would be useful to #input-files from the > client-side for analysis. Along the lines of MAPREDUCE-1403, it would be easy > to stick in the JobConf during job-submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1403) Save file-sizes of each of the artifacts in DistributedCache in the JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-1403: Release Note: Added private configuration variables: mapred.cache.files.filesizes and mapred.cache.archives.filesizes to store sizes of distributed cache artifacts per job. This can be used by tools like Gridmix in simulation runs. > Save file-sizes of each of the artifacts in DistributedCache in the JobConf > --- > > Key: MAPREDUCE-1403 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1403 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1403_yhadoop20-1.patch, > MAPREDUCE-1403_yhadoop20.patch > > > It would be a useful metric to collect... potentially GridMix could use it to > emulate jobs which use the DistributedCache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1466) FileInputFormat should save #input-files in JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-1466: Attachment: MAPREDUCE-1466_yhadoop20-2.patch Patch removing prefixes in file names generated due to incorrect diff command. > FileInputFormat should save #input-files in JobConf > --- > > Key: MAPREDUCE-1466 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1466 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1466_yhadoop20-1.patch, > MAPREDUCE-1466_yhadoop20-2.patch, MAPREDUCE-1466_yhadoop20.patch > > > We already track the amount of data consumed by MR applications > (MAP_INPUT_BYTES), alongwith, it would be useful to #input-files from the > client-side for analysis. Along the lines of MAPREDUCE-1403, it would be easy > to stick in the JobConf during job-submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1413) Improve logging of progress/errors in the job and task trackers
[
https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834105#action_12834105
]
Chris Douglas commented on MAPREDUCE-1413:
--
That seems tautological to me: if one's logger is configured to not print the
exception, then it is unavailable in the logs. But as a practical
accommodation, I see your point. OK.
Instead of defining {{toString()}} for the TaskTracker, could this just print
the relevant information in offerService? Or are you using that in the service
lifecycle branch somehow?
> Improve logging of progress/errors in the job and task trackers
> ---
>
> Key: MAPREDUCE-1413
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
>Affects Versions: 0.22.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1413.patch
>
>
> I have code that improves the logging of the trackers as they start stop and
> fail, through
> # More logging of events
> # including exception strings and stacks when things go wrong
> People's whose JTs and TTs aren't behaving may appreciate this
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1414) TestRecoveryManager can spin waiting for a job to be half done
[ https://issues.apache.org/jira/browse/MAPREDUCE-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834104#action_12834104 ] Chris Douglas commented on MAPREDUCE-1414: -- The test harness, I'd argue, is not more robust because there is a log when the test fails for ambiguous reasons. I tried running this test on my laptop, it failed because the job took more than a minute, and it was still unclear why- or whether- that was a problem. The log artifact was just as useless as nothing and the proposed patch doesn't change that; it makes an already flaky test even less reliable by introducing a failure case that is not implicit in what the test is validating. If the test is flaky, the framework and/or the test are flawed. The current patch proposes to fix neither one. > TestRecoveryManager can spin waiting for a job to be half done > --- > > Key: MAPREDUCE-1414 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1414 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1414.patch > > > This is something I've seen: the TestRecoveryManager spinning forever waiting > for a job to get half done. The test runner will eventually kill it, but that > loses any log and chance of finding the problem. > Solution: have a timeout on how long you wait for the job -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1354) Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses
[ https://issues.apache.org/jira/browse/MAPREDUCE-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1354: - Attachment: MAPREDUCE-1354_yhadoop20.patch Updated patch to incorporate MAPREDUCE-1495 alongwith fixing pretty much all uses of JobConf inside the scheduling code in JIP and the CS. > Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS > accesses > - > > Key: MAPREDUCE-1354 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1354 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Devaraj Das >Assignee: Arun C Murthy >Priority: Critical > Attachments: MAPREDUCE-1354_yhadoop20.patch, > MAPREDUCE-1354_yhadoop20.patch, MAPREDUCE-1354_yhadoop20.patch, > MAPREDUCE-1354_yhadoop20.patch, MAPREDUCE-1354_yhadoop20.patch, > MAPREDUCE-1354_yhadoop20.patch > > > It'd be nice to have the JobTracker object not be locked while accessing the > HDFS for reading the jobconf file and while writing the jobinfo file in the > submitJob method. We should see if we can avoid taking the lock altogether. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1307) Introduce the concept of Job Permissions
[ https://issues.apache.org/jira/browse/MAPREDUCE-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834092#action_12834092 ] Devaraj Das commented on MAPREDUCE-1307: Some comments summarizing an offline discussion with Vinod: 1) Protect getTaskDiagnostics() APIs also w.r.t access control. 2) ACLs for jobs should be displayed as part of jobdetails (the front page for jobs where all high level info is displayed). 3) If a user gets an access control error, he should be informed about the configured ACLs for the job. 4) Make the ACL part of the JobStatus object so that it is visible to all interested parties who might be interested (including the command line to get the job status). Also, the CompletedJobStatusStore can make use of this and enforce access control.. 5) Can we have the default ACL for job view enabled for the groups the user belongs to? Current patch makes the default to be '', meaning no one else except job-owner and superuser/supergroup can see others' jobs if mapreduce.cluster.job-authorization-enabled is set to true. > Introduce the concept of Job Permissions > > > Key: MAPREDUCE-1307 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1307 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: security >Reporter: Devaraj Das >Assignee: Vinod K V > Fix For: 0.22.0 > > Attachments: 1307-early-1.patch, MAPREDUCE-1307-20100210.txt, > MAPREDUCE-1307-20100211.txt, MAPREDUCE-1307-20100215.txt > > > It would be good to define the notion of job permissions analogous to file > permissions. Then the JobTracker can restrict who can "read" (e.g. look at > the job page) or "modify" (e.g. kill) jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Fix Version/s: 0.22.0 > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1398-1.txt, patch-1398-2.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Attachment: patch-1398-2.txt Patch with comments incorporated. bq. The default value for taskMemoryManagerEnabled was changed in the patch which seemed unnecessary. Can we instead override isTaskMemoryManagerEnabled, if we just want to short circuit this in the test case ? Instead of overriding isTaskMemoryManagerEnabled(), I made setTaskMemoryManagerEnabledFlag() method package private and called it from testcase to turn off memory management. > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1398-1.txt, patch-1398-2.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Status: Patch Available (was: Open) > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Attachments: patch-1398-1.txt, patch-1398-2.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1408) Allow customization of job submission policies
[ https://issues.apache.org/jira/browse/MAPREDUCE-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rahul k singh updated MAPREDUCE-1408: - Attachment: 1408-2.patch Implemented all the points mentioned above except: The JobMonitor::process method can probably be removed, since it doesn't do any meaningful work (the instance is instead passed to the downstream StatsCollector, which does). The monitor is then only responsible for determining when the job is done, not its outcome The unit test can subclass the downstream component instead of subclassing JobMonitor The above is not done , due to time constraints and also extend Statistics is taking time. Will open the new jira where we will remove the process method. For the question: If a job fails submission in serial mode, wil the JobFactory be woken up? yes > Allow customization of job submission policies > -- > > Key: MAPREDUCE-1408 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1408 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Reporter: rahul k singh > Attachments: 1408-1.patch, 1408-2.patch > > > Currently, gridmix3 replay job submission faithfully. For evaluation > purposes, it would be great if we can support other job submission policies > such as sequential job submission, or stress job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Status: Open (was: Patch Available) > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Attachments: patch-1398-1.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834087#action_12834087 ] Hemanth Yamijala commented on MAPREDUCE-1398: - Looks good overall. Few minor nits: - The default value for taskMemoryManagerEnabled was changed in the patch which seemed unnecessary. Can we instead override isTaskMemoryManagerEnabled, if we just want to short circuit this in the test case ? - We can use the setIndexCache API to initialize the index cache in the main TT also. - It would be really helpful to add a log line where we break the TaskLauncher's loop on detecting the killed condition. Can you please make these changes and make the patch run through Hudson ? > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Attachments: patch-1398-1.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1354) Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses
[ https://issues.apache.org/jira/browse/MAPREDUCE-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834078#action_12834078 ] Amareshwari Sriramadasu commented on MAPREDUCE-1354: Some comments on the patch: 1. JobInProgress constructor calls methods like JobTracker.getSystemDirectoryForJob(). This method is called with JobTracker lock sometimes, JobTracker lock followed by JobInProgress lock sometimes and without any lock in this case. I think this should not effect any, but we should verify all the locking orders for all the back calls from JobInProgress constructor to JobTracker. 2. Unused import for org.apache.tools.ant.taskdefs.condition.HasMethod in JobInProgress 3. Variable JobTracker.EMPTY_TASK_DIAGNOSTICS is not used anywhere. > Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS > accesses > - > > Key: MAPREDUCE-1354 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1354 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Devaraj Das >Assignee: Arun C Murthy >Priority: Critical > Attachments: MAPREDUCE-1354_yhadoop20.patch, > MAPREDUCE-1354_yhadoop20.patch, MAPREDUCE-1354_yhadoop20.patch, > MAPREDUCE-1354_yhadoop20.patch, MAPREDUCE-1354_yhadoop20.patch > > > It'd be nice to have the JobTracker object not be locked while accessing the > HDFS for reading the jobconf file and while writing the jobinfo file in the > submitJob method. We should see if we can avoid taking the lock altogether. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1398) TaskLauncher remains stuck on tasks waiting for free nodes even if task is killed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1398: --- Attachment: patch-1398-1.txt Added one more assertion to the testcase. > TaskLauncher remains stuck on tasks waiting for free nodes even if task is > killed. > -- > > Key: MAPREDUCE-1398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1398 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Hemanth Yamijala >Assignee: Amareshwari Sriramadasu > Attachments: patch-1398-1.txt, patch-1398.txt > > > Tasks could be assigned to trackers for slots that are running other tasks in > a commit pending state. This is an optimization done to pipeline task > assignment and launch. When the task reaches the tracker, it waits until > sufficient slots become free for it. This wait is done in the TaskLauncher > thread. Now, while waiting, if the task is killed externally (maybe because > the job finishes, etc), the TaskLauncher is not notified of this. So, it > continues to wait for the killed task to get sufficient slots. If slots do > not become free for a long time, this would result in considerable delay in > waking up the TaskLauncher thread. If the waiting task happens to be a high > RAM task, then it is also wasteful, because by waking up, it can make way for > normal tasks that can run on the available number of slots. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1354) Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses
[
https://issues.apache.org/jira/browse/MAPREDUCE-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-1354:
-
Attachment: MAPREDUCE-1354_yhadoop20.patch
Updated, fixed the framework to use the existing hasSpeculative{Maps|Reduces}
rather than fetch it from the JobConf each time.
> Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS
> accesses
> -
>
> Key: MAPREDUCE-1354
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1354
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Reporter: Devaraj Das
>Assignee: Arun C Murthy
>Priority: Critical
> Attachments: MAPREDUCE-1354_yhadoop20.patch,
> MAPREDUCE-1354_yhadoop20.patch, MAPREDUCE-1354_yhadoop20.patch,
> MAPREDUCE-1354_yhadoop20.patch, MAPREDUCE-1354_yhadoop20.patch
>
>
> It'd be nice to have the JobTracker object not be locked while accessing the
> HDFS for reading the jobconf file and while writing the jobinfo file in the
> submitJob method. We should see if we can avoid taking the lock altogether.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1455) Authorization for servlets
[ https://issues.apache.org/jira/browse/MAPREDUCE-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834056#action_12834056 ] Devaraj Das commented on MAPREDUCE-1455: bq. 3) As tasktracker doesn't have the job ACLs, when any one tries to access task logs of a job, I propose we store the job ACLs in a file say job-acls.xml) when task log files are created by taskTracker. And tasktracker will read this job-acls.xml when somebody tries to access task logs using web UI and does the authorization. I guess job-acls.xml can contain only the 2 config properties mapreduce.job.user.name and mapreduce.job.acl-view-job. Does it make sense to have the info in the log.info file that is already there in the task logs directory? > Authorization for servlets > -- > > Key: MAPREDUCE-1455 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1455 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker, security, tasktracker >Reporter: Devaraj Das >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > > This jira is about building the authorization for servlets (on top of > MAPREDUCE-1307). That is, the JobTracker/TaskTracker runs authorization > checks on web requests based on the configured job permissions. For e.g., if > the job permission is 600, then no one except the authenticated user can look > at the job details via the browser. The authenticated user in the servlet can > be obtained using the HttpServletRequest method. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1354) Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses
[
https://issues.apache.org/jira/browse/MAPREDUCE-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-1354:
-
Attachment: MAPREDUCE-1354_yhadoop20.patch
Updated patch to cache JobConf.getMemoryFor{Map|Reduce}Task.
> Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS
> accesses
> -
>
> Key: MAPREDUCE-1354
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1354
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Reporter: Devaraj Das
>Assignee: Arun C Murthy
>Priority: Critical
> Attachments: MAPREDUCE-1354_yhadoop20.patch,
> MAPREDUCE-1354_yhadoop20.patch, MAPREDUCE-1354_yhadoop20.patch,
> MAPREDUCE-1354_yhadoop20.patch
>
>
> It'd be nice to have the JobTracker object not be locked while accessing the
> HDFS for reading the jobconf file and while writing the jobinfo file in the
> submitJob method. We should see if we can avoid taking the lock altogether.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1495) Reduce locking contention on JobTracker.getTaskCompletionEvents()
[
https://issues.apache.org/jira/browse/MAPREDUCE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834030#action_12834030
]
Rajesh Balamohan commented on MAPREDUCE-1495:
-
Made the following changes to reduce contentions and profiled JT. This
virtually eliminated the
contentions caused by getTaskCompletionEvents
In JobTracker.java: (locking only jobs during get())
public TaskCompletionEvent[] getTaskCompletionEvents(
JobID jobid, int fromEventId, int maxEvents) throws IOException{
JobInProgress job = null;
synchronized (jobs) {
job = this.jobs.get(jobid);
}
if (null != job) {
if (job.inited()) {
return job.getTaskCompletionEvents(fromEventId, maxEvents);
} else {
return EMPTY_EVENTS;
}
}
return completedJobStatusStore.readJobTaskCompletionEvents(jobid,
fromEventId, maxEvents);
}
In Configuration.java: (eliminated synchronize on method level. This might be
required only when properties is null)
private Properties getProps() {
if (properties == null) {
synchronized(this) {
properties = new Properties();
loadResources(properties, resources, quietmode);
if (overlay!= null) {
properties.putAll(overlay);
if (storeResource) {
for (Map.Entry item: overlay.entrySet()) {
updatingResource.put((String) item.getKey(), "Unknown");
}
}
}
}
}
return properties;
}
> Reduce locking contention on JobTracker.getTaskCompletionEvents()
> -
>
> Key: MAPREDUCE-1495
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1495
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 0.20.1
>Reporter: Rajesh Balamohan
>
> While profiling JT for slow performance with small-jobs, it was observed that
> JobTracker.getTaskCompletionEvents() is attributing to 40% of lock contention
> on JT.
> This JIRA ticket is created to explore the possibilities of reducing the
> sychronized code block in this method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1307) Introduce the concept of Job Permissions
[ https://issues.apache.org/jira/browse/MAPREDUCE-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834003#action_12834003 ] Devaraj Das commented on MAPREDUCE-1307: Looks good overall. There seems to be an unintentional change in JobContext.java (to do with JOB_TOKEN_FILE). > Introduce the concept of Job Permissions > > > Key: MAPREDUCE-1307 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1307 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: security >Reporter: Devaraj Das >Assignee: Vinod K V > Fix For: 0.22.0 > > Attachments: 1307-early-1.patch, MAPREDUCE-1307-20100210.txt, > MAPREDUCE-1307-20100211.txt, MAPREDUCE-1307-20100215.txt > > > It would be good to define the notion of job permissions analogous to file > permissions. Then the JobTracker can restrict who can "read" (e.g. look at > the job page) or "modify" (e.g. kill) jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-326) The lowest level map-reduce APIs should be byte oriented
[
https://issues.apache.org/jira/browse/MAPREDUCE-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833924#action_12833924
]
Tom White commented on MAPREDUCE-326:
-
[Chris] > Section 3.3- addressing backwards compatibility- actually *introduces
a buffer copy*, unless the serializers are backed by the output collector
(which would make the call to {{collect}} redundant).
[Arun] > This is completely unworkable, it adds an extra buffer copy from the
map's into to the sort buffer (io.sort.mb) for the 'high-level'
api.
The proposal does not introduce extra buffer copies. The example in section 3.3
is really just a reorganization of the current code, since the serializers are
constructed with DataOutputStream objects - just as they are now. The point is
that the serialization is moved out of the kernel. There is no extra copying
going on.
There is a mistake in the document - it should say RawMapOutputCollector takes
DataOutputBuffer (not DataInputBuffer), or better, DataOutputStream objects -
sorry if this was misleading.
[Chris] > One can read directly from a stream into a byte-oriented record and
"serialize" that record into the MapOutputBuffer by a single buffer copy.
[Owen > Pipes *would* get much easier if it moved to the context object API.
Moving to Tom's API wouldn't help at all over the context object API.
This incurs an extra copy in the byte-oriented record (BytesWritable). In the
case of Pipes, for example, map outputs are read into a BytesWritable, which
are then written to the MapOutputBuffer. With a binary API it would be possible
to bypass the BytesWritable and write directly into the MapOutputBuffer.
[Arun] > I'm not sure I follow, what is the relation between
MapOutputBuffer.collect and IFile.append? Are you proposing we do away with the
sort?
No, of course not. Keys and values sent to MapOutputBuffer.collect() ultimately
find their way to IFile.append() (in MapOutputBuffer's sortAndSpill() method).
I'm just remarking that these become binary level interfaces, so there are no
generic types in these interfaces in this proposal.
Perhaps the term "API" is causing confusion here. As I have said this is not
intended for users - it should really be viewed as an integration point for
developers. This situation is like the one for developers writing MapReduce
schedulers. In this case the TaskScheduler class is package private, so
developers have to put their scheduler implementation into the same package.
For a binary layer, I am suggesting that we use the interface annotations to
achieve a similar effect, but without the limitation that their classes have to
go in the same package.
> The lowest level map-reduce APIs should be byte oriented
>
>
> Key: MAPREDUCE-326
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-326
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: eric baldeschwieler
> Attachments: MAPREDUCE-326-api.patch, MAPREDUCE-326.pdf
>
>
> As discussed here:
> https://issues.apache.org/jira/browse/HADOOP-1986#action_12551237
> The templates, serializers and other complexities that allow map-reduce to
> use arbitrary types complicate the design and lead to lots of object creates
> and other overhead that a byte oriented design would not suffer. I believe
> the lowest level implementation of hadoop map-reduce should have byte string
> oriented APIs (for keys and values). This API would be more performant,
> simpler and more easily cross language.
> The existing API could be maintained as a thin layer on top of the leaner API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-326) The lowest level map-reduce APIs should be byte oriented
[
https://issues.apache.org/jira/browse/MAPREDUCE-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833907#action_12833907
]
Doug Cutting commented on MAPREDUCE-326:
Chris> If the goal is a faster binary API, then this should use NIO primitives
[ ... ]
Perhaps this might instead look something like:
{code}
interface RawMapper {
void map(Split, RawMapOutput);
}
interface RawMapOutput {
// records are written as contiguous byte ranges here
WritableByteChannel() getChannel();
// call with the position of each record in the data written
void addRecord(long start, long length);
// utility to help keep track of bytes written
long getBytesWritten();
}
{code}
The goal is to permit the kernel to identify record boundaries (so that it can
compare, sort and transmit records) while at the same time minimize per-record
data copying. Getting this API right without benchmarking might prove
difficult. We should benchmark this under various scenarios: A key/value pair
of Writable instances, line-based data from a text file, and length-delimited,
raw binary data.
Chris> Better pipes/streaming workflows are explicitly considered in
MAPREDUCE-1183; one can imagine an implementation of the MapTask or ReduceTask
loading its user code in an implementation written in the native language.
Can you please elaborate? I don't see the words "pipes" or "streaming"
mentioned in that issue. How does one load Python, Ruby, C++, etc. into Java?
MAPREDUCE-1183 seems to me just to be a different way to encapsulate
configuration data, grouping it per extension point rather than centralizing it
in the job config.
> The lowest level map-reduce APIs should be byte oriented
>
>
> Key: MAPREDUCE-326
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-326
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: eric baldeschwieler
> Attachments: MAPREDUCE-326-api.patch, MAPREDUCE-326.pdf
>
>
> As discussed here:
> https://issues.apache.org/jira/browse/HADOOP-1986#action_12551237
> The templates, serializers and other complexities that allow map-reduce to
> use arbitrary types complicate the design and lead to lots of object creates
> and other overhead that a byte oriented design would not suffer. I believe
> the lowest level implementation of hadoop map-reduce should have byte string
> oriented APIs (for keys and values). This API would be more performant,
> simpler and more easily cross language.
> The existing API could be maintained as a thin layer on top of the leaner API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1496) org.apache.hadoop.mapred.lib.FieldSelectionMapReduce removes empty fields from key/value end
org.apache.hadoop.mapred.lib.FieldSelectionMapReduce removes empty fields from key/value end Key: MAPREDUCE-1496 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1496 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.1 Reporter: Maxim Zizin Priority: Critical If input record's key and/or value has empty fields in the end then these fields will be cut off by org.apache.hadoop.mapred.lib.FieldSelectionMapReduce -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1493) Authorization for job-history pages
[ https://issues.apache.org/jira/browse/MAPREDUCE-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833900#action_12833900 ] Devaraj Das commented on MAPREDUCE-1493: I think we anyway should log the job ACLs in the history file for the job. So in that sense, the first option seems to make more sense to me. > Authorization for job-history pages > --- > > Key: MAPREDUCE-1493 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1493 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker, security >Reporter: Vinod K V > Fix For: 0.22.0 > > > MAPREDUCE-1455 introduces authorization for most of the Map/Reduce jsp pages > and servlets, but left history pages. This JIRA will make sure that > authorization checks are made while accessing job-history pages also. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1476) committer.needsTaskCommit should not be called for a task cleanup attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833859#action_12833859 ] Hudson commented on MAPREDUCE-1476: --- Integrated in Hadoop-Mapreduce-trunk-Commit #241 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/241/]) . Fix the M/R framework to not call commit for special tasks like job setup/cleanup and task cleanup. Contributed by Amareshwari Sriramadasu. > committer.needsTaskCommit should not be called for a task cleanup attempt > - > > Key: MAPREDUCE-1476 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1476 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1476-1.txt, patch-1476-2.txt, > patch-1476-ydist.txt, patch-1476.txt > > > Currently, Task.done() calls committer.needsTaskCommit() to know whether it > needs a commit or not. This need not be called for task cleanup attempt as no > commit is required for a cleanup attempt. > Due to MAPREDUCE-1409, we saw a case where cleanup attempt went into > COMMIT_PENDING state. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1495) Reduce locking contention on JobTracker.getTaskCompletionEvents()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833836#action_12833836 ] Hemanth Yamijala commented on MAPREDUCE-1495: - Oh. Sorry. I was thinking you were going to do 1 above and try and modify the TaskCompletionEvents list. The jobs data structure is ordered for the view calls which are sorted by JobID (and therefore to some extent by submit time). The first option seems like a really simple easy fix. Can we attempt that first and re-run the test. > Reduce locking contention on JobTracker.getTaskCompletionEvents() > - > > Key: MAPREDUCE-1495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1495 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.20.1 >Reporter: Rajesh Balamohan > > While profiling JT for slow performance with small-jobs, it was observed that > JobTracker.getTaskCompletionEvents() is attributing to 40% of lock contention > on JT. > This JIRA ticket is created to explore the possibilities of reducing the > sychronized code block in this method. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1495) Reduce locking contention on JobTracker.getTaskCompletionEvents()
[
https://issues.apache.org/jira/browse/MAPREDUCE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833834#action_12833834
]
Rajesh Balamohan commented on MAPREDUCE-1495:
-
As of now, its implemented as follows in JobTracker
public synchronized TaskCompletionEvent[] getTaskCompletionEvents(
JobID jobid, int fromEventId, int maxEvents) throws IOException{
synchronized (this) {
JobInProgress job = this.jobs.get(jobid);
if (null != job) {
if (job.inited()) {
return job.getTaskCompletionEvents(fromEventId, maxEvents);
} else {
return EMPTY_EVENTS;
}
}
}
return completedJobStatusStore.readJobTaskCompletionEvents(jobid,
fromEventId, maxEvents);
}
Where, jobs is TreeMap().
It is possible to reduce to contention in 2 ways.
1. Reduce the synch to only JobInProgress job = this.jobs.get(jobid); Rest of
the code is independent of the synch block (asaik).
2. Change datastructure of jobs to ConcurrentHashMap().
This way, we can jobs.get(jobid) automatically becomes threadsafe and the total
synchornization itself can be eliminated. If its mandatory to maintain the
order, I have to try the 1st one.
> Reduce locking contention on JobTracker.getTaskCompletionEvents()
> -
>
> Key: MAPREDUCE-1495
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1495
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 0.20.1
>Reporter: Rajesh Balamohan
>
> While profiling JT for slow performance with small-jobs, it was observed that
> JobTracker.getTaskCompletionEvents() is attributing to 40% of lock contention
> on JT.
> This JIRA ticket is created to explore the possibilities of reducing the
> sychronized code block in this method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1495) Reduce locking contention on JobTracker.getTaskCompletionEvents()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833830#action_12833830 ] Hemanth Yamijala commented on MAPREDUCE-1495: - bq. Will try to check the possibility of using ConcurrentHashMap as it can drastically minimize the requirement of synchronized (and by default it supports 16 concurrent threads). Can you elaborate please ? The current data structure is a simple list. How are you modifying this to a map ? Also, I think the ordering of events is important. > Reduce locking contention on JobTracker.getTaskCompletionEvents() > - > > Key: MAPREDUCE-1495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1495 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.20.1 >Reporter: Rajesh Balamohan > > While profiling JT for slow performance with small-jobs, it was observed that > JobTracker.getTaskCompletionEvents() is attributing to 40% of lock contention > on JT. > This JIRA ticket is created to explore the possibilities of reducing the > sychronized code block in this method. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1495) Reduce locking contention on JobTracker.getTaskCompletionEvents()
[ https://issues.apache.org/jira/browse/MAPREDUCE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833819#action_12833819 ] Rajesh Balamohan commented on MAPREDUCE-1495: - Will try to check the possibility of using ConcurrentHashMap as it can drastically minimize the requirement of synchronized (and by default it supports 16 concurrent threads). I shall try and post the outcome here. > Reduce locking contention on JobTracker.getTaskCompletionEvents() > - > > Key: MAPREDUCE-1495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1495 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.20.1 >Reporter: Rajesh Balamohan > > While profiling JT for slow performance with small-jobs, it was observed that > JobTracker.getTaskCompletionEvents() is attributing to 40% of lock contention > on JT. > This JIRA ticket is created to explore the possibilities of reducing the > sychronized code block in this method. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1093) Java assertion failures triggered by tests
[
https://issues.apache.org/jira/browse/MAPREDUCE-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833818#action_12833818
]
Vinod K V commented on MAPREDUCE-1093:
--
bq. This patch disables the failing assert in TestTaskTrackerMemoryManager.
Instead can we just let it be and file a new issue, attaching any test logs if
possible there? I quickly looked at the code and seemed that it should work the
way it is expressed in code.
So, I am leaning towards letting it be and going ahead with MAPREDUCE-1092. If
you already have some logs, you can create a new issue rightaway; I can look
into it..
> Java assertion failures triggered by tests
> --
>
> Key: MAPREDUCE-1093
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1093
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Eli Collins
>Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1093.patch
>
>
> While running the tests with java asserts enabled the following two asserts
> fired:
> {code}
> testStateRefresh in TestQueueManager:
> try{
> Job job = submitSleepJob(10, 2, 10, 10, true,null, "default" );
> assert(job.isSuccessful()); <==
> }catch(Exception e){
> {code}
> {code}
> runJobExceedingMemoryLimit in TestTaskTrackerMemoryManager:
> for (TaskCompletionEvent tce : taskComplEvents) {
> // Every task HAS to fail
> assert (tce.getTaskStatus() == TaskCompletionEvent.Status.TIPFAILED ||
> tce <==
> .getTaskStatus() == TaskCompletionEvent.Status.FAILED);
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1494) TestJobDirCleanup verifies wrong jobcache directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833815#action_12833815 ] Vinod K V commented on MAPREDUCE-1494: -- Good catch! Looks that the null check (contents == null) was the one masking the error. > TestJobDirCleanup verifies wrong jobcache directory > --- > > Key: MAPREDUCE-1494 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1494 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker, test >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 0.22.0 > > > TestJobDirCleanup verifies tasktracker/jobcache directory to be empty. But > localization happens in tasktracker/user/jobcache after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1476) committer.needsTaskCommit should not be called for a task cleanup attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-1476: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this to trunk. Thanks, Amareshwari ! > committer.needsTaskCommit should not be called for a task cleanup attempt > - > > Key: MAPREDUCE-1476 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1476 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1476-1.txt, patch-1476-2.txt, > patch-1476-ydist.txt, patch-1476.txt > > > Currently, Task.done() calls committer.needsTaskCommit() to know whether it > needs a commit or not. This need not be called for task cleanup attempt as no > commit is required for a cleanup attempt. > Due to MAPREDUCE-1409, we saw a case where cleanup attempt went into > COMMIT_PENDING state. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1476) committer.needsTaskCommit should not be called for a task cleanup attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-1476: Release Note: Fixed Map/Reduce framework to not call commit task for special tasks like job setup/cleanup and task cleanup. > committer.needsTaskCommit should not be called for a task cleanup attempt > - > > Key: MAPREDUCE-1476 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1476 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1476-1.txt, patch-1476-2.txt, > patch-1476-ydist.txt, patch-1476.txt > > > Currently, Task.done() calls committer.needsTaskCommit() to know whether it > needs a commit or not. This need not be called for task cleanup attempt as no > commit is required for a cleanup attempt. > Due to MAPREDUCE-1409, we saw a case where cleanup attempt went into > COMMIT_PENDING state. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1495) Reduce locking contention on JobTracker.getTaskCompletionEvents()
Reduce locking contention on JobTracker.getTaskCompletionEvents() - Key: MAPREDUCE-1495 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1495 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.20.1 Reporter: Rajesh Balamohan While profiling JT for slow performance with small-jobs, it was observed that JobTracker.getTaskCompletionEvents() is attributing to 40% of lock contention on JT. This JIRA ticket is created to explore the possibilities of reducing the sychronized code block in this method. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1494) TestJobDirCleanup verifies wrong jobcache directory
TestJobDirCleanup verifies wrong jobcache directory --- Key: MAPREDUCE-1494 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1494 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker, test Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Priority: Minor Fix For: 0.22.0 TestJobDirCleanup verifies tasktracker/jobcache directory to be empty. But localization happens in tasktracker/user/jobcache after MAPREDUCE-856. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1414) TestRecoveryManager can spin waiting for a job to be half done
[ https://issues.apache.org/jira/browse/MAPREDUCE-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833757#action_12833757 ] Steve Loughran commented on MAPREDUCE-1414: --- I disagree. will timeout but it reacts by killing the process, creating an empy XML file. when the XSL transforms are applied to create HTML docs, will mention the missing file and skip it. Your test HTML reports -which may be on some remote hudson server, after all- won't contain any information as to why your build failed, except you get emails saying "junit test run failed". Having the test recognise and handle a situation which at least one person (me) has encountered would eliminate this, and I don't see why it should be rejected. The more robust your test harness is, the better. > TestRecoveryManager can spin waiting for a job to be half done > --- > > Key: MAPREDUCE-1414 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1414 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1414.patch > > > This is something I've seen: the TestRecoveryManager spinning forever waiting > for a job to get half done. The test runner will eventually kill it, but that > loses any log and chance of finding the problem. > Solution: have a timeout on how long you wait for the job -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1413) Improve logging of progress/errors in the job and task trackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833756#action_12833756 ] Steve Loughran commented on MAPREDUCE-1413: --- The log calls aren't logging the _whole_ exception twice # The log message includes the toString() value of the exception, normally the value of getMessage(), though that can be changed. This message doesn't include the stack trace # The second parameter passes in the raw exception chain with full stack trace. It is up to the log engine what it does with that; printing out the stack trace in text is one option, but I know of others (discard, save the fault as an XML-parseable). This is why log(text,Throwable) is the recommended way to log faults in all the main java log frameworks. # Sometimes I've seen loggers not log the toString() value of the exception passed in as param #2, which is less than ideal if you are trying to track down a problem and all you have are the logs. # Therefore, I do think what I've done is the best tactic to producing logs that are human and machine readable. # Will check out the javadoc error and submit a patch that corrects that. > Improve logging of progress/errors in the job and task trackers > --- > > Key: MAPREDUCE-1413 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1413 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1413.patch > > > I have code that improves the logging of the trackers as they start stop and > fail, through > # More logging of events > # including exception strings and stacks when things go wrong > People's whose JTs and TTs aren't behaving may appreciate this -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1476) committer.needsTaskCommit should not be called for a task cleanup attempt
[
https://issues.apache.org/jira/browse/MAPREDUCE-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833751#action_12833751
]
Amareshwari Sriramadasu commented on MAPREDUCE-1476:
bq. -1 core tests.
TestCommandLineJobSubmission failed with NoClassDefFoundError. See
MAPREDUCE-1275.
Stacktrace for the failure:
{noformat}
java.lang.NoClassDefFoundError: org/apache/hadoop/ipc/Server$Handler
at org.apache.hadoop.ipc.Server.start(Server.java:1343)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.activate(NameNode.java:318)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:308)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:421)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:410)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1261)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:278)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:127)
{noformat}
> committer.needsTaskCommit should not be called for a task cleanup attempt
> -
>
> Key: MAPREDUCE-1476
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1476
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.20.1
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-1476-1.txt, patch-1476-2.txt,
> patch-1476-ydist.txt, patch-1476.txt
>
>
> Currently, Task.done() calls committer.needsTaskCommit() to know whether it
> needs a commit or not. This need not be called for task cleanup attempt as no
> commit is required for a cleanup attempt.
> Due to MAPREDUCE-1409, we saw a case where cleanup attempt went into
> COMMIT_PENDING state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1476) committer.needsTaskCommit should not be called for a task cleanup attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833750#action_12833750 ] Hadoop QA commented on MAPREDUCE-1476: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435846/patch-1476-2.txt against trunk revision 909993. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/454/console This message is automatically generated. > committer.needsTaskCommit should not be called for a task cleanup attempt > - > > Key: MAPREDUCE-1476 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1476 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1476-1.txt, patch-1476-2.txt, > patch-1476-ydist.txt, patch-1476.txt > > > Currently, Task.done() calls committer.needsTaskCommit() to know whether it > needs a commit or not. This need not be called for task cleanup attempt as no > commit is required for a cleanup attempt. > Due to MAPREDUCE-1409, we saw a case where cleanup attempt went into > COMMIT_PENDING state. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1476) committer.needsTaskCommit should not be called for a task cleanup attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1476: --- Attachment: patch-1476-ydist.txt Patch for Yahoo! distribution. Ran ant test and test-patch. All tests passed except TestHdfsProxy. > committer.needsTaskCommit should not be called for a task cleanup attempt > - > > Key: MAPREDUCE-1476 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1476 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1476-1.txt, patch-1476-2.txt, > patch-1476-ydist.txt, patch-1476.txt > > > Currently, Task.done() calls committer.needsTaskCommit() to know whether it > needs a commit or not. This need not be called for task cleanup attempt as no > commit is required for a cleanup attempt. > Due to MAPREDUCE-1409, we saw a case where cleanup attempt went into > COMMIT_PENDING state. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1493) Authorization for job-history pages
[ https://issues.apache.org/jira/browse/MAPREDUCE-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833721#action_12833721 ] Vinod K V commented on MAPREDUCE-1493: -- For getting the ACLs for retired jobs, we actually have two options: - *Log the ACLs also into job-history itself* during job-submission and retrieve the same for access control. -- This avoids reading job-conf file for every request of a job specific page. -- Needs a change to JobHistory submisison event. - *Read the job-conf file on history also* to get the acls. -- Introduces extra round trip to DFS -- No change to JobHistory. I think the former is better performance wise. Thoughts? > Authorization for job-history pages > --- > > Key: MAPREDUCE-1493 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1493 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobtracker, security >Reporter: Vinod K V > Fix For: 0.22.0 > > > MAPREDUCE-1455 introduces authorization for most of the Map/Reduce jsp pages > and servlets, but left history pages. This JIRA will make sure that > authorization checks are made while accessing job-history pages also. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1307) Introduce the concept of Job Permissions
[ https://issues.apache.org/jira/browse/MAPREDUCE-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833711#action_12833711 ] Hadoop QA commented on MAPREDUCE-1307: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435845/MAPREDUCE-1307-20100215.txt against trunk revision 909993. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/323/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/323/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/323/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/323/console This message is automatically generated. > Introduce the concept of Job Permissions > > > Key: MAPREDUCE-1307 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1307 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: security >Reporter: Devaraj Das >Assignee: Vinod K V > Fix For: 0.22.0 > > Attachments: 1307-early-1.patch, MAPREDUCE-1307-20100210.txt, > MAPREDUCE-1307-20100211.txt, MAPREDUCE-1307-20100215.txt > > > It would be good to define the notion of job permissions analogous to file > permissions. Then the JobTracker can restrict who can "read" (e.g. look at > the job page) or "modify" (e.g. kill) jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
