[jira] [Updated] (MAPREDUCE-3474) NM disk failure detection only covers local dirs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3474: --- Issue Type: Bug (was: Sub-task) Parent: (was: MAPREDUCE-3121) > NM disk failure detection only covers local dirs > - > > Key: MAPREDUCE-3474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Reporter: Eli Collins > > This is the MR counterpart to HDFS-1848. Like HDFS volume failure detection, > NM disk failure detection checks a subset of the disks, and a subset of the > directories. Eg the TT and the NM do not check the root disk for errors > unless a local dir resides on them. Even if a local dir resides on the root > disk the disk checking code only checks the local dirs so a failure only seen > when accessing a part of the disk no hosting the local dirs will not be > noticed. The disk that hosts the logs, pid, tmp dirs etc is critical, so if > needs to be checked as well, and the NM should shutdown if a critical disk is > not available (to prevent MR issues similar to HDFS-1848 and HDFS-2095). > Typically people currently work around this limitation by (aside from > ignoring it) by using raid-1 for the root disk or a health script that checks > the root disk health. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3441) NodeManager should identify failed disks becoming good back again
[ https://issues.apache.org/jira/browse/MAPREDUCE-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3441: --- Issue Type: Bug (was: Sub-task) Parent: (was: MAPREDUCE-3121) > NodeManager should identify failed disks becoming good back again > - > > Key: MAPREDUCE-3441 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3441 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Reporter: Ravi Gummadi > > MAPREDUCE-3121 makes NodeManager identify disk failures. But once a disk goes > down, it is marked as failed forever. To reuse that disk (after it becomes > good), NodeManager needs restart. This JIRA is to improve NodeManager to > reuse good disks(which could be bad some time back). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3440) Add tests for testing other NM components with disk failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3440: --- Issue Type: Improvement (was: Sub-task) Parent: (was: MAPREDUCE-3121) > Add tests for testing other NM components with disk failures > > > Key: MAPREDUCE-3440 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3440 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ravi Gummadi > > Add more tests to test other components when disks fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4495) Workflow Application Master in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated MAPREDUCE-4495: --- Attachment: (was: MapReduceWorkflowAM.pdf) > Workflow Application Master in YARN > --- > > Key: MAPREDUCE-4495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha >Reporter: Bo Wang >Assignee: Bo Wang > Attachments: MapReduceWorkflowAM.pdf > > > It is useful to have a workflow application master, which will be capable of > running a DAG of jobs. The workflow client submits a DAG request to the AM > and then the AM will manage the life cycle of this application in terms of > requesting the needed resources from the RM, and starting, monitoring and > retrying the application's individual tasks. > Compared to running Oozie with the current MapReduce Application Master, > these are some of the advantages: > - Less number of consumed resources, since only one application master will > be spawned for the whole workflow. > - Reuse of resources, since the same resources can be used by multiple > consecutive jobs in the workflow (no need to request/wait for resources for > every individual job from the central RM). > - More optimization opportunities in terms of collective resource requests. > - Optimization opportunities in terms of rewriting and composing jobs in the > workflow (e.g. pushing down Mappers). > - This Application Master can be reused/extended by higher systems like Pig > and hive to provide an optimized way of running their workflows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4495) Workflow Application Master in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated MAPREDUCE-4495: --- Attachment: MapReduceWorkflowAM.pdf > Workflow Application Master in YARN > --- > > Key: MAPREDUCE-4495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha >Reporter: Bo Wang >Assignee: Bo Wang > Attachments: MapReduceWorkflowAM.pdf > > > It is useful to have a workflow application master, which will be capable of > running a DAG of jobs. The workflow client submits a DAG request to the AM > and then the AM will manage the life cycle of this application in terms of > requesting the needed resources from the RM, and starting, monitoring and > retrying the application's individual tasks. > Compared to running Oozie with the current MapReduce Application Master, > these are some of the advantages: > - Less number of consumed resources, since only one application master will > be spawned for the whole workflow. > - Reuse of resources, since the same resources can be used by multiple > consecutive jobs in the workflow (no need to request/wait for resources for > every individual job from the central RM). > - More optimization opportunities in terms of collective resource requests. > - Optimization opportunities in terms of rewriting and composing jobs in the > workflow (e.g. pushing down Mappers). > - This Application Master can be reused/extended by higher systems like Pig > and hive to provide an optimized way of running their workflows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4495) Workflow Application Master in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated MAPREDUCE-4495: --- Attachment: MapReduceWorkflowAM.pdf Hi all, Here is a design doc for the Workflow AM. The code is being developed alongside. Any feedbacks are welcome. Thanks, Bo > Workflow Application Master in YARN > --- > > Key: MAPREDUCE-4495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha >Reporter: Bo Wang >Assignee: Bo Wang > Attachments: MapReduceWorkflowAM.pdf > > > It is useful to have a workflow application master, which will be capable of > running a DAG of jobs. The workflow client submits a DAG request to the AM > and then the AM will manage the life cycle of this application in terms of > requesting the needed resources from the RM, and starting, monitoring and > retrying the application's individual tasks. > Compared to running Oozie with the current MapReduce Application Master, > these are some of the advantages: > - Less number of consumed resources, since only one application master will > be spawned for the whole workflow. > - Reuse of resources, since the same resources can be used by multiple > consecutive jobs in the workflow (no need to request/wait for resources for > every individual job from the central RM). > - More optimization opportunities in terms of collective resource requests. > - Optimization opportunities in terms of rewriting and composing jobs in the > workflow (e.g. pushing down Mappers). > - This Application Master can be reused/extended by higher systems like Pig > and hive to provide an optimized way of running their workflows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4491) Encryption and Key Protection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451050#comment-13451050 ] Benoy Antony commented on MAPREDUCE-4491: - Key Protection is simple to explain. JobClient retrieves keys from a configured Keystore ,encrypts the keys along with jobId using cluster public key , submits the encrypted blob as part of the job credentials. TaskTrackers decrypts the encrypted blob using cluster private key during job localization, verifies that jobId inside the encrypted blob matches the JobId of the task. During Task Launch, the keys are made available to the child (task) process as an environment variable. Since the JobId is part of the encrypted blob, the replay attack is prevented with the JobId verification. It is easy to add integrity protection also. Now, the scheme was designed to be used in a secure cluster. It is good to explore whether it can be used in a non-secure cluster. One issue was with the cluster private key. It should be made accessible only to TaskTracker process. If the access is determined by the user's permissions, then tasks should be run as a different user. But it need not be the job owner. It can be a fixed user. I believe , you are bringing up another issue in this regard. If a rogue task can make a TT launch another rogue task with a jobId matching the one inside encrypted blob, then the keys area available to the newly launched rogue task. That's a good point. Basically the rogue task is acting as a JT/AppMaster. I am not sure whether that is possible. Even if its possible, there should be ways to detect it. > Encryption and Key Protection > - > > Key: MAPREDUCE-4491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4491 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: documentation, security, task-controller, tasktracker >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: Hadoop_Encryption.pdf, Hadoop_Encryption.pdf > > > When dealing with sensitive data, it is required to keep the data encrypted > wherever it is stored. Common use case is to pull encrypted data out of a > datasource and store in HDFS for analysis. The keys are stored in an external > keystore. > The feature adds a customizable framework to integrate different types of > keystores, support for Java KeyStore, read keys from keystores, and transport > keys from JobClient to Tasks. > The feature adds PGP encryption as a codec and additional utilities to > perform encryption related steps. > The design document is attached. It explains the requirement, design and use > cases. > Kindly review and comment. Collaboration is very much welcome. > I have a tested patch for this for 1.1 and will upload it soon as an initial > work for further refinement. > Update: The patches are uploaded to subtasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4646) client does not receive job diagnostics for failed jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-4646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4646:
--
Affects Version/s: 0.23.0
2.0.1-alpha
> client does not receive job diagnostics for failed jobs
> ---
>
> Key: MAPREDUCE-4646
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4646
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 2.0.1-alpha
>Reporter: Jason Lowe
>
> When a job fails the client is not showing any diagnostics. For example,
> running a fail job results in this not-so-helpful message from the client:
> {noformat}
> 2012-09-07 21:12:00,649 INFO [main] mapreduce.Job
> (Job.java:monitorAndPrintJob(1308)) - Job job_1347052207658_0001 failed with
> state FAILED due to:
> {noformat}
> ...and nothing else to go with it indicating what went wrong. The job
> diagnostics are apparently not making it back to the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4646) client does not receive job diagnostics for failed jobs
Jason Lowe created MAPREDUCE-4646:
-
Summary: client does not receive job diagnostics for failed jobs
Key: MAPREDUCE-4646
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4646
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Jason Lowe
When a job fails the client is not showing any diagnostics. For example,
running a fail job results in this not-so-helpful message from the client:
{noformat}
2012-09-07 21:12:00,649 INFO [main] mapreduce.Job
(Job.java:monitorAndPrintJob(1308)) - Job job_1347052207658_0001 failed with
state FAILED due to:
{noformat}
...and nothing else to go with it indicating what went wrong. The job
diagnostics are apparently not making it back to the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4509) Make link in "Aggregation is not enabled. Try the nodemanager at"
[ https://issues.apache.org/jira/browse/MAPREDUCE-4509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4509: - Fix Version/s: (was: 2.0.1-alpha) 3.0.0 > Make link in "Aggregation is not enabled. Try the nodemanager at" > - > > Key: MAPREDUCE-4509 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4509 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: webapps >Affects Versions: 0.23.0 >Reporter: Radim Kolar >Priority: Trivial > Fix For: 0.23.3, 3.0.0 > > > if log aggregation is disabled message is displayed > *Aggregation is not enabled. Try the nodemanager at reavers.com:9006* > It would be helpfull to make link to nodemanager clickable. > This message is located in > /hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/log/AggregatedLogsBlock.java > but i could not figure out how to make link in hamlet framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4644) mapreduce-client-jobclient-tests do not run from dist tarball
[ https://issues.apache.org/jira/browse/MAPREDUCE-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450972#comment-13450972 ] Jason Lowe commented on MAPREDUCE-4644: --- HADOOP-8738 explicitly removed the junit jar from the distro which lead to the jobclient tests breakage. > mapreduce-client-jobclient-tests do not run from dist tarball > - > > Key: MAPREDUCE-4644 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4644 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Affects Versions: 2.2.0-alpha >Reporter: Jason Lowe >Priority: Blocker > > The mapreduce jobclient tests rely on junit which is missing from the dist > tarball. This prevents running often-used tests like sleep jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4607) Race condition in ReduceTask completion can result in Task being incorrectly failed
[
https://issues.apache.org/jira/browse/MAPREDUCE-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450954#comment-13450954
]
Tom White commented on MAPREDUCE-4607:
--
Please go ahead and make the changes and I'll review.
> Race condition in ReduceTask completion can result in Task being incorrectly
> failed
> ---
>
> Key: MAPREDUCE-4607
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4607
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.1.0-alpha
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Attachments: MAPREDUCE-4607.1.patch, MAPREDUCE-4607.2.patch,
> MAPREDUCE-4607.3.patch, MAPREDUCE-4607.patch
>
>
> Problem reported by chackaravarthy in MAPREDUCE-4252
> This problem has been handled when speculative task launched for map task and
> other attempt got failed (not killed)
> Can the similar kind of scenario can happen in case of reduce task?
> Consider the following scenario for reduce task in case of speculation (one
> attempt got killed):
> 1. A task attempt is started.
> 2. A speculative task attempt for the same task is started.
> 3. The first task attempt completes and causes the task to transition to
> SUCCEEDED.
> 4. Then speculative task attempt will be killed because of the completion of
> first attempt.
> As a result, internal error will be thrown from this attempt
> (TaskImpl.MapRetroactiveKilledTransition) and hence task attempt failure
> leads to job failure.
> TaskImpl.MapRetroactiveKilledTransition
> if (!TaskType.MAP.equals(task.getType())) {
> LOG.error("Unexpected event for REDUCE task " + event.getType());
> task.internalError(event.getType());
> }
> So, do we need to have following code in MapRetroactiveKilledTransition also
> just like in MapRetroactiveFailureTransition.
> if (event instanceof TaskTAttemptEvent) {
> TaskTAttemptEvent castEvent = (TaskTAttemptEvent) event;
> if (task.getState() == TaskState.SUCCEEDED &&
> !castEvent.getTaskAttemptID().equals(task.successfulAttempt)) {
> // don't allow a different task attempt to override a previous
> // succeeded state
> return TaskState.SUCCEEDED;
> }
> }
> please check whether this is a valid case and give your suggestion.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4645) Providing a random seed to Slive should make the sequence of filenames completely deterministic
[ https://issues.apache.org/jira/browse/MAPREDUCE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-4645: Attachment: MAPREDUCE-4645.branch-0.23.patch This patch changes the dummy key for the SliveMapper to be a "splitID" and the Random number generator to be seeded with that splitID + user-specified seed. Also the PathFinder which generates the path, is given its own separate instance of Random, so that if you run the same Slive command twice, all ops will succeed the first time and fail the second time (because the file would already have been created / deleted the first time) > Providing a random seed to Slive should make the sequence of filenames > completely deterministic > --- > > Key: MAPREDUCE-4645 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4645 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance, test >Affects Versions: 0.23.1, 2.0.0-alpha >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Labels: performance, test > Attachments: MAPREDUCE-4645.branch-0.23.patch > > > Using the -random seed option still doesn't produce a deterministic sequence > of filenames. Hence there's no way to replicate the performance test. If I'm > providing a seed, its obvious that I want the test to be reproducible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4645) Providing a random seed to Slive should make the sequence of filenames completely deterministic
Ravi Prakash created MAPREDUCE-4645: --- Summary: Providing a random seed to Slive should make the sequence of filenames completely deterministic Key: MAPREDUCE-4645 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4645 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, test Affects Versions: 2.0.0-alpha, 0.23.1 Reporter: Ravi Prakash Assignee: Ravi Prakash Using the -random seed option still doesn't produce a deterministic sequence of filenames. Hence there's no way to replicate the performance test. If I'm providing a seed, its obvious that I want the test to be reproducible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4554) Job Credentials are not transmitted if security is turned off
[ https://issues.apache.org/jira/browse/MAPREDUCE-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450908#comment-13450908 ] Hadoop QA commented on MAPREDUCE-4554: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544256/MR_4554_trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2832//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2832//console This message is automatically generated. > Job Credentials are not transmitted if security is turned off > - > > Key: MAPREDUCE-4554 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4554 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Affects Versions: 2.0.0-alpha >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: trunk > > Attachments: MR_4554_trunk.patch, MR_4554_trunk.patch, > MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, > MR_4554_trunk.patch > > > Credentials (secret keys) can be passed to a job via > mapreduce.job.credentials.json or mapreduce.job.credentials.binary . > These credentials get submitted during job submission and are made available > to the task processes. > In HADOOP 1, these credentials get submitted and routed to task processes > even if security was off. > In HADOOP 2 , these credentials are transmitted only when the security is > turned on. > This should be changed for two reasons: > 1) It is not backward compatible. > 2) Credentials should be passed even if security is turned off . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4554) Job Credentials are not transmitted if security is turned off
[ https://issues.apache.org/jira/browse/MAPREDUCE-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated MAPREDUCE-4554: Status: Patch Available (was: Open) > Job Credentials are not transmitted if security is turned off > - > > Key: MAPREDUCE-4554 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4554 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Affects Versions: 2.0.0-alpha >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: trunk > > Attachments: MR_4554_trunk.patch, MR_4554_trunk.patch, > MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, > MR_4554_trunk.patch > > > Credentials (secret keys) can be passed to a job via > mapreduce.job.credentials.json or mapreduce.job.credentials.binary . > These credentials get submitted during job submission and are made available > to the task processes. > In HADOOP 1, these credentials get submitted and routed to task processes > even if security was off. > In HADOOP 2 , these credentials are transmitted only when the security is > turned on. > This should be changed for two reasons: > 1) It is not backward compatible. > 2) Credentials should be passed even if security is turned off . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4554) Job Credentials are not transmitted if security is turned off
[ https://issues.apache.org/jira/browse/MAPREDUCE-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated MAPREDUCE-4554: Attachment: MR_4554_trunk.patch Attaching patch with the following changes 1) Fixed TestStagingCleanup to fix the test failures. 2) Fixed TestMRCredentials to fix the deprecation warnings > Job Credentials are not transmitted if security is turned off > - > > Key: MAPREDUCE-4554 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4554 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Affects Versions: 2.0.0-alpha >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: trunk > > Attachments: MR_4554_trunk.patch, MR_4554_trunk.patch, > MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch, > MR_4554_trunk.patch > > > Credentials (secret keys) can be passed to a job via > mapreduce.job.credentials.json or mapreduce.job.credentials.binary . > These credentials get submitted during job submission and are made available > to the task processes. > In HADOOP 1, these credentials get submitted and routed to task processes > even if security was off. > In HADOOP 2 , these credentials are transmitted only when the security is > turned on. > This should be changed for two reasons: > 1) It is not backward compatible. > 2) Credentials should be passed even if security is turned off . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4554) Job Credentials are not transmitted if security is turned off
[ https://issues.apache.org/jira/browse/MAPREDUCE-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated MAPREDUCE-4554: Status: Open (was: Patch Available) > Job Credentials are not transmitted if security is turned off > - > > Key: MAPREDUCE-4554 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4554 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Affects Versions: 2.0.0-alpha >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: trunk > > Attachments: MR_4554_trunk.patch, MR_4554_trunk.patch, > MR_4554_trunk.patch, MR_4554_trunk.patch, MR_4554_trunk.patch > > > Credentials (secret keys) can be passed to a job via > mapreduce.job.credentials.json or mapreduce.job.credentials.binary . > These credentials get submitted during job submission and are made available > to the task processes. > In HADOOP 1, these credentials get submitted and routed to task processes > even if security was off. > In HADOOP 2 , these credentials are transmitted only when the security is > turned on. > This should be changed for two reasons: > 1) It is not backward compatible. > 2) Credentials should be passed even if security is turned off . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4642) MiniMRClientClusterFactory should not use job.setJar()
[
https://issues.apache.org/jira/browse/MAPREDUCE-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450838#comment-13450838
]
Hudson commented on MAPREDUCE-4642:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #2731 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2731/])
MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar()
(rkanter via tucu) (Revision 1382093)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
> MiniMRClientClusterFactory should not use job.setJar()
> --
>
> Key: MAPREDUCE-4642
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4642
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.2.0-alpha
>
> Attachments: MAPREDUCE-4642.patch, MAPREDUCE-4642.patch
>
>
> Currently, MiniMRClientClusterFactory does {{job.setJar(callerJar)}} so that
> the {{callerJar}} is added to the cache in MR2. However, this makes the
> resulting configuration inconsistent between MR1 and MR2 as in MR1 the job
> jar is not set and in MR2 its set to the {{callerJar}}. This difference can
> also cause some tests to fail in Oozie. We should instead use the
> {{job.addCacheFile()}} method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4642) MiniMRClientClusterFactory should not use job.setJar()
[
https://issues.apache.org/jira/browse/MAPREDUCE-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450798#comment-13450798
]
Hudson commented on MAPREDUCE-4642:
---
Integrated in Hadoop-Common-trunk-Commit #2707 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2707/])
MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar()
(rkanter via tucu) (Revision 1382093)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
> MiniMRClientClusterFactory should not use job.setJar()
> --
>
> Key: MAPREDUCE-4642
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4642
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.2.0-alpha
>
> Attachments: MAPREDUCE-4642.patch, MAPREDUCE-4642.patch
>
>
> Currently, MiniMRClientClusterFactory does {{job.setJar(callerJar)}} so that
> the {{callerJar}} is added to the cache in MR2. However, this makes the
> resulting configuration inconsistent between MR1 and MR2 as in MR1 the job
> jar is not set and in MR2 its set to the {{callerJar}}. This difference can
> also cause some tests to fail in Oozie. We should instead use the
> {{job.addCacheFile()}} method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4642) MiniMRClientClusterFactory should not use job.setJar()
[
https://issues.apache.org/jira/browse/MAPREDUCE-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450799#comment-13450799
]
Hudson commented on MAPREDUCE-4642:
---
Integrated in Hadoop-Hdfs-trunk-Commit #2770 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2770/])
MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar()
(rkanter via tucu) (Revision 1382093)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382093
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
> MiniMRClientClusterFactory should not use job.setJar()
> --
>
> Key: MAPREDUCE-4642
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4642
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.2.0-alpha
>
> Attachments: MAPREDUCE-4642.patch, MAPREDUCE-4642.patch
>
>
> Currently, MiniMRClientClusterFactory does {{job.setJar(callerJar)}} so that
> the {{callerJar}} is added to the cache in MR2. However, this makes the
> resulting configuration inconsistent between MR1 and MR2 as in MR1 the job
> jar is not set and in MR2 its set to the {{callerJar}}. This difference can
> also cause some tests to fail in Oozie. We should instead use the
> {{job.addCacheFile()}} method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4642) MiniMRClientClusterFactory should not use job.setJar()
[
https://issues.apache.org/jira/browse/MAPREDUCE-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-4642:
--
Resolution: Fixed
Fix Version/s: 2.2.0-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Robert. Committed to trunk and branch-2.
> MiniMRClientClusterFactory should not use job.setJar()
> --
>
> Key: MAPREDUCE-4642
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4642
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.2.0-alpha
>
> Attachments: MAPREDUCE-4642.patch, MAPREDUCE-4642.patch
>
>
> Currently, MiniMRClientClusterFactory does {{job.setJar(callerJar)}} so that
> the {{callerJar}} is added to the cache in MR2. However, this makes the
> resulting configuration inconsistent between MR1 and MR2 as in MR1 the job
> jar is not set and in MR2 its set to the {{callerJar}}. This difference can
> also cause some tests to fail in Oozie. We should instead use the
> {{job.addCacheFile()}} method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4642) MiniMRClientClusterFactory should not use job.setJar()
[
https://issues.apache.org/jira/browse/MAPREDUCE-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450792#comment-13450792
]
Alejandro Abdelnur commented on MAPREDUCE-4642:
---
+1
> MiniMRClientClusterFactory should not use job.setJar()
> --
>
> Key: MAPREDUCE-4642
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4642
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-4642.patch, MAPREDUCE-4642.patch
>
>
> Currently, MiniMRClientClusterFactory does {{job.setJar(callerJar)}} so that
> the {{callerJar}} is added to the cache in MR2. However, this makes the
> resulting configuration inconsistent between MR1 and MR2 as in MR1 the job
> jar is not set and in MR2 its set to the {{callerJar}}. This difference can
> also cause some tests to fail in Oozie. We should instead use the
> {{job.addCacheFile()}} method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4629) Remove JobHistory.DEBUG_MODE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450787#comment-13450787 ] Hudson commented on MAPREDUCE-4629: --- Integrated in Hadoop-Hdfs-trunk-Commit #2769 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2769/]) MAPREDUCE-4629. Remove JobHistory.DEBUG_MODE (Karthik Kambatla via bobby) (Revision 1382088) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382088 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java > Remove JobHistory.DEBUG_MODE > > > Key: MAPREDUCE-4629 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4629 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 1.2.0, 2.2.0-alpha > > Attachments: branch1-mr-4629.patch, MR-4629.patch, MR-4629.patch.v1, > trunk-mr-6429.patch, trunk-mr-6429.patch > > > Remove JobHistory.DEBUG_MODE for the following reasons: > 1. No one seems to be using it - the config parameter corresponding to > enabling it does not even exist in mapred-default.xml > 2. The logging being done in DEBUG_MODE needs to move to LOG.debug() and > LOG.trace() > 3. Buggy handling of helper methods in DEBUG_MODE; e.g. directoryTime() and > timestampDirectoryComponent(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4629) Remove JobHistory.DEBUG_MODE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450786#comment-13450786 ] Hudson commented on MAPREDUCE-4629: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2730 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2730/]) MAPREDUCE-4629. Remove JobHistory.DEBUG_MODE (Karthik Kambatla via bobby) (Revision 1382088) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382088 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java > Remove JobHistory.DEBUG_MODE > > > Key: MAPREDUCE-4629 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4629 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 1.2.0, 2.2.0-alpha > > Attachments: branch1-mr-4629.patch, MR-4629.patch, MR-4629.patch.v1, > trunk-mr-6429.patch, trunk-mr-6429.patch > > > Remove JobHistory.DEBUG_MODE for the following reasons: > 1. No one seems to be using it - the config parameter corresponding to > enabling it does not even exist in mapred-default.xml > 2. The logging being done in DEBUG_MODE needs to move to LOG.debug() and > LOG.trace() > 3. Buggy handling of helper methods in DEBUG_MODE; e.g. directoryTime() and > timestampDirectoryComponent(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4629) Remove JobHistory.DEBUG_MODE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450783#comment-13450783 ] Karthik Kambatla commented on MAPREDUCE-4629: - Thanks a lot, Bobby. > Remove JobHistory.DEBUG_MODE > > > Key: MAPREDUCE-4629 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4629 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 1.2.0, 2.2.0-alpha > > Attachments: branch1-mr-4629.patch, MR-4629.patch, MR-4629.patch.v1, > trunk-mr-6429.patch, trunk-mr-6429.patch > > > Remove JobHistory.DEBUG_MODE for the following reasons: > 1. No one seems to be using it - the config parameter corresponding to > enabling it does not even exist in mapred-default.xml > 2. The logging being done in DEBUG_MODE needs to move to LOG.debug() and > LOG.trace() > 3. Buggy handling of helper methods in DEBUG_MODE; e.g. directoryTime() and > timestampDirectoryComponent(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4629) Remove JobHistory.DEBUG_MODE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450785#comment-13450785 ] Hudson commented on MAPREDUCE-4629: --- Integrated in Hadoop-Common-trunk-Commit #2706 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2706/]) MAPREDUCE-4629. Remove JobHistory.DEBUG_MODE (Karthik Kambatla via bobby) (Revision 1382088) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382088 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java > Remove JobHistory.DEBUG_MODE > > > Key: MAPREDUCE-4629 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4629 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 1.2.0, 2.2.0-alpha > > Attachments: branch1-mr-4629.patch, MR-4629.patch, MR-4629.patch.v1, > trunk-mr-6429.patch, trunk-mr-6429.patch > > > Remove JobHistory.DEBUG_MODE for the following reasons: > 1. No one seems to be using it - the config parameter corresponding to > enabling it does not even exist in mapred-default.xml > 2. The logging being done in DEBUG_MODE needs to move to LOG.debug() and > LOG.trace() > 3. Buggy handling of helper methods in DEBUG_MODE; e.g. directoryTime() and > timestampDirectoryComponent(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4629) Remove JobHistory.DEBUG_MODE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4629: --- Resolution: Fixed Fix Version/s: 2.2.0-alpha 1.2.0 Status: Resolved (was: Patch Available) Thanks Karthik, I checked this into trunk, branch-2, and branch-1 > Remove JobHistory.DEBUG_MODE > > > Key: MAPREDUCE-4629 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4629 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 1.2.0, 2.2.0-alpha > > Attachments: branch1-mr-4629.patch, MR-4629.patch, MR-4629.patch.v1, > trunk-mr-6429.patch, trunk-mr-6429.patch > > > Remove JobHistory.DEBUG_MODE for the following reasons: > 1. No one seems to be using it - the config parameter corresponding to > enabling it does not even exist in mapred-default.xml > 2. The logging being done in DEBUG_MODE needs to move to LOG.debug() and > LOG.trace() > 3. Buggy handling of helper methods in DEBUG_MODE; e.g. directoryTime() and > timestampDirectoryComponent(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4629) Remove JobHistory.DEBUG_MODE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450775#comment-13450775 ] Robert Joseph Evans commented on MAPREDUCE-4629: Jenkis came back OK and the 1.0 tests for me passed. I'll check this in. Thanks for the patch this is something I have wanted to see happen for a while. > Remove JobHistory.DEBUG_MODE > > > Key: MAPREDUCE-4629 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4629 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: branch1-mr-4629.patch, MR-4629.patch, MR-4629.patch.v1, > trunk-mr-6429.patch, trunk-mr-6429.patch > > > Remove JobHistory.DEBUG_MODE for the following reasons: > 1. No one seems to be using it - the config parameter corresponding to > enabling it does not even exist in mapred-default.xml > 2. The logging being done in DEBUG_MODE needs to move to LOG.debug() and > LOG.trace() > 3. Buggy handling of helper methods in DEBUG_MODE; e.g. directoryTime() and > timestampDirectoryComponent(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4629) Remove JobHistory.DEBUG_MODE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450765#comment-13450765 ] Hadoop QA commented on MAPREDUCE-4629: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544233/trunk-mr-6429.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2831//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2831//console This message is automatically generated. > Remove JobHistory.DEBUG_MODE > > > Key: MAPREDUCE-4629 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4629 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: branch1-mr-4629.patch, MR-4629.patch, MR-4629.patch.v1, > trunk-mr-6429.patch, trunk-mr-6429.patch > > > Remove JobHistory.DEBUG_MODE for the following reasons: > 1. No one seems to be using it - the config parameter corresponding to > enabling it does not even exist in mapred-default.xml > 2. The logging being done in DEBUG_MODE needs to move to LOG.debug() and > LOG.trace() > 3. Buggy handling of helper methods in DEBUG_MODE; e.g. directoryTime() and > timestampDirectoryComponent(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack
[ https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450752#comment-13450752 ] Tsuyoshi OZAWA commented on MAPREDUCE-4502: --- After debating with Siddharsh, I decided to implement combining against current trunk branch. I'll attach the architecture overview and design note as soon as possible(maybe day after tomorrow). > Multi-level aggregation with combining the result of maps per node/rack > --- > > Key: MAPREDUCE-4502 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Tsuyoshi OZAWA >Assignee: Tsuyoshi OZAWA > > The shuffle costs is expensive in Hadoop in spite of the existence of > combiner, because the scope of combining is limited within only one MapTask. > To solve this problem, it's a good way to aggregate the result of maps per > node/rack by launch combiner. > This JIRA is to implement the multi-level aggregation infrastructure, > including combining per container(MAPREDUCE-3902 is related), coordinating > containers by application master without breaking fault tolerance of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4629) Remove JobHistory.DEBUG_MODE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4629: --- Attachment: trunk-mr-6429.patch Uploading the trunk patch again, unchanged to Jenkins can run the precommit build on it. The changes look good to me. Assuming Jenkins comes back OK I am a +1 on the changes for trunk. Assuming my running the unit tests on branhc-1 come back OK I am +1 for that as well. > Remove JobHistory.DEBUG_MODE > > > Key: MAPREDUCE-4629 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4629 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: branch1-mr-4629.patch, MR-4629.patch, MR-4629.patch.v1, > trunk-mr-6429.patch, trunk-mr-6429.patch > > > Remove JobHistory.DEBUG_MODE for the following reasons: > 1. No one seems to be using it - the config parameter corresponding to > enabling it does not even exist in mapred-default.xml > 2. The logging being done in DEBUG_MODE needs to move to LOG.debug() and > LOG.trace() > 3. Buggy handling of helper methods in DEBUG_MODE; e.g. directoryTime() and > timestampDirectoryComponent(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4644) mapreduce-client-jobclient-tests do not run from dist tarball
[
https://issues.apache.org/jira/browse/MAPREDUCE-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450717#comment-13450717
]
Jason Lowe commented on MAPREDUCE-4644:
---
Error traceback when trying to launch the tests jar:
{noformat}
$ hadoop jar
hadoop-*-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-SNAPSHOT-tests.jar
java.lang.NoClassDefFoundError: junit/framework/TestCase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at
org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:60)
at
org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:54)
at
org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 20 more
An example program must be given as the first argument.
Valid program names are:
{noformat}
> mapreduce-client-jobclient-tests do not run from dist tarball
> -
>
> Key: MAPREDUCE-4644
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4644
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
>Affects Versions: 2.2.0-alpha
>Reporter: Jason Lowe
>Priority: Blocker
>
> The mapreduce jobclient tests rely on junit which is missing from the dist
> tarball. This prevents running often-used tests like sleep jobs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4644) mapreduce-client-jobclient-tests do not run from dist tarball
Jason Lowe created MAPREDUCE-4644: - Summary: mapreduce-client-jobclient-tests do not run from dist tarball Key: MAPREDUCE-4644 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4644 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 2.2.0-alpha Reporter: Jason Lowe Priority: Blocker The mapreduce jobclient tests rely on junit which is missing from the dist tarball. This prevents running often-used tests like sleep jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4641) Exception in commitJob marks job as successful in job history
[
https://issues.apache.org/jira/browse/MAPREDUCE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450708#comment-13450708
]
Hudson commented on MAPREDUCE-4641:
---
Integrated in Hadoop-Hdfs-0.23-Build #367 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/367/])
svn merge -c 1381784 FIXES: MAPREDUCE-4641. Exception in commitJob marks
job as successful in job history (Jason Lowe via bobby) (Revision 1381787)
Result = UNSTABLE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381787
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
> Exception in commitJob marks job as successful in job history
> -
>
> Key: MAPREDUCE-4641
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4641
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 2.0.0-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MAPREDUCE-4641.patch
>
>
> If the job committer throws an {{IOException}} from {{commitJob}} then the
> job will be marked as FINISHED/FAILED on the RM apps page, but the history
> server will show the job as SUCCEEDED.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4633) history server doesn't set permissions on all subdirs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450707#comment-13450707 ] Hudson commented on MAPREDUCE-4633: --- Integrated in Hadoop-Hdfs-0.23-Build #367 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/367/]) svn merge -c 1381614 FIXES: MAPREDUCE-4633. history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381619) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381619 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java > history server doesn't set permissions on all subdirs > -- > > Key: MAPREDUCE-4633 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4633 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-4633.patch > > > The job history server creates a bunch of subdirectories under the "done" > directory. They are like 2012/09/03/00. It only sets the permissions on > the last one, ie 00 to 770.So the 2012/09/03 aren't explicitly set so > if the umask is more restrictive, they won't be set as it expects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4641) Exception in commitJob marks job as successful in job history
[
https://issues.apache.org/jira/browse/MAPREDUCE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450635#comment-13450635
]
Hudson commented on MAPREDUCE-4641:
---
Integrated in Hadoop-Mapreduce-trunk #1189 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1189/])
MAPREDUCE-4641. Exception in commitJob marks job as successful in job
history (Jason Lowe via bobby) (Revision 1381784)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381784
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
> Exception in commitJob marks job as successful in job history
> -
>
> Key: MAPREDUCE-4641
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4641
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 2.0.0-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MAPREDUCE-4641.patch
>
>
> If the job committer throws an {{IOException}} from {{commitJob}} then the
> job will be marked as FINISHED/FAILED on the RM apps page, but the history
> server will show the job as SUCCEEDED.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4638) MR AppMaster shouldn't rely on YARN_APPLICATION_CLASSPATH providing MR jars
[ https://issues.apache.org/jira/browse/MAPREDUCE-4638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450637#comment-13450637 ] Hudson commented on MAPREDUCE-4638: --- Integrated in Hadoop-Mapreduce-trunk #1189 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1189/]) MAPREDUCE-4638. MR AM supplies MapReduce jars in classpath rather than rely on YARN. Contributed by Arun C Murthy. (Revision 1381868) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381868 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml > MR AppMaster shouldn't rely on YARN_APPLICATION_CLASSPATH providing MR jars > --- > > Key: MAPREDUCE-4638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4638 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 2.1.0-alpha > > Attachments: MAPREDUCE-4638.patch > > > Currently YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH provides > $YARN_HOME/share/hadoop/mapreduce/* & > $YARN_HOME/share/hadoop/mapreduce/lib/*. It should not depend on this post > YARN-86. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4633) history server doesn't set permissions on all subdirs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450634#comment-13450634 ] Hudson commented on MAPREDUCE-4633: --- Integrated in Hadoop-Mapreduce-trunk #1189 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1189/]) MAPREDUCE-4633. history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java > history server doesn't set permissions on all subdirs > -- > > Key: MAPREDUCE-4633 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4633 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-4633.patch > > > The job history server creates a bunch of subdirectories under the "done" > directory. They are like 2012/09/03/00. It only sets the permissions on > the last one, ie 00 to 770.So the 2012/09/03 aren't explicitly set so > if the umask is more restrictive, they won't be set as it expects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4638) MR AppMaster shouldn't rely on YARN_APPLICATION_CLASSPATH providing MR jars
[ https://issues.apache.org/jira/browse/MAPREDUCE-4638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450591#comment-13450591 ] Hudson commented on MAPREDUCE-4638: --- Integrated in Hadoop-Hdfs-trunk #1158 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1158/]) MAPREDUCE-4638. MR AM supplies MapReduce jars in classpath rather than rely on YARN. Contributed by Arun C Murthy. (Revision 1381868) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381868 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml > MR AppMaster shouldn't rely on YARN_APPLICATION_CLASSPATH providing MR jars > --- > > Key: MAPREDUCE-4638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4638 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 2.1.0-alpha > > Attachments: MAPREDUCE-4638.patch > > > Currently YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH provides > $YARN_HOME/share/hadoop/mapreduce/* & > $YARN_HOME/share/hadoop/mapreduce/lib/*. It should not depend on this post > YARN-86. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4633) history server doesn't set permissions on all subdirs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450588#comment-13450588 ] Hudson commented on MAPREDUCE-4633: --- Integrated in Hadoop-Hdfs-trunk #1158 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1158/]) MAPREDUCE-4633. history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java > history server doesn't set permissions on all subdirs > -- > > Key: MAPREDUCE-4633 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4633 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-4633.patch > > > The job history server creates a bunch of subdirectories under the "done" > directory. They are like 2012/09/03/00. It only sets the permissions on > the last one, ie 00 to 770.So the 2012/09/03 aren't explicitly set so > if the umask is more restrictive, they won't be set as it expects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4641) Exception in commitJob marks job as successful in job history
[
https://issues.apache.org/jira/browse/MAPREDUCE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450589#comment-13450589
]
Hudson commented on MAPREDUCE-4641:
---
Integrated in Hadoop-Hdfs-trunk #1158 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1158/])
MAPREDUCE-4641. Exception in commitJob marks job as successful in job
history (Jason Lowe via bobby) (Revision 1381784)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381784
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
> Exception in commitJob marks job as successful in job history
> -
>
> Key: MAPREDUCE-4641
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4641
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 2.0.0-alpha
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MAPREDUCE-4641.patch
>
>
> If the job committer throws an {{IOException}} from {{commitJob}} then the
> job will be marked as FINISHED/FAILED on the RM apps page, but the history
> server will show the job as SUCCEEDED.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4629) Remove JobHistory.DEBUG_MODE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450432#comment-13450432 ] Hadoop QA commented on MAPREDUCE-4629: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544175/branch1-mr-4629.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2830//console This message is automatically generated. > Remove JobHistory.DEBUG_MODE > > > Key: MAPREDUCE-4629 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4629 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: branch1-mr-4629.patch, MR-4629.patch, MR-4629.patch.v1, > trunk-mr-6429.patch > > > Remove JobHistory.DEBUG_MODE for the following reasons: > 1. No one seems to be using it - the config parameter corresponding to > enabling it does not even exist in mapred-default.xml > 2. The logging being done in DEBUG_MODE needs to move to LOG.debug() and > LOG.trace() > 3. Buggy handling of helper methods in DEBUG_MODE; e.g. directoryTime() and > timestampDirectoryComponent(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
