[jira] [Updated] (MAPREDUCE-2129) Job may hang if mapreduce.job.committer.setup.cleanup.needed=false and mapreduce.map/reduce.failures.maxpercent>0
[ https://issues.apache.org/jira/browse/MAPREDUCE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-2129: -- Target Version/s: 1.1.0 (was: 1.1.0, 1.2.0) Fix Version/s: (was: 1.2.0) 1.1.0 Committed to branch-1.1 for 1.1.0. Fixed location in branch-1/CHANGES.txt. I sure hope they put back the SVN/JIRA integration plugin soon! > Job may hang if mapreduce.job.committer.setup.cleanup.needed=false and > mapreduce.map/reduce.failures.maxpercent>0 > - > > Key: MAPREDUCE-2129 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2129 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.2, 0.20.3, 0.21.1 >Reporter: Kang Xiao >Assignee: Subroto Sanyal > Labels: hadoop > Fix For: 1.1.0 > > Attachments: MAPREDUCE-2129.patch, MAPREDUCE-2129.patch, > MAPREDUCE-2129.patch, MAPREDUCE-2129.patch > > > Job may hang at RUNNING state if > mapreduce.job.committer.setup.cleanup.needed=false and > mapreduce.map/reduce.failures.maxpercent>0. It happens when some tasks fail > but havent reached failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2376) test-task-controller fails if run as a userid < 1000
[ https://issues.apache.org/jira/browse/MAPREDUCE-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-2376: -- Target Version/s: 0.23.0, 1.1.0, 0.22.1 (was: 0.22.0, 0.23.0, 1.1.0) Fix Version/s: 1.1.0 Made Target Versions and Fix Versions consistent. Leaving bug open because the fix is still missing from Hadoop-2 line(s) and trunk. > test-task-controller fails if run as a userid < 1000 > > > Key: MAPREDUCE-2376 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2376 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller, test >Affects Versions: 0.20.205.0, 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 1.1.0, 0.22.1 > > Attachments: mapreduce-2376-20.txt, mr-2376-0.22.patch > > > test-task-controller tries to verify that the task-controller won't run on > behalf of users with uid < 1000. This makes the test fail when running in > some test environments - eg our hudson jobs internally run as a system user > with uid 101. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3993) Graceful handling of codec errors during decompression
[ https://issues.apache.org/jira/browse/MAPREDUCE-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-3993: -- Target Version/s: 1.2.0, 2.1.0-alpha (was: 1.1.0, 2.0.0-alpha) corrected Target Versions to match Fix Versions. > Graceful handling of codec errors during decompression > -- > > Key: MAPREDUCE-3993 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3993 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 0.23.1, 1.0.2 >Reporter: Todd Lipcon >Assignee: Karthik Kambatla > Fix For: 1.2.0, 2.1.0-alpha > > Attachments: MR-3993_branch1.patch, MR-3993_branch1.patch, > MR-3993_trunk.patch, MR-3993_trunk.patch > > > When using a compression codec for intermediate compression, some cases of > corrupt data can cause the codec to throw exceptions other than IOException > (eg java.lang.InternalError). This will currently cause the whole reduce task > to fail, instead of simply treating it like another case of a failed fetch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420452#comment-13420452 ] Hadoop QA commented on MAPREDUCE-2454: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537540/mapreduce-2454.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2647//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2647//console This message is automatically generated. > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, KeyValueIterator.java, > MR-2454-trunkPatchPreview.gz, MapOutputSorter.java, > MapOutputSorterAbstract.java, ReduceInputSorter.java, mapreduce-2454.patch, > mr-2454-on-mr-279-build82.patch.gz > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2454) Allow external sorter plugin for MR
[
https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420409#comment-13420409
]
Mariappan Asokan commented on MAPREDUCE-2454:
-
I think I found the reason why TestReduceFetch test was failing. The line:
{code}
commitMemory -= size;
{code}
in the method unreserve() in MergeManager.java was missing in my patch. I have
corrected this problem and resubmitted the patch.
> Allow external sorter plugin for MR
> ---
>
> Key: MAPREDUCE-2454
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Mariappan Asokan
>Priority: Minor
> Labels: features, performance, plugin, sort
> Attachments: HadoopSortPlugin.pdf, KeyValueIterator.java,
> MR-2454-trunkPatchPreview.gz, MapOutputSorter.java,
> MapOutputSorterAbstract.java, ReduceInputSorter.java, mapreduce-2454.patch,
> mr-2454-on-mr-279-build82.patch.gz
>
>
> Define interfaces and some abstract classes in the Hadoop framework to
> facilitate external sorter plugins both on the Map and Reduce sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariappan Asokan updated MAPREDUCE-2454: Attachment: mapreduce-2454.patch > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, KeyValueIterator.java, > MR-2454-trunkPatchPreview.gz, MapOutputSorter.java, > MapOutputSorterAbstract.java, ReduceInputSorter.java, mapreduce-2454.patch, > mr-2454-on-mr-279-build82.patch.gz > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariappan Asokan updated MAPREDUCE-2454: Status: Patch Available (was: Open) > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, KeyValueIterator.java, > MR-2454-trunkPatchPreview.gz, MapOutputSorter.java, > MapOutputSorterAbstract.java, ReduceInputSorter.java, mapreduce-2454.patch, > mr-2454-on-mr-279-build82.patch.gz > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mariappan Asokan updated MAPREDUCE-2454: Attachment: (was: mapreduce-2454.patch) > Allow external sorter plugin for MR > --- > > Key: MAPREDUCE-2454 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Mariappan Asokan >Priority: Minor > Labels: features, performance, plugin, sort > Attachments: HadoopSortPlugin.pdf, KeyValueIterator.java, > MR-2454-trunkPatchPreview.gz, MapOutputSorter.java, > MapOutputSorterAbstract.java, ReduceInputSorter.java, > mr-2454-on-mr-279-build82.patch.gz > > > Define interfaces and some abstract classes in the Hadoop framework to > facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-338) Need more complete API of JobClient class
[ https://issues.apache.org/jira/browse/MAPREDUCE-338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jian fan updated MAPREDUCE-338: --- Description: We need a programmatic way to find out the information about a map/reduce cluster and the jobs on the cluster. The current API is not complete. In particular, the following API functions are needed: 1. jobs() currently, there is an API function JobsToComplete, which returns running/waiting jobs only. jobs() should return the complete list. 2. TaskReport[] getMap/ReduceTaskReports(String jobid) 3. getStartTime() 4. getJobStatus(String jobid); 5. getJobProfile(String jobid); was: We need a programmatic way to find out the information about a map/reduce cluster and the jobs on the cluster. The current API is not complete. In particular, the following API functions are needed: 1. jobs() currently, there is an API function JobsToComplete, which returns running/waiting jobs only. jobs() should return the complete list. 2. TaskReport[] getMap/ReduceTaskReports(String jobid) 3. getStartTime() 4. getJobStatus(String jobid); 5. getJobProfile(String jobid); > Need more complete API of JobClient class > - > > Key: MAPREDUCE-338 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-338 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Runping Qi > > We need a programmatic way to find out the information about a map/reduce > cluster and the jobs on the cluster. > The current API is not complete. > In particular, the following API functions are needed: > 1. jobs() currently, there is an API function JobsToComplete, which returns > running/waiting jobs only. jobs() should return the complete list. > 2. TaskReport[] getMap/ReduceTaskReports(String jobid) > 3. getStartTime() > 4. getJobStatus(String jobid); > 5. getJobProfile(String jobid); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420390#comment-13420390
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537535/MAPREDUCE-987_branch-1.0_rev2.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2646//console
This message is automatically generated.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Affects Versions: 1.2.0, 2.2.0-alpha
>Reporter: Philip Zeyliger
>Assignee: Ahmed Radwan
>Priority: Minor
> Fix For: 2.2.0-alpha
>
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_branch-1.0_rev2.patch, MAPREDUCE-987_rev2.patch,
> MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Radwan updated MAPREDUCE-987:
---
Attachment: MAPREDUCE-987_branch-1.0_rev2.patch
Thanks Tucu! Here is the updated patch for branch-1.0 including the new docs.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Affects Versions: 1.2.0, 2.2.0-alpha
>Reporter: Philip Zeyliger
>Assignee: Ahmed Radwan
>Priority: Minor
> Fix For: 2.2.0-alpha
>
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_branch-1.0_rev2.patch, MAPREDUCE-987_rev2.patch,
> MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420366#comment-13420366 ] Hari Mankude commented on MAPREDUCE-4334: - Relevant information would be the performance impact of running maps and reduces in cgroups in terms of latency. Overall, this would be a very useful feature since it is possible to add fencing around cpu/io resources in addition to memory usage for MR tasks. > Add support for CPU isolation/monitoring of containers > -- > > Key: MAPREDUCE-4334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Arun C Murthy >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4334-pre1.patch, > MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, > MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch, > MAPREDUCE-4334-v1.patch, MAPREDUCE-4334-v2.patch > > > Once we get in MAPREDUCE-4327, it will be important to actually enforce > limits on CPU consumption of containers. > Several options spring to mind: > # taskset (RHEL5+) > # cgroups (RHEL6+) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420363#comment-13420363 ] Bikas Saha commented on MAPREDUCE-4334: --- Aside from a design proposal I would be really interested in seeing how exactly cgroups work in the context of our typical workload. Say, take a bunch of typical mappers and reducers and run them in isolation. Then run them in isolation within cgroups. Is there a difference? Now run them concurrently with and without cgroups. What are the observations? These experiments may lead to expected or unexpected results and would be a great addition to the design pros and cons. Perhaps you have already run those experiments. If yes, care sharing the results. > Add support for CPU isolation/monitoring of containers > -- > > Key: MAPREDUCE-4334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Arun C Murthy >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4334-pre1.patch, > MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, > MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch, > MAPREDUCE-4334-v1.patch, MAPREDUCE-4334-v2.patch > > > Once we get in MAPREDUCE-4327, it will be important to actually enforce > limits on CPU consumption of containers. > Several options spring to mind: > # taskset (RHEL5+) > # cgroups (RHEL6+) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420347#comment-13420347 ] Andrew Ferguson commented on MAPREDUCE-4334: Hi Arun, I feel like we've been discussing pros & cons for the length of this JIRA. :-) I think, perhaps, I proposed too large of a change across this issue and MAPREDUCE-4351: cgroups for cpu, cgroups for memory, a code refactoring, etc. Instead, I would like to make a smaller change, with just cgroups for CPUs and place them in each launcher's code, as you requested above. Perhaps a better re-factoring than I suggested with the ContainersMonitor will become clear afterwards. How does this sound to you? I was planning to finish it up on Monday. best, Andrew > Add support for CPU isolation/monitoring of containers > -- > > Key: MAPREDUCE-4334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Arun C Murthy >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4334-pre1.patch, > MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, > MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch, > MAPREDUCE-4334-v1.patch, MAPREDUCE-4334-v2.patch > > > Once we get in MAPREDUCE-4327, it will be important to actually enforce > limits on CPU consumption of containers. > Several options spring to mind: > # taskset (RHEL5+) > # cgroups (RHEL6+) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4417) add support for encrypted shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420245#comment-13420245 ] Hadoop QA commented on MAPREDUCE-4417: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537503/MAPREDUCE-4417.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. -1 javac. The applied patch generated 2049 javac compiler warnings (more than the trunk's current 2007 warnings). +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2645//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2645//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2645//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2645//console This message is automatically generated. > add support for encrypted shuffle > - > > Key: MAPREDUCE-4417 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4417 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, security >Affects Versions: 2.0.0-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.1.0-alpha > > Attachments: MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch > > > Currently Shuffle fetches go on the clear. While Kerberos provides > comprehensive authentication for the cluster, it does not provide > confidentiality. > When processing sensitive data confidentiality may be desired (at the expense > of job performance and resources utilization for doing encryption). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4417) add support for encrypted shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-4417: -- Attachment: MAPREDUCE-4417.patch rebasing patch to trunk and resolving conflict in index documentation page. Also, simplified some initialization logic in the testcases. > add support for encrypted shuffle > - > > Key: MAPREDUCE-4417 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4417 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, security >Affects Versions: 2.0.0-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.1.0-alpha > > Attachments: MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, MAPREDUCE-4417.patch, > MAPREDUCE-4417.patch, MAPREDUCE-4417.patch > > > Currently Shuffle fetches go on the clear. While Kerberos provides > comprehensive authentication for the cluster, it does not provide > confidentiality. > When processing sensitive data confidentiality may be desired (at the expense > of job performance and resources utilization for doing encryption). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4452) Several Findbugs issues with new FairScheduler in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-4452: -- Attachment: (was: MAPREDUCE-4452-v3.patch) > Several Findbugs issues with new FairScheduler in YARN > -- > > Key: MAPREDUCE-4452 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 2.1.0-alpha >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, > MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch > > > The good feature of FairScheduler is added recently to YARN. As recently > PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs > related to FairScheduler: > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() > might ignore java.lang.Exception > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; > locked 50% of time > The details are > in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with new FairScheduler in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420136#comment-13420136 ] Hadoop QA commented on MAPREDUCE-4452: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537485/MAPREDUCE-4452-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2644//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2644//console This message is automatically generated. > Several Findbugs issues with new FairScheduler in YARN > -- > > Key: MAPREDUCE-4452 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 2.1.0-alpha >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, > MAPREDUCE-4452-v3.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch > > > The good feature of FairScheduler is added recently to YARN. As recently > PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs > related to FairScheduler: > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() > might ignore java.lang.Exception > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; > locked 50% of time > The details are > in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with new FairScheduler in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420133#comment-13420133 ] Junping Du commented on MAPREDUCE-4452: --- In new v3 patch, separating monitor objects from attributes to get rid of ML_SYNC_ON_FIELD_TO_GUARD_CHANGING_THAT_FIELD which is due to updating on lock object. Also, add synchronization to getter/setter of userMaxAppsDefault and queueMaxAppsDefault to get ride of FindBugs warning.(in fact, it is not necessary because this is int type and should be finished in one CPU instruction. However, it is not much overhead as lock level is fine-grained). > Several Findbugs issues with new FairScheduler in YARN > -- > > Key: MAPREDUCE-4452 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 2.1.0-alpha >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, > MAPREDUCE-4452-v3.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch > > > The good feature of FairScheduler is added recently to YARN. As recently > PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs > related to FairScheduler: > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() > might ignore java.lang.Exception > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; > locked 50% of time > The details are > in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4452) Several Findbugs issues with new FairScheduler in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-4452: -- Attachment: MAPREDUCE-4452-v3.patch > Several Findbugs issues with new FairScheduler in YARN > -- > > Key: MAPREDUCE-4452 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 2.1.0-alpha >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, > MAPREDUCE-4452-v3.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch > > > The good feature of FairScheduler is added recently to YARN. As recently > PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs > related to FairScheduler: > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() > might ignore java.lang.Exception > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; > locked 50% of time > The details are > in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with new FairScheduler in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420120#comment-13420120 ] Junping Du commented on MAPREDUCE-4452: --- Patrick, you are right that the threads of modifying and reading are pending on different lock object/attribute in my previous patch, so I update to v3 patch so both of them are synchronized on related attributes. The intention to get rid of synchronized getter methods (as v1 patch) is that will make QueueManager instance can only be accessed by one thread per time although different threads are reading/modifying different attributes. I guess that is the right way to do it. What do you think? > Several Findbugs issues with new FairScheduler in YARN > -- > > Key: MAPREDUCE-4452 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 2.1.0-alpha >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, > MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch > > > The good feature of FairScheduler is added recently to YARN. As recently > PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs > related to FairScheduler: > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() > might ignore java.lang.Exception > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; > locked 50% of time > The details are > in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with new FairScheduler in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420119#comment-13420119 ] Hadoop QA commented on MAPREDUCE-4452: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537479/MAPREDUCE-4452-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2643//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2643//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2643//console This message is automatically generated. > Several Findbugs issues with new FairScheduler in YARN > -- > > Key: MAPREDUCE-4452 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 2.1.0-alpha >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, > MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch > > > The good feature of FairScheduler is added recently to YARN. As recently > PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs > related to FairScheduler: > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() > might ignore java.lang.Exception > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; > locked 50% of time > The details are > in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4452) Several Findbugs issues with new FairScheduler in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-4452: -- Attachment: MAPREDUCE-4452-v3.patch > Several Findbugs issues with new FairScheduler in YARN > -- > > Key: MAPREDUCE-4452 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 2.1.0-alpha >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, > MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch > > > The good feature of FairScheduler is added recently to YARN. As recently > PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs > related to FairScheduler: > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() > might ignore java.lang.Exception > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; > locked 50% of time > Inconsistent synchronization of > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; > locked 50% of time > The details are > in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
