[jira] [Commented] (MAPREDUCE-4424) The 'mapred job -list' command should show the job name as well
[
https://issues.apache.org/jira/browse/MAPREDUCE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478735#comment-13478735
]
Harsh J commented on MAPREDUCE-4424:
Hi Xiaoding,
Many thanks for the patch!
I personally feel that having Job Name near Job ID makes more sense than at the
end (which mostly wraps around on regular terminal sizes with MRv2 today).
We could cap the maximum characters we show, to do this without worrying about
the variable size issue, am fine with doing that.
Thoughts?
P.s. Please also update the test case in TestMRJobClient.java (jobclient
module).
The 'mapred job -list' command should show the job name as well
---
Key: MAPREDUCE-4424
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4424
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: trunk
Reporter: Harsh J
Priority: Trivial
Labels: newbie
Attachments: MAPREDUCE-4424.patch
Currently the {{mapred job -list}} command does not show the Job Name, just
the Job ID. It would be good to display the Job name too. Idea originally
from HADOOP-.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2446) HCE 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-2446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478758#comment-13478758 ] chao li commented on MAPREDUCE-2446: /usr/include/bits/xopen_lim.h:95:6: 错误：标识符“(”前缺少二元运算符 /usr/include/bits/xopen_lim.h:98:7: 错误：标识符“(”前缺少二元运算符 /usr/include/bits/xopen_lim.h:122:6: 错误：标识符“(”前缺少二元运算符 make[2]: *** [CompressionFactory.o] 错误 1 make[2]: Leaving directory `/root/hadoop_hce_v1/hadoop-0.20.3/src/c++/hce/impl/Compress' make[1]: *** [all-recursive] 错误 1 make[1]: Leaving directory `/root/hadoop_hce_v1/hadoop-0.20.3/src/c++/hce/impl' How can I confirm this error? HCE 2.0 --- Key: MAPREDUCE-2446 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2446 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/streaming, pipes, task Reporter: Dong Yang Enhancing MapReduce by Task-level Optimization. Except for yielding speedups of up to 130% on original Streaming Program, Hce 2.0 provides some more flexible programming interfaces including c++, java, python, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2264) Job status exceeds 100% in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-2264: -- Target Version/s: 0.22.0, 1.1.1 (was: 0.22.0, 1.1.0) Job status exceeds 100% in some cases -- Key: MAPREDUCE-2264 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2, 0.20.205.0 Reporter: Adam Kramer Assignee: Devaraj K Labels: critical-0.22.0 Attachments: MAPREDUCE-2264-0.20.205-1.patch, MAPREDUCE-2264-0.20.205.patch, MAPREDUCE-2264-0.20.3.patch, MAPREDUCE-2264-trunk.patch, more than 100%.bmp I'm looking now at my jobtracker's list of running reduce tasks. One of them is 120.05% complete, the other is 107.28% complete. I understand that these numbers are estimates, but there is no case in which an estimate of 100% for a non-complete task is better than an estimate of 99.99%, nor is there any case in which an estimate greater than 100% is valid. I suggest that whatever logic is computing these set 99.99% as a hard maximum. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2340) optimize JobInProgress.initTasks()
[ https://issues.apache.org/jira/browse/MAPREDUCE-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-2340: -- Target Version/s: 0.22.0, 1.2.0 (was: 0.22.0, 1.1.0) optimize JobInProgress.initTasks() -- Key: MAPREDUCE-2340 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2340 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 0.20.1, 0.21.0 Reporter: Kang Xiao Labels: critical-0.22.0 Attachments: MAPREDUCE-2340.patch, MAPREDUCE-2340.patch, MAPREDUCE-2340.r1.diff JobTracker's hostnameToNodeMap cache can speed up JobInProgress.initTasks() and JobInProgress.createCache() significantly. A test for 1 job with 10 maps on a 2400 cluster shows nearly 10 and 50 times speed up for initTasks() and createCache(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3751) Simplify job submission in gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-3751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-3751: -- Target Version/s: 2.0.0-alpha, 1.2.0 (was: 1.1.0, 2.0.0-alpha) Simplify job submission in gridmix -- Key: MAPREDUCE-3751 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3751 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix, mrv2 Affects Versions: 0.23.0, 1.0.0 Reporter: Arun C Murthy Currently gridmix tries to gauge cluster load etc. and throttles job submission. This makes it unpredictable and also is hard to support across MR1 and MR2. I propose we simplify it to be: # Replay mode - Just submit jobs in the interval as in the original trace. # Stress mode - Compress the interval with a given factor for all jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3936) Clients should not enforce counter limits
[ https://issues.apache.org/jira/browse/MAPREDUCE-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-3936: -- Target Version/s: 1.1.1, 2.0.3-alpha (was: 1.1.0, 2.0.3-alpha) Clients should not enforce counter limits -- Key: MAPREDUCE-3936 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3936 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-3936.patch, MAPREDUCE-3936.patch The code for enforcing counter limits (from MAPREDUCE-1943) creates a static JobConf instance to load the limits, which may throw an exception if the client limit is set to be lower than the limit on the cluster (perhaps because the cluster limit was raised from the default). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4272) SortedRanges.Range#compareTo is not spec compliant
[
https://issues.apache.org/jira/browse/MAPREDUCE-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated MAPREDUCE-4272:
--
Target Version/s: 2.0.0-alpha, 1.2.0 (was: 1.1.0, 2.0.0-alpha)
SortedRanges.Range#compareTo is not spec compliant
--
Key: MAPREDUCE-4272
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4272
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 1.0.3
Reporter: Luke Lu
Assignee: Luke Lu
Labels: alt-jdk
Attachments: mapreduce-4272-branch-1.0.patch
SortedRanges.Range#compareTo does not satisfy the requirement of
Comparable#compareTo, where the implementor must ensure
{noformat}sgn(x.compareTo(y)) == -sgn(y.compareTo(x)){noformat} for all x and
y.
This is manifested as TestStreamingBadRecords failures in alternative JDKs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4398) Fix mapred.system.dir permission error with FairScheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478887#comment-13478887 ] Luke Lu commented on MAPREDUCE-4398: The magic number of 4 is the default number of job init threads (mapred.jobinit.threads). You have to submit 4 (or precisely mapred.jobinit.threads) or more jobs as the jobtracker user at the same time to make sure the job init thread are initialized as the system user so they can access the mapred.system.dir (for security reasons, it must be 700). Otherwise, some of the job init threads will be initialized as whatever user who first submits a job. This can lead to seemingly more bizarre behavior: some time it works (the job is initialized by one of the system threads) and sometime it doesn't (the job is initialized by one of the user threads). Once you know the root cause, it's pretty trivial to come up with a patch. The default fifo scheduler and capacity scheduler do not have this bug. Fix mapred.system.dir permission error with FairScheduler - Key: MAPREDUCE-4398 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4398 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Gao Incorrect job initialization logic in FairScheduler causes mysterious intermittent mapred.system.dir permission errors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4721) Task startup time in JHS is same as job startup time.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478897#comment-13478897 ] Hudson commented on MAPREDUCE-4721: --- Integrated in Hadoop-Yarn-trunk #7 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/7/]) MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1399289 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm Task startup time in JHS is same as job startup time. - Key: MAPREDUCE-4721 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4721 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha Reporter: Ravi Prakash Assignee: Ravi Prakash Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 Attachments: MAPREDUCE-4721.patch, MAPREDUCE-4721.patch As Bobby pointed out in https://issues.apache.org/jira/browse/MAPREDUCE-4711?focusedCommentId=13471696page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13471696 In the Map and Reduce tasks page, it should print the earliest task attempt launch time as TaskImpl:getLaunchTime() does. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3881) building fail under Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478928#comment-13478928
]
caolong commented on MAPREDUCE-3881:
ok
This patch fixes the issue for me.
building fail under Windows
---
Key: MAPREDUCE-3881
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3881
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Environment: D:\os\hadoopcommonmvn --version
Apache Maven 3.0.4 (r1232337; 2012-01-17 16:44:56+0800)
Maven home: C:\portable\maven\bin\..
Java version: 1.7.0_02, vendor: Oracle Corporation
Java home: C:\Program Files (x86)\Java\jdk1.7.0_02\jre
Default locale: zh_CN, platform encoding: GBK
OS name: windows 7, version: 6.1, arch: x86, family: windows
Reporter: Changming Sun
Priority: Minor
Attachments: pom.xml.patch
Original Estimate: 1h
Remaining Estimate: 1h
hadoop-mapreduce-project\hadoop-yarn\hadoop-yarn-common\pom.xml is not
portable.
execution
idgenerate-version/id
phasegenerate-sources/phase
configuration
executablescripts/saveVersion.sh/executable
arguments
argument${project.version}/argument
argument${project.build.directory}/argument
/arguments
/configuration
goals
goalexec/goal
/goals
/execution
when I built it under windows , I got a such error:
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec
(gen
erate-version) on project hadoop-yarn-common: Command execution failed.
Cannot r
un program scripts\saveVersion.sh (in directory
D:\os\hadoopcommon\hadoop-map
reduce-project\hadoop-yarn\hadoop-yarn-common): CreateProcess error=2,
? - [Help 1]
we should modify it like this: (copied from
hadoop-common-project\hadoop-common\pom.xml)
configuration
target
mkdir
dir=${project.build.directory}/generated-sources/java/
exec executable=sh
arg
line=${basedir}/dev-support/saveVersion.sh
${project.version} ${project.build.directory}/generated-sources/java/
/exec
/target
/configuration
/execution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4721) Task startup time in JHS is same as job startup time.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478956#comment-13478956 ] Hudson commented on MAPREDUCE-4721: --- Integrated in Hadoop-Hdfs-0.23-Build #408 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/408/]) svn merge -c 1399289 FIXES: MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399293) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1399293 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm Task startup time in JHS is same as job startup time. - Key: MAPREDUCE-4721 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4721 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha Reporter: Ravi Prakash Assignee: Ravi Prakash Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 Attachments: MAPREDUCE-4721.patch, MAPREDUCE-4721.patch As Bobby pointed out in https://issues.apache.org/jira/browse/MAPREDUCE-4711?focusedCommentId=13471696page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13471696 In the Map and Reduce tasks page, it should print the earliest task attempt launch time as TaskImpl:getLaunchTime() does. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4721) Task startup time in JHS is same as job startup time.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478971#comment-13478971 ] Hudson commented on MAPREDUCE-4721: --- Integrated in Hadoop-Hdfs-trunk #1199 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1199/]) MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1399289 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm Task startup time in JHS is same as job startup time. - Key: MAPREDUCE-4721 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4721 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha Reporter: Ravi Prakash Assignee: Ravi Prakash Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 Attachments: MAPREDUCE-4721.patch, MAPREDUCE-4721.patch As Bobby pointed out in https://issues.apache.org/jira/browse/MAPREDUCE-4711?focusedCommentId=13471696page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13471696 In the Map and Reduce tasks page, it should print the earliest task attempt launch time as TaskImpl:getLaunchTime() does. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4721) Task startup time in JHS is same as job startup time.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478988#comment-13478988 ] Hudson commented on MAPREDUCE-4721: --- Integrated in Hadoop-Mapreduce-trunk #1229 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1229/]) MAPREDUCE-4721. Task startup time in JHS is same as job startup time. (Ravi Prakash via bobby) (Revision 1399289) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1399289 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestCompletedTask.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm Task startup time in JHS is same as job startup time. - Key: MAPREDUCE-4721 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4721 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha Reporter: Ravi Prakash Assignee: Ravi Prakash Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 Attachments: MAPREDUCE-4721.patch, MAPREDUCE-4721.patch As Bobby pointed out in https://issues.apache.org/jira/browse/MAPREDUCE-4711?focusedCommentId=13471696page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13471696 In the Map and Reduce tasks page, it should print the earliest task attempt launch time as TaskImpl:getLaunchTime() does. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4731) FSShell doble encodes qualified Paths
Robert Joseph Evans created MAPREDUCE-4731:
--
Summary: FSShell doble encodes qualified Paths
Key: MAPREDUCE-4731
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4731
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.2-alpha, 0.23.3
Reporter: Robert Joseph Evans
{noformat}
$ hadoop fs -mkdir /tmp/me
$ hadoop fs -touchz /tmp/me/A%3AB
$ hadoop fs -ls /tmp/me/A%3AB
Found 1 items
-rw--- 3 me hdfs 0 2012-10-18 17:47 /tmp/me/A%3AB
$ hadoop fs -ls hdfs:///tmp/me/A%3AB
Found 1 items
-rw--- 3 me hdfs 0 2012-10-18 17:47 hdfs:///tmp/me/A%253AB
$ hadoop fs -cat hdfs:///tmp/me/A%3AB
cat: File does not exist: /tmp/me/A%253AB
$ hadoop fs -cat /tmp/me/A%3AB
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4731) FSShell double encodes qualified Paths
[
https://issues.apache.org/jira/browse/MAPREDUCE-4731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4731:
---
Summary: FSShell double encodes qualified Paths (was: FSShell doble
encodes qualified Paths)
FSShell double encodes qualified Paths
--
Key: MAPREDUCE-4731
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4731
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.3, 2.0.2-alpha
Reporter: Robert Joseph Evans
{noformat}
$ hadoop fs -mkdir /tmp/me
$ hadoop fs -touchz /tmp/me/A%3AB
$ hadoop fs -ls /tmp/me/A%3AB
Found 1 items
-rw--- 3 me hdfs 0 2012-10-18 17:47 /tmp/me/A%3AB
$ hadoop fs -ls hdfs:///tmp/me/A%3AB
Found 1 items
-rw--- 3 me hdfs 0 2012-10-18 17:47 hdfs:///tmp/me/A%253AB
$ hadoop fs -cat hdfs:///tmp/me/A%3AB
cat: File does not exist: /tmp/me/A%253AB
$ hadoop fs -cat /tmp/me/A%3AB
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4731) FSShell double encodes qualified Paths
[
https://issues.apache.org/jira/browse/MAPREDUCE-4731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4731:
---
Attachment: fails.txt
This patch adds in a unit tests that exercises the failure.
FSShell double encodes qualified Paths
--
Key: MAPREDUCE-4731
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4731
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.3, 2.0.2-alpha
Reporter: Robert Joseph Evans
Attachments: fails.txt
{noformat}
$ hadoop fs -mkdir /tmp/me
$ hadoop fs -touchz /tmp/me/A%3AB
$ hadoop fs -ls /tmp/me/A%3AB
Found 1 items
-rw--- 3 me hdfs 0 2012-10-18 17:47 /tmp/me/A%3AB
$ hadoop fs -ls hdfs:///tmp/me/A%3AB
Found 1 items
-rw--- 3 me hdfs 0 2012-10-18 17:47 hdfs:///tmp/me/A%253AB
$ hadoop fs -cat hdfs:///tmp/me/A%3AB
cat: File does not exist: /tmp/me/A%253AB
$ hadoop fs -cat /tmp/me/A%3AB
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4729) job history UI not showing all job attempts
[
https://issues.apache.org/jira/browse/MAPREDUCE-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479201#comment-13479201
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-4729:
It looks like we retain the history files from all attempts, did you look at
them all? Also, please see if you find this log in any of the AM attempts:
{code}
Got an error parsing job-history file + historyFile + , ignoring incomplete
events.
{code}
job history UI not showing all job attempts
---
Key: MAPREDUCE-4729
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4729
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 0.23.3
Reporter: Thomas Graves
We are seeing a case where a job runs but the AM is running out of memory in
the first 3 attempts. The job eventually finishes on the 4th attempt. When
you go to the job history UI for that job, it only shows the last attempt.
This is bad since we want to see why the first 3 attempts failed.
The RM web ui shows all 4 attempts.
Also I tested this locally by running kill on the app master and in that
case the history server UI does show all attempts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4596) Split StateMachine state from states seen by MRClientProtocol (for Job, Task, TaskAttempt)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-4596:
---
Status: Open (was: Patch Available)
Looking through the patch again, found a bug. We no longer have a
TaskAttemptState.UNASSIGNED, so the method {{TaskAttemptState
toYarn(org.apache.hadoop.mapred.TaskStatus.State state)}} will fail. To avoid
these bugs in future, we may explicitly code all the states instead of using
.valueOf().
Split StateMachine state from states seen by MRClientProtocol (for Job, Task,
TaskAttempt)
--
Key: MAPREDUCE-4596
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4596
Project: Hadoop Map/Reduce
Issue Type: Task
Components: applicationmaster, mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: MR4596_2.txt, MR4596_3.txt, MR4596-branch23_1.txt,
MR4596.txt
State machine states are currently exposed via MRClienProtocol. This makes it
tough to modify the AM state machines, or have an alternate AM with different
state machines (MR-3902) without the changes being visible in
MRClientProtocol (MRv2 equivalent of ClientProtocol).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479257#comment-13479257 ] Jason Lowe commented on MAPREDUCE-4730: --- A little more digging and I'm a bit more confident that this is a flow control problem in the IPC layer. I think the scenario goes like this: # 1000's of reducers start asking for map completion events about the same time # IPC Server.Handler thread fields a call off the queue, makes the call and gets 900K of data # Handler thread queues up the response data to the connection, likely sees its the only thing in the queue, and tries to push out the data # It's too big to send it all without blocking so it pushes the remainder back onto the response queue for the Responder thread to deal with and moves on to another call from the call queue # Lots of reducers are queueing up in the call queue to get their 900K of data, and the handler threads are processing them and pushing that data on the response queues as fast as they can # Responder thread and/or socket I/O can't keep pace with the rate at which handlers are generating 900K responses and we eventually exhaust memory AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Priority: Blocker We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4730: -- Attachment: MAPREDUCE-4730.patch Patch to lower the number of map completion events reducers ask for at a time from 1 to 500. This is a short-term fix to allow 20x the number of reducers to run in the same IPC response footprint as before. Ran a sleep job test with 2 mappers and 3000 reducers with the fix, and it was able to complete with a standard AM size (1.5GB slot). AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4730.patch We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4730: -- Assignee: Jason Lowe Target Version/s: 2.0.3-alpha, 0.23.5 Status: Patch Available (was: Open) AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4730.patch We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479286#comment-13479286 ] Robert Joseph Evans commented on MAPREDUCE-4730: The patch is simple enough if Jenkins comes back OK I am a +1 on it. AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4730.patch We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479290#comment-13479290 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4730: Great analysis! 900K * 3000 reducers = 2.7GB, so the numbers are adding up. Instead of hard-coding it, each reducer could base it on the total number of reducers for the job (from configuration)? AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4730.patch We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479291#comment-13479291
]
Hadoop QA commented on MAPREDUCE-4730:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549736/MAPREDUCE-4730.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2939//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2939//console
This message is automatically generated.
AM crashes due to OOM while serving up map task completion events
-
Key: MAPREDUCE-4730
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.3
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
Attachments: MAPREDUCE-4730.patch
We're seeing a repeatable OOM crash in the AM for a task with around 3
maps and 3000 reducers. Details to follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479296#comment-13479296 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4730: Also, lessening it has performance implications on small jobs (not sure how much) given the fetcher loop runs every 1 second irrespective of whether there are more events or not. So, hate to propose it, but shall we add in a config to override this? AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4730.patch We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4729) job history UI not showing all job attempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479306#comment-13479306 ] Thomas Graves commented on MAPREDUCE-4729: -- Ok, so I figured this out. The job is using output format org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat, which has the OutputCommitter which is set to null. This caused the MRAppMaster recoveryService to not start: org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Not starting RecoveryService: recoveryEnabled: true recoverySupportedByCommitter: false ApplicationAttemptID: 4 Since the recovery service didn't start it didn't parse the old job history files, thus didn't have the list of old AMs. I think we should fix that so that even if recovery isn't supported we atleast parse and get the previous AM attempt info. job history UI not showing all job attempts --- Key: MAPREDUCE-4729 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4729 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.3 Reporter: Thomas Graves We are seeing a case where a job runs but the AM is running out of memory in the first 3 attempts. The job eventually finishes on the 4th attempt. When you go to the job history UI for that job, it only shows the last attempt. This is bad since we want to see why the first 3 attempts failed. The RM web ui shows all 4 attempts. Also I tested this locally by running kill on the app master and in that case the history server UI does show all attempts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4729) job history UI not showing all job attempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479309#comment-13479309 ] Thomas Graves commented on MAPREDUCE-4729: -- I'm not a pig expert but perhaps we should also follow up with pig team to see if they should support Recovery job history UI not showing all job attempts --- Key: MAPREDUCE-4729 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4729 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.3 Reporter: Thomas Graves We are seeing a case where a job runs but the AM is running out of memory in the first 3 attempts. The job eventually finishes on the 4th attempt. When you go to the job history UI for that job, it only shows the last attempt. This is bad since we want to see why the first 3 attempts failed. The RM web ui shows all 4 attempts. Also I tested this locally by running kill on the app master and in that case the history server UI does show all attempts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479311#comment-13479311 ] Jason Lowe commented on MAPREDUCE-4730: --- Is the 1 second sleep necessary? Seems like we could eliminate that sleep if we got a maximum-sized response? AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4730.patch We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4730: -- Status: Open (was: Patch Available) I think immediately turning around and asking for the next MAX_EVENTS maps if we just received MAX_EVENTS entries would be a straightforward way to eliminate the sleep penalty. Unfortunately I don't think that will work all the time due to another bug where the caller can receive less than MAX_EVENTS entries even though that many entries were processed during the call. TaskAttemptListenerImpl is calling Job.getTaskAttemptCompletionEvents with the same fromEvent and maxEvents passed in from the reducer but is then filtering the result for just map events. This means that even though we receive maxEvents in completion events the caller could see less than that if there are one or more reducer completion event mixed in there. Worse, if all of the events are reducer events then zero events will be reported back to the caller and it won't bump up fromEvent on the next call. Reducer never makes progress and we're toast. This could happen in a case where all maps complete, more than MAX_EVENTS reducers complete, but some straggling reducers get fetch failures and cause a map to be restarted. This is less likely to occur with an ask size of 1 since you'd have to have 1 reducers complete in a row, but it's more likely with an ask size of 500. AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4730.patch We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4729) job history UI not showing all job attempts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479369#comment-13479369 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4729: bq. I think we should fix that so that even if recovery isn't supported we atleast parse and get the previous AM attempt info. +1 bq. I'm not a pig expert but perhaps we should also follow up with pig team to see if they should support Recovery This is a general issue with all OutputFormats. We need to educate users that they need to start implementing recovery. job history UI not showing all job attempts --- Key: MAPREDUCE-4729 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4729 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.3 Reporter: Thomas Graves We are seeing a case where a job runs but the AM is running out of memory in the first 3 attempts. The job eventually finishes on the 4th attempt. When you go to the job history UI for that job, it only shows the last attempt. This is bad since we want to see why the first 3 attempts failed. The RM web ui shows all 4 attempts. Also I tested this locally by running kill on the app master and in that case the history server UI does show all attempts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4732) testcase testJobRetire fails using IBM JAVA 7
Amir Sanjar created MAPREDUCE-4732: -- Summary: testcase testJobRetire fails using IBM JAVA 7 Key: MAPREDUCE-4732 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4732 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.0.3 Environment: RHEL 6.2 with IBM JAVA 7 on a x86_64 system Reporter: Amir Sanjar Priority: Minor Testcase: testJobRetire took 53.352 sec Testcase: testJobRetireWithUnreportedTasks took 41.173 sec FAILED Job did not retire junit.framework.AssertionFailedError: Job did not retire at org.apache.hadoop.mapred.TestJobRetire.waitTillRetire(TestJobRetire.java:130) at org.apache.hadoop.mapred.TestJobRetire.testJobRetireWithUnreportedTasks(TestJobRetire.java:229) Testcase: testJobRemoval took 1.073 sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4596) Split StateMachine state from states seen by MRClientProtocol (for Job, Task, TaskAttempt)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4596: -- Attachment: MR4596_4.txt Split StateMachine state from states seen by MRClientProtocol (for Job, Task, TaskAttempt) -- Key: MAPREDUCE-4596 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4596 Project: Hadoop Map/Reduce Issue Type: Task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MR4596_2.txt, MR4596_3.txt, MR4596_4.txt, MR4596-branch23_1.txt, MR4596-branch23_2.txt, MR4596.txt State machine states are currently exposed via MRClienProtocol. This makes it tough to modify the AM state machines, or have an alternate AM with different state machines (MR-3902) without the changes being visible in MRClientProtocol (MRv2 equivalent of ClientProtocol). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4596) Split StateMachine state from states seen by MRClientProtocol (for Job, Task, TaskAttempt)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4596: -- Status: Patch Available (was: Open) Split StateMachine state from states seen by MRClientProtocol (for Job, Task, TaskAttempt) -- Key: MAPREDUCE-4596 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4596 Project: Hadoop Map/Reduce Issue Type: Task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: MR4596_2.txt, MR4596_3.txt, MR4596_4.txt, MR4596-branch23_1.txt, MR4596-branch23_2.txt, MR4596.txt State machine states are currently exposed via MRClienProtocol. This makes it tough to modify the AM state machines, or have an alternate AM with different state machines (MR-3902) without the changes being visible in MRClientProtocol (MRv2 equivalent of ClientProtocol). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4732) testcase testJobRetire fails using IBM JAVA 7
[
https://issues.apache.org/jira/browse/MAPREDUCE-4732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479395#comment-13479395
]
Amir Sanjar commented on MAPREDUCE-4732:
debug result:
private void waitTillRetire(JobID id, JobTracker jobtracker) {
//wait for job to get retired
JobInProgress job = jobtracker.getJob(id);
for (int i = 0; i 10 job != null; i++) {
UtilsForTests.waitFor(1000);
job = jobtracker.getJob(id);
}
assertNull(Job did not retire, job); = assertion job != null
Conclusion: this is a timing issue, build and run with IBM JAVA 7, this
particular jobs does not terminate in the given maximum time of 10sec, Works
with 15 sec on a TP520.
Having a hardcoded max time is not recommanded..
We are testing a patch.. will be up loaded shortly ..
testcase testJobRetire fails using IBM JAVA 7
-
Key: MAPREDUCE-4732
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4732
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 1.0.3
Environment: RHEL 6.2 with IBM JAVA 7 on a x86_64 system
Reporter: Amir Sanjar
Priority: Minor
Testcase: testJobRetire took 53.352 sec
Testcase: testJobRetireWithUnreportedTasks took 41.173 sec
FAILED
Job did not retire
junit.framework.AssertionFailedError: Job did not retire
at
org.apache.hadoop.mapred.TestJobRetire.waitTillRetire(TestJobRetire.java:130)
at
org.apache.hadoop.mapred.TestJobRetire.testJobRetireWithUnreportedTasks(TestJobRetire.java:229)
Testcase: testJobRemoval took 1.073 sec
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4229) Intern counter names in the JT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479396#comment-13479396 ] Miomir Boljanovic commented on MAPREDUCE-4229: -- Sorry, this has been a hectic week for me too. Unfortunately, I don't have any figures to share yet but will try to capture some heap dumps over the weekend. Intern counter names in the JT -- Key: MAPREDUCE-4229 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4229 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 1.0.2, 3.0.0, 2.0.2-alpha Reporter: Todd Lipcon Attachments: MAPREDUCE-4229-branch-0.23.patch, MAPREDUCE-4229.patch In our experience, most of the memory in production JTs goes to storing counter names (String objects and character arrays). Since most counter names are reused again and again, it would be a big memory savings to keep a hash set of already-used counter names within a job, and refer to the same object from all tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4596) Split StateMachine state from states seen by MRClientProtocol (for Job, Task, TaskAttempt)
[
https://issues.apache.org/jira/browse/MAPREDUCE-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479407#comment-13479407
]
Hadoop QA commented on MAPREDUCE-4596:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549757/MR4596_4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2940//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2940//console
This message is automatically generated.
Split StateMachine state from states seen by MRClientProtocol (for Job, Task,
TaskAttempt)
--
Key: MAPREDUCE-4596
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4596
Project: Hadoop Map/Reduce
Issue Type: Task
Components: applicationmaster, mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: MR4596_2.txt, MR4596_3.txt, MR4596_4.txt,
MR4596-branch23_1.txt, MR4596-branch23_2.txt, MR4596.txt
State machine states are currently exposed via MRClienProtocol. This makes it
tough to modify the AM state machines, or have an alternate AM with different
state machines (MR-3902) without the changes being visible in
MRClientProtocol (MRv2 equivalent of ClientProtocol).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4229) Intern counter names in the JT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4229: --- Attachment: MR-4229.txt Patch that reduces memory consumption on History Server. Intern counter names in the JT -- Key: MAPREDUCE-4229 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4229 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 1.0.2, 3.0.0, 2.0.2-alpha Reporter: Todd Lipcon Attachments: MAPREDUCE-4229-branch-0.23.patch, MAPREDUCE-4229.patch, MR-4229.txt In our experience, most of the memory in production JTs goes to storing counter names (String objects and character arrays). Since most counter names are reused again and again, it would be a big memory savings to keep a hash set of already-used counter names within a job, and refer to the same object from all tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4229) Intern counter names in the JT
[
https://issues.apache.org/jira/browse/MAPREDUCE-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479420#comment-13479420
]
Robert Joseph Evans commented on MAPREDUCE-4229:
I rand some benchmarks looking at the Job History server using a jhist file for
a job that had 9416 maps and 500 reducers. I then used a combination of
YourKit and jhat to look at the heap savings.
For Jhat I did the OQL {noformat}select
sum(map(heap.objects(java.lang.String),sizeof(it))){noformat} to get the
size of all of the strings currently reachable on the heap.
I saw that nothing changed in between the base and the first patch. Both of
them had 22MB of strings in the heap. Looking at the code that was changed to
do interning, the only code that uses it was rumen. It is still a good change,
but it did not have the impact I was looking for. So I implemented the patch I
just attached which adds in interning of Strings that are parsed out of the
jhist file. This reduced the 22MB of strings to 3MB of strings.
I want to do something similar for the AM, but it is more difficult to look at,
and I don't think I will have time in the near future. So if someone else could
review this we can check it in and file a follow up JIRA for looking at the AM.
Intern counter names in the JT
--
Key: MAPREDUCE-4229
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4229
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobtracker
Affects Versions: 1.0.2, 3.0.0, 2.0.2-alpha
Reporter: Todd Lipcon
Attachments: MAPREDUCE-4229-branch-0.23.patch, MAPREDUCE-4229.patch,
MR-4229.txt
In our experience, most of the memory in production JTs goes to storing
counter names (String objects and character arrays). Since most counter names
are reused again and again, it would be a big memory savings to keep a hash
set of already-used counter names within a job, and refer to the same object
from all tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4733) Reducer can fail to make progress during shuffle if too many reducers complete consecutively
Jason Lowe created MAPREDUCE-4733: - Summary: Reducer can fail to make progress during shuffle if too many reducers complete consecutively Key: MAPREDUCE-4733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4733 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe TaskAttemptListenerImpl implements getMapCompletionEvents by calling Job.getTaskAttemptCompletionEvents with the same fromEvent and maxEvents passed in from the reducer and then filtering the result for just map events. We can't filter the task completion event list and expect the caller's window into the list to match up. As soon as a reducer event appears in the list it means we are redundantly sending map completion events that were already seen by the reducer. Worst case the reducer will hang if all of the events in the requested window are reducer events. In that case zero events will be reported back to the caller and it won't bump up fromEvent on the next call. Reducer then never sees the final map completion events needed to complete the shuffle. This could happen in a case where all maps complete, more than MAX_EVENTS reducers complete consecutively, but some straggling reducers get fetch failures and cause a map to be restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479432#comment-13479432 ] Jason Lowe commented on MAPREDUCE-4730: --- Filed MAPREDUCE-4733 to track the filtering/windowing issue in TaskAttemptListenerImpl.getMapCompletionEvents AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4730.patch We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4229) Intern counter names in the JT
[
https://issues.apache.org/jira/browse/MAPREDUCE-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479436#comment-13479436
]
Hadoop QA commented on MAPREDUCE-4229:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549767/MR-4229.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2941//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2941//console
This message is automatically generated.
Intern counter names in the JT
--
Key: MAPREDUCE-4229
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4229
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobtracker
Affects Versions: 1.0.2, 3.0.0, 2.0.2-alpha
Reporter: Todd Lipcon
Attachments: MAPREDUCE-4229-branch-0.23.patch, MAPREDUCE-4229.patch,
MR-4229.txt
In our experience, most of the memory in production JTs goes to storing
counter names (String objects and character arrays). Since most counter names
are reused again and again, it would be a big memory savings to keep a hash
set of already-used counter names within a job, and refer to the same object
from all tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4723) Fix warnings found by findbugs 2
[ https://issues.apache.org/jira/browse/MAPREDUCE-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4723: -- Attachment: MAPREDUCE-4723.patch Fix warnings found by findbugs 2 Key: MAPREDUCE-4723 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4723 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4723.patch, MAPREDUCE-4723.patch, MAPREDUCE-4723.patch The MAPREDUCE side of HADOOP-8594. Umbrella jira for fixing the warnings found by findbugs 2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4723) Fix warnings found by findbugs 2
[
https://issues.apache.org/jira/browse/MAPREDUCE-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479489#comment-13479489
]
Hadoop QA commented on MAPREDUCE-4723:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549778/MAPREDUCE-4723.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
-4 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
hadoop-mapreduce-project/hadoop-mapreduce-examples:
org.apache.hadoop.mapreduce.v2.app.TestRecovery
org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing
org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEvents
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2942//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2942//console
This message is automatically generated.
Fix warnings found by findbugs 2
Key: MAPREDUCE-4723
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4723
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4723.patch, MAPREDUCE-4723.patch,
MAPREDUCE-4723.patch
The MAPREDUCE side of HADOOP-8594. Umbrella jira for fixing the warnings
found by findbugs 2.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2454) Allow external sorter plugin for MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479501#comment-13479501 ] Arun C Murthy commented on MAPREDUCE-2454: -- Asokan, sorry I've been busy with stuff - thanks for understanding. I've spent sometime thinking about this - and I feel we can do something far simpler to address Syncsort's goal of plugging in your proprietary sort while mitigating risk to MR itself. How about this: I feel we could accomplish both goals by something very simple... by making MapOutputBuffer pluggable by introducing a MapOutputCollector interface. That's about it. This way, you can supply a custom MapOutputBuffer which plugs in your sort for your customers while we can just keep our current implementation. Hopefully, that makes sense. What else would you need? I'm basically trying to vastly minimize the APIs we spread out, this way when we want to change our sort implementation for MAPREDUCE-4039 or Sailfish etc. we have *much* more leeway, at the same time we don't affect you at all. Thoughts? Allow external sorter plugin for MR --- Key: MAPREDUCE-2454 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2454 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 2.0.0-alpha, 3.0.0, 2.0.2-alpha Reporter: Mariappan Asokan Assignee: Mariappan Asokan Priority: Minor Labels: features, performance, plugin, sort Attachments: HadoopSortPlugin.pdf, HadoopSortPlugin.pdf, KeyValueIterator.java, MapOutputSorterAbstract.java, MapOutputSorter.java, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mapreduce-2454.patch, mr-2454-on-mr-279-build82.patch.gz, MR-2454-trunkPatchPreview.gz, ReduceInputSorter.java Define interfaces and some abstract classes in the Hadoop framework to facilitate external sorter plugins both on the Map and Reduce sides. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4734) The history server should link back to NM logs if aggregation is incomplete / disabled
Siddharth Seth created MAPREDUCE-4734: - Summary: The history server should link back to NM logs if aggregation is incomplete / disabled Key: MAPREDUCE-4734 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4734 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver, mrv2 Affects Versions: 0.23.4 Reporter: Siddharth Seth Assignee: Siddharth Seth -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479532#comment-13479532 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4730: bq. Seems like we could eliminate that sleep if we got a maximum-sized response? +1. AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4730.patch We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4733) Reducer can fail to make progress during shuffle if too many reducers complete consecutively
[ https://issues.apache.org/jira/browse/MAPREDUCE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned MAPREDUCE-4733: - Assignee: Jason Lowe Reducer can fail to make progress during shuffle if too many reducers complete consecutively Key: MAPREDUCE-4733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4733 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe TaskAttemptListenerImpl implements getMapCompletionEvents by calling Job.getTaskAttemptCompletionEvents with the same fromEvent and maxEvents passed in from the reducer and then filtering the result for just map events. We can't filter the task completion event list and expect the caller's window into the list to match up. As soon as a reducer event appears in the list it means we are redundantly sending map completion events that were already seen by the reducer. Worst case the reducer will hang if all of the events in the requested window are reducer events. In that case zero events will be reported back to the caller and it won't bump up fromEvent on the next call. Reducer then never sees the final map completion events needed to complete the shuffle. This could happen in a case where all maps complete, more than MAX_EVENTS reducers complete consecutively, but some straggling reducers get fetch failures and cause a map to be restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (MAPREDUCE-4735) Make arguments in TestDFSIO case insensitive
[ https://issues.apache.org/jira/browse/MAPREDUCE-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas moved HADOOP-8897 to MAPREDUCE-4735: Component/s: (was: test) test Key: MAPREDUCE-4735 (was: HADOOP-8897) Project: Hadoop Map/Reduce (was: Hadoop Common) Make arguments in TestDFSIO case insensitive Key: MAPREDUCE-4735 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4735 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Reporter: Robert Kanter Assignee: Brandon Li Priority: Trivial Attachments: HADOOP-8897.patch, HADOOP-8897.patch It would be convenient if the arguments in TestDFSIO were case insensitive. For example, it should allow -read, -Read, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4735) Make arguments in TestDFSIO case insensitive
[ https://issues.apache.org/jira/browse/MAPREDUCE-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-4735: --- Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 for the patch. I committed it to trunk. Thank you Brandon. Make arguments in TestDFSIO case insensitive Key: MAPREDUCE-4735 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4735 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Reporter: Robert Kanter Assignee: Brandon Li Priority: Trivial Fix For: 3.0.0 Attachments: HADOOP-8897.patch, HADOOP-8897.patch It would be convenient if the arguments in TestDFSIO were case insensitive. For example, it should allow -read, -Read, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4733) Reducer can fail to make progress during shuffle if too many reducers complete consecutively
[ https://issues.apache.org/jira/browse/MAPREDUCE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4733: -- Attachment: MAPREDUCE-4733.patch Patch to add a new method to the AM's Job interface so map task completions can be reliably iterated by reducers. Reducer can fail to make progress during shuffle if too many reducers complete consecutively Key: MAPREDUCE-4733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4733 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-4733.patch TaskAttemptListenerImpl implements getMapCompletionEvents by calling Job.getTaskAttemptCompletionEvents with the same fromEvent and maxEvents passed in from the reducer and then filtering the result for just map events. We can't filter the task completion event list and expect the caller's window into the list to match up. As soon as a reducer event appears in the list it means we are redundantly sending map completion events that were already seen by the reducer. Worst case the reducer will hang if all of the events in the requested window are reducer events. In that case zero events will be reported back to the caller and it won't bump up fromEvent on the next call. Reducer then never sees the final map completion events needed to complete the shuffle. This could happen in a case where all maps complete, more than MAX_EVENTS reducers complete consecutively, but some straggling reducers get fetch failures and cause a map to be restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4733) Reducer can fail to make progress during shuffle if too many reducers complete consecutively
[ https://issues.apache.org/jira/browse/MAPREDUCE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4733: -- Target Version/s: 2.0.3-alpha, 0.23.5 Status: Patch Available (was: Open) Reducer can fail to make progress during shuffle if too many reducers complete consecutively Key: MAPREDUCE-4733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4733 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-4733.patch TaskAttemptListenerImpl implements getMapCompletionEvents by calling Job.getTaskAttemptCompletionEvents with the same fromEvent and maxEvents passed in from the reducer and then filtering the result for just map events. We can't filter the task completion event list and expect the caller's window into the list to match up. As soon as a reducer event appears in the list it means we are redundantly sending map completion events that were already seen by the reducer. Worst case the reducer will hang if all of the events in the requested window are reducer events. In that case zero events will be reported back to the caller and it won't bump up fromEvent on the next call. Reducer then never sees the final map completion events needed to complete the shuffle. This could happen in a case where all maps complete, more than MAX_EVENTS reducers complete consecutively, but some straggling reducers get fetch failures and cause a map to be restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4735) Make arguments in TestDFSIO case insensitive
[ https://issues.apache.org/jira/browse/MAPREDUCE-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479592#comment-13479592 ] Hudson commented on MAPREDUCE-4735: --- Integrated in Hadoop-trunk-Commit #2891 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/2891/]) MAPREDUCE-4735. Make arguments in TestDFSIO case insensitive. Contributed by Brandon Li. (Revision 1399952) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1399952 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java Make arguments in TestDFSIO case insensitive Key: MAPREDUCE-4735 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4735 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Reporter: Robert Kanter Assignee: Brandon Li Priority: Trivial Fix For: 3.0.0 Attachments: HADOOP-8897.patch, HADOOP-8897.patch It would be convenient if the arguments in TestDFSIO were case insensitive. For example, it should allow -read, -Read, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4733) Reducer can fail to make progress during shuffle if too many reducers complete consecutively
[
https://issues.apache.org/jira/browse/MAPREDUCE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479597#comment-13479597
]
Hadoop QA commented on MAPREDUCE-4733:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549801/MAPREDUCE-4733.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
-4 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2943//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2943//console
This message is automatically generated.
Reducer can fail to make progress during shuffle if too many reducers
complete consecutively
Key: MAPREDUCE-4733
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4733
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.3
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: MAPREDUCE-4733.patch
TaskAttemptListenerImpl implements getMapCompletionEvents by calling
Job.getTaskAttemptCompletionEvents with the same fromEvent and maxEvents
passed in from the reducer and then filtering the result for just map events.
We can't filter the task completion event list and expect the caller's
window into the list to match up. As soon as a reducer event appears in
the list it means we are redundantly sending map completion events that were
already seen by the reducer.
Worst case the reducer will hang if all of the events in the requested window
are reducer events. In that case zero events will be reported back to the
caller and it won't bump up fromEvent on the next call. Reducer then never
sees the final map completion events needed to complete the shuffle. This
could happen in a case where all maps complete, more than MAX_EVENTS reducers
complete consecutively, but some straggling reducers get fetch failures and
cause a map to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4733) Reducer can fail to make progress during shuffle if too many reducers complete consecutively
[ https://issues.apache.org/jira/browse/MAPREDUCE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479600#comment-13479600 ] Jason Lowe commented on MAPREDUCE-4733: --- I believe the javadoc warnings are from HADOOP-8887 which added @goal and @phase tags that are generating warnings during javadoc generation. Reducer can fail to make progress during shuffle if too many reducers complete consecutively Key: MAPREDUCE-4733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4733 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-4733.patch TaskAttemptListenerImpl implements getMapCompletionEvents by calling Job.getTaskAttemptCompletionEvents with the same fromEvent and maxEvents passed in from the reducer and then filtering the result for just map events. We can't filter the task completion event list and expect the caller's window into the list to match up. As soon as a reducer event appears in the list it means we are redundantly sending map completion events that were already seen by the reducer. Worst case the reducer will hang if all of the events in the requested window are reducer events. In that case zero events will be reported back to the caller and it won't bump up fromEvent on the next call. Reducer then never sees the final map completion events needed to complete the shuffle. This could happen in a case where all maps complete, more than MAX_EVENTS reducers complete consecutively, but some straggling reducers get fetch failures and cause a map to be restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4736) Remove obsolete option [-rootDir] from TestDFSIO
Brandon Li created MAPREDUCE-4736: - Summary: Remove obsolete option [-rootDir] from TestDFSIO Key: MAPREDUCE-4736 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4736 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Reporter: Brandon Li Assignee: Brandon Li Priority: Trivial Attachments: MAPREDUCE-4736.patch Looks like this option is obsolete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4736) Remove obsolete option [-rootDir] from TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-4736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated MAPREDUCE-4736: -- Attachment: MAPREDUCE-4736.patch Remove obsolete option [-rootDir] from TestDFSIO Key: MAPREDUCE-4736 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4736 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Reporter: Brandon Li Assignee: Brandon Li Priority: Trivial Attachments: MAPREDUCE-4736.patch Looks like this option is obsolete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4736) Remove obsolete option [-rootDir] from TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-4736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated MAPREDUCE-4736: -- Status: Patch Available (was: Open) Remove obsolete option [-rootDir] from TestDFSIO Key: MAPREDUCE-4736 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4736 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Reporter: Brandon Li Assignee: Brandon Li Priority: Trivial Attachments: MAPREDUCE-4736.patch Looks like this option is obsolete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4736) Remove obsolete option [-rootDir] from TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-4736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated MAPREDUCE-4736: -- Description: Looks like this option is obsolete. Remove it to avoid confusion. (was: Looks like this option is obsolete.) Remove obsolete option [-rootDir] from TestDFSIO Key: MAPREDUCE-4736 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4736 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Priority: Trivial Attachments: MAPREDUCE-4736.patch Looks like this option is obsolete. Remove it to avoid confusion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4736) Remove obsolete option [-rootDir] from TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-4736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated MAPREDUCE-4736: -- Affects Version/s: 3.0.0 Remove obsolete option [-rootDir] from TestDFSIO Key: MAPREDUCE-4736 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4736 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Priority: Trivial Attachments: MAPREDUCE-4736.patch Looks like this option is obsolete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4730: -- Attachment: MAPREDUCE-4730.patch New patch that attempts to scale the maximum number of events a reducer will ask for per RPC call based on some fuzzy numbers. It still keeps maxEventsToFetch between 100 and 1 to avoid extremes. Since events appear to be just a little under 100 bytes each, the patch currently targets around 300MB of memory on the AM for RPC response processing. This can still be exceeded given enough reducers, but the user should be able to bump up the AM memory size at that point and buy quite a bit more reducers. This patch also implements the do-not-wait-if-we-got-a-full-response logic to avoid wasting time while trying to fetch all the completion events. Still need to do some testing at scale, but quick touch-testing on a single-node cluster seems to work so putting it out there for comment and Jenkins. AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4730.patch, MAPREDUCE-4730.patch We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4730: -- Status: Patch Available (was: Open) AM crashes due to OOM while serving up map task completion events - Key: MAPREDUCE-4730 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-4730.patch, MAPREDUCE-4730.patch We're seeing a repeatable OOM crash in the AM for a task with around 3 maps and 3000 reducers. Details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479633#comment-13479633
]
Hadoop QA commented on MAPREDUCE-4730:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549805/MAPREDUCE-4730.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
-4 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2945//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2945//console
This message is automatically generated.
AM crashes due to OOM while serving up map task completion events
-
Key: MAPREDUCE-4730
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.3
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
Attachments: MAPREDUCE-4730.patch, MAPREDUCE-4730.patch
We're seeing a repeatable OOM crash in the AM for a task with around 3
maps and 3000 reducers. Details to follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4736) Remove obsolete option [-rootDir] from TestDFSIO
[
https://issues.apache.org/jira/browse/MAPREDUCE-4736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479638#comment-13479638
]
Hadoop QA commented on MAPREDUCE-4736:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549803/MAPREDUCE-4736.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
-4 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2944//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2944//console
This message is automatically generated.
Remove obsolete option [-rootDir] from TestDFSIO
Key: MAPREDUCE-4736
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4736
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: test
Affects Versions: 3.0.0
Reporter: Brandon Li
Assignee: Brandon Li
Priority: Trivial
Attachments: MAPREDUCE-4736.patch
Looks like this option is obsolete. Remove it to avoid confusion.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
