[jira] [Commented] (MAPREDUCE-3372) HADOOP_PREFIX cannot be overriden
[ https://issues.apache.org/jira/browse/MAPREDUCE-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152673#comment-13152673 ] Hudson commented on MAPREDUCE-3372: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1308 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1308/]) MAPREDUCE-3372. HADOOP_PREFIX cannot be overridden. Contributed by Bruno Mahé. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203488 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/python/compile * /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/python/pyAbacus/compile * /hadoop/common/trunk/hadoop-mapreduce-project/src/packages/deb/init.d/hadoop-historyserver * /hadoop/common/trunk/hadoop-mapreduce-project/src/packages/rpm/init.d/hadoop-historyserver > HADOOP_PREFIX cannot be overriden > - > > Key: MAPREDUCE-3372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3372 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3372.patch > > > hadoop-config.sh forces HADOOP_prefix to a specific value: > export HADOOP_PREFIX=`dirname "$this"`/.. > It would be nice to make this overridable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3372) HADOOP_PREFIX cannot be overriden
[ https://issues.apache.org/jira/browse/MAPREDUCE-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152675#comment-13152675 ] Hudson commented on MAPREDUCE-3372: --- Integrated in Hadoop-Mapreduce-0.23-Commit #193 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/193/]) Merge -r 1203487:1203488 from trunk to branch-0.23. Fixes: MAPREDUCE-3372 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203489 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples/python/compile * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples/python/pyAbacus/compile * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/packages/deb/init.d/hadoop-historyserver * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/packages/rpm/init.d/hadoop-historyserver > HADOOP_PREFIX cannot be overriden > - > > Key: MAPREDUCE-3372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3372 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3372.patch > > > hadoop-config.sh forces HADOOP_prefix to a specific value: > export HADOOP_PREFIX=`dirname "$this"`/.. > It would be nice to make this overridable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3343) TaskTracker Out of Memory because of distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152672#comment-13152672 ] zhaoyunjiong commented on MAPREDUCE-3343: - Unit test already cover the job is removed from the archives. If we remove the call to removeTaskDistributedCacheManager in the test, the last line of testRemoveTaskDistributedCacheManager will fail. > TaskTracker Out of Memory because of distributed cache > -- > > Key: MAPREDUCE-3343 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3343 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 0.20.205.0 >Reporter: Ahmed Radwan >Assignee: zhaoyunjiong > Labels: mapreduce, patch > Attachments: MAPREDUCE-3343_rev2.patch, > mapreduce-3343-release-0.20.205.0.patch > > > This Out of Memory happens when you run large number of jobs (using the > distributed cache) on a TaskTracker. > Seems the basic issue is with the distributedCacheManager (instance of > TrackerDistributedCacheManager in TaskTracker.java), this gets created during > TaskTracker.initialize(), and it keeps references to > TaskDistributedCacheManager for every submitted job via the jobArchives Map, > also references to CacheStatus via cachedArchives map. I am not seeing these > cleaned up between jobs, so this can out of memory problems after really > large number of jobs are submitted. We have seen this issue in a number of > cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152665#comment-13152665 ] Tom White commented on MAPREDUCE-3169: -- > should MiniMRCluster be deprecated in 23/trunk in favour of > MiniMRClientClusterFactory Yes, I think that's a good idea. > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Fix For: 0.20.205.1, 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3372) HADOOP_PREFIX cannot be overriden
[ https://issues.apache.org/jira/browse/MAPREDUCE-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152658#comment-13152658 ] Hudson commented on MAPREDUCE-3372: --- Integrated in Hadoop-Hdfs-trunk-Commit #1357 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1357/]) MAPREDUCE-3372. HADOOP_PREFIX cannot be overridden. Contributed by Bruno Mahé. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203488 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/python/compile * /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/python/pyAbacus/compile * /hadoop/common/trunk/hadoop-mapreduce-project/src/packages/deb/init.d/hadoop-historyserver * /hadoop/common/trunk/hadoop-mapreduce-project/src/packages/rpm/init.d/hadoop-historyserver > HADOOP_PREFIX cannot be overriden > - > > Key: MAPREDUCE-3372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3372 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3372.patch > > > hadoop-config.sh forces HADOOP_prefix to a specific value: > export HADOOP_PREFIX=`dirname "$this"`/.. > It would be nice to make this overridable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3372) HADOOP_PREFIX cannot be overriden
[ https://issues.apache.org/jira/browse/MAPREDUCE-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152655#comment-13152655 ] Hudson commented on MAPREDUCE-3372: --- Integrated in Hadoop-Common-trunk-Commit #1283 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1283/]) MAPREDUCE-3372. HADOOP_PREFIX cannot be overridden. Contributed by Bruno Mahé. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203488 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/python/compile * /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/python/pyAbacus/compile * /hadoop/common/trunk/hadoop-mapreduce-project/src/packages/deb/init.d/hadoop-historyserver * /hadoop/common/trunk/hadoop-mapreduce-project/src/packages/rpm/init.d/hadoop-historyserver > HADOOP_PREFIX cannot be overriden > - > > Key: MAPREDUCE-3372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3372 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3372.patch > > > hadoop-config.sh forces HADOOP_prefix to a specific value: > export HADOOP_PREFIX=`dirname "$this"`/.. > It would be nice to make this overridable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3372) HADOOP_PREFIX cannot be overriden
[ https://issues.apache.org/jira/browse/MAPREDUCE-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152653#comment-13152653 ] Hudson commented on MAPREDUCE-3372: --- Integrated in Hadoop-Common-0.23-Commit #181 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/181/]) Merge -r 1203487:1203488 from trunk to branch-0.23. Fixes: MAPREDUCE-3372 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203489 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples/python/compile * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples/python/pyAbacus/compile * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/packages/deb/init.d/hadoop-historyserver * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/packages/rpm/init.d/hadoop-historyserver > HADOOP_PREFIX cannot be overriden > - > > Key: MAPREDUCE-3372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3372 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3372.patch > > > hadoop-config.sh forces HADOOP_prefix to a specific value: > export HADOOP_PREFIX=`dirname "$this"`/.. > It would be nice to make this overridable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3372) HADOOP_PREFIX cannot be overriden
[ https://issues.apache.org/jira/browse/MAPREDUCE-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152651#comment-13152651 ] Hudson commented on MAPREDUCE-3372: --- Integrated in Hadoop-Hdfs-0.23-Commit #180 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/180/]) Merge -r 1203487:1203488 from trunk to branch-0.23. Fixes: MAPREDUCE-3372 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203489 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples/python/compile * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples/python/pyAbacus/compile * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/packages/deb/init.d/hadoop-historyserver * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/packages/rpm/init.d/hadoop-historyserver > HADOOP_PREFIX cannot be overriden > - > > Key: MAPREDUCE-3372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3372 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3372.patch > > > hadoop-config.sh forces HADOOP_prefix to a specific value: > export HADOOP_PREFIX=`dirname "$this"`/.. > It would be nice to make this overridable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3372) HADOOP_PREFIX cannot be overriden
[ https://issues.apache.org/jira/browse/MAPREDUCE-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3372: - Resolution: Fixed Fix Version/s: 0.23.1 Assignee: Bruno Mahé Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 I've just committed this. Thanks Bruno! > HADOOP_PREFIX cannot be overriden > - > > Key: MAPREDUCE-3372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3372 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3372.patch > > > hadoop-config.sh forces HADOOP_prefix to a specific value: > export HADOOP_PREFIX=`dirname "$this"`/.. > It would be nice to make this overridable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152644#comment-13152644
]
Alejandro Abdelnur commented on MAPREDUCE-3427:
---
The tests failing are excluded in the streaming POM, once fixed they should be
removed from the exclusion list.
> streaming tests fail with MR2
> -
>
> Key: MAPREDUCE-3427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, mrv2
>Affects Versions: 0.24.0, 0.23.1
>Reporter: Alejandro Abdelnur
>Priority: Blocker
> Fix For: 0.24.0, 0.23.1
>
>
> After Mavenizing streaming and getting its testcases to use the MiniMRCluster
> wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
> Following is an assessment of those failures. Note that the testcases have
> been tweaked only to set the streaming JAR and yarn as the framework.
>
> (If these issues are unrelated we should create sub-tasks for each one of
> them).
> *TestStreamingCombiner*, fails because returned counters don't match
> assertion. However, counters printed in the test output indicate values that
> would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
> passing back counter information to the client API.
> *TestStreamingBadRecords*, the job is failing with the following exception
> {code}
> Application application_1321575850006_0001 failed 1 times due to AM Container
> for
> appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
> .Failing this attempt.. Failing the application.
> {code}
> Difficult to troubleshoot because there are not task logs from Mini MR/YARN
> run.
> *TestStreamingStatus* fails in validateTaskStatus() in the following assertion
> {code}
> expected:<[before consuming input > sort]> but was:<[SUCCEEDED]>
> {code}
> *TestUlimit* fails with
> {code}
> org.junit.ComparisonFailure: output is wrong expected:<[786432]> but
> was:<[unlimited]>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3427) streaming tests fail with MR2
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.24.0, 0.23.1
Reporter: Alejandro Abdelnur
Priority: Blocker
Fix For: 0.24.0, 0.23.1
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have been
tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of them).
*TestStreamingCombiner*, fails because returned counters don't match assertion.
However, counters printed in the test output indicate values that would satisfy
the assertion. As Tom has indicated it seems MR/YARN are not passing back
counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:<[before consuming input > sort]> but was:<[SUCCEEDED]>
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:<[786432]> but
was:<[unlimited]>
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2324) Job should fail if a reduce task can't be scheduled anywhere
[ https://issues.apache.org/jira/browse/MAPREDUCE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152642#comment-13152642 ] Subroto Sanyal commented on MAPREDUCE-2324: --- Hi Todd, Murthy, Robert The issue targets to fix the problem in case of Reducer. As per fix that is committed, I can see the check for *ResourceEstimator.getEstimatedReduceInputSize* is removed from *findNewReduceTask* I have the following questions for the fix committed: * How about the same problem occurring in case of Mappers? * Say for example only one TaskTracker is having low disk space, as per the fix we go ahead and assign the Reduce task to it; which ends up in failure. So one failure which could have been reduced by the check. Regards, Subroto Sanyal > Job should fail if a reduce task can't be scheduled anywhere > > > Key: MAPREDUCE-2324 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2324 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.2, 0.20.205.0 >Reporter: Todd Lipcon >Assignee: Robert Joseph Evans > Fix For: 0.20.205.0 > > Attachments: MR-2324-disable-check-v2.patch, MR-2324-security-v1.txt, > MR-2324-security-v2.txt, MR-2324-security-v3.patch, > MR-2324-secutiry-just-log-v1.patch > > > If there's a reduce task that needs more disk space than is available on any > mapred.local.dir in the cluster, that task will stay pending forever. For > example, we produced this in a QA cluster by accidentally running terasort > with one reducer - since no mapred.local.dir had 1T free, the job remained in > pending state for several days. The reason for the "stuck" task wasn't clear > from a user perspective until we looked at the JT logs. > Probably better to just fail the job if a reduce task goes through all TTs > and finds that there isn't enough space. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3373) Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152588#comment-13152588 ] Hudson commented on MAPREDUCE-3373: --- Integrated in Hadoop-Mapreduce-0.23-Commit #192 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/192/]) Merge -r 1203454:1203455 from trunk to branch-0.23. Fixes: MAPREDUCE-3373 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203458 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/slaves.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/start-all.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/stop-all.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemons.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode-remote.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode-remote.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode.sh > Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh. > - > > Key: MAPREDUCE-3373 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3373.patch > > > It would be nice to be able to specify some other location for > hadoop-config.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3343) TaskTracker Out of Memory because of distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152582#comment-13152582 ] Eli Collins commented on MAPREDUCE-3343: Ping? > TaskTracker Out of Memory because of distributed cache > -- > > Key: MAPREDUCE-3343 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3343 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 0.20.205.0 >Reporter: Ahmed Radwan >Assignee: zhaoyunjiong > Labels: mapreduce, patch > Attachments: MAPREDUCE-3343_rev2.patch, > mapreduce-3343-release-0.20.205.0.patch > > > This Out of Memory happens when you run large number of jobs (using the > distributed cache) on a TaskTracker. > Seems the basic issue is with the distributedCacheManager (instance of > TrackerDistributedCacheManager in TaskTracker.java), this gets created during > TaskTracker.initialize(), and it keeps references to > TaskDistributedCacheManager for every submitted job via the jobArchives Map, > also references to CacheStatus via cachedArchives map. I am not seeing these > cleaned up between jobs, so this can out of memory problems after really > large number of jobs are submitted. We have seen this issue in a number of > cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3373) Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152577#comment-13152577 ] Hudson commented on MAPREDUCE-3373: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1306 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1306/]) MAPREDUCE-3373. Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh. Contributed by Bruno Mahé tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203455 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/slaves.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/start-all.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/stop-all.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemons.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode-remote.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode-remote.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode.sh > Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh. > - > > Key: MAPREDUCE-3373 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3373.patch > > > It would be nice to be able to specify some other location for > hadoop-config.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3373) Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152556#comment-13152556 ] Hudson commented on MAPREDUCE-3373: --- Integrated in Hadoop-Hdfs-0.23-Commit #179 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/179/]) Merge -r 1203454:1203455 from trunk to branch-0.23. Fixes: MAPREDUCE-3373 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203458 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/slaves.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/start-all.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/stop-all.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemons.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode-remote.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode-remote.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode.sh > Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh. > - > > Key: MAPREDUCE-3373 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3373.patch > > > It would be nice to be able to specify some other location for > hadoop-config.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3373) Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152552#comment-13152552 ] Hudson commented on MAPREDUCE-3373: --- Integrated in Hadoop-Common-0.23-Commit #180 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/180/]) Merge -r 1203454:1203455 from trunk to branch-0.23. Fixes: MAPREDUCE-3373 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203458 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/slaves.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/start-all.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/stop-all.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemons.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode-remote.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode-remote.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode.sh > Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh. > - > > Key: MAPREDUCE-3373 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3373.patch > > > It would be nice to be able to specify some other location for > hadoop-config.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3373) Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3373: - Resolution: Fixed Fix Version/s: 0.23.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 I've just committed this. Thanks Bruno! > Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh. > - > > Key: MAPREDUCE-3373 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3373.patch > > > It would be nice to be able to specify some other location for > hadoop-config.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3373) Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3373: - Summary: Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh. (was: hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh) > Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh. > - > > Key: MAPREDUCE-3373 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3373.patch > > > It would be nice to be able to specify some other location for > hadoop-config.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3373) hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh
[ https://issues.apache.org/jira/browse/MAPREDUCE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152542#comment-13152542 ] Hudson commented on MAPREDUCE-3373: --- Integrated in Hadoop-Common-trunk-Commit #1282 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1282/]) MAPREDUCE-3373. Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh. Contributed by Bruno Mahé tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203455 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/slaves.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/start-all.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/stop-all.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemons.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode-remote.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode-remote.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode.sh > hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh > --- > > Key: MAPREDUCE-3373 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3373.patch > > > It would be nice to be able to specify some other location for > hadoop-config.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3373) hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh
[ https://issues.apache.org/jira/browse/MAPREDUCE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152536#comment-13152536 ] Hudson commented on MAPREDUCE-3373: --- Integrated in Hadoop-Hdfs-trunk-Commit #1356 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1356/]) MAPREDUCE-3373. Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh. Contributed by Bruno Mahé tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203455 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/slaves.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/start-all.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/stop-all.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemons.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode-remote.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/start-raidnode.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode-remote.sh * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/bin/stop-raidnode.sh > hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh > --- > > Key: MAPREDUCE-3373 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.23.1 >Reporter: Bruno Mahé >Assignee: Bruno Mahé > Labels: bigtop > Attachments: MAPREDUCE-3373.patch > > > It would be nice to be able to specify some other location for > hadoop-config.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3426) uber-jobs tried to write outputs into wrong dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152513#comment-13152513 ] Hitesh Shah commented on MAPREDUCE-3426: Map task tries to write output into top-level directory instead of user-specific appcache: 2011-11-17 00:16:38,196 WARN [uber-SubtaskRunner] org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext: /grid/0/dev/hrt_mr/tmp/mapred-local is not writable org.apache.hadoop.util.DiskChecker$DiskErrorException: directory is not writable: /grid/0/dev/hrt_mr/tmp/mapred-local at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:99) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:272) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:331) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.mapred.MROutputFiles.getSpillFileForWrite(MROutputFiles.java:147) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1503) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1397) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:649) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:716) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328) at org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:294) at org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:224) at java.lang.Thread.run(Thread.java:662) > uber-jobs tried to write outputs into wrong dir > --- > > Key: MAPREDUCE-3426 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3426 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Hitesh Shah > Fix For: 0.23.1 > > > Incorrect setup of the uber tasks causes tasks to try to write intermidiate > outputs into dirs that the user does not have permissions to write to on a > secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3426) uber-jobs tried to write outputs into wrong dir
uber-jobs tried to write outputs into wrong dir --- Key: MAPREDUCE-3426 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3426 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Fix For: 0.23.1 Incorrect setup of the uber tasks causes tasks to try to write intermidiate outputs into dirs that the user does not have permissions to write to on a secure cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3413) RM web ui applications not sorted in any order by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152508#comment-13152508 ] Jonathan Eagles commented on MAPREDUCE-3413: No new unit tests are needed for the web page display bug. Manual tests include 1) verifying that all web pages with id are sorted by default on that column. 2) visually inspecting the html output to verify jQuery column def are correct as well as aaSorting. > RM web ui applications not sorted in any order by default > - > > Key: MAPREDUCE-3413 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3413 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3413.patch, MAPREDUCE-3413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
[ https://issues.apache.org/jira/browse/MAPREDUCE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152504#comment-13152504 ] Jonathan Eagles commented on MAPREDUCE-3045: No new tests are needed for this change as it only changes html output. Manual tests include testing that tables are sorted correctly and that filter and search terms return correct results for all tables that have an elapsed time column. > Elapsed time filter on jobhistory server displays incorrect table entries > - > > Key: MAPREDUCE-3045 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3045.patch, MAPREDUCE-3045.patch, > MAPREDUCE-3045.patch > > > The elapsed time filter on the jobhistory server filters incorrect > information. > For e.g. on a cluster where the elapsed time of all the tasks is either 7 or > 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3419) Don't mark exited TT threads as dead in MiniMRCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved MAPREDUCE-3419. Resolution: Fixed Hadoop Flags: Reviewed All tests ran cleanly except for one, which is unrelated, filed HADOOP-7836 for it. I've committed this. > Don't mark exited TT threads as dead in MiniMRCluster > --- > > Key: MAPREDUCE-3419 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3419 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker, test >Affects Versions: 0.20.206.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: mapreduce-2850-1.patch > > > MAPREDUCE-2850 flagged all TT threads that exited in the MiniMRCluster as > dead, this breaks a number of the other tests that use MiniMRCluster across > restart. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-3419) Don't mark exited TT threads as dead in MiniMRCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-3419 started by Eli Collins. > Don't mark exited TT threads as dead in MiniMRCluster > --- > > Key: MAPREDUCE-3419 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3419 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker, test >Affects Versions: 0.20.206.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: mapreduce-2850-1.patch > > > MAPREDUCE-2850 flagged all TT threads that exited in the MiniMRCluster as > dead, this breaks a number of the other tests that use MiniMRCluster across > restart. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152478#comment-13152478 ] Hudson commented on MAPREDUCE-3415: --- Integrated in Hadoop-Mapreduce-0.23-Commit #190 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/190/]) Merge -r 1203410:1203411 from trunk to branch. FIXES: MAPREDUCE-3415 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203415 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/JarFinder.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestJarFinder.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3251) Network ACLs can prevent some clients to talk to MR ApplicationMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152474#comment-13152474 ] Mahadev konar commented on MAPREDUCE-3251: -- @Alejandro, With HDFS, the datanode ports acls are usually openend up to allow folks writing to HDFS from outside the cluster. The datanode ports are fixed and is easy to open acls to that but the issue here is that AM comes up on a random port and its not possible to open acls because of that. Hope that answers the questions. > Network ACLs can prevent some clients to talk to MR ApplicationMaster > - > > Key: MAPREDUCE-3251 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3251 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Anupam Seth >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3251_branch-0_23_preliminary.txt > > > In 0.20.xxx, the JobClient while polling goes to JT to get the job status. > With YARN, AM can be launched on any port and the client will have to have > ACL open to that port to talk to AM and get the job status. When the client > is within the same grid network access to AM is not a problem. But some > applications may have one installation per set of clusters and may launch > jobs even across such sets (on job trackers in another set of clusters). For > that to work only the JT port needs to be open currently. In case of YARN, > all ports will have to be opened up for things to work. That would be a > security no-no. > There are two possible solutions: > 1) Make the job client only talk to RM (as an option) to get the job > status. > 2) Limit the range of ports AM can listen on. > Option 2) may not be favorable as there is no direct OS API to find a free > port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3251) Network ACLs can prevent some clients to talk to MR ApplicationMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152473#comment-13152473 ] Anupam Seth commented on MAPREDUCE-3251: bq. Unless I'm missing something here, a client not only talk with MR side, often it does HDFS operations; and for this it needs network access to all cluster nodes. Unless you force them to use something like Hoop. Also, in MR2 land, where are splits being calculated?, this requires HDFS access. I probably do not just have any depth of understanding here, but will access to HDFS not be an issue for the client in the kind of scenario this is attempting to address since what is blocking is the port ACL not being opened up? > Network ACLs can prevent some clients to talk to MR ApplicationMaster > - > > Key: MAPREDUCE-3251 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3251 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Anupam Seth >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3251_branch-0_23_preliminary.txt > > > In 0.20.xxx, the JobClient while polling goes to JT to get the job status. > With YARN, AM can be launched on any port and the client will have to have > ACL open to that port to talk to AM and get the job status. When the client > is within the same grid network access to AM is not a problem. But some > applications may have one installation per set of clusters and may launch > jobs even across such sets (on job trackers in another set of clusters). For > that to work only the JT port needs to be open currently. In case of YARN, > all ports will have to be opened up for things to work. That would be a > security no-no. > There are two possible solutions: > 1) Make the job client only talk to RM (as an option) to get the job > status. > 2) Limit the range of ports AM can listen on. > Option 2) may not be favorable as there is no direct OS API to find a free > port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3251) Network ACLs can prevent some clients to talk to MR ApplicationMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-3251: --- Attachment: MAPREDUCE-3251_branch-0_23_preliminary.txt Attaching a preliminary patch to gauge if I am headed in the right space. Feedback will be welcomed. > Network ACLs can prevent some clients to talk to MR ApplicationMaster > - > > Key: MAPREDUCE-3251 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3251 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Anupam Seth >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3251_branch-0_23_preliminary.txt > > > In 0.20.xxx, the JobClient while polling goes to JT to get the job status. > With YARN, AM can be launched on any port and the client will have to have > ACL open to that port to talk to AM and get the job status. When the client > is within the same grid network access to AM is not a problem. But some > applications may have one installation per set of clusters and may launch > jobs even across such sets (on job trackers in another set of clusters). For > that to work only the JT port needs to be open currently. In case of YARN, > all ports will have to be opened up for things to work. That would be a > security no-no. > There are two possible solutions: > 1) Make the job client only talk to RM (as an option) to get the job > status. > 2) Limit the range of ports AM can listen on. > Option 2) may not be favorable as there is no direct OS API to find a free > port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152463#comment-13152463 ] Hudson commented on MAPREDUCE-3415: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1304 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1304/]) MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR resolution (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203411 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/JarFinder.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestJarFinder.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152446#comment-13152446 ] Hudson commented on MAPREDUCE-3415: --- Integrated in Hadoop-Hdfs-0.23-Commit #177 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/177/]) Merge -r 1203410:1203411 from trunk to branch. FIXES: MAPREDUCE-3415 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203415 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/JarFinder.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestJarFinder.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152447#comment-13152447 ] Hudson commented on MAPREDUCE-3415: --- Integrated in Hadoop-Common-0.23-Commit #178 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/178/]) Merge -r 1203410:1203411 from trunk to branch. FIXES: MAPREDUCE-3415 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203415 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/JarFinder.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestJarFinder.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3415: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and 0.23 > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3169: -- Resolution: Fixed Fix Version/s: 0.23.1 0.24.0 0.20.205.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk, 0.23 & 0.20.205 branches. Thanks Ahmed. > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Fix For: 0.20.205.1, 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152440#comment-13152440 ] Hudson commented on MAPREDUCE-3169: --- Integrated in Hadoop-Mapreduce-0.23-Commit #189 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/189/]) Merge -r 1203370:1203371 from trunk to branch. FIXES: MAPREDUCE-3169 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203402 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152430#comment-13152430 ] Hudson commented on MAPREDUCE-3415: --- Integrated in Hadoop-Common-trunk-Commit #1280 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1280/]) MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR resolution (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203411 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/JarFinder.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestJarFinder.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152428#comment-13152428 ] Hudson commented on MAPREDUCE-3415: --- Integrated in Hadoop-Hdfs-trunk-Commit #1354 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1354/]) MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR resolution (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203411 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/JarFinder.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestJarFinder.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152422#comment-13152422 ] Hudson commented on MAPREDUCE-3169: --- Integrated in Hadoop-Common-0.23-Commit #177 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/177/]) Merge -r 1203370:1203371 from trunk to branch. FIXES: MAPREDUCE-3169 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203402 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152421#comment-13152421 ] Hudson commented on MAPREDUCE-3169: --- Integrated in Hadoop-Hdfs-0.23-Commit #176 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/176/]) Merge -r 1203370:1203371 from trunk to branch. FIXES: MAPREDUCE-3169 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203402 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152414#comment-13152414 ] Alejandro Abdelnur commented on MAPREDUCE-3415: --- @Tom, Jenkins failure is because testcases are run at MR level and they are picking up a prev version of common JAR from M2 repo, not using the changes in the this patch. after the commit things would go back to normal > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3425) uber job hangs and fails to get killed properly on a job kill signal
uber job hangs and fails to get killed properly on a job kill signal Key: MAPREDUCE-3425 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3425 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0, 0.23.1 Reporter: Hitesh Shah Fix For: 0.24.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
[ https://issues.apache.org/jira/browse/MAPREDUCE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152407#comment-13152407 ] Hadoop QA commented on MAPREDUCE-3045: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504123/MAPREDUCE-3045.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1320//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1320//console This message is automatically generated. > Elapsed time filter on jobhistory server displays incorrect table entries > - > > Key: MAPREDUCE-3045 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3045.patch, MAPREDUCE-3045.patch, > MAPREDUCE-3045.patch > > > The elapsed time filter on the jobhistory server filters incorrect > information. > For e.g. on a cluster where the elapsed time of all the tasks is either 7 or > 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3369) Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169
[ https://issues.apache.org/jira/browse/MAPREDUCE-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152406#comment-13152406 ] Tom White commented on MAPREDUCE-3369: -- Ahmed, thanks for the updated patches. I think the tools and examples are being handled in HADOOP-7590 so they don't need to be in this patch. Also, we need to get a clear jenkins test run. (I guess this is waiting for MAPREDUCE-3169?) > Migrate MR1 tests to run on MR2 using the new interfaces introduced in > MAPREDUCE-3169 > - > > Key: MAPREDUCE-3369 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3369 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3369.patch, MAPREDUCE-3369_rev2_part1.patch, > MAPREDUCE-3369_rev2_part2.patch, mv-cp.sh > > > This ticket tracks the migration of MR1 tests (currently residing in > "hadoop-mapreduce-project/src/test/") to run on MR2. The migration is using > the new interfaces introduced in MAPREDUCE-3169. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
[ https://issues.apache.org/jira/browse/MAPREDUCE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3045: --- Target Version/s: 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0) Status: Patch Available (was: Open) > Elapsed time filter on jobhistory server displays incorrect table entries > - > > Key: MAPREDUCE-3045 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3045.patch, MAPREDUCE-3045.patch, > MAPREDUCE-3045.patch > > > The elapsed time filter on the jobhistory server filters incorrect > information. > For e.g. on a cluster where the elapsed time of all the tasks is either 7 or > 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
[ https://issues.apache.org/jira/browse/MAPREDUCE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3045: --- Attachment: MAPREDUCE-3045.patch > Elapsed time filter on jobhistory server displays incorrect table entries > - > > Key: MAPREDUCE-3045 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3045.patch, MAPREDUCE-3045.patch, > MAPREDUCE-3045.patch > > > The elapsed time filter on the jobhistory server filters incorrect > information. > For e.g. on a cluster where the elapsed time of all the tasks is either 7 or > 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152369#comment-13152369 ] Hudson commented on MAPREDUCE-3169: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1303 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1303/]) MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2. (Ahmed via tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203371 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles reassigned MAPREDUCE-3360: -- Assignee: Jonathan Eagles > Provide information about lost nodes in the UI. > --- > > Key: MAPREDUCE-3360 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 > Environment: NA >Reporter: Bhallamudi Venkata Siva Kamesh >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: lostNodes.png > > > Currently there is no information provided about *lost nodes*. Provide > information in the UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152361#comment-13152361 ] Siddharth Seth commented on MAPREDUCE-3169: --- As part of this, should MiniMRCluster be deprecated in 23/trunk in favour of MiniMRClientClusterFactory, or an additional note in MiniMRCluster pointing people to MiniMRClientClusterFactory. > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152340#comment-13152340 ] Hudson commented on MAPREDUCE-3169: --- Integrated in Hadoop-Hdfs-trunk-Commit #1353 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1353/]) MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2. (Ahmed via tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203371 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152338#comment-13152338 ] Hudson commented on MAPREDUCE-3169: --- Integrated in Hadoop-Common-trunk-Commit #1279 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1279/]) MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2. (Ahmed via tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203371 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152315#comment-13152315 ] Tom White commented on MAPREDUCE-3169: -- > Should we call it something other than MiniMRCluster to prevent confusion? Yes. The patch for 0.20 does not change MiniMRCluster, and for 23/trunk it provides an implementation of MiniMRCluster for the operations that are possible when running with YARN (the jobtracker/tasktracker-specific methods will throw UnsupportedOperationException). In addition, a new interface called MiniMRClientCluster is introduced (in 20/23/trunk) that downstream projects can start using. > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152313#comment-13152313 ] Ahmed Radwan commented on MAPREDUCE-3415: - +1, thanks Tucu! > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152309#comment-13152309 ] Ahmed Radwan commented on MAPREDUCE-3169: - I agree, this is how old tests can work with almost none or very minor changes. Thanks Arun & Tucu! > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152305#comment-13152305 ] Alejandro Abdelnur commented on MAPREDUCE-3169: --- MiniMRCluster is on purpose to allow downstream projects to continue using the same API for testing for all their legacy testcases with none/minimal disruption. For new stuff folks should use MiniMRClientCluster via MiniMRClientClusterFactory. I'm planning to commit this now, are we OK? > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152299#comment-13152299 ] Arun C Murthy commented on MAPREDUCE-3169: -- Should we call it something other than MiniMRCluster to prevent confusion? > Create a new MiniMRCluster equivalent which only provides client APIs cross > MR1 and MR2 > --- > > Key: MAPREDUCE-3169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3169-0.20-security.patch, > MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, > MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch > > > Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster > for writing tests. Many users do as well. MiniMRCluster, however, exposes MR > implementation details like the existence of TaskTrackers, JobTrackers, etc, > since it was used by MR1 for testing the server implementations as well. > This JIRA is to create a new interface which could be implemented either by > MR1 or MR2 that exposes only the client-side portions of the MR framework. > Ideally it would be "recompile-compatible" with MiniMRCluster for most > applications, and the MR1 implementation could be backported to 20x branch. > Thus, dependent projects like HBase could migrate to this implementation and > test against both MR1 and MR2. We can also use this to port over the current > functional tests that use only the client-side features of MiniMRCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152283#comment-13152283 ] Tom White commented on MAPREDUCE-3415: -- +1 (pending investigating the jenkins warnings) > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3424) MR-2415 cleanup
[ https://issues.apache.org/jira/browse/MAPREDUCE-3424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned MAPREDUCE-3424: -- Assignee: Eli Collins > MR-2415 cleanup > --- > > Key: MAPREDUCE-3424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Affects Versions: 0.20.205.0 >Reporter: Eli Collins >Assignee: Eli Collins >Priority: Minor > Attachments: mapreduce-3424-1.patch > > > MR-2415 had some tabs and weird indenting and spacing. Also would be more > clear if LTC explicitly overrides createLogDir. Let's clean this up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3424) MR-2415 cleanup
[ https://issues.apache.org/jira/browse/MAPREDUCE-3424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-3424: --- Attachment: mapreduce-3424-1.patch Patch attached. > MR-2415 cleanup > --- > > Key: MAPREDUCE-3424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Affects Versions: 0.20.205.0 >Reporter: Eli Collins >Priority: Minor > Attachments: mapreduce-3424-1.patch > > > MR-2415 had some tabs and weird indenting and spacing. Also would be more > clear if LTC explicitly overrides createLogDir. Let's clean this up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3424) MR-2415 cleanup
MR-2415 cleanup --- Key: MAPREDUCE-3424 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3424 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: tasktracker Affects Versions: 0.20.205.0 Reporter: Eli Collins Priority: Minor Attachments: mapreduce-3424-1.patch MR-2415 had some tabs and weird indenting and spacing. Also would be more clear if LTC explicitly overrides createLogDir. Let's clean this up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3413) RM web ui applications not sorted in any order by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152260#comment-13152260 ] Hadoop QA commented on MAPREDUCE-3413: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504098/MAPREDUCE-3413.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1319//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1319//console This message is automatically generated. > RM web ui applications not sorted in any order by default > - > > Key: MAPREDUCE-3413 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3413 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3413.patch, MAPREDUCE-3413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152239#comment-13152239 ] Hadoop QA commented on MAPREDUCE-3415: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504097/MAPREDUCE-3415.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1318//console This message is automatically generated. > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3413) RM web ui applications not sorted in any order by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3413: --- Attachment: MAPREDUCE-3413.patch > RM web ui applications not sorted in any order by default > - > > Key: MAPREDUCE-3413 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3413 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3413.patch, MAPREDUCE-3413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3423) Fetcher Thread should log proper exception
[
https://issues.apache.org/jira/browse/MAPREDUCE-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152234#comment-13152234
]
Hadoop QA commented on MAPREDUCE-3423:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12504087/MAPREDUCE-3423.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1317//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1317//console
This message is automatically generated.
> Fetcher Thread should log proper exception
> --
>
> Key: MAPREDUCE-3423
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3423
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: Bhallamudi Venkata Siva Kamesh
> Attachments: MAPREDUCE-3423.patch
>
>
> Exception should be logged properly when there is any while deserializing
> Shuffle Header information .
> {code}
> if (mapId == null || mapOutput == null) {
> LOG.info("fetcher#" + id + " failed to read map header" +
> mapId + " decomp: " +
> decompressedLength + ", " + compressedLength, ioe);
> return false;
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3413) RM web ui applications not sorted in any order by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3413: --- Status: Patch Available (was: Open) > RM web ui applications not sorted in any order by default > - > > Key: MAPREDUCE-3413 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3413 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3413.patch, MAPREDUCE-3413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3415: -- Attachment: MAPREDUCE-3415.patch as per Ahmed suggestion, not using temp dir anymore, instead using a dir within 'target', thus no need to do an explicit clean up. Had to tweak the MiniMRClusterFactory to use JarFinder as it was not working when running test cases from root level because in that case there is no JAR for for the caller class. Also, this enables running the testcases from within IDEs (for the very same reason) > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch, MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3419) Don't mark exited TT threads as dead in MiniMRCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152214#comment-13152214 ] Eli Collins commented on MAPREDUCE-3419: Thanks Ravi. Yea, this fixes the other tets. I'll re-run the whole suite on branch-20-security for sanity before committing. > Don't mark exited TT threads as dead in MiniMRCluster > --- > > Key: MAPREDUCE-3419 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3419 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker, test >Affects Versions: 0.20.206.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: mapreduce-2850-1.patch > > > MAPREDUCE-2850 flagged all TT threads that exited in the MiniMRCluster as > dead, this breaks a number of the other tests that use MiniMRCluster across > restart. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated MAPREDUCE-3366:
---
Status: Open (was: Patch Available)
> Mapreduce component should use consistent directory structure layout as
> HDFS/common
> ---
>
> Key: MAPREDUCE-3366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
> Environment: Java, Maven
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: MAPREDUCE-3366.patch
>
>
> Directory structure for MRv2 layout looks like:
> {noformat}
> hadoop-mapreduce-0.23.0-SNAPSHOT/bin
> /conf
> /lib
> /modules
> {noformat}
> The directory structure layout should be updated to reflect changes
> implemented in HADOOP-6255.
> {noformat}
> hadoop-mapreduce-0.23.0-SNAPSHOT/bin
> /etc/hadoop
> /lib
> /libexec
> /sbin
> /share/hadoop
> /share/hadoop/lib
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3366) Mapreduce component should use consistent directory structure layout as HDFS/common
[
https://issues.apache.org/jira/browse/MAPREDUCE-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152200#comment-13152200
]
Hitesh Shah commented on MAPREDUCE-3366:
For MR jobs to work properly with the above change, APPLICATION_CLASSPATH in
ApplicationConstants.java will also likely need to be changed.
> Mapreduce component should use consistent directory structure layout as
> HDFS/common
> ---
>
> Key: MAPREDUCE-3366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3366
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
> Environment: Java, Maven
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: MAPREDUCE-3366.patch
>
>
> Directory structure for MRv2 layout looks like:
> {noformat}
> hadoop-mapreduce-0.23.0-SNAPSHOT/bin
> /conf
> /lib
> /modules
> {noformat}
> The directory structure layout should be updated to reflect changes
> implemented in HADOOP-6255.
> {noformat}
> hadoop-mapreduce-0.23.0-SNAPSHOT/bin
> /etc/hadoop
> /lib
> /libexec
> /sbin
> /share/hadoop
> /share/hadoop/lib
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3423) Fetcher Thread should log proper exception
[
https://issues.apache.org/jira/browse/MAPREDUCE-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-3423:
--
Attachment: MAPREDUCE-3423.patch
Attaching a patch with proper logging
> Fetcher Thread should log proper exception
> --
>
> Key: MAPREDUCE-3423
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3423
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: Bhallamudi Venkata Siva Kamesh
> Attachments: MAPREDUCE-3423.patch
>
>
> Exception should be logged properly when there is any while deserializing
> Shuffle Header information .
> {code}
> if (mapId == null || mapOutput == null) {
> LOG.info("fetcher#" + id + " failed to read map header" +
> mapId + " decomp: " +
> decompressedLength + ", " + compressedLength, ioe);
> return false;
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3423) Fetcher Thread should log proper exception
[
https://issues.apache.org/jira/browse/MAPREDUCE-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-3423:
--
Status: Patch Available (was: Open)
> Fetcher Thread should log proper exception
> --
>
> Key: MAPREDUCE-3423
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3423
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: Bhallamudi Venkata Siva Kamesh
> Attachments: MAPREDUCE-3423.patch
>
>
> Exception should be logged properly when there is any while deserializing
> Shuffle Header information .
> {code}
> if (mapId == null || mapOutput == null) {
> LOG.info("fetcher#" + id + " failed to read map header" +
> mapId + " decomp: " +
> decompressedLength + ", " + compressedLength, ioe);
> return false;
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3423) Fetcher Thread should log proper exception
Fetcher Thread should log proper exception
--
Key: MAPREDUCE-3423
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3423
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.24.0
Reporter: Bhallamudi Venkata Siva Kamesh
Exception should be logged properly when there is any while deserializing
Shuffle Header information .
{code}
if (mapId == null || mapOutput == null) {
LOG.info("fetcher#" + id + " failed to read map header" +
mapId + " decomp: " +
decompressedLength + ", " + compressedLength, ioe);
return false;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3121) NodeManager should handle disk-failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152178#comment-13152178 ] Hitesh Shah commented on MAPREDUCE-3121: @Vinod, for failed disks, did you mean to say -101 and not -1001? -1000 denotes invalid container status whereas -100 denotes container aborted. > NodeManager should handle disk-failures > --- > > Key: MAPREDUCE-3121 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3121 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.23.1 > > Attachments: 3121.patch, 3121.v1.1.patch, 3121.v1.patch, 3121.v2.patch > > > This is akin to MAPREDUCE-2413 but for YARN's NodeManager. We want to > minimize the impact of transient/permanent disk failures on containers. With > larger number of disks per node, the ability to continue to run containers on > other disks is crucial. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2793) [MR-279] Maintain consistency in naming appIDs, jobIDs and attemptIDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne reassigned MAPREDUCE-2793: - Assignee: (was: Eric Payne) > [MR-279] Maintain consistency in naming appIDs, jobIDs and attemptIDs > -- > > Key: MAPREDUCE-2793 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2793 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Priority: Blocker > Fix For: 0.24.0 > > > appIDs, jobIDs and attempt/container ids are not consistently named in the > logs, console and UI. For consistency purpose, they all have to follow a > common naming convention. > Currently, > For appID > = > On the RM UI: app_1308259676864_5 > On the JHS UI: No appID > Console/logs: No appID > mapred-local dirs are named as: application_1308259676864_0005 > For jobID > = > On the RM UI: job_1308259676864_5_5 > JHS UI: job_1308259676864_5_5 > Console/logs: job_1308259676864_0005 > mapred-local dirs are named as: No jobID > For attemptID > > On the RM UI: attempt_1308259676864_5_5_m_24_0 > JHS attempt_1308259676864_5_5_m_24_0 > Console/logs: attempt_1308259676864_0005_m_24_0 > mapred-local dirs are named as: container_1308259676864_0005_24 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2793) [MR-279] Maintain consistency in naming appIDs, jobIDs and attemptIDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne reassigned MAPREDUCE-2793: - Assignee: Eric Payne > [MR-279] Maintain consistency in naming appIDs, jobIDs and attemptIDs > -- > > Key: MAPREDUCE-2793 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2793 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne >Priority: Blocker > Fix For: 0.24.0 > > > appIDs, jobIDs and attempt/container ids are not consistently named in the > logs, console and UI. For consistency purpose, they all have to follow a > common naming convention. > Currently, > For appID > = > On the RM UI: app_1308259676864_5 > On the JHS UI: No appID > Console/logs: No appID > mapred-local dirs are named as: application_1308259676864_0005 > For jobID > = > On the RM UI: job_1308259676864_5_5 > JHS UI: job_1308259676864_5_5 > Console/logs: job_1308259676864_0005 > mapred-local dirs are named as: No jobID > For attemptID > > On the RM UI: attempt_1308259676864_5_5_m_24_0 > JHS attempt_1308259676864_5_5_m_24_0 > Console/logs: attempt_1308259676864_0005_m_24_0 > mapred-local dirs are named as: container_1308259676864_0005_24 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
[ https://issues.apache.org/jira/browse/MAPREDUCE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152117#comment-13152117 ] Hadoop QA commented on MAPREDUCE-3045: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504073/MAPREDUCE-3045.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1316//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1316//console This message is automatically generated. > Elapsed time filter on jobhistory server displays incorrect table entries > - > > Key: MAPREDUCE-3045 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3045.patch, MAPREDUCE-3045.patch > > > The elapsed time filter on the jobhistory server filters incorrect > information. > For e.g. on a cluster where the elapsed time of all the tasks is either 7 or > 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
[ https://issues.apache.org/jira/browse/MAPREDUCE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152112#comment-13152112 ] Jonathan Eagles commented on MAPREDUCE-3045: This issue exists on many of the web pages that display elapsed time > Elapsed time filter on jobhistory server displays incorrect table entries > - > > Key: MAPREDUCE-3045 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3045.patch, MAPREDUCE-3045.patch > > > The elapsed time filter on the jobhistory server filters incorrect > information. > For e.g. on a cluster where the elapsed time of all the tasks is either 7 or > 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
[ https://issues.apache.org/jira/browse/MAPREDUCE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3045: --- Target Version/s: 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0) Status: Open (was: Patch Available) > Elapsed time filter on jobhistory server displays incorrect table entries > - > > Key: MAPREDUCE-3045 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3045.patch, MAPREDUCE-3045.patch > > > The elapsed time filter on the jobhistory server filters incorrect > information. > For e.g. on a cluster where the elapsed time of all the tasks is either 7 or > 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
[ https://issues.apache.org/jira/browse/MAPREDUCE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3045: --- Fix Version/s: (was: 0.23.1) Target Version/s: 0.24.0, 0.23.1 Status: Patch Available (was: Open) > Elapsed time filter on jobhistory server displays incorrect table entries > - > > Key: MAPREDUCE-3045 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3045.patch, MAPREDUCE-3045.patch > > > The elapsed time filter on the jobhistory server filters incorrect > information. > For e.g. on a cluster where the elapsed time of all the tasks is either 7 or > 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
[ https://issues.apache.org/jira/browse/MAPREDUCE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3045: --- Attachment: MAPREDUCE-3045.patch > Elapsed time filter on jobhistory server displays incorrect table entries > - > > Key: MAPREDUCE-3045 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Minor > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3045.patch, MAPREDUCE-3045.patch > > > The elapsed time filter on the jobhistory server filters incorrect > information. > For e.g. on a cluster where the elapsed time of all the tasks is either 7 or > 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3413) RM web ui applications not sorted in any order by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152097#comment-13152097 ] Jonathan Eagles commented on MAPREDUCE-3413: Noticed this problem in a few more places > RM web ui applications not sorted in any order by default > - > > Key: MAPREDUCE-3413 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3413 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3413) RM web ui applications not sorted in any order by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3413: --- Status: Open (was: Patch Available) > RM web ui applications not sorted in any order by default > - > > Key: MAPREDUCE-3413 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3413 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-3413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3311) Bump jetty to 6.1.26
[ https://issues.apache.org/jira/browse/MAPREDUCE-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152055#comment-13152055 ] Hudson commented on MAPREDUCE-3311: --- Integrated in Hadoop-Mapreduce-22-branch #89 (See [https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/89/]) MAPREDUCE-3311. Bump jetty to 6.1.26. Controbuted by Konstantin Boudnik cos : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202990 Files : * /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt * /hadoop/common/branches/branch-0.22/mapreduce/ivy/libraries.properties * /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/ivy.xml * /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/dynamic-scheduler/ivy.xml * /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/fairscheduler/ivy.xml * /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/gridmix/ivy.xml * /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/mumak/ivy.xml * /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/raid/ivy.xml * /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/streaming/ivy.xml > Bump jetty to 6.1.26 > > > Key: MAPREDUCE-3311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3311 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 0.22.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Fix For: 0.22.0 > > Attachments: MAPREDUCE-3311.patch, MAPREDUCE-3311.patch, > MAPREDUCE-3311.patch, MAPREDUCE-3311.patch, MAPREDUCE-3311.patch > > > MapReduce part of HADOOP-7450 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3407) Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152049#comment-13152049 ] Hudson commented on MAPREDUCE-3407: --- Integrated in Hadoop-Hdfs-0.23-Build #79 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/79/]) MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed by the integration tests. Contributed by Hitesh Shah. svn merge -c r1202766 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202768 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml > Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster > > > Key: MAPREDUCE-3407 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3407 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Fix For: 0.23.1 > > Attachments: MR-3407.1.patch > > > pom for mapreduce-client-jobclient sets system property to incorrect jar > name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3355) AM scheduling hangs frequently with sort job on 350 nodes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152048#comment-13152048
]
Hudson commented on MAPREDUCE-3355:
---
Integrated in Hadoop-Hdfs-0.23-Build #79 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/79/])
MAPREDUCE-3355. Fixed MR AM's ContainerLauncher to handle node-command
timeouts correctly. (vinodkv)
svn merge -c r1202744 --ignore-ancestry ../../trunk/
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202745
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestContainerLauncher.java
> AM scheduling hangs frequently with sort job on 350 nodes
> -
>
> Key: MAPREDUCE-3355
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3355
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3355-2009.1.txt,
> MAPREDUCE-3355-2009.txt, MAPREDUCE-3355-2015.txt, MR3355.txt
>
>
> Another collaboration with [~karams]. Sort job hangs not so rarely on a 350
> node cluster. Found this in AM logs:
> {code}
> Exception in thread "ContainerLauncher #60"
> org.apache.hadoop.yarn.YarnException: java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:170)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:168)
> ... 4 more
> Exception in thread "ContainerLauncher #53"
> org.apache.hadoop.yarn.YarnException: java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:170)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.sendContainerLaunchFailedMsg(ContainerLauncherImpl.java:405)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:330)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:168)
> ... 5 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3355) AM scheduling hangs frequently with sort job on 350 nodes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152037#comment-13152037
]
Hudson commented on MAPREDUCE-3355:
---
Integrated in Hadoop-Mapreduce-trunk #900 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/900/])
MAPREDUCE-3355. Fixed MR AM's ContainerLauncher to handle node-command
timeouts correctly. (vinodkv)
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202744
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestContainerLauncher.java
> AM scheduling hangs frequently with sort job on 350 nodes
> -
>
> Key: MAPREDUCE-3355
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3355
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3355-2009.1.txt,
> MAPREDUCE-3355-2009.txt, MAPREDUCE-3355-2015.txt, MR3355.txt
>
>
> Another collaboration with [~karams]. Sort job hangs not so rarely on a 350
> node cluster. Found this in AM logs:
> {code}
> Exception in thread "ContainerLauncher #60"
> org.apache.hadoop.yarn.YarnException: java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:170)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:168)
> ... 4 more
> Exception in thread "ContainerLauncher #53"
> org.apache.hadoop.yarn.YarnException: java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:170)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.sendContainerLaunchFailedMsg(ContainerLauncherImpl.java:405)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:330)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:168)
> ... 5 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3407) Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152038#comment-13152038 ] Hudson commented on MAPREDUCE-3407: --- Integrated in Hadoop-Mapreduce-trunk #900 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/900/]) MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed by the integration tests. Contributed by Hitesh Shah. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202766 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml > Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster > > > Key: MAPREDUCE-3407 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3407 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Fix For: 0.23.1 > > Attachments: MR-3407.1.patch > > > pom for mapreduce-client-jobclient sets system property to incorrect jar > name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3355) AM scheduling hangs frequently with sort job on 350 nodes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152032#comment-13152032
]
Hudson commented on MAPREDUCE-3355:
---
Integrated in Hadoop-Mapreduce-0.23-Build #96 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/96/])
MAPREDUCE-3355. Fixed MR AM's ContainerLauncher to handle node-command
timeouts correctly. (vinodkv)
svn merge -c r1202744 --ignore-ancestry ../../trunk/
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202745
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestContainerLauncher.java
> AM scheduling hangs frequently with sort job on 350 nodes
> -
>
> Key: MAPREDUCE-3355
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3355
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3355-2009.1.txt,
> MAPREDUCE-3355-2009.txt, MAPREDUCE-3355-2015.txt, MR3355.txt
>
>
> Another collaboration with [~karams]. Sort job hangs not so rarely on a 350
> node cluster. Found this in AM logs:
> {code}
> Exception in thread "ContainerLauncher #60"
> org.apache.hadoop.yarn.YarnException: java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:170)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:168)
> ... 4 more
> Exception in thread "ContainerLauncher #53"
> org.apache.hadoop.yarn.YarnException: java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:170)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.sendContainerLaunchFailedMsg(ContainerLauncherImpl.java:405)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:330)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:168)
> ... 5 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3407) Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152033#comment-13152033 ] Hudson commented on MAPREDUCE-3407: --- Integrated in Hadoop-Mapreduce-0.23-Build #96 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/96/]) MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed by the integration tests. Contributed by Hitesh Shah. svn merge -c r1202766 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202768 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml > Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster > > > Key: MAPREDUCE-3407 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3407 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Fix For: 0.23.1 > > Attachments: MR-3407.1.patch > > > pom for mapreduce-client-jobclient sets system property to incorrect jar > name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3355) AM scheduling hangs frequently with sort job on 350 nodes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152025#comment-13152025
]
Hudson commented on MAPREDUCE-3355:
---
Integrated in Hadoop-Hdfs-trunk #866 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/866/])
MAPREDUCE-3355. Fixed MR AM's ContainerLauncher to handle node-command
timeouts correctly. (vinodkv)
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202744
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestContainerLauncher.java
> AM scheduling hangs frequently with sort job on 350 nodes
> -
>
> Key: MAPREDUCE-3355
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3355
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3355-2009.1.txt,
> MAPREDUCE-3355-2009.txt, MAPREDUCE-3355-2015.txt, MR3355.txt
>
>
> Another collaboration with [~karams]. Sort job hangs not so rarely on a 350
> node cluster. Found this in AM logs:
> {code}
> Exception in thread "ContainerLauncher #60"
> org.apache.hadoop.yarn.YarnException: java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:170)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:168)
> ... 4 more
> Exception in thread "ContainerLauncher #53"
> org.apache.hadoop.yarn.YarnException: java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:170)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.sendContainerLaunchFailedMsg(ContainerLauncherImpl.java:405)
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:330)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:168)
> ... 5 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3407) Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152026#comment-13152026 ] Hudson commented on MAPREDUCE-3407: --- Integrated in Hadoop-Hdfs-trunk #866 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/866/]) MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed by the integration tests. Contributed by Hitesh Shah. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1202766 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml > Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster > > > Key: MAPREDUCE-3407 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3407 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Fix For: 0.23.1 > > Attachments: MR-3407.1.patch > > > pom for mapreduce-client-jobclient sets system property to incorrect jar > name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3415) improve MiniMRYarnCluster & DistributedShell JAR resolution
[ https://issues.apache.org/jira/browse/MAPREDUCE-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152000#comment-13152000 ] Ahmed Radwan commented on MAPREDUCE-3415: - Thanks Tucu! I have tested the patch and it works great. +1 I have a comment regrading clean up of created jar files. The files are now created in the temporary system directory, and the reliance will be on the default system clean up (e.g. on reboot). I think this may cause problems on test dedicated machines that will then require periodic cleanup of the tmp dir. After few minutes of testing I have a dozen of jar files sitting in my tmp dir. I think it may be better to create these jars in a maven predefined directory, that can be cleaned up through maven at the end. What do you think? > improve MiniMRYarnCluster & DistributedShell JAR resolution > > > Key: MAPREDUCE-3415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3415 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.24.0, 0.23.1 > > Attachments: MAPREDUCE-3415.patch, MAPREDUCE-3415.patch, > MAPREDUCE-3415.patch > > > Current JAR resolution assumes the following: > # The class used for JAR lookup is effectively in a JAR > # A System property is set for testing with the location of the JAR > The problem with #1 is that in some cases (when using the class in the same > Maven module where the class is, the class is not in a JAR but in a directory > 'target/test-classes'). > The problem with #2 is the JAR does not exists at the time of running the > test (packaging comes after test and we are not doing integration testing yet > thus won't work) > In addition, this is required for streaming testcases, to have the JAR with > streaming classes for testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3121) NodeManager should handle disk-failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151940#comment-13151940 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3121: Okay, still a monster patch, but in a far better shape. I anticipate just one more iteration. - Please don't include util like APIs in NodeHealthStatus record, we want to keep the record implementations to the bare essentials. - Pass DiskHandlerService everywhere instead of NodeHealthCheckerService. - Change DISKS_FAILED to not use 144. May be -1001. - Remove unused imports in ContainersLauncher. - Remove the commented out init() code in ContainerExecutor. - Rename LocalStorage to DirectoryCollection? - getHealthScriptTimer() belongs to the HealthScriptRunner itself. Let's make nodeHealthScriptRunner.getTimerTask() public and drop TimerTask getHealthScriptTimer() from NodeHealthCheckerService. - Trivial: (java)doc NodeHealthCheckerService class. ContainerLaunch: When all disks have failed, use the health-report in the exception *and* also add a diagnostics to the event. - Same in ResourceLocalizationService - DiskHandlerService: when major-percentage disks are gone, log the report. (+108) ResourceLocalizationService: - Take a snapshot of dirs before the health-check for startLocalizer()? - PublicLocalizer uses a LocalDirAllocator for downloading file. Should it instead use DiskHandlerService? May be also check for min-percentage disks to be alive for each addResource() request. You will need changes to FSDownload too. - Remove PUBCACHE_CTXT after doing above. AppLogAggregatorImpl: - Existing log message at +120 can also list the good dirs. Bad dirs can be deduced from the DHS logs. DiskHandlerService: - The APIs with size are not needed or don't need the size paramater itself. - Take a lock on the cloned config for accesses via updateDirsInConfiguration(), getLocalPathForWrite(String pathStr) etc. where configuration is accessed. MiniYARNCluster: - Change the default the numLocalDirs and numLogDirs to 4? Also, consolidate the constructors? I can see the N number of constructors pattern of MiniMRCluster, let's avoid that. conf/controller.cfg - Update to not have the removed configs. - Can you also add banned.users and min.user.id with the default values? TestDiskFailure: - verifyDisksHealth(): Loop through and wait for a max of say 10 seconds for the node to turn unhealthy. - waitForDiskHealthCheck(): We can capture DiskHandlerService's last report time and wait till it changes atleast once. Of course that should be capped by a upper limit on the wait time. Can you run the linux-container-executor tests: TestLinuxContainerExecutor and TestContainerManagerWithLCE? Create a separate ticket for handling the disks that come back up online. Create a separate ticket for having a metric for numFailedDirs. - Test plan: - RM stops scheduling when major-percentage of disks go bad: Done - Node's DiskHandler recognises bad disks: Done - Node's DiskHandler recognises minimum percentage of good disks : Done - Integration test: Run a mapreduce job (so that Shuffle is also verified), offline some disks, run one more job and verify that both the apps pass. TODO - LogAggregation test: Verify that logs written on bad disks are ignored for aggregation (augment TestLogAggregationService) TODO: - ContainerLaunch: Verify that -- new containers don't use bad directories(by testing the LOCAL_DIRS env in a custom map job): TODO -- if major percentage disks turn bad, -- container should exit with proper exit code(should be easy with a custom application). TODO -- localization for a resource fails TODO > NodeManager should handle disk-failures > --- > > Key: MAPREDUCE-3121 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3121 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.23.1 > > Attachments: 3121.patch, 3121.v1.1.patch, 3121.v1.patch, 3121.v2.patch > > > This is akin to MAPREDUCE-2413 but for YARN's NodeManager. We want to > minimize the impact of transient/permanent disk failures on containers. With > larger number of disks per node, the ability to continue to run containers on > other disks is crucial. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3349) No rack-name logged in JobHistory for unsuccessful tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151899#comment-13151899
]
Amar Kamat commented on MAPREDUCE-3349:
---
Kindly make sure that Rumen's {{TopologyBuilder}} is also fixed in this JIRA.
{{TopologyBuilder}} expects hostname in the form '/rack/host' and uses
{{TaskAttemptFinishedEvent}} and {{TaskAttemptUnsuccessfulCompletionEvent}} for
the same. It also uses split-location from {{TaskStartedEvent}}. Since the
hostnames and racknames are now separately logged, we need to make sure that
{{TopologyBuilder}} is aware of that.
> No rack-name logged in JobHistory for unsuccessful tasks
>
>
> Key: MAPREDUCE-3349
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3349
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Devaraj K
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3349.patch
>
>
> Found this while running jobs on a cluster with [~Karams].
> This is because TaskAttemptUnsuccessfulCompletionEvent history record doesn't
> have a rack field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
