[jira] [Updated] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3953: Status: Open (was: Patch Available) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3953: Status: Patch Available (was: Open) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4040) History links should use hostname rather than IP address.
[
https://issues.apache.org/jira/browse/MAPREDUCE-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234172#comment-13234172
]
Hadoop QA commented on MAPREDUCE-4040:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519175/MAPREDUCE-4040.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2077//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2077//console
This message is automatically generated.
History links should use hostname rather than IP address.
-
Key: MAPREDUCE-4040
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4040
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.2
Reporter: Bhallamudi Venkata Siva Kamesh
Priority: Minor
Attachments: MAPREDUCE-4040.patch
While navigating from web page (eg: */cluster/app/app-id* ) to HS, browser
displays IP address rather than hostname.
{code:title=JobHistoryUtils.java|borderStyle=solid}
if (address.getAddress().isAnyLocalAddress() ||
address.getAddress().isLoopbackAddress()) {
sb.append(InetAddress.getLocalHost().getHostAddress());
}
}
{code}
I *think* it is better to use hostname rather than IP address.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3955) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234170#comment-13234170 ] Hadoop QA commented on MAPREDUCE-3955: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519177/MAPREDUCE-3955-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2078//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2078//console This message is automatically generated. Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine. Key: MAPREDUCE-3955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3955 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: MAPREDUCE-3955-trunk.patch, MAPREDUCE-3955-trunk.patch We shouldn't have two rpc engines based on protocol buffers. ProtoOverHadoopRpcEngine in hadoop-yarn-common should be replaced by ProtobufRpcEngine in hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4039) Sort Avoidance
[
https://issues.apache.org/jira/browse/MAPREDUCE-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234174#comment-13234174
]
anty.rao commented on MAPREDUCE-4039:
-
I am a little confused about the implementation of Reader of IFile.
In previous hadoop version, IFile reader will read in a bunch of key/value
pairs from the disk one time, then serve it directly from in memory.I think
this strategy is common and good.However, in yarn for each requested key/value
pairs reader will go hit the disk(though pre-read will do some help). Am i miss
something？Can someone shed light on me?
Sort Avoidance
--
Key: MAPREDUCE-4039
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4039
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: mrv2
Affects Versions: 0.23.2
Reporter: anty.rao
Priority: Minor
Fix For: 0.23.2
Inspired by
[Tenzing|http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/37200.pdf],
in 5.1 MapReduce Enhanceemtns:
{quote}*Sort Avoidance*. Certain operators such as hash join
and hash aggregation require shuffling, but not sorting. The
MapReduce API was enhanced to automatically turn off
sorting for these operations. When sorting is turned off, the
mapper feeds data to the reducer which directly passes the
data to the Reduce() function bypassing the intermediate
sorting step. This makes many SQL operators significantly
more ecient.{quote}
There are a lot of applications which need aggregation only, not
sorting.Using sorting to achieve aggregation is costly and inefficient.
Without sorting, up application can make use of hash table or hash map to do
aggregation efficiently.But application should bear in mind that reduce
memory is limited, itself is committed to manage memory of reduce, guard
against out of memory. Map-side combiner is not supported, you can also do
hash aggregation in map side as a workaround.
the following is the main points of sort avoidance implementation
# add a configuration parameter ??mapreduce.sort.avoidance??, boolean type,
to turn on/off sort avoidance workflow.Two type of workflow are coexist
together.
# key/value pairs emitted by map function is sorted by partition only, using
a more efficient sorting algorithm: counting sort.
# map-side merge, use a kind of byte merge, which just concatenate bytes from
generated spills, read in bytes, write out bytes, without overhead of
key/value serialization/deserailization, comparison, which current version
incurs.
# reduce can start up as soon as there is any map output available, in
contrast to sort workflow which must wait until all map outputs are fetched
and merged.
# map output in memory can be directly consumed by reduce.When reduce can't
catch up with the speed of incoming map outputs, in-memory merge thread will
kick in, merging in-memory map outputs onto disk.
# sequentially read in on-disk files to feed reduce, in contrast to currently
implementation which read multiple files concurrently, result in many disk
seek. Map output in memory take precedence over on disk files in feeding
reduce function.
I have already implement this feature based on hadoop CDH3U3 and done some
performance evaluation, you can reference to
[https://github.com/hanborq/hadoop] for details. Now,I'm willing to port it
into yarn. Welcome for commenting.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4040) History links should use hostname rather than IP address.
[
https://issues.apache.org/jira/browse/MAPREDUCE-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234180#comment-13234180
]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-4040:
---
Verified the fix manually, and with this fix, browser displays *hostname*
rather than *IP Address*.
History links should use hostname rather than IP address.
-
Key: MAPREDUCE-4040
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4040
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.2
Reporter: Bhallamudi Venkata Siva Kamesh
Priority: Minor
Attachments: MAPREDUCE-4040.patch
While navigating from web page (eg: */cluster/app/app-id* ) to HS, browser
displays IP address rather than hostname.
{code:title=JobHistoryUtils.java|borderStyle=solid}
if (address.getAddress().isAnyLocalAddress() ||
address.getAddress().isLoopbackAddress()) {
sb.append(InetAddress.getLocalHost().getHostAddress());
}
}
{code}
I *think* it is better to use hostname rather than IP address.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3955) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated MAPREDUCE-3955: Target Version/s: 0.24.0, 0.23.3 (was: 0.23.3, 0.24.0) Status: Open (was: Patch Available) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine. Key: MAPREDUCE-3955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3955 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: MAPREDUCE-3955-trunk.patch, MAPREDUCE-3955-trunk.patch We shouldn't have two rpc engines based on protocol buffers. ProtoOverHadoopRpcEngine in hadoop-yarn-common should be replaced by ProtobufRpcEngine in hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3955) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated MAPREDUCE-3955: Target Version/s: 0.24.0, 0.23.3 (was: 0.23.3, 0.24.0) Status: Patch Available (was: Open) Fixed TestMRJobsWithHistoryService. Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine. Key: MAPREDUCE-3955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3955 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: MAPREDUCE-3955-trunk.patch, MAPREDUCE-3955-trunk.patch, MAPREDUCE-3955-trunk.patch We shouldn't have two rpc engines based on protocol buffers. ProtoOverHadoopRpcEngine in hadoop-yarn-common should be replaced by ProtobufRpcEngine in hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3955) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated MAPREDUCE-3955: Attachment: MAPREDUCE-3955-trunk.patch Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine. Key: MAPREDUCE-3955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3955 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: MAPREDUCE-3955-trunk.patch, MAPREDUCE-3955-trunk.patch, MAPREDUCE-3955-trunk.patch We shouldn't have two rpc engines based on protocol buffers. ProtoOverHadoopRpcEngine in hadoop-yarn-common should be replaced by ProtobufRpcEngine in hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3955) Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234196#comment-13234196 ] Hadoop QA commented on MAPREDUCE-3955: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519182/MAPREDUCE-3955-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2079//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2079//console This message is automatically generated. Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine. Key: MAPREDUCE-3955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3955 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: MAPREDUCE-3955-trunk.patch, MAPREDUCE-3955-trunk.patch, MAPREDUCE-3955-trunk.patch We shouldn't have two rpc engines based on protocol buffers. ProtoOverHadoopRpcEngine in hadoop-yarn-common should be replaced by ProtobufRpcEngine in hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3315) Master-Worker Application on YARN
[
https://issues.apache.org/jira/browse/MAPREDUCE-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234229#comment-13234229
]
Nikhil S. Ketkar commented on MAPREDUCE-3315:
-
I have started some work on this issue and I wanted to describe the overall
approach I am taking and get some feedback.
Let me start by describing how an application will be written using
Master-Worker. The pseudocode below illustrates the usage.
{code}
class WordCount {
// Word count implementation using Master-Worker paradigm.
// Master sends portions of the input file to workers
// Worker builds a word count hash map (word - count) and sends it as a
result unit
// Master uses the results to update master dictionary
class WorkUnit {
// A list of words
}
class ResultUnit {
// Map of word - count
}
class Master extends MasterRunnerWorkUnit, ResultUnit {
public manageWorkers() {
// Spawn a bunch of workers using spawnNewWorker();
// For each spawned worker spawnNewWorker() will return a
WorkerReference, keep it around for bookkeeping
// Read a file and create WorkUnits with a fixed number of lines to
each workers
// Assign WorkUnits to Workers using assignWork(), use the
previously obtained WorkerReference
// Wait for ResultUnits using waitForResult();
// Update master dictionary based on a received result unit
// After work is done, kill workers using terminateWorker()
}
}
class Worker extends WorkerRunnerWorkUnit, ResultUnit {
public ResultUnit doWork(WorkUnit wu) {
// Build a word count hash map (word - count) and sends it as a result
unit
}
}
}
// User writes code to setup and launch job
public static void main(String[] args) throws Exception {
MWJobConf conf = new MWJobConf(WordCount.class);
conf.setJobName(WordCount);
conf.setMasterClass(Master.class);
conf.setWorkerClass(Worker.class);
conf.setWorkUnitClass(WorkUnit.class);
conf.setResultUnitClass(ResultUnit.class);
MWJobClient.runJob(conf);
}
}
{code}
Key functionality for the Master Worker will be implemented in the following
classes.
{code}
class MasterRunnerW, R {
// Spawn new Worker
protected WorkerReference spawnNewWorker();
// Spawn new Workers
protected ArrayListWorkerReference spawnNewWorkers();
// Assign work to any Worker, returns WorkerReference to whom it was
assigned
protected WorkerReference assignWork(WorkUnit wu);
// Assign work to a specific Worker
protected void assignWork(WorkerReference wf, WorkUnit wu);
// Wait for result from any Worker. Blocking Call
protected ResultUnit waitForResult();
// Wait for result from a specific worker. Blocking Call
protected ResultUnit waitForResult(WorkerReference wf);
// Is this specific worker alive? Blocking Call
protected boolean isWorkerAlive(WorkerReference wf);
// Get alive workers? Blocking Call
protected ArrayListWorkerReference isWorkerAlive(WorkerReference wf);
// Terminate a specific Worker
protected void terminateWorker(WorkerReference wf);
// To be implemented by user
public manageWorkers() = 0;
}
class WorkerRunnerW, R {
// To be implemented by user
public ResultUnit doWork(WorkUnit wu) = 0;
}
class WorkUnitContainerW {
// The framework passes around WorkUnitContainers which contain the user
defined
// WorkUnit and some additional bookkeeping information
}
class ResultUnitContainerR {
// The framework passes around ResultUnitContainers which contain the user
defined
// ResultUnit and some additional bookkeeping information
}
class WorkerReference {
// Uniquely identifies the Workers
}
{code}
A few questions I have been thinking about are:
# Should I use Hadoop IPC or RMI or something else?
# Should the Master be in the ApplicationManager or be run as a Container?
Master-Worker Application on YARN
-
Key: MAPREDUCE-3315
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3315
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Sharad Agarwal
Assignee: Sharad Agarwal
Fix For: 0.24.0
Currently master worker scenarios are forced fit into Map-Reduce. Now with
YARN, these can be first class and would benefit real/near realtime workloads
and be more effective in using the cluster resources.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234232#comment-13234232 ] Ravi Gummadi commented on MAPREDUCE-3953: - Gridmix unit tests passed on my local machine. test-patch seems to be not working properly. That is why it gave that error above. I just committed the patch to trunk. Thanks Amar for the review. Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3953: Resolution: Fixed Status: Resolved (was: Patch Available) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234234#comment-13234234 ] Hudson commented on MAPREDUCE-3953: --- Integrated in Hadoop-Hdfs-trunk-Commit #1984 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1984/]) MAPREDUCE-3953. [Gridmix] Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task.(ravigummadi) (Revision 1303330) Result = SUCCESS ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1303330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobFactory.java * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/ZombieJob.java Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4047) NullPointerException during the map task merge phase
NullPointerException during the map task merge phase
Key: MAPREDUCE-4047
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4047
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Devaraj K
While executing the tera sort with 1TB data, got the below exception
{code:xml}
2012-03-20 21:02:18,526 FATAL [IPC Server handler 14 on 31776]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1332261815858_0002_m_003124_0 - exited : java.lang.NullPointerException
at org.apache.hadoop.mapred.IFile$Writer.init(IFile.java:135)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1845)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1438)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:663)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:730)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4047) NullPointerException during the map task merge phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4047:
-
Affects Version/s: 0.23.1
Assignee: Devaraj K
NullPointerException during the map task merge phase
Key: MAPREDUCE-4047
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4047
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
While executing the tera sort with 1TB data, got the below exception
{code:xml}
2012-03-20 21:02:18,526 FATAL [IPC Server handler 14 on 31776]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1332261815858_0002_m_003124_0 - exited :
java.lang.NullPointerException
at org.apache.hadoop.mapred.IFile$Writer.init(IFile.java:135)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1845)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1438)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:663)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:730)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234242#comment-13234242 ] Hudson commented on MAPREDUCE-3953: --- Integrated in Hadoop-Common-trunk-Commit #1910 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1910/]) MAPREDUCE-3953. [Gridmix] Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task.(ravigummadi) (Revision 1303330) Result = SUCCESS ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1303330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobFactory.java * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/ZombieJob.java Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4048) NullPointerException exception while accessing the Application Master UI
NullPointerException exception while accessing the Application Master UI
Key: MAPREDUCE-4048
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4048
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
{code:xml}
2012-03-21 10:21:31,838 ERROR [2145015588@qtp-957250718-801]
org.apache.hadoop.yarn.webapp.Dispatcher: error handling URI:
/mapreduce/attempts/job_1332261815858_2_8/m/KILLED
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:150)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
...
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.NullPointerException
at com.google.common.base.Joiner.toString(Joiner.java:317)
at com.google.common.base.Joiner.appendTo(Joiner.java:97)
at com.google.common.base.Joiner.appendTo(Joiner.java:127)
at com.google.common.base.Joiner.join(Joiner.java:158)
at com.google.common.base.Joiner.join(Joiner.java:166)
at org.apache.hadoop.yarn.util.StringHelper.join(StringHelper.java:102)
at
org.apache.hadoop.mapreduce.v2.app.webapp.AppController.badRequest(AppController.java:319)
at
org.apache.hadoop.mapreduce.v2.app.webapp.AppController.attempts(AppController.java:286)
... 36 more
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234271#comment-13234271 ] Hudson commented on MAPREDUCE-3953: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1919 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1919/]) MAPREDUCE-3953. [Gridmix] Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task.(ravigummadi) (Revision 1303330) Result = ABORTED ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1303330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobFactory.java * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/ZombieJob.java Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4049) plugin for generic shuffle service
plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 1.0.1, 0.23.1 Reporter: Avner BenHanoch Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we (Mellanox with Prof. Weikuan Yu from Auburn University) are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). Reference: Hadoop Acceleration through Network Levitated Merging, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Description: Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we (Mellanox with Prof. Weikuan Yu from Auburn University) are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I will soon attach document with suggested API for the plugin was: Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we (Mellanox with Prof. Weikuan Yu from Auburn University) are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). Reference: Hadoop Acceleration through Network Levitated Merging, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] Target Version/s: 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0) plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we (Mellanox with Prof. Weikuan Yu from Auburn University) are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I will soon attach document with suggested API for the plugin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234319#comment-13234319 ] Hudson commented on MAPREDUCE-3953: --- Integrated in Hadoop-Hdfs-trunk #991 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/991/]) MAPREDUCE-3953. [Gridmix] Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task.(ravigummadi) (Revision 1303330) Result = UNSTABLE ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1303330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobFactory.java * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/ZombieJob.java Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234360#comment-13234360 ] Hudson commented on MAPREDUCE-3953: --- Integrated in Hadoop-Mapreduce-trunk #1026 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1026/]) MAPREDUCE-3953. [Gridmix] Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task.(ravigummadi) (Revision 1303330) Result = SUCCESS ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1303330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobFactory.java * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/ZombieJob.java Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4050) Invalid node link
[ https://issues.apache.org/jira/browse/MAPREDUCE-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-4050: -- Attachment: MAPREDUCE-4050.png Attaching the snapshot Invalid node link - Key: MAPREDUCE-4050 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4050 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Attachments: MAPREDUCE-4050.png When a task is in *UNASSIGNED* state, node link is displayed as +null+. But I think it is better to display the link as *N/A* rather than +null+. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4050) Invalid node link
Invalid node link - Key: MAPREDUCE-4050 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4050 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Attachments: MAPREDUCE-4050.png When a task is in *UNASSIGNED* state, node link is displayed as +null+. But I think it is better to display the link as *N/A* rather than +null+. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4051) Remove the empty hadoop-mapreduce-project/assembly/all.xml file
Remove the empty hadoop-mapreduce-project/assembly/all.xml file --- Key: MAPREDUCE-4051 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4051 Project: Hadoop Map/Reduce Issue Type: Task Affects Versions: 0.23.2 Reporter: Ravi Prakash Jenkins picks up this XML and looks for test results in it. This file should be empty and removed. I could tell Jenkins not to pick this file up, but I'd rather also remove this empty file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4051) Remove the empty hadoop-mapreduce-project/assembly/all.xml file
[ https://issues.apache.org/jira/browse/MAPREDUCE-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-4051: Status: Patch Available (was: Open) $ svn rm hadoop-mapreduce-project/assembly/all.xml ought to do it by anyone weilding committer powers. I wasn't able to generate a patch using svn diff after rming the file. Remove the empty hadoop-mapreduce-project/assembly/all.xml file --- Key: MAPREDUCE-4051 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4051 Project: Hadoop Map/Reduce Issue Type: Task Affects Versions: 0.23.2 Reporter: Ravi Prakash Jenkins picks up this XML and looks for test results in it. This file should be empty and removed. I could tell Jenkins not to pick this file up, but I'd rather also remove this empty file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4040) History links should use hostname rather than IP address.
[
https://issues.apache.org/jira/browse/MAPREDUCE-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234467#comment-13234467
]
Hadoop QA commented on MAPREDUCE-4040:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519175/MAPREDUCE-4040.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2080//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2080//console
This message is automatically generated.
History links should use hostname rather than IP address.
-
Key: MAPREDUCE-4040
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4040
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.2
Reporter: Bhallamudi Venkata Siva Kamesh
Priority: Minor
Attachments: MAPREDUCE-4040.patch
While navigating from web page (eg: */cluster/app/app-id* ) to HS, browser
displays IP address rather than hostname.
{code:title=JobHistoryUtils.java|borderStyle=solid}
if (address.getAddress().isAnyLocalAddress() ||
address.getAddress().isLoopbackAddress()) {
sb.append(InetAddress.getLocalHost().getHostAddress());
}
}
{code}
I *think* it is better to use hostname rather than IP address.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234493#comment-13234493 ] Milind Bhandarkar commented on MAPREDUCE-4049: -- There has been an effort recently (https://github.com/hanborq/hadoop) to backport shuffle from 0.23 to 1.0 branch. Would it be pluggable with your patch ? plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I will soon attach document with suggested API for the plugin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4043) Secret keys set in Credentials are not seen by tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234553#comment-13234553 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4043: Can you please check and see if the patch for MAPREDUCE-3727 is the cause? I remember something like this, credentials being removed from the client as one of the proposals there. Don't have bandwidth to confirm this myself. Thanks. Secret keys set in Credentials are not seen by tasks Key: MAPREDUCE-4043 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4043 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, security Affects Versions: 0.23.2 Reporter: Jason Lowe Priority: Blocker The following scenario works in 0.20.205 but no longer works in 0.23: 1) During job submission, a secret key is set by calling jobConf.getCredentials().addSecretKey(Text, byte[]) 2) A map task retrieves the secret key by calling jobConf.getCredentials().getSecretKey(Text) In 205 the secret key is retrieved successfully but in 0.23 the secret key is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4004) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/MAPREDUCE-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated MAPREDUCE-4004: Attachment: MAPREDUCE-4004-branch-0.23.1.patch MAPREDUCE-4004.patch Renamed patches. Should apply to trunk and branches, identically. DistCp fails when invoked by Oozie -- Key: MAPREDUCE-4004 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4004 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.1 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: MAPREDUCE-4004-branch-0.23.1.patch, MAPREDUCE-4004.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4004) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/MAPREDUCE-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated MAPREDUCE-4004: Status: Open (was: Patch Available) DistCp fails when invoked by Oozie -- Key: MAPREDUCE-4004 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4004 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.1 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: MAPREDUCE-4004-branch-0.23.1.patch, MAPREDUCE-4004.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4004) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/MAPREDUCE-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated MAPREDUCE-4004: Attachment: (was: DistCp_HadoopTokenFileLocation.patch) DistCp fails when invoked by Oozie -- Key: MAPREDUCE-4004 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4004 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.1 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: MAPREDUCE-4004-branch-0.23.1.patch, MAPREDUCE-4004.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4004) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/MAPREDUCE-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mithun Radhakrishnan updated MAPREDUCE-4004: Status: Patch Available (was: Open) DistCp fails when invoked by Oozie -- Key: MAPREDUCE-4004 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4004 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.1 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: MAPREDUCE-4004-branch-0.23.1.patch, MAPREDUCE-4004.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4004) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/MAPREDUCE-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13235046#comment-13235046 ] Hadoop QA commented on MAPREDUCE-4004: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519313/MAPREDUCE-4004-branch-0.23.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2081//console This message is automatically generated. DistCp fails when invoked by Oozie -- Key: MAPREDUCE-4004 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4004 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.1 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: MAPREDUCE-4004-branch-0.23.1.patch, MAPREDUCE-4004.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4052) Windows eclpise can not submit the job
Windows eclpise can not submit the job -- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4053) Counters group names deprecation is wrong, iterating over group names deprecated names don't show up
Counters group names deprecation is wrong, iterating over group names deprecated names don't show up Key: MAPREDUCE-4053 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4053 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0, 0.23.3 Reporter: Alejandro Abdelnur Priority: Critical Fix For: 0.23.3 This is similar to the deprecation of Configuration properties bug HADOOP-8167, interator() retrieval of counter names only returns new names. Oozie breaks here because it is using the deprecate name and iterating over values (OOZIE-777). While it can be worked around easily in Oozie, this is breaking backwards compatibility. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
