[jira] [Commented] (MAPREDUCE-5905) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
[
https://issues.apache.org/jira/browse/MAPREDUCE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526293#comment-14526293
]
Tsuyoshi Ozawa commented on MAPREDUCE-5905:
---
+1, committing this shortly. This patch fixes a mistake about the management of
array indices.
Before the fix:
{quote}
{(group1)(group1)[(counter1)(counter1)(45)][(counter2)(counter2)(45)][(counter\{\}\(\)\[\])(counter\{\}\(\)\[\])(45)]}{(group2)(group2)[(counter1)(counter1)(45)][(counter2)(counter2)(45)][(counter\{\}\(\)\[\])(counter\{\}\(\)\[\])(45)]}{(group\{\}\(\)\[\])(group\{\}\(\)\[\])[(counter1)(counter1)(45)][(counter2)(counter2)(45)][(counter\{\}\(\)\[\])(counter\{\}\(\)\[\])(45)]}nullnullnullnullnullnull
{quote}
After the fix:
{quote}
{(group1)(group1)[(counter1)(counter1)(45)][(counter2)(counter2)(45)][(counter\{\}\(\)\[\])(counter\{\}\(\)\[\])(45)]}{(group2)(group2)[(counter1)(counter1)(45)][(counter2)(counter2)(45)][(counter\{\}\(\)\[\])(counter\{\}\(\)\[\])(45)]}{(group\{\}\(\)\[\])(group\{\}\(\)\[\])[(counter1)(counter1)(45)][(counter2)(counter2)(45)][(counter\{\}\(\)\[\])(counter\{\}\(\)\[\])(45)]}
{quote}
CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
--
Key: MAPREDUCE-5905
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5905
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0, 2.7.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Attachments: MAPREDUCE-5905.patch
CountersStrings.toEscapedCompactStrings outputs null strings if a
CounterGroup has more than one Counter.
That way there are some null strings in MRv1(CDH) job history log.
https://issues.cloudera.org/browse/DISTRO-598
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5905) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-5905: -- Resolution: Fixed Fix Version/s: 2.8.0 Target Version/s: 2.8.0 (was: 2.6.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk and branch-2. Thanks [~ajisakaa] for your contribution. CountersStrings.toEscapedCompactStrings outputs unnecessary null strings -- Key: MAPREDUCE-5905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5905 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.0, 2.7.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-5905.patch CountersStrings.toEscapedCompactStrings outputs null strings if a CounterGroup has more than one Counter. That way there are some null strings in MRv1(CDH) job history log. https://issues.cloudera.org/browse/DISTRO-598 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5905) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526300#comment-14526300 ] Hudson commented on MAPREDUCE-5905: --- FAILURE: Integrated in Hadoop-trunk-Commit #7719 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7719/]) MAPREDUCE-5905. CountersStrings.toEscapedCompactStrings outputs unnecessary null strings. Contributed by Akira AJISAKA. (ozawa: rev 3ba18362f2a4b83635b89aa0adc5ebaf27d9ca83) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/CountersStrings.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java * hadoop-mapreduce-project/CHANGES.txt CountersStrings.toEscapedCompactStrings outputs unnecessary null strings -- Key: MAPREDUCE-5905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5905 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.0, 2.7.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-5905.patch CountersStrings.toEscapedCompactStrings outputs null strings if a CounterGroup has more than one Counter. That way there are some null strings in MRv1(CDH) job history log. https://issues.cloudera.org/browse/DISTRO-598 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6349: -- Labels: BB2015-05-TBR backward-incompatible newbie (was: BB2015-05-TBR newbie) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, backward-incompatible, newbie Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6349: -- Affects Version/s: 2.7.0 Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, backward-incompatible, newbie Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6349: -- Labels: BB2015-05-TBR newbie (was: BB2015-05-TBR backward-incompatible newbie) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, newbie Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526312#comment-14526312 ] Tsuyoshi Ozawa commented on MAPREDUCE-6349: --- +1, confirmed compatibility - it's OK to change REDUCER_INPUT_VALUE_CLASS since REDUCER_INPUT_VALUE_CLASS is protected value and only used in Chain.java. The value set as REDUCER_INPUT_VALUE_CLASS isn't used from anywhere. Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, newbie Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6349: -- Fix Version/s: 2.8.0 Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, newbie Fix For: 2.8.0 Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6349: -- Resolution: Fixed Target Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk and branch-2. Thanks [~rchiang] for your contribution! Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, newbie Fix For: 2.8.0 Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526318#comment-14526318 ] Hudson commented on MAPREDUCE-6349: --- FAILURE: Integrated in Hadoop-trunk-Commit #7720 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7720/]) MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. Contributed by Ray Chiang. (ozawa: rev bb6ef2984d8f117711b806c4ebdc757bd182c06e) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/Chain.java * hadoop-mapreduce-project/CHANGES.txt Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, newbie Fix For: 2.8.0 Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6312) Hive fails due to stale proxy in ClientServiceDelegate
[
https://issues.apache.org/jira/browse/MAPREDUCE-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526434#comment-14526434
]
Radim Kubacki commented on MAPREDUCE-6312:
--
To Khaja: clue to your problem is IOException 'no space left on device' and it
is apparently a different issue. What I reported can be reproduced on a working
cluster with a simple map-reduce client.
Hive fails due to stale proxy in ClientServiceDelegate
--
Key: MAPREDUCE-6312
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6312
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.5.0
Reporter: Radim Kubacki
ClientServiceDelegate initializes its realProxy field to AMProxy for a new or
running job. Later when the job finishes it will not update this proxy to
query history server and AM will not return valid data for this job.
We found this while investigating
https://issues.cloudera.org/browse/DISTRO-631 that describes Hive failure
because it uses loop like
{code}
progress(JobClient jc, RunningJob rj) { ...
while (!rj.isComplete() || (extraRounds 0)) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
}
RunningJob newRj = jc.getJob(rj.getID());
if (newRj == null) {
// under exceptional load, hadoop may not be able to look up
status
// of finished jobs (because it has purged them from memory).
From
// hive's perspective - it's equivalent to the job having
failed.
// So raise a meaningful exception
throw new IOException(Could not find status of job: +
rj.getID());
} else {
rj = newRj;
}
}
{code}
In this snippet JobClient.getJob will try to create RunningJob instance
referring to job file in /user/$USER/.staging even when job is finished and
the file is moved to /user/history/done (or /user/history/done_intermediate).
Note that Hive queries can succeed if there is a timing where HDFS performs
actual file delete with a delay.
We can try to write a patch if there is an agreement that this should be
fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6351) Reducer hung in copy phase.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laxman updated MAPREDUCE-6351:
--
Description:
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long time.
After killing this task for couple of times manually, it gets completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs and
they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
{deleted}
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like a
classic case of multi-threading issue. Thread goes to wait state after it has
been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
{deleted}
was:
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long time.
After killing this task for couple of times manually, it gets completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs and
they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like a
classic case of multi-threading issue. Thread goes to wait state after it has
been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
Reducer hung in copy phase.
---
Key: MAPREDUCE-6351
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6351
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.6.0
Reporter: Laxman
Attachments: jstat-gc.log, reducer-container-partial.log.zip,
thread-dumps.out
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long
time. After killing this task for couple of times manually, it gets
completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs
and they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
{deleted}
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like
a classic case of multi-threading issue. Thread goes to wait state after it
has been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
{deleted}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6351) Reducer hung in copy phase.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laxman updated MAPREDUCE-6351:
--
Description:
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long time.
After killing this task for couple of times manually, it gets completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs and
they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
-
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like a
classic case of multi-threading issue. Thread goes to wait state after it has
been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}-
was:
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long time.
After killing this task for couple of times manually, it gets completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs and
they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
{deleted}
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like a
classic case of multi-threading issue. Thread goes to wait state after it has
been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
{deleted}
Reducer hung in copy phase.
---
Key: MAPREDUCE-6351
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6351
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.6.0
Reporter: Laxman
Attachments: jstat-gc.log, reducer-container-partial.log.zip,
thread-dumps.out
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long
time. After killing this task for couple of times manually, it gets
completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs
and they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
-
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like
a classic case of multi-threading issue. Thread goes to wait state after it
has been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}-
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6351) Reducer hung in copy phase.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laxman updated MAPREDUCE-6351:
--
Description:
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long time.
After killing this task for couple of times manually, it gets completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs and
they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like a
classic case of multi-threading issue. Thread goes to wait state after it has
been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
was:
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long time.
After killing this task for couple of times manually, it gets completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs and
they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
-
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like a
classic case of multi-threading issue. Thread goes to wait state after it has
been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}-
Reducer hung in copy phase.
---
Key: MAPREDUCE-6351
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6351
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.6.0
Reporter: Laxman
Attachments: jstat-gc.log, reducer-container-partial.log.zip,
thread-dumps.out
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long
time. After killing this task for couple of times manually, it gets
completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs
and they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like
a classic case of multi-threading issue. Thread goes to wait state after it
has been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6351) Reducer hung in copy phase.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526438#comment-14526438
]
Laxman commented on MAPREDUCE-6351:
---
Threads analysis mentioned in description above found to be incorrect when I
retrace the code flow. Pre-notification is not a problem as merger wait is
guarded by size check.
However, problem exists, fetchers are not proceeding and waiting for merger to
free some memory and merge doing nothing.
Reducer hung in copy phase.
---
Key: MAPREDUCE-6351
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6351
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.6.0
Reporter: Laxman
Attachments: jstat-gc.log, reducer-container-partial.log.zip,
thread-dumps.out
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long
time. After killing this task for couple of times manually, it gets
completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs
and they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like
a classic case of multi-threading issue. Thread goes to wait state after it
has been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (MAPREDUCE-6348) JobHistoryEventHandler could not flush every 30 secondes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reopened MAPREDUCE-6348: -- JobHistoryEventHandler could not flush every 30 secondes Key: MAPREDUCE-6348 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6348 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Reporter: qus-jiawei Priority: Minor JobHistoryEventHandler could not flush the event every 30 seconds. cause the var isTimerActive is never set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6348) JobHistoryEventHandler could not flush every 30 secondes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526485#comment-14526485 ] Devaraj K commented on MAPREDUCE-6348: -- Dup of MAPREDUCE-6339 JobHistoryEventHandler could not flush every 30 secondes Key: MAPREDUCE-6348 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6348 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Reporter: qus-jiawei Priority: Minor JobHistoryEventHandler could not flush the event every 30 seconds. cause the var isTimerActive is never set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6348) JobHistoryEventHandler could not flush every 30 secondes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K resolved MAPREDUCE-6348. -- Resolution: Duplicate JobHistoryEventHandler could not flush every 30 secondes Key: MAPREDUCE-6348 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6348 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Reporter: qus-jiawei Priority: Minor JobHistoryEventHandler could not flush the event every 30 seconds. cause the var isTimerActive is never set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526515#comment-14526515 ] Hudson commented on MAPREDUCE-6349: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #183 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/183/]) MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. Contributed by Ray Chiang. (ozawa: rev bb6ef2984d8f117711b806c4ebdc757bd182c06e) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/Chain.java * hadoop-mapreduce-project/CHANGES.txt Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, newbie Fix For: 2.8.0 Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5905) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526517#comment-14526517 ] Hudson commented on MAPREDUCE-5905: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #183 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/183/]) MAPREDUCE-5905. CountersStrings.toEscapedCompactStrings outputs unnecessary null strings. Contributed by Akira AJISAKA. (ozawa: rev 3ba18362f2a4b83635b89aa0adc5ebaf27d9ca83) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/CountersStrings.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java CountersStrings.toEscapedCompactStrings outputs unnecessary null strings -- Key: MAPREDUCE-5905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5905 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.0, 2.7.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-5905.patch CountersStrings.toEscapedCompactStrings outputs null strings if a CounterGroup has more than one Counter. That way there are some null strings in MRv1(CDH) job history log. https://issues.cloudera.org/browse/DISTRO-598 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5905) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526522#comment-14526522 ] Hudson commented on MAPREDUCE-5905: --- FAILURE: Integrated in Hadoop-Yarn-trunk #917 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/917/]) MAPREDUCE-5905. CountersStrings.toEscapedCompactStrings outputs unnecessary null strings. Contributed by Akira AJISAKA. (ozawa: rev 3ba18362f2a4b83635b89aa0adc5ebaf27d9ca83) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/CountersStrings.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java CountersStrings.toEscapedCompactStrings outputs unnecessary null strings -- Key: MAPREDUCE-5905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5905 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.0, 2.7.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-5905.patch CountersStrings.toEscapedCompactStrings outputs null strings if a CounterGroup has more than one Counter. That way there are some null strings in MRv1(CDH) job history log. https://issues.cloudera.org/browse/DISTRO-598 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526520#comment-14526520 ] Hudson commented on MAPREDUCE-6349: --- FAILURE: Integrated in Hadoop-Yarn-trunk #917 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/917/]) MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. Contributed by Ray Chiang. (ozawa: rev bb6ef2984d8f117711b806c4ebdc757bd182c06e) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/Chain.java Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, newbie Fix For: 2.8.0 Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-6279: -- Attachment: (was: MAPREDUCE-6279.v2.txt) AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6279.v1.txt Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-6279: -- Status: Open (was: Patch Available) AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6279.v1.txt Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-6279: -- Attachment: MAPREDUCE-6279.v2.txt Re-uploading version 2 of the patch and re-submitting the patch in order to kick the build. AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-6279: -- Status: Patch Available (was: Open) AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6351) Reducer hung in copy phase.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526636#comment-14526636
]
Jason Lowe commented on MAPREDUCE-6351:
---
I suspect this is a duplicate of MAPREDUCE-6334. I see a lot of these types of
messages in the reducer log:
{noformat}
2015-05-01 19:59:37,632 WARN [fetcher#13]
org.apache.hadoop.mapreduce.task.reduce.Fetcher: Shuffle output from
glgs1190.grid.uh1.inmobi.com:13562 failed, retry it.
{noformat}
I think it is leaking memory allocations from the shuffle errors and the
shuffle buffer runs out of available memory (hence fetchers told to WAIT) but
there isn't enough data in the shuffle buffer to trigger a merge. All of the
memory that was leaked will never complete to kick off the merge and unblock
the other threads.
Reducer hung in copy phase.
---
Key: MAPREDUCE-6351
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6351
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.6.0
Reporter: Laxman
Attachments: jstat-gc.log, reducer-container-partial.log.zip,
thread-dumps.out
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long
time. After killing this task for couple of times manually, it gets
completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs
and they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like
a classic case of multi-threading issue. Thread goes to wait state after it
has been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526665#comment-14526665 ] Hudson commented on MAPREDUCE-6349: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2115 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2115/]) MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. Contributed by Ray Chiang. (ozawa: rev bb6ef2984d8f117711b806c4ebdc757bd182c06e) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/Chain.java * hadoop-mapreduce-project/CHANGES.txt Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, newbie Fix For: 2.8.0 Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5905) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526667#comment-14526667 ] Hudson commented on MAPREDUCE-5905: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2115 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2115/]) MAPREDUCE-5905. CountersStrings.toEscapedCompactStrings outputs unnecessary null strings. Contributed by Akira AJISAKA. (ozawa: rev 3ba18362f2a4b83635b89aa0adc5ebaf27d9ca83) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/CountersStrings.java CountersStrings.toEscapedCompactStrings outputs unnecessary null strings -- Key: MAPREDUCE-5905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5905 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.0, 2.7.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-5905.patch CountersStrings.toEscapedCompactStrings outputs null strings if a CounterGroup has more than one Counter. That way there are some null strings in MRv1(CDH) job history log. https://issues.cloudera.org/browse/DISTRO-598 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5905) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526698#comment-14526698 ] Hudson commented on MAPREDUCE-5905: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #174 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/174/]) MAPREDUCE-5905. CountersStrings.toEscapedCompactStrings outputs unnecessary null strings. Contributed by Akira AJISAKA. (ozawa: rev 3ba18362f2a4b83635b89aa0adc5ebaf27d9ca83) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/CountersStrings.java * hadoop-mapreduce-project/CHANGES.txt CountersStrings.toEscapedCompactStrings outputs unnecessary null strings -- Key: MAPREDUCE-5905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5905 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.0, 2.7.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-5905.patch CountersStrings.toEscapedCompactStrings outputs null strings if a CounterGroup has more than one Counter. That way there are some null strings in MRv1(CDH) job history log. https://issues.cloudera.org/browse/DISTRO-598 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526696#comment-14526696 ] Hudson commented on MAPREDUCE-6349: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #174 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/174/]) MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. Contributed by Ray Chiang. (ozawa: rev bb6ef2984d8f117711b806c4ebdc757bd182c06e) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/Chain.java Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, newbie Fix For: 2.8.0 Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526726#comment-14526726 ] Hudson commented on MAPREDUCE-6349: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/184/]) MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. Contributed by Ray Chiang. (ozawa: rev bb6ef2984d8f117711b806c4ebdc757bd182c06e) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/Chain.java Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, newbie Fix For: 2.8.0 Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5905) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526728#comment-14526728 ] Hudson commented on MAPREDUCE-5905: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #184 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/184/]) MAPREDUCE-5905. CountersStrings.toEscapedCompactStrings outputs unnecessary null strings. Contributed by Akira AJISAKA. (ozawa: rev 3ba18362f2a4b83635b89aa0adc5ebaf27d9ca83) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/CountersStrings.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java CountersStrings.toEscapedCompactStrings outputs unnecessary null strings -- Key: MAPREDUCE-5905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5905 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.0, 2.7.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-5905.patch CountersStrings.toEscapedCompactStrings outputs null strings if a CounterGroup has more than one Counter. That way there are some null strings in MRv1(CDH) job history log. https://issues.cloudera.org/browse/DISTRO-598 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5905) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
[ https://issues.apache.org/jira/browse/MAPREDUCE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526752#comment-14526752 ] Hudson commented on MAPREDUCE-5905: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2133 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2133/]) MAPREDUCE-5905. CountersStrings.toEscapedCompactStrings outputs unnecessary null strings. Contributed by Akira AJISAKA. (ozawa: rev 3ba18362f2a4b83635b89aa0adc5ebaf27d9ca83) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/CountersStrings.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java CountersStrings.toEscapedCompactStrings outputs unnecessary null strings -- Key: MAPREDUCE-5905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5905 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.0, 2.7.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-5905.patch CountersStrings.toEscapedCompactStrings outputs null strings if a CounterGroup has more than one Counter. That way there are some null strings in MRv1(CDH) job history log. https://issues.cloudera.org/browse/DISTRO-598 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6349) Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526750#comment-14526750 ] Hudson commented on MAPREDUCE-6349: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2133 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2133/]) MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. Contributed by Ray Chiang. (ozawa: rev bb6ef2984d8f117711b806c4ebdc757bd182c06e) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/Chain.java * hadoop-mapreduce-project/CHANGES.txt Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS -- Key: MAPREDUCE-6349 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6349 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: BB2015-05-TBR, newbie Fix For: 2.8.0 Attachments: MAPREDUCE-6349.001.patch Ran across this typo in a property. It doesn't look like it's used anywhere externally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5649) Reduce cannot use more than 2G memory for the final merge
[
https://issues.apache.org/jira/browse/MAPREDUCE-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526796#comment-14526796
]
Hadoop QA commented on MAPREDUCE-5649:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 59s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 43s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 48s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 48s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 17s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 41s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 38m 51s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12729994/MAPREDUCE-5649.003.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f65c79 |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5633/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5633/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5633/console |
This message was automatically generated.
Reduce cannot use more than 2G memory for the final merge
--
Key: MAPREDUCE-5649
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5649
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: stanley shi
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5649.001.patch, MAPREDUCE-5649.002.patch,
MAPREDUCE-5649.003.patch
In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in
the finalMerge method:
int maxInMemReduce = (int)Math.min(
Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE);
This means no matter how much memory user has, reducer will not retain more
than 2G data in memory before the reduce phase starts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6354) shuffle handler should log connection info
Chang Li created MAPREDUCE-6354: --- Summary: shuffle handler should log connection info Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5649) Reduce cannot use more than 2G memory for the final merge
[
https://issues.apache.org/jira/browse/MAPREDUCE-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526868#comment-14526868
]
Hadoop QA commented on MAPREDUCE-5649:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 45s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 16s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 38s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 37m 50s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12729994/MAPREDUCE-5649.003.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f65c79 |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5635/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5635/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5635/console |
This message was automatically generated.
Reduce cannot use more than 2G memory for the final merge
--
Key: MAPREDUCE-5649
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5649
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: stanley shi
Assignee: Gera Shegalov
Attachments: MAPREDUCE-5649.001.patch, MAPREDUCE-5649.002.patch,
MAPREDUCE-5649.003.patch
In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in
the finalMerge method:
int maxInMemReduce = (int)Math.min(
Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE);
This means no matter how much memory user has, reducer will not retain more
than 2G data in memory before the reduce phase starts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6354: Attachment: MAPREDUCE-6354.patch shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6354: Status: Patch Available (was: Open) [~jlowe], could you please help review this patch? Thanks shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526880#comment-14526880 ] Allen Wittenauer commented on MAPREDUCE-6354: - I don't think this is a good idea. This is going to result in MASSIVE log increases with large MR jobs. shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[
https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526940#comment-14526940
]
Hadoop QA commented on MAPREDUCE-6354:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 48s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 21s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 37s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 20s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 35m 57s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730212/MAPREDUCE-6354.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f65c79 |
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5636/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5636/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5636/console |
This message was automatically generated.
shuffle handler should log connection info
--
Key: MAPREDUCE-6354
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chang Li
Assignee: Chang Li
Attachments: MAPREDUCE-6354.patch
currently, shuffle handler only log connection info in debug mode, we want to
log that info in a more concise way
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6251) JobClient needs additional retries at a higher level to address not-immediately-consistent dfs corner cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-6251: --- Target Version/s: 2.8.0 Status: Open (was: Patch Available) Okay, reviewing again after your responses - Configuration usually goes into MRJobConfig. Unless you explicitly don't want them publicly visible to end-users naturally. - Once you move them to MRJobConfig, the naming convention also changes. Names there follow MR_CLIENT_*, DEFAULT_* patterns - Document them in mapred-default.xml? Stating when they are needed, and how they should be used in contrast to the lower level retries. JobClient needs additional retries at a higher level to address not-immediately-consistent dfs corner cases --- Key: MAPREDUCE-6251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6251 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 2.6.0 Reporter: Craig Welch Assignee: Craig Welch Attachments: MAPREDUCE-6251.0.patch, MAPREDUCE-6251.1.patch, MAPREDUCE-6251.2.patch, MAPREDUCE-6251.3.patch The JobClient is used to get job status information for running and completed jobs. Final state and history for a job is communicated from the application master to the job history server via a distributed file system - where the history is uploaded by the application master to the dfs and then scanned/loaded by the jobhistory server. While HDFS has strong consistency guarantees not all Hadoop DFS's do. When used in conjunction with a distributed file system which does not have this guarantee there will be cases where the history server may not see an uploaded file, resulting in the dreaded no such job and a null value for the RunningJob in the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6192) Create unit test to automatically compare MR related classes and mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527011#comment-14527011 ] Robert Kanter commented on MAPREDUCE-6192: -- LGTM +1 Create unit test to automatically compare MR related classes and mapred-default.xml --- Key: MAPREDUCE-6192 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6192 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6192.001.patch, MAPREDUCE-6192.002.patch, MAPREDUCE-6192.003.patch, MAPREDUCE-6192.004.patch, MAPREDUCE-6192.005.patch, MAPREDUCE-6192.006.patch, MAPREDUCE-6192.007.patch Create a unit test that will automatically compare the fields in the various MapReduce related classes and mapred-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527018#comment-14527018 ] Jason Lowe commented on MAPREDUCE-6354: --- bq. I don't think this is a good idea. This is going to result in MASSIVE log increases with large MR jobs. It will create a lot of logging for a large job, but we're _already_ logging a not-so-useful message per shuffle connection after MAPREDUCE-5787. And having a useful message per shuffle connection is very useful for tracking down abusive jobs whose shuffle phase causes NM file descriptors, network or disk to go haywire. Allen, would it mitigate your concerns if this were logged with a separately configurable logger, e.g.: ShuffleHandlerAuditLogger? That way users could configure it on when they want to audit shuffle transfers or off when they don't. Other comments on the patch: please don't add newlines to the output, it just makes the logs visibly long. I'd prefer a brief one-line message per connection, and IMHO it's redundant to label a job ID with job: since the jobID has job_ in it already. shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5649) Reduce cannot use more than 2G memory for the final merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527070#comment-14527070 ] Jason Lowe commented on MAPREDUCE-5649: --- +1 lgtm. Committing this. Reduce cannot use more than 2G memory for the final merge -- Key: MAPREDUCE-5649 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5649 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: stanley shi Assignee: Gera Shegalov Attachments: MAPREDUCE-5649.001.patch, MAPREDUCE-5649.002.patch, MAPREDUCE-5649.003.patch In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in the finalMerge method: int maxInMemReduce = (int)Math.min( Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE); This means no matter how much memory user has, reducer will not retain more than 2G data in memory before the reduce phase starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5649) Reduce cannot use more than 2G memory for the final merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5649: -- Resolution: Fixed Fix Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Gera! I committed this to trunk and branch-2. Reduce cannot use more than 2G memory for the final merge -- Key: MAPREDUCE-5649 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5649 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: stanley shi Assignee: Gera Shegalov Fix For: 2.8.0 Attachments: MAPREDUCE-5649.001.patch, MAPREDUCE-5649.002.patch, MAPREDUCE-5649.003.patch In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in the finalMerge method: int maxInMemReduce = (int)Math.min( Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE); This means no matter how much memory user has, reducer will not retain more than 2G data in memory before the reduce phase starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5649) Reduce cannot use more than 2G memory for the final merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527091#comment-14527091 ] Hudson commented on MAPREDUCE-5649: --- FAILURE: Integrated in Hadoop-trunk-Commit #7725 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7725/]) MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge. Contributed by Gera Shegalov (jlowe: rev 7dc3c1203d1ab14c09d0aaf0869a5bcdfafb0a5a) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMergeManager.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java Reduce cannot use more than 2G memory for the final merge -- Key: MAPREDUCE-5649 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5649 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: stanley shi Assignee: Gera Shegalov Fix For: 2.8.0 Attachments: MAPREDUCE-5649.001.patch, MAPREDUCE-5649.002.patch, MAPREDUCE-5649.003.patch In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in the finalMerge method: int maxInMemReduce = (int)Math.min( Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE); This means no matter how much memory user has, reducer will not retain more than 2G data in memory before the reduce phase starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6312) Hive fails due to stale proxy in ClientServiceDelegate
[
https://issues.apache.org/jira/browse/MAPREDUCE-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527098#comment-14527098
]
Khaja Hussain commented on MAPREDUCE-6312:
--
To Radim: My failure had the same exception triggered. It is clear from the log
that my issue is different from your. In this case you can ignore the comments.
Thanks.
Hive fails due to stale proxy in ClientServiceDelegate
--
Key: MAPREDUCE-6312
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6312
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.5.0
Reporter: Radim Kubacki
ClientServiceDelegate initializes its realProxy field to AMProxy for a new or
running job. Later when the job finishes it will not update this proxy to
query history server and AM will not return valid data for this job.
We found this while investigating
https://issues.cloudera.org/browse/DISTRO-631 that describes Hive failure
because it uses loop like
{code}
progress(JobClient jc, RunningJob rj) { ...
while (!rj.isComplete() || (extraRounds 0)) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
}
RunningJob newRj = jc.getJob(rj.getID());
if (newRj == null) {
// under exceptional load, hadoop may not be able to look up
status
// of finished jobs (because it has purged them from memory).
From
// hive's perspective - it's equivalent to the job having
failed.
// So raise a meaningful exception
throw new IOException(Could not find status of job: +
rj.getID());
} else {
rj = newRj;
}
}
{code}
In this snippet JobClient.getJob will try to create RunningJob instance
referring to job file in /user/$USER/.staging even when job is finished and
the file is moved to /user/history/done (or /user/history/done_intermediate).
Note that Hive queries can succeed if there is a timing where HDFS performs
actual file delete with a delay.
We can try to write a patch if there is an agreement that this should be
fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[
https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated MAPREDUCE-6279:
--
Attachment: MAPREDUCE-6279.v3.patch
Version 3 has the {{.patch}} extension.
AM should explicity exit JVM after all services have stopped
Key: MAPREDUCE-6279
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.5.0
Reporter: Jason Lowe
Assignee: Eric Payne
Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt,
MAPREDUCE-6279.v3.patch
Occasionally the MapReduce AM can get stuck trying to shut down.
MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been
fixed, but this can also occur with uber jobs if the task code inadvertently
leaves non-daemon threads lingering.
We should explicitly shutdown the JVM after the MapReduce AM has unregistered
and all services have been stopped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[
https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527754#comment-14527754
]
Hudson commented on MAPREDUCE-6165:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #7729 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7729/])
MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed
by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
[JDK8] TestCombineFileInputFormat failed on JDK8
Key: MAPREDUCE-6165
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Wei Yan
Assignee: Akira AJISAKA
Priority: Minor
Fix For: 2.8.0
Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch,
MAPREDUCE-6165-003.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-004.patch,
MAPREDUCE-6165-reproduce.patch
The error msg:
{noformat}
testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
Time elapsed: 2.487 sec FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:1
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911)
testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
Time elapsed: 0.985 sec FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:1
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6192) Create unit test to automatically compare MR related classes and mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6192: -- Attachment: MAPREDUCE-6192.branch-2.007.patch MAPREDUCE-6057 (Remove obsolete entries from mapred-default.xml) was committed to trunk, but not to branch-2. Those obsolete entries need a waiver in branch-2 version of this patch. Create unit test to automatically compare MR related classes and mapred-default.xml --- Key: MAPREDUCE-6192 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6192 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6192.001.patch, MAPREDUCE-6192.002.patch, MAPREDUCE-6192.003.patch, MAPREDUCE-6192.004.patch, MAPREDUCE-6192.005.patch, MAPREDUCE-6192.006.patch, MAPREDUCE-6192.007.patch, MAPREDUCE-6192.branch-2.007.patch Create a unit test that will automatically compare the fields in the various MapReduce related classes and mapred-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6192) Create unit test to automatically compare MR related classes and mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527868#comment-14527868 ] Ray Chiang commented on MAPREDUCE-6192: --- I've put up a separate branch-2 patch. It makes the TestMapreduceConfigFields equivalent in branch-2 and trunk. Create unit test to automatically compare MR related classes and mapred-default.xml --- Key: MAPREDUCE-6192 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6192 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6192.001.patch, MAPREDUCE-6192.002.patch, MAPREDUCE-6192.003.patch, MAPREDUCE-6192.004.patch, MAPREDUCE-6192.005.patch, MAPREDUCE-6192.006.patch, MAPREDUCE-6192.007.patch, MAPREDUCE-6192.branch-2.007.patch Create a unit test that will automatically compare the fields in the various MapReduce related classes and mapred-default.xml. It should throw an error if a property is missing in either the class or the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527701#comment-14527701
]
Naganarasimha G R commented on MAPREDUCE-6304:
--
hi [~Wangda],
I meant actually only deletion of {{value/value}} for example
{quote}
property
namemapreduce.job.am.node-label-expression/name
{color:red} !-- value/value --{color}
descriptionOverrides mapreduce.job.node-label-expression for
application
master containers
/description
/property
{quote}
which is similar to the existing configs like yarn.ipc.*.factory.class,
yarn.resourcemanager.cluster-id,yarn.resourcemanager.ha.rm-ids,yarn.resourcemanager.ha.id
In general your approach is fine but as we can make use of the above approach
of configuration did not want to bring in additional default configs. thoughts
?
Specifying node labels when submitting MR jobs
--
Key: MAPREDUCE-6304
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Jian Fang
Assignee: Naganarasimha G R
Fix For: 2.8.0
Attachments: MAPREDUCE-6304.20150410-1.patch,
MAPREDUCE-6304.20150411-1.patch, MAPREDUCE-6304.20150501-1.patch
Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify
node labels when submitting MR jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6259) IllegalArgumentException due to missing job submit time
[
https://issues.apache.org/jira/browse/MAPREDUCE-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527808#comment-14527808
]
zhihai xu commented on MAPREDUCE-6259:
--
thanks [~jlowe] for the review and commit.
IllegalArgumentException due to missing job submit time
---
Key: MAPREDUCE-6259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6259
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: zhihai xu
Assignee: zhihai xu
Fix For: 2.7.1
Attachments: MAPREDUCE-6259.000.patch
-1 job submit time cause IllegalArgumentException when parse the Job history
file name and JOB_INIT_FAILED cause -1 job submit time in JobIndexInfo.
We found the following job history file name which cause
IllegalArgumentException when parse the job status in the job history file
name.
{code}
job_1418398645407_115853--1-worun-kafka%2Dto%2Dhdfs%5Btwo%5D%5B15+topic%28s%29%5D-1423572836007-0-0-FAILED-root.journaling-1423572836007.jhist
{code}
The stack trace for the IllegalArgumentException is
{code}
2015-02-10 04:54:01,863 WARN org.apache.hadoop.mapreduce.v2.hs.PartialJob:
Exception while parsing job state. Defaulting to KILLED
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.v2.api.records.JobState.0
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.v2.api.records.JobState.valueOf(JobState.java:21)
at
org.apache.hadoop.mapreduce.v2.hs.PartialJob.getState(PartialJob.java:82)
at
org.apache.hadoop.mapreduce.v2.hs.PartialJob.init(PartialJob.java:59)
at
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getAllPartialJobs(CachedHistoryStorage.java:159)
at
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getPartialJobs(CachedHistoryStorage.java:173)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistory.getPartialJobs(JobHistory.java:284)
at
org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:212)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
[jira] [Commented] (MAPREDUCE-6353) Divide by zero error in MR AM when calculating available containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527840#comment-14527840
]
Hadoop QA commented on MAPREDUCE-6353:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 42s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 10s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 57s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 8m 57s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 45m 24s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730335/MAPREDUCE-6353.001.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 551615f |
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5641/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5641/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5641/console |
This message was automatically generated.
Divide by zero error in MR AM when calculating available containers
---
Key: MAPREDUCE-6353
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6353
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Attachments: MAPREDUCE-6353.001.patch
When running a sleep job with zero CPU vcores i see the following exception
2015-04-30 06:41:06,954 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN
CONTACTING RM.
java.lang.ArithmeticException: / by zero
at
org.apache.hadoop.mapreduce.v2.app.rm.ResourceCalculatorUtils.computeAvailableContainers(ResourceCalculatorUtils.java:38)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assign(RMContainerAllocator.java:947)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.access$200(RMContainerAllocator.java:840)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:247)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:282)
at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527862#comment-14527862
]
Hadoop QA commented on MAPREDUCE-6174:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 52s | The applied patch generated 4
new checkstyle issues (total was 8, now 10). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 40s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 38m 8s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730348/MAPREDUCE-6174.002.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 551615f |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5642/console |
This message was automatically generated.
Combine common stream code into parent class for InMemoryMapOutput and
OnDiskMapOutput.
---
Key: MAPREDUCE-6174
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 3.0.0, 2.6.0
Reporter: Eric Payne
Assignee: Eric Payne
Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.v1.txt
Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing
similar things with regards to IFile streams.
In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are
different from 3rd-party implementations, this JIRA will make them subclass a
common class (see
https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6353) Divide by zero error in MR AM when calculating available containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anubhav Dhoot updated MAPREDUCE-6353: - Status: Patch Available (was: Open) Divide by zero error in MR AM when calculating available containers --- Key: MAPREDUCE-6353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6353 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: MAPREDUCE-6353.001.patch When running a sleep job with zero CPU vcores i see the following exception 2015-04-30 06:41:06,954 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN CONTACTING RM. java.lang.ArithmeticException: / by zero at org.apache.hadoop.mapreduce.v2.app.rm.ResourceCalculatorUtils.computeAvailableContainers(ResourceCalculatorUtils.java:38) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assign(RMContainerAllocator.java:947) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.access$200(RMContainerAllocator.java:840) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:247) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:282) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6353) Divide by zero error in MR AM when calculating available containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anubhav Dhoot updated MAPREDUCE-6353: - Attachment: MAPREDUCE-6353.001.patch Fix that handles zero resource value Divide by zero error in MR AM when calculating available containers --- Key: MAPREDUCE-6353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6353 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: MAPREDUCE-6353.001.patch When running a sleep job with zero CPU vcores i see the following exception 2015-04-30 06:41:06,954 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN CONTACTING RM. java.lang.ArithmeticException: / by zero at org.apache.hadoop.mapreduce.v2.app.rm.ResourceCalculatorUtils.computeAvailableContainers(ResourceCalculatorUtils.java:38) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assign(RMContainerAllocator.java:947) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.access$200(RMContainerAllocator.java:840) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:247) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:282) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6192) Create unit test to automatically compare MR related classes and mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527685#comment-14527685
]
Robert Kanter commented on MAPREDUCE-6192:
--
[~rchiang], it works fine on trunk, but it looks like there's some differences
in branch-2. Can you look into this?
{noformat}
Running org.apache.hadoop.mapreduce.TestMapreduceConfigFields
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.624 sec
FAILURE! - in org.apache.hadoop.mapreduce.TestMapreduceConfigFields
testCompareXmlAgainstConfigurationClass(org.apache.hadoop.mapreduce.TestMapreduceConfigFields)
Time elapsed: 0.497 sec FAILURE!
java.lang.AssertionError: mapred-default.xml has 4 properties missing in
interface org.apache.hadoop.mapreduce.MRJobConfig interface
org.apache.hadoop.mapreduce.MRConfig class
org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig class
org.apache.hadoop.mapred.ShuffleHandler class
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat class
org.apache.hadoop.mapreduce.lib.input.FileInputFormat class
org.apache.hadoop.mapreduce.Job class
org.apache.hadoop.mapreduce.lib.input.NLineInputFormat class
org.apache.hadoop.mapred.JobConf class
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass(TestConfigurationFieldsBase.java:468)
...
{noformat}
Create unit test to automatically compare MR related classes and
mapred-default.xml
---
Key: MAPREDUCE-6192
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6192
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
Labels: supportability
Attachments: MAPREDUCE-6192.001.patch, MAPREDUCE-6192.002.patch,
MAPREDUCE-6192.003.patch, MAPREDUCE-6192.004.patch, MAPREDUCE-6192.005.patch,
MAPREDUCE-6192.006.patch, MAPREDUCE-6192.007.patch
Create a unit test that will automatically compare the fields in the various
MapReduce related classes and mapred-default.xml. It should throw an error if
a property is missing in either the class or the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[
https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527720#comment-14527720
]
Tsuyoshi Ozawa commented on MAPREDUCE-6165:
---
+1, committing this.
[JDK8] TestCombineFileInputFormat failed on JDK8
Key: MAPREDUCE-6165
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Wei Yan
Assignee: Akira AJISAKA
Priority: Minor
Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch,
MAPREDUCE-6165-003.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-004.patch,
MAPREDUCE-6165-reproduce.patch
The error msg:
{noformat}
testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
Time elapsed: 2.487 sec FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:1
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911)
testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
Time elapsed: 0.985 sec FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:1
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[
https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated MAPREDUCE-6165:
--
Resolution: Fixed
Fix Version/s: 2.8.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed this to trunk and branch-2. Thanks [~ajisakaa] for your contribution
and thanks [~rkanter] for your review!
[JDK8] TestCombineFileInputFormat failed on JDK8
Key: MAPREDUCE-6165
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Wei Yan
Assignee: Akira AJISAKA
Priority: Minor
Fix For: 2.8.0
Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch,
MAPREDUCE-6165-003.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-004.patch,
MAPREDUCE-6165-reproduce.patch
The error msg:
{noformat}
testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
Time elapsed: 2.487 sec FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:1
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911)
testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
Time elapsed: 0.985 sec FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:1
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527790#comment-14527790 ] Karthik Kambatla commented on MAPREDUCE-6354: - I have the same concerns as Allen. We should also improve existing logs to avoid logging for every connection. bq. Allen, would it mitigate your concerns if this were logged with a separately configurable logger, e.g.: ShuffleHandlerAuditLogger? That way users could configure it on when they want to audit shuffle transfers or off when they don't. +1. Sounds like a good compromise. One could look at the shuffle connections if they want to and roll up as aggressively as desired. shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6192) Create unit test to automatically compare MR related classes and mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527922#comment-14527922
]
Hadoop QA commented on MAPREDUCE-6192:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 6s | Pre-patch branch-2 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | patch | 0m 54s | The patch command could not apply
the patch. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730361/MAPREDUCE-6192.branch-2.007.patch
|
| Optional Tests | javac unit findbugs checkstyle |
| git revision | branch-2 / aefadb8 |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5643/console |
This message was automatically generated.
Create unit test to automatically compare MR related classes and
mapred-default.xml
---
Key: MAPREDUCE-6192
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6192
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
Labels: supportability
Attachments: MAPREDUCE-6192.001.patch, MAPREDUCE-6192.002.patch,
MAPREDUCE-6192.003.patch, MAPREDUCE-6192.004.patch, MAPREDUCE-6192.005.patch,
MAPREDUCE-6192.006.patch, MAPREDUCE-6192.007.patch,
MAPREDUCE-6192.branch-2.007.patch
Create a unit test that will automatically compare the fields in the various
MapReduce related classes and mapred-default.xml. It should throw an error if
a property is missing in either the class or the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6337) add a mode to replay MR job history files to the timeline service
[ https://issues.apache.org/jira/browse/MAPREDUCE-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527436#comment-14527436 ] Zhijie Shen commented on MAPREDUCE-6337: Sangjin, thanks for the patch. Here're some high level comments: 1. I've a concern about the way to replay MR job history. Now the approach is to read all the history and convert it into entities, and write it once for a job. This may not reflect the realistic workload pattern, at least different from the current way MR puts the timeline data. Shall we add one more option to control 1) put all entities once per job, 2) put one entity per call and 3) repeatedly put entity per event. The third option is more close to current MR putting method, though it doesn't mean to be the optimal approach. Perhaps different options may affect the write performance. 2. TimelineEntityConverter is doing something similar to what we've done in MAPREDUCE-6237, but in a bit different way, and the entity composition is also slightly different, such as saving counter in metric. I think the reason why MAPREDUCE-6237 may not be reused is that we convert from Info to entity while MAPREDUCE-6237 converts Event to entity. Perhaps we want to refactor the code and consolidate the conversion later. add a mode to replay MR job history files to the timeline service - Key: MAPREDUCE-6337 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6337 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-6337-YARN-2928.001.patch, YARN-3438.000.patch The subtask covers the work on top of YARN-3437 to add a mode to replay MR job history files to the timeline service storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6259) IllegalArgumentException due to missing job submit time
[
https://issues.apache.org/jira/browse/MAPREDUCE-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527407#comment-14527407
]
Hudson commented on MAPREDUCE-6259:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #7727 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7727/])
MAPREDUCE-6259. IllegalArgumentException due to missing job submit time.
Contributed by zhihai xu (jlowe: rev bf70c5ae2824a9139c1aa9d7c14020018881cec2)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/AMStartedEvent.java
IllegalArgumentException due to missing job submit time
---
Key: MAPREDUCE-6259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6259
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: zhihai xu
Assignee: zhihai xu
Fix For: 2.7.1
Attachments: MAPREDUCE-6259.000.patch
-1 job submit time cause IllegalArgumentException when parse the Job history
file name and JOB_INIT_FAILED cause -1 job submit time in JobIndexInfo.
We found the following job history file name which cause
IllegalArgumentException when parse the job status in the job history file
name.
{code}
job_1418398645407_115853--1-worun-kafka%2Dto%2Dhdfs%5Btwo%5D%5B15+topic%28s%29%5D-1423572836007-0-0-FAILED-root.journaling-1423572836007.jhist
{code}
The stack trace for the IllegalArgumentException is
{code}
2015-02-10 04:54:01,863 WARN org.apache.hadoop.mapreduce.v2.hs.PartialJob:
Exception while parsing job state. Defaulting to KILLED
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.v2.api.records.JobState.0
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.v2.api.records.JobState.valueOf(JobState.java:21)
at
org.apache.hadoop.mapreduce.v2.hs.PartialJob.getState(PartialJob.java:82)
at
org.apache.hadoop.mapreduce.v2.hs.PartialJob.init(PartialJob.java:59)
at
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getAllPartialJobs(CachedHistoryStorage.java:159)
at
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getPartialJobs(CachedHistoryStorage.java:173)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistory.getPartialJobs(JobHistory.java:284)
at
org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:212)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[
https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527564#comment-14527564
]
Hadoop QA commented on MAPREDUCE-6354:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 28s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 33s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 18s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 36s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 19s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 35m 21s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730294/MAPREDUCE-6354.3.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 71f4de2 |
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5640/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5640/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5640/console |
This message was automatically generated.
shuffle handler should log connection info
--
Key: MAPREDUCE-6354
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chang Li
Assignee: Chang Li
Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch,
MAPREDUCE-6354.patch
currently, shuffle handler only log connection info in debug mode, we want to
log that info in a more concise way
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6337) add a mode to replay MR job history files to the timeline service
[ https://issues.apache.org/jira/browse/MAPREDUCE-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527577#comment-14527577 ] Zhijie Shen commented on MAPREDUCE-6337: My bad. I meant MAPREDUCE-6327, where we implemented MR writing to ATS v2. add a mode to replay MR job history files to the timeline service - Key: MAPREDUCE-6337 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6337 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-6337-YARN-2928.001.patch The subtask covers the work on top of YARN-3437 to add a mode to replay MR job history files to the timeline service storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated MAPREDUCE-6304: -- Status: Open (was: Patch Available) Specifying node labels when submitting MR jobs -- Key: MAPREDUCE-6304 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Jian Fang Assignee: Naganarasimha G R Fix For: 2.8.0 Attachments: MAPREDUCE-6304.20150410-1.patch, MAPREDUCE-6304.20150411-1.patch, MAPREDUCE-6304.20150501-1.patch Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify node labels when submitting MR jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527488#comment-14527488 ] Wangda Tan commented on MAPREDUCE-6304: --- [~Naganarasimha], remove from mapred-default.xml is not good. mapred-default.xml is a way to document all options we have, how about make its default to be USE_QUEUE_DEFINED_DEFAULT or better name, it's not not specified or null actually, it's using queue defined node label expression. Thoughts? Specifying node labels when submitting MR jobs -- Key: MAPREDUCE-6304 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Jian Fang Assignee: Naganarasimha G R Fix For: 2.8.0 Attachments: MAPREDUCE-6304.20150410-1.patch, MAPREDUCE-6304.20150411-1.patch, MAPREDUCE-6304.20150501-1.patch Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify node labels when submitting MR jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6337) add a mode to replay MR job history files to the timeline service
[ https://issues.apache.org/jira/browse/MAPREDUCE-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527571#comment-14527571 ] Sangjin Lee commented on MAPREDUCE-6337: Thanks for the comments [~zjshen]. I think #1 is a good idea. While we will never mimic the real life usage precisely (and one man's real life may be different from another's), we could improve the pattern some more. Let me work on that a little bit. Regarding your second point, I don't think MAPREDUCE-6237 is the right JIRA. Were you thinking of some other JIRA? add a mode to replay MR job history files to the timeline service - Key: MAPREDUCE-6337 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6337 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-6337-YARN-2928.001.patch The subtask covers the work on top of YARN-3437 to add a mode to replay MR job history files to the timeline service storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6337) add a mode to replay MR job history files to the timeline service
[ https://issues.apache.org/jira/browse/MAPREDUCE-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6337: --- Attachment: (was: YARN-3438.000.patch) add a mode to replay MR job history files to the timeline service - Key: MAPREDUCE-6337 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6337 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: MAPREDUCE-6337-YARN-2928.001.patch The subtask covers the work on top of YARN-3437 to add a mode to replay MR job history files to the timeline service storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[
https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527319#comment-14527319
]
Hadoop QA commented on MAPREDUCE-6279:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 34s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 58s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 30s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 45m 22s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730255/MAPREDUCE-6279.v3.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3fe79e1 |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5637/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5637/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5637/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5637/console |
This message was automatically generated.
AM should explicity exit JVM after all services have stopped
Key: MAPREDUCE-6279
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.5.0
Reporter: Jason Lowe
Assignee: Eric Payne
Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt,
MAPREDUCE-6279.v3.patch
Occasionally the MapReduce AM can get stuck trying to shut down.
MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been
fixed, but this can also occur with uber jobs if the task code inadvertently
leaves non-daemon threads lingering.
We should explicitly shutdown the JVM after the MapReduce AM has unregistered
and all services have been stopped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6354: Attachment: MAPREDUCE-6354.3.patch shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[
https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527215#comment-14527215
]
Chang Li commented on MAPREDUCE-6354:
-
[~jlowe] Thanks for review! Have updated my patch. Have tested the logging and
the connection info is logged as following
{code}
shuffle for job_1430770193094_0002 reducer 0,
maps=[attempt_1430770193094_0002_m_00_0,
attempt_1430770193094_0002_m_01_0]
{code}
shuffle handler should log connection info
--
Key: MAPREDUCE-6354
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chang Li
Assignee: Chang Li
Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.patch
currently, shuffle handler only log connection info in debug mode, we want to
log that info in a more concise way
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6259) IllegalArgumentException due to missing job submit time
[
https://issues.apache.org/jira/browse/MAPREDUCE-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-6259:
--
Summary: IllegalArgumentException due to missing job submit time (was: -1
job submit time cause IllegalArgumentException when parse the Job history file
name and JOB_INIT_FAILED cause -1 job submit time in JobIndexInfo.)
IllegalArgumentException due to missing job submit time
---
Key: MAPREDUCE-6259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6259
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6259.000.patch
-1 job submit time cause IllegalArgumentException when parse the Job history
file name and JOB_INIT_FAILED cause -1 job submit time in JobIndexInfo.
We found the following job history file name which cause
IllegalArgumentException when parse the job status in the job history file
name.
{code}
job_1418398645407_115853--1-worun-kafka%2Dto%2Dhdfs%5Btwo%5D%5B15+topic%28s%29%5D-1423572836007-0-0-FAILED-root.journaling-1423572836007.jhist
{code}
The stack trace for the IllegalArgumentException is
{code}
2015-02-10 04:54:01,863 WARN org.apache.hadoop.mapreduce.v2.hs.PartialJob:
Exception while parsing job state. Defaulting to KILLED
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.v2.api.records.JobState.0
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.v2.api.records.JobState.valueOf(JobState.java:21)
at
org.apache.hadoop.mapreduce.v2.hs.PartialJob.getState(PartialJob.java:82)
at
org.apache.hadoop.mapreduce.v2.hs.PartialJob.init(PartialJob.java:59)
at
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getAllPartialJobs(CachedHistoryStorage.java:159)
at
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getPartialJobs(CachedHistoryStorage.java:173)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistory.getPartialJobs(JobHistory.java:284)
at
org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:212)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
[jira] [Commented] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[
https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527394#comment-14527394
]
Robert Kanter commented on MAPREDUCE-6165:
--
I ran the tests in hadoop-mapreduce-client-jobclient and they all passed
locally (took about 1.5 hours). I've kicked off Jenkins again just to be extra
sure, but it looks like this patch is probably fine.
[JDK8] TestCombineFileInputFormat failed on JDK8
Key: MAPREDUCE-6165
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Wei Yan
Assignee: Akira AJISAKA
Priority: Minor
Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch,
MAPREDUCE-6165-003.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-004.patch,
MAPREDUCE-6165-reproduce.patch
The error msg:
{noformat}
testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
Time elapsed: 2.487 sec FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:1
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911)
testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
Time elapsed: 0.985 sec FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:1
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6354: Attachment: MAPREDUCE-6354.2.patch shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6259) -1 job submit time cause IllegalArgumentException when parse the Job history file name and JOB_INIT_FAILED cause -1 job submit time in JobIndexInfo.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527236#comment-14527236
]
Jason Lowe commented on MAPREDUCE-6259:
---
Apologies for the long delay.
+1 lgtm. Committing this.
-1 job submit time cause IllegalArgumentException when parse the Job history
file name and JOB_INIT_FAILED cause -1 job submit time in JobIndexInfo.
Key: MAPREDUCE-6259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6259
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6259.000.patch
-1 job submit time cause IllegalArgumentException when parse the Job history
file name and JOB_INIT_FAILED cause -1 job submit time in JobIndexInfo.
We found the following job history file name which cause
IllegalArgumentException when parse the job status in the job history file
name.
{code}
job_1418398645407_115853--1-worun-kafka%2Dto%2Dhdfs%5Btwo%5D%5B15+topic%28s%29%5D-1423572836007-0-0-FAILED-root.journaling-1423572836007.jhist
{code}
The stack trace for the IllegalArgumentException is
{code}
2015-02-10 04:54:01,863 WARN org.apache.hadoop.mapreduce.v2.hs.PartialJob:
Exception while parsing job state. Defaulting to KILLED
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.v2.api.records.JobState.0
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.v2.api.records.JobState.valueOf(JobState.java:21)
at
org.apache.hadoop.mapreduce.v2.hs.PartialJob.getState(PartialJob.java:82)
at
org.apache.hadoop.mapreduce.v2.hs.PartialJob.init(PartialJob.java:59)
at
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getAllPartialJobs(CachedHistoryStorage.java:159)
at
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getPartialJobs(CachedHistoryStorage.java:173)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistory.getPartialJobs(JobHistory.java:284)
at
org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:212)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
[jira] [Commented] (MAPREDUCE-5649) Reduce cannot use more than 2G memory for the final merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527133#comment-14527133 ] Gera Shegalov commented on MAPREDUCE-5649: -- Thanks [~jlowe] for review and commit. Thanks [~jerrychenhf] for your patch in MAPREDUCE-4883, I resolve it as a duplicate. Reduce cannot use more than 2G memory for the final merge -- Key: MAPREDUCE-5649 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5649 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: stanley shi Assignee: Gera Shegalov Fix For: 2.8.0 Attachments: MAPREDUCE-5649.001.patch, MAPREDUCE-5649.002.patch, MAPREDUCE-5649.003.patch In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in the finalMerge method: int maxInMemReduce = (int)Math.min( Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE); This means no matter how much memory user has, reducer will not retain more than 2G data in memory before the reduce phase starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[
https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527337#comment-14527337
]
Hadoop QA commented on MAPREDUCE-6354:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 0s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 43s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 49s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 20s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 0m 40s | The patch appears to introduce 2
new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 20s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 36m 26s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-mapreduce-client-shuffle |
| | Nullcheck of jobQ at line 746 of value previously dereferenced in
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext,
MessageEvent) At ShuffleHandler.java:746 of value previously dereferenced in
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext,
MessageEvent) At ShuffleHandler.java:[line 746] |
| | Nullcheck of reduceQ at line 746 of value previously dereferenced in
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext,
MessageEvent) At ShuffleHandler.java:746 of value previously dereferenced in
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ChannelHandlerContext,
MessageEvent) At ShuffleHandler.java:[line 746] |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730274/MAPREDUCE-6354.2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / bf70c5a |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
|
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5638/console |
This message was automatically generated.
shuffle handler should log connection info
--
Key: MAPREDUCE-6354
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chang Li
Assignee: Chang Li
Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.patch
currently, shuffle handler only log connection info in debug mode, we want to
log that info in a more concise way
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6355) 2.5 client cannot communicate with 2.5 job on 2.6 cluster
Jason Lowe created MAPREDUCE-6355: - Summary: 2.5 client cannot communicate with 2.5 job on 2.6 cluster Key: MAPREDUCE-6355 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6355 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Jason Lowe Trying to run a job on a Hadoop 2.6 cluster from a Hadoop 2.5 client submitting a job that uses Hadoop 2.5 jars results in a job that succeeds but the client cannot communicate with the AM while the job is running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4883) Reducer's Maximum Shuffle Buffer Size should be enlarged for 64bit JVM
[
https://issues.apache.org/jira/browse/MAPREDUCE-4883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-4883:
-
Resolution: Duplicate
Target Version/s: (was: )
Status: Resolved (was: Patch Available)
This idea from this patch was folded into MAPREDUCE-5649. Thanks [~jerrychenhf]!
Reducer's Maximum Shuffle Buffer Size should be enlarged for 64bit JVM
--
Key: MAPREDUCE-4883
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4883
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 0.20.2, 1.0.3
Environment: Especially for 64bit JVM
Reporter: Lijie Xu
Assignee: Jerry Chen
Labels: patch
Attachments: MAPREDUCE-4883.patch
Original Estimate: 12h
Remaining Estimate: 12h
In hadoop-0.20.2, hadoop-1.0.3 or other versions, reducer's shuffle buffer
size cannot exceed 2048MB (i.e., Integer.MAX_VALUE). This is reasonable for
32bit JVM.
But for 64bit JVM, although reducer's JVM size can be set more than 2048MB
(e.g., mapred.child.java.opts=-Xmx4000m), the heap size used for shuffle
buffer is at most 2048MB * maxInMemCopyUse (default 0.7) not 4000MB *
maxInMemCopyUse.
So the pointed piece of code in ReduceTask.java needs modification for 64bit
JVM.
---
private final long maxSize;
private final long maxSingleShuffleLimit;
private long size = 0;
private Object dataAvailable = new Object();
private long fullSize = 0;
private int numPendingRequests = 0;
private int numRequiredMapOutputs = 0;
private int numClosed = 0;
private boolean closed = false;
public ShuffleRamManager(Configuration conf) throws IOException {
final float maxInMemCopyUse =
conf.getFloat(mapred.job.shuffle.input.buffer.percent, 0.70f);
if (maxInMemCopyUse 1.0 || maxInMemCopyUse 0.0) {
throw new IOException(mapred.job.shuffle.input.buffer.percent +
maxInMemCopyUse);
}
// Allow unit tests to fix Runtime memory
-- maxSize = (int)(conf.getInt(mapred.job.reduce.total.mem.bytes,
--(int)Math.min(Runtime.getRuntime().maxMemory(), Integer.MAX_VALUE))
-- * maxInMemCopyUse);
maxSingleShuffleLimit = (long)(maxSize *
MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION);
LOG.info(ShuffleRamManager: MemoryLimit= + maxSize +
, MaxSingleShuffleLimit= + maxSingleShuffleLimit);
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6259) IllegalArgumentException due to missing job submit time
[
https://issues.apache.org/jira/browse/MAPREDUCE-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-6259:
--
Resolution: Fixed
Fix Version/s: 2.7.1
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks, [~zxu]! I committed this to trunk, branch-2, and branch-2.7.
IllegalArgumentException due to missing job submit time
---
Key: MAPREDUCE-6259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6259
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: zhihai xu
Assignee: zhihai xu
Fix For: 2.7.1
Attachments: MAPREDUCE-6259.000.patch
-1 job submit time cause IllegalArgumentException when parse the Job history
file name and JOB_INIT_FAILED cause -1 job submit time in JobIndexInfo.
We found the following job history file name which cause
IllegalArgumentException when parse the job status in the job history file
name.
{code}
job_1418398645407_115853--1-worun-kafka%2Dto%2Dhdfs%5Btwo%5D%5B15+topic%28s%29%5D-1423572836007-0-0-FAILED-root.journaling-1423572836007.jhist
{code}
The stack trace for the IllegalArgumentException is
{code}
2015-02-10 04:54:01,863 WARN org.apache.hadoop.mapreduce.v2.hs.PartialJob:
Exception while parsing job state. Defaulting to KILLED
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.v2.api.records.JobState.0
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.v2.api.records.JobState.valueOf(JobState.java:21)
at
org.apache.hadoop.mapreduce.v2.hs.PartialJob.getState(PartialJob.java:82)
at
org.apache.hadoop.mapreduce.v2.hs.PartialJob.init(PartialJob.java:59)
at
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getAllPartialJobs(CachedHistoryStorage.java:159)
at
org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getPartialJobs(CachedHistoryStorage.java:173)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistory.getPartialJobs(JobHistory.java:284)
at
org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:212)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
[jira] [Commented] (MAPREDUCE-6355) 2.5 client cannot communicate with 2.5 job on 2.6 cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-6355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527367#comment-14527367
]
Jason Lowe commented on MAPREDUCE-6355:
---
Here's a snippet from a 2.5 client submitting a sleep job using 2.5 jars for
the job on a 2.6 cluster:
{noformat}
15/05/04 21:34:00 INFO mapreduce.Job: Running job: job_1429081185257_495794
15/05/04 21:34:13 WARN ipc.Client: Exception encountered while connecting to
the server :
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.
15/05/04 21:34:13 WARN ipc.Client: Exception encountered while connecting to
the server :
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.
15/05/04 21:34:14 WARN ipc.Client: Exception encountered while connecting to
the server :
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.
15/05/04 21:34:14 WARN ipc.Client: Exception encountered while connecting to
the server :
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
DIGEST-MD5: digest response format violation. Mismatched response.
[... many mismatched response errors removed for brevity ...]
15/05/04 21:34:39 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
15/05/04 21:34:39 INFO mapreduce.Job: Job job_1429081185257_495794 running in
uber mode : false
15/05/04 21:34:39 INFO mapreduce.Job: map 100% reduce 100%
15/05/04 21:34:39 INFO mapreduce.Job: Job job_1429081185257_495794 completed
successfully
15/05/04 21:34:39 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=28
FILE: Number of bytes written=249357
[... and the rest of the job counters ...]
{noformat}
2.5 client cannot communicate with 2.5 job on 2.6 cluster
-
Key: MAPREDUCE-6355
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6355
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Jason Lowe
Trying to run a job on a Hadoop 2.6 cluster from a Hadoop 2.5 client
submitting a job that uses Hadoop 2.5 jars results in a job that succeeds but
the client cannot communicate with the AM while the job is running.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6165) [JDK8] TestCombineFileInputFormat failed on JDK8
[
https://issues.apache.org/jira/browse/MAPREDUCE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527672#comment-14527672
]
Hadoop QA commented on MAPREDUCE-6165:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 40s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 43s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 57s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 3s | The applied patch generated 1
new checkstyle issues (total was 30, now 29). |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 57s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 48s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 100m 34s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 140m 34s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12729508/MAPREDUCE-6165-004.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0d6aa5d |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/console |
This message was automatically generated.
[JDK8] TestCombineFileInputFormat failed on JDK8
Key: MAPREDUCE-6165
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6165
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Wei Yan
Assignee: Akira AJISAKA
Priority: Minor
Attachments: MAPREDUCE-6165-001.patch, MAPREDUCE-6165-002.patch,
MAPREDUCE-6165-003.patch, MAPREDUCE-6165-003.patch, MAPREDUCE-6165-004.patch,
MAPREDUCE-6165-reproduce.patch
The error msg:
{noformat}
testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
Time elapsed: 2.487 sec FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:1
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911)
testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
Time elapsed: 0.985 sec FAILURE!
junit.framework.AssertionFailedError: expected:2 but was:1
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated MAPREDUCE-6174:
--
Attachment: MAPREDUCE-6174.002.patch
Thank you, [~jira.shegalov], for your help with this issue. Version 2 of the
patch now contains a test that verifies that {{inMemoryMapOutput}} and
{{OnDiskMapOutput}} are derived from the new class.
Combine common stream code into parent class for InMemoryMapOutput and
OnDiskMapOutput.
---
Key: MAPREDUCE-6174
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 3.0.0, 2.6.0
Reporter: Eric Payne
Assignee: Eric Payne
Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.v1.txt
Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing
similar things with regards to IFile streams.
In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are
different from 3rd-party implementations, this JIRA will make them subclass a
common class (see
https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
