[jira] [Commented] (MAPREDUCE-6607) .staging dir is not cleanup if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15101498#comment-15101498
]
Hadoop QA commented on MAPREDUCE-6607:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 12s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 49s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 54s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12782477/MAPREDUCE-6607.01.patch
|
| JIRA Issue | MAPREDUCE-6607 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a5dcf0affe81 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1da762c |
| Default Java | 1.7.0_91
[jira] [Updated] (MAPREDUCE-6338) MR AppMaster does not honor ephemeral port range
[ https://issues.apache.org/jira/browse/MAPREDUCE-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandr updated MAPREDUCE-6338: Attachment: MAPREDUCE-6338.003.patch cleared > MR AppMaster does not honor ephemeral port range > > > Key: MAPREDUCE-6338 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6338 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 2.6.0 >Reporter: Frank Nguyen >Assignee: Frank Nguyen > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-6338.002.patch, MAPREDUCE-6338.003.patch > > > The MR AppMaster should only use port ranges defined in the > yarn.app.mapreduce.am.job.client.port-range property. On initial startup of > the MRAppMaster, it does use the port range defined in the property. > However, it also opens up a listener on a random ephemeral port. This is not > the Jetty listener. It is another listener opened by the MRAppMaster via > another thread and is recognized by the RM. Other nodes will try to > communicate to it via that random port. With firewall settings on, the MR > job will fail because the random port is not opened. This problem has caused > others to have all OS ephemeral ports opened to have MR jobs run. > This is related to MAPREDUCE-4079 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6338) MR AppMaster does not honor ephemeral port range
[ https://issues.apache.org/jira/browse/MAPREDUCE-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandr updated MAPREDUCE-6338: Status: Patch Available (was: Open) > MR AppMaster does not honor ephemeral port range > > > Key: MAPREDUCE-6338 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6338 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 2.6.0 >Reporter: Frank Nguyen >Assignee: Frank Nguyen > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-6338.002.patch, MAPREDUCE-6338.003.patch > > > The MR AppMaster should only use port ranges defined in the > yarn.app.mapreduce.am.job.client.port-range property. On initial startup of > the MRAppMaster, it does use the port range defined in the property. > However, it also opens up a listener on a random ephemeral port. This is not > the Jetty listener. It is another listener opened by the MRAppMaster via > another thread and is recognized by the RM. Other nodes will try to > communicate to it via that random port. With firewall settings on, the MR > job will fail because the random port is not opened. This problem has caused > others to have all OS ephemeral ports opened to have MR jobs run. > This is related to MAPREDUCE-4079 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6338) MR AppMaster does not honor ephemeral port range
[ https://issues.apache.org/jira/browse/MAPREDUCE-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandr updated MAPREDUCE-6338: Status: Open (was: Patch Available) > MR AppMaster does not honor ephemeral port range > > > Key: MAPREDUCE-6338 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6338 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 2.6.0 >Reporter: Frank Nguyen >Assignee: Frank Nguyen > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-6338.002.patch, MAPREDUCE-6338.003.patch > > > The MR AppMaster should only use port ranges defined in the > yarn.app.mapreduce.am.job.client.port-range property. On initial startup of > the MRAppMaster, it does use the port range defined in the property. > However, it also opens up a listener on a random ephemeral port. This is not > the Jetty listener. It is another listener opened by the MRAppMaster via > another thread and is recognized by the RM. Other nodes will try to > communicate to it via that random port. With firewall settings on, the MR > job will fail because the random port is not opened. This problem has caused > others to have all OS ephemeral ports opened to have MR jobs run. > This is related to MAPREDUCE-4079 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6338) MR AppMaster does not honor ephemeral port range
[
https://issues.apache.org/jira/browse/MAPREDUCE-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102027#comment-15102027
]
Hadoop QA commented on MAPREDUCE-6338:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
(total was 27, now 28). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 43s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 54s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 2s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12782545/MAPREDUCE-6338.003.patch
|
| JIRA Issue | MAPREDUCE-6338 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6ef076597e6b 3.13.0-36-lowlatency #63-U
[jira] [Commented] (MAPREDUCE-6554) MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory
[
https://issues.apache.org/jira/browse/MAPREDUCE-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102053#comment-15102053
]
Jason Lowe commented on MAPREDUCE-6554:
---
+1 lgtm. The checkstyle issue was present before the patch as well, and the
whitespace, tab, and license warnings are not due to this patch.
Committing this.
> MRAppMaster servicestart failing with NPE in
> MRAppMaster#parsePreviousJobHistory
> -
>
> Key: MAPREDUCE-6554
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6554
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: 0001-MAPREDUCE-6554.patch, 0002-MAPREDUCE-6554.patch,
> 0003-MAPREDUCE-6554.patch, MAPREDUCE-6554-branch-2.7.001.patch,
> MAPREDUCE-6554-branch-2.7.patch
>
>
> Create scenario so that MR app master gets preempted.
> On next MRAppMaster launch tried to recover previous job history file
> {{MRAppMaster#parsePreviousJobHistory}}
> {noformat}
> 2015-11-21 13:52:27,722 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED;
> cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at java.io.StringReader.(StringReader.java:50)
> at org.apache.avro.Schema$Parser.parse(Schema.java:917)
> at org.apache.avro.Schema.parse(Schema.java:966)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:75)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1256)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1225)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1087)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1570)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1566)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1499)
> 2015-11-21 13:52:27,725 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 0
> {noformat}
> EventReader(EventReader stream)
> {noformat}
> this.version = in.readLine();
> ...
> Schema myschema = new
> SpecificData(Event.class.getClassLoader()).getSchema(Event.class);
> this.schema = Schema.parse(in.readLine());
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6554) MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory
[
https://issues.apache.org/jira/browse/MAPREDUCE-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102075#comment-15102075
]
Hudson commented on MAPREDUCE-6554:
---
FAILURE: Integrated in Hadoop-trunk-Commit #9120 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9120/])
MAPREDUCE-6554. MRAppMaster servicestart failing with NPE in (jlowe: rev
9fbd579ab50f051c8ad63d316ca5a3e24822dcdd)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
* hadoop-mapreduce-project/CHANGES.txt
> MRAppMaster servicestart failing with NPE in
> MRAppMaster#parsePreviousJobHistory
> -
>
> Key: MAPREDUCE-6554
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6554
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: 0001-MAPREDUCE-6554.patch, 0002-MAPREDUCE-6554.patch,
> 0003-MAPREDUCE-6554.patch, MAPREDUCE-6554-branch-2.7.001.patch,
> MAPREDUCE-6554-branch-2.7.patch
>
>
> Create scenario so that MR app master gets preempted.
> On next MRAppMaster launch tried to recover previous job history file
> {{MRAppMaster#parsePreviousJobHistory}}
> {noformat}
> 2015-11-21 13:52:27,722 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED;
> cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at java.io.StringReader.(StringReader.java:50)
> at org.apache.avro.Schema$Parser.parse(Schema.java:917)
> at org.apache.avro.Schema.parse(Schema.java:966)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:75)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1256)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1225)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1087)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1570)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1566)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1499)
> 2015-11-21 13:52:27,725 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 0
> {noformat}
> EventReader(EventReader stream)
> {noformat}
> this.version = in.readLine();
> ...
> Schema myschema = new
> SpecificData(Event.class.getClassLoader()).getSchema(Event.class);
> this.schema = Schema.parse(in.readLine());
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6554) MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory
[
https://issues.apache.org/jira/browse/MAPREDUCE-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-6554:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.6.4
2.7.3
Status: Resolved (was: Patch Available)
Thanks, Bibin! I committed this to trunk, branch-2, branch-2.8, branch-2.7,
and branch-2.6.
> MRAppMaster servicestart failing with NPE in
> MRAppMaster#parsePreviousJobHistory
> -
>
> Key: MAPREDUCE-6554
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6554
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Fix For: 2.7.3, 2.6.4
>
> Attachments: 0001-MAPREDUCE-6554.patch, 0002-MAPREDUCE-6554.patch,
> 0003-MAPREDUCE-6554.patch, MAPREDUCE-6554-branch-2.7.001.patch,
> MAPREDUCE-6554-branch-2.7.patch
>
>
> Create scenario so that MR app master gets preempted.
> On next MRAppMaster launch tried to recover previous job history file
> {{MRAppMaster#parsePreviousJobHistory}}
> {noformat}
> 2015-11-21 13:52:27,722 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED;
> cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at java.io.StringReader.(StringReader.java:50)
> at org.apache.avro.Schema$Parser.parse(Schema.java:917)
> at org.apache.avro.Schema.parse(Schema.java:966)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:75)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1256)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1225)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1087)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1570)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1566)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1499)
> 2015-11-21 13:52:27,725 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 0
> {noformat}
> EventReader(EventReader stream)
> {noformat}
> this.version = in.readLine();
> ...
> Schema myschema = new
> SpecificData(Event.class.getClassLoader()).getSchema(Event.class);
> this.schema = Schema.parse(in.readLine());
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6338) MR AppMaster does not honor ephemeral port range
[ https://issues.apache.org/jira/browse/MAPREDUCE-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102172#comment-15102172 ] Alexandr commented on MAPREDUCE-6338: - Hello, [~bleuleon], you said : "Rohith, you are correct, the web port that opens up temporarily while the job is in progress is also random. I think a separate JIRA can be created for it since it does not stop the Job from completing successfully.". So, do we have the solution for random web ports ? I'm trying to configure hadoop with 2 nodes and with firewall enable. I can not do this because CantainerLauncher opens random port: 2015-12-16 16:00:14,775 INFO [ContainerLauncher #4] org.apache.hadoop.ipc.Client: Retrying connect to server: ub2simpl500/192.168.121.94:45353. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) > MR AppMaster does not honor ephemeral port range > > > Key: MAPREDUCE-6338 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6338 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 2.6.0 >Reporter: Frank Nguyen >Assignee: Frank Nguyen > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-6338.002.patch, MAPREDUCE-6338.003.patch > > > The MR AppMaster should only use port ranges defined in the > yarn.app.mapreduce.am.job.client.port-range property. On initial startup of > the MRAppMaster, it does use the port range defined in the property. > However, it also opens up a listener on a random ephemeral port. This is not > the Jetty listener. It is another listener opened by the MRAppMaster via > another thread and is recognized by the RM. Other nodes will try to > communicate to it via that random port. With firewall settings on, the MR > job will fail because the random port is not opened. This problem has caused > others to have all OS ephemeral ports opened to have MR jobs run. > This is related to MAPREDUCE-4079 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6536) hadoop-pipes doesn't use maven properties for openssl
[ https://issues.apache.org/jira/browse/MAPREDUCE-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102510#comment-15102510 ] Chris Nauroth commented on MAPREDUCE-6536: -- Hi [~aw]. This looks good to me, but we'll need one more patch revision for the pom.xml change, now that HADOOP-8887 has been committed. > hadoop-pipes doesn't use maven properties for openssl > - > > Key: MAPREDUCE-6536 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6536 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 3.0.0 > Environment: OS X >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-12518.00.patch, HADOOP-12518.01.patch, > HADOOP-12518.02.patch, HADOOP-12518.03.patch, MAPREDUCE-6536.04.patch > > > hadoop-common has some maven properties that are used to define where OpenSSL > lives. hadoop-pipes should also use them so we can enable automated testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
