[jira] [Commented] (MAPREDUCE-6754) Container Ids for an yarn application should be monotonically increasing in the scope of the application
[ https://issues.apache.org/jira/browse/MAPREDUCE-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436757#comment-15436757 ] Srikanth Sampath commented on MAPREDUCE-6754: - Thanks much [[~jlowe][[~vinodkv][[~jianhe] Given the discussion above, we can just add the attemptId to the JvmId for now. I will consider nuking out JvmId separately. > Container Ids for an yarn application should be monotonically increasing in > the scope of the application > > > Key: MAPREDUCE-6754 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6754 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Srikanth Sampath >Assignee: Srikanth Sampath > > Currently across application attempts, container Ids are reused. The > container id is stored in AppSchedulingInfo and it is reinitialized with > every application attempt. So the containerId scope is limited to the > application attempt. > In the MR Framework, It is important to note that the containerId is being > used as part of the JvmId. JvmId has 3 components containerId>. The JvmId is used in datastructures in TaskAttemptListener and > is passed between the AppMaster and the individual tasks. For an application > attempt, no two tasks have the same JvmId. > This works well currently, since inflight tasks get killed if the AppMaster > goes down. However, if we want to enable WorkPreserving nature for the AM, > containers (and hence containerIds) live across application attempts. If we > recycle containerIds across attempts, then two independent tasks (one > inflight from a previous attempt and another new in a succeeding attempt) > can have the same JvmId and cause havoc. > This can be solved in two ways: > *Approach A*: Include attempt id as part of the JvmId. This is a viable > solution, however, there is a change in the format of the JVMid. Changing > something that has existed so long for an optional feature is not persuasive. > *Approach B*: Keep the container id to be a monotonically increasing id for > the life of an application. So, container ids are not reused across > application attempts containers should be able to outlive an application > attempt. This is the preferred approach as it is clean, logical and is > backwards compatible. Nothing changes for existing applications or the > internal workings. > *How this is achieved:* > Currently, we maintain latest containerId only for application attempts and > reinitialize for new attempts. With this approach, we retrieve the latest > containerId from the just-failed attempt and initialize the new attempt with > the latest containerId (instead of 0). I can provide the patch if it helps. > It currently exists in MAPREDUCE-6726 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6764) Teragen LOG initialization bug
[ https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437222#comment-15437222 ] Wei-Chiu Chuang commented on MAPREDUCE-6764: Committed this patch to trunk, branch-2 and branch-2.8. Thanks [~yufeigu] for reporting and submitting a patch, and thanks [~Naganarasimha] for the reviewing the patch! > Teragen LOG initialization bug > -- > > Key: MAPREDUCE-6764 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Reporter: Yufei Gu >Assignee: Yufei Gu >Priority: Minor > Attachments: MAPREDUCE-6764.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6764) Teragen LOG initialization bug
[ https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated MAPREDUCE-6764: Affects Version/s: 3.0.0-alpha1 > Teragen LOG initialization bug > -- > > Key: MAPREDUCE-6764 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Affects Versions: 3.0.0-alpha1 >Reporter: Yufei Gu >Assignee: Yufei Gu >Priority: Minor > Attachments: MAPREDUCE-6764.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6764) Teragen LOG initialization bug
[ https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated MAPREDUCE-6764: Resolution: Fixed Status: Resolved (was: Patch Available) > Teragen LOG initialization bug > -- > > Key: MAPREDUCE-6764 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Affects Versions: 3.0.0-alpha1 >Reporter: Yufei Gu >Assignee: Yufei Gu >Priority: Minor > Labels: reviewed > Attachments: MAPREDUCE-6764.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6764) Teragen LOG initialization bug
[ https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated MAPREDUCE-6764: Labels: reviewed (was: ) > Teragen LOG initialization bug > -- > > Key: MAPREDUCE-6764 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Affects Versions: 3.0.0-alpha1 >Reporter: Yufei Gu >Assignee: Yufei Gu >Priority: Minor > Labels: reviewed > Attachments: MAPREDUCE-6764.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6764) Teragen LOG initialization bug
[ https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437235#comment-15437235 ] Yufei Gu edited comment on MAPREDUCE-6764 at 8/25/16 5:06 PM: -- Thanks [~jojochuang] for the review and commit! Thanks [~Naganarasimha] for the review! was (Author: yufeigu): Thanks [~jojochuang] for the review and commit! Thanks [~garlanaganarasi...@huawei.com] for the review! > Teragen LOG initialization bug > -- > > Key: MAPREDUCE-6764 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Affects Versions: 3.0.0-alpha1 >Reporter: Yufei Gu >Assignee: Yufei Gu >Priority: Minor > Labels: reviewed > Attachments: MAPREDUCE-6764.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6764) Teragen LOG initialization bug
[ https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437235#comment-15437235 ] Yufei Gu commented on MAPREDUCE-6764: - Thanks [~jojochuang] for the review and commit! Thanks [~garlanaganarasi...@huawei.com] for the review! > Teragen LOG initialization bug > -- > > Key: MAPREDUCE-6764 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Affects Versions: 3.0.0-alpha1 >Reporter: Yufei Gu >Assignee: Yufei Gu >Priority: Minor > Labels: reviewed > Attachments: MAPREDUCE-6764.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6764) Teragen LOG initialization bug
[ https://issues.apache.org/jira/browse/MAPREDUCE-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437263#comment-15437263 ] Hudson commented on MAPREDUCE-6764: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10348 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10348/]) MAPREDUCE-6764. Teragen LOG initialization bug. Contributed by Yufei Gu. (weichiu: rev 1360bd2d545134b582e70f2add33a105710dc80b) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraGen.java > Teragen LOG initialization bug > -- > > Key: MAPREDUCE-6764 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6764 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Affects Versions: 3.0.0-alpha1 >Reporter: Yufei Gu >Assignee: Yufei Gu >Priority: Minor > Labels: reviewed > Attachments: MAPREDUCE-6764.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6762) ControlledJob#toString failed with NPE when job status is not successfully updated
[ https://issues.apache.org/jira/browse/MAPREDUCE-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6762: Assignee: Weiwei Yang > ControlledJob#toString failed with NPE when job status is not successfully > updated > -- > > Key: MAPREDUCE-6762 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6762 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6762.001.patch > > > This issue was found from a cluster where Pig query occasionally failed on > NPE. Pig uses JobControl API to track MR job status, but sometimes Job > History Server failed to flush job meta files to HDFS which caused the status > update failed. Then we get NPE in > {{org.apache.hadoop.mapreduce.Job.getJobName}}. The result of this situation > is quite confusing: Pig query failed, job history is missing, but the job > status on Yarn is succeed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6768) TestRecovery.testSpeculative failed with NPE
Haibo Chen created MAPREDUCE-6768: - Summary: TestRecovery.testSpeculative failed with NPE Key: MAPREDUCE-6768 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6768 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Haibo Chen Assignee: Haibo Chen 1 tests failed. REGRESSION: org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative Error Message: null Stack Trace: java.lang.NullPointerException: null at org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative(TestRecovery.java:1201) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6768) TestRecovery.testSpeculative failed with NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438260#comment-15438260 ] Haibo Chen commented on MAPREDUCE-6768: --- {code:java} ContainerId t1a2contId = task1Attempt2.getAssignedContainerID(); LOG.info(t1a2contId.toString()); {code} TaskAttempt2 may not have been assigned a container, this is why there is a NPE. Uploading a trivial patch to fix this. > TestRecovery.testSpeculative failed with NPE > > > Key: MAPREDUCE-6768 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6768 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > > 1 tests failed. > REGRESSION: org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative > Error Message: > null > Stack Trace: > java.lang.NullPointerException: null > at > org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative(TestRecovery.java:1201) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6768) TestRecovery.testSpeculative failed with NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6768: -- Attachment: mapreduce6768.001.patch > TestRecovery.testSpeculative failed with NPE > > > Key: MAPREDUCE-6768 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6768 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6768.001.patch > > > 1 tests failed. > REGRESSION: org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative > Error Message: > null > Stack Trace: > java.lang.NullPointerException: null > at > org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative(TestRecovery.java:1201) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6768) TestRecovery.testSpeculative failed with NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6768: -- Status: Patch Available (was: Open) > TestRecovery.testSpeculative failed with NPE > > > Key: MAPREDUCE-6768 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6768 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: mapreduce6768.001.patch > > > 1 tests failed. > REGRESSION: org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative > Error Message: > null > Stack Trace: > java.lang.NullPointerException: null > at > org.apache.hadoop.mapreduce.v2.app.TestRecovery.testSpeculative(TestRecovery.java:1201) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6768) TestRecovery.testSpeculative failed with NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438291#comment-15438291 ] Hadoop QA commented on MAPREDUCE-6768: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 59s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 37s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 24s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s {color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 1 new + 118 unchanged - 1 fixed = 119 total (was 119) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 45s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 4s {color} | {color:green} hadoop-mapreduce-client-app in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 13s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12825581/mapreduce6768.001.patch | | JIRA Issue | MAPREDUCE-6768 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 51ccb7d6be4d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 81485db | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6698/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt | | Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6698/testReport/ | | modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app | | Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6698/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > TestRecovery.testSpeculative failed with NPE > > > Key: MAPREDUCE-6768 > URL: https://issues.