[jira] [Commented] (MAPREDUCE-5663) Add an interface to Input/Ouput Formats to obtain delegation tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14534851#comment-14534851 ] Roman Shaposhnik commented on MAPREDUCE-5663: - [~sseth] I'm looking at it as part of bugbash (and also b/c krb is something I'm pretty familiar with). Given the discussion so far, I'm not quite sure where this stands. The last comment from [~tucu00] makes a lot of sense to me, but obviously I'm looking at this for the first time. > Add an interface to Input/Ouput Formats to obtain delegation tokens > --- > > Key: MAPREDUCE-5663 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5663 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Michael Weng > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-5663.4.txt, MAPREDUCE-5663.5.txt, > MAPREDUCE-5663.6.txt, MAPREDUCE-5663.patch.txt, MAPREDUCE-5663.patch.txt2, > MAPREDUCE-5663.patch.txt3 > > > Currently, delegation tokens are obtained as part of the getSplits / > checkOutputSpecs calls to the InputFormat / OutputFormat respectively. > This works as long as the splits are generated on a node with kerberos > credentials. For split generation elsewhere (AM for example), an explicit > interface is required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3202) Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13861748#comment-13861748 ] Roman Shaposhnik commented on MAPREDUCE-3202: - [~vitthal_gogate] I did the first pass over the patch and it looks reasonable. Here's a few bits of feedback I've got so far: * I would like to suggest that an extra JIRA (perhaps a subjira of this one) gets created to track *just* the changes required in existing HS webUI to support Vaidya. One half of the patch will go there. * please make sure to add unit tests for that first half of the patch * on this JIRA we shall keep track of things under hadoop-tools/hadoop-vaidya * we would also need to make sure that hadoop-tools/hadoop-vaidya bits end up in maven assembly > Integrating Hadoop Vaidya with Job History UI in Hadoop 2.0 > > > Key: MAPREDUCE-3202 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3202 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: vitthal (Suhas) Gogate >Assignee: Evan (Yiming) Meng > Fix For: trunk > > Attachments: MAPREDUCE-3202.patch, MAPREDUCE-3202.patch > > > Hadoop Vaidya provides a detailed analysis of the M/R job in terms of various > execution inefficiencies and the associated remedies that user can easily > understand and fix. This Jira patch integrates it with Job History UI under > Hadoop 2.0 branch. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5362) clean up POM dependencies
[ https://issues.apache.org/jira/browse/MAPREDUCE-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13751399#comment-13751399 ] Roman Shaposhnik commented on MAPREDUCE-5362: - I'm cleaning up a few build issues (Bigtop 0.8.0 related) and I was wondering whether you'd think it would be a good idea for me to tackle this one as well. Please assign it to me if you think it is. > clean up POM dependencies > - > > Key: MAPREDUCE-5362 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5362 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 2.1.0-beta >Reporter: Alejandro Abdelnur > > Intermediate 'pom' modules define dependencies inherited by leaf modules. > This is causing issues in intellij IDE. > We should normalize the leaf modules like in common, hdfs and tools where all > dependencies are defined in each leaf module and the intermediate 'pom' > module do not define any dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated MAPREDUCE-5240:
Status: Patch Available (was: Reopened)
> inside of FileOutputCommitter the initialized Credentials cache appears to be
> empty
> ---
>
> Key: MAPREDUCE-5240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.4-alpha
>Reporter: Roman Shaposhnik
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Labels: 2.0.4.1
> Fix For: 2.0.5-beta, 2.0.4.1-alpha
>
> Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt,
> MAPREDUCE-5240-20130513.txt, MAPREDUCE-5240.2.0.4.rvs.patch.txt
>
>
> I am attaching a modified wordcount job that clearly demonstrates the problem
> we've encountered in running Sqoop2 on YARN (BIGTOP-949).
> Here's what running it produces:
> {noformat}
> $ hadoop fs -mkdir in
> $ hadoop fs -put /etc/passwd in
> $ hadoop jar ./bug.jar org.myorg.LostCreds
> 13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
> longer used.
> numberOfSecretKeys: 1
> numberOfTokens: 0
> ..
> ..
> ..
> 13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
> state FAILED due to: Job commit failed: java.io.IOException:
> numberOfSecretKeys: 0
> numberOfTokens: 0
> at
> org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
> As you can see, even though we've clearly initialized the creds via:
> {noformat}
> job.getCredentials().addSecretKey(new Text("mykey"), "mysecret".getBytes());
> {noformat}
> It doesn't seem to appear later in the job.
> This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
> YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated MAPREDUCE-5240:
Attachment: MAPREDUCE-5240.2.0.4.rvs.patch.txt
Attaching a modified patch for branch-2.0.4
> inside of FileOutputCommitter the initialized Credentials cache appears to be
> empty
> ---
>
> Key: MAPREDUCE-5240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.4-alpha
>Reporter: Roman Shaposhnik
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Labels: 2.0.4.1
> Fix For: 2.0.5-beta, 2.0.4.1-alpha
>
> Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt,
> MAPREDUCE-5240-20130513.txt, MAPREDUCE-5240.2.0.4.rvs.patch.txt
>
>
> I am attaching a modified wordcount job that clearly demonstrates the problem
> we've encountered in running Sqoop2 on YARN (BIGTOP-949).
> Here's what running it produces:
> {noformat}
> $ hadoop fs -mkdir in
> $ hadoop fs -put /etc/passwd in
> $ hadoop jar ./bug.jar org.myorg.LostCreds
> 13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
> longer used.
> numberOfSecretKeys: 1
> numberOfTokens: 0
> ..
> ..
> ..
> 13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
> state FAILED due to: Job commit failed: java.io.IOException:
> numberOfSecretKeys: 0
> numberOfTokens: 0
> at
> org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
> As you can see, even though we've clearly initialized the creds via:
> {noformat}
> job.getCredentials().addSecretKey(new Text("mykey"), "mysecret".getBytes());
> {noformat}
> It doesn't seem to appear later in the job.
> This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
> YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655768#comment-13655768
]
Roman Shaposhnik commented on MAPREDUCE-5240:
-
Vinod, thanks a million for attaching the patch! I'm building it right now in
Bigtop and hope to run validation tests tomorrow! Will keep you posted on a
JIRA.
> inside of FileOutputCommitter the initialized Credentials cache appears to be
> empty
> ---
>
> Key: MAPREDUCE-5240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.4-alpha
>Reporter: Roman Shaposhnik
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 2.0.5-beta
>
> Attachments: LostCreds.java, MAPREDUCE-5240-20130512.txt
>
>
> I am attaching a modified wordcount job that clearly demonstrates the problem
> we've encountered in running Sqoop2 on YARN (BIGTOP-949).
> Here's what running it produces:
> {noformat}
> $ hadoop fs -mkdir in
> $ hadoop fs -put /etc/passwd in
> $ hadoop jar ./bug.jar org.myorg.LostCreds
> 13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
> longer used.
> numberOfSecretKeys: 1
> numberOfTokens: 0
> ..
> ..
> ..
> 13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
> state FAILED due to: Job commit failed: java.io.IOException:
> numberOfSecretKeys: 0
> numberOfTokens: 0
> at
> org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
> As you can see, even though we've clearly initialized the creds via:
> {noformat}
> job.getCredentials().addSecretKey(new Text("mykey"), "mysecret".getBytes());
> {noformat}
> It doesn't seem to appear later in the job.
> This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
> YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655638#comment-13655638
]
Roman Shaposhnik commented on MAPREDUCE-5240:
-
Vinod, if it is not too difficult I'd greatly appreciate a small patch for
specifically this very issue. Thanks!
> inside of FileOutputCommitter the initialized Credentials cache appears to be
> empty
> ---
>
> Key: MAPREDUCE-5240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.4-alpha
>Reporter: Roman Shaposhnik
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 2.0.5-beta
>
> Attachments: LostCreds.java
>
>
> I am attaching a modified wordcount job that clearly demonstrates the problem
> we've encountered in running Sqoop2 on YARN (BIGTOP-949).
> Here's what running it produces:
> {noformat}
> $ hadoop fs -mkdir in
> $ hadoop fs -put /etc/passwd in
> $ hadoop jar ./bug.jar org.myorg.LostCreds
> 13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
> longer used.
> numberOfSecretKeys: 1
> numberOfTokens: 0
> ..
> ..
> ..
> 13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
> state FAILED due to: Job commit failed: java.io.IOException:
> numberOfSecretKeys: 0
> numberOfTokens: 0
> at
> org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
> As you can see, even though we've clearly initialized the creds via:
> {noformat}
> job.getCredentials().addSecretKey(new Text("mykey"), "mysecret".getBytes());
> {noformat}
> It doesn't seem to appear later in the job.
> This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
> YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated MAPREDUCE-5240:
As was mentioned on the Bigtop JIRA this looks related to HADOOP-8726
> inside of FileOutputCommitter the initialized Credentials cache appears to be
> empty
> ---
>
> Key: MAPREDUCE-5240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1
>Affects Versions: 2.0.4-alpha
>Reporter: Roman Shaposhnik
>Priority: Blocker
> Fix For: 2.0.5-beta
>
> Attachments: LostCreds.java
>
>
> I am attaching a modified wordcount job that clearly demonstrates the problem
> we've encountered in running Sqoop2 on YARN (BIGTOP-949).
> Here's what running it produces:
> {noformat}
> $ hadoop fs -mkdir in
> $ hadoop fs -put /etc/passwd in
> $ hadoop jar ./bug.jar org.myorg.LostCreds
> 13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
> longer used.
> numberOfSecretKeys: 1
> numberOfTokens: 0
> ..
> ..
> ..
> 13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
> state FAILED due to: Job commit failed: java.io.IOException:
> numberOfSecretKeys: 0
> numberOfTokens: 0
> at
> org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
> As you can see, even though we've clearly initialized the creds via:
> {noformat}
> job.getCredentials().addSecretKey(new Text("mykey"), "mysecret".getBytes());
> {noformat}
> It doesn't seem to appear later in the job.
> This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
> YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated MAPREDUCE-5240:
Attachment: LostCreds.java
> inside of FileOutputCommitter the initialized Credentials cache appears to be
> empty
> ---
>
> Key: MAPREDUCE-5240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1
>Affects Versions: 2.0.4-alpha
>Reporter: Roman Shaposhnik
>Priority: Blocker
> Fix For: 2.0.5-beta
>
> Attachments: LostCreds.java
>
>
> I am attaching a modified wordcount job that clearly demonstrates the problem
> we've encountered in running Sqoop2 on YARN (BIGTOP-949).
> Here's what running it produces:
> {noformat}
> $ hadoop fs -mkdir in
> $ hadoop fs -put /etc/passwd in
> $ hadoop jar ./bug.jar org.myorg.LostCreds
> 13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
> longer used.
> numberOfSecretKeys: 1
> numberOfTokens: 0
> ..
> ..
> ..
> 13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
> state FAILED due to: Job commit failed: java.io.IOException:
> numberOfSecretKeys: 0
> numberOfTokens: 0
> at
> org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
> As you can see, even though we've clearly initialized the creds via:
> {noformat}
> job.getCredentials().addSecretKey(new Text("mykey"), "mysecret".getBytes());
> {noformat}
> It doesn't seem to appear later in the job.
> This is a pretty critical issue for Sqoop 2 since it appears to be DOA for
> YARN in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated MAPREDUCE-5240:
Description:
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text("mykey"), "mysecret".getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for YARN
in Hadoop 2.0.4-alpha
was:
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text("mykey"), "mysecret".getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for YARN
in Hadoop 2.0.4-alpha
> inside of FileOutputCommitter the initialized Credentials cache appears to be
> empty
> ---
>
> Key: MAPREDUCE-5240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1
>Affects Versions: 2.0.4-alpha
>Reporter: Roman Shaposhnik
>Priority: Blocker
> Fix For: 2.0.5-beta
>
> Attachments: LostCreds.java
>
>
> I am attaching a modified wordcount job that clearly demonstrates the problem
> we've encountered in running Sqoop2 on YARN (BIGTOP-949).
> Here's what running it produces:
> {noformat}
> $ hadoop fs -mkdir in
> $ hadoop fs -put /etc/passwd in
> $ hadoop jar ./bug.jar org.myorg.LostCreds
> 13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
> longer used.
> numberOfSecretKeys: 1
> numberOfTokens: 0
> ..
> ..
> ..
> 13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
> state FAILED due to: Job commit failed: java.io.IOException:
> numberOfSecretKeys: 0
> numberOfTokens: 0
> at
> org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concur
[jira] [Created] (MAPREDUCE-5240) inside of FileOutputCommitter the initialized Credentials cache appears to be empty
Roman Shaposhnik created MAPREDUCE-5240:
---
Summary: inside of FileOutputCommitter the initialized Credentials
cache appears to be empty
Key: MAPREDUCE-5240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Priority: Blocker
Fix For: 2.0.5-beta
Attachments: LostCreds.java
I am attaching a modified wordcount job that clearly demonstrates the problem
we've encountered in running Sqoop2 on YARN (BIGTOP-949).
Here's what running it produces:
{noformat}
$ hadoop fs -mkdir in
$ hadoop fs -put /etc/passwd in
$ hadoop jar ./bug.jar org.myorg.LostCreds
13/05/12 03:13:46 WARN mapred.JobConf: The variable mapred.child.ulimit is no
longer used.
numberOfSecretKeys: 1
numberOfTokens: 0
..
..
..
13/05/12 03:05:35 INFO mapreduce.Job: Job job_1368318686284_0013 failed with
state FAILED due to: Job commit failed: java.io.IOException:
numberOfSecretKeys: 0
numberOfTokens: 0
at
org.myorg.LostCreds$DestroyerFileOutputCommitter.commitJob(LostCreds.java:43)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
As you can see, even though we've clearly initialized the creds via:
{noformat}
job.getCredentials().addSecretKey(new Text("mykey"), "mysecret".getBytes());
{noformat}
It doesn't seem to appear later in the job.
This is a pretty critical issue for Sqoop 2 since it appears to be DOA for YARN
in Hadoop 2.0.4-alpha
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5088) MR Client gets an renewer token exception while Oozie is submitting a job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13616835#comment-13616835
]
Roman Shaposhnik commented on MAPREDUCE-5088:
-
Sid, would you mind taking a look at MAPREDUCE-5117 ?
> MR Client gets an renewer token exception while Oozie is submitting a job
> -
>
> Key: MAPREDUCE-5088
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5088
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha
>Reporter: Roman Shaposhnik
>Assignee: Daryn Sharp
>Priority: Blocker
> Fix For: 2.0.4-alpha
>
> Attachments: HADOOP-9409.patch, HADOOP-9409.patch,
> MAPREDUCE-5088.patch, MAPREDUCE-5088.patch, MAPREDUCE-5088.txt
>
>
> After the fix for HADOOP-9299 I'm now getting the following bizzare exception
> in Oozie while trying to submit a job. This also seems to be KRB related:
> {noformat}
> 2013-03-15 13:34:16,555 WARN ActionStartXCommand:542 - USER[hue] GROUP[-]
> TOKEN[] APP[MapReduce] JOB[001-130315123130987-oozie-oozi-W]
> ACTION[001-130315123130987-oozie-oozi-W@Sleep] Error starting action
> [Sleep]. ErrorType [ERROR], ErrorCode [UninitializedMessageException],
> Message [UninitializedMessageException: Message missing required fields:
> renewer]
> org.apache.oozie.action.ActionExecutorException:
> UninitializedMessageException: Message missing required fields: renewer
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:401)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:738)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
> at org.apache.oozie.command.XCommand.call(XCommand.java:277)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: com.google.protobuf.UninitializedMessageException: Message missing
> required fields: renewer
> at
> com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:605)
> at
> org.apache.hadoop.security.proto.SecurityProtos$GetDelegationTokenRequestProto$Builder.build(SecurityProtos.java:973)
> at
> org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.mergeLocalToProto(GetDelegationTokenRequestPBImpl.java:84)
> at
> org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.getProto(GetDelegationTokenRequestPBImpl.java:67)
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:200)
> at
> org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:194)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:273)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:581)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:576)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:723)
> ... 10 more
> 2013-03-15 13:34:16,555 WARN ActionStartXCommand:542 - USER[hue] GROUP[-]
>
[jira] [Updated] (MAPREDUCE-5117) With security enabled HS delegation token renewer fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated MAPREDUCE-5117:
Attachment: yarn.log
> With security enabled HS delegation token renewer fails
> ---
>
> Key: MAPREDUCE-5117
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5117
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: security
>Affects Versions: 2.0.4-alpha
>Reporter: Roman Shaposhnik
>Priority: Blocker
> Fix For: 2.0.4-alpha
>
> Attachments: yarn.log
>
>
> It seems that the HSClientProtocolPBClientImpl should implement Closeable as
> per the attached stack trace. The problem can be observed on a cluster
> running the latest branch-2.0.4-alpha with MAPREDUCE-5088 applied on top. The
> easiest way to reproduce it is to run an oozie pig job:
> {noformat}
> $ oozie job -oozie http://`hostname -f`:11000/oozie -run
> -DjobTracker=`hostname -f`:8032 -DnameNode=hdfs://`hostname -f`:17020
> -DexamplesRoot=examples -config /tmp/examples/apps/pig/job.properties
> {noformat}
> Please also note that I can successfully submit simple jobs (Pi/Sleep) from a
> command line using hadoop jar command. Thus it *seems* related to
> MAPREDUCE-5088 change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5117) With security enabled HS delegation token renewer fails
Roman Shaposhnik created MAPREDUCE-5117:
---
Summary: With security enabled HS delegation token renewer fails
Key: MAPREDUCE-5117
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5117
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: security
Affects Versions: 2.0.4-alpha
Reporter: Roman Shaposhnik
Priority: Blocker
Fix For: 2.0.4-alpha
It seems that the HSClientProtocolPBClientImpl should implement Closeable as
per the attached stack trace. The problem can be observed on a cluster running
the latest branch-2.0.4-alpha with MAPREDUCE-5088 applied on top. The easiest
way to reproduce it is to run an oozie pig job:
{noformat}
$ oozie job -oozie http://`hostname -f`:11000/oozie -run -DjobTracker=`hostname
-f`:8032 -DnameNode=hdfs://`hostname -f`:17020 -DexamplesRoot=examples -config
/tmp/examples/apps/pig/job.properties
{noformat}
Please also note that I can successfully submit simple jobs (Pi/Sleep) from a
command line using hadoop jar command. Thus it *seems* related to
MAPREDUCE-5088 change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4549) Distributed cache conflicts breaks backwards compatability
[ https://issues.apache.org/jira/browse/MAPREDUCE-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated MAPREDUCE-4549: Resolution: Fixed Status: Resolved (was: Patch Available) > Distributed cache conflicts breaks backwards compatability > -- > > Key: MAPREDUCE-4549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.2-alpha >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 2.0.4-alpha, 0.23.5 > > Attachments: MAPREDUCE-4549-trunk.patch, MR-4549-branch-0.23.txt > > > I recently put in MAPREDUCE-4503 which went a bit too far, and broke > backwards compatibility with 1.0 in distribtued cache entries. instead of > changing the behavior of the distributed cache to more closely match 1.0 > behavior I want to just change the exception to a warning message informing > the users that it will become an error in 2.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4820) MRApps distributed-cache duplicate checks are incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13614482#comment-13614482 ] Roman Shaposhnik commented on MAPREDUCE-4820: - [~revans2] I think you may be absolutely right. This could be a residual effect of enabling OOZIE-1089 (which we probably shouldn't do now that the actual issue seems to be fixed). Let me poke on the Oozie side and report back. Thanks for your help so far! > MRApps distributed-cache duplicate checks are incorrect > --- > > Key: MAPREDUCE-4820 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4820 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.2-alpha >Reporter: Alejandro Abdelnur >Priority: Blocker > Fix For: 2.0.4-alpha > > Attachments: launcher-job.conf.xml, launcher-job.logs.txt, > mr-job.conf.xml, mr-job.logs.txt > > > This seems a combination of issues that are being exposed in 2.0.2-alpha by > MAPREDUCE-4549. > MAPREDUCE-4549 introduces a check to to ensure there are not duplicate JARs > in the distributed-cache (using the JAR name as identity). > In Hadoop 2 (different from Hadoop 1), all JARs in the distributed-cache are > symlink-ed to the current directory of the task. > MRApps, when setting up the DistributedCache > (MRApps#setupDistributedCache->parseDistributedCacheArtifacts) assumes that > the local resources (this includes files in the CURRENT_DIR/, > CURRENT_DIR/classes/ and files in CURRENT_DIR/lib/) are part of the > distributed-cache already. > For systems, like Oozie, which use a launcher job to submit the real job this > poses a problem because MRApps is run from the launcher job to submit the > real job. The configuration of the real job has the correct distributed-cache > entries (no duplicates), but because the current dir has the same files, the > submission fails. > It seems that MRApps should not be checking dups in the distributed-cached > against JARs in the CURRENT_DIR/ or CURRENT_DIR/lib/. The dup check should be > done among distributed-cached entries only. > It seems YARNRunner is symlink-ing all files in the distributed cached in the > current directory. In Hadoop 1 this was done only for files added to the > distributed-cache using a fragment (ie "#FOO") to trigger a symlink creation. > Marking as a blocker because without a fix for this, Oozie cannot submit jobs > to Hadoop 2 (i've debugged Oozie in a live cluster being used by BigTop > -thanks Roman- to test their release work, and I've verified that Oozie 3.3 > does not create duplicated entries in the distributed-cache) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4549) Distributed cache conflicts breaks backwards compatability
[ https://issues.apache.org/jira/browse/MAPREDUCE-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13614367#comment-13614367 ] Roman Shaposhnik commented on MAPREDUCE-4549: - All the findings are attached over here: MAPREDUCE-4820 At this point I'd agree with Arun and close this one. We can track what's left over at MAPREDUCE-4820 > Distributed cache conflicts breaks backwards compatability > -- > > Key: MAPREDUCE-4549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.2-alpha >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.5, 2.0.4-alpha > > Attachments: MAPREDUCE-4549-trunk.patch, MR-4549-branch-0.23.txt > > > I recently put in MAPREDUCE-4503 which went a bit too far, and broke > backwards compatibility with 1.0 in distribtued cache entries. instead of > changing the behavior of the distributed cache to more closely match 1.0 > behavior I want to just change the exception to a warning message informing > the users that it will become an error in 2.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5088) MR Client gets an renewer token exception while Oozie is submitting a job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13612326#comment-13612326
]
Roman Shaposhnik commented on MAPREDUCE-5088:
-
Guys, what's the latest on this patch? I've tested it in Bigtop and it seems to
do exactly what it needs as far as unblocking Oozie is concerned. What else is
left before we can commit it to trunk and all the relevant branches?
> MR Client gets an renewer token exception while Oozie is submitting a job
> -
>
> Key: MAPREDUCE-5088
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5088
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha
>Reporter: Roman Shaposhnik
>Assignee: Daryn Sharp
>Priority: Blocker
> Fix For: 2.0.4-alpha
>
> Attachments: HADOOP-9409.patch, HADOOP-9409.patch,
> MAPREDUCE-5088.patch, MAPREDUCE-5088.patch
>
>
> After the fix for HADOOP-9299 I'm now getting the following bizzare exception
> in Oozie while trying to submit a job. This also seems to be KRB related:
> {noformat}
> 2013-03-15 13:34:16,555 WARN ActionStartXCommand:542 - USER[hue] GROUP[-]
> TOKEN[] APP[MapReduce] JOB[001-130315123130987-oozie-oozi-W]
> ACTION[001-130315123130987-oozie-oozi-W@Sleep] Error starting action
> [Sleep]. ErrorType [ERROR], ErrorCode [UninitializedMessageException],
> Message [UninitializedMessageException: Message missing required fields:
> renewer]
> org.apache.oozie.action.ActionExecutorException:
> UninitializedMessageException: Message missing required fields: renewer
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:401)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:738)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
> at org.apache.oozie.command.XCommand.call(XCommand.java:277)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: com.google.protobuf.UninitializedMessageException: Message missing
> required fields: renewer
> at
> com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:605)
> at
> org.apache.hadoop.security.proto.SecurityProtos$GetDelegationTokenRequestProto$Builder.build(SecurityProtos.java:973)
> at
> org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.mergeLocalToProto(GetDelegationTokenRequestPBImpl.java:84)
> at
> org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.getProto(GetDelegationTokenRequestPBImpl.java:67)
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:200)
> at
> org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:194)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:273)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:581)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:576)
> at
> org.apache.oozie.action.hadoop.JavaAct
[jira] [Commented] (MAPREDUCE-4820) MRApps distributed-cache duplicate checks are incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13611154#comment-13611154 ] Roman Shaposhnik commented on MAPREDUCE-4820: - Robert, I just tried it with the tip of the branch-2.0.4-alpha and I still see the problem. I'm building the following: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.0.4-alpha@1457977 Where do you say it is fixed? > MRApps distributed-cache duplicate checks are incorrect > --- > > Key: MAPREDUCE-4820 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4820 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.2-alpha >Reporter: Alejandro Abdelnur >Priority: Blocker > Fix For: 2.0.4-alpha > > Attachments: launcher-job.conf.xml, launcher-job.logs.txt, > mr-job.conf.xml, mr-job.logs.txt > > > This seems a combination of issues that are being exposed in 2.0.2-alpha by > MAPREDUCE-4549. > MAPREDUCE-4549 introduces a check to to ensure there are not duplicate JARs > in the distributed-cache (using the JAR name as identity). > In Hadoop 2 (different from Hadoop 1), all JARs in the distributed-cache are > symlink-ed to the current directory of the task. > MRApps, when setting up the DistributedCache > (MRApps#setupDistributedCache->parseDistributedCacheArtifacts) assumes that > the local resources (this includes files in the CURRENT_DIR/, > CURRENT_DIR/classes/ and files in CURRENT_DIR/lib/) are part of the > distributed-cache already. > For systems, like Oozie, which use a launcher job to submit the real job this > poses a problem because MRApps is run from the launcher job to submit the > real job. The configuration of the real job has the correct distributed-cache > entries (no duplicates), but because the current dir has the same files, the > submission fails. > It seems that MRApps should not be checking dups in the distributed-cached > against JARs in the CURRENT_DIR/ or CURRENT_DIR/lib/. The dup check should be > done among distributed-cached entries only. > It seems YARNRunner is symlink-ing all files in the distributed cached in the > current directory. In Hadoop 1 this was done only for files added to the > distributed-cache using a fragment (ie "#FOO") to trigger a symlink creation. > Marking as a blocker because without a fix for this, Oozie cannot submit jobs > to Hadoop 2 (i've debugged Oozie in a live cluster being used by BigTop > -thanks Roman- to test their release work, and I've verified that Oozie 3.3 > does not create duplicated entries in the distributed-cache) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4820) MRApps distributed-cache duplicate checks are incorrect
[
https://issues.apache.org/jira/browse/MAPREDUCE-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated MAPREDUCE-4820:
Attachment: launcher-job.conf.xml
launcher-job.logs.txt
mr-job.logs.txt
mr-job.conf.xml
To make the debugging easier I'm attaching logs and XML confs that I see on my
cluster after submitting:
{noformat}
$ oozie job -oozie http://localhost:11000/oozie -run -DjobTracker=bigtop:8032
-DnameNode=hdfs://bigtop:17020 -DexamplesRoot=examples -config
/tmp/examples/apps/map-reduce/job.properties
{noformat}
> MRApps distributed-cache duplicate checks are incorrect
> ---
>
> Key: MAPREDUCE-4820
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4820
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.0.2-alpha
>Reporter: Alejandro Abdelnur
>Priority: Blocker
> Fix For: 2.0.4-alpha
>
> Attachments: launcher-job.conf.xml, launcher-job.logs.txt,
> mr-job.conf.xml, mr-job.logs.txt
>
>
> This seems a combination of issues that are being exposed in 2.0.2-alpha by
> MAPREDUCE-4549.
> MAPREDUCE-4549 introduces a check to to ensure there are not duplicate JARs
> in the distributed-cache (using the JAR name as identity).
> In Hadoop 2 (different from Hadoop 1), all JARs in the distributed-cache are
> symlink-ed to the current directory of the task.
> MRApps, when setting up the DistributedCache
> (MRApps#setupDistributedCache->parseDistributedCacheArtifacts) assumes that
> the local resources (this includes files in the CURRENT_DIR/,
> CURRENT_DIR/classes/ and files in CURRENT_DIR/lib/) are part of the
> distributed-cache already.
> For systems, like Oozie, which use a launcher job to submit the real job this
> poses a problem because MRApps is run from the launcher job to submit the
> real job. The configuration of the real job has the correct distributed-cache
> entries (no duplicates), but because the current dir has the same files, the
> submission fails.
> It seems that MRApps should not be checking dups in the distributed-cached
> against JARs in the CURRENT_DIR/ or CURRENT_DIR/lib/. The dup check should be
> done among distributed-cached entries only.
> It seems YARNRunner is symlink-ing all files in the distributed cached in the
> current directory. In Hadoop 1 this was done only for files added to the
> distributed-cache using a fragment (ie "#FOO") to trigger a symlink creation.
> Marking as a blocker because without a fix for this, Oozie cannot submit jobs
> to Hadoop 2 (i've debugged Oozie in a live cluster being used by BigTop
> -thanks Roman- to test their release work, and I've verified that Oozie 3.3
> does not create duplicated entries in the distributed-cache)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4549) Distributed cache conflicts breaks backwards compatability
[ https://issues.apache.org/jira/browse/MAPREDUCE-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated MAPREDUCE-4549: Priority: Blocker (was: Critical) Fix Version/s: (was: 2.0.5-beta) 2.0.4-alpha Making this a blocker for 2.0.4-alpha as agreed. I will keep attaching logs/debugging data from Oozie runs to MAPREDUCE-4820 > Distributed cache conflicts breaks backwards compatability > -- > > Key: MAPREDUCE-4549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.2-alpha >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.5, 2.0.4-alpha > > Attachments: MAPREDUCE-4549-trunk.patch, MR-4549-branch-0.23.txt > > > I recently put in MAPREDUCE-4503 which went a bit too far, and broke > backwards compatibility with 1.0 in distribtued cache entries. instead of > changing the behavior of the distributed cache to more closely match 1.0 > behavior I want to just change the exception to a warning message informing > the users that it will become an error in 2.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4820) MRApps distributed-cache duplicate checks are incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated MAPREDUCE-4820: Priority: Blocker (was: Major) Fix Version/s: (was: 2.0.5-beta) 2.0.4-alpha > MRApps distributed-cache duplicate checks are incorrect > --- > > Key: MAPREDUCE-4820 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4820 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.2-alpha >Reporter: Alejandro Abdelnur >Priority: Blocker > Fix For: 2.0.4-alpha > > > This seems a combination of issues that are being exposed in 2.0.2-alpha by > MAPREDUCE-4549. > MAPREDUCE-4549 introduces a check to to ensure there are not duplicate JARs > in the distributed-cache (using the JAR name as identity). > In Hadoop 2 (different from Hadoop 1), all JARs in the distributed-cache are > symlink-ed to the current directory of the task. > MRApps, when setting up the DistributedCache > (MRApps#setupDistributedCache->parseDistributedCacheArtifacts) assumes that > the local resources (this includes files in the CURRENT_DIR/, > CURRENT_DIR/classes/ and files in CURRENT_DIR/lib/) are part of the > distributed-cache already. > For systems, like Oozie, which use a launcher job to submit the real job this > poses a problem because MRApps is run from the launcher job to submit the > real job. The configuration of the real job has the correct distributed-cache > entries (no duplicates), but because the current dir has the same files, the > submission fails. > It seems that MRApps should not be checking dups in the distributed-cached > against JARs in the CURRENT_DIR/ or CURRENT_DIR/lib/. The dup check should be > done among distributed-cached entries only. > It seems YARNRunner is symlink-ing all files in the distributed cached in the > current directory. In Hadoop 1 this was done only for files added to the > distributed-cache using a fragment (ie "#FOO") to trigger a symlink creation. > Marking as a blocker because without a fix for this, Oozie cannot submit jobs > to Hadoop 2 (i've debugged Oozie in a live cluster being used by BigTop > -thanks Roman- to test their release work, and I've verified that Oozie 3.3 > does not create duplicated entries in the distributed-cache) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5088) MR Client gets an renewer token exception while Oozie is submitting a job
[
https://issues.apache.org/jira/browse/MAPREDUCE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609137#comment-13609137
]
Roman Shaposhnik commented on MAPREDUCE-5088:
-
Guys, thanks a million for the patch. So far I tried the one from yesterday
(20/Mar/13) and it completely unblocks Oozie in my Bigtop fully distributed
testing.
I will try your latest patch today and report back.
On a related note it seems like MAPREDUCE-4820 is biting us again -- I'll
reopen it.
> MR Client gets an renewer token exception while Oozie is submitting a job
> -
>
> Key: MAPREDUCE-5088
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5088
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.3-alpha
>Reporter: Roman Shaposhnik
>Assignee: Daryn Sharp
>Priority: Blocker
> Fix For: 2.0.4-alpha
>
> Attachments: HADOOP-9409.patch, HADOOP-9409.patch,
> MAPREDUCE-5088.patch
>
>
> After the fix for HADOOP-9299 I'm now getting the following bizzare exception
> in Oozie while trying to submit a job. This also seems to be KRB related:
> {noformat}
> 2013-03-15 13:34:16,555 WARN ActionStartXCommand:542 - USER[hue] GROUP[-]
> TOKEN[] APP[MapReduce] JOB[001-130315123130987-oozie-oozi-W]
> ACTION[001-130315123130987-oozie-oozi-W@Sleep] Error starting action
> [Sleep]. ErrorType [ERROR], ErrorCode [UninitializedMessageException],
> Message [UninitializedMessageException: Message missing required fields:
> renewer]
> org.apache.oozie.action.ActionExecutorException:
> UninitializedMessageException: Message missing required fields: renewer
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:401)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:738)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
> at org.apache.oozie.command.XCommand.call(XCommand.java:277)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: com.google.protobuf.UninitializedMessageException: Message missing
> required fields: renewer
> at
> com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:605)
> at
> org.apache.hadoop.security.proto.SecurityProtos$GetDelegationTokenRequestProto$Builder.build(SecurityProtos.java:973)
> at
> org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.mergeLocalToProto(GetDelegationTokenRequestPBImpl.java:84)
> at
> org.apache.hadoop.mapreduce.v2.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl.getProto(GetDelegationTokenRequestPBImpl.java:67)
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:200)
> at
> org.apache.hadoop.mapred.YARNRunner.getDelegationTokenFromHS(YARNRunner.java:194)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:273)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:581)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1439)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:576)
[jira] [Commented] (MAPREDUCE-3635) Improve Hadoop subcomponent integration in Hadoop 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597323#comment-13597323 ] Roman Shaposhnik commented on MAPREDUCE-3635: - I think we're good. > Improve Hadoop subcomponent integration in Hadoop 0.23 > -- > > Key: MAPREDUCE-3635 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3635 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build, client >Affects Versions: 0.23.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 0.24.0 > > > Please see HADOOP-7939 for a complete description and discussion. This JIRA > is for patch tracking purposes only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4820) MRApps distributed-cache duplicate checks are incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated MAPREDUCE-4820: Priority: Blocker (was: Critical) Fix Version/s: 2.0.3-alpha > MRApps distributed-cache duplicate checks are incorrect > --- > > Key: MAPREDUCE-4820 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4820 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.2-alpha >Reporter: Alejandro Abdelnur >Priority: Blocker > Fix For: 2.0.3-alpha > > > This seems a combination of issues that are being exposed in 2.0.2-alpha by > MAPREDUCE-4549. > MAPREDUCE-4549 introduces a check to to ensure there are not duplicate JARs > in the distributed-cache (using the JAR name as identity). > In Hadoop 2 (different from Hadoop 1), all JARs in the distributed-cache are > symlink-ed to the current directory of the task. > MRApps, when setting up the DistributedCache > (MRApps#setupDistributedCache->parseDistributedCacheArtifacts) assumes that > the local resources (this includes files in the CURRENT_DIR/, > CURRENT_DIR/classes/ and files in CURRENT_DIR/lib/) are part of the > distributed-cache already. > For systems, like Oozie, which use a launcher job to submit the real job this > poses a problem because MRApps is run from the launcher job to submit the > real job. The configuration of the real job has the correct distributed-cache > entries (no duplicates), but because the current dir has the same files, the > submission fails. > It seems that MRApps should not be checking dups in the distributed-cached > against JARs in the CURRENT_DIR/ or CURRENT_DIR/lib/. The dup check should be > done among distributed-cached entries only. > It seems YARNRunner is symlink-ing all files in the distributed cached in the > current directory. In Hadoop 1 this was done only for files added to the > distributed-cache using a fragment (ie "#FOO") to trigger a symlink creation. > Marking as a blocker because without a fix for this, Oozie cannot submit jobs > to Hadoop 2 (i've debugged Oozie in a live cluster being used by BigTop > -thanks Roman- to test their release work, and I've verified that Oozie 3.3 > does not create duplicated entries in the distributed-cache) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4219) make default container-executor.conf.dir be a path relative to the container-executor binary
[
https://issues.apache.org/jira/browse/MAPREDUCE-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266945#comment-13266945
]
Roman Shaposhnik commented on MAPREDUCE-4219:
-
I tested Hadoop examples on the fully distributed secure cluster with ../conf
baked in as a HADOOP_CONF
> make default container-executor.conf.dir be a path relative to the
> container-executor binary
>
>
> Key: MAPREDUCE-4219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4219
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: security
>Affects Versions: 0.23.1
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
> Fix For: 2.0.0
>
> Attachments: HADOOP-8332.patch.txt, MAPREDUCE-4219.patch.txt
>
>
> Currently, container-executor binary has an absolute pathname of its
> configuration file baked in. This prevents an easy relocation of the
> configuration files when dealing with multiple Hadoop installs on the same
> node. It would be nice to at least allow for a relative path resolution
> starting from the location of the container-executor binary itself. Something
> like
> {noformat}
> ../etc/hadoop/
> {noformat}
> Thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4219) make default container-executor.conf.dir be a path relative to the container-executor binary
[
https://issues.apache.org/jira/browse/MAPREDUCE-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated MAPREDUCE-4219:
Attachment: MAPREDUCE-4219.patch.txt
Please take a look at the new patch.
> make default container-executor.conf.dir be a path relative to the
> container-executor binary
>
>
> Key: MAPREDUCE-4219
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4219
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: security
>Affects Versions: 0.23.1
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
> Fix For: 2.0.0
>
> Attachments: HADOOP-8332.patch.txt, MAPREDUCE-4219.patch.txt
>
>
> Currently, container-executor binary has an absolute pathname of its
> configuration file baked in. This prevents an easy relocation of the
> configuration files when dealing with multiple Hadoop installs on the same
> node. It would be nice to at least allow for a relative path resolution
> starting from the location of the container-executor binary itself. Something
> like
> {noformat}
> ../etc/hadoop/
> {noformat}
> Thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2314) configure files that are generated as part of the released tarball need to have executable bit set
[ https://issues.apache.org/jira/browse/MAPREDUCE-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated MAPREDUCE-2314: Component/s: build Affects Version/s: 0.22.0 > configure files that are generated as part of the released tarball need to > have executable bit set > -- > > Key: MAPREDUCE-2314 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2314 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Affects Versions: 0.22.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-2314.patch > > > Currently the configure files that are packaged in a tarball are -rw-rw-r-- -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2314) configure files that are generated as part of the released tarball need to have executable bit set
[ https://issues.apache.org/jira/browse/MAPREDUCE-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994993#comment-12994993 ] Roman Shaposhnik commented on MAPREDUCE-2314: - To answer a chmod question -- I followed the already established pattern of not doing chmods in parallel. > configure files that are generated as part of the released tarball need to > have executable bit set > -- > > Key: MAPREDUCE-2314 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2314 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-2314.patch > > > Currently the configure files that are packaged in a tarball are -rw-rw-r-- -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2314) configure files that are generated as part of the released tarball need to have executable bit set
[ https://issues.apache.org/jira/browse/MAPREDUCE-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994991#comment-12994991 ] Roman Shaposhnik commented on MAPREDUCE-2314: - $ ant veryclean test-patch -Dpatch.file=/home/rvs/MAPREDUCE-2314.patch -Dscratch.dir=/tmp/mapred/scratch -Dfindbugs.home=/home/rvs/findbugs-1.3.9 -Dforrest.home=/home/rvs/src/apache-forrest-0.8 . [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. And given that it is a build change " -1 tests included" can be ignored > configure files that are generated as part of the released tarball need to > have executable bit set > -- > > Key: MAPREDUCE-2314 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2314 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-2314.patch > > > Currently the configure files that are packaged in a tarball are -rw-rw-r-- -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2314) configure files that are generated as part of the released tarball need to have executable bit set
[ https://issues.apache.org/jira/browse/MAPREDUCE-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated MAPREDUCE-2314: Attachment: MAPREDUCE-2314.patch > configure files that are generated as part of the released tarball need to > have executable bit set > -- > > Key: MAPREDUCE-2314 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2314 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-2314.patch > > > Currently the configure files that are packaged in a tarball are -rw-rw-r-- -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2314) configure files that are generated as part of the released tarball need to have executable bit set
[ https://issues.apache.org/jira/browse/MAPREDUCE-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated MAPREDUCE-2314: Status: Patch Available (was: Open) > configure files that are generated as part of the released tarball need to > have executable bit set > -- > > Key: MAPREDUCE-2314 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2314 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-2314.patch > > > Currently the configure files that are packaged in a tarball are -rw-rw-r-- -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2314) configure files that are generated as part of the released tarball need to have executable bit set
configure files that are generated as part of the released tarball need to have executable bit set -- Key: MAPREDUCE-2314 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2314 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Currently the configure files that are packaged in a tarball are -rw-rw-r-- -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2260) Remove auto-generated native build files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988287#action_12988287 ] Roman Shaposhnik commented on MAPREDUCE-2260: - To finish up my previous answer -- the command used to build hadoop-mapreduce was: ant -Dcompile.native=true -Dcompile.c++=true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop-0.20/conf -Dlibrecordio=true veryclean api-report task-controller compile-c++ tar Finally, as for testing -- examples (pipes) were executed > Remove auto-generated native build files > > > Key: MAPREDUCE-2260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2260 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-2260.diff > > Original Estimate: 24h > Remaining Estimate: 24h > > The repo currently includes the automake and autoconf generated files for the > native build. Per discussion on HADOOP-6421 let's remove them and use the > host's automake and autoconf. We should also do this for libhdfs and > fuse-dfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2260) Remove auto-generated native build files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988276#action_12988276 ] Roman Shaposhnik commented on MAPREDUCE-2260: - bq. it would be good to verify that ... ant tar produces a tarball which contains the generated files (ie a dist style tarball). Since one has to specify extra properties in order for native bits to be included in a dist style tarball here's a list of builds that were done: cd hadoop-common ; ant -Dcompile.native=true -Dcompile.c++=true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop/conf veryclean api-report compile-core-native tar cd hadoop-hdfs ; ant -Dcompile.native=true -Dcompile.c++=true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop/conf -Dlibhdfs=1 veryclean api-report tar cd hadoop-mapreduce ; That resulted in the following native bits being created: c++/Linux-amd64-64/include/hadoop/Pipes.hh c++/Linux-amd64-64/include/hadoop/SerialUtils.hh c++/Linux-amd64-64/include/hadoop/TemplateFactory.hh c++/Linux-amd64-64/include/hadoop/StringUtils.hh c++/Linux-amd64-64/lib/libhadooppipes.a c++/Linux-amd64-64/lib/libhadooputils.a c++/Linux-amd64-64/lib/libhdfs.a c++/Linux-amd64-64/lib/libhdfs.so.0.0.0 c++/Linux-amd64-64/lib/libhdfs.so.0 c++/Linux-amd64-64/lib/libhdfs.so c++/Linux-amd64-64/lib/libhdfs.la c++/lib/libhdfs.a c++/lib/libhdfs.so.0.0.0 c++/lib/libhdfs.so.0 c++/lib/libhdfs.so c++/lib/libhdfs.la librecordio/librecordio.a bin/task-controller lib/native/Linux-amd64-64/libhadoop.so lib/native/Linux-amd64-64/libhadoop.a lib/native/Linux-amd64-64/libhadoop.so.1 lib/native/Linux-amd64-64/libhadoop.so.1.0.0 lib/native/Linux-amd64-64/libhadoop.la > Remove auto-generated native build files > > > Key: MAPREDUCE-2260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2260 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-2260.diff > > Original Estimate: 24h > Remaining Estimate: 24h > > The repo currently includes the automake and autoconf generated files for the > native build. Per discussion on HADOOP-6421 let's remove them and use the > host's automake and autoconf. We should also do this for libhdfs and > fuse-dfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2260) Remove auto-generated native build files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated MAPREDUCE-2260: Attachment: MAPREDUCE-2260.diff > Remove auto-generated native build files > > > Key: MAPREDUCE-2260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2260 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-2260.diff > > Original Estimate: 24h > Remaining Estimate: 24h > > The repo currently includes the automake and autoconf generated files for the > native build. Per discussion on HADOOP-6421 let's remove them and use the > host's automake and autoconf. We should also do this for libhdfs and > fuse-dfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2260) Remove auto-generated native build files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated MAPREDUCE-2260: Status: Patch Available (was: Open) I'm attaching a first cut at the patch. Please let me know what do you think about the approach taken. Notes: 1. since none of the C++ projects is imported (they are all developed as part of the Hadoop mapreduce workspace) it seems that we all would be better off defining a single common umbrella C++ project with a single autotools metastructure and 5 different subdirectories corresponding to pipes, pipes-examples, task-controller, utils and librecordio. We don't really gain anything by splitting C++ sources at the level of Ant. 2. the build (at least the release build) now depends on the entire autoconf > Remove auto-generated native build files > > > Key: MAPREDUCE-2260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2260 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Original Estimate: 24h > Remaining Estimate: 24h > > The repo currently includes the automake and autoconf generated files for the > native build. Per discussion on HADOOP-6421 let's remove them and use the > host's automake and autoconf. We should also do this for libhdfs and > fuse-dfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-2260) Remove auto-generated native build files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik reassigned MAPREDUCE-2260: --- Assignee: Roman Shaposhnik > Remove auto-generated native build files > > > Key: MAPREDUCE-2260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2260 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Original Estimate: 24h > Remaining Estimate: 24h > > The repo currently includes the automake and autoconf generated files for the > native build. Per discussion on HADOOP-6421 let's remove them and use the > host's automake and autoconf. We should also do this for libhdfs and > fuse-dfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2260) Remove auto-generated native build files
Remove auto-generated native build files Key: MAPREDUCE-2260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2260 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Reporter: Roman Shaposhnik The repo currently includes the automake and autoconf generated files for the native build. Per discussion on HADOOP-6421 let's remove them and use the host's automake and autoconf. We should also do this for libhdfs and fuse-dfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
