[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade
[ https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266461#comment-16266461 ] Ray Chiang commented on HADOOP-15059: - Yeah, I'm pretty sure with HDFS-11096, we were launching jobs entirely from a 2.8.0 machine and not the reverse (i.e. not testing the 3.0 client with the 2.9 MR jars). That's my error for not catching that. I had to read up a bit on delegation tokens before I could understand how things worked. I'm +1 on [~jlowe]'s first suggestion (writing out two credential files). That seems like the simplest workaround. I don't think it will be that much of a burden and the file should be removed during container cleanup. > 3.0 deployment cannot work with old version MR tar ball which break rolling > upgrade > --- > > Key: HADOOP-15059 > URL: https://issues.apache.org/jira/browse/HADOOP-15059 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: Junping Du >Priority: Blocker > > I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed > because following error: > {noformat} > 2017-11-21 12:42:50,911 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for > application appattempt_1511295641738_0003_01 > 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > 2017-11-21 12:42:51,118 FATAL [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster > java.lang.RuntimeException: Unable to determine current user > at > org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254) > at > org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220) > at > org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212) > at > org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638) > Caused by: java.io.IOException: Exception reading > /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens > at > org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208) > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907) > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820) > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689) > at > org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252) > ... 4 more > Caused by: java.io.IOException: Unknown version 1 in token storage. > at > org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226) > at > org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205) > ... 8 more > 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting > with status 1: java.lang.RuntimeException: Unable to determine current user > {noformat} > I think it is due to token incompatiblity change between 2.9 and 3.0. As we > claim "rolling upgrade" is supported in Hadoop 3, we should fix this before > we ship 3.0 otherwise all MR running applications will get stuck during/after > upgrade. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3A committer for zero-rename commits to S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266376#comment-16266376 ] Rohith Sharma K S commented on HADOOP-13786: This patch generates java doc errors. See MAPREDUCE-7014. > Add S3A committer for zero-rename commits to S3 endpoints > - > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 3.1.0 > > Attachments: HADOOP-13786-036.patch, HADOOP-13786-037.patch, > HADOOP-13786-038.patch, HADOOP-13786-039.patch, > HADOOP-13786-HADOOP-13345-001.patch, HADOOP-13786-HADOOP-13345-002.patch, > HADOOP-13786-HADOOP-13345-003.patch, HADOOP-13786-HADOOP-13345-004.patch, > HADOOP-13786-HADOOP-13345-005.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-007.patch, > HADOOP-13786-HADOOP-13345-009.patch, HADOOP-13786-HADOOP-13345-010.patch, > HADOOP-13786-HADOOP-13345-011.patch, HADOOP-13786-HADOOP-13345-012.patch, > HADOOP-13786-HADOOP-13345-013.patch, HADOOP-13786-HADOOP-13345-015.patch, > HADOOP-13786-HADOOP-13345-016.patch, HADOOP-13786-HADOOP-13345-017.patch, > HADOOP-13786-HADOOP-13345-018.patch, HADOOP-13786-HADOOP-13345-019.patch, > HADOOP-13786-HADOOP-13345-020.patch, HADOOP-13786-HADOOP-13345-021.patch, > HADOOP-13786-HADOOP-13345-022.patch, HADOOP-13786-HADOOP-13345-023.patch, > HADOOP-13786-HADOOP-13345-024.patch, HADOOP-13786-HADOOP-13345-025.patch, > HADOOP-13786-HADOOP-13345-026.patch, HADOOP-13786-HADOOP-13345-027.patch, > HADOOP-13786-HADOOP-13345-028.patch, HADOOP-13786-HADOOP-13345-028.patch, > HADOOP-13786-HADOOP-13345-029.patch, HADOOP-13786-HADOOP-13345-030.patch, > HADOOP-13786-HADOOP-13345-031.patch, HADOOP-13786-HADOOP-13345-032.patch, > HADOOP-13786-HADOOP-13345-033.patch, HADOOP-13786-HADOOP-13345-035.patch, > MAPREDUCE-6823-003.patch, cloud-intergration-test-failure.log, > objectstore.pdf, s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266372#comment-16266372 ] SammiChen commented on HADOOP-14964: Hi [~djp], thanks for you feedback and approve. I'm so glad that current design timely addressed your concern. The patch for 2.8.3 will be cherry picked from branch-2. I would like to wait 2~3 days to see if there is other feedback from the community. If no further feedback, I will commit the patch to 2.8.3 and update the release note accordingly. > AliyunOSS: backport Aliyun OSS module to branch-2 > - > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Fix For: 2.9.1 > > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch, > HADOOP-14964-branch-2.9.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266371#comment-16266371 ] SammiChen commented on HADOOP-14964: Hi [~chris.douglas], thanks for the suggestion. I recommitted the code to branch-2.9 with the "cherry-pick" note. The release not is added. Currently, it's just for 2.9.1. Once the patch goes into 2.8.3, I will update the release note. > AliyunOSS: backport Aliyun OSS module to branch-2 > - > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Fix For: 2.9.1 > > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch, > HADOOP-14964-branch-2.9.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-14964: --- Release Note: OSS is widely used among China’s cloud users and this work implemented a new Hadoop compatible filesystem AliyunOSSFileSystem with oss scheme, similar to the s3a and azure support. Currently, the feature is support in 2.9.1. > AliyunOSS: backport Aliyun OSS module to branch-2 > - > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Fix For: 2.9.1 > > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch, > HADOOP-14964-branch-2.9.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-14964: --- Fix Version/s: 2.9.1 > AliyunOSS: backport Aliyun OSS module to branch-2 > - > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Fix For: 2.9.1 > > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch, > HADOOP-14964-branch-2.9.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14964) AliyunOSS: backport Aliyun OSS module to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-14964: --- Summary: AliyunOSS: backport Aliyun OSS module to branch-2 (was: AliyunOSS: backport Aliyun OSS module to branch-2 and 2.8+ branches) > AliyunOSS: backport Aliyun OSS module to branch-2 > - > > Key: HADOOP-14964 > URL: https://issues.apache.org/jira/browse/HADOOP-14964 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Reporter: Genmao Yu >Assignee: SammiChen > Attachments: HADOOP-14964-branch-2.000.patch, > HADOOP-14964-branch-2.8.000.patch, HADOOP-14964-branch-2.8.001.patch, > HADOOP-14964-branch-2.9.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14875) Create end user documentation from the compatibility guidelines
[ https://issues.apache.org/jira/browse/HADOOP-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266286#comment-16266286 ] Daniel Templeton commented on HADOOP-14875: --- Any other comments, [~anu], [~rkanter], [~steve_l], [~leftnoteasy], or [~sunilg]? > Create end user documentation from the compatibility guidelines > --- > > Key: HADOOP-14875 > URL: https://issues.apache.org/jira/browse/HADOOP-14875 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: AdminCompatibilityGuide.pdf, Compatibility.htm, > HADOOP-14875.001.patch, HADOOP-14875.002.patch, HADOOP-14875.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12956) Inevitable Log4j2 migration via slf4j
[ https://issues.apache.org/jira/browse/HADOOP-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266158#comment-16266158 ] Ralph Goers commented on HADOOP-12956: -- The above patch should simplify some of the challenges other projects have in moving to Log4j 2 as Hadoop won't require Log4j 1's AppenderSkeleton unless Log4j 1 is being used. It also eliminates the need to exclude the log4j 1.x and slf4j-log4j12 jars since they are now optional. > Inevitable Log4j2 migration via slf4j > - > > Key: HADOOP-12956 > URL: https://issues.apache.org/jira/browse/HADOOP-12956 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Gopal V >Assignee: Haohui Mai > > {{5 August 2015 --The Apache Logging Services™ Project Management Committee > (PMC) has announced that the Log4j™ 1.x logging framework has reached its end > of life (EOL) and is no longer officially supported.}} > https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces > A whole framework log4j2 upgrade has to be synchronized, partly for improved > performance brought about by log4j2. > https://logging.apache.org/log4j/2.x/manual/async.html#Performance -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org