Joe McDonnell created IMPALA-9677: ------------------------------------- Summary: FE Analysis tests using fake S3 bucket fail with AnalysisException Key: IMPALA-9677 URL: https://issues.apache.org/jira/browse/IMPALA-9677 Project: IMPALA Issue Type: Bug Components: Frontend Affects Versions: Impala 4.0 Reporter: Joe McDonnell
It looks like Hadoop added extra validation for initializing the S3AFilesystem. [HADOOP-16711|https://issues.apache.org/jira/browse/HADOOP-16711] now defaults to checking for the existence of the bucket and permissions on it. The level of validation is controlled by fs.s3a.bucket.probe. We use a bogus S3 bucket in our analysis tests with the expectation that all we require is a valid URI. Instead, we get this error: {noformat} Error during analysis: org.apache.impala.common.AnalysisException: bucket: doesBucketExistV2 on bucket: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 10988D12BA86A604; S3 Extended Request ID: qbfw3pMHGGiPTZyLiAOIAvMlJLpy5HOgkHSN4I1n1pBhX2H3BYSwDXdt+gIi3+xiA+E2w5Itv5o=), S3 Extended Request ID: qbfw3pMHGGiPTZyLiAOIAvMlJLpy5HOgkHSN4I1n1pBhX2H3BYSwDXdt+gIi3+xiA+E2w5Itv5o=:InvalidAccessKeyId sql: create table tbl (i int) location 's3a://bucket/test-warehouse/new_table'{noformat} Developers frequently don't have S3 credentials set up, so this will always fail. Currently, these tests fails: {noformat} org.apache.impala.analysis.AnalyzeDDLTest.TestAlterTableAddDropPartition org.apache.impala.analysis.AnalyzeDDLTest.TestCreateDataSource org.apache.impala.analysis.AnalyzeDDLTest.TestCreateDb org.apache.impala.analysis.AnalyzeDDLTest.TestAlterTableSet org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLike org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTable org.apache.impala.planner.ExplainTest.testScanNodeFsScheme org.apache.impala.planner.PlannerTest.testScanNodeFsScheme{noformat} It reproduces consistently on jenkins.impala.io: [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch-cdp-hive/69/] -- This message was sent by Atlassian Jira (v8.3.4#803005)