Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/8294 )
Change subject: IMPALA-6067: Enable s3 access via IAM roles for EC2 VMs ...................................................................... Patch Set 3: >[...]Please do answer the question about changing > the FE tests to just use s3a so that we can remove the hackery > altogether.[...] In short: it depends on your end goal. If the end goal is to get rid of the s3n: provider and its supporting menagerie (jets3t, etc.), then yes, it can be done with a small price: a single negative test in AnalyzeStmtsTest.java will have to go. The code there in https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java#L3000 performs a test step that should fail for s3n -- but if I remove the s3n properties from core-site.xml, this line blows up complaining about missing credentials. The stack trace reveals that Impala's HdfsUri::analyze() calls into Hadoop's Path::getFileSystem() to find out what kind of filesystem the URI points to; this in turn calls down all the way to org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:74) being called from Jets3t. If your end goal is to remove dummy credentials from core-site.xml.tmpl, this will probably not be complete success: the s3a: provider behaves in a similar () though not identical) way. When the front-end tests are run without an S3 container in sight (e.g. your local dev box with no S3 access set up), AnalyzeDDLTest's s3a: URIs still require similar dummy credentials, but with s3a: these can be supplied in environment variables. This means that core-site.xml[.tmpl] could be cleaned up, but dummy credentials in envireonment variables would still be there. -- To view, visit http://gerrit.cloudera.org:8080/8294 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I14cd9d4453a91baad3c379aa7e4944993fca95ae Gerrit-Change-Number: 8294 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal <laszlo.g...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: David Knupp <dkn...@cloudera.com> Gerrit-Reviewer: Jim Apple <jbapple-imp...@apache.org> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Lars Volker <l...@cloudera.com> Gerrit-Reviewer: Laszlo Gaal <laszlo.g...@cloudera.com> Gerrit-Reviewer: Michael Brown <mi...@cloudera.com> Gerrit-Reviewer: Philip Zeyliger <phi...@cloudera.com> Gerrit-Reviewer: Sailesh Mukil <sail...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Comment-Date: Wed, 22 Nov 2017 21:04:24 +0000 Gerrit-HasComments: No
