I must have forgotten my original issue when I sent that last response. While I am able to use drill on AWS, my original problem was that I cannot use it locally.
I actually have not solved this problem yet. When I run drill locally, it is still using BasicAWSCredentialsProvider and InstanceProfileCredentialsProvider, neither of which will work on my local machine. I need it to use “com.amazonaws.auth.profile.ProfileCredentialsProvider”, but it is not. Despite all the properties I have set in core-site.xml. The source code you have is very complicated so it’s hard for me to track down exactly what is happening, but after browsing it I decided to try two things: 1. I set HADOOP_HOME to point to my drill home directory. This is because in the FileSystemPlugin source code (line 71 of version 1.10), I see that you merely create a new Configuration. At no point do you tell it what core-site.xml file to use, which surprised me. Are you sure that the drill conf/core-site.xml file is even used here? I am not using Hadoop, so there was no HADOOP_HOME set, I decided to set it in hopes that the Drill conf would pick up the creds provider chain I set. 2. I decided to copy the core-site settings into the storage plugin definition on the UI. I did this because I see that these values override the Hadoop configuration in the FileSystemPlugin. Despite that, Drill is still using the wrong credentials providers. Would somebody please tell me how to set the AWS credentials providers for Drill? I am seriously struggling with this, nothing I am trying is working. Michael Knapp On 4/20/17, 5:51 PM, "Knapp, Michael" <[email protected]> wrote: I am still not sure why Zeppelin is not using my credentials provider chain, but I discovered that InstanceProfileCredentialsProvider is actually what I need, it loads credentials from IAM roles. In my storage plugin, I left the bucket as the default value because I provided the whole path in my query: SELECT * FROM s3.`s3a://my-bucket/path/to/my/file` limit 3; But it seems like when Zeppelin tries to query S3, it is checking the bucket defined in the storage plugin. It seems like incorrect logic to me. In any case, when I corrected the bucket in the storage plugin, I am able to query S3 from drill inside docker now. On 4/19/17, 9:44 AM, "Knapp, Michael" <[email protected]> wrote: Yes I have. The s3 storage plugin does not work with session tokens, that is not an option. On 4/18/17, 7:51 PM, "Abhishek Girish" <[email protected]> wrote: Hey Michael, Have you copied over the core-site.xml file onto Drill's conf directory? You could also set the credentials directly in the s3 storage plugin [1]. [1] https://drill.apache.org/docs/s3-storage-plugin/ -Abhishek On Tue, Apr 18, 2017 at 4:17 PM Knapp, Michael <[email protected]> wrote: > Drill Developers, > > I have been struggling to change the aws credentials when running drill. > I am using session tokens in a local profile. I also run the app using IAM > roles, but right now I just want it to work locally. This is in my > core-site.xml: > > > <property> > <key>fs.s3a.aws.credentials.provider</key> > > <value>com.amazonaws.auth.profile.ProfileCredentialsProvider,org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider</value> > </property> > <property> > <key>fs.s3a.security.credential.provider.path</key> > <value>com.amazonaws.auth.profile.ProfileCredentialsProvider,org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider</value> > </property> > <property> > <key>hadoop.security.credential.provider.path</key> > <value>com.amazonaws.auth.profile.ProfileCredentialsProvider,org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider</value> > </property> > > > but unfortunately when I run the application (with several loggers set to > trace) I still see this: > 2017-04-18 23:07:31,725 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] > TRACE o.a.d.exec.util.ImpersonationUtil - Creating DrillFileSystem for > proxy user: drill (auth:SIMPLE) > 2017-04-18 23:07:31,986 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] > DEBUG c.a.auth.AWSCredentialsProviderChain - Unable to load credentials > from BasicAWSCredentialsProvider: Access key or secret key is null > 2017-04-18 23:07:34,003 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] > DEBUG c.a.auth.AWSCredentialsProviderChain - Unable to load credentials > from InstanceProfileCredentialsProvider: Unable to load credentials from > Amazon EC2 metadata service > 2017-04-18 23:07:34,030 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] > DEBUG o.a.drill.exec.work.foreman.Foreman - > 270963cb-cb2a-aa04-3ad1-1a92384a31f5: State change requested STARTING --> > FAILED > org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception > during fragment initialization: Unable to load AWS credentials from any > provider in the chain > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:298) > [drill-java-exec-1.10.0.jar:1.10.0] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] > Caused by: com.amazonaws.AmazonClientException: Unable to load AWS > credentials from any provider in the chain > > > The application is only checking BasicAWDCredentialsProvider and > InstanceProfileCredentialsProvider. Neither of those will work for me. I > have been searching the source code for a while now and none of the > properties I try to set seem to actually work. > > Would somebody please tell me how to configure the credentials provider > chain in drill? > > Michael Knapp > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. > ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer. ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
