Drill Developers,
I have been struggling to change the aws credentials when running drill. I am
using session tokens in a local profile. I also run the app using IAM roles,
but right now I just want it to work locally. This is in my core-site.xml:
<property>
<key>fs.s3a.aws.credentials.provider</key>
<value>com.amazonaws.auth.profile.ProfileCredentialsProvider,org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider</value>
</property>
<property>
<key>fs.s3a.security.credential.provider.path</key>
<value>com.amazonaws.auth.profile.ProfileCredentialsProvider,org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider</value>
</property>
<property>
<key>hadoop.security.credential.provider.path</key>
<value>com.amazonaws.auth.profile.ProfileCredentialsProvider,org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider</value>
</property>
but unfortunately when I run the application (with several loggers set to
trace) I still see this:
2017-04-18 23:07:31,725 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] TRACE
o.a.d.exec.util.ImpersonationUtil - Creating DrillFileSystem for proxy user:
drill (auth:SIMPLE)
2017-04-18 23:07:31,986 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] DEBUG
c.a.auth.AWSCredentialsProviderChain - Unable to load credentials from
BasicAWSCredentialsProvider: Access key or secret key is null
2017-04-18 23:07:34,003 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] DEBUG
c.a.auth.AWSCredentialsProviderChain - Unable to load credentials from
InstanceProfileCredentialsProvider: Unable to load credentials from Amazon EC2
metadata service
2017-04-18 23:07:34,030 [270963cb-cb2a-aa04-3ad1-1a92384a31f5:foreman] DEBUG
o.a.drill.exec.work.foreman.Foreman - 270963cb-cb2a-aa04-3ad1-1a92384a31f5:
State change requested STARTING --> FAILED
org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
during fragment initialization: Unable to load AWS credentials from any
provider in the chain
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:298)
[drill-java-exec-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials
from any provider in the chain
The application is only checking BasicAWDCredentialsProvider and
InstanceProfileCredentialsProvider. Neither of those will work for me. I have
been searching the source code for a while now and none of the properties I try
to set seem to actually work.
Would somebody please tell me how to configure the credentials provider chain
in drill?
Michael Knapp
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to
Capital One and/or its affiliates and may only be used solely in performance of
work or services for Capital One. The information transmitted herewith is
intended only for use by the individual or entity to which it is addressed. If
the reader of this message is not the intended recipient, you are hereby
notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is
strictly prohibited. If you have received this communication in error, please
contact the sender and delete the material from your computer.
