Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

[Anup Tiwari]

looping dev in for suggestions.

Regards,
Anup Tiwari

On Thu, Apr 12, 2018 at 8:39 PM, Richard A. Bross 
wrote:

> To be honest, my advice was just based on your post - we only use Hive in
> AWS EMR, so I couldn't tell you.
>
> Glad that at least you're back up though.
>
> - Original Message -
> From: "Anup Tiwari" 
> To: user@hive.apache.org
> Sent: Thursday, April 12, 2018 10:50:23 AM
> Subject: Re: Unable to read hive external table data which is linked to s3
> after upgradation from 2.1.1 to 2.3.3
>
>
>
>
>
>
>
>
> Hi Richard ,
>
> After looking at hive 2.3.3 logs i found that when we are loading all
> configuration parameters then below message is getting printed :-
>
> Values omitted for security reason if present: [fs.s3n.awsAccessKeyId,
> fs.s3a.access.key, fs.s3.awsAccessKeyId, hive.server2.keystore.password, f
> s.s3a.proxy.password, javax.jdo.option.ConnectionPassword,
> fs.s3.awsSecretAccessKey, fs.s3n.awsSecretAccessKey, fs.s3a.secret.key]
>
>
> while in hive 2.1.1 logs i found below message :-
>
> Values omitted for security reason if present: 
> [hive.server2.keystore.password,
> javax.jdo.option.ConnectionPassword]
>
> Can this be the reason why hive 2.3.3 is not able to read s3 related
> params from hive-site.xml?
>
> I found one of JIRA : https://issues.apache.org/jira/browse/HIVE-14588
>
>
>
> Also i have set below property in hive-site.xml by excluding above s3
> variables(fs.s3a.access.key,fs.s3a.secret.key) from list and it worked.
>
> 
> hive.conf.hidden.list
> javax.jdo.option.ConnectionPassword,hive.
> server2.keystore.password,fs.s3.awsAccessKeyId,fs.s3.
> awsSecretAccessKey,fs.s3n.awsAccessKeyId,fs.s3n.awsSecretAccessKey,fs.s3a.
> proxy.password
> Comma separated list of configuration options which should
> not be read by normal user like passwords.Anup has excluded 2 variable
> which were fs.s3a.access.key,fs.s3a.secret.key
> 
>
> Let me know if there is any other solution because i think if these
> variables are by default part of hidden.list then there will be some other
> proper workaround for this.
>
>
>
>
>
>
>
>
>
> Regards,
> Anup Tiwari
>
>
> On Thu, Apr 12, 2018 at 7:44 PM, Richard A. Bross < r...@oaktreepeak.com >
> wrote:
>
>
> I hear you, but given the exception log, it does seem that it can't
> authenticate you. You can try using the AWS environment variables. If that
> resolves the issue then you'll have some more to go on. According to
> Hortonworks here:
>
> https://hortonworks.github.io/hdp-aws/s3-security/
>
> "AWS CLI supports authentication through environment variables. These same
> environment variables will be used by Hadoop if no configuration properties
> are set."
>
>
> - Original Message -
> From: "Anup Tiwari" < anupsdtiw...@gmail.com >
> To: user@hive.apache.org
> Sent: Thursday, April 12, 2018 10:06:33 AM
> Subject: Re: Unable to read hive external table data which is linked to s3
> after upgradation from 2.1.1 to 2.3.3
>
>
> We are not using EMR. Also we have set below params for accessing s3
> bucket in hive-site.xml which are same as what we have set in hive 2.1.1.
>
>
> * fs.s3a.access.key
> * fs.s3a.secret.key
> * fs.s3a.connection.maximum
> * fs.s3a.impl
>
>
>
>
>
>
>
>
> Regards,
> Anup Tiwari
>
>
> On Thu, Apr 12, 2018 at 7:19 PM, Richard A. Bross < r...@oaktreepeak.com >
> wrote:
>
>
> Based on the exception, it looks more like an AWS credentials issue than a
> Hive issue. Are you running in AWS EMR, on-prem?
>
> In AWS the resource accessing the S3 bucket would have to have an IAM that
> gave permission. If you are running somewhere else whatever AWS login you
> use would have to have the correct permissions in the IAM.
>
>
>
> - Original Message -
> From: "Anup Tiwari" < anupsdtiw...@gmail.com >
> To: user@hive.apache.org
> Sent: Thursday, April 12, 2018 9:11:37 AM
> Subject: Unable to read hive external table data which is linked to s3
> after upgradation from 2.1.1 to 2.3.3
>
>
>
> Hi All,
>
> When i am trying to read s3 linked external table in 2.3.3 ; i am getting
> errors.It was working properly in 2.1.1. please find below details and let
> me know if i am missing something :-
>
>
> Hadoop Version :- 2.8.0
>
> Query :-
>
> select log_date,count(1) as cnt from test.tt1 group by log_date;
>
> Error :-
>
> Vertex failed, vertexName=Map 1, vertexId=vertex_1523502631429_0029_3_00,
> diagnostics=[Vertex vertex_1523502631429_0029_3_00 [Map 1] killed/failed
> due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tt1 initializer failed,
> vertex=vertex_1523502631429_0029_3_00 [Map 1], 
> org.apache.hadoop.fs.s3a.AWSClientIOException:
> doesBucketExist on g24x7.new-analytics: com.amazonaws.AmazonClientException:
> No AWS Credentials provided by BasicAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider
> : com.amazonaws.AmazonClientException: The requested metadata is not
> found at 

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

[Richard A. Bross]

To be honest, my advice was just based on your post - we only use Hive in AWS 
EMR, so I couldn't tell you.

Glad that at least you're back up though.

- Original Message -
From: "Anup Tiwari" 
To: user@hive.apache.org
Sent: Thursday, April 12, 2018 10:50:23 AM
Subject: Re: Unable to read hive external table data which is linked to s3 
after upgradation from 2.1.1 to 2.3.3








Hi Richard , 

After looking at hive 2.3.3 logs i found that when we are loading all 
configuration parameters then below message is getting printed :- 

Values omitted for security reason if present: [fs.s3n.awsAccessKeyId, 
fs.s3a.access.key, fs.s3.awsAccessKeyId, hive.server2.keystore.password, f 
s.s3a.proxy.password, javax.jdo.option.ConnectionPassword, 
fs.s3.awsSecretAccessKey, fs.s3n.awsSecretAccessKey, fs.s3a.secret.key] 


while in hive 2.1.1 logs i found below message :- 

Values omitted for security reason if present: [hive.server2.keystore.password, 
javax.jdo.option.ConnectionPassword] 

Can this be the reason why hive 2.3.3 is not able to read s3 related params 
from hive-site.xml? 

I found one of JIRA : https://issues.apache.org/jira/browse/HIVE-14588 



Also i have set below property in hive-site.xml by excluding above s3 
variables(fs.s3a.access.key,fs.s3a.secret.key) from list and it worked. 

 
hive.conf.hidden.list 
javax.jdo.option.ConnectionPassword,hive.server2.keystore.password,fs.s3.awsAccessKeyId,fs.s3.awsSecretAccessKey,fs.s3n.awsAccessKeyId,fs.s3n.awsSecretAccessKey,fs.s3a.proxy.password
 
Comma separated list of configuration options which should not be 
read by normal user like passwords.Anup has excluded 2 variable which were 
fs.s3a.access.key,fs.s3a.secret.key 
 

Let me know if there is any other solution because i think if these variables 
are by default part of hidden.list then there will be some other proper 
workaround for this. 









Regards, 
Anup Tiwari 


On Thu, Apr 12, 2018 at 7:44 PM, Richard A. Bross < r...@oaktreepeak.com > 
wrote: 


I hear you, but given the exception log, it does seem that it can't 
authenticate you. You can try using the AWS environment variables. If that 
resolves the issue then you'll have some more to go on. According to 
Hortonworks here: 

https://hortonworks.github.io/hdp-aws/s3-security/ 

"AWS CLI supports authentication through environment variables. These same 
environment variables will be used by Hadoop if no configuration properties are 
set." 


- Original Message - 
From: "Anup Tiwari" < anupsdtiw...@gmail.com > 
To: user@hive.apache.org 
Sent: Thursday, April 12, 2018 10:06:33 AM 
Subject: Re: Unable to read hive external table data which is linked to s3 
after upgradation from 2.1.1 to 2.3.3 


We are not using EMR. Also we have set below params for accessing s3 bucket in 
hive-site.xml which are same as what we have set in hive 2.1.1. 


* fs.s3a.access.key 
* fs.s3a.secret.key 
* fs.s3a.connection.maximum 
* fs.s3a.impl 








Regards, 
Anup Tiwari 


On Thu, Apr 12, 2018 at 7:19 PM, Richard A. Bross < r...@oaktreepeak.com > 
wrote: 


Based on the exception, it looks more like an AWS credentials issue than a Hive 
issue. Are you running in AWS EMR, on-prem? 

In AWS the resource accessing the S3 bucket would have to have an IAM that gave 
permission. If you are running somewhere else whatever AWS login you use would 
have to have the correct permissions in the IAM. 



- Original Message - 
From: "Anup Tiwari" < anupsdtiw...@gmail.com > 
To: user@hive.apache.org 
Sent: Thursday, April 12, 2018 9:11:37 AM 
Subject: Unable to read hive external table data which is linked to s3 after 
upgradation from 2.1.1 to 2.3.3 



Hi All, 

When i am trying to read s3 linked external table in 2.3.3 ; i am getting 
errors.It was working properly in 2.1.1. please find below details and let me 
know if i am missing something :- 


Hadoop Version :- 2.8.0 

Query :- 

select log_date,count(1) as cnt from test.tt1 group by log_date; 

Error :- 

Vertex failed, vertexName=Map 1, vertexId=vertex_1523502631429_0029_3_00, 
diagnostics=[Vertex vertex_1523502631429_0029_3_00 [Map 1] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tt1 initializer failed, 
vertex=vertex_1523502631429_0029_3_00 [Map 1], 
org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on 
g24x7.new-analytics: com.amazonaws.AmazonClientException: No AWS Credentials 
provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
SharedInstanceProfileCredentialsProvider : com.amazonaws.AmazonClientException: 
The requested metadata is not found at 
http://16x.xxx.xxx.xx4/latest/meta-data/iam/security-credentials/ : No AWS 
Credentials provided by BasicAWSCredentialsProvider 
EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider 
: com.amazonaws.AmazonClientException: The requested metadata is not found at 
http://16x.xxx.xxx.xx4/latest/meta-data/iam/security-credentials/ 
at 

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

[Anup Tiwari]

Hi Richard,

After looking at hive *2.3.3* logs i found that when we are loading all
configuration parameters then below message is getting printed :-

Values omitted for security reason if present: [fs.s3n.awsAccessKeyId,
fs.s3a.access.key, fs.s3.awsAccessKeyId, hive.server2.keystore.password, f
s.s3a.proxy.password, javax.jdo.option.ConnectionPassword,
fs.s3.awsSecretAccessKey, fs.s3n.awsSecretAccessKey, fs.s3a.secret.key]


while in hive *2.1.1* logs i found below message :-

Values omitted for security reason if present:
[hive.server2.keystore.password, javax.jdo.option.ConnectionPassword]

Can this be the reason why hive 2.3.3 is not able to read s3 related params
from hive-site.xml?

I found one of JIRA : https://issues.apache.org/jira/browse/HIVE-14588



Also i have set below property in hive-site.xml by excluding above s3
variables(fs.s3a.access.key,fs.s3a.secret.key) from list and it worked.


hive.conf.hidden.list

javax.jdo.option.ConnectionPassword,hive.server2.keystore.password,fs.s3.awsAccessKeyId,fs.s3.awsSecretAccessKey,fs.s3n.awsAccessKeyId,fs.s3n.awsSecretAccessKey,fs.s3a.proxy.password
Comma separated list of configuration options which should
not be read by normal user like passwords.Anup has excluded 2 variable
which were fs.s3a.access.key,fs.s3a.secret.key


Let me know if there is any other solution because i think if these
variables are by default part of hidden.list then there will be some other
proper workaround for this.



Regards,
Anup Tiwari

On Thu, Apr 12, 2018 at 7:44 PM, Richard A. Bross 
wrote:

> I hear you, but given the exception log, it does seem that it can't
> authenticate you.  You can try using the AWS environment variables.  If
> that resolves the issue then you'll have some more to go on.  According to
> Hortonworks here:
>
> https://hortonworks.github.io/hdp-aws/s3-security/
>
> "AWS CLI supports authentication through environment variables. These same
> environment variables will be used by Hadoop if no configuration properties
> are set."
>
>
> - Original Message -
> From: "Anup Tiwari" 
> To: user@hive.apache.org
> Sent: Thursday, April 12, 2018 10:06:33 AM
> Subject: Re: Unable to read hive external table data which is linked to s3
> after upgradation from 2.1.1 to 2.3.3
>
>
> We are not using EMR. Also we have set below params for accessing s3
> bucket in hive-site.xml which are same as what we have set in hive 2.1.1.
>
>
> * fs.s3a.access.key
> * fs.s3a.secret.key
> * fs.s3a.connection.maximum
> * fs.s3a.impl
>
>
>
>
>
>
> Regards,
> Anup Tiwari
>
>
> On Thu, Apr 12, 2018 at 7:19 PM, Richard A. Bross < r...@oaktreepeak.com >
> wrote:
>
>
> Based on the exception, it looks more like an AWS credentials issue than a
> Hive issue. Are you running in AWS EMR, on-prem?
>
> In AWS the resource accessing the S3 bucket would have to have an IAM that
> gave permission. If you are running somewhere else whatever AWS login you
> use would have to have the correct permissions in the IAM.
>
>
>
> - Original Message -
> From: "Anup Tiwari" < anupsdtiw...@gmail.com >
> To: user@hive.apache.org
> Sent: Thursday, April 12, 2018 9:11:37 AM
> Subject: Unable to read hive external table data which is linked to s3
> after upgradation from 2.1.1 to 2.3.3
>
>
>
> Hi All,
>
> When i am trying to read s3 linked external table in 2.3.3 ; i am getting
> errors.It was working properly in 2.1.1. please find below details and let
> me know if i am missing something :-
>
>
> Hadoop Version :- 2.8.0
>
> Query :-
>
> select log_date,count(1) as cnt from test.tt1 group by log_date;
>
> Error :-
>
> Vertex failed, vertexName=Map 1, vertexId=vertex_1523502631429_0029_3_00,
> diagnostics=[Vertex vertex_1523502631429_0029_3_00 [Map 1] killed/failed
> due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tt1 initializer failed,
> vertex=vertex_1523502631429_0029_3_00 [Map 1], 
> org.apache.hadoop.fs.s3a.AWSClientIOException:
> doesBucketExist on g24x7.new-analytics: com.amazonaws.AmazonClientException:
> No AWS Credentials provided by BasicAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider
> : com.amazonaws.AmazonClientException: The requested metadata is not
> found at http://16x.xxx.xxx.xx4/latest/meta-data/iam/security-credentials/
> : No AWS Credentials provided by BasicAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider
> : com.amazonaws.AmazonClientException: The requested metadata is not
> found at http://16x.xxx.xxx.xx4/latest/meta-data/iam/security-credentials/
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:128)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(
> S3AFileSystem.java:288)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(
> S3AFileSystem.java:236)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811)
> at 

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

[Richard A. Bross]

I hear you, but given the exception log, it does seem that it can't 
authenticate you.  You can try using the AWS environment variables.  If that 
resolves the issue then you'll have some more to go on.  According to 
Hortonworks here:

https://hortonworks.github.io/hdp-aws/s3-security/

"AWS CLI supports authentication through environment variables. These same 
environment variables will be used by Hadoop if no configuration properties are 
set."


- Original Message -
From: "Anup Tiwari" 
To: user@hive.apache.org
Sent: Thursday, April 12, 2018 10:06:33 AM
Subject: Re: Unable to read hive external table data which is linked to s3 
after upgradation from 2.1.1 to 2.3.3


We are not using EMR. Also we have set below params for accessing s3 bucket in 
hive-site.xml which are same as what we have set in hive 2.1.1. 


* fs.s3a.access.key 
* fs.s3a.secret.key 
* fs.s3a.connection.maximum 
* fs.s3a.impl 






Regards, 
Anup Tiwari 


On Thu, Apr 12, 2018 at 7:19 PM, Richard A. Bross < r...@oaktreepeak.com > 
wrote: 


Based on the exception, it looks more like an AWS credentials issue than a Hive 
issue. Are you running in AWS EMR, on-prem? 

In AWS the resource accessing the S3 bucket would have to have an IAM that gave 
permission. If you are running somewhere else whatever AWS login you use would 
have to have the correct permissions in the IAM. 



- Original Message - 
From: "Anup Tiwari" < anupsdtiw...@gmail.com > 
To: user@hive.apache.org 
Sent: Thursday, April 12, 2018 9:11:37 AM 
Subject: Unable to read hive external table data which is linked to s3 after 
upgradation from 2.1.1 to 2.3.3 



Hi All, 

When i am trying to read s3 linked external table in 2.3.3 ; i am getting 
errors.It was working properly in 2.1.1. please find below details and let me 
know if i am missing something :- 


Hadoop Version :- 2.8.0 

Query :- 

select log_date,count(1) as cnt from test.tt1 group by log_date; 

Error :- 

Vertex failed, vertexName=Map 1, vertexId=vertex_1523502631429_0029_3_00, 
diagnostics=[Vertex vertex_1523502631429_0029_3_00 [Map 1] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tt1 initializer failed, 
vertex=vertex_1523502631429_0029_3_00 [Map 1], 
org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on 
g24x7.new-analytics: com.amazonaws.AmazonClientException: No AWS Credentials 
provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
SharedInstanceProfileCredentialsProvider : com.amazonaws.AmazonClientException: 
The requested metadata is not found at 
http://16x.xxx.xxx.xx4/latest/meta-data/iam/security-credentials/ : No AWS 
Credentials provided by BasicAWSCredentialsProvider 
EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider 
: com.amazonaws.AmazonClientException: The requested metadata is not found at 
http://16x.xxx.xxx.xx4/latest/meta-data/iam/security-credentials/ 
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:128) 
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:288)
 
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:236) 
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) 
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) 
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848) 
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830) 
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) 
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356) 
at 
org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:265)
 
at 
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236) 
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) 
at org.apache.hadoop.hive.ql.io 
.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:442) 
at org.apache.hadoop.hive.ql.io 
.HiveInputFormat.getSplits(HiveInputFormat.java:561) 
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
 
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
 
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:422) 
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
 
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
 
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
 
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
at 

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

[Anup Tiwari]

We are not using EMR. Also we have set below params for accessing s3 bucket
in hive-site.xml which are same as what we have set in hive 2.1.1.


   - fs.s3a.access.key
   - fs.s3a.secret.key
   - fs.s3a.connection.maximum
   - fs.s3a.impl


Regards,
Anup Tiwari

On Thu, Apr 12, 2018 at 7:19 PM, Richard A. Bross 
wrote:

> Based on the exception, it looks more like an AWS credentials issue than a
> Hive issue.  Are you running in AWS EMR, on-prem?
>
> In AWS the resource accessing the S3 bucket would have to have an IAM that
> gave permission.  If you are running somewhere else whatever AWS login you
> use would have to have the correct permissions in the IAM.
>
> - Original Message -
> From: "Anup Tiwari" 
> To: user@hive.apache.org
> Sent: Thursday, April 12, 2018 9:11:37 AM
> Subject: Unable to read hive external table data which is linked to s3
> after upgradation from 2.1.1 to 2.3.3
>
>
>
> Hi All,
>
> When i am trying to read s3 linked external table in 2.3.3 ; i am getting
> errors.It was working properly in 2.1.1. please find below details and let
> me know if i am missing something :-
>
>
> Hadoop Version :- 2.8.0
>
> Query :-
>
> select log_date,count(1) as cnt from test.tt1 group by log_date;
>
> Error :-
>
> Vertex failed, vertexName=Map 1, vertexId=vertex_1523502631429_0029_3_00,
> diagnostics=[Vertex vertex_1523502631429_0029_3_00 [Map 1] killed/failed
> due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tt1 initializer failed,
> vertex=vertex_1523502631429_0029_3_00 [Map 1], 
> org.apache.hadoop.fs.s3a.AWSClientIOException:
> doesBucketExist on g24x7.new-analytics: com.amazonaws.AmazonClientException:
> No AWS Credentials provided by BasicAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider
> : com.amazonaws.AmazonClientException: The requested metadata is not
> found at http://16x.xxx.xxx.xx4/latest/meta-data/iam/security-credentials/
> : No AWS Credentials provided by BasicAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider 
> SharedInstanceProfileCredentialsProvider
> : com.amazonaws.AmazonClientException: The requested metadata is not
> found at http://16x.xxx.xxx.xx4/latest/meta-data/iam/security-credentials/
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:128)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(
> S3AFileSystem.java:288)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(
> S3AFileSystem.java:236)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(
> FileInputFormat.java:265)
> at org.apache.hadoop.mapred.FileInputFormat.listStatus(
> FileInputFormat.java:236)
> at org.apache.hadoop.mapred.FileInputFormat.getSplits(
> FileInputFormat.java:322)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(
> HiveInputFormat.java:442)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(
> HiveInputFormat.java:561)
> at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.
> initialize(HiveSplitGenerator.java:196)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$
> InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$
> InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1807)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$
> InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at org.apache.tez.dag.app.dag.RootInputInitializerManager$
> InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.amazonaws.AmazonClientException: No AWS Credentials
> provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider
> SharedInstanceProfileCredentialsProvider : 
> com.amazonaws.AmazonClientException:
> The requested metadata is not found at http://16x.xxx.xxx.xx4/latest/
> meta-data/iam/security-credentials/
> at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(
> AWSCredentialProviderList.java:151)
> at 

Re: Unable to read hive external table data which is linked to s3 after upgradation from 2.1.1 to 2.3.3

[Richard A. Bross]

Based on the exception, it looks more like an AWS credentials issue than a Hive 
issue.  Are you running in AWS EMR, on-prem?

In AWS the resource accessing the S3 bucket would have to have an IAM that gave 
permission.  If you are running somewhere else whatever AWS login you use would 
have to have the correct permissions in the IAM.

- Original Message -
From: "Anup Tiwari" 
To: user@hive.apache.org
Sent: Thursday, April 12, 2018 9:11:37 AM
Subject: Unable to read hive external table data which is linked to s3 after 
upgradation from 2.1.1 to 2.3.3



Hi All, 

When i am trying to read s3 linked external table in 2.3.3 ; i am getting 
errors.It was working properly in 2.1.1. please find below details and let me 
know if i am missing something :- 


Hadoop Version :- 2.8.0 

Query :- 

select log_date,count(1) as cnt from test.tt1 group by log_date; 

Error :- 

Vertex failed, vertexName=Map 1, vertexId=vertex_1523502631429_0029_3_00, 
diagnostics=[Vertex vertex_1523502631429_0029_3_00 [Map 1] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tt1 initializer failed, 
vertex=vertex_1523502631429_0029_3_00 [Map 1], 
org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on 
g24x7.new-analytics: com.amazonaws.AmazonClientException: No AWS Credentials 
provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
SharedInstanceProfileCredentialsProvider : com.amazonaws.AmazonClientException: 
The requested metadata is not found at 
http://16x.xxx.xxx.xx4/latest/meta-data/iam/security-credentials/ : No AWS 
Credentials provided by BasicAWSCredentialsProvider 
EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider 
: com.amazonaws.AmazonClientException: The requested metadata is not found at 
http://16x.xxx.xxx.xx4/latest/meta-data/iam/security-credentials/ 
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:128) 
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:288)
 
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:236) 
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) 
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) 
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848) 
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830) 
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) 
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356) 
at 
org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:265)
 
at 
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236) 
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) 
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:442)
 
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:561)
 
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
 
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
 
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:422) 
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
 
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
 
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
 
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745) 
Caused by: com.amazonaws.AmazonClientException: No AWS Credentials provided by 
BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
SharedInstanceProfileCredentialsProvider : com.amazonaws.AmazonClientException: 
The requested metadata is not found at 
http://16x.xxx.xxx.xx4/latest/meta-data/iam/security-credentials/ 
at 
org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:151)
 
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3779) 
at 
com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1107) 
at 
com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1070)
 
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:276)
 
... 24 more 
Caused by: com.amazonaws.AmazonClientException: The requested metadata is not 
found at