Hi Team,
I want to setup hive4 standalone to store table on S3 and Adls gen2 as a
storage .
Could you please help me as with proper steps and configurations required for
this.
Because we are facing multiple issue on this please help me here ASPA.
What we tried.
I am trying to configure AWS S3 configuration with the Hadoop and Hive setup.
But while trying so we are seeing following exception :
hadoop fs -ls s3a://somesh.qa.bucket/ -:
Fatal internal error java.lang.RuntimeException:
java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem
not found
To resolve this I have added hadoop-aws-3.3.6.jar and
aws-java-sdk-bundle-1.12.770.jar in Hadoop classpath.
i.e is under : /usr/local/hadoop/share/hadoop/common/lib
And S3 related configurations in the core-site.xml file: under
/usr/local/hadoop/etc/hadoop directory.
fs.default.name s3a://somesh.qa.bucket fs.s3a.impl
org.apache.hadoop.fs.s3a.S3AFileSystem fs.s3a.endpoint
s3.us-west-2.amazonaws.com fs.s3a.access.key {Access _Key_Value}
fs.s3a.secret.key {Secret_Key_Value} fs.s3a.path.style.access false
Now when we try hadoop fs -ls s3a://somesh.qa.bucket/
We are observing following exception :
2024-08-22 13:50:11,294 INFO impl.MetricsConfig: Loaded properties from
hadoop-metrics2.properties
2024-08-22 13:50:11,376 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
period at 10 second(s).
2024-08-22 13:50:11,376 INFO impl.MetricsSystemImpl: s3a-file-system metrics
system started
2024-08-22 13:50:11,434 WARN util.VersionInfoUtils: The AWS SDK for Java 1.x
entered maintenance mode starting July 31, 2024 and will reach end of support
on December 31, 2025. For more information, see
https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-1-x-is-in-maintenance-mode-effective-july-31-2024/
You can print where on the file system the AWS SDK for Java 1.x core runtime is
located by setting the AWS_JAVA_V1_PRINT_LOCATION environment variable or
aws.java.v1.printLocation system property to 'true'.
This message can be disabled by setting the
AWS_JAVA_V1_DISABLE_DEPRECATION_ANNOUNCEMENT environment variable or
aws.java.v1.disableDeprecationAnnouncement system property to 'true'.
The AWS SDK for Java 1.x is being used here:
at java.lang.Thread.getStackTrace(Thread.java:1564)
at
com.amazonaws.util.VersionInfoUtils.printDeprecationAnnouncement(VersionInfoUtils.java:81)
at com.amazonaws.util.VersionInfoUtils.(VersionInfoUtils.java:59)
at com.amazonaws.internal.EC2ResourceFetcher.(EC2ResourceFetcher.java:44)
at
com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.(InstanceMetadataServiceCredentialsFetcher.java:38)
at
com.amazonaws.auth.InstanceProfileCredentialsProvider.(InstanceProfileCredentialsProvider.java:111)
at
com.amazonaws.auth.InstanceProfileCredentialsProvider.(InstanceProfileCredentialsProvider.java:91)
at
com.amazonaws.auth.InstanceProfileCredentialsProvider.(InstanceProfileCredentialsProvider.java:75)
at
com.amazonaws.auth.InstanceProfileCredentialsProvider.(InstanceProfileCredentialsProvider.java:58)
at
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.initializeProvider(EC2ContainerCredentialsProviderWrapper.java:66)
at
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.(EC2ContainerCredentialsProviderWrapper.java:55)
at
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider.(IAMInstanceCredentialsProvider.java:53)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:727)
at org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList(S3AUtils.java:659)
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:585)
at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:959)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:586)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3712)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3663)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:347)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:264)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:247)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:105)
at org.apache.hadoop.fs.shell.Command.run(Command.java:191)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:97)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
ls: s3a://infa.qa.bucket/:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
com.amazonaws.SdkClientException: Unable to load AWS credentials from
environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY
(or AWS_SECRET_ACCESS_KEY))
2024-08-22 13:50:14,248 INFO impl.MetricsSystemImpl: Stopping s3a-file-system
metrics system...
2024-08-22 13:50:14,248 INFO impl.MetricsSystemImpl: s3a-file-system metrics
system stopped.
2024-08-22 13:50:14,248 INFO impl.MetricsSystemImpl: s3a-file-system metrics
syst
Could you please help us to resolve this issue as soon as possible
Thanks,
Somesh
