I did try it with the command below, followed by the error. I am not sure how
to specify that it should go to our s3 cluster, but I tried it after the @ sign
and then the bucket. It goes to Amazon, which seems to be by default. I know
it isn't an issue with Riak and that technically it should work, but I was
looking for someone that might have tried this and gotten it to work. It seems
like a configuration in Hadoop more than anything. So I am asking there also.
hadoop distcp -update /user/test/client/part-m-0
s3a://access_key:secret_...@s3.test.com/test-bucket
16/08/31 09:37:38 INFO s3a.S3AFileSystem: Caught an AmazonServiceException,
which means your request made it to Amazon S3, but was rejected with an error
response for some reason.
16/08/31 09:37:38 INFO s3a.S3AFileSystem: Error Message: Status Code: 403, AWS
Service: Amazon S3, AWS Request ID: 94AE01A87CE82C24, AWS Error Code: null, AWS
Error Message: Forbidden
16/08/31 09:37:38 INFO s3a.S3AFileSystem: HTTP Status Code: 403
16/08/31 09:37:38 INFO s3a.S3AFileSystem: AWS Error Code: null
16/08/31 09:37:38 INFO s3a.S3AFileSystem: Error Type: Client
16/08/31 09:37:38 INFO s3a.S3AFileSystem: Request ID: 94AE01A87CE82C24
16/08/31 09:37:38 INFO s3a.S3AFileSystem: Class Name:
com.amazonaws.services.s3.model.AmazonS3Exception
16/08/31 09:37:38 ERROR tools.DistCp: Invalid arguments:
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 403, AWS
Service: Amazon S3, AWS Request ID: 94AE01A87CE82C24, AWS Error Code: null, AWS
Error Message: Forbidden, S3 Extended Request ID:
qYOsZBJ5O3cFaKpbdKEpbou0Zx5hSgsHJHum/kPvjZu6CAvJ2jqVoUVUUXpquEtw
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:976)
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:956)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:892)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:217)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:116)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:430)
Invalid arguments: Status Code: 403, AWS Service: Amazon S3, AWS Request ID:
94AE01A87CE82C24, AWS Error Code: null, AWS Error Message: Forbidden
-Original Message-
From: Luke Bakken [mailto:lbak...@basho.com]
Sent: Wednesday, August 31, 2016 1:18 PM
To: Valenti, Anthony
Cc: riak-users@lists.basho.com
Subject: Re: Using Riak CS with Hadoop
Riak CS provides an S3 capable API, so theoretically it could work.
Have you tried? If so and you're having issues, follow up here.
--
Luke Bakken
Engineer
lbak...@basho.com
On Wed, Aug 31, 2016 at 7:38 AM, Valenti, Anthony
wrote:
> Has anyone setup Hadoop to be able use Raik CS as an S3
> source/destination instead of or in addition to Amazon S3? Hadoop
> assumes that it should go to Amazon S3 by default. Specifically, I am
> trying to use Hadoop distcp to copy files to Riak CS.
Inmar Confidentiality Note: This e-mail and any attachments are confidential
and intended to be viewed and used solely by the intended recipient. If you
are not the intended recipient, be aware that any disclosure, dissemination,
distribution, copying or use of this e-mail or any attachment is prohibited.
If you received this e-mail in error, please notify us immediately by returning
it to the sender and delete this copy and all attachments from your system and
destroy any printed copies. Thank you for your cooperation.
Notice of Protected Rights: The removal of any copyright, trademark, or
proprietary legend contained in this e-mail or any attachment is prohibited
without the express, written permission of Inmar, Inc. Furthermore, the
intended recipient must maintain all copyright notices, trademarks, and
proprietary legends within this e-mail and any attachments in their original
form and location if the e-mail or any attachments are reproduced, printed or
distributed.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
