Hi ,
I used spark-ec2 script to create ec2 cluster.
Now I am trying copy data from s3 into hdfs.
I am doing this
*root@ip-172-31-21-160 ephemeral-hdfs]$ bin/hadoop distcp
s3://xxx/home/mydata/small.sam
hdfs://ec2-52-11-148-31.us-west-2.compute.amazonaws.com:9010/data1
It works pretty fine for me with the script comes with 1.2.0 release.
Here's a few things which you can try:
- Add your s3 credentials to the core-site.xml
property namefs.s3.awsAccessKeyId/name
valueID/value/propertyproperty
namefs.s3.awsSecretAccessKey/name
valueSECRET/value/property
- Do a
