Does anyone know if I can save a RDD as a text file to a pre-created directory
in S3 bucket?
I have a directory created in S3 bucket: //nexgen-software/dev
When I tried to save a RDD as text file in this directory:
rdd.saveAsTextFile("s3n://nexgen-software/dev/output");
I got following exception at runtime:
Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception:
org.jets3t.service.S3ServiceException: S3 HEAD request failed for '/dev' -
ResponseCode=403, ResponseMessage=Forbidden
I have verified /dev has write permission. However, if I grant the bucket
//nexgen-software write permission, I don't get exception. But the output is
not created under dev. Rather, a different /dev/output directory is created
directory in the bucket (//nexgen-software). Is this how saveAsTextFile
behalves in S3? Is there anyway I can have output created under a pre-defied
directory.
Thanks in advance.
