+1 for Sab's thoughtful answer... Yasemin: As Gourav said, using IAM roles is considered best practice and generally will give you fewer headaches in the end... but you may have a reason for doing it the way you are, and certainly the way you posted should be supported and not cause the error you described.
-- Chris Miller On Tue, Mar 15, 2016 at 11:22 PM, Sabarish Sasidharan < sabarish.sasidha...@manthan.com> wrote: > There are many solutions to a problem. > > Also understand that sometimes your situation might be such. For ex what > if you are accessing S3 from your Spark job running in your continuous > integration server sitting in your data center or may be a box under your > desk. And sometimes you are just trying something. > > Also understand that sometimes you want answers to solve your problem at > hand without redirecting you to something else. Understand what you > suggested is an appropriate way of doing it, which I myself have proposed > before, but that doesn't solve the OP's problem at hand. > > Regards > Sab > On 15-Mar-2016 8:27 pm, "Gourav Sengupta" <gourav.sengu...@gmail.com> > wrote: > >> Oh!!! What the hell!!!! >> >> Please never use the URI >> >> *s3n://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY.*That is a major cause of >> pain, security issues, code maintenance issues and ofcourse something that >> Amazon strongly suggests that we do not use. Please use roles and you will >> not have to worry about security. >> >> Regards, >> Gourav Sengupta >> >> On Tue, Mar 15, 2016 at 2:38 PM, Sabarish Sasidharan < >> sabarish....@gmail.com> wrote: >> >>> You have a slash before the bucket name. It should be @<bucket name>. >>> >>> Regards >>> Sab >>> On 15-Mar-2016 4:03 pm, "Yasemin Kaya" <godo...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I am using Spark 1.6.0 standalone and I want to read a txt file from S3 >>>> bucket named yasemindeneme and my file name is deneme.txt. But I am getting >>>> this error. Here is the simple code >>>> <https://gist.github.com/anonymous/6d174f8587f0f3fd2334> >>>> Exception in thread "main" java.lang.IllegalArgumentException: Invalid >>>> hostname in URI s3n://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY@ >>>> /yasemindeneme/deneme.txt >>>> at >>>> org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:45) >>>> at >>>> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:55) >>>> >>>> >>>> I try 2 options >>>> *sc.hadoopConfiguration() *and >>>> *sc.textFile("s3n://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY@/yasemindeneme/deneme.txt/");* >>>> >>>> Also I did export AWS_ACCESS_KEY_ID= ..... >>>> export AWS_SECRET_ACCESS_KEY= >>>> But there is no change about error. >>>> >>>> Could you please help me about this issue? >>>> >>>> >>>> -- >>>> hiç ender hiç >>>> >>> >>
