Re: reading file from S3

+1 for Sab's thoughtful answer... Yasemin: As Gourav said, using IAM roles is considered best practice and generally will give you fewer headaches in the end... but you may have a reason for doing it the way you are, and certainly the way you posted should be supported and not cause the error you

Re: reading file from S3

Hi, Thanx a lot all, I understand my problem comes from *hadoop version* and I move the spark 1.6.0 *hadoop 2.4 *version and there is no problem. Best, yasemin 2016-03-15 17:31 GMT+02:00 Gourav Sengupta : > Once again, please use roles, there is no way that you have to specify the > access keys

Re: reading file from S3

Once again, please use roles, there is no way that you have to specify the access keys in the URI under any situation. Please read Amazon documentation and they will say the same. The only situation when you use the access keys in URI is when you have not read the Amazon documentation :) Regards,

Re: reading file from S3

There are many solutions to a problem. Also understand that sometimes your situation might be such. For ex what if you are accessing S3 from your Spark job running in your continuous integration server sitting in your data center or may be a box under your desk. And sometimes you are just trying s

Re: reading file from S3

Oh!!! What the hell Please never use the URI *s3n://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY.*That is a major cause of pain, security issues, code maintenance issues and ofcourse something that Amazon strongly suggests that we do not use. Please use roles and you will not have to worry about s

Re: reading file from S3

You have a slash before the bucket name. It should be @. Regards Sab On 15-Mar-2016 4:03 pm, "Yasemin Kaya" wrote: > Hi, > > I am using Spark 1.6.0 standalone and I want to read a txt file from S3 > bucket named yasemindeneme and my file name is deneme.txt. But I am getting > this error. Here is

Re: reading file from S3

Hi, Try starting your clusters with roles, and you will not have to configure, hard code anything at all. Let me know in case you need any help with this. Regards, Gourav Sengupta On Tue, Mar 15, 2016 at 11:32 AM, Yasemin Kaya wrote: > Hi Safak, > > I changed the Keys but there is no change.

Re: reading file from S3

Hi Safak, I changed the Keys but there is no change. Best, yasemin 2016-03-15 12:46 GMT+02:00 Şafak Serdar Kapçı : > Hello Yasemin, > Maybe your key id or access key has special chars like backslash or > something. You need to change it. > Best Regards, > Safak. > > 2016-03-15 12:33 GMT+02:00

Re: reading file from S3

Hello Yasemin, Maybe your key id or access key has special chars like backslash or something. You need to change it. Best Regards, Safak. 2016-03-15 12:33 GMT+02:00 Yasemin Kaya : > Hi, > > I am using Spark 1.6.0 standalone and I want to read a txt file from S3 > bucket named yasemindeneme and my

reading file from S3

Hi, I am using Spark 1.6.0 standalone and I want to read a txt file from S3 bucket named yasemindeneme and my file name is deneme.txt. But I am getting this error. Here is the simple code Exception in thread "main" java.lang.IllegalArgumentE

Re: Reading file from S3, facing java.lang.NoClassDefFoundError: org/jets3t/service/ServiceException

Thanks Akhil, it solved the problem. best /Shahab On Fri, Jun 12, 2015 at 8:50 PM, Akhil Das wrote: > Looks like your spark is not able to pick up the HADOOP_CONF. To fix this, > you can actually add jets3t-0.9.0.jar to the classpath > (sc.addJar(/path/to/jets3t-0.9.0.jar). > > Thanks > Best Re

Re: Reading file from S3, facing java.lang.NoClassDefFoundError: org/jets3t/service/ServiceException

Looks like your spark is not able to pick up the HADOOP_CONF. To fix this, you can actually add jets3t-0.9.0.jar to the classpath (sc.addJar(/path/to/jets3t-0.9.0.jar). Thanks Best Regards On Thu, Jun 11, 2015 at 6:44 PM, shahab wrote: > Hi, > > I tried to read a csv file from amazon s3, but I

Reading file from S3, facing java.lang.NoClassDefFoundError: org/jets3t/service/ServiceException

Hi, I tried to read a csv file from amazon s3, but I get the following exception which I have no clue how to solve this. I tried both spark 1.3.1 and 1.2.1, but no success. Any idea how to solve this is appreciated. best, /Shahab the code: val hadoopConf=sc.hadoopConfiguration; hadoopConf.s