I think it may have worked, but I am getting an error. I added this line to site-core.xml: <property><name>fs.defaultFS</name><value>s3n</value></property>
and I see the following contents in my S3 directory after running sqoop: _SUCCESS part-m-00000 part-m-00001 part-m-00002 part-m-00003 part-m-00004 part-m-00005 I'm running sqoop version 1.4.4. But I still get this error after running sqoop: http://pastebin.com/5AYCsd78 any ideas? thanks for the help so far imran On Tue, Feb 4, 2014 at 11:24 AM, Venkat Ranganathan < [email protected]> wrote: > Which version of sqoop are you using. Sqoop 1.4.4 addressed use of other > filesystems with the fix mentioned in SQOOP-1033 > > Thanks > Venkat > > > On Tue, Feb 4, 2014 at 8:14 AM, Jarek Jarcec Cecho <[email protected]>wrote: > >> Yes Imran, >> I would try to define the fs.defaultFS for the S3 in core-site.xml and >> see if it will help Sqoop to accept the S3 path. >> >> Jarcec >> >> On Tue, Feb 04, 2014 at 08:08:17AM -0800, Imran Akbar wrote: >> > thanks Jarek, >> > How would I do that? Do I need to set fs.defaultFS in >> core-site.xml, or >> > is it something else? Is there a document somewhere which describes >> this? >> > >> > yours, >> > imran >> > >> > >> > On Mon, Feb 3, 2014 at 9:31 PM, Jarek Jarcec Cecho <[email protected] >> >wrote: >> > >> > > Would you mind trying to set the S3 filesystem as the default one for >> > > Sqoop? >> > > >> > > Jarcec >> > > >> > > On Mon, Feb 03, 2014 at 10:25:50AM -0800, Imran Akbar wrote: >> > > > Hi, >> > > > I've been able to sqoop from MySQL into HDFS, but I was >> wondering if >> > > it >> > > > was possible to send the data directly to S3 instead. I've read >> some >> > > posts >> > > > on this forum and others that indicate that it's not possible to do >> this >> > > - >> > > > could someone confirm? >> > > > >> > > > I tried to get it to work by setting: >> > > > --warehouse-dir s3n://MYS3APIKEY:MYS3SECRETKEY@bucketname/folder/ >> > > > or >> > > > --target-dir s3n://MYS3APIKEY:MYS3SECRETKEY@bucketname/folder/ >> > > > >> > > > options but I get the error: >> > > > ERROR tool.ImportTool: Imported Failed: This file system object >> (hdfs:// >> > > > 10.168.22.133:9000) does not support access to the request path >> > > > 's3n://****:****@iakbar.emr/new-hive-output/_logs' You possibly >> called >> > > > FileSystem.get(conf) when you should have called FileSystem.get(uri, >> > > conf) >> > > > to obtain a file system supporting your path >> > > > >> > > > If it's not possible to do this, should I just import to HDFS and >> then >> > > > output to S3? Is there an easy way to do this without having to >> specify >> > > > the schema of the whole table again? >> > > > >> > > > thanks, >> > > > imran >> > > >> > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
