see if you can --target-dir s3n://****:*********@iakbar.emr/tmp see if you can try the target dir in /tmp to see if it is a permissions problem. usually everybody can create files in /tmp
On Tue, Feb 4, 2014 at 4:50 PM, Imran Akbar <[email protected]> wrote: > That doesn't seem to be the issue, because I just manually created a > folder called "_logs" in S3 and it worked. > Any ideas why the sqoop import would work, but would fail when trying to > create a "_logs" folder after its done? > > > On Tue, Feb 4, 2014 at 1:44 PM, Imran Akbar <[email protected]>wrote: > >> Hey Venkat, >> Sorry, I meant to say I made that change in core-site.xml, not >> site-core.xml. >> >> I'm trying to do a hive import from MySQL to S3, but I think the error is >> popping up because sqoop is trying to create a "_logs" directory, but >> according to S3's naming conventions you can't start the name of a bucket >> with an underscore: >> >> "Bucket names can contain lowercase letters, numbers, and dashes. Each >> label must start and end with a lowercase letter or a number." >> http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html >> >> this is the error i'm getting (the iakbar.emr/dump2/ location on S3 >> contains files, so I know sqoop works up to this point): >> "This file system object (hdfs://10.202.163.18:9000) does not support >> access to the request path 's3n://****:****@iakbar.emr/dump2/_logs'" >> >> thanks, >> imran >> >> >> On Tue, Feb 4, 2014 at 12:45 PM, Venkat Ranganathan < >> [email protected]> wrote: >> >>> I think you are trying to do a hive import from the S3 location. I >>> think it may not be supported - As Jarcec said you may want to change the >>> core-site to point to S3 on your Hadoop cluster. But I have not tested >>> this so not sure if that will work >>> >>> Venkat >>> >>> >>> On Tue, Feb 4, 2014 at 12:04 PM, Imran Akbar <[email protected]>wrote: >>> >>>> I think it may have worked, but I am getting an error. >>>> >>>> I added this line to site-core.xml: >>>> <property><name>fs.defaultFS</name><value>s3n</value></property> >>>> >>>> and I see the following contents in my S3 directory after running sqoop: >>>> _SUCCESS >>>> part-m-00000 >>>> part-m-00001 >>>> part-m-00002 >>>> part-m-00003 >>>> part-m-00004 >>>> part-m-00005 >>>> >>>> I'm running sqoop version 1.4.4. >>>> >>>> But I still get this error after running sqoop: >>>> http://pastebin.com/5AYCsd78 >>>> >>>> any ideas? >>>> thanks for the help so far >>>> >>>> imran >>>> >>>> >>>> On Tue, Feb 4, 2014 at 11:24 AM, Venkat Ranganathan < >>>> [email protected]> wrote: >>>> >>>>> Which version of sqoop are you using. Sqoop 1.4.4 addressed use of >>>>> other filesystems with the fix mentioned in SQOOP-1033 >>>>> >>>>> Thanks >>>>> Venkat >>>>> >>>>> >>>>> On Tue, Feb 4, 2014 at 8:14 AM, Jarek Jarcec Cecho >>>>> <[email protected]>wrote: >>>>> >>>>>> Yes Imran, >>>>>> I would try to define the fs.defaultFS for the S3 in core-site.xml >>>>>> and see if it will help Sqoop to accept the S3 path. >>>>>> >>>>>> Jarcec >>>>>> >>>>>> On Tue, Feb 04, 2014 at 08:08:17AM -0800, Imran Akbar wrote: >>>>>> > thanks Jarek, >>>>>> > How would I do that? Do I need to set fs.defaultFS in >>>>>> core-site.xml, or >>>>>> > is it something else? Is there a document somewhere which >>>>>> describes this? >>>>>> > >>>>>> > yours, >>>>>> > imran >>>>>> > >>>>>> > >>>>>> > On Mon, Feb 3, 2014 at 9:31 PM, Jarek Jarcec Cecho < >>>>>> [email protected]>wrote: >>>>>> > >>>>>> > > Would you mind trying to set the S3 filesystem as the default one >>>>>> for >>>>>> > > Sqoop? >>>>>> > > >>>>>> > > Jarcec >>>>>> > > >>>>>> > > On Mon, Feb 03, 2014 at 10:25:50AM -0800, Imran Akbar wrote: >>>>>> > > > Hi, >>>>>> > > > I've been able to sqoop from MySQL into HDFS, but I was >>>>>> wondering if >>>>>> > > it >>>>>> > > > was possible to send the data directly to S3 instead. I've >>>>>> read some >>>>>> > > posts >>>>>> > > > on this forum and others that indicate that it's not possible >>>>>> to do this >>>>>> > > - >>>>>> > > > could someone confirm? >>>>>> > > > >>>>>> > > > I tried to get it to work by setting: >>>>>> > > > --warehouse-dir s3n://MYS3APIKEY:MYS3SECRETKEY@bucketname >>>>>> /folder/ >>>>>> > > > or >>>>>> > > > --target-dir s3n://MYS3APIKEY:MYS3SECRETKEY@bucketname/folder/ >>>>>> > > > >>>>>> > > > options but I get the error: >>>>>> > > > ERROR tool.ImportTool: Imported Failed: This file system object >>>>>> (hdfs:// >>>>>> > > > 10.168.22.133:9000) does not support access to the request path >>>>>> > > > 's3n://****:****@iakbar.emr/new-hive-output/_logs' You possibly >>>>>> called >>>>>> > > > FileSystem.get(conf) when you should have called >>>>>> FileSystem.get(uri, >>>>>> > > conf) >>>>>> > > > to obtain a file system supporting your path >>>>>> > > > >>>>>> > > > If it's not possible to do this, should I just import to HDFS >>>>>> and then >>>>>> > > > output to S3? Is there an easy way to do this without having >>>>>> to specify >>>>>> > > > the schema of the whole table again? >>>>>> > > > >>>>>> > > > thanks, >>>>>> > > > imran >>>>>> > > >>>>>> >>>>> >>>>> >>>>> CONFIDENTIALITY NOTICE >>>>> NOTICE: This message is intended for the use of the individual or >>>>> entity to which it is addressed and may contain information that is >>>>> confidential, privileged and exempt from disclosure under applicable law. >>>>> If the reader of this message is not the intended recipient, you are >>>>> hereby >>>>> notified that any printing, copying, dissemination, distribution, >>>>> disclosure or forwarding of this communication is strictly prohibited. If >>>>> you have received this communication in error, please contact the sender >>>>> immediately and delete it from your system. Thank You. >>>> >>>> >>>> >>> >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. >>> >> >> >
