Hey Venkat,
Sorry, I meant to say I made that change in core-site.xml, not
site-core.xml.
I'm trying to do a hive import from MySQL to S3, but I think the error is
popping up because sqoop is trying to create a "_logs" directory, but
according to S3's naming conventions you can't start the name of a bucket
with an underscore:
"Bucket names can contain lowercase letters, numbers, and dashes. Each
label must start and end with a lowercase letter or a number."
http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
this is the error i'm getting (the iakbar.emr/dump2/ location on S3
contains files, so I know sqoop works up to this point):
"This file system object (hdfs://10.202.163.18:9000) does not support
access to the request path 's3n://****:****@iakbar.emr/dump2/_logs'"
thanks,
imran
On Tue, Feb 4, 2014 at 12:45 PM, Venkat Ranganathan <
[email protected]> wrote:
> I think you are trying to do a hive import from the S3 location. I
> think it may not be supported - As Jarcec said you may want to change the
> core-site to point to S3 on your Hadoop cluster. But I have not tested
> this so not sure if that will work
>
> Venkat
>
>
> On Tue, Feb 4, 2014 at 12:04 PM, Imran Akbar <[email protected]>wrote:
>
>> I think it may have worked, but I am getting an error.
>>
>> I added this line to site-core.xml:
>> <property><name>fs.defaultFS</name><value>s3n</value></property>
>>
>> and I see the following contents in my S3 directory after running sqoop:
>> _SUCCESS
>> part-m-00000
>> part-m-00001
>> part-m-00002
>> part-m-00003
>> part-m-00004
>> part-m-00005
>>
>> I'm running sqoop version 1.4.4.
>>
>> But I still get this error after running sqoop:
>> http://pastebin.com/5AYCsd78
>>
>> any ideas?
>> thanks for the help so far
>>
>> imran
>>
>>
>> On Tue, Feb 4, 2014 at 11:24 AM, Venkat Ranganathan <
>> [email protected]> wrote:
>>
>>> Which version of sqoop are you using. Sqoop 1.4.4 addressed use of
>>> other filesystems with the fix mentioned in SQOOP-1033
>>>
>>> Thanks
>>> Venkat
>>>
>>>
>>> On Tue, Feb 4, 2014 at 8:14 AM, Jarek Jarcec Cecho <[email protected]>wrote:
>>>
>>>> Yes Imran,
>>>> I would try to define the fs.defaultFS for the S3 in core-site.xml and
>>>> see if it will help Sqoop to accept the S3 path.
>>>>
>>>> Jarcec
>>>>
>>>> On Tue, Feb 04, 2014 at 08:08:17AM -0800, Imran Akbar wrote:
>>>> > thanks Jarek,
>>>> > How would I do that? Do I need to set fs.defaultFS in
>>>> core-site.xml, or
>>>> > is it something else? Is there a document somewhere which describes
>>>> this?
>>>> >
>>>> > yours,
>>>> > imran
>>>> >
>>>> >
>>>> > On Mon, Feb 3, 2014 at 9:31 PM, Jarek Jarcec Cecho <[email protected]
>>>> >wrote:
>>>> >
>>>> > > Would you mind trying to set the S3 filesystem as the default one
>>>> for
>>>> > > Sqoop?
>>>> > >
>>>> > > Jarcec
>>>> > >
>>>> > > On Mon, Feb 03, 2014 at 10:25:50AM -0800, Imran Akbar wrote:
>>>> > > > Hi,
>>>> > > > I've been able to sqoop from MySQL into HDFS, but I was
>>>> wondering if
>>>> > > it
>>>> > > > was possible to send the data directly to S3 instead. I've read
>>>> some
>>>> > > posts
>>>> > > > on this forum and others that indicate that it's not possible to
>>>> do this
>>>> > > -
>>>> > > > could someone confirm?
>>>> > > >
>>>> > > > I tried to get it to work by setting:
>>>> > > > --warehouse-dir s3n://MYS3APIKEY:MYS3SECRETKEY@bucketname/folder/
>>>> > > > or
>>>> > > > --target-dir s3n://MYS3APIKEY:MYS3SECRETKEY@bucketname/folder/
>>>> > > >
>>>> > > > options but I get the error:
>>>> > > > ERROR tool.ImportTool: Imported Failed: This file system object
>>>> (hdfs://
>>>> > > > 10.168.22.133:9000) does not support access to the request path
>>>> > > > 's3n://****:****@iakbar.emr/new-hive-output/_logs' You possibly
>>>> called
>>>> > > > FileSystem.get(conf) when you should have called
>>>> FileSystem.get(uri,
>>>> > > conf)
>>>> > > > to obtain a file system supporting your path
>>>> > > >
>>>> > > > If it's not possible to do this, should I just import to HDFS and
>>>> then
>>>> > > > output to S3? Is there an easy way to do this without having to
>>>> specify
>>>> > > > the schema of the whole table again?
>>>> > > >
>>>> > > > thanks,
>>>> > > > imran
>>>> > >
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
