Re: Streaming data into Hadoop

[Ryan LeCompte]

Even better! I'll try this out tomorrow.

Thanks,
Ryan




On Dec 9, 2008, at 10:36 PM, "Aaron Kimball" <[EMAIL PROTECTED]> wrote:

Note also that "cat foo | bin/hadoop fs -put - some/hdfs/path" will  
use

stdin.
- Aaron

On Mon, Dec 8, 2008 at 5:56 PM, Ryan LeCompte <[EMAIL PROTECTED]>  
wrote:



Just what I need -- thanks!

On Mon, Dec 8, 2008 at 7:31 PM, Alex Loddengaard <[EMAIL PROTECTED]>
wrote:

This should answer your questions:



Alex

On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte <[EMAIL PROTECTED]>

wrote:



Hello all,

I normally upload files into hadoop via bin/hadoop fs -put file  
dest.


However, is there a way to somehow stream data into Hadoop?

For example, I'd love to do something like this:

zcat xxx >> HADOOP_HDFS_DESTINATION

This would save me a ton of time since I don't have to first unpack
the .tgz file and upload the raw file into HDFS.

Is this possible with Hadoop 0.19?

Thanks,
Ryan

Re: Streaming data into Hadoop

[Aaron Kimball]

Note also that "cat foo | bin/hadoop fs -put - some/hdfs/path" will use
stdin.
- Aaron

On Mon, Dec 8, 2008 at 5:56 PM, Ryan LeCompte <[EMAIL PROTECTED]> wrote:

> Just what I need -- thanks!
>
> On Mon, Dec 8, 2008 at 7:31 PM, Alex Loddengaard <[EMAIL PROTECTED]>
> wrote:
> > This should answer your questions:
> >
> > 
> >
> > Alex
> >
> > On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hello all,
> >>
> >> I normally upload files into hadoop via bin/hadoop fs -put file dest.
> >>
> >> However, is there a way to somehow stream data into Hadoop?
> >>
> >> For example, I'd love to do something like this:
> >>
> >> zcat xxx >> HADOOP_HDFS_DESTINATION
> >>
> >> This would save me a ton of time since I don't have to first unpack
> >> the .tgz file and upload the raw file into HDFS.
> >>
> >> Is this possible with Hadoop 0.19?
> >>
> >> Thanks,
> >> Ryan
> >>
> >
>

Re: Streaming data into Hadoop

[Ryan LeCompte]

Just what I need -- thanks!

On Mon, Dec 8, 2008 at 7:31 PM, Alex Loddengaard <[EMAIL PROTECTED]> wrote:
> This should answer your questions:
>
> 
>
> Alex
>
> On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte <[EMAIL PROTECTED]> wrote:
>
>> Hello all,
>>
>> I normally upload files into hadoop via bin/hadoop fs -put file dest.
>>
>> However, is there a way to somehow stream data into Hadoop?
>>
>> For example, I'd love to do something like this:
>>
>> zcat xxx >> HADOOP_HDFS_DESTINATION
>>
>> This would save me a ton of time since I don't have to first unpack
>> the .tgz file and upload the raw file into HDFS.
>>
>> Is this possible with Hadoop 0.19?
>>
>> Thanks,
>> Ryan
>>
>

Re: Streaming data into Hadoop

[Alex Loddengaard]

This should answer your questions:



Alex

On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte <[EMAIL PROTECTED]> wrote:

> Hello all,
>
> I normally upload files into hadoop via bin/hadoop fs -put file dest.
>
> However, is there a way to somehow stream data into Hadoop?
>
> For example, I'd love to do something like this:
>
> zcat xxx >> HADOOP_HDFS_DESTINATION
>
> This would save me a ton of time since I don't have to first unpack
> the .tgz file and upload the raw file into HDFS.
>
> Is this possible with Hadoop 0.19?
>
> Thanks,
> Ryan
>

Streaming data into Hadoop

[Ryan LeCompte]

Hello all,

I normally upload files into hadoop via bin/hadoop fs -put file dest.

However, is there a way to somehow stream data into Hadoop?

For example, I'd love to do something like this:

zcat xxx >> HADOOP_HDFS_DESTINATION

This would save me a ton of time since I don't have to first unpack
the .tgz file and upload the raw file into HDFS.

Is this possible with Hadoop 0.19?

Thanks,
Ryan