I think it will be great to introduce RCFile to the community and learn more
feedbacks. I will try to get some test results.

To Jeff
>> Might be worth checking out what's in TFile also and
>> see if you can merge one into the other.
Tfile is quite different with RCFile in that
1) TFile's data layout is a little complicated
2) TFile support InputStream pointint to value to avoid large memory copy or
OOM
3) I think TFile does not support append once it is created?
I took a look at Tfile long time ago, so not sure what I am saying is right
for the current Tfile work.

I think merging one into the other is not an easy work. Hadoop currently has
many file formats, such as SequenceFile, Tfile, Ifile, MapFile, Hfile,
RCFile etc. Instead of merging them, I think a better way may be to put them
into a same collection, for example, put them into a same package.



On 09-6-15 下午4:23, "Zheng Shao" <[email protected]> wrote:

> Yes we did take a look at TFile before we started. The conclusion at the
> time was that TFile was not ready yet, and as a result the RCFile is
> implemented on top of SequenceFile (most of the SequenceFile code got
> reused).
> Once the TFile is ready, we should think about adding Columnar Support to
> TFile just as the same way we do to SequenceFile.
> 
> 
> Another thing is that I think as soon as we get time, we should publish the
> current RCFile work to hadoop user mailing lists to see the feedback from
> there. We should include some introduction of the design as well as
> performance numbers. What do you think Yongqiang?
> 
> Zheng
> 
> 2009/6/14 Jeff Hammerbacher <[email protected]>
> 
>> Hey,
>> 
>> The folks from Yahoo say that their work on TFile (see
>> http://issues.apache.org/jira/browse/HADOOP-3315) is quite similar to the
>> Hive work on RCFile. Might be worth checking out what's in TFile also and
>> see if you can merge one into the other.
>> 
>> Later,
>> Jeff
>> 
>> 2009/6/12 He Yongqiang <[email protected]>
>> 
>>> I think this would be great. And we should avoid duplicate work and
>> should
>>> put together the develop forces instead of isolating them.  I completely
>>> support this :).
>>> 
>>> 
>>> On 09-6-13 上午5:42, "Joydeep Sen Sarma" <[email protected]> wrote:
>>> 
>>>> Is this columnar format reusable by other folks?
>>>> 
>>>> Is it possible to demonstrate the use (in case of simple delimited text
>>> data)
>>>> by writing a configurable inputformat on top of this that can project
>> out
>>> the
>>>> configured columns?
>>>> 
>>>> The reason I am asking is that there is some parallel work going on in
>>>> yahoo/pig on columnar format and no one's aware of this format -
>> whereas
>>> it
>>>> should be usable by other people (and not just Hive).
>>>> 
>>>> Comments?
>>>> 
>>>> 
>>> 
>>> 
>>> 
>> 
> 
> 


Reply via email to