It should be re usable. The work in yahoo/pig is on different line though as Alan explained to me. They are actually storing the columns in different files (I think they are using the TFile stuff) rather than having a columnar storage within a block...
Ashish ________________________________________ From: Joydeep Sen Sarma [[email protected]] Sent: Friday, June 12, 2009 2:42 PM To: [email protected] Subject: can RCFile* be also exported as a hadoop contrib project? Is this columnar format reusable by other folks? Is it possible to demonstrate the use (in case of simple delimited text data) by writing a configurable inputformat on top of this that can project out the configured columns? The reason I am asking is that there is some parallel work going on in yahoo/pig on columnar format and no one's aware of this format - whereas it should be usable by other people (and not just Hive). Comments?
