No, no reason for a single file - just a little simpler to think about. By the way, can multiple MapReduce workers read the same SequenceFile simultaneously?
On Mon, Feb 2, 2009 at 9:42 AM, Tom White <[email protected]> wrote: > Is there any reason why it has to be a single SequenceFile? You could > write a local program to write several block compressed SequenceFiles > in parallel (to HDFS), each containing a portion of the files on your > PC. > > Tom > > On Mon, Feb 2, 2009 at 3:24 PM, Mark Kerzner <[email protected]> > wrote: > > Truly, I do not see any advantage to doing this, as opposed to writing > > (Java) code which will copy files to HDFS, because then tarring becomes > my > > bottleneck. Unless I write code measure the file sizes and prepare > pointers > > for multiple tarring tasks. It becomes pretty complex though, and I > thought > > of something simple. I might as well accept that copying one hard drive > to > > HDFS is not going to be parallelized. > > Mark > > > > On Sun, Feb 1, 2009 at 11:44 PM, Philip (flip) Kromer > > <[email protected]>wrote: > > > >> Could you tar.bz2 them up (setting up the tar so that it made a few > dozen > >> files), toss them onto the HDFS, and use > >> http://stuartsierra.com/2008/04/24/a-million-little-files > >> to go into SequenceFile? > >> > >> This lets you preserve the originals and do the sequence file conversion > >> across the cluster. It's only really helpful, of course, if you also > want > >> to > >> prepare a .tar.bz2 so you can clear out the sprawl > >> > >> flip > >> > >> On Sun, Feb 1, 2009 at 11:22 PM, Mark Kerzner <[email protected]> > >> wrote: > >> > >> > Hi, > >> > > >> > I am writing an application to copy all files from a regular PC to a > >> > SequenceFile. I can surely do this by simply recursing all directories > on > >> > my > >> > PC, but I wonder if there is any way to parallelize this, a MapReduce > >> task > >> > even. Tom White's books seems to imply that it will have to be a > custom > >> > application. > >> > > >> > Thank you, > >> > Mark > >> > > >> > >> > >> > >> -- > >> http://www.infochimps.org > >> Connected Open Free Data > >> > > >
