On Feb 5, 2008 5:04 PM, Michael Sharpe <[EMAIL PROTECTED]> wrote: > There are several assumptions in your statement that are not accurate: > > 1. different clients are *already* writing to a single networked file > system > This is not true. We have a final process called our Merge that takes the > individual pieces and recombines them into a new file. Not very > efficient. If I have 1000 files that are each 1MB in size and I need a > single 1GB file, I need to read in 1000 files and write out the contents > of the 1000 files to the 1 new file. Then I have to delete the 1000 old > files. Now, assume that this needs to be done 100 times to generate 100 > separate 1GB files. This has a huge processing time overhead, not to > mention impact on the file server while our cluster is still doing other > operations for the remaining nodes. In the near future, it will not be > uncommon for us to end up with TBs worth of data over the numerous data > files.
Write a FileSystemDriver that understands the relationship of these linked files and make it "appear" to have a file that is all of them concatenated logically. Or you could abstract it higher up into the reader having a Stream that knows to switch to the next file as needed. Either way, it is the consumption of the file that is interesting, not the production... =================================== This list is hosted by DevelopMentor� http://www.develop.com View archives and manage your subscription(s) at http://discuss.develop.com
