Well I am currently looking at merging too. In my application merging will occur against a filer (read as higher latency device). I am currently working on how to stage indices on local disk before moving to a filer. Assume I must move to a filer eventually for whatever crazzy reason I need to....dont ask it aint funny :-}
In that case I have a different performance issue, that is that FSInputStream and FSOutputStream inherit the buffer size of 1k from OS and IS This would be useful to increase to reduce the amount of RPC's to the filer when doing merges ..... assuming that reads and writes are sequential (CIFS supports a 64k block and NFS supports upto I think 32k). I haven't spent much time on this so far so its not like I know its hard todo. From preliminary experiments its obvious that changing the OS buffersize is not the thing todo. If anyone has successfully increased the FSOutputStream and FSInputStream buffers and got it not to blow up on array copies I would love to know the short cut. Chris --- Kevin Burton <[EMAIL PROTECTED]> wrote: > Chris Collins wrote: > > >To follow up. I was surprised to find that from the experiment of indexing > 4k > >documents to local disk (Dell PE with onboard RAID with 256MB cache). I got > the > >following data from my profile: > > > >70 % time was spent in inverting the document > >30 % in merge > > > > > > > Oh.. yeah.. thats indexing. I'm more interested in merging multiple > indexes... > > Kevin > > -- > > > Use Rojo (RSS/Atom aggregator)! - visit http://rojo.com. > See irc.freenode.net #rojo if you want to chat. > > Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html > > Kevin A. Burton, Location - San Francisco, CA > AIM/YIM - sfburtonator, Web - http://peerfear.org/ > GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
