Re: index update (was Re: Large InputStream.BUFFER_SIZE causes OutOfMemoryError.. FYI)

2004-04-14 Thread Kevin A. Burton
petite_abeille wrote: On Apr 13, 2004, at 02:45, Kevin A. Burton wrote: He mentioned that I might be able to squeeze 5-10% out of index merges this way. Talking of which... what strategy(ies) do people use to minimize downtime when updating an index? This should probably be a wiki page. Any

Re: index update (was Re: Large InputStream.BUFFER_SIZE causes OutOfMemoryError.. FYI)

2004-04-13 Thread Stephane James Vaucher
I'm actually pretty lazy about index updates, and haven't had the need for efficiency, since my requirement is that new documents should be available on a next working day basis. I reindex everything from scatch every night (400,000 docs) and store it in an timestamped index. When the reindexin

index update (was Re: Large InputStream.BUFFER_SIZE causes OutOfMemoryError.. FYI)

2004-04-13 Thread petite_abeille
On Apr 13, 2004, at 02:45, Kevin A. Burton wrote: He mentioned that I might be able to squeeze 5-10% out of index merges this way. Talking of which... what strategy(ies) do people use to minimize downtime when updating an index? My current "strategy" is as follow: (1) use a temporary RAMDirect

Large InputStream.BUFFER_SIZE causes OutOfMemoryError.. FYI

2004-04-12 Thread Kevin A. Burton
Not sure if this is a bug or expected behavior. I took Doug's suggestion and migrated to a large BUFFER_SIZE of 1024^2 . He mentioned that I might be able to squeeze 5-10% out of index merges this way. I'm not sure if this is expected behavior but this requires a LOT of memory. Without this