Simply breaking up your index into separate pieces on the same machine buys you nothing, in fact it costs you considerably. Have you put a profiler on the system to see what's happening? I expect you're swapping all over the place and are memory-constrained.
Have you considered sharding your index across multiple machines? That is often an approach to use assuming that logically this is a single index... Best Erick On Fri, Jul 8, 2011 at 10:39 AM, Chris Bamford <chris.bamf...@talktalk.net> wrote: > > Hi > > > I was wondering how to improve search performance over a set of indexes like > this: > > 27G K1-1/index > 19G K1-2/index > 24G K1-3/index > 15G K1-4/index > 19G K1-5/index > 31G K2-1/index > 16G K2-2/index > 8.1G K2-3/index > 12G K2-4/index > 15G K2-5/index > > In total it is ~192Gb. All have the same schema, with data partitioned by my > application into separate > indexes (rather than just 1 big one). > I have tried searching with both a MultiSearcher and a ParallelMultiSearcher > (with lots of memory), but it > is way too slow (sometimes hangs, otherwise it takes tens of minutes). Do > you have any tips for improving speed? > I have reduced the stored data as much as possible (without using > compression) and use a FieldSelector when querying > but neither seem to help.. > > Is this the right approach? If I query them one by one I response time is > reasonable (a few seconds each), but I was > kind of hoping to treat them as one large index if possible. > > Any thoughts welcomed. > > Thanks, > > - Chris > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
