2011/3/21 Brian Hurt <bhur...@gmail.com>: > I'm having a problem with the performance of lazily-loaded fields with > lucene. The basic structure of the code is that I get a set of documents > back from a query, then iterate through them, reading most fields to collect > fragments. This is taking an excessively long amount of time- mostly in my > call to getField on the document. Each such call takes milliseconds, > meaning a query that even returns only a few dozen items takes me hundreds > of milliseconds to process, which is causing problems elsewhere. So the > question I have is if there is some way I can optimize the loading of > fields? I am calling the doc function on the index search with a null > FieldSelector, but this does not seem to reduce the cost of getting fields > (indeed, it seems to slow down the whole query processing by a significant > factor). Is there any help anyone can give me?
Hi, you should not use a null FieldSelector, but create one instead, for example the org.apache.lucene.document.MapFieldSelector, so to make sure you're only going to initialize the fields you actually need (assuming you need a subset). Also you might have a look into FieldCache usage, there's an excellent chapter on it in Lucene in Action (second edition); this approach assumes you can dedicate some memory to cache extracted field values in memory, and you're in a read-mostly scenario (as filling the FieldCache might be more costly than a single extraction: it affects all elements in the index, not your Hits only). I've also seen that the cost of Field extraction is quite cheap on solid state disks, compared to rotating disks at least. Regards, Sanne > > Thanks. > > Brian > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
