On Wed, Mar 16, 2011 at 03:17:41PM +0000, Chris Bamford wrote:
> Hi
>
> I am trying to speed up access to the data in my results Documents
> and was wondering if FieldSelector might be the way forward?
> After my search, I end up with an ArrayList of Documents, from each
> of which I need to extract certain fields and their values as
> key/value pairs in a HashMap. Normally this is quick, but with
> large data sets (10,000+) it can take several seconds. Here is the
> essence of the current code:
>
> for (Document doc : resultDocs) {
> keyVals = getDocKeyValuePairs(doc, fields);
> if (keyVals != null) {
> resultsList.add(keyVals);
> }
> }
Hi,
at this point it's too late, the field selector must be applied when you
fill resultDocs and call IndexReder.document(int id, FieldSelector fs) or
Searcher.doc(int id, FieldSelector fs).
Another point is that your code is too close from the dataset volume,
more doc means more heap. If you don't need yo have the whole dataset
loaded into RAM maybe you could implement a custom Collector that do the
whole job on a doc by doc basis and not collecting and saving all docs
in a Collection.
--
David Causse
Spotter
http://www.spotter.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
