Hi,
Thanks for your reply (an apologies for the orig msg being ent multiple times
to the list - googlemail problems).
I actually meant to put 'maxBufferredDocs'. I admit I'm not that familar with
this parameter, but as I understand it, it is the number of documents that are
held in ram before flushing to disk. I've noticed the ramBufferSizeMB is a
similar parameter, but using memory as the threshold rather than number of docs.
Is it best not to set these too high on indexers?
In my environment, all writes are done via SolrJ, where documents are placed in
a SolrDocumentList and commit()ed when the list reaches 1000 (default value),
or a configured commit thread interval is reached (default is 20s, whichever
comes first). I suppose this is a SolrJ-side version of 'maxBufferedDocs', so
maybe I don't need to set maxBufferedDocs in solrconfig? (the SolrJ 'client' is
on the same machine as the index)
For the indexer cores (essentially write-only indexes), I wasn't planning on
configuring extra memory for read cache (Lucene value cache or filter cache),
as no queries would/should be received on these. Should I reconsider this?
They'll be plenty of RAM available for indexers to use and still leave enough
for the OS file system cache to do its thing. Do you have any suggestions as to
what would be the best way to use this memory to achieve optimal indexing
speed?
The main things I do now to tune for fast indexing are:
* commiting lists of docs rather than each one separately
* not optimizing too often
* bump up the mergeFactor (I use a value of 25)
Many Thanks!
Peter
> Date: Thu, 11 Mar 2010 09:19:12 -0800
> From: hossman_luc...@fucit.org
> To: solr-user@lucene.apache.org
> Subject: Re: Scaling indexes with high document count
>
>
> : I wonder if anyone might have some insight/advice on index scaling for high
> : document count vs size deployments...
>
> Your general approach sounds reasonable, although specifics of how you'll
> need to tune the caches and how much hardware you'll need will largely
> depend on the specifics of the data and the queries.
>
> I'm not sure what you mean by this though...
>
>
> : As searching would always be performed on replicas - the indexing cores
> : wouldn't be tuned with much autowarming/read cache, but have loads of
> : 'maxdocs' cache. The searchers would be the other way 'round - lots of
>
> what do you mean by "'maxdocs' cache" ?
>
>
>
> -Hoss
>
_________________________________________________________________
Tell us your greatest, weirdest and funniest Hotmail stories
http://clk.atdmt.com/UKM/go/195013117/direct/01/
