On Tue, Jan 12, 2010 at 2:53 PM, Smith G <gudumba.sm...@gmail.com> wrote: > 4) queuesize parameter of Streaming constructer: What could be the > rough-value when it comes > to real time application having a million+ documents to be indexed ? .. > So what does "queuesize" is exactly for ? , if we can go on > adding as many as we can.
The queue provides a buffer between the document producer(s) (your application code) and the consumers (the solrj impl that sends the doc to solr) and helps to further increase concurrency. Consider the following scenario if you had only 1 document producer thread (the common case) and no queue inbetween. Your producer creates a document, and adds it... but since all of the consumer threads are busy, it blocks. Next, consumer thread #1 and #2 both become ready at the same time. Consumer thread takes your document, unblocking your producer thread to create another document. But in the meantime, consumer thread #2 sits idle. It probably doesn't make sense to set the buffer above 10, unless your document creation is really "bursty" and you need the additional buffering to make sure that indexing threads are kept busy. -Yonik http://www.lucidimagination.com
