Here's a great blog explaining one major difference between 3.x and 4.x: http://www.searchworkings.org/blog/-/blogs/gimme-all-resources-you-have-i-can-use-them!/
In a nutshell, 3.x blocks on segment merges (which can be triggered by commits). I've heard anecdotal accounts that pushing your rambuffer size much above 256M doesn't help, you may want to lower it just to see if there's any difference.... But the key would be looking at CPU utilization on your server. If that's near 100% (when not merging in 3.x), you can't go any faster. FWIW, Erick On Fri, Oct 26, 2012 at 2:05 AM, adityab <aditya_ba...@yahoo.com> wrote: > thanks for the replies, > > the reason we have the custom program is we need to gather data from > different sources, massage it and then post to solr. Our program is > multithreaded with 10 threads fetching data and putting it in blocking Queue > and 5 threads posting the data from Queue to Solr. Each thread writes 1000 > documents at a time. > We have observed that with 1024 rambuffer set on master we are able to > publish 20M documents in around 3.5 to 4.5 hrs > Every 1000 document posted to Solr on an average takes 3 seconds (based on > QT) > > Any advice on speeding up the process. > > I will try doing commit at the end to see if there is any difference. > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Best-way-to-commit-data-to-Solr-tp4015921p4015992.html > Sent from the Solr - User mailing list archive at Nabble.com.
