Batch replica transfer from master shard solr cloud

2014-10-17 Thread Cp Mishra
Hi, We were getting pretty low ingest throughput when using replicas. We traced it to the following logic in indexing code: - Read SolrInputDocument from stream - Index a document - Add to ConcurrentUpdateSolrServer instance for sending to replica So, we

Re: Batch replica transfer from master shard solr cloud

2014-10-17 Thread Shawn Heisey
On 10/17/2014 8:50 AM, Cp Mishra wrote: So, we changed the logic to: -Read SolrInputDocument objects from stream in batches of 500. -Add documents to ConcurrentUpdateSolrServer instance -Index documents in a loop This has improved indexing speed significantly. What are the caveats to this

Re: Batch replica transfer from master shard solr cloud

2014-10-17 Thread Erick Erickson
Pretty sure that's never been made configurable. I've seen anecdotal evidence of a 30-40% slowdown when adding the first replica, from there the penalty is much less. Cp Mishra: Any time you change code you're absolutely invited to open a JIRA and attach the code for people to look at. Please