On 1/8/2013 11:19 PM, vijeshnair wrote:
Yes Shawn, the batchSize is -1 only and I also have the mergeScheduler exactly same as you mentioned. When I had this problem in SOLR 3.4, I did an extensive googling and gathered much of the tweaks and tuning from different blogs and forums and configured the 4.0 instance. My next full run is scheduled for this weekend, I will try with a higher mysql wait_timeout value and update you the outcome.
With maxThreadCount at 1 and maxMergeCount at 6, I was able to complete full-import with no problems. All mysql (5.1.61) server-side timeouts are at their defaults - they don't show up in my.cnf and I haven't tweaked them anywhere else either.
A full import for me consists of six simultaneous imports into six Solr cores, each of which is over 12 million rows. It takes three hours, and each of those six imports creates a 16GB index on Solr 4.1-SNAPSHOT, 22GB on Solr 3.5.0. There is a seventh import as well, but it only does a few hundred thousand rows. That one finishes before any major merging takes place.
Thanks, Shawn
