Shawn, can you open an issue so we don't forget about this?
On Thu, May 22, 2014 at 9:31 PM, Shawn Heisey <s...@elyograg.org> wrote: > On 5/22/2014 8:31 AM, Aniket Bhoi wrote: > > On Thu, May 22, 2014 at 7:13 PM, Shawn Heisey <s...@elyograg.org> wrote: > > > >> On 5/22/2014 1:53 AM, Aniket Bhoi wrote: > >>> Details: > >>> > >>> *Solr Version:* > >>> Solr Specification Version: 3.4.0.2012.01.23.14.08.01 > >>> Solr Implementation Version: 3.4 > >>> Lucene Specification Version: 3.4 > >>> Lucene Implementation Version: 3.4 > >>> > >>> *Tomcat version:* > >>> Apache Tomcat/6.0.18 > >>> > >>> *OS details:* > >>> SUSE Linux Enterprise Server 11 (x86_64) > >>> VERSION = 11 > >>> PATCHLEVEL = 1 > >>> > >>> While running indexing on this server,It failed. > >>> > >>> Log excerpt: > >>> > >>> Caused by: > org.apache.solr.handler.dataimport.DataImportHandlerException: > >>> com.microsoft.sqlserver.jdbc.SQLServerException: The result set is > >> closed. > >>> > >>> Out intial hypothesis was that there is a problem with the connection > >>> thread,so we made changes to the context.xml and added > >>> validationQuery,testOnBorrow etc..to make sure the thread doesnt time > >> out. > >>> We also killed a lot of sleeping sessions from the server to the > >> database. > >>> All of the above still didnt work > >> I have reduced your log excerpt to what I think is the important part. > >> > >> Removing the multithreaded support as others have suggested is a good > >> idea, but what I think is really happening here is that Solr is engaging > >> in a multi-tier merge, so it stops indexing for a while ... and > >> meanwhile, JDBC times out and closes your database connection because of > >> inactivity. When the largest merge tier finishes, indexing tries to > >> resume, which it can't do because the database connection is closed. > >> > >> The solution is to allow more simultaneous merges to happen, which > >> allows indexing to continue while a multi-tier merge is underway. This > >> is my indexConfig section from solrconfig.xml: > >> > >> <indexConfig> > >> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy"> > >> <int name="maxMergeAtOnce">35</int> > >> <int name="segmentsPerTier">35</int> > >> <int name="maxMergeAtOnceExplicit">105</int> > >> </mergePolicy> > >> <mergeScheduler > class="org.apache.lucene.index.ConcurrentMergeScheduler"> > >> <int name="maxThreadCount">1</int> > >> <int name="maxMergeCount">6</int> > >> </mergeScheduler> > >> <ramBufferSizeMB>48</ramBufferSizeMB> > >> <infoStream file="INFOSTREAM-${solr.core.name > }.txt">false</infoStream> > >> </indexConfig> > >> > >> The important part for your purposes is the mergeScheduler config, and > >> in particular, maxMergeCount. Increase that to 6. If you are using > >> standard spinning hard disks, do not increase maxThreadCount beyond 1. > >> If you are using SSD, you can safely increase that a small amount, but I > >> don't think I'd go above 2 or 3. > >> > >> Thanks, > >> Shawn > >> > >> > > I may be missing something ,or looking in the wrong place,But I cannot > find > > an indexConfig section or any other mentioned detail above in the > > solrconfig.xml file > > Solr will work without one, in which case it will simply use the > defaults. With older 3.x versions the mergeScheduler config will > actually need to go in an indexDefaults section. The mainIndex and > indexDefaults sections were deprecated in 3.6 and removed entirely in 4.x. > > https://issues.apache.org/jira/browse/SOLR-1052 > > If you don't have indexDefaults either, you may need to add the config > as a top level element under <config>. If you do this, here's what you > should add: > > <indexConfig> > <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"> > <int name="maxThreadCount">1</int> > <int name="maxMergeCount">6</int> > </mergeScheduler> > </indexConfig> > > I think we should probably change the default value that Solr uses for > maxMergeCount. This problem comes up fairly often. As long as > maxThreadCount is 1, I cannot think of a really good reason to limit > maxMergeCount at the level that we currently do. > > Thanks, > Shawn > > -- Regards, Shalin Shekhar Mangar.