On Thu, May 22, 2014 at 9:31 PM, Shawn Heisey <s...@elyograg.org> wrote:
> On 5/22/2014 8:31 AM, Aniket Bhoi wrote:
> > On Thu, May 22, 2014 at 7:13 PM, Shawn Heisey <s...@elyograg.org> wrote:
> >
> >> On 5/22/2014 1:53 AM, Aniket Bhoi wrote:
> >>> Details:
> >>>
> >>> *Solr Version:*
> >>> Solr Specification Version: 3.4.0.2012.01.23.14.08.01
> >>> Solr Implementation Version: 3.4
> >>> Lucene Specification Version: 3.4
> >>> Lucene Implementation Version: 3.4
> >>>
> >>> *Tomcat version:*
> >>> Apache Tomcat/6.0.18
> >>>
> >>> *OS details:*
> >>> SUSE Linux Enterprise Server 11 (x86_64)
> >>> VERSION = 11
> >>> PATCHLEVEL = 1
> >>>
> >>> While running indexing on this server,It failed.
> >>>
> >>> Log excerpt:
> >>>
> >>> Caused by:
> org.apache.solr.handler.dataimport.DataImportHandlerException:
> >>> com.microsoft.sqlserver.jdbc.SQLServerException: The result set is
> >> closed.
> >>>
> >>> Out intial hypothesis was that there is a problem with the connection
> >>> thread,so we made changes to the context.xml and added
> >>> validationQuery,testOnBorrow etc..to make sure the thread doesnt time
> >> out.
> >>> We also killed a lot of sleeping sessions from the server to the
> >> database.
> >>> All of the above still didnt work
> >> I have reduced your log excerpt to what I think is the important part.
> >>
> >> Removing the multithreaded support as others have suggested is a good
> >> idea, but what I think is really happening here is that Solr is engaging
> >> in a multi-tier merge, so it stops indexing for a while ... and
> >> meanwhile, JDBC times out and closes your database connection because of
> >> inactivity. When the largest merge tier finishes, indexing tries to
> >> resume, which it can't do because the database connection is closed.
> >>
> >> The solution is to allow more simultaneous merges to happen, which
> >> allows indexing to continue while a multi-tier merge is underway. This
> >> is my indexConfig section from solrconfig.xml:
> >>
> >> <indexConfig>
> >> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> >> <int name="maxMergeAtOnce">35</int>
> >> <int name="segmentsPerTier">35</int>
> >> <int name="maxMergeAtOnceExplicit">105</int>
> >> </mergePolicy>
> >> <mergeScheduler
> class="org.apache.lucene.index.ConcurrentMergeScheduler">
> >> <int name="maxThreadCount">1</int>
> >> <int name="maxMergeCount">6</int>
> >> </mergeScheduler>
> >> <ramBufferSizeMB>48</ramBufferSizeMB>
> >> <infoStream file="INFOSTREAM-${solr.core.name
> }.txt">false</infoStream>
> >> </indexConfig>
> >>
> >> The important part for your purposes is the mergeScheduler config, and
> >> in particular, maxMergeCount. Increase that to 6. If you are using
> >> standard spinning hard disks, do not increase maxThreadCount beyond 1.
> >> If you are using SSD, you can safely increase that a small amount, but I
> >> don't think I'd go above 2 or 3.
> >>
> >> Thanks,
> >> Shawn
> >>
> >>
> > I may be missing something ,or looking in the wrong place,But I cannot
> find
> > an indexConfig section or any other mentioned detail above in the
> > solrconfig.xml file
>
> Solr will work without one, in which case it will simply use the
> defaults. With older 3.x versions the mergeScheduler config will
> actually need to go in an indexDefaults section. The mainIndex and
> indexDefaults sections were deprecated in 3.6 and removed entirely in 4.x.
>
> https://issues.apache.org/jira/browse/SOLR-1052
>
> If you don't have indexDefaults either, you may need to add the config
> as a top level element under <config>. If you do this, here's what you
> should add:
>
> <indexConfig>
> <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
> <int name="maxThreadCount">1</int>
> <int name="maxMergeCount">6</int>
> </mergeScheduler>
> </indexConfig>
>
> I think we should probably change the default value that Solr uses for
> maxMergeCount. This problem comes up fairly often. As long as
> maxThreadCount is 1, I cannot think of a really good reason to limit
> maxMergeCount at the level that we currently do.
>
> Thanks,
> Shawn
>
>
I changed the solrconfig.xml file and included the changes you
suggested.However,this didnt work out,the Indexing still fails..I have also
added this
*:maxActive="100" minIdle="10" maxWait="10000" initialSize="10"
logAbandoned="true" validationQuery="select 1" testOnBorrow="true"
testOnReturn="true" validationQueryTimeout="30" removeAbandoned="true"
removeAbandonedTimeout="3600" *
to the* context.xml* file.This didnt work etiher.The thing to note is after
I changed the solrconfig.xml to add the merge config changes,the indexing
failed after 6 hours.Earlier,It used to fail after 1 hour .
