In case anyone is having the same problem, I finally got this working, using
the nightly build link that Yonik sent around:
http://people.apache.org/builds/lucene/solr/nightly/
Thanks,
Gio.
-----Original Message-----
From: Giovanni Fernandez-Kincade
Sent: Wednesday, October 14, 2009 2:10 PM
To: Giovanni Fernandez-Kincade; solr-user@lucene.apache.org;
noble.p...@gmail.com
Subject: RE: Lucene Merge Threads
Does anyone know the correct syntax to specify the maximum number of threads
for the ConcurrentMergeScheduler?
Also, is there any concrete way to know when the merge is actually complete
(aside from profiling the machine)?
Thanks,
Gio.
-----Original Message-----
From: Giovanni Fernandez-Kincade
Sent: Tuesday, October 13, 2009 7:59 PM
To: Giovanni Fernandez-Kincade; 'solr-user@lucene.apache.org';
'noble.p...@gmail.com'
Subject: RE: Lucene Merge Threads
I'm still getting the error after getting the latest from trunk and building
it.
This is what I added to the solrconfig.xml:
<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
<int name="maxThreadCount">5</int>
</mergeScheduler>
Any other ideas?
Thanks,
Gio.
SEVERE: org.apache.solr.common.SolrException: Error loading class '
5
'
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:310)
at
org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:325)
at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:81)
at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:178)
at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:123)
at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:172)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:400)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException:
5
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
at java.security.AccessController.doPrivileged(Unknown Source)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)
at java.lang.Class.$$YJP$$forName0(Native Method)
at java.lang.Class.forName0(Unknown Source)
at java.lang.Class.forName(Unknown Source)
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:294)
... 28 more
-----Original Message-----
From: Giovanni Fernandez-Kincade
Sent: Tuesday, October 13, 2009 10:50 AM
To: solr-user@lucene.apache.org; 'noble.p...@gmail.com'
Subject: RE: Lucene Merge Threads
Here's the version information from the admin page:
Solr Specification Version: 1.3.0.2009.07.28.18.51.06
Solr Implementation Version: 1.4-dev ${svnversion} - gkincade - 2009-07-28
18:51:06
Lucene Specification Version: 2.9-dev
Lucene Implementation Version: 2.9-dev 794238 - 2009-07-15 18:05:08
-----Original Message-----
From: noble.p...@gmail.com [mailto:noble.p...@gmail.com] On Behalf Of Noble
Paul ??????? ??????
Sent: Tuesday, October 13, 2009 2:32 AM
To: solr-user@lucene.apache.org
Subject: Re: Lucene Merge Threads
which version of Solr are you using? the <int
name="maxThreadCount">1</int> syntax was added recently
On Tue, Oct 13, 2009 at 8:08 AM, Giovanni Fernandez-Kincade
<gfernandez-kinc...@capitaliq.com> wrote:
> This didn't end up working. I got the following error when I tried to commit:
>
> Oct 12, 2009 8:36:42 PM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Error loading class '
> 5
> '
> at
> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:310)
> at
> org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:325)
> at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:81)
> at
> org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:178)
> at
> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:123)
> at
> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:172)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:400)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
> at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
> at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.ClassNotFoundException:
> 5
>
> at java.net.URLClassLoader$1.run(Unknown Source)
> at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
> at java.security.AccessController.doPrivileged(Unknown Source)
> at java.net.URLClassLoader.findClass(Unknown Source)
> at java.lang.ClassLoader.loadClass(Unknown Source)
> at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
> at java.lang.ClassLoader.loadClass(Unknown Source)
> at java.lang.ClassLoader.loadClassInternal(Unknown Source)
> at java.lang.Class.$$YJP$$forName0(Native Method)
> at java.lang.Class.forName0(Unknown Source)
> at java.lang.Class.forName(Unknown Source)
> at
> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:294)
> ... 28 more
>
>
> I believe it's because the MaxThreadCount is not a public property of the
> ConcurrentMergeSchedulerClass. You have to call this method to set it:
>
> public void setMaxThreadCount(int count) {
> if (count < 1)
> throw new IllegalArgumentException("count should be at least 1");
> maxThreadCount = count;
> }
>
> Is that possible through the solrconfig?
>
> Thanks,
> Gio.
>
> -----Original Message-----
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
> Sent: Monday, October 12, 2009 7:53 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Lucene Merge Threads
>
> Do you have to make a new call to optimize to make it start the merge again?
>
> -----Original Message-----
> From: Jason Rutherglen [mailto:jason.rutherg...@gmail.com]
> Sent: Monday, October 12, 2009 7:28 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Lucene Merge Threads
>
> Try this in solrconfig.xml:
>
> <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
> <int name="maxThreadCount">1</int>
> </mergeScheduler>
>
> Yes you can stop the process mid-merge. The partially merged files
> will be deleted on restart.
>
> We need to update the wiki?
>
> On Mon, Oct 12, 2009 at 4:05 PM, Giovanni Fernandez-Kincade
> <gfernandez-kinc...@capitaliq.com> wrote:
>> Hi,
>> I'm attempting to optimize a pretty large index, and even though the
>> optimize request timed out, I watched it using a profiler and saw that the
>> optimize thread continued executing. Eventually it completed, but in the
>> background I still see a thread performing a merge:
>>
>> Lucene Merge Thread #0 [RUNNABLE, IN_NATIVE] CPU time: 17:51
>> java.io.RandomAccessFile.readBytes(byte[], int, int)
>> java.io.RandomAccessFile.read(byte[], int, int)
>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
>> int, int)
>> org.apache.lucene.store.BufferedIndexInput.refill()
>> org.apache.lucene.store.BufferedIndexInput.readByte()
>> org.apache.lucene.store.IndexInput.readVInt()
>> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
>> org.apache.lucene.index.SegmentTermEnum.next()
>> org.apache.lucene.index.SegmentMergeInfo.next()
>> org.apache.lucene.index.SegmentMerger.mergeTermInfos(FormatPostingsFieldsConsumer)
>> org.apache.lucene.index.SegmentMerger.mergeTerms()
>> org.apache.lucene.index.SegmentMerger.merge(boolean)
>> org.apache.lucene.index.IndexWriter.mergeMiddle(MergePolicy$OneMerge)
>> org.apache.lucene.index.IndexWriter.merge(MergePolicy$OneMerge)
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(MergePolicy$OneMerge)
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run()
>>
>>
>> This has taken quite a while, and hasn't really been fully utilizing the
>> machine's resources. After looking at the Lucene source, I noticed that you
>> can set a MaxThreadCount parameter in this class. Is this parameter exposed
>> by Solr somehow? I see the class mentioned, commented out, in my
>> solrconfig.xml, but I'm not sure of the correct way to specify the parameter:
>>
>> <!--
>> Expert:
>> The Merge Scheduler in Lucene controls how merges are performed. The
>> ConcurrentMergeScheduler (Lucene 2.3 default)
>> can perform merges in the background using separate threads. The
>> SerialMergeScheduler (Lucene 2.2 default) does not.
>> -->
>>
>> <!--<mergeScheduler>org.apache.lucene.index.ConcurrentMergeScheduler</mergeScheduler>-->
>>
>>
>> Also, if I can specify this parameter, is it safe to just start/stop my
>> servlet server (Tomcat) mid-merge?
>>
>> Thanks in advance,
>> Gio.
>>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
