Re: Lucene Merge Threads

2009-10-14 Thread Jason Rutherglen 
Gio,

> Also, is there any concrete way to know when the merge is actually complete 
> (aside from profiling the machine)?

This would be a great feature to add to the Solr web UI.  The ability
to monitor merges in progress and log how much time each used.

-J

RE: Lucene Merge Threads

2009-10-14 Thread Giovanni Fernandez-Kincade 
In case anyone is having the same problem, I finally got this working, using 
the nightly build link that Yonik sent around:
http://people.apache.org/builds/lucene/solr/nightly/

Thanks,
Gio.
-Original Message-
From: Giovanni Fernandez-Kincade 
Sent: Wednesday, October 14, 2009 2:10 PM
To: Giovanni Fernandez-Kincade; solr-user@lucene.apache.org; 
noble.p...@gmail.com
Subject: RE: Lucene Merge Threads

Does anyone know the correct syntax to specify the maximum number of threads 
for the ConcurrentMergeScheduler?

Also, is there any concrete way to know when the merge is actually complete 
(aside from profiling the machine)?

Thanks,
Gio.

-Original Message-
From: Giovanni Fernandez-Kincade 
Sent: Tuesday, October 13, 2009 7:59 PM
To: Giovanni Fernandez-Kincade; 'solr-user@lucene.apache.org'; 
'noble.p...@gmail.com'
Subject: RE: Lucene Merge Threads

I'm still getting the error after getting the latest from trunk and building 
it. 
This is what I added to the solrconfig.xml:

5



Any other ideas?

Thanks,
Gio.

SEVERE: org.apache.solr.common.SolrException: Error loading class '
5
'
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:310)
at 
org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:325)
at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:81)
at 
org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:178)
at 
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:123)
at 
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:172)
at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:400)
at 
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at 
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at 
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at 
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: 
5

at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
at java.security.AccessController.doPrivileged(Unknown Source)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)
at java.lang.Class.$$YJP$$forName0(Native Method)
at java.lang.Class.forName0(Unknown Source)
at java.lang.Class.forName(Unknown Source)
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:294)
... 28 more

-Original Message-
From: Giovanni Fernandez-Kincade 
Sent: Tuesday, October 13, 2009 10:50 AM
To: solr-user@lucene.apache.org; 'noble.p...@gmail.com'
Subject: RE: Lucene Merge Threads

Here's the version informatio

RE: Lucene Merge Threads

2009-10-14 Thread Giovanni Fernandez-Kincade 
Does anyone know the correct syntax to specify the maximum number of threads 
for the ConcurrentMergeScheduler?

Also, is there any concrete way to know when the merge is actually complete 
(aside from profiling the machine)?

Thanks,
Gio.

-Original Message-
From: Giovanni Fernandez-Kincade 
Sent: Tuesday, October 13, 2009 7:59 PM
To: Giovanni Fernandez-Kincade; 'solr-user@lucene.apache.org'; 
'noble.p...@gmail.com'
Subject: RE: Lucene Merge Threads

I'm still getting the error after getting the latest from trunk and building 
it. 
This is what I added to the solrconfig.xml:

5



Any other ideas?

Thanks,
Gio.

SEVERE: org.apache.solr.common.SolrException: Error loading class '
5
'
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:310)
at 
org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:325)
at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:81)
at 
org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:178)
at 
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:123)
at 
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:172)
at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:400)
at 
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at 
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at 
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at 
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: 
5

at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
at java.security.AccessController.doPrivileged(Unknown Source)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)
at java.lang.Class.$$YJP$$forName0(Native Method)
at java.lang.Class.forName0(Unknown Source)
at java.lang.Class.forName(Unknown Source)
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:294)
... 28 more

-Original Message-
From: Giovanni Fernandez-Kincade 
Sent: Tuesday, October 13, 2009 10:50 AM
To: solr-user@lucene.apache.org; 'noble.p...@gmail.com'
Subject: RE: Lucene Merge Threads

Here's the version information from the admin page:

Solr Specification Version: 1.3.0.2009.07.28.18.51.06
Solr Implementation Version: 1.4-dev ${svnversion} - gkincade - 2009-07-28 
18:51:06
Lucene Specification Version: 2.9-dev
Lucene Implementation Version: 2.9-dev 794238 - 2009-07-15 18:05:08




-Original Message-
From: noble.p...@gmail.com [mailto:noble.p...@gmail.com] On Behalf Of Noble 
Paul ??? ??
Sent: Tuesday, October 13, 2009 2

RE: Lucene Merge Threads

2009-10-13 Thread Giovanni Fernandez-Kincade 
I'm still getting the error after getting the latest from trunk and building 
it. 
This is what I added to the solrconfig.xml:

5



Any other ideas?

Thanks,
Gio.

SEVERE: org.apache.solr.common.SolrException: Error loading class '
5
'
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:310)
at 
org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:325)
at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:81)
at 
org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:178)
at 
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:123)
at 
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:172)
at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:400)
at 
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at 
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at 
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at 
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: 
5

at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
at java.security.AccessController.doPrivileged(Unknown Source)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)
at java.lang.Class.$$YJP$$forName0(Native Method)
at java.lang.Class.forName0(Unknown Source)
at java.lang.Class.forName(Unknown Source)
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:294)
... 28 more

-Original Message-
From: Giovanni Fernandez-Kincade 
Sent: Tuesday, October 13, 2009 10:50 AM
To: solr-user@lucene.apache.org; 'noble.p...@gmail.com'
Subject: RE: Lucene Merge Threads

Here's the version information from the admin page:

Solr Specification Version: 1.3.0.2009.07.28.18.51.06
Solr Implementation Version: 1.4-dev ${svnversion} - gkincade - 2009-07-28 
18:51:06
Lucene Specification Version: 2.9-dev
Lucene Implementation Version: 2.9-dev 794238 - 2009-07-15 18:05:08




-Original Message-
From: noble.p...@gmail.com [mailto:noble.p...@gmail.com] On Behalf Of Noble 
Paul ??? ??
Sent: Tuesday, October 13, 2009 2:32 AM
To: solr-user@lucene.apache.org
Subject: Re: Lucene Merge Threads

which version of Solr are you using? the 1 syntax was added recently

On Tue, Oct 13, 2009 at 8:08 AM, Giovanni Fernandez-Kincade
 wrote:
> This didn't end up working. I got the following error when I tried to commit:
>
> Oct 12, 2009 8:36:42 PM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Error loading cl

RE: Lucene Merge Threads

2009-10-13 Thread Giovanni Fernandez-Kincade 
Will do. Thanks!

-Original Message-
From: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com] 
Sent: Tuesday, October 13, 2009 11:48 AM
To: solr-user@lucene.apache.org
Subject: Re: Lucene Merge Threads

On Tue, Oct 13, 2009 at 8:19 PM, Giovanni Fernandez-Kincade <
gfernandez-kinc...@capitaliq.com> wrote:

> Here's the version information from the admin page:
>
> Solr Specification Version: 1.3.0.2009.07.28.18.51.06
> Solr Implementation Version: 1.4-dev ${svnversion} - gkincade - 2009-07-28
> 18:51:06
> Lucene Specification Version: 2.9-dev
> Lucene Implementation Version: 2.9-dev 794238 - 2009-07-15 18:05:08
>
>
This was built on 2009-07-28. The property injection stuff was committed to
trunk on 2009-09-27. Can you please try using the latest nightly?

-- 
Regards,
Shalin Shekhar Mangar.

Re: Lucene Merge Threads

2009-10-13 Thread Shalin Shekhar Mangar 
On Tue, Oct 13, 2009 at 8:19 PM, Giovanni Fernandez-Kincade <
gfernandez-kinc...@capitaliq.com> wrote:

> Here's the version information from the admin page:
>
> Solr Specification Version: 1.3.0.2009.07.28.18.51.06
> Solr Implementation Version: 1.4-dev ${svnversion} - gkincade - 2009-07-28
> 18:51:06
> Lucene Specification Version: 2.9-dev
> Lucene Implementation Version: 2.9-dev 794238 - 2009-07-15 18:05:08
>
>
This was built on 2009-07-28. The property injection stuff was committed to
trunk on 2009-09-27. Can you please try using the latest nightly?

-- 
Regards,
Shalin Shekhar Mangar.

RE: Lucene Merge Threads

2009-10-13 Thread Giovanni Fernandez-Kincade 
Here's the version information from the admin page:

Solr Specification Version: 1.3.0.2009.07.28.18.51.06
Solr Implementation Version: 1.4-dev ${svnversion} - gkincade - 2009-07-28 
18:51:06
Lucene Specification Version: 2.9-dev
Lucene Implementation Version: 2.9-dev 794238 - 2009-07-15 18:05:08




-Original Message-
From: noble.p...@gmail.com [mailto:noble.p...@gmail.com] On Behalf Of Noble 
Paul ??? ??
Sent: Tuesday, October 13, 2009 2:32 AM
To: solr-user@lucene.apache.org
Subject: Re: Lucene Merge Threads

which version of Solr are you using? the 1 syntax was added recently

On Tue, Oct 13, 2009 at 8:08 AM, Giovanni Fernandez-Kincade
 wrote:
> This didn't end up working. I got the following error when I tried to commit:
>
> Oct 12, 2009 8:36:42 PM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Error loading class '
>                5
>        '
>        at 
> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:310)
>        at 
> org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:325)
>        at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:81)
>        at 
> org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:178)
>        at 
> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:123)
>        at 
> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:172)
>        at 
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:400)
>        at 
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>        at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
>        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>        at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
>        at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
>        at 
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>        at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
>        at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>        at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>        at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>        at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
>        at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>        at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>        at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>        at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
>        at 
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
>        at 
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
>        at 
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
>        at 
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
>        at 
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
>        at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.ClassNotFoundException:
>                5
>
>        at java.net.URLClassLoader$1.run(Unknown Source)
>        at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
>        at java.security.AccessController.doPrivileged(Unknown Source)
>        at java.net.URLClassLoader.findClass(Unknown Source)
>        at java.lang.ClassLoader.loadClass(Unknown Source)
>        at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
>        at java.lang.ClassLoader.loadClass(Unknown Source)
>        at java.lang.ClassLoader.loadClassInternal(Unknown Source)
>        at java.lang.Class.$$YJP$$forName0(Native Method)
>        at java.lang.Class.forName0(Unknown Source)
>        at java.lang.Class.forName(Unknown Source)
>        at 
> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:294)
>        ... 28 more
>
>
> I believe it's because the MaxThreadCount is not a public property of the 
> ConcurrentMergeSchedulerClass. You have to call this method to set it:
>
> public voi

Re: Lucene Merge Threads

2009-10-12 Thread Noble Paul നോബിള്‍ नोब्ळ् 
which version of Solr are you using? the 1 syntax was added recently

On Tue, Oct 13, 2009 at 8:08 AM, Giovanni Fernandez-Kincade
 wrote:
> This didn't end up working. I got the following error when I tried to commit:
>
> Oct 12, 2009 8:36:42 PM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Error loading class '
>                5
>        '
>        at 
> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:310)
>        at 
> org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:325)
>        at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:81)
>        at 
> org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:178)
>        at 
> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:123)
>        at 
> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:172)
>        at 
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:400)
>        at 
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>        at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
>        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>        at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
>        at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
>        at 
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>        at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
>        at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>        at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>        at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>        at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
>        at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>        at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>        at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>        at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
>        at 
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
>        at 
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
>        at 
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
>        at 
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
>        at 
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
>        at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.ClassNotFoundException:
>                5
>
>        at java.net.URLClassLoader$1.run(Unknown Source)
>        at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
>        at java.security.AccessController.doPrivileged(Unknown Source)
>        at java.net.URLClassLoader.findClass(Unknown Source)
>        at java.lang.ClassLoader.loadClass(Unknown Source)
>        at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
>        at java.lang.ClassLoader.loadClass(Unknown Source)
>        at java.lang.ClassLoader.loadClassInternal(Unknown Source)
>        at java.lang.Class.$$YJP$$forName0(Native Method)
>        at java.lang.Class.forName0(Unknown Source)
>        at java.lang.Class.forName(Unknown Source)
>        at 
> org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:294)
>        ... 28 more
>
>
> I believe it's because the MaxThreadCount is not a public property of the 
> ConcurrentMergeSchedulerClass. You have to call this method to set it:
>
> public void setMaxThreadCount(int count) {
>    if (count < 1)
>      throw new IllegalArgumentException("count should be at least 1");
>    maxThreadCount = count;
>  }
>
> Is that possible through the solrconfig?
>
> Thanks,
> Gio.
>
> -Original Message-
> From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com]
> Sent: Monday, October 12, 2009 7:53 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Lucene Merge Threads
>
> Do you have to make a new call

RE: Lucene Merge Threads

2009-10-12 Thread Giovanni Fernandez-Kincade 
This didn't end up working. I got the following error when I tried to commit:

Oct 12, 2009 8:36:42 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Error loading class '
5
'
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:310)
at 
org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:325)
at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:81)
at 
org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:178)
at 
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:123)
at 
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:172)
at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:400)
at 
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at 
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at 
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at 
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: 
5

at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
at java.security.AccessController.doPrivileged(Unknown Source)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)
at java.lang.Class.$$YJP$$forName0(Native Method)
at java.lang.Class.forName0(Unknown Source)
at java.lang.Class.forName(Unknown Source)
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:294)
... 28 more


I believe it's because the MaxThreadCount is not a public property of the 
ConcurrentMergeSchedulerClass. You have to call this method to set it:

public void setMaxThreadCount(int count) {
if (count < 1)
  throw new IllegalArgumentException("count should be at least 1");
maxThreadCount = count;
  }

Is that possible through the solrconfig?

Thanks,
Gio.

-Original Message-
From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com] 
Sent: Monday, October 12, 2009 7:53 PM
To: solr-user@lucene.apache.org
Subject: RE: Lucene Merge Threads

Do you have to make a new call to optimize to make it start the merge again?

-Original Message-
From: Jason Rutherglen [mailto:jason.rutherg...@gmail.com] 
Sent: Monday, October 12, 2009 7:28 PM
To: solr-user@lucene.apache.org
Subject: Re: Lucene Merge Threads

Try this in solrconfig.xml:


  1


Yes you can stop the process mid-merge.  The partially merged files
will be deleted on restart.

We need to update the wiki?

On Mon, Oct 12, 2009 at 4:05 PM, Giovanni Fernandez-Kincade
 wrote:
> Hi,
> I'm attempting to optimize a pretty large index, and even though the optimiz

RE: Lucene Merge Threads

2009-10-12 Thread Giovanni Fernandez-Kincade 
Do you have to make a new call to optimize to make it start the merge again?

-Original Message-
From: Jason Rutherglen [mailto:jason.rutherg...@gmail.com] 
Sent: Monday, October 12, 2009 7:28 PM
To: solr-user@lucene.apache.org
Subject: Re: Lucene Merge Threads

Try this in solrconfig.xml:


  1


Yes you can stop the process mid-merge.  The partially merged files
will be deleted on restart.

We need to update the wiki?

On Mon, Oct 12, 2009 at 4:05 PM, Giovanni Fernandez-Kincade
 wrote:
> Hi,
> I'm attempting to optimize a pretty large index, and even though the optimize 
> request timed out, I watched it using a profiler and saw that the optimize 
> thread continued executing. Eventually it completed, but in the background I 
> still see a thread performing a merge:
>
> Lucene Merge Thread #0 [RUNNABLE, IN_NATIVE] CPU time: 17:51
> java.io.RandomAccessFile.readBytes(byte[], int, int)
> java.io.RandomAccessFile.read(byte[], int, int)
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
>  int, int)
> org.apache.lucene.store.BufferedIndexInput.refill()
> org.apache.lucene.store.BufferedIndexInput.readByte()
> org.apache.lucene.store.IndexInput.readVInt()
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
> org.apache.lucene.index.SegmentTermEnum.next()
> org.apache.lucene.index.SegmentMergeInfo.next()
> org.apache.lucene.index.SegmentMerger.mergeTermInfos(FormatPostingsFieldsConsumer)
> org.apache.lucene.index.SegmentMerger.mergeTerms()
> org.apache.lucene.index.SegmentMerger.merge(boolean)
> org.apache.lucene.index.IndexWriter.mergeMiddle(MergePolicy$OneMerge)
> org.apache.lucene.index.IndexWriter.merge(MergePolicy$OneMerge)
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(MergePolicy$OneMerge)
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run()
>
>
> This has taken quite a while, and hasn't really been fully utilizing the 
> machine's resources. After looking at the Lucene source, I noticed that you 
> can set a MaxThreadCount parameter in this class. Is this parameter exposed 
> by Solr somehow? I see the class mentioned, commented out, in my 
> solrconfig.xml, but I'm not sure of the correct way to specify the parameter:
>
> 
>    
> 
>
>
> Also, if I can specify this parameter, is it safe to just start/stop my 
> servlet server (Tomcat) mid-merge?
>
> Thanks in advance,
> Gio.
>

Re: Lucene Merge Threads

2009-10-12 Thread Jason Rutherglen 
Try this in solrconfig.xml:


  1


Yes you can stop the process mid-merge.  The partially merged files
will be deleted on restart.

We need to update the wiki?

On Mon, Oct 12, 2009 at 4:05 PM, Giovanni Fernandez-Kincade
 wrote:
> Hi,
> I'm attempting to optimize a pretty large index, and even though the optimize 
> request timed out, I watched it using a profiler and saw that the optimize 
> thread continued executing. Eventually it completed, but in the background I 
> still see a thread performing a merge:
>
> Lucene Merge Thread #0 [RUNNABLE, IN_NATIVE] CPU time: 17:51
> java.io.RandomAccessFile.readBytes(byte[], int, int)
> java.io.RandomAccessFile.read(byte[], int, int)
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
>  int, int)
> org.apache.lucene.store.BufferedIndexInput.refill()
> org.apache.lucene.store.BufferedIndexInput.readByte()
> org.apache.lucene.store.IndexInput.readVInt()
> org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
> org.apache.lucene.index.SegmentTermEnum.next()
> org.apache.lucene.index.SegmentMergeInfo.next()
> org.apache.lucene.index.SegmentMerger.mergeTermInfos(FormatPostingsFieldsConsumer)
> org.apache.lucene.index.SegmentMerger.mergeTerms()
> org.apache.lucene.index.SegmentMerger.merge(boolean)
> org.apache.lucene.index.IndexWriter.mergeMiddle(MergePolicy$OneMerge)
> org.apache.lucene.index.IndexWriter.merge(MergePolicy$OneMerge)
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(MergePolicy$OneMerge)
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run()
>
>
> This has taken quite a while, and hasn't really been fully utilizing the 
> machine's resources. After looking at the Lucene source, I noticed that you 
> can set a MaxThreadCount parameter in this class. Is this parameter exposed 
> by Solr somehow? I see the class mentioned, commented out, in my 
> solrconfig.xml, but I'm not sure of the correct way to specify the parameter:
>
> 
>    
> 
>
>
> Also, if I can specify this parameter, is it safe to just start/stop my 
> servlet server (Tomcat) mid-merge?
>
> Thanks in advance,
> Gio.
>

Lucene Merge Threads

2009-10-12 Thread Giovanni Fernandez-Kincade 
Hi,
I'm attempting to optimize a pretty large index, and even though the optimize 
request timed out, I watched it using a profiler and saw that the optimize 
thread continued executing. Eventually it completed, but in the background I 
still see a thread performing a merge:

Lucene Merge Thread #0 [RUNNABLE, IN_NATIVE] CPU time: 17:51
java.io.RandomAccessFile.readBytes(byte[], int, int)
java.io.RandomAccessFile.read(byte[], int, int)
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(byte[],
 int, int)
org.apache.lucene.store.BufferedIndexInput.refill()
org.apache.lucene.store.BufferedIndexInput.readByte()
org.apache.lucene.store.IndexInput.readVInt()
org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos)
org.apache.lucene.index.SegmentTermEnum.next()
org.apache.lucene.index.SegmentMergeInfo.next()
org.apache.lucene.index.SegmentMerger.mergeTermInfos(FormatPostingsFieldsConsumer)
org.apache.lucene.index.SegmentMerger.mergeTerms()
org.apache.lucene.index.SegmentMerger.merge(boolean)
org.apache.lucene.index.IndexWriter.mergeMiddle(MergePolicy$OneMerge)
org.apache.lucene.index.IndexWriter.merge(MergePolicy$OneMerge)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(MergePolicy$OneMerge)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run()


This has taken quite a while, and hasn't really been fully utilizing the 
machine's resources. After looking at the Lucene source, I noticed that you can 
set a MaxThreadCount parameter in this class. Is this parameter exposed by Solr 
somehow? I see the class mentioned, commented out, in my solrconfig.xml, but 
I'm not sure of the correct way to specify the parameter:






Also, if I can specify this parameter, is it safe to just start/stop my servlet 
server (Tomcat) mid-merge?

Thanks in advance,
Gio.