NumberFormatException upon reading a Trie field during search
Hi there, I have recently upgraded our Solr instance and have reindexed all of the items in our store, and for at least one search I am getting some unusual error messages back for a search that previously worked. It reads as follows: HTTP Status 500 - Invalid shift value in prefixCoded string (is encoded value really an INT?) java.lang.NumberFormatException: Invalid shift value in prefixCoded string (is encoded value really an INT?) at org.apache.lucene.util.NumericUtils.prefixCodedToInt(NumericUtils.java:233) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:237) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:441) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:208) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:414) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$MultiComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:435) at org.apache.solr.search.DocSetDelegateCollector.setNextReader(DocSetHitCollector.java:140) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:253) at org.apache.lucene.search.Searcher.search(Searcher.java:171) at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1101) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:880) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:619) java.lang.NumberFormatException: Invalid shift value in prefixCoded string (is encoded value really an INT?) at org.apache.lucene.util.NumericUtils.prefixCodedToInt(NumericUtils.java:233) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:237) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:441) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:208) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:414) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$MultiComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:435) at org.apache.solr.search.DocSetDelegateCollector.setNextReader(DocSetHitCollector.java:140) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:253) at org.apache.lucene.search.Searcher.search(Searcher.java:171) at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1101) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:880) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at
Re: NumberFormatException upon reading a Trie field during search
Hi all, Just to let you know, deleting the index and reindexing our data appears to have fixed this problem, at least for the moment. My guess is that the old index wasn't deleted cleanly, as I assumed it had been. Thanks Jon On 8 Oct 2010, at 12:14, Jon Poulton wrote: Hi there, I have recently upgraded our Solr instance and have reindexed all of the items in our store, and for at least one search I am getting some unusual error messages back for a search that previously worked. It reads as follows: HTTP Status 500 - Invalid shift value in prefixCoded string (is encoded value really an INT?) java.lang.NumberFormatException: Invalid shift value in prefixCoded string (is encoded value really an INT?) at org.apache.lucene.util.NumericUtils.prefixCodedToInt(NumericUtils.java:233) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:237) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:441) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:208) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:414) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$MultiComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:435) at org.apache.solr.search.DocSetDelegateCollector.setNextReader(DocSetHitCollector.java:140) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:253) at org.apache.lucene.search.Searcher.search(Searcher.java:171) at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1101) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:880) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:619) java.lang.NumberFormatException: Invalid shift value in prefixCoded string (is encoded value really an INT?) at org.apache.lucene.util.NumericUtils.prefixCodedToInt(NumericUtils.java:233) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:237) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:441) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:208) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:414) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$MultiComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:435) at org.apache.solr.search.DocSetDelegateCollector.setNextReader(DocSetHitCollector.java:140) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:253) at org.apache.lucene.search.Searcher.search(Searcher.java:171) at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1101) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:880) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182
Re: Query: URl too long
Hi there, We had a similar issue. It's an easy fix, simply change the request type from GET to POST. Jon On 12 Jul 2010, at 11:18, Frederico Azeiteiro wrote: Hi, I need to perform a search using a list of values (about 2000). I'm using SolrNET QueryInList function that creates the searchstring like: fieldName: value1 OR fieldName: value2 OR fieldName: value3... (2000 values) This method created a string with about 100 000 chars and the Web Request fails with URI too long (C#). I'm trying to update an old Lucene app that performs this kind of searches. How can I achieve this with Solr? What are my options here? Thank you, Frederico
schema.xml XSD/DTD
Morning all, I was wondering if anyone had written an XSD/DTD for schema.xml? A quick look at the wiki (http://wiki.apache.org/solr/SchemaXml) suggests that this has yet to be done. I'm starting to research how our application could create and configure Solr cores at runtime; and I think schema.xml validation would probably be an essential part of this. Any thoughts or comments? Cheers Jon Poulton
Re: Some indexing requests to Solr fail
Hi there,
Thanks for the reply!
Our backend code is currently set to commit every time it sends over a
batch of documents - so it depends on how big the batch is and how
often edits occur - probably too often. I've looked at the code, and
the SolrJ commit() method takes two parameters - one is called
waitSearcher, and another waitFlush. They aren't really documented too
well, but I assume that the waitSearcher bool (currently set to false)
may be part of the problem.
I am considering removing the code that calls the commit() method
altogether and relying on the settings for DirectUpdateHandler2 to
determine when commits actually get done. That way we can tweak it on
the Solr side without having to recompile and redeploy our main app
(or by having to add new settings and code to handle them to our main
app).
Out of curiosity; how are people doing optimize() calls? Are you doing
them immediately after every commit(), or periodically as part of a job?
Jon
On 31 Mar 2010, at 05:11, Lance Norskog wrote:
How often do you commit? New searchers are only created after a
commit. You notice that handleCommit is in the stack trace :) This
means that commits are happening too often for the amount of other
traffic currently happening, and so it can't finishing creating the
searcher before the next commit starts the next searcher.
The service unavailable messages are roughly the same problem: these
commits might be timing out because the other end is too busy doing
commits. You might try using autocommit instead: commits can happen
every N documents, every T seconds, or both. This keeps the commit
overhead to a controlled amount and commits should stay behind warming
up previous searchers.
On Tue, Mar 30, 2010 at 7:15 AM, Jon Poulton jon.poul...@vyre.com
wrote:
Hi there,
We have a setup in which our main application (running on a
separate Tomcat instance on the same machine) uses SolrJ calls to
an instance of Solr running on the same box. SolrJ is used both for
indexing and searching Solr. Searching seems to be working fine,
but quite frequently we see the following stack trace in our
application logs:
org.apache.solr.common.SolrException: Service Unavailable
Service Unavailable
request: http://localhost:8070/solr/unify/update/javabin
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request
(CommonsHttpSolrServer.java:424)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request
(CommonsHttpSolrServer.java:243)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process
(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:
86)
at vyre.content.rabida.index.RemoteIndexingThread.sendIndexRequest
(RemoteIndexingThread.java:283)
at vyre.content.rabida.index.RemoteIndexingThread.commitBatch
(RemoteIndexingThread.java:195)
at vyre.util.thread.AbstractBatchProcessor.commit
(AbstractBatchProcessor.java:93)
at vyre.util.thread.AbstractBatchProcessor.run
(AbstractBatchProcessor.java:117)
at java.lang.Thread.run(Thread.java:619)
Looking in the Solr logs, there does not appear to be any problems.
The host and port number are correct, its just sometimes our
content gets indexed (visible in the solr logs), and sometimes it
doesn't (nothing visible in solr logs). I'm not sure what could be
causing this problem, but I can hazard a couple of guesses; is
there any upper llimit on the size of a javabin request, or any
point at which the service would decide that the POST was too
large? Has any one else encountered a similar problem?
On a final note, scrolling back through the solr logs does reveal
the following:
29-Mar-2010 17:05:25 org.apache.solr.core.SolrCore getSearcher
WARNING: [unify] Error opening new searcher. exceeded limit of
maxWarmingSearchers=2, try again later.
29-Mar-2010 17:05:25
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {} 0 22
29-Mar-2010 17:05:25 org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Error opening new
searcher. exceeded limit of maxWarmingSearchers=2, try again later.
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:
1029)
at org.apache.solr.update.DirectUpdateHandler2.commit
(DirectUpdateHandler2.java:418)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit
(RunUpdateProcessorFactory.java:85)
at org.apache.solr.handler.RequestHandlerUtils.handleCommit
(RequestHandlerUtils.java:107)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody
(ContentStreamHandlerBase.java:48)
at org.apache.solr.handler.RequestHandlerBase.handleRequest
(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at org.apache.solr.servlet.SolrDispatchFilter.execute
(SolrDispatchFilter.java:338
Some indexing requests to Solr fail
Hi there,
We have a setup in which our main application (running on a separate Tomcat
instance on the same machine) uses SolrJ calls to an instance of Solr running
on the same box. SolrJ is used both for indexing and searching Solr. Searching
seems to be working fine, but quite frequently we see the following stack trace
in our application logs:
org.apache.solr.common.SolrException: Service Unavailable
Service Unavailable
request: http://localhost:8070/solr/unify/update/javabin
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:424)
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:243)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:86)
at
vyre.content.rabida.index.RemoteIndexingThread.sendIndexRequest(RemoteIndexingThread.java:283)
at
vyre.content.rabida.index.RemoteIndexingThread.commitBatch(RemoteIndexingThread.java:195)
at
vyre.util.thread.AbstractBatchProcessor.commit(AbstractBatchProcessor.java:93)
at
vyre.util.thread.AbstractBatchProcessor.run(AbstractBatchProcessor.java:117)
at java.lang.Thread.run(Thread.java:619)
Looking in the Solr logs, there does not appear to be any problems. The host
and port number are correct, its just sometimes our content gets indexed
(visible in the solr logs), and sometimes it doesn't (nothing visible in solr
logs). I'm not sure what could be causing this problem, but I can hazard a
couple of guesses; is there any upper llimit on the size of a javabin request,
or any point at which the service would decide that the POST was too large? Has
any one else encountered a similar problem?
On a final note, scrolling back through the solr logs does reveal the following:
29-Mar-2010 17:05:25 org.apache.solr.core.SolrCore getSearcher
WARNING: [unify] Error opening new searcher. exceeded limit of
maxWarmingSearchers=2, try again later.
29-Mar-2010 17:05:25 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {} 0 22
29-Mar-2010 17:05:25 org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Error opening new searcher.
exceeded limit of maxWarmingSearchers=2, try again later.
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1029)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:48)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:619)
This appears to be an unrelated problem, as the timing is different from the
rejected indexing requests. As there is a large number of concurrent searching
and indexing going on constantly, I'm guessing that I've set the number of
maxWarmingSearches too low, and that as two searchers are warming up, further
indexing causes more searches to be warmed, violating this maximum value - does
this sound like a reasonable conclusion?
Thanks in advance for any help.
Jon
SolJ and query parameters
Hi there,
I'm trying to understand how the query syntax specified on the Solr Wiki (
http://wiki.apache.org/solr/SolrQuerySyntax ) fits in with the usage of the
SolJ class SolrQuery. There are not too many examples of usage to be found.
For example. Say I wanted to replicate the following query using SolrQuery.
q={!lucene q.op=AND df=text}myfield:foo +bar -baz
How would I do it so that q.op was set to OR instead of AND? There is no
method I can see on SolrQuery to set q.op, only a query string, which is
presumably in this case is the text +bar -baz, as the rest can be specified
by calling set methods on SolrQuery.
Thanks in advance for any help.
Jon
RE: SolJ and query parameters
Thanks for the reply.
Using SolrQuery.setQuery({!lucene q.op=AND df=text}myfield:foo +bar -baz});
would make more sense if it were not for the other methods available on
SolrQuery.
For example, there is a setFields(String..) method. So what happens if I call
setFields(title, description) after having set the query to the above
value? What do I end up with? Something like this:
{!lucene q.op=AND df=text}title:(foo +bar -baz) description:(foo +bar baz)}
I'm still having trouble understanding how the class is intended to be used.
Cheers
Jon
-Original Message-
From: Ahmet Arslan [mailto:iori...@yahoo.com]
Sent: 07 January 2010 17:38
To: solr-user@lucene.apache.org
Subject: Re: SolJ and query parameters
--- On Thu, 1/7/10, Jon Poulton jon.poul...@vyre.com wrote:
From: Jon Poulton jon.poul...@vyre.com
Subject: SolJ and query parameters
To: 'solr-user@lucene.apache.org' solr-user@lucene.apache.org
Date: Thursday, January 7, 2010, 7:25 PM
Hi there,
I'm trying to understand how the query syntax specified on
the Solr Wiki ( http://wiki.apache.org/solr/SolrQuerySyntax ) fits in
with the usage of the SolJ class SolrQuery. There are not
too many examples of usage to be found.
For example. Say I wanted to replicate the following query
using SolrQuery.
q={!lucene q.op=AND df=text}myfield:foo +bar -baz
Whole string is the value of the parameter q.
SolrQuery.setQuery({!lucene q.op=AND df=text}myfield:foo +bar -baz);
How would I do it so that q.op was set to OR instead of
AND? There is no method I can see on SolrQuery to set
q.op, only a query string, which is presumably in this
case is the text +bar -baz, as the rest can be specified
by calling set methods on SolrQuery.
if you want to set q.op to AND you can use SolrQuery.set(QueryParsing.OP,AND);
hope this helps.
RE: SolJ and query parameters
I've also just noticed that QueryParsing is not in the SolrJ API. It's in one
of the other Solr jar dependencies.
I'm beginning to think that maybe the best approach it to write a query string
generator which can generate strings of the form:
q={!lucene q.op=AND df=text}myfield:foo +bar -baz
Then just set this on a SolrQuery instance and send it over the wire. It not
the kind of string you'd want an end user to have to type out.
-Original Message-
From: Ahmet Arslan [mailto:iori...@yahoo.com]
Sent: 07 January 2010 17:38
To: solr-user@lucene.apache.org
Subject: Re: SolJ and query parameters
--- On Thu, 1/7/10, Jon Poulton jon.poul...@vyre.com wrote:
From: Jon Poulton jon.poul...@vyre.com
Subject: SolJ and query parameters
To: 'solr-user@lucene.apache.org' solr-user@lucene.apache.org
Date: Thursday, January 7, 2010, 7:25 PM
Hi there,
I'm trying to understand how the query syntax specified on
the Solr Wiki ( http://wiki.apache.org/solr/SolrQuerySyntax ) fits in
with the usage of the SolJ class SolrQuery. There are not
too many examples of usage to be found.
For example. Say I wanted to replicate the following query
using SolrQuery.
q={!lucene q.op=AND df=text}myfield:foo +bar -baz
Whole string is the value of the parameter q.
SolrQuery.setQuery({!lucene q.op=AND df=text}myfield:foo +bar -baz);
How would I do it so that q.op was set to OR instead of
AND? There is no method I can see on SolrQuery to set
q.op, only a query string, which is presumably in this
case is the text +bar -baz, as the rest can be specified
by calling set methods on SolrQuery.
if you want to set q.op to AND you can use SolrQuery.set(QueryParsing.OP,AND);
hope this helps.
SolrCore has a large number of SolrIndexSearchers retained in infoRegistry
Hi there, I'm looking at some problems we are having with some legacy code which uses Solr (1.3) under the hood. We seem to get repeated OutOfMemory errors on a 24-48 hour basis searching a relatively small index of 70,000 documents. This may be an error in the way Solr is configured, or it may be a problem with the way it's being invoked, I don't know, as I'm fairly new to Solr. The OutOfMemoryError is an unusual one: java.lang.OutOfMemoryError: GC overhead limit exceeded There are also a few stack traces in the solr logs. The first is: SEVERE: Error during auto-warming of key:root_show_stop_range:[2009-12-22T08:59:37.254 TO *]:java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOfRange(Arrays.java:3209) at java.lang.String.init(String.java:215) at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122) at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:251) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:218) at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:55) at org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.termDocs(MultiSegmentReader.java:609) at org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.next(MultiSegmentReader.java:560) at org.apache.lucene.search.RangeFilter.getDocIdSet(RangeFilter.java:268) at org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.init(ConstantScoreQuery.java:116) at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:81) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:131) at org.apache.lucene.search.Searcher.search(Searcher.java:126) at org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:601) at org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:507) at org.apache.solr.search.SolrIndexSearcher.cacheDocSet(SolrIndexSearcher.java:482) at org.apache.solr.search.SolrIndexSearcher$1.regenerateItem(SolrIndexSearcher.java:224) at org.apache.solr.search.LRUCache.warm(LRUCache.java:194) at org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1518) at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1018) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) I took a memory dump of the running application, and found that vast amounts of memory (over 400MB) was being consumed by nine or ten large SolrIndexSearcher objects; references to which are being held within a LinkedHashMap in SolrCore called infoRegistry. I had a quick look at the Solr 1.3.0 source code to try and figure out what was going wrong and whether SolrCore was being used incorrectly in our own source. It looks like whenever a new Searcher is created, it registers itself with SolrCore, and this registration places a reference to the Searcher in a LinkedHashMap (the infoRegistry). What is puzzling me is why so many SolrIndexSearcher objects are being created, and what the conditions are for their creation and removal. The code I can see in our own product does not use SolrIndexSearcher directly, it simply makes calls to execute on SolrCore; so I would normally expect that that class would be managing the Searcher life cycle internally. Does anyone have any idea as to what may be going on here? I have had a look at the solrconfig.xml file, but it does not seem to depart significantly from the defaults provided. Thanks in advance for any help. Jon
Concurrent access to EmbeddedSolrServer
Hi there, I'm about to start implementing some code which will access a Solr instance via a ThreadPool concurrently. I've been looking at the solrj API docs ( particularly http://lucene.apache.org/solr/api/index.html?org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.html ) and I just want to make sure what I have in mind makes sense. The Javadoc is a bit sparse, so I thought I'd ask a couple of questions here. 1) I'm assuming that EmbeddedSolrServer can be accessed concurrently by several threads at once for add, delete and query operations (on the SolrServer parent interface). Is that right? I don't have to enforce single-threaded access? 2) What happens if multiple threads simultaneously call commit? 3) What happens if multiple threads simultaneously call optimize? 4) Both commit and optimise have optional parameters called waitFlush and waitSearcher. These are undocumented in the Javadoc. What do they signify? Thanks in advance for any help. Cheers Jon
