Re: Issue with field collapsing in solr 4 while performing distributed search
Martijn, How do we add a custom algorithm for distributing documents in Solr Cloud? According to this discussion http://lucene.472066.n3.nabble.com/SolrCloud-how-to-index-documents-into-a-specific-core-and-how-to-search-against-that-core-td3985262.html , Mark discourages users from using custom distribution mechanism in Solr Cloud. Load balancing is not an issue for us at the moment. In that case, how should we implement a custom partitioning algorithm. On Mon, Jun 11, 2012 at 6:23 PM, Martijn v Groningen < martijn.v.gronin...@gmail.com> wrote: > The ngroups returns the number of groups that have matched with the > query. However if you want ngroups to be correct in a distributed > environment you need > to put document belonging to the same group into the same shard. > Groups can't cross shard boundaries. I guess you need to do > some manual document partitioning. > > Martijn > > On 11 June 2012 14:29, Nitesh Nandy wrote: > > Version: Solr 4.0 (svn build 30th may, 2012) with Solr Cloud (2 slices > and > > 2 shards) > > > > The setup was done as per the wiki: > http://wiki.apache.org/solr/SolrCloud > > > > We are doing distributed search. While querying, we use field collapsing > > with "ngroups" set as true as we need the number of search results. > > > > However, there is a difference in the number of "result list" returned > and > > the "ngroups" value returned. > > > > Ex: > > > http://localhost:8983/solr/select?q=message:blah%20AND%20userid:3&&group=true&group.field=id&group.ngroups=true > > > > > > The response XMl looks like > > > > > > > > > > 0 > > 46 > > > > id > > true > > true > > messagebody:monit AND usergroupid:3 > > > > > > > > > > 10 > > 9 > > > > > > 320043 > > > > ... > > > > > > > > 398807 > > ... > > > > > > > > 346878 > > ... > > > > > > 346880 > > ... > > > > > > > > > > > > > > So you can see that the ngroups value returned is 9 and the actual number > > of groups returned is 4 > > > > Why do we have this discrepancy in the ngroups, matches and actual number > > of groups. Is this an open issue ? > > > > Any kind of help is appreciated. > > > > -- > > Regards, > > > > Nitesh Nandy > > > > -- > Met vriendelijke groet, > > Martijn van Groningen > -- Regards, Nitesh Nandy
Issue with field collapsing in solr 4 while performing distributed search
Version: Solr 4.0 (svn build 30th may, 2012) with Solr Cloud (2 slices and 2 shards) The setup was done as per the wiki: http://wiki.apache.org/solr/SolrCloud We are doing distributed search. While querying, we use field collapsing with "ngroups" set as true as we need the number of search results. However, there is a difference in the number of "result list" returned and the "ngroups" value returned. Ex: http://localhost:8983/solr/select?q=message:blah%20AND%20userid:3&&group=true&group.field=id&group.ngroups=true The response XMl looks like 0 46 id true true messagebody:monit AND usergroupid:3 10 9 320043 ... 398807 ... 346878 ... 346880 ... So you can see that the ngroups value returned is 9 and the actual number of groups returned is 4 Why do we have this discrepancy in the ngroups, matches and actual number of groups. Is this an open issue ? Any kind of help is appreciated. -- Regards, Nitesh Nandy
Issue with Solrcloud /solr 4.0 : Discrepancy in number of groups and ngroups value
We are using Solr 4.0 (svn build 30th may, 2012) with Solr Cloud. While querying, we use field collpasing with ngroups set to true. However, there is a difference in the number of results got and the "ngroups" value returned. Ex: http://localhost:8983/solr/select?q=messagebody:monit%20AND%20usergroupid:3&&group=true&group.field=id&facet.limit=20&group.ngroups=true The values returned are like 10 9 Actual groups returned :4 Why do we have this discrepancy in the ngroups, matches and actual number of groups. Earlier we were using the same query with solr 3.5 (without solr cloud) and it was giving correct results. Any kind of help is appreciated. -- Regards, Nitesh Nandy
LockObtainedFailedException on solr update
My solr index is updated simultaneously by multiple clients via REST. I use "commitWithing" attribute in the command to direct auto commits. I start getting this error after a couple of days of usage. How do i fix this ? Please find the error log below. Using solr 3.1 with tomcat Thanks -- HTTP Status 500 - Lock obtain timed out: NativeFSLock@ /var/lib/solr/data/index/write.lock org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/lib/solr/data/index/write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1097) at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:83) at org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:102) at org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:174) at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:222) at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61) at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:147) at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:55) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:662) -- Regards, Nitesh Nandy
OverlappingFileLockException when concurrent commits in solr
talina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:636) description The server encountered an internal error (null java.nio.channels.OverlappingFileLockException at sun.nio.ch.FileChannelImpl$SharedFileLockTable.checkList(FileChannelImpl.java:1215) at sun.nio.ch.FileChannelImpl$SharedFileLockTable.add(FileChannelImpl.java:1117) at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:923) at java.nio.channels.FileChannel.tryLock(FileChannel.java:978) at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:233) at org.apache.lucene.store.Lock.obtain(Lock.java:73) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1550) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1407) at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:190) at org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:98) at org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173) at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:220) at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61) at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139) at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1317) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:636) ) that prevented it from fulfilling this request. -- Regards, Nitesh Nandy
OverlappingFileLockException when concurrent commits in solr
talina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:636) description The server encountered an internal error (null java.nio.channels.OverlappingFileLockException at sun.nio.ch.FileChannelImpl$SharedFileLockTable.checkList(FileChannelImpl.java:1215) at sun.nio.ch.FileChannelImpl$SharedFileLockTable.add(FileChannelImpl.java:1117) at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:923) at java.nio.channels.FileChannel.tryLock(FileChannel.java:978) at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:233) at org.apache.lucene.store.Lock.obtain(Lock.java:73) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1550) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1407) at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:190) at org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:98) at org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173) at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:220) at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61) at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139) at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1317) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:636) ) that prevented it from fulfilling this request. -- Regards, Nitesh Nandy
