Hi Mikhall, Actually I have topK=1200, reRankDocs=1000000 and reRankWeight=1000 - so MUCH bigger numbers.
I've found by trial and error that I have to push topK up much higher than the amount of results (rows) I really want (which is 60 in this case) - if I use a lower topK (like 60 or even 600) then sometimes the reranking will stop before the rows (for example I might get 40 reranked documents in the 60 and the last 20 look like natural order and not reranked). Derek On Thu, Aug 25, 2022 at 7:32 AM Mikhail Khludnev <[email protected]> wrote: > have you specified topK=10 reRankDocs=4? > > On Wed, Aug 24, 2022 at 10:20 AM Derek C <[email protected]> wrote: > > > I actually found an example in the logs ! > > > > 2022-08-23 19:45:28.273 ERROR (qtp837073696-15) [c:getlocal_live_products > > s:shard1 r:core_node8 x:getlocal_live_products_shard1_replica_n7] > > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: > > java.lang.IllegalArgumentException: maxSize must be > 0 and < 2147483630; > > got: 0 => org.apache.solr.common.SolrException: > > java.lang.IllegalArgumentException: maxSize must be > 0 and < 2147483630; > > got: 0 > > at > > org.apache.solr.search.ReRankCollector.topDocs(ReRankCollector.java:163) > > org.apache.solr.common.SolrException: java.lang.IllegalArgumentException: > > maxSize must be > 0 and < 2147483630; got: 0 > > at > > org.apache.solr.search.ReRankCollector.topDocs(ReRankCollector.java:163) > > ~[?:?] > > at > > > > > org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1696) > > ~[?:?] > > at > > > > > org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1510) > > ~[?:?] > > at > > > org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:681) > > ~[?:?] > > at > > > > > org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1618) > > ~[?:?] > > at > > > > > org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:425) > > ~[?:?] > > at > > > > > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:420) > > ~[?:?] > > at > > > > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:207) > > ~[?:?] > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2866) > ~[?:?] > > at > > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:887) > ~[?:?] > > at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:606) > > ~[?:?] > > at > > > > > org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:239) > > ~[?:?] > > at > > > > > org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:207) > > ~[?:?] > > at > > > > > org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257) > > ~[?:?] > > at > > > > > org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227) > > ~[?:?] > > at > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202) > > ~[?:?] > > at > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184) > > ~[?:?] > > at > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) > > ~[jetty-servlet-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) > > ~[jetty-servlet-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) > > ~[jetty-servlet-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) > > ~[jetty-security-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) > > ~[jetty-servlet-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at org.eclipse.jetty.server.Server.handle(Server.java:516) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) > > ~[jetty-server-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > org.eclipse.jetty.io > > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) > > ~[jetty-io-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) > > ~[jetty-io-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) > > ~[jetty-io-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) > > ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) > > ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > > ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) > > ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) > > ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > > ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] > > at > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > > ~[jetty-util-9.4.44.v20210927.jar:9.4.44.v20210927] > > at java.lang.Thread.run(Thread.java:829) [?:?] > > Caused by: java.lang.IllegalArgumentException: maxSize must be > 0 and < > > 2147483630; got: 0 > > at org.apache.lucene.util.LongHeap.<init>(LongHeap.java:57) > ~[?:?] > > at org.apache.lucene.util.LongHeap$1.<init>(LongHeap.java:69) > > ~[?:?] > > at org.apache.lucene.util.LongHeap.create(LongHeap.java:69) > ~[?:?] > > at > > org.apache.lucene.util.hnsw.NeighborQueue.<init>(NeighborQueue.java:41) > > ~[?:?] > > at > org.apache.lucene.util.hnsw.HnswGraph.search(HnswGraph.java:105) > > ~[?:?] > > at > > > > > org.apache.lucene.codecs.lucene90.Lucene90HnswVectorsReader.search(Lucene90HnswVectorsReader.java:250) > > ~[?:?] > > at > > > > > org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsReader.search(PerFieldKnnVectorsFormat.java:241) > > ~[?:?] > > at > > > > > org.apache.lucene.index.CodecReader.searchNearestVectors(CodecReader.java:233) > > ~[?:?] > > at > > > > > org.apache.lucene.index.FilterLeafReader.searchNearestVectors(FilterLeafReader.java:350) > > ~[?:?] > > at > > > > > org.apache.lucene.index.FilterLeafReader.searchNearestVectors(FilterLeafReader.java:350) > > ~[?:?] > > at > > > org.apache.lucene.search.KnnVectorQuery.searchLeaf(KnnVectorQuery.java:75) > > ~[?:?] > > at > > org.apache.lucene.search.KnnVectorQuery.rewrite(KnnVectorQuery.java:63) > > ~[?:?] > > at > > org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:775) > > ~[?:?] > > at > > org.apache.lucene.search.QueryRescorer.rescore(QueryRescorer.java:64) > > ~[?:?] > > at > > org.apache.solr.search.ReRankCollector.topDocs(ReRankCollector.java:118) > > ~[?:?] > > ... 55 more > > > > (It's truncated there after the "55 more") > > > > > > > > On Wed, Aug 24, 2022 at 2:01 AM Joel Bernstein <[email protected]> > wrote: > > > > > Can you post the stack trace from the logs? > > > > > > > > > Joel Bernstein > > > http://joelsolr.blogspot.com/ > > > > > > > > > On Tue, Aug 23, 2022 at 7:01 PM Derek C <[email protected]> wrote: > > > > > > > Hi all, > > > > > > > > We have 3 X SOLR instances running SOLR cloud and I'm doing KNN > > > > searches and KNN reranking queries across a couple of million > > documents. > > > > > > > > Every now and then the KNN searches will all fail with the error > > > > "java.lang.IllegalArgumentException: maxSize must be > 0 and < > > > 2147483630; > > > > got: 0". > > > > > > > > After a few minutes the problem goes away and the searches work > again. > > > > > > > > Does anyone know what might be causing this? > > > > > > > > thanks v much > > > > > > > > Derek > > > > > > > > -- > > > > -- > > > > Derek Conniffe > > > > Harvey Software Systems Ltd T/A HSSL > > > > Telephone (IRL): 086 856 3823 > > > > Telephone (US): (650) 443 8285 > > > > Skype: dconnrt > > > > Email: [email protected] > > > > > > > > > > > > *Disclaimer:* This email and any files transmitted with it are > > > confidential > > > > and intended solely for the use of the individual or entity to whom > > they > > > > are addressed. If you have received this email in error please delete > > it > > > > (if you are not the intended recipient you are notified that > > disclosing, > > > > copying, distributing or taking any action in reliance on the > contents > > of > > > > this information is strictly prohibited). > > > > *Warning*: Although HSSL have taken reasonable precautions to ensure > no > > > > viruses are present in this email, HSSL cannot accept responsibility > > for > > > > any loss or damage arising from the use of this email or attachments. > > > > P For the Environment, please only print this email if necessary. > > > > > > > > > > > > > -- > > -- > > Derek Conniffe > > Harvey Software Systems Ltd T/A HSSL > > Telephone (IRL): 086 856 3823 > > Telephone (US): (650) 443 8285 > > Skype: dconnrt > > Email: [email protected] > > > > > > *Disclaimer:* This email and any files transmitted with it are > confidential > > and intended solely for the use of the individual or entity to whom they > > are addressed. If you have received this email in error please delete it > > (if you are not the intended recipient you are notified that disclosing, > > copying, distributing or taking any action in reliance on the contents of > > this information is strictly prohibited). > > *Warning*: Although HSSL have taken reasonable precautions to ensure no > > viruses are present in this email, HSSL cannot accept responsibility for > > any loss or damage arising from the use of this email or attachments. > > P For the Environment, please only print this email if necessary. > > > > > -- > Sincerely yours > Mikhail Khludnev > -- -- Derek Conniffe Harvey Software Systems Ltd T/A HSSL Telephone (IRL): 086 856 3823 Telephone (US): (650) 443 8285 Skype: dconnrt Email: [email protected] *Disclaimer:* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please delete it (if you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited). *Warning*: Although HSSL have taken reasonable precautions to ensure no viruses are present in this email, HSSL cannot accept responsibility for any loss or damage arising from the use of this email or attachments. P For the Environment, please only print this email if necessary.
