Thank you for the detailed response Eric. Very much appreciated. The reason I am looking into splitting the index into two is because it’s much faster to search across a smaller index than a larger one.
On Wed, Jun 20, 2018 at 10:46 AM Erick Erickson <erickerick...@gmail.com> wrote: > You still haven't answered _why_ you think splitting even a 20G index > is desirable. We regularly see 200G+ indexes per replica in the field, > so what's the point? Have you measured different setups to see if it's > a good idea? A 200G index needs some beefy hardware admittedly.... > > If you have adequate response times with a 20G index and need to > increase the QPS rate, just add more replicas. Having more than one > shard inevitably adds overhead which may (or may not) be made up for > by parallelizing some of the work. It's nearly always better to use > only one shard if it meets your response time requirements. > > Best, > Erick > > On Wed, Jun 20, 2018 at 10:39 AM, Sushant Vengurlekar > <svengurle...@curvolabs.com> wrote: > > The index size is small because this is my local development copy. The > > production index is more than 20GB. So I am working on getting the index > > split and replicated on different nodes. Our current instance on prod is > > single instance solr 6 which we are working on moving towards solrcloud 7 > > > > On Wed, Jun 20, 2018 at 10:30 AM Erick Erickson <erickerick...@gmail.com > > > > wrote: > > > >> Use the indexupgrader tool or optimize your index before using > splitshard. > >> > >> Since this is a small index (< 5G), optimizing will not create an > >> overly-large segment, so that pitfall is avoided. > >> > >> You haven't yet explained why you think splitting the index would be > >> beneficial. Splitting an index this small is unlikely to improve query > >> performance appreciably. This feels a lot like an "XY" problem, you're > >> asking how to do X thinking it will solve Y but not telling us what Y > >> is. > >> > >> Best, > >> Erick > >> > >> On Wed, Jun 20, 2018 at 9:40 AM, Sushant Vengurlekar > >> <svengurle...@curvolabs.com> wrote: > >> > How can I resolve this error? > >> > > >> > On Wed, Jun 20, 2018 at 9:11 AM, Alexandre Rafalovitch < > >> arafa...@gmail.com> > >> > wrote: > >> > > >> >> This seems more related to an old index upgraded to latest Solr > rather > >> than > >> >> the split itself. > >> >> > >> >> Regards, > >> >> Alex > >> >> > >> >> On Wed, Jun 20, 2018, 12:07 PM Sushant Vengurlekar, < > >> >> svengurle...@curvolabs.com> wrote: > >> >> > >> >> > Thanks for the reply Alessandro! Appreciate it. > >> >> > > >> >> > Below is the full request and the error received > >> >> > > >> >> > curl ' > >> >> > > >> >> > http://localhost:8081/solr/admin/collections?action= > >> >> SPLITSHARD&collection=dev-transactions&shard=shard1 > >> >> > ' > >> >> > > >> >> > { > >> >> > > >> >> > "responseHeader":{ > >> >> > > >> >> > "status":500, > >> >> > > >> >> > "QTime":7920}, > >> >> > > >> >> > "success":{ > >> >> > > >> >> > "solr-1:8081_solr":{ > >> >> > > >> >> > "responseHeader":{ > >> >> > > >> >> > "status":0, > >> >> > > >> >> > "QTime":1190}, > >> >> > > >> >> > "core":"dev-transactions_shard1_0_replica_n3"}, > >> >> > > >> >> > "solr-1:8081_solr":{ > >> >> > > >> >> > "responseHeader":{ > >> >> > > >> >> > "status":0, > >> >> > > >> >> > "QTime":1047}, > >> >> > > >> >> > "core":"dev-transactions_shard1_1_replica_n4"}, > >> >> > > >> >> > "solr-1:8081_solr":{ > >> >> > > >> >> > "responseHeader":{ > >> >> > > >> >> > "status":0, > >> >> > > >> >> > "QTime":6}}, > >> >> > > >> >> > "solr-1:8081_solr":{ > >> >> > > >> >> > "responseHeader":{ > >> >> > > >> >> > "status":0, > >> >> > > >> >> > "QTime":1009}}}, > >> >> > > >> >> > "failure":{ > >> >> > > >> >> > > >> >> > > "solr-1:8081_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$ > >> >> RemoteSolrException:Error > >> >> > from server at http://solr-1:8081/solr: > >> >> > java.lang.IllegalArgumentException: > >> >> > Cannot merge a segment that has been created with major version 6 > into > >> >> this > >> >> > index which has been created by major version 7"}, > >> >> > > >> >> > "Operation splitshard caused > >> >> > > >> >> > exception:":"org.apache.solr.common.SolrException:org. > >> >> apache.solr.common.SolrException: > >> >> > SPLITSHARD failed to invoke SPLIT core admin command", > >> >> > > >> >> > "exception":{ > >> >> > > >> >> > "msg":"SPLITSHARD failed to invoke SPLIT core admin command", > >> >> > > >> >> > "rspCode":500}, > >> >> > > >> >> > "error":{ > >> >> > > >> >> > "metadata":[ > >> >> > > >> >> > "error-class","org.apache.solr.common.SolrException", > >> >> > > >> >> > "root-error-class","org.apache.solr.common.SolrException"], > >> >> > > >> >> > "msg":"SPLITSHARD failed to invoke SPLIT core admin command", > >> >> > > >> >> > "trace":"org.apache.solr.common.SolrException: SPLITSHARD > failed > >> to > >> >> > invoke SPLIT core admin command\n\tat > >> >> > > >> >> > org.apache.solr.client.solrj.SolrResponse.getException( > >> >> SolrResponse.java:53)\n\tat > >> >> > > >> >> > org.apache.solr.handler.admin.CollectionsHandler.invokeAction( > >> >> CollectionsHandler.java:258)\n\tat > >> >> > > >> >> > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody( > >> >> CollectionsHandler.java:230)\n\tat > >> >> > > >> >> > org.apache.solr.handler.RequestHandlerBase.handleRequest( > >> >> RequestHandlerBase.java:195)\n\tat > >> >> > > >> >> > org.apache.solr.servlet.HttpSolrCall.handleAdmin( > >> >> HttpSolrCall.java:736)\n\tat > >> >> > > >> >> > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest( > >> >> HttpSolrCall.java:717)\n\tat > >> >> > > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)\n\tat > >> >> > > >> >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter( > >> >> SolrDispatchFilter.java:384)\n\tat > >> >> > > >> >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter( > >> >> SolrDispatchFilter.java:330)\n\tat > >> >> > > >> >> > org.eclipse.jetty.servlet.ServletHandler$CachedChain. > >> >> doFilter(ServletHandler.java:1629)\n\tat > >> >> > > >> >> > org.eclipse.jetty.servlet.ServletHandler.doHandle( > >> >> ServletHandler.java:533)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.ScopedHandler.handle( > >> >> ScopedHandler.java:143)\n\tat > >> >> > > >> >> > org.eclipse.jetty.security.SecurityHandler.handle( > >> >> SecurityHandler.java:548)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle( > >> >> HandlerWrapper.java:132)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.ScopedHandler. > >> >> nextHandle(ScopedHandler.java:190)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.session.SessionHandler. > >> >> doHandle(SessionHandler.java:1595)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.ScopedHandler. > >> >> nextHandle(ScopedHandler.java:188)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.ContextHandler. > >> >> doHandle(ContextHandler.java:1253)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.ScopedHandler. > >> >> nextScope(ScopedHandler.java:168)\n\tat > >> >> > > >> >> > org.eclipse.jetty.servlet.ServletHandler.doScope( > >> >> ServletHandler.java:473)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.session.SessionHandler. > >> >> doScope(SessionHandler.java:1564)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.ScopedHandler. > >> >> nextScope(ScopedHandler.java:166)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.ContextHandler. > >> >> doScope(ContextHandler.java:1155)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.ScopedHandler.handle( > >> >> ScopedHandler.java:141)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle( > >> >> ContextHandlerCollection.java:219)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.HandlerCollection. > >> >> handle(HandlerCollection.java:126)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle( > >> >> HandlerWrapper.java:132)\n\tat > >> >> > > >> >> > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle( > >> >> RewriteHandler.java:335)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle( > >> >> HandlerWrapper.java:132)\n\tat > >> >> > org.eclipse.jetty.server.Server.handle(Server.java:530)\n\tat > >> >> > > >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)\n\tat > >> >> > > >> >> > org.eclipse.jetty.server.HttpConnection.onFillable( > >> >> HttpConnection.java:256)\n\tat > >> >> > org.eclipse.jetty.io > >> >> > > >> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)\ > >> >> n\tat > >> >> > org.eclipse.jetty.io > >> .FillInterest.fillable(FillInterest.java:102)\n\tat > >> >> > org.eclipse.jetty.io > >> .ChannelEndPoint$2.run(ChannelEndPoint.java:124)\n\ > >> >> tat > >> >> > > >> >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill. > >> >> doProduce(EatWhatYouKill.java:247)\n\tat > >> >> > > >> >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill. > >> >> produce(EatWhatYouKill.java:140)\n\tat > >> >> > > >> >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run( > >> >> EatWhatYouKill.java:131)\n\tat > >> >> > > >> >> > > >> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run( > >> >> ReservedThreadExecutor.java:382)\n\tat > >> >> > > >> >> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > >> >> QueuedThreadPool.java:708)\n\tat > >> >> > > >> >> > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( > >> >> QueuedThreadPool.java:626)\n\tat > >> >> > java.lang.Thread.run(Thread.java:748)\n", > >> >> > > >> >> > "code":500}} > >> >> > > >> >> > > >> >> > On Wed, Jun 20, 2018 at 2:25 AM, Alessandro Benedetti < > >> >> > a.benede...@sease.io> > >> >> > wrote: > >> >> > > >> >> > > Hi, > >> >> > > in the first place, why do you want to split 2 Gb indexes ? > >> >> > > Nowadays is a fairly small index. > >> >> > > > >> >> > > Secondly what you reported is incomplete. > >> >> > > I would expect a Caused By section in the stacktrace. > >> >> > > > >> >> > > This are generic recommendations, always spend time in analysing > the > >> >> > > problem > >> >> > > you had scrupulously. > >> >> > > - SolrCloud problems often involve more than one node. Be sure to > >> check > >> >> > the > >> >> > > logs of all the nodes possibly involved. > >> >> > > - Report the full stack trace to the community > >> >> > > - Report your full request which provoked the exception > >> >> > > > >> >> > > Help is much easier this way :) > >> >> > > > >> >> > > Regards > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > ----- > >> >> > > --------------- > >> >> > > Alessandro Benedetti > >> >> > > Search Consultant, R&D Software Engineer, Director > >> >> > > Sease Ltd. - www.sease.io > >> >> > > -- > >> >> > > Sent from: > >> http://lucene.472066.n3.nabble.com/Solr-User-f472068.html > >> >> > > > >> >> > > >> >> > >> >
