low qps with high load averages on solrcloud
Hi Noticed that a solrcloud cluster doesn't scale linearly with # of nodes unlike the unsharded solr cluster. We are seeing a 10 fold drop in QPS in multi sharded mode. While some overhead is understandable, the pattern that we see is, some nodes load averages shoot up and that in turn skews up the overall latencies of the cluster since those nodes become the weaklings in the system. We tried balancing things out by having atleast a shard(collection) for each node but there is still lot of unevenness in load averages across shards. Is there a way to avoid these kind of hotspots? Suchi
updateNumericDocValue in solr 4.6.1
All The following code changes don't seem to really update the docValue in my case. IndexWriter iw = core.getSolrCoreState().getIndexWriter(core).get(); value = Long.parseLong(score); Term term = new Term(ID, id1); iw.updateNumericDocValue(term, 'rank', value); iw.commit() Schema changes: Does any one have a working sample of updateNumericDocValue in solr 4.6.1? Suchi
Exception in unit tests for distributed search component
Hi I am trying to test a custom distributed component with solr 4.6.1 which extends BaseDistributedSearchTestCase but end up with the following error. There are lot of tests in the solr code base which extend BaseDistributedSearchTestCase. Not sure what is wrong here. Suchi testDistribSearch(com.test.DistributedTest) Time elapsed: 2.288 sec <<< ERROR! org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: java.lang.AssertionError at __randomizedtesting.SeedInfo.seed([EB2AD095C59CFFE7:6ACC5E8DB2C39FDB]:0) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102) at org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:436)
Re: fl rename of unique key in solrcloud
Thx Jeon. That worked. Now both the fields are returned in the response.
Its a bit inefficient but works neverthless.
Suchi
On Sat, Nov 15, 2014 at 10:44 PM, Jeon Woosung
wrote:
> I guess that I caused by shard which return renamed field.
>
> following code is source code of solr 4.6
>
> ===
> 986:if ((sreq.purpose & ShardRequest.PURPOSE_GET_FIELDS) != 0) {
> 987: boolean returnScores = (rb.getFieldFlags() &
> SolrIndexSearcher.GET_SCORES) != 0;
> 988:
> 989: assert(sreq.responses.size() == 1);
> 990: ShardResponse srsp = sreq.responses.get(0);
> 991: SolrDocumentList docs =
> (SolrDocumentList)srsp.getSolrResponse().getResponse().get("response");
> 992:
> 993: String keyFieldName =
> rb.req.getSchema().getUniqueKeyField().getName();
> 994: boolean removeKeyField =
> !rb.rsp.getReturnFields().wantsField(keyFieldName);
> 995:
> 996: for (SolrDocument doc : docs) {
> 997:Object id = doc.getFieldValue(keyFieldName);
> 998:ShardDoc sdoc = rb.resultIds.get(id.toString());
>
>
>
> If each shard return renamed field name instead of keyFieldName(UniqueKey),
> "id" of 998 line could be null. Because the doc of 996 line wouldn't have
> "keyFieldName"
>
>
> So if you are urgent or you can not wait for patch, you can add unique
> field like this.
> eg) http://
> /solr//select?q=dress&fl=a1:p1&fl=p1
>
>
>
>
> On Sat, Nov 15, 2014 at 11:26 PM, Garth Grimm <
> garthgr...@averyranchconsulting.com> wrote:
>
> > https://issues.apache.org/jira/browse/SOLR-6744 created.
> >
> > And hopefully correctly, since that’s my first.
> > On Nov 15, 2014, at 9:12 AM, Garth Grimm <
> > garthgr...@averyranchconsulting.com > garthgr...@averyranchconsulting.com>> wrote:
> >
> > I see the same issue on 4.10.1.
> >
> > I’ll open a JIRA if I don’t see one.
> >
> > I guess the best immediate work around is to copy the unique field, and
> > use that field for renaming?
> > On Nov 15, 2014, at 3:18 AM, Suchi Amalapurapu > <mailto:su...@bloomreach.com>> wrote:
> >
> > Solr version:4.6.1
> >
> > On Sat, Nov 15, 2014 at 12:24 PM, Jeon Woosung > <mailto:jeonwoos...@gmail.com>>
> > wrote:
> >
> > Could you let me know version of the solr?
> >
> > On Sat, Nov 15, 2014 at 5:05 AM, Suchi Amalapurapu > <mailto:su...@bloomreach.com>>
> > wrote:
> >
> > Hi
> > Getting the following exception when using fl renaming with unique key in
> > the schema.
> > http:///solr//select?q=dress&fl=a1:p1
> >
> > where p1 is the unique key for
> > For collections with single shard, this works flawlessly but results in
> > the
> > following exception in case of multiple shards.
> >
> > How do we fix this? Stack trace below.
> > Suchi
> >
> > error": {"trace": "java.lang.NullPointerException\n\tat
> >
> >
> >
> >
> org.apache.solr.handler.component.QueryComponent.returnFields(QueryComponent.java:998)\n\tat
> >
> >
> >
> >
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:653)\n\tat
> >
> >
> >
> >
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:628)\n\tat
> >
> >
> >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)\n\tat
> >
> >
> >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\tat
> > org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)\n\tat
> >
> >
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)\n\tat
> >
> >
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)\n\tat
> >
> >
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)\n\tat
> >
> >
> >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)\n\tat
> >
> >
> >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)\n\tat
> >
> >
> >
> >
> org.eclipse.jetty.server
Re: fl rename of unique key in solrcloud
Solr version:4.6.1
On Sat, Nov 15, 2014 at 12:24 PM, Jeon Woosung
wrote:
> Could you let me know version of the solr?
>
> On Sat, Nov 15, 2014 at 5:05 AM, Suchi Amalapurapu
> wrote:
>
> > Hi
> > Getting the following exception when using fl renaming with unique key in
> > the schema.
> > http:///solr//select?q=dress&fl=a1:p1
> >
> > where p1 is the unique key for
> > For collections with single shard, this works flawlessly but results in
> the
> > following exception in case of multiple shards.
> >
> > How do we fix this? Stack trace below.
> > Suchi
> >
> > error": {"trace": "java.lang.NullPointerException\n\tat
> >
> >
> org.apache.solr.handler.component.QueryComponent.returnFields(QueryComponent.java:998)\n\tat
> >
> >
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:653)\n\tat
> >
> >
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:628)\n\tat
> >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)\n\tat
> >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\tat
> > org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)\n\tat
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)\n\tat
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)\n\tat
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)\n\tat
> >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)\n\tat
> >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)\n\tat
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)\n\tat
> >
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)\n\tat
> >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\n\tat
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)\n\tat
> >
> >
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)\n\tat
> >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)\n\tat
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)\n\tat
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)\n\tat
> >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)\n\tat
> >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)\n\tat
> >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)\n\tat
> > org.eclipse.jetty.server.Server.handle(Server.java:368)\n\tat
> >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)\n\tat
> >
> >
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)\n\tat
> >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)\n\tat
> >
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)\n\tat
> > org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)\n\tat
> >
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)\n\tat
> >
> >
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)\n\tat
> >
> >
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)\n\tat
> >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)\n\tat
> >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)\n\tat
> > java.lang.Thread.run(Thread.java:662)\n","code": 500
> >
>
>
>
> --
> *God bless U*
>
fl rename of unique key in solrcloud
Hi
Getting the following exception when using fl renaming with unique key in
the schema.
http:///solr//select?q=dress&fl=a1:p1
where p1 is the unique key for
For collections with single shard, this works flawlessly but results in the
following exception in case of multiple shards.
How do we fix this? Stack trace below.
Suchi
error": {"trace": "java.lang.NullPointerException\n\tat
org.apache.solr.handler.component.QueryComponent.returnFields(QueryComponent.java:998)\n\tat
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:653)\n\tat
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:628)\n\tat
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\tat
org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:368)\n\tat
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)\n\tat
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)\n\tat
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)\n\tat
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)\n\tat
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)\n\tat
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)\n\tat
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)\n\tat
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)\n\tat
java.lang.Thread.run(Thread.java:662)\n","code": 500
solrcore.properties in solrcloud
Hi Can some one clarify the role of solrcore.properties in Solr 4.3+? We can define custom properties in solrcore.properties and use them in solrconfig.xml Will this functionality be retained going forward? Suchi
Re: Query time boosts vs static boost function defined in solrconfig.xml
I have a boost function defined in solrconfig.xml which is used to rank
results and need additional reranking based on some weights passed in the
url.
The function using weights passed in the url is parsed into local params
and overrides the function defined in the solrconfig.xml
Ideally I want to use both.
I tried extending edismax parser but wasn't able to over ride this part.
Wasn't able to inject these boost functions in Solrparams either.
Suchi
On Sun, Feb 17, 2013 at 1:13 AM, Jack Krupansky wrote:
> Could you give a more specific example of what you are trying to
> accomplish. I mean, what issue you are trying to address.
>
> -- Jack Krupansky
>
> -Original Message- From: Suchi Amalapurapu
> Sent: Friday, February 15, 2013 2:21 AM
> To: solr-user@lucene.apache.org
>
> Subject: Query time boosts vs static boost function defined in
> solrconfig.xml
>
> Query time boost function seem to be loaded via local params while boost
> functions defined in solrconfig.xml get added to a request globally.
> QParser.geParams
> public String getParam(String name) {
>String val;
>if (localParams != null) {
> val = localParams.get(name);
> if (val != null) return val;
>}
>return params.get(name);
> }
> indicates we can use only one of them with local params overriding the
> global one's.
>
> Is there a way to use both? The use case is that I want to apply
> parameterized boosts on top of boost functions in solrconfig.xml
>
> To simulate this functionality I can either
> - Define a custom QParser.
> - Define a custom RequestHandler/SearchComponent
> In both cases I can inject local params into request's params and further
> do string replace's for some args.
> Is there a better way of doing things?
>
> Also are static boost methods in solrconfig.xml loaded only once vs
> frequent parsing of queries in query time boosts?
> Suchi
>
Query time boosts vs static boost function defined in solrconfig.xml
Query time boost function seem to be loaded via local params while boost
functions defined in solrconfig.xml get added to a request globally.
QParser.geParams
public String getParam(String name) {
String val;
if (localParams != null) {
val = localParams.get(name);
if (val != null) return val;
}
return params.get(name);
}
indicates we can use only one of them with local params overriding the
global one's.
Is there a way to use both? The use case is that I want to apply
parameterized boosts on top of boost functions in solrconfig.xml
To simulate this functionality I can either
- Define a custom QParser.
- Define a custom RequestHandler/SearchComponent
In both cases I can inject local params into request's params and further
do string replace's for some args.
Is there a better way of doing things?
Also are static boost methods in solrconfig.xml loaded only once vs
frequent parsing of queries in query time boosts?
Suchi
