low qps with high load averages on solrcloud
Hi Noticed that a solrcloud cluster doesn't scale linearly with # of nodes unlike the unsharded solr cluster. We are seeing a 10 fold drop in QPS in multi sharded mode. While some overhead is understandable, the pattern that we see is, some nodes load averages shoot up and that in turn skews up the overall latencies of the cluster since those nodes become the weaklings in the system. We tried balancing things out by having atleast a shard(collection) for each node but there is still lot of unevenness in load averages across shards. Is there a way to avoid these kind of hotspots? Suchi
Exception in unit tests for distributed search component
Hi I am trying to test a custom distributed component with solr 4.6.1 which extends BaseDistributedSearchTestCase but end up with the following error. There are lot of tests in the solr code base which extend BaseDistributedSearchTestCase. Not sure what is wrong here. Suchi testDistribSearch(com.test.DistributedTest) Time elapsed: 2.288 sec ERROR! org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: java.lang.AssertionError at __randomizedtesting.SeedInfo.seed([EB2AD095C59CFFE7:6ACC5E8DB2C39FDB]:0) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102) at org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:436)
updateNumericDocValue in solr 4.6.1
All The following code changes don't seem to really update the docValue in my case. IndexWriter iw = core.getSolrCoreState().getIndexWriter(core).get(); value = Long.parseLong(score); Term term = new Term(ID, id1); iw.updateNumericDocValue(term, 'rank', value); iw.commit() Schema changes: field name=rank type=long indexed=true stored=true required=false docValues=true/ Does any one have a working sample of updateNumericDocValue in solr 4.6.1? Suchi
Re: fl rename of unique key in solrcloud
Thx Jeon. That worked. Now both the fields are returned in the response.
Its a bit inefficient but works neverthless.
Suchi
On Sat, Nov 15, 2014 at 10:44 PM, Jeon Woosung jeonwoos...@gmail.com
wrote:
I guess that I caused by shard which return renamed field.
following code is source code of solr 4.6
===
986:if ((sreq.purpose ShardRequest.PURPOSE_GET_FIELDS) != 0) {
987: boolean returnScores = (rb.getFieldFlags()
SolrIndexSearcher.GET_SCORES) != 0;
988:
989: assert(sreq.responses.size() == 1);
990: ShardResponse srsp = sreq.responses.get(0);
991: SolrDocumentList docs =
(SolrDocumentList)srsp.getSolrResponse().getResponse().get(response);
992:
993: String keyFieldName =
rb.req.getSchema().getUniqueKeyField().getName();
994: boolean removeKeyField =
!rb.rsp.getReturnFields().wantsField(keyFieldName);
995:
996: for (SolrDocument doc : docs) {
997:Object id = doc.getFieldValue(keyFieldName);
998:ShardDoc sdoc = rb.resultIds.get(id.toString());
If each shard return renamed field name instead of keyFieldName(UniqueKey),
id of 998 line could be null. Because the doc of 996 line wouldn't have
keyFieldName
So if you are urgent or you can not wait for patch, you can add unique
field like this.
eg) http://
host_name/solr/collection_name/select?q=dressfl=a1:p1fl=p1
On Sat, Nov 15, 2014 at 11:26 PM, Garth Grimm
garthgr...@averyranchconsulting.com wrote:
https://issues.apache.org/jira/browse/SOLR-6744 created.
And hopefully correctly, since that’s my first.
On Nov 15, 2014, at 9:12 AM, Garth Grimm
garthgr...@averyranchconsulting.commailto:
garthgr...@averyranchconsulting.com wrote:
I see the same issue on 4.10.1.
I’ll open a JIRA if I don’t see one.
I guess the best immediate work around is to copy the unique field, and
use that field for renaming?
On Nov 15, 2014, at 3:18 AM, Suchi Amalapurapu su...@bloomreach.com
mailto:su...@bloomreach.com wrote:
Solr version:4.6.1
On Sat, Nov 15, 2014 at 12:24 PM, Jeon Woosung jeonwoos...@gmail.com
mailto:jeonwoos...@gmail.com
wrote:
Could you let me know version of the solr?
On Sat, Nov 15, 2014 at 5:05 AM, Suchi Amalapurapu su...@bloomreach.com
mailto:su...@bloomreach.com
wrote:
Hi
Getting the following exception when using fl renaming with unique key in
the schema.
http://host_name/solr/collection_name/select?q=dressfl=a1:p1
where p1 is the unique key for collection_name
For collections with single shard, this works flawlessly but results in
the
following exception in case of multiple shards.
How do we fix this? Stack trace below.
Suchi
error: {trace: java.lang.NullPointerException\n\tat
org.apache.solr.handler.component.QueryComponent.returnFields(QueryComponent.java:998)\n\tat
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:653)\n\tat
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:628)\n\tat
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\tat
org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)\n\tat
Re: fl rename of unique key in solrcloud
Solr version:4.6.1
On Sat, Nov 15, 2014 at 12:24 PM, Jeon Woosung jeonwoos...@gmail.com
wrote:
Could you let me know version of the solr?
On Sat, Nov 15, 2014 at 5:05 AM, Suchi Amalapurapu su...@bloomreach.com
wrote:
Hi
Getting the following exception when using fl renaming with unique key in
the schema.
http://host_name/solr/collection_name/select?q=dressfl=a1:p1
where p1 is the unique key for collection_name
For collections with single shard, this works flawlessly but results in
the
following exception in case of multiple shards.
How do we fix this? Stack trace below.
Suchi
error: {trace: java.lang.NullPointerException\n\tat
org.apache.solr.handler.component.QueryComponent.returnFields(QueryComponent.java:998)\n\tat
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:653)\n\tat
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:628)\n\tat
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\tat
org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:368)\n\tat
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)\n\tat
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)\n\tat
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)\n\tat
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)\n\tat
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)\n\tat
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)\n\tat
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)\n\tat
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)\n\tat
java.lang.Thread.run(Thread.java:662)\n,code: 500
--
*God bless U*
fl rename of unique key in solrcloud
Hi
Getting the following exception when using fl renaming with unique key in
the schema.
http://host_name/solr/collection_name/select?q=dressfl=a1:p1
where p1 is the unique key for collection_name
For collections with single shard, this works flawlessly but results in the
following exception in case of multiple shards.
How do we fix this? Stack trace below.
Suchi
error: {trace: java.lang.NullPointerException\n\tat
org.apache.solr.handler.component.QueryComponent.returnFields(QueryComponent.java:998)\n\tat
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:653)\n\tat
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:628)\n\tat
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\tat
org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:368)\n\tat
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)\n\tat
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)\n\tat
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)\n\tat
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)\n\tat
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)\n\tat
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)\n\tat
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)\n\tat
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)\n\tat
java.lang.Thread.run(Thread.java:662)\n,code: 500
solrcore.properties in solrcloud
Hi Can some one clarify the role of solrcore.properties in Solr 4.3+? We can define custom properties in solrcore.properties and use them in solrconfig.xml Will this functionality be retained going forward? Suchi
Re: Query time boosts vs static boost function defined in solrconfig.xml
I have a boost function defined in solrconfig.xml which is used to rank
results and need additional reranking based on some weights passed in the
url.
The function using weights passed in the url is parsed into local params
and overrides the function defined in the solrconfig.xml
Ideally I want to use both.
I tried extending edismax parser but wasn't able to over ride this part.
Wasn't able to inject these boost functions in Solrparams either.
Suchi
On Sun, Feb 17, 2013 at 1:13 AM, Jack Krupansky j...@basetechnology.comwrote:
Could you give a more specific example of what you are trying to
accomplish. I mean, what issue you are trying to address.
-- Jack Krupansky
-Original Message- From: Suchi Amalapurapu
Sent: Friday, February 15, 2013 2:21 AM
To: solr-user@lucene.apache.org
Subject: Query time boosts vs static boost function defined in
solrconfig.xml
Query time boost function seem to be loaded via local params while boost
functions defined in solrconfig.xml get added to a request globally.
QParser.geParams
public String getParam(String name) {
String val;
if (localParams != null) {
val = localParams.get(name);
if (val != null) return val;
}
return params.get(name);
}
indicates we can use only one of them with local params overriding the
global one's.
Is there a way to use both? The use case is that I want to apply
parameterized boosts on top of boost functions in solrconfig.xml
To simulate this functionality I can either
- Define a custom QParser.
- Define a custom RequestHandler/SearchComponent
In both cases I can inject local params into request's params and further
do string replace's for some args.
Is there a better way of doing things?
Also are static boost methods in solrconfig.xml loaded only once vs
frequent parsing of queries in query time boosts?
Suchi
Query time boosts vs static boost function defined in solrconfig.xml
Query time boost function seem to be loaded via local params while boost
functions defined in solrconfig.xml get added to a request globally.
QParser.geParams
public String getParam(String name) {
String val;
if (localParams != null) {
val = localParams.get(name);
if (val != null) return val;
}
return params.get(name);
}
indicates we can use only one of them with local params overriding the
global one's.
Is there a way to use both? The use case is that I want to apply
parameterized boosts on top of boost functions in solrconfig.xml
To simulate this functionality I can either
- Define a custom QParser.
- Define a custom RequestHandler/SearchComponent
In both cases I can inject local params into request's params and further
do string replace's for some args.
Is there a better way of doing things?
Also are static boost methods in solrconfig.xml loaded only once vs
frequent parsing of queries in query time boosts?
Suchi
