Re: Solrj 4.7.2 - slowing down over time
Hi,
thanks for your response!
We use javamelody for some basic statistics about app. Here are some graphs
from last 24 hours:
http://imgur.com/a/OQxnb
First graph is memory used by application. Second graph shows how seach
time rapidly increased.
At 13:40 there was neccessary app restart and at 1:30 index optimize job
started. Everything has been fine today, no restart needed.
Based on these graphs, I don't think problem is tight memory.
Can you explain that point with autoCommit?
Now we create list of SolrInputDocuments, add them to EmbeddedSolrServer
and then call explicit hard commit EmbeddedSolrServer.commit() with
both waitFlush and waitSearcher set to true (could be this that problem?).
So no autoCommit is used.
Main reason for it is that after somebody does some document modifications,
we insert document's ID into database. After some time all these documents
are reindexed, commited and removed from database. So everything is
persistent - modified documents are in database or already commited.
Roman
2016-05-18 23:24 GMT+02:00 Joel Bernstein <joels...@gmail.com>:
> One thing to investigate is whether your caches are too large and gradually
> filling up memory. It does sound like memory is getting tighter over time.
> A memory profiler would be helpful in figuring out memory issues.
>
> Moving to autoCommits would also eliminate any slowness due to overlapping
> searchers from committing too frequently.
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Wed, May 18, 2016 at 10:39 AM, Roman Slavík <slaav...@gmail.com> wrote:
>
> > Hi all,
> >
> > we're using solr in our application and have problem that both searching
> > and indexing is slowing down over time.
> >
> > Versions:
> > -
> > Java 1.7
> > Solr 4.7.2
> > Lucene 4.1 (luceneMatchVersion param in solrconfig.xml)
> >
> > App architecture:
> > -
> > We don't use solr as standalone application, but we integrated him with
> > solrj library into our app.
> > It has 1 CoreContainer with 3 cores (let's call them alpha, beta, gamma).
> > EmbeddedSolrServer is used as java class to work with each core.
> > Application does both searching and indexing.
> >
> > What's going on:
> >
> > After Tomcat restart everything is working great for 2-3 days. But after
> > this time solr (all cores) starts slw down until it's unusable and we
> > have to restart Tomcat. Then it works fine.
> > For example search time for really complex query is 1,5 s when it works
> > fine. But then it rises to more than 1 min. Same issue is with indexing -
> > first fast but then slow.
> >
> > Searching:
> > --
> > Core apha is used mainly for normal search. But sometimes for faceting
> too.
> > Beta and gamma are only for facets.
> > alpha: 25000 queries/day
> > beta: 7000 queries/day
> > gamma: 7000 queries/day
> > We do lots of query joins, sometimes cross cores.
> >
> > Indexing:
> > -
> > We commit changes continuously over day. Number of commits is limited to
> 1
> > commit/min for all three cores. So we do max 1440 commits daily. One
> commit
> > contains between 1 and 100 docs.
> > Method EmbeddedSolrServer.add(SolrInputDocument) is used and in the end
> > EmbeddedSolrServer.commit().
> > Every night we call EmbeddedSolrServer.optimize() on each core.
> >
> > Index size:
> > ---
> > alpha: 13,5 GB
> > beta: 300 MB
> > gamma: 600 MB
> >
> > Hardware:
> > -
> > Ubuntu 14.04
> > 8 core CPU
> > java heap space 22 GB RAM
> > SSD drive with more than 50 GB free space
> >
> > Solr config (Same configuration is used for all cores):
> > ---
> > > class="${solr.directoryFactory:solr.MMapDirectoryFactory}"/>
> > LUCENE_41
> >
> > false
> > 10
> >
> > 32
> > 1
> > 1000
> > 1
> >
> > native
> > false
> > true
> >
> >
> > 1
> > 0
> >
> >
> >
> >
> >
> > > autowarmCount="0"/>
> > > autowarmCount="0"/>
> > > autowarmCount="0"/>
> >
> > true
> > false
> > 2
> >
> >
> > Conclusion:
> > ---
> > Is something wrong in configuration? Or is this some kind of bug? Or...?
> > Can you give me some advice how to resolve this problem please?
> >
> >
> > Roman
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
Solrj 4.7.2 - slowing down over time
Hi all,
we're using solr in our application and have problem that both searching
and indexing is slowing down over time.
Versions:
-
Java 1.7
Solr 4.7.2
Lucene 4.1 (luceneMatchVersion param in solrconfig.xml)
App architecture:
-
We don't use solr as standalone application, but we integrated him with
solrj library into our app.
It has 1 CoreContainer with 3 cores (let's call them alpha, beta,
gamma). EmbeddedSolrServer is used as java class to work with each core.
Application does both searching and indexing.
What's going on:
After Tomcat restart everything is working great for 2-3 days. But after
this time solr (all cores) starts slw down until it's unusable and
we have to restart Tomcat. Then it works fine.
For example search time for really complex query is 1,5 s when it works
fine. But then it rises to more than 1 min. Same issue is with indexing
- first fast but then slow.
Searching:
--
Core apha is used mainly for normal search. But sometimes for faceting too.
Beta and gamma are only for facets.
alpha: 25000 queries/day
beta: 7000 queries/day
gamma: 7000 queries/day
We do lots of query joins, sometimes cross cores.
Indexing:
-
We commit changes continuously over day. Number of commits is limited to
1 commit/min for all three cores. So we do max 1440 commits daily. One
commit contains between 1 and 100 docs.
Method EmbeddedSolrServer.add(SolrInputDocument) is used and in the end
EmbeddedSolrServer.commit().
Every night we call EmbeddedSolrServer.optimize() on each core.
Index size:
---
alpha: 13,5 GB
beta: 300 MB
gamma: 600 MB
Hardware:
-
Ubuntu 14.04
8 core CPU
java heap space 22 GB RAM
SSD drive with more than 50 GB free space
Solr config (Same configuration is used for all cores):
---
class="${solr.directoryFactory:solr.MMapDirectoryFactory}"/>
LUCENE_41
false
10
32
1
1000
1
native
false
true
1
0
autowarmCount="0"/>
autowarmCount="0"/>
autowarmCount="0"/>
true
false
2
Conclusion:
---
Is something wrong in configuration? Or is this some kind of bug? Or...?
Can you give me some advice how to resolve this problem please?
Roman
[Solrj 4.0] How use JOIN
Hi,
I can't found any good example how use Join function with Solrj 4.0 api.
Let's have this example data:
doc
field name=id1/field
field name=nameThomas/field
field name=age40/field
/doc
doc
field name=id2/field
field name=nameJohn/field
field name=age17/field
field name=parent1/field
/doc
And code:
String stringQuery = (name:Thomas) AND (age:40);
SolrQuery query = new SolrQuery();
query.setQuery(stringQuery);
QueryResponse response = solrServer.query(query);
Result is doc with id=1. Now I want extend query above with JOIN similar
to this:
{!join from=to=idparent }(name:John AND age:17)
and expect same result - doc with id=1. But I have no idea how add this
Join condition to SolrQuery object. Is there any method I missed? Or
must I extend my base stringQuery somehow?
Thanks for any advice!
Roman
[Solrj 4.0] No group response
Hi guys, I have problem with grouping in Solr 4.0 using Solrj api. I need this: search some documents limited with solr query, group them by one field and return total count of groups. There is param 'group.ngroups' for adding groups count into group response. Sounds easy, so I wrote something like this: SolrQuery query = new SolrQuery().setQuery(queryString); query.addField(score); query.setParam(GroupParams.GROUP, true); query.setParam(GroupParams.GROUP_MAIN, true); query.setParam(GroupParams.GROUP_FIELD, group_field); query.setParam(GroupParams.GROUP_LIMIT, 1); query.setParam(GroupParams.GROUP_TOTAL_COUNT, true); QueryResponse response = solrServer.query(query); // contains found docs GroupResponse groupResponse = response.getGroupResponse(); // null Search result is ok, QueryResponse contains docs I searched for. But group response is always null. Did I miss something, some magic parameter for enabling group response? Thanks for any advice Roman
Re: Solr4 BETA group.ngroups count
Hi, did you find where is problem? Dne 23.8.2012 16:40, lenz...@gfi.ihk.de napsal(a): Hello, I have a problem using grouped queries and the 'group.ngroups' parameter. When I run the following request myURL/select?q=myQuerygroup=truegroup.field=personIdgroup.ngroups=truewt=xml the response looks like this: lst name=grouped lst name=personId int name=matches11/int int name=ngroups6/int arr name=groups lst str name=groupValue106.12345/str result name=doclist numFound=3 start=0.../result /lst lst str name=groupValue106.12312/str result name=doclist numFound=3 start=0.../result /lst lst str name=groupValue101.12313/str result name=doclist numFound=2 start=0 maxScore= 0.08318384.../result /lst lst str name=groupValue101.12312/str result name=doclist numFound=3 start=0 maxScore= 0.08318384.../result /lst /arr /lst /lst I expected, that the ngroups results in 4, because it is the total count of all groups, that match my query. The result of 'matches' is right, and the 11 docs are distributed on the 4 groups of my response, but I have no idea, what ngroups is counting in this case. Can anybody explain to me, what's the meaning of ngroups is? regards Norman Lenzner
Re: Group count in SOLR 3.3
Thanks, I found GroupResponse was added in Solr 3.4 so I have to update version :) Roman Dne 24.8.2012 18:09, Erick Erickson napsal(a): 3.6 has a getNGroups, does that do what you want? Best Erick On Thu, Aug 23, 2012 at 2:23 AM, Roman Slavík sla...@effectiva.cz wrote: Hi guys, we are using SOLR 3.3 with Solrj inside our java project. In actual version we had to add some grouping support, so we add parameters into SolrQuery object like this: query.setParam(GroupParams.GROUP, true); query.setParam(GroupParams.GROUP_MAIN, true); query.setParam(GroupParams.GROUP_FIELD, OUR_GROUP_FIELD); and we get QueryResponse with results we need. Awesome! But now I have one remaining problem, I don't know how get number of groups from QueryResponse. I found I must add group.ngroups=true param into query. So I did it: query.setParam(GroupParams.GROUP_TOTAL_COUNT, true); But QueryResponse seems same. There's no method like getGroupCount() or group count param in header. Am I doing something wrong? Or is it SOLR 3.3 problem? If we upgrade to newer version, will it works? Thanks for any advice! Roman
Group count in SOLR 3.3
Hi guys, we are using SOLR 3.3 with Solrj inside our java project. In actual version we had to add some grouping support, so we add parameters into SolrQuery object like this: query.setParam(GroupParams.GROUP, true); query.setParam(GroupParams.GROUP_MAIN, true); query.setParam(GroupParams.GROUP_FIELD, OUR_GROUP_FIELD); and we get QueryResponse with results we need. Awesome! But now I have one remaining problem, I don't know how get number of groups from QueryResponse. I found I must add group.ngroups=true param into query. So I did it: query.setParam(GroupParams.GROUP_TOTAL_COUNT, true); But QueryResponse seems same. There's no method like getGroupCount() or group count param in header. Am I doing something wrong? Or is it SOLR 3.3 problem? If we upgrade to newer version, will it works? Thanks for any advice! Roman
