Re: improve performance after commit
: back in just now. Here's an example trying to warm using a sort on : field name subject. I tried query of : allMessageContent:trying;subject+asc as well as : allMessageContent:trying;subject (without +asc) - either way when expressing params in XML (either as init params for a request handler, or in a QuerySenderListener the params don't need to be URL escaped ... they just need to be XML escaped, try something like... listener event=newSearcher class=solr.QuerySenderListener arr name=queries lst str name=qallMessageContent:test; subject asc/str str name=start0/str str name=rows10/str /lst /arr /listener -Hoss
improve performance after commit
hello, I'm looking for some tips / suggestions around reducing the query time for Solr after I've post'ed a commit request. My Lucene index contains around 2,000,000 documents, and I have a job that periodically removes artibrary documents from Lucene and replaces them with fresh copies from a database. Whenever that cycle occurs, I send a commit to Solr to expose the updates. The problem is that immediately after the commit, a Solr query that previously took 5-20ms now takes 20-25 seconds. Ouch. I know that commit can be expensive, although I don't know by how much, or what I might do to mitigate the expense. I haven't much doc around this topic. I've also tried different cache settings (basically using high values for cache and auto-warm sizes) but that doesn't seem to make much of a difference. I'll keep investigating on my own, but if anyone has any suggestions or additional info, I would greatly appreciate it. thanks, Kaan Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace Managed Hosting. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at [EMAIL PROTECTED], and delete the original message. Your cooperation is appreciated.
Re: improve performance after commit
On 3/6/07, Kaan Erdener [EMAIL PROTECTED] wrote: I'm looking for some tips / suggestions around reducing the query time for Solr after I've post'ed a commit request. My Lucene index contains around 2,000,000 documents, and I have a job that periodically removes artibrary documents from Lucene and replaces them with fresh copies from a database. Whenever that cycle occurs, I send a commit to Solr to expose the updates. The problem is that immediately after the commit, a Solr query that previously took 5-20ms now takes 20-25 seconds. Ouch. If this is a normal query (no faceting) then most likely the time is spent populating a lucene FieldCache entry used for sorting results. Put a static warming entry in solrconfig.xml that queries for a small number of documents and sorts that query by all the fields you commonly sort by. -Yonik
Re: improve performance after commit
On Mar 6, 2007, at 1:55 PM, Yonik Seeley wrote: On 3/6/07, Kaan Erdener [EMAIL PROTECTED] wrote: I'm looking for some tips / suggestions around reducing the query time for Solr after I've post'ed a commit request. My Lucene index contains around 2,000,000 documents, and I have a job that periodically removes artibrary documents from Lucene and replaces them with fresh copies from a database. Whenever that cycle occurs, I send a commit to Solr to expose the updates. The problem is that immediately after the commit, a Solr query that previously took 5-20ms now takes 20-25 seconds. Ouch. If this is a normal query (no faceting) then most likely the time is spent populating a lucene FieldCache entry used for sorting results. Put a static warming entry in solrconfig.xml that queries for a small number of documents and sorts that query by all the fields you commonly sort by. -Yonik I'm not exactly sure this is what you meant, but I did some more research and it looks close. I added the following to my solrconfig.xml: listener event=newSearcher class=solr.QuerySenderListener arr name=queries lst str name=qallMessageContent:test/str str name=start0/str str name=rows10/str /lst /arr /listener and also: listener event=firstSearcher class=solr.QuerySenderListener arr name=queries lst str name=qallMessageContent:trying/str str name=start0/str str name=rows10/str /lst /arr /listener From what I can see in the logs, these are both invoked after the commit. However, the query times after a commit are still slow (around 20 seconds). I'm guessing I didn't set up the warming correctly? I had some sorting parameters in there, but the syntax was wrong, produced errors on startup, so I took them out for now. Mar 6, 2007 4:51:52 PM org.apache.solr.update.DirectUpdateHandler2 commit INFO: end_commit_flush Mar 6, 2007 4:51:52 PM org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming [EMAIL PROTECTED] main from [EMAIL PROTECTED] main documentCache {lookups=10,hits=0,hitratio=0.00,inserts=20,evictions=0,size=20,cumulati ve_lookups=120,cumulative_hits=68,cumulative_hitratio=0.56,cumulative_in serts=52,cumulative_evictions=0} Mar 6, 2007 4:51:52 PM org.apache.solr.search.SolrIndexSearcher warm INFO: autowarming result for [EMAIL PROTECTED] main documentCache {lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,cumulative_ lookups=120,cumulative_hits=68,cumulative_hitratio=0.56,cumulative_inser ts=52,cumulative_evictions=0} Mar 6, 2007 4:51:52 PM org.apache.solr.core.QuerySenderListener newSearcher INFO: QuerySenderListener sending requests to [EMAIL PROTECTED] main Mar 6, 2007 4:51:52 PM org.apache.solr.core.SolrCore execute INFO: rows=10start=0q=allMessageContent:trying 0 410 Mar 6, 2007 4:51:52 PM org.apache.solr.core.QuerySenderListener newSearcher Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace Managed Hosting. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at [EMAIL PROTECTED], and delete the original message. Your cooperation is appreciated.
Re: improve performance after commit
On 3/6/07, Kaan Erdener [EMAIL PROTECTED] wrote: From what I can see in the logs, these are both invoked after the commit. However, the query times after a commit are still slow (around 20 seconds). Your warming script didn't do any sorts. Why don't you also show the part of the log with the slow query... that would make it much easier for people to help. -Yonik
Re: improve performance after commit
str name=qallMessageContent:test;subject+asc/str there should be a space between subject and asc, try: http://host/select?q=allMessageContent:test;subject%20asc + is supposed to become a space, but it looks like it is staying +