CCU of Solr?
Hi all! I am developing an online dictionary application by using Solr, but I wonder that how many concurrent request that Solr can be process? -- View this message in context: http://old.nabble.com/CCU-of-Solr--tp26840318p26840318.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: CCU of Solr?
it is very difficult to say. It depends on the cache hit ratio. If everything is served out of cache you may go upto arounbf 1000 req/sec On Fri, Dec 18, 2009 at 1:39 PM, Olala wrote: > > Hi all! > > I am developing an online dictionary application by using Solr, but I wonder > that how many concurrent request that Solr can be process? > -- > View this message in context: > http://old.nabble.com/CCU-of-Solr--tp26840318p26840318.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- - Noble Paul | Systems Architect| AOL | http://aol.com
Re: CCU of Solr?
Thanks for your answer! But how i can test this??? Do you know any tool that help me do that? :confused: Noble Paul നോബിള് नोब्ळ्-2 wrote: > > it is very difficult to say. It depends on the cache hit ratio. If > everything is served out of cache you may go upto arounbf 1000 req/sec > > On Fri, Dec 18, 2009 at 1:39 PM, Olala wrote: >> >> Hi all! >> >> I am developing an online dictionary application by using Solr, but I >> wonder >> that how many concurrent request that Solr can be process? >> -- >> View this message in context: >> http://old.nabble.com/CCU-of-Solr--tp26840318p26840318.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > - > Noble Paul | Systems Architect| AOL | http://aol.com > > -- View this message in context: http://old.nabble.com/CCU-of-Solr--tp26840318p26840598.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Huge load and long response times during search
Try solr.FastLRUCache instead of solr.LRUCache it's the new cache
gesture for solr 1.4.
And maybe true in main index section or
diminish mergefactor
see http://wiki.apache.org/lucene-java/ImproveSearchingSpeed
Tomasz Kępski a écrit :
Hi,
I'm using SOLR(1.4) to search among about 3,500,000 documents. After
the server kernel was updated to 64bit system has started to suffer.
Our server has 8G of RAM and double Intel Core 2 DUO.
We used to have average loads around 2-2,5. It was not as good as it
should but as long HTTP response times was acceptable we do not care
to much ;-)
Since few days avg loads are usually around 6, sometimes goes even to
20. PHP, Mysql and Postgresql based application is rather fine, but
when tries to access SOLR it takes ages to load page. In top java
process (Jetty) takes 200-250% of CPU, iotop shows that most of the
disk operations are done by SOLR threads as well.
When we do shut down Jetty load goes down to 1,5 or even less than 1.
My index has ~12G below is a part of my solrconf.xml:
1024
true
true
40
200
solr 0 name="rows">10
solr price
0 10
solr name="sort">rekomendacja 0 name="rows">10
static newSearcher warming query from
solrconfig.xml
fast_warm 0
10
static firstSearcher warming query from
solrconfig.xml
false
dismax
explicit
0.01
name^90.0 scategory^450.0 brand^90.0 text^0.01 description^30
brand,description,id,name,price,score
4<100% 5<90%
100
*:*
sample query parameters from log looks like this:
2009-11-20 21:07:15 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select
params={spellcheck=true&wt=json&rows=20&json.nl=map&start=520&facet=true&spellcheck.collate=true&fl=id,name,description,preparation,url,shop_id&q=camera&qt=dismax&version=1.3&hl.fl=name,description,atributes,brand,url&facet.field=shop_id&facet.field=brand&hl.fragsize=200&spellcheck.count=5&hl.snippets=3&hl=true}
hits=3784 status=0 QTime=83
2009-11-20 21:07:15 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/spellCheckCompRH
params={spellcheck=true&wt=json&rows=20&json.nl=map&start=520&facet=true&spellcheck.collate=true&fl=id,name,description,preparation,url,shop_id&q=camera&qt=dismax&version=1.3&hl.fl=name,description,atributes,brand,url&facet.field=shop_id&facet.field=brand&hl.fragsize=200&spellcheck.count=5&hl.snippets=3&hl=true}
hits=3784 status=0 QTime=16
And at last the question ;-)
How to speed up the search?
Which parameters should I check first to find out what is the bottleneck?
Sorry for verbose entry but I would like to give as clear point of
view as possible
Thanks in advance,
Tom
Re: Problem with words thats amost similar
Den 17. des. 2009 kl. 13.48 skrev Shalin Shekhar Mangar: > 2009/12/17 Steinar Asbjørnsen > >> Den 17. des. 2009 kl. 12.42 skrev Shalin Shekhar Mangar: >> >>> For specific cases like this, you can add the word to a file and specify >> it >>> in schema, for example: >>> >>> >> protected="protwords.txt"/> >> >> Ty Shalin. >> >> This is my schema.xml file >> >> >> >> > words="stopwords.txt" enablePositionIncrements="true"/> >> > generateWordParts="1" generateNumberParts="1" catenateWords="1" >> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> >> >> > protected="protwords.txt"/> >> >> >> >> >> > ignoreCase="true" expand="true"/> >> > words="stopwords.txt"/> >> > generateWordParts="1" generateNumberParts="1" catenateWords="0" >> catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> >> >> > protected="protwords.txt"/> >> >> >> >> >> I added restaurant and restaurering to protwords.txt, restarted Tomcat, but >> no dice. >> Do I need to use the SnowballPorterFilterFactory? >> And do I need to reindex the documents? >> >> > Actually EnglishPorterFilterFactory is the same as > SnowballPorterFilterFactory with language="English". Both will work. You > will need to re-index the documents. What I've done so far is to add both restaurant and restaurering to protwords.txt. I've also refeed a single document (with the keyword "restaurering") to check that it no longer appears in a search result for "restaurant". Do i have to refeed every document in the index? Or restart so that solr re-reads the protwords.txt-file (this is on a different installation(prod) then the one i restarted earlier(dev))? Steinar
Re: Simple Wildcard Search Question
Thanks for hanging in there and helping me when wildcard-queries aren't analysed it makes sense for this one. But i'm still stuck at the problem mentioned above with "AndererName09". I'm not using any wildcards, the query-string and the index-value clearly show - a lot of - matches but still I get no results search for that very same query-string. e.G. Index-Value: "Herrn AndererName09 Vorname09" split into "herrn|ander|name|09|nochmal|name|09 anderernam" Query-Value: "AndererName09" split into: "ander|name|09 andernam" but no results?! -- View this message in context: http://old.nabble.com/Simple-Wildcard-Search-Question-tp26747482p26842359.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem with words thats amost similar
2009/12/18 Steinar Asbjørnsen > > What I've done so far is to add both restaurant and restaurering to > protwords.txt. > I've also refeed a single document (with the keyword "restaurering") to > check that it no longer appears in a search result for "restaurant". > Do i have to refeed every document in the index? > Or restart so that solr re-reads the protwords.txt-file (this is on a > different installation(prod) then the one i restarted earlier(dev))? > > Documents already added in the index have already gone through the analysis step and terms such as restaurering would have already been stemmed. Therefore, you'll need to re-index all documents which contained the words you have specified in protwords.txt. -- Regards, Shalin Shekhar Mangar.
Optimize not having any effect on my index
Hey guys, I'm getting some strange behavior here, and I'm wondering if I'm doing anything wrong.. I've got an unoptimized index, and I'm trying to run the following command: http://server:8983/solr/update?optimize=true&maxSegments=10&waitFlush=false Tried it first directly in the browser, it obviously took quite a bit of time, but once it was finished I see no difference in my index. Same number of files, same size etc. So i tried with curl: curl http://server:8983/solr/update --data-binary '' -H 'Content-type:text/xml; charset=utf-8' No difference here either... Am I doing anything wrong? Do i need to issue a commit after the optimize? Any pointers would be greatly appreciated. Cheers, Aleks
Re: shards parameter
My goal is to hide configuration from client application. So when I
distribute index, application does not know this.
I added new search handler and it works fine. Thanks to everybody
Jacob Elder-4 wrote:
>
> If the goal is to save time when using the admin interface, you can just
> add
> this to conf/admin-extra.html:
>
> http://www.google.com/jsapi";>
>
> google.load("prototype", "1.6");
>
>
> Event.observe(
> window,
> 'load',
> function() {
> elements = document.getElementsByName('queryForm')
> elements[0].insert(" value=\"shard01,shard02\">")
> });
>
>
> You will get an editable field with sensible defaults under the query box.
>
> On Thu, Dec 17, 2009 at 4:09 PM, Yonik Seeley
> wrote:
>
>> You're setting up an infinite loop by adding a shards parameter on the
>> default search handler.
>> Create a new search handler and put your default under that.
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>> On Thu, Dec 17, 2009 at 7:47 AM, pcurila wrote:
>> >
>> > I tried it out. But there is another issue I can not cope with.
>> > I have two shards:
>> > localhost:8983/solr
>> > localhost:8984/solr
>> >
>> > If I write this into the defaults section
>> > localhost:8983/solr,localhost:8984/solr
>> > and than I issue a query on localhost:8983, solr do not respond.
>> >
>> > If I write this
>> > localhost:8984/solr
>> > it works but there is just half of the index.
>> >
>> >
>> >
>> >
>> > Noble Paul നോബിള് नोब्ळ्-2 wrote:
>> >>
>> >> yes.
>> >> put it under the "defaults" section in your standard requesthandler.
>> >>
>> >> On Thu, Dec 17, 2009 at 5:22 PM, pcurila wrote:
>> >>>
>> >>> Hello, is there any way to configure shards parameter in
>> solrconfig.xml?
>> >>> So I
>> >>> do not need provide it in the url. Thanks Peter
>> >>> --
>> >>> View this message in context:
>> >>> http://old.nabble.com/shards-parameter-tp26826908p26826908.html
>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> -
>> >> Noble Paul | Systems Architect| AOL | http://aol.com
>> >>
>> >>
>> >
>> > --
>> > View this message in context:
>> http://old.nabble.com/shards-parameter-tp26826908p26827527.html
>> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >
>> >
>>
>
>
>
> --
> Jacob Elder
>
>
--
View this message in context:
http://old.nabble.com/shards-parameter-tp26826908p26843690.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: CCU of Solr?
You can't test it until you have a working SOLR instance in your specific problem space. But assuming you have a SOLR setup, there are a plethora of tools, just google "SOLR load testing". JMeter has been mentioned, as well as others. You can also write your own load tester that just spawns a bunch of threads that query your SOLR server, should take you about a day. Erick On Fri, Dec 18, 2009 at 3:42 AM, Olala wrote: > > Thanks for your answer! But how i can test this??? Do you know any tool > that > help me do that? :confused: > > > Noble Paul നോബിള് नोब्ळ्-2 wrote: > > > > it is very difficult to say. It depends on the cache hit ratio. If > > everything is served out of cache you may go upto arounbf 1000 req/sec > > > > On Fri, Dec 18, 2009 at 1:39 PM, Olala wrote: > >> > >> Hi all! > >> > >> I am developing an online dictionary application by using Solr, but I > >> wonder > >> that how many concurrent request that Solr can be process? > >> -- > >> View this message in context: > >> http://old.nabble.com/CCU-of-Solr--tp26840318p26840318.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > >> > > > > > > > > -- > > - > > Noble Paul | Systems Architect| AOL | http://aol.com > > > > > > -- > View this message in context: > http://old.nabble.com/CCU-of-Solr--tp26840318p26840598.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
Re: Simple Wildcard Search Question
Hmmm. What do you get when you use ?debugQuery=true? Have you gone in through the SOLR admin page and tries queries that way? what do you see? Puzzles me too Erick On Fri, Dec 18, 2009 at 6:52 AM, QBasti wrote: > > Thanks for hanging in there and helping me > > when wildcard-queries aren't analysed it makes sense for this one. But i'm > still stuck at the problem mentioned above with "AndererName09". I'm not > using any wildcards, the query-string and the index-value clearly show - a > lot of - matches but still I get no results search for that very same > query-string. > e.G. > Index-Value: "Herrn AndererName09 Vorname09" split into > "herrn|ander|name|09|nochmal|name|09 >anderernam" > > Query-Value: "AndererName09" split into: > "ander|name|09 > andernam" > > but no results?! > > -- > View this message in context: > http://old.nabble.com/Simple-Wildcard-Search-Question-tp26747482p26842359.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
Re: Optimize not having any effect on my index
Hi Aleksander, Aleksander Stensby wrote: So i tried with curl: curl http://server:8983/solr/update --data-binary '' -H 'Content-type:text/xml; charset=utf-8' No difference here either... Am I doing anything wrong? Do i need to issue a commit after the optimize? Did you restart the Solr server instance after the optimize operation was completed? BTW: You could initiate the optimization operation by POSTing optimize=true directly, i.e., curl http://server:8983/solr/update/update --form-string optimize=true -Sascha
Re: Adaptive search?
You can add click counts to your index as additional field and boost results based on that value. http://wiki.apache.org/solr/SolrRelevancyFAQ#How_can_I_change_the_score_of_a_document_based_on_the_.2Avalue.2A_of_a_field_.28say.2C_.22popularity.22.29 You can keep some kind of buffer for clicks and update click count field for documents in the index periodically. If you don't want to update whole documents in the index then you probably should look at ExternalFileField or Lucene ParallelReader as a custom Solr IndexReader, but this is complex low level Lucene stuff and requires some hacking. Alex On Thu, Dec 17, 2009 at 6:46 PM, Siddhant Goel wrote: > Let say we have a search engine (a simple front end - web app kind of a > thing - responsible for querying Solr and then displaying the results in a > human readable form) based on Solr. If a user searches for something, gets > quite a few search results, and then clicks on one such result - is there > any mechanism by which we can notify Solr to boost the score/relevance of > that particular result in future searches? If not, then any pointers on how > to go about doing that would be very helpful. > > Thanks, > > On Thu, Dec 17, 2009 at 7:50 PM, Paul Libbrecht wrote: > >> What can it mean to "adapt to user clicks" ? Quite many things in my head. >> Do you have maybe a citation that inspires you here? >> >> paul >> >> >> Le 17-déc.-09 à 13:52, Siddhant Goel a écrit : >> >> >> Does Solr provide adaptive searching? Can it adapt to user clicks within >>> the >>> search results it provides? Or that has to be done externally? >>> >> >> > > > -- > - Siddhant >
need of projects
hi all, I am disparately in need some webpages that uses solr at backend and display the results.. Plz can u send some PHP codes tht resembles like Google that uses Solr. Im new to solr.
Re: need of projects
Hello, There are many sites that are using Solr YoAuto.com - Auto parts shopping comparison engine is using solr on the backend. Its not a PHP site. Paul On Fri, Dec 18, 2009 at 9:34 AM, Naga raja wrote: > hi all, > I am disparately in need some webpages that uses solr at backend and > display the results.. > Plz can u send some PHP codes tht resembles like Google that uses Solr. > Im new to solr. >
RE: need of projects
http://our.gop.com/app/render/go.aspx?xsl=search.xslt&searchTerm=*:*&searc hFilter=SearchObjectCategory:USER_PROFILE It returns well formed XML or JSoN, simply utilize it as a rest-like web service. Most sites you visit that utilize it will not expose that consumption logic in client visible code. We proxy it, as I suspect most do. -Original Message- From: Naga raja [mailto:jollyn...@gmail.com] Sent: Friday, December 18, 2009 12:35 PM To: solr-user Subject: need of projects hi all, I am disparately in need some webpages that uses solr at backend and display the results.. Plz can u send some PHP codes tht resembles like Google that uses Solr. Im new to solr.
Re: CCU of Solr?
I used ab(apache bench) to test with handle 1000 requests, with a maximum of 300 requests running concurrently (ab -n 1000 -c 300), and then I received the output as follows: Concurrency Level: 300 Time taken for tests: 6.797 seconds Complete requests: 1000 Failed requests:0 Write errors: 0 Non-2xx responses: 1000 Total transferred: 162000 bytes HTML transferred: 0 bytes Requests per second:147.13 [#/sec] (mean) Time per request: 2039.063 [ms] (mean) Time per request: 6.797 [ms] (mean, across all concurrent requests) Transfer rate: 23.28 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:01 4.7 0 78 Processing: 234 979 310.6 10164141 Waiting: 219 632 230.16094141 Total:234 980 310.7 10164141 Percentage of the requests served within a certain time (ms) 50% 1016 66% 1141 75% 1203 80% 1219 90% 1313 95% 1453 98% 1469 99% 1484 100% 4141 (longest request) I wonder that 147 requests per second is too low? Erick Erickson wrote: > > You can't test it until you have a working SOLR instance in > your specific problem space. > > But assuming you have a SOLR setup, there are a plethora of > tools, just google "SOLR load testing". JMeter has been mentioned, > as well as others. > > You can also write your own load tester that just spawns a bunch of > threads that query your SOLR server, should take you about a day. > > Erick > > On Fri, Dec 18, 2009 at 3:42 AM, Olala wrote: > >> >> Thanks for your answer! But how i can test this??? Do you know any tool >> that >> help me do that? :confused: >> >> >> Noble Paul നോബിള് नोब्ळ्-2 wrote: >> > >> > it is very difficult to say. It depends on the cache hit ratio. If >> > everything is served out of cache you may go upto arounbf 1000 req/sec >> > >> > On Fri, Dec 18, 2009 at 1:39 PM, Olala wrote: >> >> >> >> Hi all! >> >> >> >> I am developing an online dictionary application by using Solr, but I >> >> wonder >> >> that how many concurrent request that Solr can be process? >> >> -- >> >> View this message in context: >> >> http://old.nabble.com/CCU-of-Solr--tp26840318p26840318.html >> >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> >> >> >> > >> > >> > >> > -- >> > - >> > Noble Paul | Systems Architect| AOL | http://aol.com >> > >> > >> >> -- >> View this message in context: >> http://old.nabble.com/CCU-of-Solr--tp26840318p26840598.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > -- View this message in context: http://old.nabble.com/CCU-of-Solr--tp26840318p26852460.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: CCU of Solr?
ab is not the best testing tool to test website performance try loadrunner or apache httpclient instead, they act more like an browser On Sat, Dec 19, 2009 at 11:20 AM, Olala wrote: > > I used ab(apache bench) to test with handle 1000 requests, with a maximum > of > 300 requests running concurrently (ab -n 1000 -c 300), and then I received > the output as follows: > > Concurrency Level: 300 > Time taken for tests: 6.797 seconds > Complete requests: 1000 > Failed requests:0 > Write errors: 0 > Non-2xx responses: 1000 > Total transferred: 162000 bytes > HTML transferred: 0 bytes > Requests per second:147.13 [#/sec] (mean) > Time per request: 2039.063 [ms] (mean) > Time per request: 6.797 [ms] (mean, across all concurrent requests) > Transfer rate: 23.28 [Kbytes/sec] received > > Connection Times (ms) > min mean[+/-sd] median max > Connect:01 4.7 0 78 > Processing: 234 979 310.6 10164141 > Waiting: 219 632 230.16094141 > Total:234 980 310.7 10164141 > > Percentage of the requests served within a certain time (ms) > 50% 1016 > 66% 1141 > 75% 1203 > 80% 1219 > 90% 1313 > 95% 1453 > 98% 1469 > 99% 1484 > 100% 4141 (longest request) > > > I wonder that 147 requests per second is too low? > > > > Erick Erickson wrote: > > > > You can't test it until you have a working SOLR instance in > > your specific problem space. > > > > But assuming you have a SOLR setup, there are a plethora of > > tools, just google "SOLR load testing". JMeter has been mentioned, > > as well as others. > > > > You can also write your own load tester that just spawns a bunch of > > threads that query your SOLR server, should take you about a day. > > > > Erick > > > > On Fri, Dec 18, 2009 at 3:42 AM, Olala wrote: > > > >> > >> Thanks for your answer! But how i can test this??? Do you know any tool > >> that > >> help me do that? :confused: > >> > >> > >> Noble Paul നോബിള് नोब्ळ्-2 wrote: > >> > > >> > it is very difficult to say. It depends on the cache hit ratio. If > >> > everything is served out of cache you may go upto arounbf 1000 req/sec > >> > > >> > On Fri, Dec 18, 2009 at 1:39 PM, Olala wrote: > >> >> > >> >> Hi all! > >> >> > >> >> I am developing an online dictionary application by using Solr, but I > >> >> wonder > >> >> that how many concurrent request that Solr can be process? > >> >> -- > >> >> View this message in context: > >> >> http://old.nabble.com/CCU-of-Solr--tp26840318p26840318.html > >> >> Sent from the Solr - User mailing list archive at Nabble.com. > >> >> > >> >> > >> > > >> > > >> > > >> > -- > >> > - > >> > Noble Paul | Systems Architect| AOL | http://aol.com > >> > > >> > > >> > >> -- > >> View this message in context: > >> http://old.nabble.com/CCU-of-Solr--tp26840318p26840598.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: > http://old.nabble.com/CCU-of-Solr--tp26840318p26852460.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- Regards Qiu - chiqiu@gmail.com
