You're just going to have to accept it being slow. Think of it this way: you have 4M (say) buckets that have to be counted into. Then the top 500 have to be collected to return. That's just going to take some time unless you have very beefy machines.
I'd _really_ back up and consider whether this is a good thing or whether this is one of those ideas that doesn't have much use to the user. If your results rarely if ever show counts for a URL more than, say, 5, is it really giving your users useful info? Best, Erick On Mon, Nov 4, 2013 at 6:54 PM, Mingfeng Yang <mfy...@wisewindow.com> wrote: > Erick, > > It could have more than 4M distinct values. The purpose of this facet is > to display the most frequent, say top 500, urls to users. > > Sascha, > > Thanks for the info. I will look into this thread thing. > > Mingfeng > > > On Mon, Nov 4, 2013 at 4:47 AM, Erick Erickson <erickerick...@gmail.com > >wrote: > > > How many unique URLs do you have in your 9M > > docs? If your 9M hits have 4M distinct URLs, then > > this is not very valuable to the user. > > > > Sascha: > > Was that speedup on a single field or were you faceting over > > multiple fields? Because as I remember that code spins off > > threads on a per-field basis, and if I'm mis-remembering I need > > to look again! > > > > Best, > > Erick > > > > > > On Sat, Nov 2, 2013 at 5:07 AM, Sascha SZOTT <sz...@gmx.de> wrote: > > > > > Hi Ming, > > > > > > which Solr version are you using? In case you use one of the latest > > > versions (4.5 or above) try the new parameter facet.threads with a > > > reasonable value (4 to 8 gave me a massive performance speedup when > > > working with large facets, i.e. nTerms >> 10^7). > > > > > > -Sascha > > > > > > > > > Mingfeng Yang wrote: > > > > I have an index with 170M documents, and two of the fields for each > > > > doc is "source" and "url". And I want to know the top 500 most > > > > frequent urls from Video source. > > > > > > > > So I did a facet with > > > > "fq=source:Video&facet=true&facet.field=url&facet.limit=500", and > > > > the matching documents are about 9 millions. > > > > > > > > The solr cluster is hosted on two ec2 instances each with 4 cpu, and > > > > 32G memory. 16G is allocated tfor java heap. 4 master shards on one > > > > machine, and 4 replica on another machine. Connected together via > > > > zookeeper. > > > > > > > > Whenever I did the query above, the response is just taking too long > > > > and the client will get timed out. Sometimes, when the end user is > > > > impatient, so he/she may wait for a few second for the results, and > > > > then kill the connection, and then issue the same query again and > > > > again. Then the server will have to deal with multiple such heavy > > > > queries simultaneously and being so busy that we got "no server > > > > hosting shard" error, probably due to lost communication between solr > > > > node and zookeeper. > > > > > > > > Is there any way to deal with such problem? > > > > > > > > Thanks, Ming > > > > > > > > > >
