Hi Yuval, quick question. You say that your code has 750k docs and around 400mb? Is this some kind of test dataset and you expect it to grow significantly? For an index of this size, I wouldn't use distributed search, single shard should be fine.
Tomás On Sun, Nov 17, 2013 at 6:50 AM, Yuval Dotan <yuvaldo...@gmail.com> wrote: > Hi, > > I isolated the case > > Installed on a new machine (2 x Xeon E5410 2.33GHz) > > I have an environment with 12Gb of memory. > > I assigned 6gb of memory to Solr and I’m not running any other memory > consuming process so no memory issues should arise. > > Removed all indexes apart from two: > > emptyCore – empty – used for routing > > core1 – holds the stored data – has ~750,000 docs and size of 400Mb > > Again this is a single machine that holds both indexes. > > The query > > http://localhost:8210/solr/emptyCore/select?rows=5000&q=*:*&shards=127.0.0.1:8210/solr/core1&wt=jsonQTime > takes ~3 seconds > > and direct query > http://localhost:8210/solr/core1/select?rows=5000&q=*:*&wt=json Qtime > takes > ~15 ms - a magnitude difference. > > I ran the long query several times and got an improvement of about a sec > (33%) but that’s it. > > I need to better understand why this is happening. > > I tried looking at Solr code and debugging the issue but with no success. > > The one thing I did notice is that the getFirstMatch method which receives > the doc id, searches the term dict and returns the internal id takes most > of the time for some reason. > > I am pretty stuck and would appreciate any ideas > > My only solution for the moment is to bypass the distributed query, > implement code in my own app that directly queries the relevant cores and > handles the sorting etc.. > > Thanks > > > > > On Sat, Nov 16, 2013 at 2:39 PM, Michael Sokolov < > msoko...@safaribooksonline.com> wrote: > > > Did you say what the memory profile of your machine is? How much memory, > > and how large are the shards? This is just a random guess, but it might > be > > that if you are memory-constrained, there is a lot of thrashing caused by > > paging (swapping?) in and out the sharded indexes while a single index > can > > be scanned linearly, even if it does need to be paged in. > > > > -Mike > > > > > > On 11/14/2013 8:10 AM, Elran Dvir wrote: > > > >> Hi, > >> > >> We tried returning just the id field and got exactly the same > performance. > >> Our system is distributed but all shards are in a single machine so > >> network issues are not a factor. > >> The code we found where Solr is spending its time is on the shard and > not > >> on the routing core, again all shards are local. > >> We investigated the getFirstMatch() method and noticed that the > >> MultiTermEnum.reset (inside MultiTerm.iterator) and MultiTerm.seekExact > >> take 99% of the time. > >> Inside these methods, the call to BlockTreeTermsReader$ > >> FieldReader$SegmentTermsEnum$Frame.loadBlock takes most of the time. > >> Out of the 7 seconds run these methods take ~5 and > >> BinaryResponseWriter.write takes the rest(~ 2 seconds). > >> > >> We tried increasing cache sizes and got hits, but it only improved the > >> query time by a second (~6), so no major effect. > >> We are not indexing during our tests. The performance is similar. > >> (How do we measure doc size? Is it important due to the fact that the > >> performance is the same when returning only id field?) > >> > >> We still don't completely understand why the query takes this much > longer > >> although the cores are on the same machine. > >> > >> Is there a way to improve the performance (code, configuration, query)? > >> > >> -----Original Message----- > >> From: idokis...@gmail.com [mailto:idokis...@gmail.com] On Behalf Of > >> Manuel Le Normand > >> Sent: Thursday, November 14, 2013 1:30 AM > >> To: solr-user@lucene.apache.org > >> Subject: Re: distributed search is significantly slower than direct > search > >> > >> It's surprising such a query takes a long time, I would assume that > after > >> trying consistently q=*:* you should be getting cache hits and times > should > >> be faster. Try see in the adminUI how do your query/doc cache perform. > >> Moreover, the query in itself is just asking the first 5000 docs that > >> were indexed (returing the first [docid]), so seems all this time is > wasted > >> on transfer. Out of these 7 secs how much is spent on the above method? > >> What do you return by default? How big is every doc you display in your > >> results? > >> Might be the matter that both collections work on the same ressources. > >> Try elaborating your use-case. > >> > >> Anyway, it seems like you just made a test to see what will be the > >> performance hit in a distributed environment so I'll try to explain some > >> things we encountered in our benchmarks, with a case that has at least > the > >> similarity of the num of docs fetched. > >> > >> We reclaim 2000 docs every query, running over 40 shards. This means > >> every shard is actually transfering to our frontend 2000 docs every > >> document-match request (the first you were referring to). Even if lazily > >> loaded, reading 2000 id's (on 40 servers) and lazy loading the fields > is a > >> tough job. Waiting for the slowest shard to respond, then sorting the > docs > >> and reloading (lazy or not) the top 2000 docs might take a long time. > >> > >> Our times are 4-8 secs, but do it's not possible comparing cases. We've > >> done few steps that improved it along the way, steps that led to others. > >> These were our starters: > >> > >> 1. Profile these queries from different servers and solr instances, > >> try > >> putting your finger what collection is working hard and why. Check > if > >> you're stuck on components that don't have an added value for you > but > >> are > >> used by default. > >> 2. Consider eliminating the doc cache. It loads lots of (partly) > lazy > >> documents that their probability of secondary usage is low. There's > >> no such > >> thing "popular docs" when requesting so many docs. You may be using > >> your > >> memory in a better way. > >> 3. Bottleneck check - inner server metrics as cpu user / iowait, > >> packets > >> transferred over the network, page faults etc. are excellent in > order > >> to > >> understand if the disk/network/cpu is slowing you down. Then upgrade > >> hardware in one of the shards to check if it helps by looking at the > >> upgraded shard qTime compared to other. > >> 4. Warm up the index after commiting - try to benchmark how do > queries > >> performs before and after some warm-up, let's say some few hundreds > of > >> queries (from your previous system) in order to warm up the os cache > >> (assuming your using NRTDirectoryFactory) > >> > >> > >> Good luck, > >> Manu > >> > >> > >> On Wed, Nov 13, 2013 at 2:38 PM, Erick Erickson < > erickerick...@gmail.com> > >> wrote: > >> > >> One thing you can try, and this is more diagnostic than a cure, is > >>> return just the id field (and insure that lazy field loading is true). > >>> That'll tell you whether the issue is actually fetching the document > >>> off disk and decompressing, although frankly that's unlikely since you > >>> can get your 5,000 rows from a single machine quickly. > >>> > >>> The code you found where Solr is spending its time, is that on the > >>> "routing" core or on the shards? I actually have a hard time > >>> understanding how that code could take a long time, doesn't seem > >>> right. > >>> > >>> You are transferring 5,000 docs across the network, so it's possible > >>> that your network is just slow, that's certainly a difference between > >>> the local and remote case, but that's a stab in the dark. > >>> > >>> Not much help I know, > >>> Erick > >>> > >>> > >>> > >>> On Wed, Nov 13, 2013 at 2:52 AM, Elran Dvir <elr...@checkpoint.com> > >>> wrote: > >>> > >>> Erick, Thanks for your response. > >>>> > >>>> We are upgrading our system using Solr. > >>>> We need to preserve old functionality. Our client displays 5K > >>>> document and groups them. > >>>> > >>>> Is there a way to refactor code in order to improve distributed > >>>> documents fetching? > >>>> > >>>> Thanks. > >>>> > >>>> -----Original Message----- > >>>> From: Erick Erickson [mailto:erickerick...@gmail.com] > >>>> Sent: Wednesday, October 30, 2013 3:17 AM > >>>> To: solr-user@lucene.apache.org > >>>> Subject: Re: distributed search is significantly slower than direct > >>>> > >>> search > >>> > >>>> You can't. There will inevitably be some overhead in the distributed > >>>> > >>> case. > >>> > >>>> That said, 7 seconds is quite long. > >>>> > >>>> 5,000 rows is excessive, and probably where your issue is. You're > >>>> having to go out and fetch the docs across the wire. Perhaps there > >>>> is some batching that could be done there, I don't know whether this > >>>> is one document per request or not. > >>>> > >>>> Why 5K docs? > >>>> > >>>> Best, > >>>> Erick > >>>> > >>>> > >>>> On Tue, Oct 29, 2013 at 2:54 AM, Elran Dvir <elr...@checkpoint.com> > >>>> > >>> wrote: > >>> > >>>> Hi all, > >>>>> > >>>>> I am using Solr 4.4 with multi cores. One core (called template) > >>>>> is my "routing" core. > >>>>> > >>>>> When I run > >>>>> > http://127.0.0.1:8983/solr/template/select?rows=5000&q=*:*&shards=127. > >>>>> 0.0.1:8983/solr/core1, > >>>>> it consistently takes about 7s. > >>>>> When I run > >>>>> http://127.0.0.1:8983/solr/core1/select?rows=5000&q=*:*, it > >>>>> consistently takes about 40ms. > >>>>> > >>>>> I profiled the distributed query. > >>>>> This is the distributed query process (I hope the terms are > accurate): > >>>>> When solr identifies a distributed query, it sends the query to > >>>>> the shard and get matched shard docs. > >>>>> Then it sends another query to the shard to get the Solr documents. > >>>>> Most time is spent in the last stage in the function "process" of > >>>>> "QueryComponent" in: > >>>>> > >>>>> for (int i=0; i<idArr.size(); i++) { > >>>>> int id = req.getSearcher().getFirstMatch( > >>>>> new Term(idField.getName(), > >>>>> idField.getType().toInternal(idArr.get(i)))); > >>>>> > >>>>> How can I make my distributed query as fast as the direct one? > >>>>> > >>>>> Thanks. > >>>>> > >>>>> > >>>> Email secured by Check Point > >>>> > >>>> > >> Email secured by Check Point > >> > > > > >
