Interesting info.
You should look into using Solid State Drives. I moved my search engine to
SSD and saw dramatic improvements.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Huge-Performance-Solr-distributed-search-tp3530627p346.html
Sent from the Solr - User
Problem has been resolved. My disk subsystem been a bottleneck for quick search.
I put my indexes to RAM and I see very nice QTimes :)
Sorry for your time, guys.
On Mon, Nov 28, 2011 at 4:02 PM, Artem Lokotosh wrote:
> Hi all again. Thanks to all for your replies.
>
> On this weekend I'd made som
Hi all again. Thanks to all for your replies.
On this weekend I'd made some interesting tests, and I would like to share it
with you.
First of all I made speed test of my hdd:
root@LSolr:~# hdparm -t /dev/sda9
/dev/sda9:
Timing buffered disk reads: 146 MB in 3.01 seconds = 48.54 MB/se
in general terms, when your Java heap is so large, it is beneficial to
set mx and ms to the same size.
On Wed, Nov 23, 2011 at 5:12 AM, Artem Lokotosh wrote:
> Hi!
>
> * Data:
> - Solr 3.4;
> - 30 shards ~ 13GB, 27-29M docs each shard.
>
> * Machine parameters (Ubuntu 10.04 LTS):
> user@Solr:~$ u
On 11/25/2011 3:13 AM, Mark Miller wrote:
When you search each shard, are you positive that you are using all of the
same parameters? You are sure you are hitting request handlers that are
configured exactly the same and sending exactly the same queries?
I'm my experience, the overhead for dist
45 000 000 per shard approx, Tomcat, caching was tweaked in solrconfig and
shard given 12GB of RAM max.
filterCache class="solr.FastLRUCache" size="1200" initialSize="1200"
autowarmCount="128"/>
true
50
200
In you case I would first check if the network throu
On Thu, Nov 24, 2011 at 12:09 PM, Artem Lokotosh wrote:
> >How big are the documents you return (how many fields, avg KB per doc,
> etc.)?
> I have a following schema in my solr configuration name="field1" type="text" indexed="true" stored="false"/> name="field2" type="text" indexed="true" stored
>How big are the documents you return (how many fields, avg KB per doc, etc.)?
I have a following schema in my solr configuration
27M–30M docs and 12-15 GB for each shard, 0.5KB per doc
>Does performance get much better if you only request top 100, or top>10
>documents instead of top 1000?
>> Can you merge, e.g. 3 shards together or is it much effort for your
>> team?>Yes, we can merge. We'll try to do this and review how it will works
Merge does not help :(I've tried to merge two shards in one, three
shards in one, but results are similar to results first configuration
with 30 shar
If you request 1000 docs from each shard, then aggregator is really
fetching 30,000 total documents, which then it must merge (re-sort
results, and take top 1000 to return to client). Its possible that
SOLR merging implementation needs optimized, but it does not seem like
it could be that slow. H
> If the response time from each shard shows decent figures, then aggregator>
> seems to be a bottleneck. Do you btw have a lot of concurrent users?For now
> is not a problem, but we expect from 1K to 10K of concurrent users and maybe
> more
On Wed, Nov 23, 2011 at 4:43 PM, Dmitry Kan wrote:
>
If the response time from each shard shows decent figures, then aggregator
seems to be a bottleneck. Do you btw have a lot of concurrent users?
On Wed, Nov 23, 2011 at 4:38 PM, Artem Lokotosh wrote:
> > Is this log from the frontend SOLR (aggregator) or from a shard?
> from aggregator
>
> > Can
> Is this log from the frontend SOLR (aggregator) or from a shard?
from aggregator
> Can you merge, e.g. 3 shards together or is it much effort for your team?
Yes, we can merge. We'll try to do this and review how it will works
Thanks, Dmitry
Any another ideas?
On Wed, Nov 23, 2011 at 4:01 PM, D
Hello,
Is this log from the frontend SOLR (aggregator) or from a shard?
Can you merge, e.g. 3 shards together or is it much effort for your team?
In our setup we currently have 16 shards with ~30GB each, but we rarely
search in all of them at once.
Best,
Dmitry
On Wed, Nov 23, 2011 at 3:12 PM,
Hi!
* Data:
- Solr 3.4;
- 30 shards ~ 13GB, 27-29M docs each shard.
* Machine parameters (Ubuntu 10.04 LTS):
user@Solr:~$ uname -a
Linux Solr 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011
x86_64 GNU/Linux
user@Solr:~$ cat /proc/cpuinfo
processor : 0 - 3
vendor_id : Genui
15 matches
Mail list logo
