Hello, Is this log from the frontend SOLR (aggregator) or from a shard? Can you merge, e.g. 3 shards together or is it much effort for your team?
In our setup we currently have 16 shards with ~30GB each, but we rarely search in all of them at once. Best, Dmitry On Wed, Nov 23, 2011 at 3:12 PM, Artem Lokotosh <arco...@gmail.com> wrote: > Hi! > > * Data: > - Solr 3.4; > - 30 shards ~ 13GB, 27-29M docs each shard. > > * Machine parameters (Ubuntu 10.04 LTS): > user@Solr:~$ uname -a > Linux Solr 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011 > x86_64 GNU/Linux > user@Solr:~$ cat /proc/cpuinfo > processor : 0 - 3 > vendor_id : GenuineIntel > cpu family : 6 > model : 44 > model name : Intel(R) Xeon(R) CPU X5690 @ 3.47GHz > stepping : 2 > cpu MHz : 3458.000 > cache size : 12288 KB > fpu : yes > fpu_exception : yes > cpuid level : 11 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge > mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx > rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology > tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 > sse4_2 popcnt aes hypervisor lahf_lm ida arat > bogomips : 6916.00 > clflush size : 64 > cache_alignment : 64 > address sizes : 40 bits physical, 48 bits virtual > power management: > user@Solr:~$ cat /proc/meminfo > MemTotal: 16992680 kB > MemFree: 110424 kB > Buffers: 9976 kB > Cached: 11588380 kB > SwapCached: 41952 kB > Active: 9860764 kB > Inactive: 6198668 kB > Active(anon): 4062144 kB > Inactive(anon): 398972 kB > Active(file): 5798620 kB > Inactive(file): 5799696 kB > Unevictable: 0 kB > Mlocked: 0 kB > SwapTotal: 46873592 kB > SwapFree: 46810712 kB > Dirty: 36 kB > Writeback: 0 kB > AnonPages: 4424756 kB > Mapped: 940660 kB > Shmem: 40 kB > Slab: 362344 kB > SReclaimable: 350372 kB > SUnreclaim: 11972 kB > KernelStack: 2488 kB > PageTables: 68568 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 55369932 kB > Committed_AS: 5740556 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 350532 kB > VmallocChunk: 34359384964 kB > HardwareCorrupted: 0 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > DirectMap4k: 10240 kB > DirectMap2M: 17299456 kB > > - Apache Tomcat 6.0.32: > <!-- java arguments --> > -XX:+DisableExplicitGC > -XX:PermSize=512M > -XX:MaxPermSize=512M > -Xmx12G > -Xms3G > -XX:NewSize=128M > -XX:MaxNewSize=128M > -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC > -XX:+CMSClassUnloadingEnabled > -XX:CMSInitiatingOccupancyFraction=50 > -XX:GCTimeRatio=9 > -XX:MinHeapFreeRatio=25 > -XX:MaxHeapFreeRatio=25 > -verbose:gc > -XX:+PrintGCTimeStamps > -Xloggc:/opt/search/tomcat/logs/gc.log > > Out search schema is: > - 5 servers with configuration above; > - one tomcat6 application on each server with 6 solr applications. > > - Full addresses are: > 1) http://192.168.1.85:8080/solr1,http://192.168.1.85:8080/solr2,..., > http://192.168.1.85:8080/solr6 > 2) http://192.168.1.86:8080/solr7,http://192.168.1.86:8080/solr8,..., > http://192.168.1.86:8080/solr12 > ... > 5) http://192.168.1.89:8080/solr25,http://192.168.1.89:8080/solr26,..., > http://192.168.1.89:8080/solr30 > - At another server there is a additional "common" application with > shards paramerter: > <requestHandler name="search" class="solr.SearchHandler" default="true"> > <lst name="defaults"> > <str name="echoParams">explicit</str> > <str name="shards">192.168.1.85:8080/solr1,192.168.1.85:8080/solr2,..., > 192.168.1.89:8080/solr30</str> > <int name="rows">10</int> > </lst> > </requestHandler> > - schema and solrconfig are identical for all shards, for first shard > see attach; > - on these servers are only search, indexation is on another > (optimized to 2 segments shards replicate with ssh/rsync scripts). > > So now the major problem is huge performance on distributed search. > Take look on, for example, these logs: > This is on 30 shards: > INFO: [] webapp=/solr > path=/select/params={fl=*,score&ident=true&start=0&q=(barium)&rows=2000} > status=0 QTime=40712 > INFO: [] webapp=/solr > path=/select/params={fl=*,score&ident=true&start=0&q=(pittances)&rows=2000} > status=0 QTime=36097 > INFO: [] webapp=/solr > > path=/select/params={fl=*,score&ident=true&start=0&q=(reliability)&rows=2000} > status=0 QTime=75756 > INFO: [] webapp=/solr > > path=/select/params={fl=*,score&ident=true&start=0&q=(blessing's)&rows=2000} > status=0 QTime=30342 > INFO: [] webapp=/solr > > path=/select/params={fl=*,score&ident=true&start=0&q=(reiterated)&rows=2000} > status=0 QTime=55690 > > Sometimes QTime is more than 150000. But when we run identical queries > on one shard separately, QTime is between 200 and 1500. > Does ditributed solr search really slow or our architecture is non > optimal? Or maybe need to use any third-party applications? > Thanks for any replies. > > -- > Best regards, > Artem >
