On Thu, Jan 13, 2011 at 10:10 PM, supersoft <elarab...@gmail.com> wrote: > > On the one hand, I found really interesting those comments about the reasons > for sharding. Documentation agrees you about why to split an index in > several shards (big sizes problems) but I don't find any explanation about > the inconvenients as an Access Control List. I guess there should be some > and they can be critical in this design. Any example? [...]
Can I ask what might be a stupid question? How are you measuring the numbers below, and what do they mean? As your hit ratio is close to 1 (i.e., everything after the first query is coming from the cache), these numbers seem a little strange. Are these really the time for each of the N simultaneous queries? They seem to be monotonically increasing (though with a couple of strange exceptions), which leads me to suspect that they are some kind of cumulative times, e.g., by this interpretation, for the case of the 10 simultaneous queries, the first one takes 1047ms, the second 268ms, the third 125ms, and so on. We have run performance tests with pg_bench on a index of size 40GB on a single Solr server with about 6GB of RAM allocated to Solr, and see what I would think of as expected behaviour, i.e., for every fresh query term, the first query takes the longest, and the time for subsequent queries with the same term goes down dramatically, as the result is coming out of the cache. This is at odds to what you describe here, so I have to go back and check that we did not miss something important. > 1 simultaneous query: 3437 ms (cache fails) > > 2 simultaneous queries: 594, 954 ms > > 10 simultaneous queries: 1047, 1313, 1438, 1797, 1922, 2094, 2250, 2500, > 2938, 3000 ms > > 50 simultaneous queries: 1203, 1453, 1453, 1437, 1625, 1953, 5688, 12938, > 14953, 16281, 15984, 16453, 15812, 16469, 16563, 16844, 17703, 16843, 17359, > 16828, 18235, 18219, 18172, 18203, 17672, 17344, 17453, 18484, 18157, 18531, > 18297, 18359, 18063, 18516, 18125, 17516, 18562, 18016, 18187, 18610, 18703, > 18672, 17829, 18344, 18797, 18781, 18265, 18875, 18250, 18812 > > 100 simultaneous queries: 1297, 1531, 1969, 2203, 2375, 2891, 3937, 4109, > 4703, 4890, 5047, 5312, 5563, 6422, 6437, 7063, 7093, 7391, 7594, 7672, > 8172, 8547, 8750, 8984, 9265, 9609, 9907, 10344, 11406, 11484, 11484, 11500, > 11547, 11703, 11797, 11875, 11922, 12328, 12375, 12875, 12922, 13187, 13219, > 13407, 13500, 13562, 13719, 13828, 13875, 14016, 14078, 14672, 15922, 16328, > 16625, 16953, 17282, 18172, 18484, 18985, 20594, 20625, 20860, 21281, 21469, > 21625, 21875, 21875, 22141, 22157, 22172, 23125, 23125, 23141, 23203, 23203, > 23328, 24625, 24641, 24672, 24797, 24985, 25031, 25188, 25844, 25937, 26016, > 26437, 26453, 26437, 26485, 28297, 28687, 31782, 31985, 31969, 32016, 32031, > 32016, 32281 ms [...] Regards, Gora
