in this case i pool them as well, which doesn't seem to make any difference (compared to when i just reuse them -- but i am not writing but outside of the test i do so i do pool them using techniques similar to those in HTablePool, CAS-based queues etc. )
On Thu, Apr 21, 2011 at 11:09 PM, Ted Dunning <tdunn...@maprtech.com> wrote: > Dmitriy, > > Did I hear you say that you are instantiating a new Htable for each request? > Or was that somebody else? > > On Thu, Apr 21, 2011 at 11:04 PM, Stack <st...@duboce.net> wrote: > >> On Thu, Apr 21, 2011 at 10:49 PM, Dmitriy Lyubimov <dlie...@gmail.com> >> wrote: >> > Anyway. For a million requests shot at a region server at various >> > speeds between 300 and 500 qps the picture is not pretty. RPC metrics >> > are arctually good -- no more than 1ms average per next() and 0 per >> > get(). So region server is lightning fast. >> > >> >> This is 3-500 queries per second of 40 rows each? >> >> > What doesn't seem so fast is RPC. >> >> OK. >> >> > As i reported before, i was getting >> > 25ms TTLB under the circumstances. In this case all the traffic to the >> > node goes thru same client (but in reality of course the node's >> > portion per client should be much less). All that traffic is using >> > single regionserver node rpc queue as HConnection would not open more >> > than one socket to same region. >> >> You saw "HBASE-2939 Allow Client-Side Connection Pooling"? Would that >> help? >> >> >> > And tcp doesn't seem to perform very >> > well for some reason in this scenario. >> > >> >> What do you see here D? >> >> >> > The next thing i did was to enable tcp_nodelay on both client and >> > server. That got us down even more to 13ms average. >> > >> >> Thats a big difference. >> >> >> > However, it is still about two times slower if i run all processes at >> > the same machine (i get around 6-7ms average TTLBs for the same type >> > of scan). >> > >> > Ping time for about same packet size between hosts involved seems to >> > revolve around 1ms. Where another 5ms average time are getting lost is >> > still a mystery. But oh well i guess it is as good as it gets. >> > In real life hbase applications traffic would be much more uniformly >> > distributed among regions and this would be much less of an issue >> > perhaps. >> > >> > I also suspect that using udp for short scans and gets might reduce >> > latency a bit as well. >> > >> >> Thank you Dmitriy for digging in. Good stuff. >> St.Ack >> >
