I have a mapreduce task put together for experimentation which does a lot of
Increments over three tables and Puts to another. I set writeToWAL to false. My
HBase includes the patch that fixes serialization of writeToWAL for Increments.
MemstoreLAB is enabled but is probably not a factor, but still need to test to
exclude it.
After starting a job up on a test cluster on EC2 with 20 mappers over 10 slaves
I see initially 10-15K/ops/sec/server. This performance drops over a short time
to stabilize around 1K/ops/sec/server. So I flush the tables with the shell.
Immediately after flushing the tables, performance is back up to
10-15K/ops/sec/server. If I don't flush, performance remains low indefinitely.
If I flush only the table receiving the Gets, performance remains low.
If I set the shell to flush in a loop every 60 seconds, performance repeatedly
drops during that interval, then recovers after flushing.
When Gary and I went to NCHC in Taiwan, we saw a guy from PhiCloud present
something similar to this regarding 0.89DR. He measured the performance of the
memstore for a get-and-put use case over time and graphed it, looked like time
increased on a staircase with a trend to O(n). This was a surprising result.
ConcurrentSkipListMap#put is supposed to run in O(log n). His workaround was to
flush after some fixed number of gets+puts, 1000 I think. At the time we
weren't sure what was going on given the language barrier.
Sound familiar?
I don't claim to really understand what is going on, but need to get to the
bottom of this. Going to look at it in depth starting Monday.
- Andy
