Luke,
I tried to set IN MEMORY access group option for RandomTest table.
The result for random read test improved by ~5x:
random_read_test 1000000000
0% 10 20 30 40 50 60 70 80 90 100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Elapsed time: 1971.06 s
Total scanned: 1000000
Throughput: 513430.49 bytes/s
Throughput: 507.34 scanned cells/s
However, it is still ~2x worse than the number from BigTable paper for
non-mem tables and ~20x worse than mem-table.
Thanks,
Alex
On Oct 19, 2:18 pm, Luke <[EMAIL PROTECTED]> wrote:
> The random_read_test used to score 4k qps for comparable benchmark (vs
> 1.2k qps in the bigtable paper, note the 10k qps number is for
> memtable, which is very different from regular table. In hypertable
> you can use the IN MEMORY access group option to get memtable) The
> regular table scanner needs to merge scan cell cache, cell stores, so
> it's much more expensive than memtable scanner which just scan the
> cellcache, regardless the size of the table.
>
> Doug pushed out .11, which contains major changes to the way cellcache
> and compaction works before he went on a vacation. There might be a
> performance regression in recent releases. Thanks for the note, we'll
> look into it.
>
> __Luke
>
> On Oct 16, 9:59 pm, Alex <[EMAIL PROTECTED]> wrote:
>
> > Hi All,
>
> > Could somebody explain/describe CellCache allocation/eviction policy?
>
> > After running random read/write tests, I came to the conclusion that
> > CellCache operation is very different from what is described in Google
> > BigTable paper, i.e. CellCache works as a write buffer rather than a
> > cache. CellCache seems to help a lot for optimizing writes but it
> > doesn't help reads.
>
> > Here are the results:
>
> > random_write_test 100000000
>
> > Elapsed time: 8.16 s
> > Total inserts: 100000
> > Throughput: 12403559.87 bytes/s
> > Throughput: 12256.48 inserts/s
>
> > random_read_test 100000000
>
> > Elapsed time: 1038.47 s
> > Total scanned: 100000
> > Throughput: 97451.43 bytes/s
> > Throughput: 96.30 scanned cells/s
>
> > Random read speed is ~100x slower than the result in Google BigTable
> > for random read test which fits in the memory. In this case the data
> > set size should be ~100MB and should comfortably fit in the DRAM
> > (8GB).
>
> > Also, tcmalloc heap profiling shows that the usage memory actually
> > decreases to ~50MB during random read test while it is >700MB during
> > random write test (although top instead shows increase).
>
> > I apologize if I am missing something very basic, I have very little
> > experience in this area.
>
> > Thanks,
> > Alex
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
