[ https://issues.apache.org/jira/browse/HBASE-10094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-10094: -------------------------- Attachment: 10094v2.txt Add some metrics on sync and rate at which we are writing. Below is sample of what we have now going against localfs and then same passing a syncInterval flag set to 100: org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation: append: count = 680000000 mean rate = 67244.75 bytes/ms 1-minute rate = 52488.70 bytes/ms 5-minute rate = 50267.51 bytes/ms 15-minute rate = 49882.70 bytes/ms sync: count = 1000000 mean rate = 98.88 syncs/ms 1-minute rate = 77.16 syncs/ms 5-minute rate = 73.89 syncs/ms 15-minute rate = 73.33 syncs/ms # Every 100 appends. org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation: append: count = 585820000 mean rate = 85046.52 bytes/ms 1-minute rate = 74908.80 bytes/ms 5-minute rate = 74908.80 bytes/ms 15-minute rate = 74908.80 bytes/ms sync: count = 8643 mean rate = 1.25 syncs/ms 1-minute rate = 1.11 syncs/ms 5-minute rate = 1.11 syncs/ms 15-minute rate = 1.11 syncs/ms We probably want to settle somewhere at syncing about every 10ms or so. > Add batching to HLogPerformanceEvaluation > ----------------------------------------- > > Key: HBASE-10094 > URL: https://issues.apache.org/jira/browse/HBASE-10094 > Project: HBase > Issue Type: Sub-task > Components: Performance, wal > Reporter: stack > Assignee: Himanshu Vashishtha > Attachments: 10094v2.txt > > > As Himanshu points out in the the parent issue, HLogPE is using an unorthodox > API appending edits to the WAL; it is using an API that is meant for tests > only that does an append immediately followed by a sync call. > In normal deploy, WAL appends are done as a bunch of appends followed by a > sync on the tail of the transaction -- not a sync per append. > This issue is about changing HLogPE to use append and then sync. It also > adds an argument so you can specifying batching of a set of appends before > the sync is called. The latter lets HLogPE mimic multi puts that use the > minibatch... which appends, appends, appends.. and then syncs. > Assigning to Himanshu for review. -- This message was sent by Atlassian JIRA (v6.1#6144)