I had the same issue. I ended up writing a coprocessor that did this under the BatchMutate hood using a special attribute in the Put operation, and that increased performance significantly to be almost equal to the regular batchMutate(Put) version.
On Sat, Jul 9, 2016 at 7:59 AM, 陆巍 <luwei...@163.com> wrote: > The performance of batch(Increment) was poor, which is why I do the test. > > > I use spark stream to process data per one second (for about 50 records) > and use HBase for aggregations (sum, max, min etc). All the spark executors > are running on the same machines as the RegionServers, so the network env > is fine. > In the hbase caller, there are Put, CheckAndPut, and Increment operations. > Put and Increment are called thru Table.batch(...). > Put and CheckAndPut cost less was around 50ms, but Increment costs around > 1 minute. > > > > At 2016-07-09 22:34:05, la...@apache.org wrote: > >Note that you're doing single RPCs for each Increment. You should batch > them (see HTable.batch(...)) > > > >What version of HBase? What's the ping time between the client and server > machines?Is Nagle's disables (tcpnodelay set to true)? Otherwise each RPC > will take at least 45 ms. > > > >Lastly you'll likely hit the same server for every request in a > microbenchmark like this. > > > >You are also not actually incrementing anything since each one is a new > row (I realize this is just a test),so to test whether this is related to > Increment, can you replace the Increment by Put and see how long that takes? > > > >-- Lars > > > > From: 陆巍 <luwei...@163.com> > > To: user@hbase.apache.org > > Sent: Saturday, July 9, 2016 3:14 AM > > Subject: Poor performance of table Increment > > > >Hi, > > > >I had a test for Increment operation, and find the performance is really > bad: 94809ms for 1000 increment operaions. > >The testing cluster is pretty small with only three region servers. > > > >Below is the code for testing. Is there any bentchmark result for htable > increment? Thanks! > > > >import java.io.IOException; > >import java.util.Random; > > > >import org.apache.hadoop.conf.Configuration; > >import org.apache.hadoop.hbase.HBaseConfiguration; > >import org.apache.hadoop.hbase.TableName; > >import org.apache.hadoop.hbase.client.Connection; > >import org.apache.hadoop.hbase.client.ConnectionFactory; > >import org.apache.hadoop.hbase.client.Increment; > >import org.apache.hadoop.hbase.client.Table; > >import org.apache.hadoop.hbase.util.Bytes; > > > >class HBaseTableFactory { > > private Configuration config = HBaseConfiguration.create(); > > private Connection connection = null; > > > > public Table createTable(String tableName) throws IOException { > > if (connection == null) { > > connection = ConnectionFactory.createConnection(config); > > } > > return create(tableName, config); > > } > > > > protected Table create(String tableName, Configuration config) > > throws IOException { > > return connection.getTable(TableName.valueOf(tableName)); > > } > > > >} > > > >public class TestBytes { > > public static void main(String[] args) throws IOException { > > long t1 = System.currentTimeMillis(); > > Table table = new HBaseTableFactory().createTable("xyz"); > > for (int i = 0; i < 1000; i++) { > > > > byte[] row = Bytes.toBytes("row-" + i); > > Increment p = new Increment(row); > > byte[] f = Bytes.toBytes("cf1"); > > byte[] q = Bytes.toBytes("q1"); > > long v = 1L; > > p.addColumn(f, q, v); > > table.increment(p); > > System.out.println(i); > > } > > table.close(); > > System.out.println(System.currentTimeMillis() - t1); > > } > >} > > > > > > >