[ https://issues.apache.org/jira/browse/HBASE-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Jasani resolved HBASE-24757. ---------------------------------- Hadoop Flags: Reviewed Resolution: Fixed > ReplicationSink should limit the batch size for batch mutations based on > hbase.rpc.rows.warning.threshold > --------------------------------------------------------------------------------------------------------- > > Key: HBASE-24757 > URL: https://issues.apache.org/jira/browse/HBASE-24757 > Project: HBase > Issue Type: Improvement > Reporter: Viraj Jasani > Assignee: Viraj Jasani > Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7 > > > At times there are quite a large no of WAL Edits to ship as part of > Replication and sometimes replication queues accumulate huge list of Edits to > process. ReplicationSink at the sink server usually goes through all Edits > and creates map of table -> list of rows grouped by clusterIds, and performs > batch mutation of all rows per table level. However, there is no limit to no > of Rows that are sent as part of batch mutate call. If no of rows > limit > threshold defined by hbase.rpc.rows.warning.threshold, we usually get warn > "Large batch operation detected". If hbase.rpc.rows.size.threshold.reject is > turned on, RS will reject the whole batch without processing. > We should let Replication Sink honour this threshold value and accordingly > keep the size lower per batch mutation call. > Replication triggered batch mutations should always be consumed but keeping > limit of mutation low enough will let the system function at the same pace > and without triggering redundant warnings. This will also restrict > exploitation of cpu cycles at the destination server. -- This message was sent by Atlassian Jira (v8.3.4#803005)