[ https://issues.apache.org/jira/browse/HBASE-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162908#comment-17162908 ]
Wellington Chevreuil commented on HBASE-24757: ---------------------------------------------- Wasn't this supposedly addressed by HBASE-18027? > ReplicationSink should limit the batch size for batch mutations based on > hbase.rpc.rows.warning.threshold > --------------------------------------------------------------------------------------------------------- > > Key: HBASE-24757 > URL: https://issues.apache.org/jira/browse/HBASE-24757 > Project: HBase > Issue Type: Improvement > Reporter: Viraj Jasani > Assignee: Viraj Jasani > Priority: Major > > At times there are quite a large no of WAL Edits to ship as part of > Replication and sometimes replication queues accumulate huge list of Edits to > process. ReplicationSink at the sink server usually goes through all Edits > and creates map of table -> list of rows grouped by clusterIds, and performs > batch mutation of all rows per table level. However, there is no limit to no > of Rows that are sent as part of batch mutate call. If no of rows > limit > threshold defined by hbase.rpc.rows.warning.threshold, we usually get warn > "Large batch operation detected". If hbase.rpc.rows.size.threshold.reject is > turned on, RS will reject the whole batch without processing. > We should let Replication Sink honour this threshold value and accordingly > keep the size lower per batch mutation call. > Replication triggered batch mutations should always be consumed but keeping > limit of mutation low enough will let the system function at the same pace > and without triggering warnings. This will also restrict exploitation of heap > and cpu cycles at the destination. -- This message was sent by Atlassian Jira (v8.3.4#803005)