Shinya Yoshida created HBASE-26238:
--------------------------------------
Summary: OOME in VerifyReplication for the table contains rows
with 10M+ cells
Key: HBASE-26238
URL: https://issues.apache.org/jira/browse/HBASE-26238
Project: HBase
Issue Type: Bug
Reporter: Shinya Yoshida
Assignee: Shinya Yoshida
We have a table that contains rows with 10M+ cells.
When we run VerifyReplication for that table, we got OOME.
VerifyReplication cannot be complete without OOME even though we provide 31GB
heap for each mapper despite of RS can handle such get request.
{code:java}
org.apache.hadoop.mapred.YarnChild: Error running child :
java.lang.OutOfMemoryError
at
java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
at
java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at org.apache.hadoop.hbase.client.Result.compareResults(Result.java:844)
at
org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier.map(VerifyReplication.java:184)
at
org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier.map(VerifyReplication.java:95)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
{code}
The interesting thing is always failing at
AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161) in
Result.compareResults.
This is an application-side OOME and caused by the max size of Java array, so
we cannot avoid this error whatever heap size we use.
{code:java}
private int hugeCapacity(int minCapacity) {
if (Integer.MAX_VALUE - minCapacity < 0) { // overflow
throw new OutOfMemoryError();
}
return (minCapacity > MAX_ARRAY_SIZE)
? minCapacity : MAX_ARRAY_SIZE;
}
{code}
When we see Result.compareResults, it generates a string representation of all
cells in 2 results for the exception message.
This could be a very large string despite cells in the result could consist
from multiple byte arrays and are well optimized in the heap.
(Repeated rowkeys in string, 4 chars for 1 byte if it cannot be represented by
ascii char, long timestamp(8 bytes) vs string timestamp(13 bytes so far), and
so on.)
{code:java}
public static void compareResults(Result res1, Result res2)
throws Exception {
if (res2 == null) {
throw new Exception("There wasn't enough rows, we stopped at "
+ Bytes.toStringBinary(res1.getRow()));
}
if (res1.size() != res2.size()) {
throw new Exception("This row doesn't have the same number of KVs: "
+ res1.toString() + " compared to " + res2.toString());
}
Cell[] ourKVs = res1.rawCells();
Cell[] replicatedKVs = res2.rawCells();
for (int i = 0; i < res1.size(); i++) {
if (!ourKVs[i].equals(replicatedKVs[i]) ||
!CellUtil.matchingValue(ourKVs[i], replicatedKVs[i]) ||
!CellUtil.matchingTags(ourKVs[i], replicatedKVs[i])) {
throw new Exception("This result was different: "
+ res1.toString() + " compared to " + res2.toString());
}
}
}
{code}
In VerifyReplication, the exception thrown is never used.
So this message is useless and a white elephant for VerifyReplication and us.
Can we provide a version that produces lightweight message (or returns boolean
instead of exception, but I think it's confusing similar method that returns
boolean and another throws exception)?
(Such hot row can have mutations often and be considered as inconsistent due to
timing issues, so difficult to avoid this OOME by just a luck)
{code:java}
try {
Result.compareResults(value, currentCompareRowInPeerTable);
context.getCounter(Counters.GOODROWS).increment(1);
if (verbose) {
LOG.info("Good row key: " + delimiter
+ Bytes.toStringBinary(value.getRow()) + delimiter);
}
} catch (Exception e) {
logFailRowAndIncreaseCounter(context,
Counters.CONTENT_DIFFERENT_ROWS, value);
}
try {
Result sourceResult = sourceTable.get(new Get(row.getRow()));
Result replicatedResult = replicatedTable.get(new Get(row.getRow()));
Result.compareResults(sourceResult, replicatedResult);
if (!sourceResult.isEmpty()) {
context.getCounter(Counters.GOODROWS).increment(1);
if (verbose) {
LOG.info("Good row key (with recompare): " + delimiter +
Bytes.toStringBinary(row.getRow())
+ delimiter);
}
}
return;
} catch (Exception e) {
LOG.error("recompare fail after sleep, rowkey=" + delimiter +
Bytes.toStringBinary(row.getRow()) + delimiter);
}
{code}
VerifyReplication with the patch that changes the message to include rowkey
only can handle our table without OOME and evel smaller heap.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
