[ https://issues.apache.org/jira/browse/HBASE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451445#comment-15451445 ]
Devaraj Das edited comment on HBASE-16255 at 8/31/16 7:30 AM: -------------------------------------------------------------- Sorry, I was off on my previous guess at the issue. I dug deeper and this seems like a yarn issue to do with the minimum memory for the containers. The default heap size (-Xmx200m) for the tasks was too low ([~dspivak], curious, if you ran other IT tests that do mapreduce, and did/didn't see this issue) I added the following in yarn-site.xml: {noformat} <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>mapreduce.map.memory.mb</name> <value>1024</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>1024</value> </property> <property> <name>mapred.child.java.opts</name> <value>-Xmx1024m</value> </property> <property> <name>mapred.map.child.java.opts</name> <value>-Xmx1024m</value> </property> <property> <name>mapred.reduce.child.java.opts</name> <value>-Xmx1024m</value> </property> {noformat} And in mapred-site.xml, added the following: {noformat} <property> <name>mapred.map.child.java.opts</name> <value>-Xmx1024m</value> </property> <property> <name>mapred.reduce.child.java.opts</name> <value>-Xmx1024m</value> </property> <property> <name>mapred.child.java.opts</name> <value>-Xmx1024m</value> </property> {noformat} At the time of this writing, the test was still running (it proceeded beyond your failure point [~dspivak]). Fingers crossed. was (Author: devaraj): Sorry, I was off on my previous guess at the issue. I dug deeper and this seems like a yarn issue to do with the minimum memory for the containers. The default heap size (-Xmx200m) for the tasks was too low ([~dspivak], curious, if you ran other IT tests that do mapreduce, and did/didn't see this issue) I added the following in yarn-site.xml: {noformat} <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>mapreduce.map.memory.mb</name> <value>1024</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>1024</value> </property> <property> <name>mapred.child.java.opts</name> <value>-Xmx1024m</value> </property> <property> <name>mapred.map.child.java.opts</name> <value>-Xmx1024m</value> </property> <property> <name>mapred.reduce.child.java.opts</name> <value>-Xmx1024m</value> </property> </configuration> {noformat} And in mapred-site.xml, added the following: {noformat} <property> <name>mapred.map.child.java.opts</name> <value>-Xmx1024m</value> </property> <property> <name>mapred.reduce.child.java.opts</name> <value>-Xmx1024m</value> </property> <property> <name>mapred.child.java.opts</name> <value>-Xmx1024m</value> </property> {noformat} At the time of this writing, the test was still running (it proceeded beyond your failure point [~dspivak]). Fingers crossed. > Backup/Restore IT > ----------------- > > Key: HBASE-16255 > URL: https://issues.apache.org/jira/browse/HBASE-16255 > Project: HBase > Issue Type: Task > Reporter: Vladimir Rodionov > Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16255-addendum.3.txt, 16255.addendum, 16255.addendum2, > 16255.addendum4, 16255.addendum5, 16255.addendum6, HBASE-16255-v1.patch, > HBASE-16255-v2.patch, HBASE-16255-v3.patch, HBASE-16255-v4.patch, > HBASE-16255-v5.patch, HBASE-16255-v6.patch, backup-it-7912-8-30.out, > backup-it-8-30.out, backup-it-success.out > > > Integration test for backup restore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)