The problem is OOM at reducer copy phase. Some general suggestions. 1) Increase JVM heap size of reducer. 2) Set lesser "shuffle.input.buffer.percentage" in mr job config. Reducer will use less mem and more disk to do shuffling. 3) Increase the number of reducers. Set a smaller " kylin.job.mapreduce.default.reduce.input.mb" in kylin.properties, e.g. 500. Kylin will estimate shuffle size and allocate one reducer for every 500 MB shuffle data. This way you can control how much data be handled by each reducer.
Try 3) and 1) first. On Wed, Nov 11, 2015 at 10:36 AM, 杨海乐 <[email protected]> wrote: > 数据量是500G，15台机器，每台hadoop节点内存调整的是89G ，hbase集群共五台，内存是18G，500G测试数据共5163328507 > 条数据，一张事实表，一共7个维度，这里没有设置维度表 > Kylin有关内存的配置： > > kylin.job.mapreduce.default.reduce.input.mb=8192//这个参数不太确定是做什么的，如果太小会引发什么后果 > > 报错： > Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error > in shuffle in fetcher#3 > at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.OutOfMemoryError: Java heap space at > org.apache.hadoop.io.BoundedByteArrayOutputStream. > <init>(BoundedByteArrayOutputStream.java:56)at > org.apache.hadoop.io.BoundedByteArrayOutputStream. > <init>(BoundedByteArrayOutputStream.java:46) at > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63) > at > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:304) > at > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:294) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:511) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:329) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193) > > >
