Certain disk is full: Caused by: java.io.IOException: *No space left on device at* sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) at
2017-05-15 15:05 GMT+08:00 suresh m <suresh.m8...@gmail.com>: > Hi All, > > Trying to build cube but failing at building cube stage in process. I can > see mapper after more attempts its getting failed. > > Sending you the error message, please help me to resolve, > > attempt_1493717156723_1006_m_000000_0 100.00 FAILED > hdfs://IALERT2HDP/kylin/kylin_metadata/kylin-ce08d124-99a1- > 4ee8-aa71-c02c8285dc1e/kylin_intermediate_AL_SFTY_RPT_CUBE_ > e00df75e_7cdb_4adc_af8d_ecb9db2bc8ab/000000_0:0+214404 > > map pm10hdn02.ashokleyland.com:8042 logs > <http://pm10hdn02.ashokleyland.com:8042/node/containerlogs/container_e35_ > 1493717156723_1006_01_000002/hdfs> > Mon > May 15 11:17:53 +0550 2017 Mon May 15 11:21:32 +0550 2017 3mins, 38sec > Error: > java.io.IOException: Failed to build cube in mapper 0 at > org.apache.kylin.engine.mr.steps.InMemCuboidMapper. > cleanup(InMemCuboidMapper.java:145) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) at > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1724) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: > java.util.concurrent.ExecutionException: java.lang.RuntimeException: > java.io.IOException: java.io.IOException: java.lang.RuntimeException: > java.io.IOException: 4 exceptions during in-mem cube build, cause set to > the first, check log for more at > java.util.concurrent.FutureTask.report(FutureTask.java:122) at > java.util.concurrent.FutureTask.get(FutureTask.java:192) at > org.apache.kylin.engine.mr.steps.InMemCuboidMapper. > cleanup(InMemCuboidMapper.java:143) > ... 8 more Caused by: java.lang.RuntimeException: java.io.IOException: > java.io.IOException: java.lang.RuntimeException: java.io.IOException: 4 > exceptions during in-mem cube build, cause set to the first, check log for > more at > org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1. > run(AbstractInMemCubeBuilder.java:82) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: > java.io.IOException: java.lang.RuntimeException: java.io.IOException: 4 > exceptions during in-mem cube build, cause set to the first, check log for > more at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build( > DoggedCubeBuilder.java:126) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder. > build(DoggedCubeBuilder.java:73) > at > org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1. > run(AbstractInMemCubeBuilder.java:80) > ... 5 more Caused by: java.io.IOException: java.lang.RuntimeException: > java.io.IOException: 4 exceptions during in-mem cube build, cause set to > the first, check log for more at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort( > DoggedCubeBuilder.java:194) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$ > BuildOnce.checkException(DoggedCubeBuilder.java:167) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build( > DoggedCubeBuilder.java:114) > ... 7 more Caused by: java.lang.RuntimeException: java.io.IOException: 4 > exceptions during in-mem cube build, cause set to the first, check log for > more at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run( > DoggedCubeBuilder.java:287) > Caused by: java.io.IOException: 4 exceptions during in-mem cube build, > cause set to the first, check log for more at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.throwExceptionIfAny( > InMemCubeBuilder.java:228) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > build(InMemCubeBuilder.java:184) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > build(InMemCubeBuilder.java:135) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run( > DoggedCubeBuilder.java:282) > Caused by: java.io.IOException: No space left on device at > sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) at > sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66) at > sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) at > sun.nio.ch.IOUtil.write(IOUtil.java:65) at > sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745) at > sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731) at > org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore$Writer$1. > write(ConcurrentDiskStore.java:298) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at > java.io.DataOutputStream.flush(DataOutputStream.java:123) at > java.io.FilterOutputStream.close(FilterOutputStream.java:158) at > org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore$Writer. > close(ConcurrentDiskStore.java:320) > at org.apache.kylin.gridtable.GTBuilder.close(GTBuilder.java:53) at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > scanAndAggregateGridTable(InMemCubeBuilder.java:448) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.aggregateCuboid( > InMemCubeBuilder.java:397) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > buildCuboid(InMemCubeBuilder.java:389) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder. > access$300(InMemCubeBuilder.java:63) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$CuboidTaskThread.run( > InMemCubeBuilder.java:268) > Suppressed: java.io.IOException: No space left on device at > sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) at > sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66) at > sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) at > sun.nio.ch.IOUtil.write(IOUtil.java:65) at > sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745) at > sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731) at > org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore$Writer$1. > write(ConcurrentDiskStore.java:298) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at > java.io.FilterOutputStream.close(FilterOutputStream.java:158) at > java.io.FilterOutputStream.close(FilterOutputStream.java:159) ... 7 more > Container killed by the ApplicationMaster. Container killed on request. > Exit code is 143 Container exited with a non-zero exit code 143Regards, > Suresh > -- Best regards, Shaofeng Shi 史少锋
