One of our cubes is failing at "#12 Step Name: Build N-Dimension Cuboid Data
: 22-Dimension" step and irrespective of how much i bump up the mapper and
reducer memory and other configs, the jobs fails at this step everytime.
Looked at the yarn application logs and it doesnt give any useful info
there. Can someone help.
The cube has 8 measures 27 dimensions. Source data is ~42 million.
FYI.. this cube has been running fine for a few months now, its just started
failing a few days back and the data has not increased by much as well.
Kylin version= 1.6.0
Hortonworks = 2.4.0
Yarn MR Error:
java.lang.RuntimeException: java.io.IOException: I failed to find the one of
the right cookies.
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
at org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
at
org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:93)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:42)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.io.IOException: I failed to find the one of the right
cookies.
at
org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
at
org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
at
org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
... 17 more
2020-01-31 21:49:16,955 INFO [IPC Server handler 24 on 38174]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from
attempt_1568236776824_87076_m_000900_0: Error: java.lang.RuntimeException:
java.io.IOException: I failed to find the one of the right cookies.
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
at org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
at
org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:93)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:42)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.io.IOException: I failed to find the one of the right
cookies.
at
org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
at
org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
at
org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
... 17 more
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
