[ https://issues.apache.org/jira/browse/KYLIN-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812384#comment-16812384 ]
Shaofeng SHI commented on KYLIN-3930: ------------------------------------- The non-sharded storage are not support after v1.5 I think, though it didn't report error. Please keep in old version, or switch to the sharded HBase storage (storage type = 2). > ArrayIndexOutOfBoundsException when building > -------------------------------------------- > > Key: KYLIN-3930 > URL: https://issues.apache.org/jira/browse/KYLIN-3930 > Project: Kylin > Issue Type: Bug > Components: Job Engine > Affects Versions: all > Reporter: Jacky Woo > Priority: Major > Fix For: v2.6.2 > > Attachments: KYLIN-3930.master.01.patch > > > h2. ArrayIndexOutOfBoundsException when building. > I hive a cube building error with kylin-2.5.0: > {code:java} > 2019-03-31 02:45:18,460 ERROR [main] org.apache.kylin.engine.mr.KylinMapper: > java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method) > at > org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106) > at > org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71) > at > org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112) > at > org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47) > at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > {code} > I checked the code of "NDCuboidBuilder.buildKeyInternal" method > {code:java} > private void buildKeyInternal(Cuboid parentCuboid, Cuboid childCuboid, > ByteArray[] splitBuffers, ByteArray newKeyBodyBuf) { > RowKeyEncoder rowkeyEncoder = > rowKeyEncoderProvider.getRowkeyEncoder(childCuboid); > // rowkey columns > long mask = Long.highestOneBit(parentCuboid.getId()); > long parentCuboidId = parentCuboid.getId(); > long childCuboidId = childCuboid.getId(); > long parentCuboidIdActualLength = (long)Long.SIZE - > Long.numberOfLeadingZeros(parentCuboid.getId()); > int index = rowKeySplitter.getBodySplitOffset(); // skip shard and > cuboidId > int offset = RowConstants.ROWKEY_SHARDID_LEN + > RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId > for (int i = 0; i < parentCuboidIdActualLength; i++) { > if ((mask & parentCuboidId) > 0) {// if the this bit position > equals > // 1 > if ((mask & childCuboidId) > 0) {// if the child cuboid has > this > // column > System.arraycopy(splitBuffers[index].array(), > splitBuffers[index].offset(), newKeyBodyBuf.array(), offset, > splitBuffers[index].length()); > offset += splitBuffers[index].length(); > } > index++; > } > mask = mask >> 1; > } > rowkeyEncoder.fillHeader(newKeyBodyBuf.array()); > } > {code} > Found that "offset = SHARDID_LEN + CUBOIDID_LEN" , which is wrong when cube > is not sharding. In my case my cube's storage type is 0, which means it is > not sharding. > So, I set offset according to cube sharding, like below: > {code:java} > int offset = rowKeySplitter.getHeaderLength(); // skip shard and cuboidId > {code} > After modifying building succeeds in my environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)