[
https://issues.apache.org/jira/browse/KYLIN-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14975872#comment-14975872
]
Shaofeng SHI commented on KYLIN-1104:
-------------------------------------
Hi Yu Feng, thanks for the analysis and hot-fix, the patch looks good;
While I have a concern, because this change may hide the dirty data, and then
bypass those records into the cube. Usually 4K space should be enough for 1
row; We can enlarge it to 8K or 16K, but letting it be unlimited increased
(considering there is one extreme big line), that might be bad;
> Long dimension value casue ArrayIndexOutOfBoundsException
> ---------------------------------------------------------
>
> Key: KYLIN-1104
> URL: https://issues.apache.org/jira/browse/KYLIN-1104
> Project: Kylin
> Issue Type: Bug
> Reporter: fengYu
> Fix For: v1.1, v1.0
>
> Attachments: 0001-.patch
>
>
> I get ArrayIndexOutOfBoundsException while building cube, and the error is
> caused by one dimension(I think measure can not be so long) value is longer
> than 4096 bytes.
> I check source code and find the reason:
> 1、it specify max split length is 4096 when create BytesSplitter object in
> setup.
> 2、in BytesSplitter.split function which split all items in one row with
> delimiter, and using System.arraycopy(bytes, offset, split.value, 0, length);
> copy every item to a SplittedBytes object named split. and before this line ,
> it execute split.length = length; to set the split length.
> 3、while executing the copy code, it throw ArrayIndexOutOfBoundsException
> because the real item length is bigger than 4096!
> 4、in mapper function, the exception will catched and execute
> handleErrorRecord function to handle it, However, in this function it will
> call use splits again(in toString function), ArrayIndexOutOfBoundsException
> will throw again.
> 5、There is not one catch this exception, which casue the mapper execute
> failed.
> That is all my situation and analysis, I will modify source code to continue
> my job building and I will upload my patch later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
