[
https://issues.apache.org/jira/browse/KYLIN-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-2231:
--------------------------------
Description:
User pengwen yuan in [email protected] mailing list reports that, he got
error when using "in-mem" cubing, but got success after switch to "layer"
cubing algorithm.
Checked the code I think it is because the layer cubing tolerates on minor
insane records (see BaseCuboidMapperBase.handleErrorRecord()), but in-mem
doesn't.
There should be no difference in the behavior. Need make them consistent.
The error in in-mem cubing is:
{code}
2016-11-25 16:23:10,198 ERROR [Thread-12] org.apache.kylin.dict.TrieDictionary:
Not a valid value: 532
2016-11-25 16:23:11,199 ERROR [pool-5-thread-1]
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build error
java.io.IOException: java.lang.IllegalArgumentException: Value not exists!
at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:194)
at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:167)
at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:114)
at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:73)
at
org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:80)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Value not exists!
at
org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162)
at
org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:167)
at
org.apache.kylin.common.util.Dictionary.getIdFromValue(Dictionary.java:98)
at
org.apache.kylin.dimension.DictionaryDimEnc$DictionarySerializer.serialize(DictionaryDimEnc.java:121)
at
org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:121)
at
org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:110)
at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:93)
at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:81)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilderInputConverter.convert(InMemCubeBuilderInputConverter.java:74)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:542)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:523)
at
org.apache.kylin.gridtable.GTAggregateScanner.iterator(GTAggregateScanner.java:139)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.createBaseCuboid(InMemCubeBuilder.java:339)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:166)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:135)
at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:282)
2016-11-25 16:23:11,202 INFO [pool-5-thread-1]
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build end,
totally took 3410 ms
{code}
was:
User pengwen yuan in [email protected] mailing list reports that, he got
error when using "in-mem" cubing, but got success after switch to "layer"
cubing algorithm.
Checked the code I think it is because the layer cubing tolerates on minor
insane records (see BaseCuboidMapperBase.handleErrorRecord()), but in-mem
doesn't.
There should be no difference in the behavior. Need make them consistent.
> Layer cubing tolerates some insane records, but in-mem cubing doesn't
> ---------------------------------------------------------------------
>
> Key: KYLIN-2231
> URL: https://issues.apache.org/jira/browse/KYLIN-2231
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Reporter: Shaofeng SHI
> Priority: Minor
>
> User pengwen yuan in [email protected] mailing list reports that, he got
> error when using "in-mem" cubing, but got success after switch to "layer"
> cubing algorithm.
> Checked the code I think it is because the layer cubing tolerates on minor
> insane records (see BaseCuboidMapperBase.handleErrorRecord()), but in-mem
> doesn't.
> There should be no difference in the behavior. Need make them consistent.
> The error in in-mem cubing is:
> {code}
> 2016-11-25 16:23:10,198 ERROR [Thread-12]
> org.apache.kylin.dict.TrieDictionary: Not a valid value: 532
> 2016-11-25 16:23:11,199 ERROR [pool-5-thread-1]
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build error
> java.io.IOException: java.lang.IllegalArgumentException: Value not exists!
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:194)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:167)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:114)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:73)
> at
> org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:80)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Value not exists!
> at
> org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162)
> at
> org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:167)
> at
> org.apache.kylin.common.util.Dictionary.getIdFromValue(Dictionary.java:98)
> at
> org.apache.kylin.dimension.DictionaryDimEnc$DictionarySerializer.serialize(DictionaryDimEnc.java:121)
> at
> org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:121)
> at
> org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:110)
> at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:93)
> at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:81)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilderInputConverter.convert(InMemCubeBuilderInputConverter.java:74)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:542)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:523)
> at
> org.apache.kylin.gridtable.GTAggregateScanner.iterator(GTAggregateScanner.java:139)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.createBaseCuboid(InMemCubeBuilder.java:339)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:166)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:135)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:282)
> 2016-11-25 16:23:11,202 INFO [pool-5-thread-1]
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build end,
> totally took 3410 ms
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
