is it possible that one of the dimensions specified as "dict encoding" turns out to be having huge cardinality/dimension length? if that's the case, the generated dictionary might be too large to fix. You should think about using other dimension encodings like "fixed length" or "extended column (https://issues.apache.org/jira/browse/KYLIN-1313)"
On Tue, May 3, 2016 at 5:39 PM, ShaoFeng Shi <[email protected]> wrote: > What's the "hbase.client.keyvalue.maxsize" in your hbase configuration? > Guess it is bigger than 10485760. Please get that value and then set > "kylin.hbase.client.keyvalue.maxsize" conf/kylin.properteis to the same or > less than hbase's. Kylin uses the later one to deciside whether put a > resource to hbase or not; > > > > 2016-05-03 17:09 GMT+08:00 彭丽 <[email protected]>: > > > Hi, > > > > I meet a problem when the step of building dimension dictionary. Fact > > table has 400000 records. 3 lookup tables have 200000 , 10000, 200 > records > > separately. > > > > > > > > java.lang.IllegalArgumentException: KeyValue size too large > > > > at > > org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1413) > > > > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:1035) > > > > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:999) > > > > at > > > > > org.apache.kylin.storage.hbase.HBaseResourceStore.putResourceImpl(HBaseResou > > rceStore.java:244) > > > > at > > > > > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore. > > java:210) > > > > at > > org.apache.kylin.dict.DictionaryManager.save(DictionaryManager.java:378) > > > > at > > > > > org.apache.kylin.dict.DictionaryManager.saveNewDict(DictionaryManager.java:1 > > 99) > > > > at > > > > > org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.jav > > a:166) > > > > at > > > > > org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.ja > > va:293) > > > > at > > org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:167) > > > > at > > > > > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGe > > neratorCLI.java:51) > > > > at > > > > > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGe > > neratorCLI.java:42) > > > > at > > > > > org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob > > .java:56) > > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > > > > at > > > > > org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellEx > > ecutable.java:60) > > > > at > > > > > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable > > .java:114) > > > > at > > > > > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChaine > > dExecutable.java:50) > > > > at > > > > > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable > > .java:114) > > > > at > > > > > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultS > > cheduler.java:124) > > > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11 > > 45) > > > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6 > > 15) > > > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > result code:2 > > > > > > > > Thanks, > > > > > > > > Laney. > > > > > > > > > > > -- > Best regards, > > Shaofeng Shi > -- Regards, *Bin Mahone | 马洪宾* Apache Kylin: http://kylin.io Github: https://github.com/binmahone
