Hi,
The first step in the cube to merge, #1 Step Name: Merge Cuboid Dictionary
Error Log info:
2016-11-10 14:08:00,798 DEBUG [pool-7-thread-1] dict.DictionaryGenerator:91 :
Dictionary value samples: 10101001120172=>
479, 10101003212212=>480, 10101003812579=>481, 10101005033448=>482,
10101005046605=>483
2016-11-10 14:08:00,798 DEBUG [pool-7-thread-1] dict.DictionaryGenerator:92 :
Dictionary cardinality: 108330611
2016-11-10 14:08:00,798 DEBUG [pool-7-thread-1] dict.DictionaryGenerator:93 :
Dictionary builder class: org.apache.kylin
.dict.DictionaryGenerator$StringDictBuilder
2016-11-10 14:08:00,798 DEBUG [pool-7-thread-1] dict.DictionaryGenerator:94 :
Dictionary class: org.apache.kylin.dict.Tr
ieDictionary
2016-11-10 14:08:00,798 INFO [pool-7-thread-1] dict.DictionaryManager:169 :
Growing dict is not enabled
2016-11-10 14:08:00,803 INFO [pool-7-thread-1] dict.DictionaryManager:186 : 5
existing dictionaries of the same column
2016-11-10 14:08:00,804 INFO [pool-7-thread-1] dict.DictionaryManager:420 :
DictionaryManager(1455639799) loading Dicti
onaryInfo(loadDictObj:true) at
/dict/DMT.DMT_KYLIN_PAY_SYT_ORDR_DET_I_D/OUTBIZNO/6ab19cbd-93f6-48b2-9eb8-a4
2016-11-10 14:09:00,651 INFO [pool-7-thread-1] dict.DictionaryManager:404 :
Saving dictionary at /dict/DMT.DMT_KYLIN_PA
Y_SYT_ORDR_DET_I_D/OUTBIZNO/745317e4-a682-4f6a-8ac3-cec2a466e419.dict
2016-11-10 14:09:02,658 DEBUG [pool-7-thread-1] persistence.ResourceStore:207 :
Directly saving resource /dict/DMT.DMT_K
YLIN_PAY_SYT_ORDR_DET_I_D/OUTBIZNO/745317e4-a682-4f6a-8ac3-cec2a466e419.dict
(Store kylinDev_metadata@hbase)
2016-11-10 14:09:04,263 ERROR [pool-7-thread-1]
execution.AbstractExecutable:115 : error running Executable: MergeDictio
naryStep{id=b0d32a86-3516-4232-b041-aabe127cccc5-00, name=Merge Cuboid
Dictionary, state=RUNNING}
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2147)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.putResourceImpl(HBaseResourceStore.java:239)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:208)
at
org.apache.kylin.dict.DictionaryManager.save(DictionaryManager.java:413)
at
org.apache.kylin.dict.DictionaryManager.saveNewDict(DictionaryManager.java:209)
at
org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:176)
at
org.apache.kylin.dict.DictionaryManager.mergeDictionary(DictionaryManager.java:269)
at
org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:145)
at
org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:135)
at
org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:67)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
In /bin/ setenv.sh :
export KYLIN_JVM_SETTINGS="-Xmx100g -Xms100g -Xmn2g -XX:+UseG1GC
-XX:MaxPermSize=128M -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLo
gFileSize=64M"
but this error still appear,so Where is to modify the parameters can be solved
??
