[ https://issues.apache.org/jira/browse/KYLIN-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388770#comment-15388770 ]
Dong Li commented on KYLIN-1912: -------------------------------- Merged to master branch. > Create Intermediate Flat Hive Table failed when using beeline > ------------------------------------------------------------- > > Key: KYLIN-1912 > URL: https://issues.apache.org/jira/browse/KYLIN-1912 > Project: Kylin > Issue Type: Bug > Components: Job Engine > Affects Versions: v1.5.2 > Reporter: Billy(Yiming) Liu > Assignee: Dong Li > Priority: Critical > Attachments: KYLIN-1912.patch > > > When beeline is enabled with parameters: > kylin.hive.client = beeline > kylin.hive.beeline.params=-n root --hiveconf > hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*' > -u 'jdbc:hive2://localhost:10000' > The build cube will fail with following exceptions: > {code} > java.lang.IllegalArgumentException: Map input splits are 0 bytes, something > is wrong! > at > org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:528) > at > org.apache.kylin.engine.mr.steps.CuboidJob.setReduceTaskNum(CuboidJob.java:183) > at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:139) > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91) > at > org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 2016-07-21 15:05:38,615 INFO [pool-5-thread-6] common.AbstractHadoopJob:479 > : tempMetaFileString is : > file:///opt/client/kylin/apache-kylin-1.5.2.1-bin/bin/./../tomcat/temp/kylin_job_meta5547890224035214519/meta > 2016-07-21 15:05:38,619 ERROR [pool-5-thread-6] > execution.AbstractExecutable:128 : error execute > MapReduceExecutable{id=782765e2-5357-4cc3-8430-b4ee9ffc69de-05, name=Build > Base Cuboid Data, state=RUNNING} > java.lang.IllegalArgumentException: Map input splits are 0 bytes, something > is wrong! > at > org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:528) > at > org.apache.kylin.engine.mr.steps.CuboidJob.setReduceTaskNum(CuboidJob.java:183) > at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:139) > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91) > at > org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > After checking the detail logs, I found the intermediate table may not create > succeed. > {code} > 2016-07-21 15:15:50,361 INFO [pool-5-thread-1] > execution.AbstractExecutable:100 : Executing >>>>>>>>>>>>> Create > Intermediate Flat Hive Table <<<<<<<<<<<<< > 2016-07-21 15:15:50,376 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262 > : Update row /execute_output/8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from > oldTs: 1469085306322, to newTs: 1469085350362, operation result: true > 2016-07-21 15:15:50,376 INFO [pool-5-thread-1] manager.ExecutableManager:274 > : job id:8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from READY to RUNNING > 2016-07-21 15:15:50,376 INFO [pool-5-thread-1] > execution.AbstractExecutable:49 : executing:beeline -f > /opt/client/kylin/apache-kylin-1.5.2.1-bin/bin/./../tomcat/temp/beeline_346879081651464139.hql;rm > -f > /opt/client/kylin/apache-kylin-1.5.2.1-bin/bin/./../tomcat/temp/beeline_346879081651464139.hql > 2016-07-21 15:15:53,999 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262 > : Update row /execute_output/8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from > oldTs: 1469085350362, to newTs: 1469085353989, operation result: true > 2016-07-21 15:15:54,028 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262 > : Update row /execute_output/8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from > oldTs: 1469085353989, to newTs: 1469085354001, operation result: true > 2016-07-21 15:15:54,062 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262 > : Update row /execute_output/8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from > oldTs: 1469085354001, to newTs: 1469085354030, operation result: true > 2016-07-21 15:15:54,062 INFO [pool-5-thread-1] manager.ExecutableManager:274 > : job id:8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from RUNNING to SUCCEED > {code} > Now use beeline command to create the intermediate table, return SUCCEED in > only 4 seconds, normally build is 40+ seconds with Hive CLI. > Then check the code, found beeline could not process ";" properly. Beeline > execute each statement for each line, not split by ";". But Kylin has the > variable 'useDatabaseHql' with value 'USE default;" without the 'new line'. > That's the root cause. Not a difficult problem, buy may stop many new beeline > user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)