[jira] [Commented] (KYLIN-1912) Create Intermediate Flat Hive Table failed when using beeline
[
https://issues.apache.org/jira/browse/KYLIN-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388770#comment-15388770
]
Dong Li commented on KYLIN-1912:
Merged to master branch.
> Create Intermediate Flat Hive Table failed when using beeline
> -
>
> Key: KYLIN-1912
> URL: https://issues.apache.org/jira/browse/KYLIN-1912
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.5.2
>Reporter: Billy(Yiming) Liu
>Assignee: Dong Li
>Priority: Critical
> Attachments: KYLIN-1912.patch
>
>
> When beeline is enabled with parameters:
> kylin.hive.client = beeline
> kylin.hive.beeline.params=-n root --hiveconf
> hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*'
> -u 'jdbc:hive2://localhost:1'
> The build cube will fail with following exceptions:
> {code}
> java.lang.IllegalArgumentException: Map input splits are 0 bytes, something
> is wrong!
> at
> org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:528)
> at
> org.apache.kylin.engine.mr.steps.CuboidJob.setReduceTaskNum(CuboidJob.java:183)
> at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:139)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-21 15:05:38,615 INFO [pool-5-thread-6] common.AbstractHadoopJob:479
> : tempMetaFileString is :
> file:///opt/client/kylin/apache-kylin-1.5.2.1-bin/bin/./../tomcat/temp/kylin_job_meta5547890224035214519/meta
> 2016-07-21 15:05:38,619 ERROR [pool-5-thread-6]
> execution.AbstractExecutable:128 : error execute
> MapReduceExecutable{id=782765e2-5357-4cc3-8430-b4ee9ffc69de-05, name=Build
> Base Cuboid Data, state=RUNNING}
> java.lang.IllegalArgumentException: Map input splits are 0 bytes, something
> is wrong!
> at
> org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:528)
> at
> org.apache.kylin.engine.mr.steps.CuboidJob.setReduceTaskNum(CuboidJob.java:183)
> at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:139)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> After checking the detail logs, I found the intermediate table may not create
> succeed.
> {code}
> 2016-07-21 15:15:50,361 INFO [pool-5-thread-1]
> execution.AbstractExecutable:100 : Executing > Create
> Intermediate Flat Hive Table <
> 2016-07-21 15:15:50,376 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262
> : Update row /execute_output/8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from
> oldTs: 1469085306322, to newTs: 1469085350362, operation result: true
> 2016-07-21 15:15:50,376 INFO [pool-5-thread-1] manager.ExecutableManager:274
> : job id:8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from READY to RUNNING
> 2016-07-21 15:15:50,376 INFO [pool-5-thread-1]
> execution.AbstractExecutable:49 : executing:beeline -f
> /opt/client/kylin/apache-kylin-1.5.2.1-bin/bin/./../tomcat/temp/beeline_346879081651464139.hql;rm
> -f
> /opt/client/kylin/apache-kylin-1.5.2.1-bin/bin/./../tomcat/temp/beeline_346879081651464139.hql
> 2016-07-21 15:15:53,999 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262
> : Update row
[jira] [Commented] (KYLIN-1912) Create Intermediate Flat Hive Table failed when using beeline
[
https://issues.apache.org/jira/browse/KYLIN-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388742#comment-15388742
]
Billy(Yiming) Liu commented on KYLIN-1912:
--
Thank you for the review. HiveCmdBuilder will append line.seperator, but
HiveMRInput line 198, there is only one statement including "use, drop, create,
insert". So it's a bug. Agree with you, line.seperator is better than hard code
'\n'. I will update the patch.
> Create Intermediate Flat Hive Table failed when using beeline
> -
>
> Key: KYLIN-1912
> URL: https://issues.apache.org/jira/browse/KYLIN-1912
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.5.2
>Reporter: Billy(Yiming) Liu
>Assignee: Dong Li
>Priority: Critical
> Attachments: KYLIN-1912.patch
>
>
> When beeline is enabled with parameters:
> kylin.hive.client = beeline
> kylin.hive.beeline.params=-n root --hiveconf
> hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*'
> -u 'jdbc:hive2://localhost:1'
> The build cube will fail with following exceptions:
> {code}
> java.lang.IllegalArgumentException: Map input splits are 0 bytes, something
> is wrong!
> at
> org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:528)
> at
> org.apache.kylin.engine.mr.steps.CuboidJob.setReduceTaskNum(CuboidJob.java:183)
> at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:139)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-21 15:05:38,615 INFO [pool-5-thread-6] common.AbstractHadoopJob:479
> : tempMetaFileString is :
> file:///opt/client/kylin/apache-kylin-1.5.2.1-bin/bin/./../tomcat/temp/kylin_job_meta5547890224035214519/meta
> 2016-07-21 15:05:38,619 ERROR [pool-5-thread-6]
> execution.AbstractExecutable:128 : error execute
> MapReduceExecutable{id=782765e2-5357-4cc3-8430-b4ee9ffc69de-05, name=Build
> Base Cuboid Data, state=RUNNING}
> java.lang.IllegalArgumentException: Map input splits are 0 bytes, something
> is wrong!
> at
> org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:528)
> at
> org.apache.kylin.engine.mr.steps.CuboidJob.setReduceTaskNum(CuboidJob.java:183)
> at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:139)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> After checking the detail logs, I found the intermediate table may not create
> succeed.
> {code}
> 2016-07-21 15:15:50,361 INFO [pool-5-thread-1]
> execution.AbstractExecutable:100 : Executing > Create
> Intermediate Flat Hive Table <
> 2016-07-21 15:15:50,376 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262
> : Update row /execute_output/8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from
> oldTs: 1469085306322, to newTs: 1469085350362, operation result: true
> 2016-07-21 15:15:50,376 INFO [pool-5-thread-1] manager.ExecutableManager:274
> : job id:8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from READY to RUNNING
> 2016-07-21 15:15:50,376 INFO [pool-5-thread-1]
> execution.AbstractExecutable:49 : executing:beeline -f
>
[jira] [Commented] (KYLIN-1912) Create Intermediate Flat Hive Table failed when using beeline
[
https://issues.apache.org/jira/browse/KYLIN-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388718#comment-15388718
]
Dong Li commented on KYLIN-1912:
Reviewed the patch, in fact, with beeline, HiveCmdBuilder will append
line.seperator(not hard code \n) to each statement, please refer to UT:
org.apache.kylin.source.hive.HiveCmdBuilderTest#testBeeline
Does beeline split lines by \n or system property line.seperator? Maybe should
look into this.
> Create Intermediate Flat Hive Table failed when using beeline
> -
>
> Key: KYLIN-1912
> URL: https://issues.apache.org/jira/browse/KYLIN-1912
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.5.2
>Reporter: Billy(Yiming) Liu
>Assignee: Dong Li
>Priority: Critical
> Attachments: KYLIN-1912.patch
>
>
> When beeline is enabled with parameters:
> kylin.hive.client = beeline
> kylin.hive.beeline.params=-n root --hiveconf
> hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*'
> -u 'jdbc:hive2://localhost:1'
> The build cube will fail with following exceptions:
> {code}
> java.lang.IllegalArgumentException: Map input splits are 0 bytes, something
> is wrong!
> at
> org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:528)
> at
> org.apache.kylin.engine.mr.steps.CuboidJob.setReduceTaskNum(CuboidJob.java:183)
> at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:139)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-21 15:05:38,615 INFO [pool-5-thread-6] common.AbstractHadoopJob:479
> : tempMetaFileString is :
> file:///opt/client/kylin/apache-kylin-1.5.2.1-bin/bin/./../tomcat/temp/kylin_job_meta5547890224035214519/meta
> 2016-07-21 15:05:38,619 ERROR [pool-5-thread-6]
> execution.AbstractExecutable:128 : error execute
> MapReduceExecutable{id=782765e2-5357-4cc3-8430-b4ee9ffc69de-05, name=Build
> Base Cuboid Data, state=RUNNING}
> java.lang.IllegalArgumentException: Map input splits are 0 bytes, something
> is wrong!
> at
> org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:528)
> at
> org.apache.kylin.engine.mr.steps.CuboidJob.setReduceTaskNum(CuboidJob.java:183)
> at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:139)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> After checking the detail logs, I found the intermediate table may not create
> succeed.
> {code}
> 2016-07-21 15:15:50,361 INFO [pool-5-thread-1]
> execution.AbstractExecutable:100 : Executing > Create
> Intermediate Flat Hive Table <
> 2016-07-21 15:15:50,376 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262
> : Update row /execute_output/8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from
> oldTs: 1469085306322, to newTs: 1469085350362, operation result: true
> 2016-07-21 15:15:50,376 INFO [pool-5-thread-1] manager.ExecutableManager:274
> : job id:8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from READY to RUNNING
> 2016-07-21 15:15:50,376 INFO [pool-5-thread-1]
> execution.AbstractExecutable:49 : executing:beeline -f
>
