[jira] [Created] (KYLIN-5198) kylin无法build，org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:file:/kylin/kylin_metadata/kylin-xxxxxx/kylin_intermediate_empcubexxxxis not a direct

2022-06-21 Thread gene (Jira) 
gene created KYLIN-5198:
---

 Summary: 
kylin无法build，org.apache.hadoop.hive.ql.metadata.HiveException: 
MetaException(message:file:/kylin/kylin_metadata/kylin-xx/kylin_intermediate_empcubeis
 not a directory or unable to create one
 Key: KYLIN-5198
 URL: https://issues.apache.org/jira/browse/KYLIN-5198
 Project: Kylin
  Issue Type: Bug
Affects Versions: v3.0.2
Reporter: gene


*你好，问下 我kylin启动和构建cube是正常的，下面包括了我个人的分析，但是问题仍然无法解决，这个问题困扰了我几天了，希望能获得帮助*

*前置条件：hbase用集群启动，但是只配置了一个节点， 写hivesql 时 感觉用的本地方式而不是 dfs方式，但是hbase配置的是hdfs方式的， 
kylin启动为啥用file:///方式呢*

*1 但是 在build的时候出错了：*
2022-06-22 10:34:51,476 INFO  [04b2d097-0a4f-4dd0-bf14-a0bc69435398 main] 
metastore.HiveMetaStoreClient: Connected to metastore.
2022-06-22 10:34:51,476 INFO  [04b2d097-0a4f-4dd0-bf14-a0bc69435398 main] 
metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hadoop 
(auth:SIMPLE) retries=1 delay=1 lifetime=0
2022-06-22 10:34:51,512 ERROR [04b2d097-0a4f-4dd0-bf14-a0bc69435398 main] 
exec.DDLTask: Failed
org.apache.hadoop.hive.ql.metadata.HiveException: 
MetaException(message:file:/kylin/kylin_metadata/kylin-3204c1d5-c244-7873-07b5-d9fb8ae38996/kylin_intermediate_empcube3_8285e2f7_4de0_827f_2644_7c391ccf661b
 is not a directory or unable to create one)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:939)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:944)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4957)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:428)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: 
MetaException(message:file:/kylin/kylin_metadata/kylin-3204c1d5-c244-7873-07b5-d9fb8ae38996/kylin_intermediate_empcube3_8285e2f7_4de0_827f_2644_7c391ccf661b
 is not a directory or unable to create one)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:54908)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:54876)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result.read(ThriftHiveMetastore.java:54802)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_with_environment_context(ThriftHiveMetastore.java:1556)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_with_environment_context(ThriftHiveMetastore.java:1542)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2867)
at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:121)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:837

[jira] [Created] (KYLIN-5197) kylin无法build，报错org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:file:/kylin/kylin_metadata/kylin-xxxx/kylin_intermediate_empcubexxxx is not a direc

2022-06-21 Thread gene (Jira) 
gene created KYLIN-5197:
---

 Summary: 
kylin无法build，报错org.apache.hadoop.hive.ql.metadata.HiveException: 
MetaException(message:file:/kylin/kylin_metadata/kylin-/kylin_intermediate_empcube
 is not a directory or unable to create one
 Key: KYLIN-5197
 URL: https://issues.apache.org/jira/browse/KYLIN-5197
 Project: Kylin
  Issue Type: Bug
Affects Versions: v3.0.2
Reporter: gene






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-05 Thread ShaoFeng Shi 
50)
> ... 21 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
> ... 31 more
> Caused by: java.lang.NoClassDefFoundError:
> javax/jdo/JDOObjectNotFoundException
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:274)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getClass(MetaStoreUtils.java:1489)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:63)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:593)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:571)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:66)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:199)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:181)
> at
> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.(HiveClientCache.java:330)
> ... 36 more
> Caused by: java.lang.ClassNotFoundException:
> javax.jdo.JDOObjectNotFoundException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 50 more
>
>
>
>
>
>
> -- 原始邮件 --
> 发件人: "ShaoFeng Shi";<shaofeng...@apache.org>;
> 发送时间: 2016年1月5日(星期二) 下午3:21
> 收件人: "dev"<dev@kylin.apache.org>;
>
> 主题: Re: org.apache.hadoop.hive.ql.metadata.HiveException
>
>
>
> Hi hefeng,
>
> It seems the hcatalog jars doesn't exist on your hadoop node; The solution
> is to upload the jar files to an HDFS folder, and then set  that patch as
> the value of "kylin.job.mr.lib.dir" in kylin.properties, you can checkout
> this JIRA: https://issues.apache.org/jira/browse/KYLIN-1021
>
> In our env, the "kylin.job.mr.lib.dir" folder has the following 4 jar
> files, just for your reference:
> hive-common-xx.jar
> hive-exec-xx.jar
> hive-hcatalog-core-xx.jar
> hive-metastore-xx.jar
>
> Here "xx" means the version number;
>
> Just take a try and let us know whether it works.
>
> 2016-01-05 14:47 GMT+08:00 和风 <363938...@qq.com>:
>
> > Thanks for your help. error logs:
> > [pool-7-thread-1]:[2016-01-05
> >
> 14:45:31,312][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
> > - job id:d0e2f259-9541-4b6f-9f54-c502781549e2-00 from RUNNING to SUCCEED
> > [pool-7-thread-1]:[2016-01-05
> >
> 14:45:31,438][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
> > - job id:d0e2f259-9541-4b6f-9f54-c502781549e2 from RUNNING to READY
> > [pool-6-thread-1]:[2016-01-05
> >
> 14:45:31,483][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:102)]
> > - CubingJob{id=d0e2f259-9541-4b6f-9f54-c502781549e2,
> name=learn_kylin_four
> > - 2015020100_2015122900 - BUILD - GMT-08:00 2016-01-04 22:44:05,
> > state=READY} prepare to schedule
> > [pool-6-thread-1]:[2016-01-05
> >
> 14:45:31,484][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:106)]
> > - CubingJob{id=d0e2f259-9541-4b6f-9f54-c50278154

?????? org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-05 Thread ???? 
hi,
  I,m from hive copy  all jars to hdfs folder. run have a error.


java.io.IOException: 
NoSuchObjectException(message:default.kylin_intermediate_learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_c8d9323ae277
 table not found)
at 
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
at 
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
at 
org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
at 
org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at 
org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at 
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at 
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: 
NoSuchObjectException(message:default.kylin_intermediate_learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_c8d9323ae277
 table not found)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1808)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1778)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy47.get_table(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1208)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152)
at com.sun.proxy.$Proxy48.getTable(Unknown Source)
at org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:180)
at 
org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:105)
at 
org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
at 
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
... 13 more





--  --
??: "ShaoFeng Shi";<shaofeng...@apache.org>;
: 2016??1??5??(??) 9:48
??: "dev"<dev@kylin.apache.org>; 

: Re: org.apache.hadoop.hive.ql.metadata.HiveException



It makes progress, this time the root cause is
"java.lang.ClassNotFoundException:
javax.jdo.JDOObjectNotFoundException".

It indicates that 4 jars are not enough, jars which are needed by HCatalog
and not appeared in your MR nodes need be added to the HDFS folder as well,
like jdo-api-xx.jar. Just take a try.

2016-01-05 19:42 GMT+08:00  <363938...@qq.com>:

> thanks,but i add
> hive-common-xx.jar,hive-exec-xx.jar,hive-hcatalog-core-xx.jar,hive-metastore-xx.jar
> to hdfs folder.
> run have a new error;
>
>
> logs:
> [pool-7-thread-4]:[2016-01-05
> 19:19:59,443][ERROR][org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:123)]
> - error execute
> MapReduceExecutable{id=9c9d1e69-678c-45a0-a054-4dac9d07503a-01,
> name=Extract Fact Table Distinct Columns, state=RUNNING}
> java.io.IOException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
> at
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
> at
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
>

Re: org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread Kiriti Sai 
Hi,
This error is caused because there is no Snappy compression codec available
in your setup and Kylin expects it by default.
As a work around, you can disable the use of snappy in the configuration
files of Kylin.
> Comment the compression.codec line in kylin.properties
> comment the properties in the kylin_job_conf.xml which are related to
compression. I guess there are around 4 properties to be commented.

This was the work around I used for a while but its recommended to use
compression techniques to minimize the memory shuffling between reducers.

Thank you.
Sai Kiriti B
On Jan 5, 2016 12:31 PM, "和风" <363938...@qq.com> wrote:

> hi:
>   execution "bulid" cube, jobs exception :
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException
>
>
> logs:
>
>
> OS command error exit with 2 -- hive  -e "USE default;
> DROP TABLE IF EXISTS
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db;
>
>
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> (
> DEFAULT_KYLIN_CAL_DT_AGE_FOR_QTR_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_MONTH_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_DT_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_RTL_MONTH_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_CS_WEEK_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_YEAR_ID string
> )
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
> STORED AS SEQUENCEFILE
> LOCATION
> '/kylin/kylin_metadata/kylin-d22e7c10-032a-4d22-a802-3b74937e86db/kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db';
>
>
> SET mapreduce.job.split.metainfo.maxsize=-1;
> SET mapred.compress.map.output=true;
> SET
> mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET mapred.output.compress=true;
> SET
> mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET mapred.output.compression.type=BLOCK;
> SET mapreduce.job.max.split.locations=2000;
> SET dfs.replication=2;
> SET hive.merge.mapfiles=true;
> SET hive.merge.mapredfiles=true;
> SET hive.merge.size.per.task=268435456;
> SET hive.support.concurrency=false;
> SET hive.exec.compress.output=true;
> SET hive.auto.convert.join.noconditionaltask = true;
> SET hive.auto.convert.join.noconditionaltask.size = 3;
> INSERT OVERWRITE TABLE
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> SELECT
> KYLIN_CAL_DT.AGE_FOR_QTR_ID
> ,KYLIN_CAL_DT.AGE_FOR_MONTH_ID
> ,KYLIN_CAL_DT.AGE_FOR_DT_ID
> ,KYLIN_CAL_DT.AGE_FOR_RTL_MONTH_ID
> ,KYLIN_CAL_DT.AGE_FOR_CS_WEEK_ID
> ,KYLIN_CAL_DT.YEAR_ID
> FROM DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
> WHERE (KYLIN_CAL_DT.CAL_DT >= '2013-12-29' AND KYLIN_CAL_DT.CAL_DT <
> '2016-01-12')
> ;
>
>
> "
>
>
> Logging initialized using configuration in
> jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> OK
> Time taken: 0.936 seconds
> OK
> Time taken: 0.112 seconds
> OK
> Time taken: 0.438 seconds
> Query ID = root_20160105105405_88149f4a-a970-47d0-ba32-9a21ee5afde3
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1449731904014_1636, Tracking URL =
> http://cloud001:8088/proxy/application_1449731904014_1636/
> Kill Command = /usr/local/hadoop/bin/hadoop job  -kill
> job_1449731904014_1636
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> 2016-01-05 10:54:26,177 Stage-1 map = 0%,  reduce = 0%
> 2016-01-05 10:54:27,236 Stage-1 map = 100%,  reduce = 0%
> Ended Job = job_1449731904014_1636 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1449731904014_1636_m_00 (and more) from job
> job_1449731904014_1636
>
>
> Task with the most failures(1):
> -
> Task ID:
>   task_1449731904014_1636_m_00
>
>
> URL:
>
> http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1449731904014_1636=task_1449731904014_1636_m_00
> -
> Diagnostic M

org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread ???? 
hi:
  execution "bulid" cube, jobs exception : 
org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.hadoop.hive.ql.metadata.HiveException


logs:


OS command error exit with 2 -- hive  -e "USE default;
DROP TABLE IF EXISTS 
kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db;


CREATE EXTERNAL TABLE IF NOT EXISTS 
kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
(
DEFAULT_KYLIN_CAL_DT_AGE_FOR_QTR_ID smallint
,DEFAULT_KYLIN_CAL_DT_AGE_FOR_MONTH_ID smallint
,DEFAULT_KYLIN_CAL_DT_AGE_FOR_DT_ID smallint
,DEFAULT_KYLIN_CAL_DT_AGE_FOR_RTL_MONTH_ID smallint
,DEFAULT_KYLIN_CAL_DT_AGE_FOR_CS_WEEK_ID smallint
,DEFAULT_KYLIN_CAL_DT_YEAR_ID string
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
STORED AS SEQUENCEFILE
LOCATION 
'/kylin/kylin_metadata/kylin-d22e7c10-032a-4d22-a802-3b74937e86db/kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db';


SET mapreduce.job.split.metainfo.maxsize=-1;
SET mapred.compress.map.output=true;
SET 
mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapred.output.compress=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapred.output.compression.type=BLOCK;
SET mapreduce.job.max.split.locations=2000;
SET dfs.replication=2;
SET hive.merge.mapfiles=true;
SET hive.merge.mapredfiles=true;
SET hive.merge.size.per.task=268435456;
SET hive.support.concurrency=false;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join.noconditionaltask = true;
SET hive.auto.convert.join.noconditionaltask.size = 3;
INSERT OVERWRITE TABLE 
kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
 SELECT
KYLIN_CAL_DT.AGE_FOR_QTR_ID
,KYLIN_CAL_DT.AGE_FOR_MONTH_ID
,KYLIN_CAL_DT.AGE_FOR_DT_ID
,KYLIN_CAL_DT.AGE_FOR_RTL_MONTH_ID
,KYLIN_CAL_DT.AGE_FOR_CS_WEEK_ID
,KYLIN_CAL_DT.YEAR_ID
FROM DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT 
WHERE (KYLIN_CAL_DT.CAL_DT >= '2013-12-29' AND KYLIN_CAL_DT.CAL_DT < 
'2016-01-12')
;


"


Logging initialized using configuration in 
jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/local/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
OK
Time taken: 0.936 seconds
OK
Time taken: 0.112 seconds
OK
Time taken: 0.438 seconds
Query ID = root_20160105105405_88149f4a-a970-47d0-ba32-9a21ee5afde3
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1449731904014_1636, Tracking URL = 
http://cloud001:8088/proxy/application_1449731904014_1636/
Kill Command = /usr/local/hadoop/bin/hadoop job  -kill job_1449731904014_1636
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2016-01-05 10:54:26,177 Stage-1 map = 0%,  reduce = 0%
2016-01-05 10:54:27,236 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_1449731904014_1636 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1449731904014_1636_m_00 (and more) from job 
job_1449731904014_1636


Task with the most failures(1): 
-
Task ID:
  task_1449731904014_1636_m_00


URL:
  
http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1449731904014_1636=task_1449731904014_1636_m_00
-
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing row 
{"cal_dt":"2013-12-31","year_beg_dt":"2013-01-01","qtr_beg_dt":"2013-10-01","month_beg_dt":"2013-12-01","week_beg_dt":"2013-12-29","age_for_year_id":0,"age_for_qtr_id":0,"age_for_month_id":1,"age_for_week_id":5,"age_for_dt_id":34,"age_for_rtl_year_id":1,"age_for_rtl_qtr_id":1,"age_for_rtl_month_id":1,"age_for_rtl_week_id":5,"age_for_cs_week_id":5,"day_of_cal_id":41638,"day_of_year_id":365,"day_of_qtr_id":92,"day_of_month_id":31,"day_of_week_id":3,"week_of_year_id":53,"week_of_cal_id":5948,"month_of_qtr_id":3,"month_of_year_id":12,"month_of_cal_id":1368,"qtr_of_year_id":4,"qtr_of_cal_id":456,"year_of_cal_id":114,"year_end_dt":"2013-12-31","qtr_end_dt":"2013-12-31","mon

Re: org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread ShaoFeng Shi 
Hi 和风, screenshot is search engine unfriendly, please use text as much as
possible

2016-01-05 14:34 GMT+08:00 hongbin ma :

> can't see attachment. please provide detailed log
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>



-- 
Best regards,

Shaofeng Shi

Re: org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread hongbin ma 
can't see attachment. please provide detailed log


-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

?????? org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread ???? 
HI,
   I remove compression.codec in the kylin_job_conf.xml;but build cube have a 
new error;
error tip: failed to generate insert data sql  for intermediate table;










 




--  --
??: "hongbin ma";<mahong...@apache.org>;
: 2016??1??5??(??) 1:04
??: "dev"<dev@kylin.apache.org>; 

: Re: org.apache.hadoop.hive.ql.metadata.HiveException



agree with Sai

On Tue, Jan 5, 2016 at 12:24 PM, Kiriti Sai <kiriti163.i...@gmail.com>
wrote:

> Hi,
> This error is caused because there is no Snappy compression codec available
> in your setup and Kylin expects it by default.
> As a work around, you can disable the use of snappy in the configuration
> files of Kylin.
> > Comment the compression.codec line in kylin.properties
> > comment the properties in the kylin_job_conf.xml which are related to
> compression. I guess there are around 4 properties to be commented.
>
> This was the work around I used for a while but its recommended to use
> compression techniques to minimize the memory shuffling between reducers.
>
> Thank you.
> Sai Kiriti B
> On Jan 5, 2016 12:31 PM, "" <363938...@qq.com> wrote:
>
> > hi:
> >   execution "bulid" cube, jobs exception :
> > org.apache.hadoop.hive.ql.metadata.HiveException:
> > org.apache.hadoop.hive.ql.metadata.HiveException
> >
> >
> > logs:
> >
> >
> > OS command error exit with 2 -- hive  -e "USE default;
> > DROP TABLE IF EXISTS
> >
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db;
> >
> >
> > CREATE EXTERNAL TABLE IF NOT EXISTS
> >
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> > (
> > DEFAULT_KYLIN_CAL_DT_AGE_FOR_QTR_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_MONTH_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_DT_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_RTL_MONTH_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_CS_WEEK_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_YEAR_ID string
> > )
> > ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
> > STORED AS SEQUENCEFILE
> > LOCATION
> >
> '/kylin/kylin_metadata/kylin-d22e7c10-032a-4d22-a802-3b74937e86db/kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db';
> >
> >
> > SET mapreduce.job.split.metainfo.maxsize=-1;
> > SET mapred.compress.map.output=true;
> > SET
> >
> mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> > SET mapred.output.compress=true;
> > SET
> >
> mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> > SET mapred.output.compression.type=BLOCK;
> > SET mapreduce.job.max.split.locations=2000;
> > SET dfs.replication=2;
> > SET hive.merge.mapfiles=true;
> > SET hive.merge.mapredfiles=true;
> > SET hive.merge.size.per.task=268435456;
> > SET hive.support.concurrency=false;
> > SET hive.exec.compress.output=true;
> > SET hive.auto.convert.join.noconditionaltask = true;
> > SET hive.auto.convert.join.noconditionaltask.size = 3;
> > INSERT OVERWRITE TABLE
> >
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> > SELECT
> > KYLIN_CAL_DT.AGE_FOR_QTR_ID
> > ,KYLIN_CAL_DT.AGE_FOR_MONTH_ID
> > ,KYLIN_CAL_DT.AGE_FOR_DT_ID
> > ,KYLIN_CAL_DT.AGE_FOR_RTL_MONTH_ID
> > ,KYLIN_CAL_DT.AGE_FOR_CS_WEEK_ID
> > ,KYLIN_CAL_DT.YEAR_ID
> > FROM DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
> > WHERE (KYLIN_CAL_DT.CAL_DT >= '2013-12-29' AND KYLIN_CAL_DT.CAL_DT <
> > '2016-01-12')
> > ;
> >
> >
> > "
> >
> >
> > Logging initialized using configuration in
> > jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> >
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> [jar:file:/usr/local/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> > OK
> > Time taken: 0.936 seconds
> > OK
> > Time taken: 0.112 seconds
> > OK
> > Time taken: 0.438 seconds
> > Query ID = root_20160105105405_8814