Re: Can not load hive table in Kylin
Today in a customer's env, I encountered this problem; His environment
looks good: bin/find-hive-dependency.sh can identify the jars correctly;
but after Kylin start up, still got the ClassNotFound error:
java.lang.ClassNotFoundException:
org.apache.hadoop.hive.ql.session.SessionState
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
at
org.apache.kylin.rest.controller.TableController.showHiveDatabases(TableController.java:301)
Finally identified the root case is in hbase-env.sh; in which the customer
overwrited the "HBASE_CLASSPATH" variable, like:
export HBASE_CLASSPATH=/home/hadoop/hbase/conf
This is bad as it overwrites the variable totally, while Kylin depends on
it to tell JVM which jars need be on classpath. After changing it to:
export HBASE_CLASSPATH=/home/hadoop/hbase/conf:${HBASE_CLASSPATH}
Kylin can run well without the ClassNotFound error anymore.
2016-06-13 16:30 GMT+08:00 蔡冉 :
> Thanks Shaofeng, I checked my chrome console and found below error, and I
> also found the error in tomcat log
> kylin/tomcat/logs/localhost.2016-06-13.log, which should be the root cause:
>
> org.springframework.web.util.NestedServletException: Handler processing
> failed; nested exception is java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/ql/session/SessionState
>
> but I can find hive-exec-1.2.1.jar in the result
> of find-hive-dependency.sh and the process, is there any other
> configuration I missed?
>
>
> 2016-06-12 22:05 GMT+08:00 ShaoFeng Shi :
>
>> It seems the request wasn't received by the server at all; Most likely be
>> a client side or network error; If you're using firefox or chrome, please
>> press F12 to open the console, and then check whether there is any JS or
>> HTTP error on the console.
>>
>> 2016-06-12 17:08 GMT+08:00 蔡冉 :
>>
>>> I got error 'Oops... Failed to take action.' when trying to load hive
>>> table in Kylin web ui, and no obvious error could be found in the log file,
>>> anyone have met the problem before?
>>>
>>> My environment:
>>>
>>> Version:
>>> hadoop: 2.7.1
>>> hive: 1.2.1
>>> hbase: 1.1.2
>>> kylin: 1.5.2
>>>
>>> We have total 20 physical servers in hadoop cluster, including 2
>>> masters( for HA) and 18 slaves.
>>>
>>> Kylin installed in a docker container (for facilitating the test), and
>>> in this docker container, whole of hadoop, hive and hbase installation dir
>>> were copied from hadoop cluster servers. Commands like 'hadoop fs -ls',
>>> 'hive', 'hbase shell' can run and response successfully. Hive use a remote
>>> mysql server for metastore, the configuration was properly specified in
>>> hive-site.xml.
>>>
>>> Attachment were the log files, any help would be appreciated.
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi
>>
>>
>
--
Best regards,
Shaofeng Shi
[jira] [Created] (KYLIN-1915) no counters for job
wanglaichao created KYLIN-1915:
--
Summary: no counters for job
Key: KYLIN-1915
URL: https://issues.apache.org/jira/browse/KYLIN-1915
Project: Kylin
Issue Type: Bug
Affects Versions: v1.5.2.1
Environment: hadoop-2.4.1 hbase-1.1.5 apache-hive-1.2.1-bin
Reporter: wanglaichao
when i run a simple table , i don't find excetion ,but it runs failed.my
hadoop version is 2.4.1 .who can help me ,thanks.
this is our logs:
2016-07-22 17:41:09,588 DEBUG [pool-5-thread-25] common.HadoopStatusChecker:72
: Hadoop job job_1468901022191_96470 status :
{"app":{"id":"application_1468901022191_96470","user":"hadoop","name":"Kylin_Fact_Distinct_Columns_test_cube_0722_Step","queue":"root.hadoop","state":"FINISHED","finalStatus":"FAILED","progress":100.0,"trackingUI":"History","trackingUrl":"http://namenode171:8088/proxy/application_1468901022191_96470/jobhistory/job/job_1468901022191_96470","diagnostics":"","clusterId":1468901022191,"applicationType":"MAPREDUCE","applicationTags":"","startedTime":1469180339135,"finishedTime":1469180468493,"elapsedTime":129358,"amContainerLogs":"http://datanode94:8042/node/containerlogs/container_1468901022191_96470_01_01/hadoop","amHostHttpAddress":"datanode94:8042","allocatedMB":2048,"allocatedVCores":1,"runningContainers":1}}
2016-07-22 17:41:09,588 DEBUG [pool-5-thread-25] common.HadoopStatusChecker:59
: State of Hadoop job: job_1468901022191_96470:FINISHED-FAILED
2016-07-22 17:41:09,592 DEBUG [pool-5-thread-25] hbase.HBaseResourceStore:262 :
Update row /execute_output/2eee7f50-93ac-4c8b-9a41-bbe7c6846798-01 from oldTs:
1469180459579, to newTs: 1469180469589, operation result: true
2016-07-22 17:41:18,347 INFO [pool-4-thread-1] threadpool.DefaultScheduler:106
: Job Fetcher: 1 running, 1 actual running, 0 ready, 21 others
2016-07-22 17:41:41,704 WARN [pool-5-thread-25] common.HadoopCmdOutput:89 : no
counters for job job_1468901022191_96470
2016-07-22 17:41:41,710 DEBUG [pool-5-thread-25] hbase.HBaseResourceStore:262 :
Update row /execute_output/2eee7f50-93ac-4c8b-9a41-bbe7c6846798-01 from oldTs:
1469180469589, to newTs: 1469180501706, operation result: true
2016-07-22 17:41:41,710 INFO [pool-5-thread-25]
execution.AbstractExecutable:109 : Retry 1
2016-07-22 17:41:41,762 INFO [pool-5-thread-25]
execution.AbstractExecutable:107 : mr_job_id:job_1468901022191_96470 resumed
2016-07-22 17:41:41,767 WARN [pool-5-thread-25] httpclient.HttpMethodBase:682
: Going to buffer response body of large or unknown size. Using
getResponseBodyAsStream instead is recommended.
2016-07-22 17:41:41,767 DEBUG [pool-5-thread-25] common.HadoopStatusChecker:209
: Job job_1468901022191_96470 get status check result.
2016-07-22 17:41:41,767 DEBUG [pool-5-thread-25] common.HadoopStatusChecker:72
: Hadoop job job_1468901022191_96470 status :
{"app":{"id":"application_1468901022191_96470","user":"hadoop","name":"Kylin_Fact_Distinct_Columns_test_cube_0722_Step","queue":"root.hadoop","state":"FINISHED","finalStatus":"FAILED","progress":100.0,"trackingUI":"History","trackingUrl":"http://namenode171:8088/proxy/application_1468901022191_96470/jobhistory/job/job_1468901022191_96470","diagnostics":"","clusterId":1468901022191,"applicationType":"MAPREDUCE","applicationTags":"","startedTime":1469180339135,"finishedTime":1469180468493,"elapsedTime":129358,"amContainerLogs":"http://datanode94:8042/node/containerlogs/container_1468901022191_96470_01_01/hadoop","amHostHttpAddress":"datanode94:8042","allocatedMB":2048,"allocatedVCores":1,"runningContainers":1}}
2016-07-22 17:41:41,767 DEBUG [pool-5-thread-25] common.HadoopStatusChecker:59
: State of Hadoop job: job_1468901022191_96470:FINISHED-FAILED
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KYLIN-1916) no counters for job
wanglaichao created KYLIN-1916: -- Summary: no counters for job Key: KYLIN-1916 URL: https://issues.apache.org/jira/browse/KYLIN-1916 Project: Kylin Issue Type: Bug Components: Web Reporter: wanglaichao Assignee: Zhong,Jason -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1917) TopN counter merge performance improvement
Shaofeng SHI created KYLIN-1917: --- Summary: TopN counter merge performance improvement Key: KYLIN-1917 URL: https://issues.apache.org/jira/browse/KYLIN-1917 Project: Kylin Issue Type: Improvement Components: Job Engine Affects Versions: v1.5.2, v1.5.3 Reporter: Shaofeng SHI Fix For: Backlog Observed the reduce phase of cube build is slow when there is TopN counter. There should be room for performance improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
