Re: Can not load hive table in Kylin
Today in a customer's env, I encountered this problem; His environment looks good: bin/find-hive-dependency.sh can identify the jars correctly; but after Kylin start up, still got the ClassNotFound error: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.session.SessionState at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571) at org.apache.kylin.rest.controller.TableController.showHiveDatabases(TableController.java:301) Finally identified the root case is in hbase-env.sh; in which the customer overwrited the "HBASE_CLASSPATH" variable, like: export HBASE_CLASSPATH=/home/hadoop/hbase/conf This is bad as it overwrites the variable totally, while Kylin depends on it to tell JVM which jars need be on classpath. After changing it to: export HBASE_CLASSPATH=/home/hadoop/hbase/conf:${HBASE_CLASSPATH} Kylin can run well without the ClassNotFound error anymore. 2016-06-13 16:30 GMT+08:00 蔡冉 : > Thanks Shaofeng, I checked my chrome console and found below error, and I > also found the error in tomcat log > kylin/tomcat/logs/localhost.2016-06-13.log, which should be the root cause: > > org.springframework.web.util.NestedServletException: Handler processing > failed; nested exception is java.lang.NoClassDefFoundError: > org/apache/hadoop/hive/ql/session/SessionState > > but I can find hive-exec-1.2.1.jar in the result > of find-hive-dependency.sh and the process, is there any other > configuration I missed? > > > 2016-06-12 22:05 GMT+08:00 ShaoFeng Shi : > >> It seems the request wasn't received by the server at all; Most likely be >> a client side or network error; If you're using firefox or chrome, please >> press F12 to open the console, and then check whether there is any JS or >> HTTP error on the console. >> >> 2016-06-12 17:08 GMT+08:00 蔡冉 : >> >>> I got error 'Oops... Failed to take action.' when trying to load hive >>> table in Kylin web ui, and no obvious error could be found in the log file, >>> anyone have met the problem before? >>> >>> My environment: >>> >>> Version: >>> hadoop: 2.7.1 >>> hive: 1.2.1 >>> hbase: 1.1.2 >>> kylin: 1.5.2 >>> >>> We have total 20 physical servers in hadoop cluster, including 2 >>> masters( for HA) and 18 slaves. >>> >>> Kylin installed in a docker container (for facilitating the test), and >>> in this docker container, whole of hadoop, hive and hbase installation dir >>> were copied from hadoop cluster servers. Commands like 'hadoop fs -ls', >>> 'hive', 'hbase shell' can run and response successfully. Hive use a remote >>> mysql server for metastore, the configuration was properly specified in >>> hive-site.xml. >>> >>> Attachment were the log files, any help would be appreciated. >>> >>> >>> >>> >> >> >> -- >> Best regards, >> >> Shaofeng Shi >> >> > -- Best regards, Shaofeng Shi
[jira] [Created] (KYLIN-1915) no counters for job
wanglaichao created KYLIN-1915: -- Summary: no counters for job Key: KYLIN-1915 URL: https://issues.apache.org/jira/browse/KYLIN-1915 Project: Kylin Issue Type: Bug Affects Versions: v1.5.2.1 Environment: hadoop-2.4.1 hbase-1.1.5 apache-hive-1.2.1-bin Reporter: wanglaichao when i run a simple table , i don't find excetion ,but it runs failed.my hadoop version is 2.4.1 .who can help me ,thanks. this is our logs: 2016-07-22 17:41:09,588 DEBUG [pool-5-thread-25] common.HadoopStatusChecker:72 : Hadoop job job_1468901022191_96470 status : {"app":{"id":"application_1468901022191_96470","user":"hadoop","name":"Kylin_Fact_Distinct_Columns_test_cube_0722_Step","queue":"root.hadoop","state":"FINISHED","finalStatus":"FAILED","progress":100.0,"trackingUI":"History","trackingUrl":"http://namenode171:8088/proxy/application_1468901022191_96470/jobhistory/job/job_1468901022191_96470","diagnostics":"","clusterId":1468901022191,"applicationType":"MAPREDUCE","applicationTags":"","startedTime":1469180339135,"finishedTime":1469180468493,"elapsedTime":129358,"amContainerLogs":"http://datanode94:8042/node/containerlogs/container_1468901022191_96470_01_01/hadoop","amHostHttpAddress":"datanode94:8042","allocatedMB":2048,"allocatedVCores":1,"runningContainers":1}} 2016-07-22 17:41:09,588 DEBUG [pool-5-thread-25] common.HadoopStatusChecker:59 : State of Hadoop job: job_1468901022191_96470:FINISHED-FAILED 2016-07-22 17:41:09,592 DEBUG [pool-5-thread-25] hbase.HBaseResourceStore:262 : Update row /execute_output/2eee7f50-93ac-4c8b-9a41-bbe7c6846798-01 from oldTs: 1469180459579, to newTs: 1469180469589, operation result: true 2016-07-22 17:41:18,347 INFO [pool-4-thread-1] threadpool.DefaultScheduler:106 : Job Fetcher: 1 running, 1 actual running, 0 ready, 21 others 2016-07-22 17:41:41,704 WARN [pool-5-thread-25] common.HadoopCmdOutput:89 : no counters for job job_1468901022191_96470 2016-07-22 17:41:41,710 DEBUG [pool-5-thread-25] hbase.HBaseResourceStore:262 : Update row /execute_output/2eee7f50-93ac-4c8b-9a41-bbe7c6846798-01 from oldTs: 1469180469589, to newTs: 1469180501706, operation result: true 2016-07-22 17:41:41,710 INFO [pool-5-thread-25] execution.AbstractExecutable:109 : Retry 1 2016-07-22 17:41:41,762 INFO [pool-5-thread-25] execution.AbstractExecutable:107 : mr_job_id:job_1468901022191_96470 resumed 2016-07-22 17:41:41,767 WARN [pool-5-thread-25] httpclient.HttpMethodBase:682 : Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended. 2016-07-22 17:41:41,767 DEBUG [pool-5-thread-25] common.HadoopStatusChecker:209 : Job job_1468901022191_96470 get status check result. 2016-07-22 17:41:41,767 DEBUG [pool-5-thread-25] common.HadoopStatusChecker:72 : Hadoop job job_1468901022191_96470 status : {"app":{"id":"application_1468901022191_96470","user":"hadoop","name":"Kylin_Fact_Distinct_Columns_test_cube_0722_Step","queue":"root.hadoop","state":"FINISHED","finalStatus":"FAILED","progress":100.0,"trackingUI":"History","trackingUrl":"http://namenode171:8088/proxy/application_1468901022191_96470/jobhistory/job/job_1468901022191_96470","diagnostics":"","clusterId":1468901022191,"applicationType":"MAPREDUCE","applicationTags":"","startedTime":1469180339135,"finishedTime":1469180468493,"elapsedTime":129358,"amContainerLogs":"http://datanode94:8042/node/containerlogs/container_1468901022191_96470_01_01/hadoop","amHostHttpAddress":"datanode94:8042","allocatedMB":2048,"allocatedVCores":1,"runningContainers":1}} 2016-07-22 17:41:41,767 DEBUG [pool-5-thread-25] common.HadoopStatusChecker:59 : State of Hadoop job: job_1468901022191_96470:FINISHED-FAILED -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1916) no counters for job
wanglaichao created KYLIN-1916: -- Summary: no counters for job Key: KYLIN-1916 URL: https://issues.apache.org/jira/browse/KYLIN-1916 Project: Kylin Issue Type: Bug Components: Web Reporter: wanglaichao Assignee: Zhong,Jason -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1917) TopN counter merge performance improvement
Shaofeng SHI created KYLIN-1917: --- Summary: TopN counter merge performance improvement Key: KYLIN-1917 URL: https://issues.apache.org/jira/browse/KYLIN-1917 Project: Kylin Issue Type: Improvement Components: Job Engine Affects Versions: v1.5.2, v1.5.3 Reporter: Shaofeng SHI Fix For: Backlog Observed the reduce phase of cube build is slow when there is TopN counter. There should be room for performance improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)