Re: [Announce] Apache Kylin 2.3.1 released

2018-03-27 Thread zhenya Sun 
That's great!

> 在 2018年3月28日，上午11:44，Billy Liu  写道：
> 
> The Apache Kylin team is pleased to announce the immediate
> availability of the 2.3.1 release.
> 
> This is a bug fix release after 2.3.0 with 12 bug fixes and
> enhancements; All of the changes in this release can be found in:
> https://kylin.apache.org/docs23/release_notes.html
> 
> You can download the source release and binary packages from Apache
> Kylin's download page: https://kylin.apache.org/download/
> 
> Apache Kylin is an open source Distributed Analytics Engine designed
> to provide SQL interface and multi-dimensional analysis (OLAP) on
> Apache Hadoop, supporting extremely large datasets.
> 
> Apache Kylin lets you query massive data set at sub-second latency in 3 steps:
> 1. Identify a star schema or snowflake schema data set on Hadoop.
> 2. Build Cube on Hadoop.
> 3. Query data with ANSI-SQL and get results in sub-second, via ODBC,
> JDBC or RESTful API.
> 
> Thanks everyone who have contributed to the 2.3.1 release.
> 
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kylin.apache.org/
> 
> 
> With Warm regards
> 
> Billy Liu

[Announce] Apache Kylin 2.3.1 released

2018-03-27 Thread Billy Liu 
The Apache Kylin team is pleased to announce the immediate
availability of the 2.3.1 release.

This is a bug fix release after 2.3.0 with 12 bug fixes and
enhancements; All of the changes in this release can be found in:
https://kylin.apache.org/docs23/release_notes.html

You can download the source release and binary packages from Apache
Kylin's download page: https://kylin.apache.org/download/

Apache Kylin is an open source Distributed Analytics Engine designed
to provide SQL interface and multi-dimensional analysis (OLAP) on
Apache Hadoop, supporting extremely large datasets.

Apache Kylin lets you query massive data set at sub-second latency in 3 steps:
1. Identify a star schema or snowflake schema data set on Hadoop.
2. Build Cube on Hadoop.
3. Query data with ANSI-SQL and get results in sub-second, via ODBC,
JDBC or RESTful API.

Thanks everyone who have contributed to the 2.3.1 release.

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kylin.apache.org/


With Warm regards

Billy Liu

[RESULT] [VOTE] Release apache-kylin-2.3.1

2018-03-27 Thread Billy Liu 
Thanks to everyone who has tested the release candidate and given
their comments and votes.

The tally is as follows.

4 binding +1s:
Billy Liu
Dong Li
Shaofeng Shi
Luke Han

3 non-binding +1s:
Yonghong Zhong
Jianhua Peng
Zhenya Sun

No 0s or -1s.

Therefore I am delighted to announce that the proposal to release
Apache-Kylin-2.3.1 has passed.


With Warm regards

Billy Liu


2018-03-26 15:05 GMT+08:00 zhenya Sun :
> Agree to the proposal
>
> +1
>> 在 2018年3月26日，下午2:50，Jianhua Peng  写道：
>>
>> +1
>>
>> On 2018/03/24 17:36:30, Billy Liu  wrote:
>>> Hi all,
>>>
>>> I have created a build for Apache Kylin 2.3.1, release candidate 1.
>>> This is a bug fix release for Kylin 2.3.0
>>>
>>> Changes highlights:
>>> [KYLIN-3270] Fix the blocking issue in cube optimizing jobs
>>> [KYLIN-3276] Fix the query cache bug with dynamic parameters
>>> [KYLIN-3287] When a shard by column is in dict encoding, dict building 
>>> error.
>>> [KYLIN-3288] "Sqoop To Flat Hive Table" step should specify
>>> "mapreduce.queue.name"
>>> [KYLIN-3278] Kylin should not distribute hive table by random at Step1
>>> [KYLIN-3300] Upgrade jackson-databind for security issue
>>> [KYLIN-3301] Upgrade opensaml for security issue
>>>
>>> 12 issues are fixed in this release.
>>>
>>> Thanks to everyone who has contributed to this release.
>>> Here’s release notes:
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12342940
>>>
>>> The commit to be voted upon:
>>>
>>> https://github.com/apache/kylin/commit/928cc1628c546efe7ec348c467f5d26607c8d7e6
>>>
>>> Its hash is 928cc1628c546efe7ec348c467f5d26607c8d7e6.
>>>
>>> The artifacts to be voted on are located here:
>>> https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.3.1-rc1/
>>>
>>> The hashes of the artifacts are as follows:
>>> apache-kylin-2.3.1-src.tar.gz.md5 541d8dff9e5caaae349cf61ad394b0d6
>>> apache-kylin-2.3.1-src.tar.gz.sha1 7ff10d1770d05a8324252547f95adde990bde9dc
>>>
>>> A staged Maven repository is available for review at:
>>> https://repository.apache.org/content/repositories/orgapachekylin-1049/
>>>
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/billyliu.asc
>>>
>>> Please vote on releasing this package as Apache Kylin 2.3.1.
>>>
>>> The vote is open for the next 72 hours and passes if a majority of
>>> at least three +1 PPMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Kylin 2.3.1
>>> [ ]  0 I don't feel strongly about it, but I'm okay with the release
>>> [ ] -1 Do not release this package because...
>>>
>>>
>>> Here is my vote:
>>>
>>> +1 (binding)
>>>
>>>
>>> With Warm regards
>>>
>>> Billy Liu
>>>
>
>

Build failed in Jenkins: Kylin-SonarQube #436

2018-03-27 Thread Apache Jenkins Server 
See 


Changes:

[nju_yaho] KYLIN-3314 refactor code for cube planner algorithm

--
[...truncated 21.58 KB...]
[CHECKSTYLE] Computing warning deltas based on reference build #435
[PMD] No report found for mojo check
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.0:prepare-agent (pre-test) @ kylin-external 
---
[INFO] surefireArgLine set to 
-javaagent:/home/jenkins/.m2/repository/org/jacoco/org.jacoco.agent/0.8.0/org.jacoco.agent-0.8.0-runtime.jar=destfile=
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ kylin-external 
---
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:exec (install_htrace) @ kylin-external ---
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building Apache Kylin - External HTrace 2.4.0-SNAPSHOT
[INFO] 
Downloading: 
http://repo.maven.apache.org/maven2/org/apache/htrace/htrace-zipkin/3.1.0-incubating/htrace-zipkin-3.1.0-incubating.pom
4/4 KBDownloaded: 
http://repo.maven.apache.org/maven2/org/apache/htrace/htrace-zipkin/3.1.0-incubating/htrace-zipkin-3.1.0-incubating.pom
 (4 KB at 26.1 KB/sec)
Downloading: 
http://repo.maven.apache.org/maven2/org/apache/htrace/htrace-zipkin/3.1.0-incubating/htrace-zipkin-3.1.0-incubating.jar
4/114 KB   8/114 KB   12/114 KB   14/114 KB   18/114 KB   22/114 KB   23/114 KB 
  27/114 KB   27/114 KB   31/114 KB   31/114 KB   35/114 KB   37/114 KB   
41/114 KB   42/114 KB   46/114 KB   50/114 KB   51/114 KB   55/114 KB   59/114 
KB   59/114 KB   63/114 KB   67/114 KB   71/114 KB   72/114 KB   76/114 KB   
80/114 KB   84/114 KB   88/114 KB   92/114 KB   93/114 KB   97/114 KB   101/114 
KB   102/114 KB   106/114 KB   110/114 KB   113/114 KB   114/114 KB 
   Downloaded: 
http://repo.maven.apache.org/maven2/org/apache/htrace/htrace-zipkin/3.1.0-incubating/htrace-zipkin-3.1.0-incubating.jar
 (114 KB at 3231.0 KB/sec)
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ kylin-external-htrace 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.17:check (check-style) @ 
kylin-external-htrace ---
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.0:prepare-agent (pre-test) @ 
kylin-external-htrace ---
[INFO] surefireArgLine set to 
-javaagent:/home/jenkins/.m2/repository/org/jacoco/org.jacoco.agent/0.8.0/org.jacoco.agent-0.8.0-runtime.jar=destfile=
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
kylin-external-htrace ---
[INFO] 
[INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ 
kylin-external-htrace ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ 
kylin-external-htrace ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:copy (copy-jamm) @ 
kylin-external-htrace ---
[INFO] Configured Artifact: com.github.jbellis:jamm:?:jar
[INFO] Copying jamm-0.3.1.jar to 

[INFO] 
[INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ 
kylin-external-htrace ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:testCompile (default-testCompile) @ 
kylin-external-htrace ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ 
kylin-external-htrace ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.0:report (post-test) @ kylin-external-htrace 
---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO] 
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ kylin-external-htrace ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:3.5.1:attach-descriptor (attach-descriptor) @ 
kylin-external-htrace ---
[INFO] 
[INFO] --- maven-shade-plugin:3.0.0:shade (default) @ kylin-external-htrace ---
[INFO] Including org.apache.htrace:htrace-core:jar:3.1.0-incubating in the 
shaded jar.
[INFO] Including org.apache.htrace:htrace-zipkin:jar:3.1.0-incubating in the 
shaded jar.

Error on loading sample and starting Kylin

2018-03-27 Thread Asim Ali 
Hi,
We are evaluating Apache Kylin. I need some help setting up Kylin and
understand its working.

 I have Clouder VM 5.13 and I started most of the services using cloudera
manager and executed
sample scripts

Command history

  export KYLIN_HOME=/usr/local/kylin/apache-kylin-2.3.0-bin
  124  export PATH=$KYLIN_HOME/bin:$PATH
  125  echo $PATH
  128  sudo su -p hdf bin/check-env.sh
  129  sudo su -p hdfs bin/check-env.sh
  130  sudo su -p hdfs bin/sample.sh
Got a permission error gave "hdfs" permission on all the folders in kylin

  131  sudo su -p hdfs mkdir
/usr/local/kylin/apache-kylin-2.3.0-bin/sample_cube/metadata
  132  sudo su -p hdfs
  135  chown -R . hdfs
  136  chown hdfs -R /usr/local/kylin/
  137  sudo chown hdfs -R /usr/local/kylin/
  138  ls -all
Executed sample script
  139  sudo su -p hdfs bin/sample.sh
*System failed on the last step  *

2018-03-27 17:12:05,492 INFO  [main] zookeeper.ZooKeeper:100 : Client
environment:user.dir=/usr/local/kylin/apache-kylin-2.3.0-bin/bin
2018-03-27 17:12:05,492 INFO  [main] zookeeper.ZooKeeper:438 : Initiating
client connection, connectString=quickstart.cloudera:2181
sessionTimeout=6 watcher=hconnection-0x6fd1046d0x0,
quorum=quickstart.cloudera:2181, baseZNode=/hbase
2018-03-27 17:12:05,507 INFO  [main-SendThread(quickstart.cloudera:2181)]
zookeeper.ClientCnxn:975 : Opening socket connection to server
quickstart.cloudera/10.0.2.15:2181. Will not attempt to authenticate using
SASL (unknown error)
2018-03-27 17:12:05,516 INFO  [main-SendThread(quickstart.cloudera:2181)]
zookeeper.ClientCnxn:852 : Socket connection established, initiating
session, client: /10.0.2.15:39397, server: quickstart.cloudera/
10.0.2.15:2181
2018-03-27 17:12:05,521 INFO  [main-SendThread(quickstart.cloudera:2181)]
zookeeper.ClientCnxn:1235 : Session establishment complete on server
quickstart.cloudera/10.0.2.15:2181, sessionid = 0x16269cf569c000a,
negotiated timeout = 6
Exception in thread "main" java.lang.IllegalArgumentException:
*Failed to find metadata store by url: kylin_metadata@hbaseat *
org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:96)
at
org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:108)
at
org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)
at
org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:90)
... 3 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't
get the location for replica 0
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:363)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:160)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:62)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at
org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:862)
at
org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at
org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at
org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:421)
at
org.apache.kylin.storage.hbase.HBaseConnection.tableExists(HBaseConnection.java:281)
at
org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:306)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:111)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:92)
... 8 more
2018-03-27 17:12:10,530 INFO  [close-hbase-conn] hbase.HBaseConnection:137
: Closing HBase connections...
2018-03-27 17:12:10,530 INFO  [close-hbase-conn]
client.ConnectionManager$HConnectionImplementation:1843 : Closing zookeeper
sessionid=0x16269cf569c000a
2018-03-27 17:12:10,533 INFO  [main-EventThread] zookeeper.ClientCnxn:512 :
EventThread shut down
2018-03-27 17:12:10,534 INFO

[jira] [Created] (KYLIN-3322) TopN requires a SUM to work

2018-03-27 Thread liyang (JIRA) 
liyang created KYLIN-3322:
-

 Summary: TopN requires a SUM to work
 Key: KYLIN-3322
 URL: https://issues.apache.org/jira/browse/KYLIN-3322
 Project: Kylin
  Issue Type: Bug
Reporter: liyang
Assignee: liyang


Currently if user creates a measure of TopN seller by sum of price, it is 
required that user also creates a measure of SUM(price). Otherwise, NPE will be 
thrown at query time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3321) Set MALLOC_ARENA_MAX in script

2018-03-27 Thread Ted Yu (JIRA) 
Ted Yu created KYLIN-3321:
-

 Summary: Set MALLOC_ARENA_MAX in script
 Key: KYLIN-3321
 URL: https://issues.apache.org/jira/browse/KYLIN-3321
 Project: Kylin
  Issue Type: Task
Reporter: Ted Yu


conf/setenv.sh would be good place to set MALLOC_ARENA_MAX which prevents 
native memory OOM.

See https://github.com/prestodb/presto/issues/8993



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Upgrade Kylin version to 2.3.0 with some trouble that throws UncheckedExecutionException of HiveClientCache

2018-03-27 Thread zhenya Sun 
It is wonderful!


> 在 2018年3月27日，下午6:30，凡梦星尘  写道：
> 
> Hi zhenya：
> 
> Why not sure?   Now all things is going well.
> 
> 2018-03-27 16:45 GMT+08:00 zhenya Sun :
> 
>> excuse me are you sure this is ready to use according to what you said?
>> 
>> 
>>> 在 2018年3月27日，下午4:37，凡梦星尘  写道：
>>> 
>>> Hi guys:
>>> 
>>> Sorry, It's my mistake. Because the Kylin check environment and said
>>> without HCAT_HOME in path. So I had add it environment setting by manual.
>>> But I add the hive2-hcat libs path then you were this error. Now I change
>>> it down to hive-hcat libs that Kylin2.3 working well.
>>> 
>>> Thanks.
>>> 
>>> 2018-03-27 14:34 GMT+08:00 凡梦星尘 :
>>> 
 At noon I had restore all settings in kylin.properties file, then cube
 build under MR mode without any problems. So there were very strange, I
 think the default setting for use same hadoop configure. Whats different
 about action copy the hive-site in custom hadoop conf directory. As
>> well  I
 had found deep about Hive source code, make sure the NoSuchMethodError
>> were
 right. Because the constructor function is build in Hive2.1x version.
>> Then
 try to upgrade the hive jars under spark jars folder, but still not
 working. So what can I do now?
 
 
 
 PS:
 
 1. Use the Kylin2.0 spark guide line: https://kylin.apache.
 org/docs20/tutorial/cube_spark.html
 2. Use beeline replace for CLI to connect Hive.
 3. Upload the hive-site.xml file for your information.
 4. Throws error source code of Kylin
 
 public void configureJob(Job job) {
 try {
 job.getConfiguration().addResource("hive-site.xml");
 
 HCatInputFormat.setInput(job, dbName, tableName);
 job.setInputFormatClass(HCatInputFormat.class);
 } catch (IOException e) {
 throw new RuntimeException(e);
 }
 }
 
 
 
 Thanks.
 
 
 2018-03-27 9:47 GMT+08:00 凡梦星尘 :
 
> Hi Billy:
> 
> Thanks for your suggest, I will try it. Hope can response good news.
> 
> 2018-03-27 9:11 GMT+08:00 Billy Liu :
> 
>> For the first question, it seems some classpath conflict issue
>> existing there "NoSuchMethodError", try google "HiveMetaStoreClient
>> NoSuchMethodError"
>> 
>> For the second question, to enable the dashboard, please follow the
>> http://kylin.apache.org/docs23/howto/howto_setup_systemcube.html
>> first.
>> 
>> With Warm regards
>> 
>> Billy Liu
>> 
>> 
>> 2018-03-26 17:59 GMT+08:00 凡梦星尘 :
>>> Hi guys:
>>> 
>>> Congratulation  to Kylin release 2.3.0 with so many features.
>>> 
>>> The last weekend I had try to upgrade this newest version to test
>> ENV.
>> All
>>> things going is good and run well sample cube under MR mode. But when
>> I try
>>> to switch Spark mode not success. After fixed some problems that
>> meet a
>>> hard trouble not fixed. The error log see below:
>>> 
>>> java.lang.RuntimeException: java.io.IOException:
>>> com.google.common.util.concurrent.UncheckedExecutionException:
>>> java.lang.RuntimeException: Unable to instantiate
>>> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
>> eMetaStoreClient
>>>   at org.apache.kylin.source.hive.H
>> iveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:116)
>>>   at org.apache.kylin.engine.mr.ste
>> ps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:
>> 121)
>>>   at org.apache.kylin.engine.mr.ste
>> ps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:99)
>>>   at org.apache.kylin.engine.mr.com
>> mon.MapReduceExecutable.doWork(MapReduceExecutable.java:130)
>>>   at org.apache.kylin.job.execution
>> .AbstractExecutable.execute(AbstractExecutable.java:162)
>>>   at org.apache.kylin.job.execution
>> .DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
>>>   at org.apache.kylin.job.execution
>> .AbstractExecutable.execute(AbstractExecutable.java:162)
>>>   at org.apache.kylin.job.impl.thre
>> adpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
>>>   at java.util.concurrent.ThreadPoo
>> lExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>   at java.util.concurrent.ThreadPoo
>> lExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>   at java.lang.Thread.run(Thread.java:748)
>>> Caused by: java.io.IOException:
>>> com.google.common.util.concurrent.UncheckedExecutionException:
>>> java.lang.RuntimeException: Unable to instantiate
>>> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
>> eMetaStoreClient
>>>   at org.apache.hive.hcatalog.mapre
>>

Re: Upgrade Kylin version to 2.3.0 with some trouble that throws UncheckedExecutionException of HiveClientCache

2018-03-27 Thread 凡梦星尘 
Hi zhenya：

Why not sure?   Now all things is going well.

2018-03-27 16:45 GMT+08:00 zhenya Sun :

> excuse me are you sure this is ready to use according to what you said?
>
>
> > 在 2018年3月27日，下午4:37，凡梦星尘  写道：
> >
> > Hi guys:
> >
> > Sorry, It's my mistake. Because the Kylin check environment and said
> > without HCAT_HOME in path. So I had add it environment setting by manual.
> > But I add the hive2-hcat libs path then you were this error. Now I change
> > it down to hive-hcat libs that Kylin2.3 working well.
> >
> > Thanks.
> >
> > 2018-03-27 14:34 GMT+08:00 凡梦星尘 :
> >
> >> At noon I had restore all settings in kylin.properties file, then cube
> >> build under MR mode without any problems. So there were very strange, I
> >> think the default setting for use same hadoop configure. Whats different
> >> about action copy the hive-site in custom hadoop conf directory. As
> well  I
> >> had found deep about Hive source code, make sure the NoSuchMethodError
> were
> >> right. Because the constructor function is build in Hive2.1x version.
> Then
> >> try to upgrade the hive jars under spark jars folder, but still not
> >> working. So what can I do now?
> >>
> >>
> >>
> >> PS:
> >>
> >> 1. Use the Kylin2.0 spark guide line: https://kylin.apache.
> >> org/docs20/tutorial/cube_spark.html
> >> 2. Use beeline replace for CLI to connect Hive.
> >> 3. Upload the hive-site.xml file for your information.
> >> 4. Throws error source code of Kylin
> >>
> >> public void configureJob(Job job) {
> >> try {
> >> job.getConfiguration().addResource("hive-site.xml");
> >>
> >> HCatInputFormat.setInput(job, dbName, tableName);
> >> job.setInputFormatClass(HCatInputFormat.class);
> >> } catch (IOException e) {
> >> throw new RuntimeException(e);
> >> }
> >> }
> >>
> >>
> >>
> >> Thanks.
> >>
> >>
> >> 2018-03-27 9:47 GMT+08:00 凡梦星尘 :
> >>
> >>> Hi Billy:
> >>>
> >>> Thanks for your suggest, I will try it. Hope can response good news.
> >>>
> >>> 2018-03-27 9:11 GMT+08:00 Billy Liu :
> >>>
>  For the first question, it seems some classpath conflict issue
>  existing there "NoSuchMethodError", try google "HiveMetaStoreClient
>  NoSuchMethodError"
> 
>  For the second question, to enable the dashboard, please follow the
>  http://kylin.apache.org/docs23/howto/howto_setup_systemcube.html
>  first.
> 
>  With Warm regards
> 
>  Billy Liu
> 
> 
>  2018-03-26 17:59 GMT+08:00 凡梦星尘 :
> > Hi guys:
> >
> > Congratulation  to Kylin release 2.3.0 with so many features.
> >
> > The last weekend I had try to upgrade this newest version to test
> ENV.
>  All
> > things going is good and run well sample cube under MR mode. But when
>  I try
> > to switch Spark mode not success. After fixed some problems that
> meet a
> > hard trouble not fixed. The error log see below:
> >
> > java.lang.RuntimeException: java.io.IOException:
> > com.google.common.util.concurrent.UncheckedExecutionException:
> > java.lang.RuntimeException: Unable to instantiate
> > org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
>  eMetaStoreClient
> >at org.apache.kylin.source.hive.H
>  iveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:116)
> >at org.apache.kylin.engine.mr.ste
>  ps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:
> 121)
> >at org.apache.kylin.engine.mr.ste
>  ps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:99)
> >at org.apache.kylin.engine.mr.com
>  mon.MapReduceExecutable.doWork(MapReduceExecutable.java:130)
> >at org.apache.kylin.job.execution
>  .AbstractExecutable.execute(AbstractExecutable.java:162)
> >at org.apache.kylin.job.execution
>  .DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
> >at org.apache.kylin.job.execution
>  .AbstractExecutable.execute(AbstractExecutable.java:162)
> >at org.apache.kylin.job.impl.thre
>  adpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
> >at java.util.concurrent.ThreadPoo
>  lExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >at java.util.concurrent.ThreadPoo
>  lExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >at java.lang.Thread.run(Thread.java:748)
> > Caused by: java.io.IOException:
> > com.google.common.util.concurrent.UncheckedExecutionException:
> > java.lang.RuntimeException: Unable to instantiate
> > org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
>  eMetaStoreClient
> >at org.apache.hive.hcatalog.mapre
>  duce.HCatInputFormat.setInput(HCatInputFormat.java:97)
> >at org.apache.hive.hcatalog.mapre
>  duce.HCatInputFormat.setInput(HCatInputFormat.java:51)
>

Re: Some problem when set up kylin

2018-03-27 Thread ShaoFeng Shi 
Hadoop 3.0 is not supported yet; If you have figured out how to run Kylin
on Hadoop 3.0, welcome to share with the community!

2017-11-14 11:23 GMT+08:00 Li Yang :

> Classpath issue. Must be conflicting jars on classpath.
>
> Run "*bin/kylin.sh -v start*" to print full Kylin classpath (find 'kylin
> classpath is') on console. Should get you started to debug classpath.
>
>
> On Mon, Nov 13, 2017 at 12:13 PM, Feng, Zhutian 
> wrote:
>
> > Hadoop 3.0.0
> > Hbase 1.2.6
> > Hive 3.0.0
> > Java 1.8
> > Maven 3.5.2
> >
> > I use "bin/kylin.sh start" to start kylin, but I can not visit
> > 127.0.0.1:7070/kylin
> > There are some errors in "logs/kylin.out":
> >
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> > MaxPermSize=128M; support was removed in 8.0
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in [jar:file:/opt/hbase/lib/
> > slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in [jar:file:/opt/hadoop-3.0.0/
> > share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/
> > impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-
> > slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in [jar:file:/opt/kylin/spark/
> > jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> > usage: java org.apache.catalina.startup.Catalina [ -config {pathname} ]
> [
> > -nonaming ]  { -help | start | stop }
> > Nov 13, 2017 11:13:35 AM org.apache.catalina.core.AprLifecycleListener
> > lifecycleEvent
> > INFO: The APR based Apache Tomcat Native library which allows optimal
> > performance in production environments was not found on the
> > java.library.path: /opt/hadoop-3.0.0/lib/native
> > Nov 13, 2017 11:13:35 AM org.apache.catalina.startup.Catalina load
> > SEVERE: Catalina.start
> > org.apache.catalina.LifecycleException: Failed to initialize component
> > [StandardServer[9005]]
> > at org.apache.catalina.util.LifecycleBase.init(
> > LifecycleBase.java:107)
> > at org.apache.catalina.startup.Catalina.load(Catalina.java:643)
> > at org.apache.catalina.startup.Catalina.start(Catalina.java:682)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:62)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at org.apache.catalina.startup.Bootstrap.start(Bootstrap.
> java:294)
> > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:
> 428)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:62)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> > Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> > org.apache.derby.jdbc.AutoloadedDriver40
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:348)
> > at java.sql.DriverManager.isDriverAllowed(DriverManager.
> java:556)
> > at java.sql.DriverManager.isDriverAllowed(DriverManager.
> java:548)
> > at java.sql.DriverManager.getDrivers(DriverManager.java:446)
> > at org.apache.catalina.core.JreMemoryLeakPreventionListene
> > r.lifecycleEvent(JreMemoryLeakPreventionListener.java:288)
> > at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(
> > LifecycleSupport.java:117)
> > at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(
> > LifecycleBase.java:90)
> > at org.apache.catalina.util.LifecycleBase.setStateInternal(
> > LifecycleBase.java:388)
> > at org.apache.catalina.util.LifecycleBase.init(
> > LifecycleBase.java:101)
> > ... 13 more
> >
> > Nov 13, 2017 11:13:35 AM org.apache.catalina.startup.Catalina load
> > INFO: Initialization processed in 515 ms
> > Nov 13, 2017 11:13:35 AM org.apache.catalina.core.StandardService
> > startInternal
> > INFO: Starting service Catalina
> > Nov 13, 2017 11:13:35 AM org.apache.catalina.core.StandardEngine
> > startInternal
> > INFO: Starting Servlet Engine: Apache Tomcat/7.0.82
> > Nov 13, 2017 11:13:35 AM org.apache.catalina.startup.HostConfig
> deployWAR
> > INFO: Deploying web application archive /opt/kylin/tomcat/webapps/
> > kylin.war
> > Nov 13, 2017 11:13:36 AM org.apache.catalina.core.ContainerBase
> > addChildInternal
>

Re: Upgrade Kylin version to 2.3.0 with some trouble that throws UncheckedExecutionException of HiveClientCache

2018-03-27 Thread Billy Liu 
Thank you for the updates.

With Warm regards

Billy Liu


2018-03-27 16:37 GMT+08:00 凡梦星尘 :
> Hi guys:
>
> Sorry, It's my mistake. Because the Kylin check environment and said
> without HCAT_HOME in path. So I had add it environment setting by manual.
> But I add the hive2-hcat libs path then you were this error. Now I change
> it down to hive-hcat libs that Kylin2.3 working well.
>
> Thanks.
>
> 2018-03-27 14:34 GMT+08:00 凡梦星尘 :
>
>> At noon I had restore all settings in kylin.properties file, then cube
>> build under MR mode without any problems. So there were very strange, I
>> think the default setting for use same hadoop configure. Whats different
>> about action copy the hive-site in custom hadoop conf directory. As well  I
>> had found deep about Hive source code, make sure the NoSuchMethodError were
>> right. Because the constructor function is build in Hive2.1x version. Then
>> try to upgrade the hive jars under spark jars folder, but still not
>> working. So what can I do now?
>>
>>
>>
>> PS:
>>
>> 1. Use the Kylin2.0 spark guide line: https://kylin.apache.
>> org/docs20/tutorial/cube_spark.html
>> 2. Use beeline replace for CLI to connect Hive.
>> 3. Upload the hive-site.xml file for your information.
>> 4. Throws error source code of Kylin
>>
>> public void configureJob(Job job) {
>> try {
>> job.getConfiguration().addResource("hive-site.xml");
>>
>> HCatInputFormat.setInput(job, dbName, tableName);
>> job.setInputFormatClass(HCatInputFormat.class);
>> } catch (IOException e) {
>> throw new RuntimeException(e);
>> }
>> }
>>
>>
>>
>> Thanks.
>>
>>
>> 2018-03-27 9:47 GMT+08:00 凡梦星尘 :
>>
>>> Hi Billy:
>>>
>>> Thanks for your suggest, I will try it. Hope can response good news.
>>>
>>> 2018-03-27 9:11 GMT+08:00 Billy Liu :
>>>
 For the first question, it seems some classpath conflict issue
 existing there "NoSuchMethodError", try google "HiveMetaStoreClient
 NoSuchMethodError"

 For the second question, to enable the dashboard, please follow the
 http://kylin.apache.org/docs23/howto/howto_setup_systemcube.html
 first.

 With Warm regards

 Billy Liu


 2018-03-26 17:59 GMT+08:00 凡梦星尘 :
 > Hi guys:
 >
 > Congratulation  to Kylin release 2.3.0 with so many features.
 >
 > The last weekend I had try to upgrade this newest version to test ENV.
 All
 > things going is good and run well sample cube under MR mode. But when
 I try
 > to switch Spark mode not success. After fixed some problems that meet a
 > hard trouble not fixed. The error log see below:
 >
 > java.lang.RuntimeException: java.io.IOException:
 > com.google.common.util.concurrent.UncheckedExecutionException:
 > java.lang.RuntimeException: Unable to instantiate
 > org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
 eMetaStoreClient
 > at org.apache.kylin.source.hive.H
 iveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:116)
 > at org.apache.kylin.engine.mr.ste
 ps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:121)
 > at org.apache.kylin.engine.mr.ste
 ps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:99)
 > at org.apache.kylin.engine.mr.com
 mon.MapReduceExecutable.doWork(MapReduceExecutable.java:130)
 > at org.apache.kylin.job.execution
 .AbstractExecutable.execute(AbstractExecutable.java:162)
 > at org.apache.kylin.job.execution
 .DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
 > at org.apache.kylin.job.execution
 .AbstractExecutable.execute(AbstractExecutable.java:162)
 > at org.apache.kylin.job.impl.thre
 adpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
 > at java.util.concurrent.ThreadPoo
 lExecutor.runWorker(ThreadPoolExecutor.java:1149)
 > at java.util.concurrent.ThreadPoo
 lExecutor$Worker.run(ThreadPoolExecutor.java:624)
 > at java.lang.Thread.run(Thread.java:748)
 > Caused by: java.io.IOException:
 > com.google.common.util.concurrent.UncheckedExecutionException:
 > java.lang.RuntimeException: Unable to instantiate
 > org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
 eMetaStoreClient
 > at org.apache.hive.hcatalog.mapre
 duce.HCatInputFormat.setInput(HCatInputFormat.java:97)
 > at org.apache.hive.hcatalog.mapre
 duce.HCatInputFormat.setInput(HCatInputFormat.java:51)
 > at org.apache.kylin.source.hive.H
 iveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:113)
 > ... 10 more
 > Caused by: com.google.common.util.concurr
 ent.UncheckedExecutionException:
 > java.lang.RuntimeException: Unable to instantiate
 >

Re: Upgrade Kylin version to 2.3.0 with some trouble that throws UncheckedExecutionException of HiveClientCache

2018-03-27 Thread zhenya Sun 
excuse me are you sure this is ready to use according to what you said?


> 在 2018年3月27日，下午4:37，凡梦星尘  写道：
> 
> Hi guys:
> 
> Sorry, It's my mistake. Because the Kylin check environment and said
> without HCAT_HOME in path. So I had add it environment setting by manual.
> But I add the hive2-hcat libs path then you were this error. Now I change
> it down to hive-hcat libs that Kylin2.3 working well.
> 
> Thanks.
> 
> 2018-03-27 14:34 GMT+08:00 凡梦星尘 :
> 
>> At noon I had restore all settings in kylin.properties file, then cube
>> build under MR mode without any problems. So there were very strange, I
>> think the default setting for use same hadoop configure. Whats different
>> about action copy the hive-site in custom hadoop conf directory. As well  I
>> had found deep about Hive source code, make sure the NoSuchMethodError were
>> right. Because the constructor function is build in Hive2.1x version. Then
>> try to upgrade the hive jars under spark jars folder, but still not
>> working. So what can I do now?
>> 
>> 
>> 
>> PS:
>> 
>> 1. Use the Kylin2.0 spark guide line: https://kylin.apache.
>> org/docs20/tutorial/cube_spark.html
>> 2. Use beeline replace for CLI to connect Hive.
>> 3. Upload the hive-site.xml file for your information.
>> 4. Throws error source code of Kylin
>> 
>> public void configureJob(Job job) {
>> try {
>> job.getConfiguration().addResource("hive-site.xml");
>> 
>> HCatInputFormat.setInput(job, dbName, tableName);
>> job.setInputFormatClass(HCatInputFormat.class);
>> } catch (IOException e) {
>> throw new RuntimeException(e);
>> }
>> }
>> 
>> 
>> 
>> Thanks.
>> 
>> 
>> 2018-03-27 9:47 GMT+08:00 凡梦星尘 :
>> 
>>> Hi Billy:
>>> 
>>> Thanks for your suggest, I will try it. Hope can response good news.
>>> 
>>> 2018-03-27 9:11 GMT+08:00 Billy Liu :
>>> 
 For the first question, it seems some classpath conflict issue
 existing there "NoSuchMethodError", try google "HiveMetaStoreClient
 NoSuchMethodError"
 
 For the second question, to enable the dashboard, please follow the
 http://kylin.apache.org/docs23/howto/howto_setup_systemcube.html
 first.
 
 With Warm regards
 
 Billy Liu
 
 
 2018-03-26 17:59 GMT+08:00 凡梦星尘 :
> Hi guys:
> 
> Congratulation  to Kylin release 2.3.0 with so many features.
> 
> The last weekend I had try to upgrade this newest version to test ENV.
 All
> things going is good and run well sample cube under MR mode. But when
 I try
> to switch Spark mode not success. After fixed some problems that meet a
> hard trouble not fixed. The error log see below:
> 
> java.lang.RuntimeException: java.io.IOException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
 eMetaStoreClient
>at org.apache.kylin.source.hive.H
 iveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:116)
>at org.apache.kylin.engine.mr.ste
 ps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:121)
>at org.apache.kylin.engine.mr.ste
 ps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:99)
>at org.apache.kylin.engine.mr.com
 mon.MapReduceExecutable.doWork(MapReduceExecutable.java:130)
>at org.apache.kylin.job.execution
 .AbstractExecutable.execute(AbstractExecutable.java:162)
>at org.apache.kylin.job.execution
 .DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
>at org.apache.kylin.job.execution
 .AbstractExecutable.execute(AbstractExecutable.java:162)
>at org.apache.kylin.job.impl.thre
 adpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
>at java.util.concurrent.ThreadPoo
 lExecutor.runWorker(ThreadPoolExecutor.java:1149)
>at java.util.concurrent.ThreadPoo
 lExecutor$Worker.run(ThreadPoolExecutor.java:624)
>at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
 eMetaStoreClient
>at org.apache.hive.hcatalog.mapre
 duce.HCatInputFormat.setInput(HCatInputFormat.java:97)
>at org.apache.hive.hcatalog.mapre
 duce.HCatInputFormat.setInput(HCatInputFormat.java:51)
>at org.apache.kylin.source.hive.H
 iveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:113)
>... 10 more
> Caused by: com.google.common.util.concurr
 ent.UncheckedExecutionException:
> java.lang.RuntimeException: Unable to instantiate
>

Re: Upgrade Kylin version to 2.3.0 with some trouble that throws UncheckedExecutionException of HiveClientCache

2018-03-27 Thread 凡梦星尘 
Hi guys:

Sorry, It's my mistake. Because the Kylin check environment and said
without HCAT_HOME in path. So I had add it environment setting by manual.
But I add the hive2-hcat libs path then you were this error. Now I change
it down to hive-hcat libs that Kylin2.3 working well.

Thanks.

2018-03-27 14:34 GMT+08:00 凡梦星尘 :

> At noon I had restore all settings in kylin.properties file, then cube
> build under MR mode without any problems. So there were very strange, I
> think the default setting for use same hadoop configure. Whats different
> about action copy the hive-site in custom hadoop conf directory. As well  I
> had found deep about Hive source code, make sure the NoSuchMethodError were
> right. Because the constructor function is build in Hive2.1x version. Then
> try to upgrade the hive jars under spark jars folder, but still not
> working. So what can I do now?
>
>
>
> PS:
>
> 1. Use the Kylin2.0 spark guide line: https://kylin.apache.
> org/docs20/tutorial/cube_spark.html
> 2. Use beeline replace for CLI to connect Hive.
> 3. Upload the hive-site.xml file for your information.
> 4. Throws error source code of Kylin
>
> public void configureJob(Job job) {
> try {
> job.getConfiguration().addResource("hive-site.xml");
>
> HCatInputFormat.setInput(job, dbName, tableName);
> job.setInputFormatClass(HCatInputFormat.class);
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
> }
>
>
>
> Thanks.
>
>
> 2018-03-27 9:47 GMT+08:00 凡梦星尘 :
>
>> Hi Billy:
>>
>> Thanks for your suggest, I will try it. Hope can response good news.
>>
>> 2018-03-27 9:11 GMT+08:00 Billy Liu :
>>
>>> For the first question, it seems some classpath conflict issue
>>> existing there "NoSuchMethodError", try google "HiveMetaStoreClient
>>> NoSuchMethodError"
>>>
>>> For the second question, to enable the dashboard, please follow the
>>> http://kylin.apache.org/docs23/howto/howto_setup_systemcube.html
>>> first.
>>>
>>> With Warm regards
>>>
>>> Billy Liu
>>>
>>>
>>> 2018-03-26 17:59 GMT+08:00 凡梦星尘 :
>>> > Hi guys:
>>> >
>>> > Congratulation  to Kylin release 2.3.0 with so many features.
>>> >
>>> > The last weekend I had try to upgrade this newest version to test ENV.
>>> All
>>> > things going is good and run well sample cube under MR mode. But when
>>> I try
>>> > to switch Spark mode not success. After fixed some problems that meet a
>>> > hard trouble not fixed. The error log see below:
>>> >
>>> > java.lang.RuntimeException: java.io.IOException:
>>> > com.google.common.util.concurrent.UncheckedExecutionException:
>>> > java.lang.RuntimeException: Unable to instantiate
>>> > org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
>>> eMetaStoreClient
>>> > at org.apache.kylin.source.hive.H
>>> iveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:116)
>>> > at org.apache.kylin.engine.mr.ste
>>> ps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:121)
>>> > at org.apache.kylin.engine.mr.ste
>>> ps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:99)
>>> > at org.apache.kylin.engine.mr.com
>>> mon.MapReduceExecutable.doWork(MapReduceExecutable.java:130)
>>> > at org.apache.kylin.job.execution
>>> .AbstractExecutable.execute(AbstractExecutable.java:162)
>>> > at org.apache.kylin.job.execution
>>> .DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
>>> > at org.apache.kylin.job.execution
>>> .AbstractExecutable.execute(AbstractExecutable.java:162)
>>> > at org.apache.kylin.job.impl.thre
>>> adpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
>>> > at java.util.concurrent.ThreadPoo
>>> lExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> > at java.util.concurrent.ThreadPoo
>>> lExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> > at java.lang.Thread.run(Thread.java:748)
>>> > Caused by: java.io.IOException:
>>> > com.google.common.util.concurrent.UncheckedExecutionException:
>>> > java.lang.RuntimeException: Unable to instantiate
>>> > org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
>>> eMetaStoreClient
>>> > at org.apache.hive.hcatalog.mapre
>>> duce.HCatInputFormat.setInput(HCatInputFormat.java:97)
>>> > at org.apache.hive.hcatalog.mapre
>>> duce.HCatInputFormat.setInput(HCatInputFormat.java:51)
>>> > at org.apache.kylin.source.hive.H
>>> iveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:113)
>>> > ... 10 more
>>> > Caused by: com.google.common.util.concurr
>>> ent.UncheckedExecutionException:
>>> > java.lang.RuntimeException: Unable to instantiate
>>> > org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
>>> eMetaStoreClient
>>> > at com.google.common.cache.LocalC
>>> ache$Segment.get(LocalCache.java:2256)
>>> > at com.google.common.cache.LocalC
>>> ache.get(LocalCache.java:3985)
>>> > at

[GitHub] yiming187 closed pull request #123: KYLIN-3293, fix FixedLenHexDimEnc that return a wrong code length lea…

2018-03-27 Thread GitBox 
yiming187 closed pull request #123: KYLIN-3293, fix FixedLenHexDimEnc that 
return a wrong code length lea…
URL: https://github.com/apache/kylin/pull/123
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/core-cube/src/main/java/org/apache/kylin/cube/gridtable/TrimmedCubeCodeSystem.java
 
b/core-cube/src/main/java/org/apache/kylin/cube/gridtable/TrimmedCubeCodeSystem.java
index a0b230e9b0..261e501d8c 100644
--- 
a/core-cube/src/main/java/org/apache/kylin/cube/gridtable/TrimmedCubeCodeSystem.java
+++ 
b/core-cube/src/main/java/org/apache/kylin/cube/gridtable/TrimmedCubeCodeSystem.java
@@ -51,7 +51,7 @@ public void encodeColumnValue(int col, Object value, int 
roundingFlag, ByteBuffe
 serializer.serialize(value, buf);
 }
 
-private static void writeDimensionEncoding(DimensionEncoding encoding, 
ByteBuffer out) {
+public static void writeDimensionEncoding(DimensionEncoding encoding, 
ByteBuffer out) {
 try {
 if (encoding == null) {
 BytesUtil.writeVInt(1, out);
@@ -71,7 +71,7 @@ private static void writeDimensionEncoding(DimensionEncoding 
encoding, ByteBuffe
 }
 }
 
-private static DimensionEncoding readDimensionEncoding(ByteBuffer in) {
+public static DimensionEncoding readDimensionEncoding(ByteBuffer in) {
 try {
 int isNull = BytesUtil.readVInt(in);
 if (isNull == 1) {
diff --git 
a/core-cube/src/test/java/org/apache/kylin/gridtable/TrimmedCubeCodeSystemTest.java
 
b/core-cube/src/test/java/org/apache/kylin/gridtable/TrimmedCubeCodeSystemTest.java
new file mode 100644
index 00..dc3c762119
--- /dev/null
+++ 
b/core-cube/src/test/java/org/apache/kylin/gridtable/TrimmedCubeCodeSystemTest.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.gridtable;
+
+import static 
org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem.readDimensionEncoding;
+import static 
org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem.writeDimensionEncoding;
+
+import java.nio.ByteBuffer;
+
+import org.apache.kylin.dimension.DimensionEncoding;
+import org.apache.kylin.dimension.FixedLenHexDimEnc;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TrimmedCubeCodeSystemTest {
+@Test
+public void testFixLenHexEncSerDser() {
+FixedLenHexDimEnc enc = new FixedLenHexDimEnc(6);
+ByteBuffer buff = ByteBuffer.allocate(1024);
+writeDimensionEncoding(enc, buff);
+buff.flip();
+DimensionEncoding dimensionEncoding = readDimensionEncoding(buff);
+Assert.assertEquals(3, 
dimensionEncoding.asDataTypeSerializer().peekLength(null));
+}
+}
diff --git 
a/core-metadata/src/main/java/org/apache/kylin/dimension/FixedLenHexDimEnc.java 
b/core-metadata/src/main/java/org/apache/kylin/dimension/FixedLenHexDimEnc.java
index a931450a0b..1d7e3c983e 100644
--- 
a/core-metadata/src/main/java/org/apache/kylin/dimension/FixedLenHexDimEnc.java
+++ 
b/core-metadata/src/main/java/org/apache/kylin/dimension/FixedLenHexDimEnc.java
@@ -40,7 +40,7 @@
  * 
  * 1. "" will become null encode and decode
  * 2. "AB" will become "AB00"
- * 
+ *
  * 
  * Due to these limitations hex representation of hash values(with no padding, 
better with even characters) is more suitable
  */
@@ -166,7 +166,7 @@ public void encode(String valueStr, byte[] output, int 
outputOffset) {
 byte[] value = Bytes.toBytes(valueStr);
 int valueLen = value.length;
 int endOffset = outputOffset + bytelen;
-
+
 if (valueLen > hexLength) {
 if (avoidVerbose++ % 1 == 0) {
 logger.warn("Expect at most " + hexLength + " bytes, but got " 
+ valueLen + ", will truncate, value string: " + Bytes.toString(value, 0, 
valueLen) + " times:" + avoidVerbose);
@@ -274,11 +274,13 @@ public Object valueOf(String str) {
 @Override
 public void writeExternal(ObjectOutput out) throws IOException {

Re: Upgrade Kylin version to 2.3.0 with some trouble that throws UncheckedExecutionException of HiveClientCache

2018-03-27 Thread 凡梦星尘 
At noon I had restore all settings in kylin.properties file, then cube
build under MR mode without any problems. So there were very strange, I
think the default setting for use same hadoop configure. Whats different
about action copy the hive-site in custom hadoop conf directory. As well  I
had found deep about Hive source code, make sure the NoSuchMethodError were
right. Because the constructor function is build in Hive2.1x version. Then
try to upgrade the hive jars under spark jars folder, but still not
working. So what can I do now?



PS:

1. Use the Kylin2.0 spark guide line:
https://kylin.apache.org/docs20/tutorial/cube_spark.html
2. Use beeline replace for CLI to connect Hive.
3. Upload the hive-site.xml file for your information.
4. Throws error source code of Kylin

public void configureJob(Job job) {
try {
job.getConfiguration().addResource("hive-site.xml");

HCatInputFormat.setInput(job, dbName, tableName);
job.setInputFormatClass(HCatInputFormat.class);
} catch (IOException e) {
throw new RuntimeException(e);
}
}



Thanks.


2018-03-27 9:47 GMT+08:00 凡梦星尘 :

> Hi Billy:
>
> Thanks for your suggest, I will try it. Hope can response good news.
>
> 2018-03-27 9:11 GMT+08:00 Billy Liu :
>
>> For the first question, it seems some classpath conflict issue
>> existing there "NoSuchMethodError", try google "HiveMetaStoreClient
>> NoSuchMethodError"
>>
>> For the second question, to enable the dashboard, please follow the
>> http://kylin.apache.org/docs23/howto/howto_setup_systemcube.html
>> first.
>>
>> With Warm regards
>>
>> Billy Liu
>>
>>
>> 2018-03-26 17:59 GMT+08:00 凡梦星尘 :
>> > Hi guys:
>> >
>> > Congratulation  to Kylin release 2.3.0 with so many features.
>> >
>> > The last weekend I had try to upgrade this newest version to test ENV.
>> All
>> > things going is good and run well sample cube under MR mode. But when I
>> try
>> > to switch Spark mode not success. After fixed some problems that meet a
>> > hard trouble not fixed. The error log see below:
>> >
>> > java.lang.RuntimeException: java.io.IOException:
>> > com.google.common.util.concurrent.UncheckedExecutionException:
>> > java.lang.RuntimeException: Unable to instantiate
>> > org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
>> eMetaStoreClient
>> > at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputForma
>> t.configureJob(HiveMRInput.java:116)
>> > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setu
>> pMapper(FactDistinctColumnsJob.java:121)
>> > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(
>> FactDistinctColumnsJob.java:99)
>> > at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork
>> (MapReduceExecutable.java:130)
>> > at org.apache.kylin.job.execution.AbstractExecutable.execute(
>> AbstractExecutable.java:162)
>> > at org.apache.kylin.job.execution.DefaultChainedExecutable.doWo
>> rk(DefaultChainedExecutable.java:67)
>> > at org.apache.kylin.job.execution.AbstractExecutable.execute(
>> AbstractExecutable.java:162)
>> > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRun
>> ner.run(DefaultScheduler.java:300)
>> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1149)
>> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:624)
>> > at java.lang.Thread.run(Thread.java:748)
>> > Caused by: java.io.IOException:
>> > com.google.common.util.concurrent.UncheckedExecutionException:
>> > java.lang.RuntimeException: Unable to instantiate
>> > org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
>> eMetaStoreClient
>> > at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(
>> HCatInputFormat.java:97)
>> > at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(
>> HCatInputFormat.java:51)
>> > at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputForma
>> t.configureJob(HiveMRInput.java:113)
>> > ... 10 more
>> > Caused by: com.google.common.util.concurrent.UncheckedExecutionExceptio
>> n:
>> > java.lang.RuntimeException: Unable to instantiate
>> > org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiv
>> eMetaStoreClient
>> > at com.google.common.cache.LocalCache$Segment.get(LocalCache.
>> java:2256)
>> > at com.google.common.cache.LocalCache.get(LocalCache.java:3985)
>> > at com.google.common.cache.LocalCache$LocalManualCache.get(
>> LocalCache.java:4788)
>> > at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(
>> HiveClientCache.java:292)
>> > at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClie
>> ntCache.java:267)
>> > at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreCli
>> ent(HCatUtil.java:558)
>> > at org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJ
>> obInfo(InitializeInput.java:104)