Re: Support for Hive on Tez or Hive on Spark, cube build automation and best practices

[Li Yang]

In principal, Kylin does not do any scheduling stuff. Because only upstream
ETL knows when the data lands in hive. That's why Kylin provides Rest API
for upstream to call when data is ready.

On Sat, Jan 16, 2016 at 8:18 PM, hongbin ma  wrote:

> ​kylin invokes shell to execute hive commands, so I think Hive on Tez or
> Hive on Spark is transparent to Kylin, have you ever tried out before
> asking? If there's known issue here please open a ticket, the community
> would love to help.
>
> Kylin does not provided complicated scheduling for the moment. However we
> do plan to do something like
> https://issues.apache.org/jira/browse/KYLIN-1249, although it may still
> require external scheduling systems like crond.
>
> Cube consists of segments, cubing job is essentially appending segments to
> it(or refreshing existing segments). Cubing job will not affect cube
> segments util the job is done.
>
> ​
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Re: I got error first on kylin

[Dong Li]

Hi Kevin,

For FileNotFoundException on startup, this is not problem.
See https://issues.apache.org/jira/browse/KYLIN-1298

Thanks,
Dong Li

2016-01-21 18:26 GMT+08:00 kevin :

> I will try hbase0.98 later ,but the error I got when start up is why? why
> kylin need the jars belowe?
>
>  java.io.FileNotFoundException: /{the url of my
> hadoop_home}/contrib/capacity-scheduler/*.jar
>
> 2016-01-21 17:48 GMT+08:00 hongbin ma :
>
> > we don't have stable release for hbase 1.1.2 or higher.
> > "Also, please aware this is not a formal release, and it is not fully
> > tested:"
> >
> > the latest stable version is now 1.2, but it does not work with hbase
> 1.0+
> >
> > On Thu, Jan 21, 2016 at 5:39 PM, kevin  wrote:
> >
> > >  hi,
> > >I install kylin1.3 with hadoop2.7.1 、hive1.2.1 、hbase1.1.2 .
> > >   when I start up the server ,I got error:
> > >
> > > java.io.FileNotFoundException: /{the url of my
> > > hadoop_home}/contrib/capacity-scheduler/*.jar
> > >
> > >   but I can login the kylin webset,but when I build then example cube,I
> > got
> > > another error:
> > >
> > >   [pool-7-thread-1]:[2016-01-21
> > >
> > >
> >
> 16:53:19,367][ERROR][org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:64)]
> > > - error execute
> > > HadoopShellExecutable{id=13f4922e-c352-496e-816c-80ff7bbe0c4f-02,
> > > name=Build Dimension Dictionary, state=RUNNING}
> > > java.lang.NullPointerException
> > >  at
> > >
> > >
> >
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:47)
> > >  at
> > >
> > >
> >
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
> > >  at
> > >
> > >
> >
> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
> > >  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > >  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > >  at
> > >
> > >
> >
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
> > >  at
> > >
> > >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > >  at
> > >
> > >
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> > >  at
> > >
> > >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > >  at
> > >
> > >
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> > >  at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > >  at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > >  at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > > I want to know is this my  environment problem,thanks.
> > >
> > > 2016-01-21
> > > --
> > > kai.shen
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
> >
>



-- 
Thanks,
Dong

[jira] [Created] (KYLIN-1355) Add script to build a java-hotfix package, instead of a full package

[Shaofeng SHI (JIRA)]

Shaofeng SHI created KYLIN-1355:
---

 Summary: Add script to build a java-hotfix package, instead of a 
full package
 Key: KYLIN-1355
 URL: https://issues.apache.org/jira/browse/KYLIN-1355
 Project: Kylin
  Issue Type: Improvement
  Components: Environment 
Reporter: Shaofeng SHI
Assignee: hongbin ma


Sometimes we need deploy only java code change, but he ./scripts/package.sh 
will make a full binary package which cost extra time on building the web 
resources. It would be more efficient if we have a way to package only jar 
files into a tar ball. Then in the server side, we only replace the jars to 
take effective.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KYLIN-1356) use exec-maven-plugin for IT environment provision

[hongbin ma (JIRA)]

hongbin ma created KYLIN-1356:
-

 Summary: use exec-maven-plugin for IT environment provision
 Key: KYLIN-1356
 URL: https://issues.apache.org/jira/browse/KYLIN-1356
 Project: Kylin
  Issue Type: Bug
Reporter: hongbin ma
Assignee: hongbin ma


currently we rely on test cases like BuildCubeWithEngineTest for IT environment 
provision. This is not the right way, trying to leverage exec-maven-plugin for 
this



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: I got error first on kylin

[kevin]

I will try hbase0.98 later ,but the error I got when start up is why? why
kylin need the jars belowe?

 java.io.FileNotFoundException: /{the url of my
hadoop_home}/contrib/capacity-scheduler/*.jar

2016-01-21 17:48 GMT+08:00 hongbin ma :

> we don't have stable release for hbase 1.1.2 or higher.
> "Also, please aware this is not a formal release, and it is not fully
> tested:"
>
> the latest stable version is now 1.2, but it does not work with hbase 1.0+
>
> On Thu, Jan 21, 2016 at 5:39 PM, kevin  wrote:
>
> >  hi,
> >I install kylin1.3 with hadoop2.7.1 、hive1.2.1 、hbase1.1.2 .
> >   when I start up the server ,I got error:
> >
> > java.io.FileNotFoundException: /{the url of my
> > hadoop_home}/contrib/capacity-scheduler/*.jar
> >
> >   but I can login the kylin webset,but when I build then example cube,I
> got
> > another error:
> >
> >   [pool-7-thread-1]:[2016-01-21
> >
> >
> 16:53:19,367][ERROR][org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:64)]
> > - error execute
> > HadoopShellExecutable{id=13f4922e-c352-496e-816c-80ff7bbe0c4f-02,
> > name=Build Dimension Dictionary, state=RUNNING}
> > java.lang.NullPointerException
> >  at
> >
> >
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:47)
> >  at
> >
> >
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
> >  at
> >
> >
> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
> >  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >  at
> >
> >
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
> >  at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> >  at
> >
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> >  at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> >  at
> >
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> >  at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >  at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >  at java.lang.Thread.run(Thread.java:745)
> >
> >
> > I want to know is this my  environment problem,thanks.
> >
> > 2016-01-21
> > --
> > kai.shen
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Re: I got error first on kylin

[hongbin ma]

we don't have stable release for hbase 1.1.2 or higher.
"Also, please aware this is not a formal release, and it is not fully
tested:"

the latest stable version is now 1.2, but it does not work with hbase 1.0+

On Thu, Jan 21, 2016 at 5:39 PM, kevin  wrote:

>  hi,
>I install kylin1.3 with hadoop2.7.1 、hive1.2.1 、hbase1.1.2 .
>   when I start up the server ,I got error:
>
> java.io.FileNotFoundException: /{the url of my
> hadoop_home}/contrib/capacity-scheduler/*.jar
>
>   but I can login the kylin webset,but when I build then example cube,I got
> another error:
>
>   [pool-7-thread-1]:[2016-01-21
>
> 16:53:19,367][ERROR][org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:64)]
> - error execute
> HadoopShellExecutable{id=13f4922e-c352-496e-816c-80ff7bbe0c4f-02,
> name=Build Dimension Dictionary, state=RUNNING}
> java.lang.NullPointerException
>  at
>
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:47)
>  at
>
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
>  at
>
> org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:52)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>  at
>
> org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
>  at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>  at
>
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
>  at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
>  at
>
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
>  at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  at java.lang.Thread.run(Thread.java:745)
>
>
> I want to know is this my  environment problem,thanks.
>
> 2016-01-21
> --
> kai.shen
>



-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

Re: From the Build Base Cuboid Data step to Build N-Dimension steps, Too much time is taken.

[杨海乐]

I find that The reason is the precision of count distinct measure. The 
precision  is 1.2% . So the steps is too slow even though the data is
little(million).Can I sloue the problem by reducing the value of
kylin.job.mapreduce.default.reduce.input.mb？

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/From-the-Build-Base-Cuboid-Data-step-to-Build-N-Dimension-steps-Too-much-time-is-taken-tp3351p3368.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

how does Kylin decide which cube to use for the SQL query?

[zhong zhang]

Hi All,

After several cubes are built, we put a query in the UI.
How does Kylin decide which cube to use for this query?
My guess is that it is based on the join conditions in the
data model?

If we create two cubes with the exactly same data model
(same join conditions) but with different dimensions and
measures, how does Kylin know which cube to use for
a query?

Best regards,
Zhong

TableExistsException: kylin_metadata_acl

[kevin]

HI,all
   when I start up kylin , I got the error below:

  org.apache.hadoop.hbase.TableExistsException: kylin_metadata_acl

  but when I use hbase shell : desc 'kylin_metadata_acl'  got nothing.

  I have reformat hdfs , rebuild hive's metastore database,but it's not
work.

Re: which hadoop version is matched?

[Luke Han]

Hi Kevin,
Please refer to latest document here:
http://kylin.apache.org/docs/install/index.html

Thanks.
Luke


Best Regards!
-

Luke Han

On Fri, Jan 22, 2016 at 2:36 PM, kevin  wrote:

> *hi,all :*
> *I installed hadoop2.7.1 \ hive1.2.1 \ zookpper3.4.6 \
> hbase-0.98.16.1-hadoop2 \ apache-kylin-1.2 *
>
> *when I build the sample cube . I got error : *
>
>
>
> *[pool-5-thread-2]:[2016-01-22
> 14:20:32,671][ERROR][org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:134)]
> - ExecuteException
> job:46b5d7ca-b68a-486b-8750-f34a45396c51org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.NoSuchMethodError:
> org.apache.hadoop.yarn.conf.YarnConfiguration.getServiceAddressConfKeys(Lorg/apache/hadoop/conf/Configuration;)Ljava/util/List;
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:111)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)Caused by:
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.NoSuchMethodError:
> org.apache.hadoop.yarn.conf.YarnConfiguration.getServiceAddressConfKeys(Lorg/apache/hadoop/conf/Configuration;)Ljava/util/List;
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:111)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> ... 4 more*
>
>
> Maybe the hadoop version 2.7 is too hight ，I want to kown the highest
> version of hadoop be supported ?
>
>
> thank,
>
> kevin.
>

Re: Re: how does Kylin decide which cube to use for the SQL query?

[13802880...@139.com]

we have a case like this:
CubeA : date_id,hour_id,service_type,user,count1,count2
the rowkey sequence is : date_id+hour_id+service_type+user
this is ok when i select all the users who use serviceA, but if we want to find 
all the services that userA used, it's became very slow; 
so we create another cubeB, erverthing is the same only the rowkey sequence 
changed:  
CubeB: date_id+hour_id+user+service_type

now the problem comes, if i put cubeB in the same project with cubeA, query 
scene in cubeA become very slow, so we have to build two project!



From: hongbin ma
Date: 2016-01-22 15:13
To: dev
Subject: Re: how does Kylin decide which cube to use for the SQL query?
this is somewhere kylin can improve.
 
i opened a ticket KYLIN-1358 - revisit on cube selection within same project
 ,please comment what
you're expecting,and let's discuss to improve it.
 
On Fri, Jan 22, 2016 at 8:59 AM, 13802880...@139.com <13802880...@139.com>
wrote:
 
> kylin will evaluate the cost and select the best way, but in our case, the
> evaluation is far from perfect, so we have to create another project and
> cube;
>
>

>
> From: zhong zhang
> Date: 2016-01-22 07:50
> To: dev
> Subject: how does Kylin decide which cube to use for the SQL query?
> Hi All,
>
> After several cubes are built, we put a query in the UI.
> How does Kylin decide which cube to use for this query?
> My guess is that it is based on the join conditions in the
> data model?
>
> If we create two cubes with the exactly same data model
> (same join conditions) but with different dimensions and
> measures, how does Kylin know which cube to use for
> a query?
>
> Best regards,
> Zhong
>
 
 
 
-- 
Regards,
 
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

Re: kylin job压缩支持的参数！

[Li Yang]

http://kylin.apache.org/docs/install/advance_settings.html

Here you find settings about compression.

On Tue, Jan 19, 2016 at 8:33 PM, hongbin ma  wrote:

> ​you can comment out all entries containing snappy in kylin_hive_conf.xml,
> kylin_job_conf.xml and kylin.properties (find them in $KYLIN_HOME/conf​)
>
> in that case MR will use no compression, and storage in hbase will not use
> any, either.
>
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Re: TableExistsException: kylin_metadata_acl

[kevin]

it's a problem of hbase's zookeeper

2016-01-22 13:25 GMT+08:00 kevin :

> HI,all
>when I start up kylin , I got the error below:
>
>   org.apache.hadoop.hbase.TableExistsException: kylin_metadata_acl
>
>   but when I use hbase shell : desc 'kylin_metadata_acl'  got nothing.
>
>   I have reformat hdfs , rebuild hive's metastore database,but it's not
> work.
>

Re: Kylin service crash easily while building cube in HDP sandbox

[Li Yang]

What's the HDP version?  Kylin 1.2 only works with HDP 2.2.4.  Make sure
you are NOT running on latest HDP 2.3

On Tue, Jan 19, 2016 at 1:34 PM, 宋轶  wrote:

> I remember we can config the service to be a job engine or a query engine.
>
> > From: mahong...@apache.org
> > Date: Mon, 18 Jan 2016 15:50:07 +0800
> > Subject: Re: Kylin service crash easily while building cube in HDP
> sandbox
> > To: dev@kylin.apache.org
> >
> > btw "Calculate HTable Region Splits" should not cause OOM, can you do an
> > analysis on the dump?
> >
> > On Mon, Jan 18, 2016 at 3:48 PM, hongbin ma 
> wrote:
> >
> > > what's the size of the cube you're building? the default configuration
> for
> > > Xmx and Xms is not very big.
> > >
> > > kylin does provided sth similar to what you meant by "different
> > > lifecycles", checkout
> > > http://kylin.apache.org/docs/install/kylin_cluster.html. The reason
> why
> > > we put them into same tomcat instance is to ease the installation for
> > > newbies.
> > >
> > >
> > > On Mon, Jan 18, 2016 at 2:44 PM, Edward Zhang  >
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I was trying to play Kylin 1.2 in HDP sandbox, but unfortunately it
> always
> > >> fails while it builds cube through Kylin service. The symptom is that
> > >> Kylin
> > >> tomcat service failed each time when it is about in the step 12, which
> > >> is "Calculate
> > >> HTable Region Splits"
> > >>
> > >> I suspected it is out-of-memory issue, so I enabled GC log but did
> not see
> > >> any full gc and no dump also.
> > >>
> > >> What is the possible reason for Kylin service crash? Do you have
> similar
> > >> experience.
> > >>
> > >> Also what I observed is that Kylin service is used for both job
> client and
> > >> ad-hoc query, any consideration of why not decouple the 2
> functionalities
> > >> with different lifecycles into different processes?
> > >>
> > >>
> > >> Thanks
> > >> Edward
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > *Bin Mahone | 马洪宾*
> > > Apache Kylin: http://kylin.io
> > > Github: https://github.com/binmahone
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
>
>

[jira] [Created] (KYLIN-1358) revisit on cube selection within same project

[hongbin ma (JIRA)]

hongbin ma created KYLIN-1358:
-

 Summary: revisit on cube selection within same project
 Key: KYLIN-1358
 URL: https://issues.apache.org/jira/browse/KYLIN-1358
 Project: Kylin
  Issue Type: Improvement
Reporter: hongbin ma
Assignee: hongbin ma


current cost based selection algorithm is far form perfect. need to collect 
requirements from the users and rework on it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[Suggestion] Kylin Dev Document would explicitly tell dependent hive job jars

[Edward Zhang]

Hi,
When start cubing jobs in Kylin dev environment for example from Intellij,
we should explicitly set hive dependent jars which
include FactDistinctColumnsMapper etc.

This is missing in http://kylin.apache.org/development/dev_env.html. We can
tell developer to do the following:

1) In IDE,
set -Dkylin.hive.dependency=/usr/hdp/current/kylinjob/kylin-job-1.2-job.jar

2) copy the corresponding jar file
under kylin/job/target/kylin-job-1.2-job.jar to sandbox if you use sandbox
as development env.

I understand BuildCubeWithEngineTest may directly run some scrip to get
hive dependent jars, but in IDE, looks we have to do manual work above.

Please suggest if that is correct.

Thanks
Edward

which hadoop version is matched?

[kevin]

*hi,all :*
*I installed hadoop2.7.1 \ hive1.2.1 \ zookpper3.4.6 \
hbase-0.98.16.1-hadoop2 \ apache-kylin-1.2 *

*when I build the sample cube . I got error : *



*[pool-5-thread-2]:[2016-01-22
14:20:32,671][ERROR][org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:134)]
- ExecuteException
job:46b5d7ca-b68a-486b-8750-f34a45396c51org.apache.kylin.job.exception.ExecuteException:
org.apache.kylin.job.exception.ExecuteException:
java.lang.NoSuchMethodError:
org.apache.hadoop.yarn.conf.YarnConfiguration.getServiceAddressConfKeys(Lorg/apache/hadoop/conf/Configuration;)Ljava/util/List;
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:111)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)Caused by:
org.apache.kylin.job.exception.ExecuteException:
java.lang.NoSuchMethodError:
org.apache.hadoop.yarn.conf.YarnConfiguration.getServiceAddressConfKeys(Lorg/apache/hadoop/conf/Configuration;)Ljava/util/List;
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:111)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
... 4 more*


Maybe the hadoop version 2.7 is too hight ，I want to kown the highest
version of hadoop be supported ?


thank,

kevin.

Re: [Suggestion] Kylin Dev Document would explicitly tell dependent hive job jars

[Luke Han]

Hi Edward,
Would you mind to open JIRA and submit a patch for such improvement?

Thanks.
Luke


Best Regards!
-

Luke Han

On Fri, Jan 22, 2016 at 2:07 PM, Edward Zhang 
wrote:

> Hi,
> When start cubing jobs in Kylin dev environment for example from Intellij,
> we should explicitly set hive dependent jars which
> include FactDistinctColumnsMapper etc.
>
> This is missing in http://kylin.apache.org/development/dev_env.html. We
> can
> tell developer to do the following:
>
> 1) In IDE,
> set -Dkylin.hive.dependency=/usr/hdp/current/kylinjob/kylin-job-1.2-job.jar
>
> 2) copy the corresponding jar file
> under kylin/job/target/kylin-job-1.2-job.jar to sandbox if you use sandbox
> as development env.
>
> I understand BuildCubeWithEngineTest may directly run some scrip to get
> hive dependent jars, but in IDE, looks we have to do manual work above.
>
> Please suggest if that is correct.
>
> Thanks
> Edward
>

Re: Re: how does Kylin decide which cube to use for the SQL query?

[hongbin ma]

i see​, so cube selection should honor cubes with a better rowkey order
respecting the current query.

any other scenarios?

-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone