Re: org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread ShaoFeng Shi
Hi hefeng,

It seems the hcatalog jars doesn't exist on your hadoop node; The solution
is to upload the jar files to an HDFS folder, and then set  that patch as
the value of "kylin.job.mr.lib.dir" in kylin.properties, you can checkout
this JIRA: https://issues.apache.org/jira/browse/KYLIN-1021

In our env, the "kylin.job.mr.lib.dir" folder has the following 4 jar
files, just for your reference:
hive-common-xx.jar
hive-exec-xx.jar
hive-hcatalog-core-xx.jar
hive-metastore-xx.jar

Here "xx" means the version number;

Just take a try and let us know whether it works.

2016-01-05 14:47 GMT+08:00 和风 <363938...@qq.com>:

> Thanks for your help. error logs:
> [pool-7-thread-1]:[2016-01-05
> 14:45:31,312][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
> - job id:d0e2f259-9541-4b6f-9f54-c502781549e2-00 from RUNNING to SUCCEED
> [pool-7-thread-1]:[2016-01-05
> 14:45:31,438][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
> - job id:d0e2f259-9541-4b6f-9f54-c502781549e2 from RUNNING to READY
> [pool-6-thread-1]:[2016-01-05
> 14:45:31,483][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:102)]
> - CubingJob{id=d0e2f259-9541-4b6f-9f54-c502781549e2, name=learn_kylin_four
> - 2015020100_2015122900 - BUILD - GMT-08:00 2016-01-04 22:44:05,
> state=READY} prepare to schedule
> [pool-6-thread-1]:[2016-01-05
> 14:45:31,484][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:106)]
> - CubingJob{id=d0e2f259-9541-4b6f-9f54-c502781549e2, name=learn_kylin_four
> - 2015020100_2015122900 - BUILD - GMT-08:00 2016-01-04 22:44:05,
> state=READY} scheduled
> [pool-6-thread-1]:[2016-01-05
> 14:45:31,490][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:112)]
> - Job Fetcher: 0 running, 1 actual running, 1 ready, 5 others
> [pool-7-thread-2]:[2016-01-05
> 14:45:31,560][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
> - job id:d0e2f259-9541-4b6f-9f54-c502781549e2 from READY to RUNNING
> [pool-7-thread-2]:[2016-01-05
> 14:45:31,586][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
> - job id:d0e2f259-9541-4b6f-9f54-c502781549e2-01 from READY to RUNNING
> [pool-7-thread-2]:[2016-01-05
> 14:45:31,599][INFO][org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:115)]
> - parameters of the MapReduceExecutable:
> [pool-7-thread-2]:[2016-01-05
> 14:45:31,599][INFO][org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:116)]
> -  -conf /usr/local/kylin/conf/kylin_job_conf.xml -cubename
> learn_kylin_four -output
> /kylin/kylin_metadata/kylin-d0e2f259-9541-4b6f-9f54-c502781549e2/learn_kylin_four/fact_distinct_columns
> -jobname Kylin_Fact_Distinct_Columns_learn_kylin_four_Step -tablename
> default.kylin_intermediate_learn_kylin_four_2015020100_2015122900_d0e2f259_9541_4b6f_9f54_c502781549e2
> [pool-7-thread-2]:[2016-01-05
> 14:45:31,661][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:137)]
> - append job jar: /usr/local/kylin/lib/kylin-job-1.2.jar
> [pool-7-thread-2]:[2016-01-05
> 14:45:31,661][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:144)]
> - append kylin.hive.dependency:
> /usr/local/hive/conf:/usr/local/hive/lib/jsr305-1.3.9.jar:/usr/local/hive/lib/jetty-all-7.6.0.v20120127.jar:/usr/local/hive/lib/servlet-api-2.5.jar:/usr/local/hive/lib/jets3t-0.9.0.jar:/usr/local/hive/lib/accumulo-core-1.6.0.jar:/usr/local/hive/lib/libfb303-0.9.2.jar:/usr/local/hive/lib/json-serde-1.3.6-jar-with-dependencies.jar:/usr/local/hive/lib/commons-httpclient-3.0.1.jar:/usr/local/hive/lib/ivy-2.4.0.jar:/usr/local/hive/lib/hbase-examples-0.98.16.1-hadoop2.jar:/usr/local/hive/lib/jersey-client-1.9.jar:/usr/local/hive/lib/jersey-server-1.9.jar:/usr/local/hive/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hive/lib/zookeeper-3.4.6.jar:/usr/local/hive/lib/bonecp-0.8.0.RELEASE.jar:/usr/local/hive/lib/activation-1.1.jar:/usr/local/hive/lib/snappy-java-1.0.5.jar:/usr/local/hive/lib/commons-cli-1.2.jar:/usr/local/hive/lib/ST4-4.0.4.jar:/usr/local/hive/lib/asm-3.1.jar:/usr/local/hive/lib/hive-common-1.2.1.jar:/usr/local/hive/lib/avro-1.7.5.jar:/usr/local/hive/lib/findbugs-annotations-1.3.9-1.jar:/usr/local/hive/lib/accumulo-trace-1.6.0.jar:/usr/local/hive/lib/jcommander-1.32.jar:/usr/local/hive/lib/commons-lang-2.6.jar:/usr/local/hive/lib/hadoop-yarn-client-2.7.1.jar:/usr/local/hive/lib/hadoop-mapreduce-client-common-2.7.1.jar:/usr/local/hive/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hive/lib/hbase-hadoop-compat-0.98.16.1-hadoop2.jar:/usr/local/hive/lib/stringtemplate-3.2.1.jar:/usr/local/hive/lib/hbase-it-0.98.16.1-hadoop2-tests.jar:/usr/local/hive/lib/hadoop-yarn-api-2.7.1.jar:/u

[jira] [Created] (KYLIN-1287) UI update for streaming build action

2016-01-04 Thread Zhong,Jason (JIRA)
Zhong,Jason created KYLIN-1287:
--

 Summary: UI update for streaming build action
 Key: KYLIN-1287
 URL: https://issues.apache.org/jira/browse/KYLIN-1287
 Project: Kylin
  Issue Type: Improvement
  Components: Web 
Affects Versions: v2.0
Reporter: Zhong,Jason
Assignee: Zhong,Jason
 Fix For: 2.0


for streaming cube, it's not build from GUI, each user can schedule it in their 
own environment,when user click build on GUI,need a tip guide user to know how 
to schedule streaming cube build.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


?????? org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread ????
Thanks for your help. error logs:
[pool-7-thread-1]:[2016-01-05 
14:45:31,312][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
 - job id:d0e2f259-9541-4b6f-9f54-c502781549e2-00 from RUNNING to SUCCEED
[pool-7-thread-1]:[2016-01-05 
14:45:31,438][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
 - job id:d0e2f259-9541-4b6f-9f54-c502781549e2 from RUNNING to READY
[pool-6-thread-1]:[2016-01-05 
14:45:31,483][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:102)]
 - CubingJob{id=d0e2f259-9541-4b6f-9f54-c502781549e2, name=learn_kylin_four - 
2015020100_2015122900 - BUILD - GMT-08:00 2016-01-04 22:44:05, 
state=READY} prepare to schedule
[pool-6-thread-1]:[2016-01-05 
14:45:31,484][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:106)]
 - CubingJob{id=d0e2f259-9541-4b6f-9f54-c502781549e2, name=learn_kylin_four - 
2015020100_2015122900 - BUILD - GMT-08:00 2016-01-04 22:44:05, 
state=READY} scheduled
[pool-6-thread-1]:[2016-01-05 
14:45:31,490][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:112)]
 - Job Fetcher: 0 running, 1 actual running, 1 ready, 5 others
[pool-7-thread-2]:[2016-01-05 
14:45:31,560][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
 - job id:d0e2f259-9541-4b6f-9f54-c502781549e2 from READY to RUNNING
[pool-7-thread-2]:[2016-01-05 
14:45:31,586][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
 - job id:d0e2f259-9541-4b6f-9f54-c502781549e2-01 from READY to RUNNING
[pool-7-thread-2]:[2016-01-05 
14:45:31,599][INFO][org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:115)]
 - parameters of the MapReduceExecutable:
[pool-7-thread-2]:[2016-01-05 
14:45:31,599][INFO][org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:116)]
 -  -conf /usr/local/kylin/conf/kylin_job_conf.xml -cubename learn_kylin_four 
-output 
/kylin/kylin_metadata/kylin-d0e2f259-9541-4b6f-9f54-c502781549e2/learn_kylin_four/fact_distinct_columns
 -jobname Kylin_Fact_Distinct_Columns_learn_kylin_four_Step -tablename 
default.kylin_intermediate_learn_kylin_four_2015020100_2015122900_d0e2f259_9541_4b6f_9f54_c502781549e2
[pool-7-thread-2]:[2016-01-05 
14:45:31,661][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:137)]
 - append job jar: /usr/local/kylin/lib/kylin-job-1.2.jar
[pool-7-thread-2]:[2016-01-05 
14:45:31,661][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:144)]
 - append kylin.hive.dependency: 
/usr/local/hive/conf:/usr/local/hive/lib/jsr305-1.3.9.jar:/usr/local/hive/lib/jetty-all-7.6.0.v20120127.jar:/usr/local/hive/lib/servlet-api-2.5.jar:/usr/local/hive/lib/jets3t-0.9.0.jar:/usr/local/hive/lib/accumulo-core-1.6.0.jar:/usr/local/hive/lib/libfb303-0.9.2.jar:/usr/local/hive/lib/json-serde-1.3.6-jar-with-dependencies.jar:/usr/local/hive/lib/commons-httpclient-3.0.1.jar:/usr/local/hive/lib/ivy-2.4.0.jar:/usr/local/hive/lib/hbase-examples-0.98.16.1-hadoop2.jar:/usr/local/hive/lib/jersey-client-1.9.jar:/usr/local/hive/lib/jersey-server-1.9.jar:/usr/local/hive/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hive/lib/zookeeper-3.4.6.jar:/usr/local/hive/lib/bonecp-0.8.0.RELEASE.jar:/usr/local/hive/lib/activation-1.1.jar:/usr/local/hive/lib/snappy-java-1.0.5.jar:/usr/local/hive/lib/commons-cli-1.2.jar:/usr/local/hive/lib/ST4-4.0.4.jar:/usr/local/hive/lib/asm-3.1.jar:/usr/local/hive/lib/hive-common-1.2.1.jar:/usr/local/hive/lib/avro-1.7.5.jar:/usr/local/hive/lib/findbugs-annotations-1.3.9-1.jar:/usr/local/hive/lib/accumulo-trace-1.6.0.jar:/usr/local/hive/lib/jcommander-1.32.jar:/usr/local/hive/lib/commons-lang-2.6.jar:/usr/local/hive/lib/hadoop-yarn-client-2.7.1.jar:/usr/local/hive/lib/hadoop-mapreduce-client-common-2.7.1.jar:/usr/local/hive/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hive/lib/hbase-hadoop-compat-0.98.16.1-hadoop2.jar:/usr/local/hive/lib/stringtemplate-3.2.1.jar:/usr/local/hive/lib/hbase-it-0.98.16.1-hadoop2-tests.jar:/usr/local/hive/lib/hadoop-yarn-api-2.7.1.jar:/usr/local/hive/lib/hadoop-yarn-common-2.7.1.jar:/usr/local/hive/lib/log4j-1.2.16.jar:/usr/local/hive/lib/hadoop-mapreduce-client-core-2.7.1.jar:/usr/local/hive/lib/protobuf-java-2.5.0.jar:/usr/local/hive/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hive/lib/jetty-6.1.26.jar:/usr/local/hive/lib/hbase-common-0.98.16.1-hadoop2.jar:/usr/local/hive/lib/jasper-compiler-5.5.23.jar:/usr/local/hive/lib/paranamer-2.3.jar:/usr/local/hive/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hive/lib/java-xmlbuilder-0.4.jar:/usr/local/hive/lib/hbase-thrift-0.98.16.1-hadoop2.jar:/usr/local/hive/lib/snappy-java-1.0.4.1.jar:/usr/local/hive/lib/netty-all-4.0.23.Final.jar:/usr/local/hive

Re: org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread ShaoFeng Shi
Hi 和风, screenshot is search engine unfriendly, please use text as much as
possible

2016-01-05 14:34 GMT+08:00 hongbin ma :

> can't see attachment. please provide detailed log
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>



-- 
Best regards,

Shaofeng Shi


Re: org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread hongbin ma
can't see attachment. please provide detailed log


-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone


?????? org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread ????
HI,
   I remove compression.codec in the kylin_job_conf.xml;but build cube have a 
new error;
error tip: failed to generate insert data sql  for intermediate table;










 




--  --
??: "hongbin ma";;
: 2016??1??5??(??) 1:04
??: "dev"; 

: Re: org.apache.hadoop.hive.ql.metadata.HiveException



agree with Sai

On Tue, Jan 5, 2016 at 12:24 PM, Kiriti Sai 
wrote:

> Hi,
> This error is caused because there is no Snappy compression codec available
> in your setup and Kylin expects it by default.
> As a work around, you can disable the use of snappy in the configuration
> files of Kylin.
> > Comment the compression.codec line in kylin.properties
> > comment the properties in the kylin_job_conf.xml which are related to
> compression. I guess there are around 4 properties to be commented.
>
> This was the work around I used for a while but its recommended to use
> compression techniques to minimize the memory shuffling between reducers.
>
> Thank you.
> Sai Kiriti B
> On Jan 5, 2016 12:31 PM, "" <363938...@qq.com> wrote:
>
> > hi:
> >   execution "bulid" cube, jobs exception :
> > org.apache.hadoop.hive.ql.metadata.HiveException:
> > org.apache.hadoop.hive.ql.metadata.HiveException
> >
> >
> > logs:
> >
> >
> > OS command error exit with 2 -- hive  -e "USE default;
> > DROP TABLE IF EXISTS
> >
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db;
> >
> >
> > CREATE EXTERNAL TABLE IF NOT EXISTS
> >
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> > (
> > DEFAULT_KYLIN_CAL_DT_AGE_FOR_QTR_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_MONTH_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_DT_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_RTL_MONTH_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_CS_WEEK_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_YEAR_ID string
> > )
> > ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
> > STORED AS SEQUENCEFILE
> > LOCATION
> >
> '/kylin/kylin_metadata/kylin-d22e7c10-032a-4d22-a802-3b74937e86db/kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db';
> >
> >
> > SET mapreduce.job.split.metainfo.maxsize=-1;
> > SET mapred.compress.map.output=true;
> > SET
> >
> mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> > SET mapred.output.compress=true;
> > SET
> >
> mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> > SET mapred.output.compression.type=BLOCK;
> > SET mapreduce.job.max.split.locations=2000;
> > SET dfs.replication=2;
> > SET hive.merge.mapfiles=true;
> > SET hive.merge.mapredfiles=true;
> > SET hive.merge.size.per.task=268435456;
> > SET hive.support.concurrency=false;
> > SET hive.exec.compress.output=true;
> > SET hive.auto.convert.join.noconditionaltask = true;
> > SET hive.auto.convert.join.noconditionaltask.size = 3;
> > INSERT OVERWRITE TABLE
> >
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> > SELECT
> > KYLIN_CAL_DT.AGE_FOR_QTR_ID
> > ,KYLIN_CAL_DT.AGE_FOR_MONTH_ID
> > ,KYLIN_CAL_DT.AGE_FOR_DT_ID
> > ,KYLIN_CAL_DT.AGE_FOR_RTL_MONTH_ID
> > ,KYLIN_CAL_DT.AGE_FOR_CS_WEEK_ID
> > ,KYLIN_CAL_DT.YEAR_ID
> > FROM DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
> > WHERE (KYLIN_CAL_DT.CAL_DT >= '2013-12-29' AND KYLIN_CAL_DT.CAL_DT <
> > '2016-01-12')
> > ;
> >
> >
> > "
> >
> >
> > Logging initialized using configuration in
> > jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> >
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> [jar:file:/usr/local/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> > OK
> > Time taken: 0.936 seconds
> > OK
> > Time taken: 0.112 seconds
> > OK
> > Time taken: 0.438 seconds
> > Query ID = root_20160105105405_88149f4a-a970-47d0-ba32-9a21ee5afde3
> > Total jobs = 3
> > Launching Job 1 out of 3
> > Number of reduce tasks is set to 0 since there's no reduce operator
> > Starting Job = job_1449731904014_1636, Tracking URL =
> > http://cloud001:8088/proxy/application_1449731904014_1636/
> > Kill Command = /usr/local/hadoop/bin/hadoop job  -kill
> > job_1449731904014_1636
> > Hadoop job information for Stage-1: number of mappers: 1; number of
> > reducers: 0
> > 2016-01-05 10:54:26,177 Stage-1 map = 0%,  reduce = 0%
> > 2016-01-05 10:54:27,236 Stage-1 map = 100%,  reduce = 0%
> > Ended Job = job_1449731904014_1636 with errors
> > Error during job, obtaining debugging information...
> > Examining task ID: task_1449731904014_1636_m

RE: cut size for hbase region

2016-01-04 Thread Zhang, Zhong
Hongbin,

It's 1.1. Shaofeng pointed that  it's a bug in the configuration in 1.1.

Thanks,
Zhong

-Original Message-
From: hongbin ma [mailto:mahong...@apache.org] 
Sent: Monday, January 04, 2016 9:01 PM
To: dev@kylin.apache.org
Subject: Re: cut size for hbase region

btw, what is the version you're using? in theory region count of a single 
segment will not exceed 500 by default, we want to check that

On Tue, Jan 5, 2016 at 9:40 AM, hongbin ma  wrote:

> the cutting is based on estimation. due to hbase compression and 
> encoding, the estimation might be not very accurate. one recent ticket 
> on this is
> https://issues.apache.org/jira/browse/KYLIN-1237
>
> On Tue, Jan 5, 2016 at 12:30 AM, Zhang, Zhong 
> wrote:
>
>> Hi All,
>>
>> Happy new year!
>>
>> Kylin provides three options for cut size. Please see the following:
>>
>> # The cut size for hbase region, in GB.
>> # E.g, for cube whose capacity be marked as "SMALL", split region per 
>> 10GB by default
>> kylin.hbase.region.cut.small=10
>> kylin.hbase.region.cut.medium=20
>> kylin.hbase.region.cut.large=100
>>
>> I choose cube size as small to build the cube and the following is 
>> one of the HTable I got.
>> HTable: KYLIN_O03ZWB4DK9
>>
>>   *   Region Count: 979
>>   *   Size: 5.75 TB
>>   *   Start Time: 2011-12-31 00:00:00
>>   *   End Time: 2014-05-01 01:00:00
>> So the size of the HTable is 5.75TB and there are 979 regions in total?
>>
>> Let's do a little bit math. 979*10GB (since split region per 10GB 
>> when cube size is marked as small) definitely does not equal 5.75TB. 
>> Do I understand correctly?
>>
>> Best regards
>> Zhong
>>
>>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>



--
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone


Re: org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread hongbin ma
agree with Sai

On Tue, Jan 5, 2016 at 12:24 PM, Kiriti Sai 
wrote:

> Hi,
> This error is caused because there is no Snappy compression codec available
> in your setup and Kylin expects it by default.
> As a work around, you can disable the use of snappy in the configuration
> files of Kylin.
> > Comment the compression.codec line in kylin.properties
> > comment the properties in the kylin_job_conf.xml which are related to
> compression. I guess there are around 4 properties to be commented.
>
> This was the work around I used for a while but its recommended to use
> compression techniques to minimize the memory shuffling between reducers.
>
> Thank you.
> Sai Kiriti B
> On Jan 5, 2016 12:31 PM, "和风" <363938...@qq.com> wrote:
>
> > hi:
> >   execution "bulid" cube, jobs exception :
> > org.apache.hadoop.hive.ql.metadata.HiveException:
> > org.apache.hadoop.hive.ql.metadata.HiveException
> >
> >
> > logs:
> >
> >
> > OS command error exit with 2 -- hive  -e "USE default;
> > DROP TABLE IF EXISTS
> >
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db;
> >
> >
> > CREATE EXTERNAL TABLE IF NOT EXISTS
> >
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> > (
> > DEFAULT_KYLIN_CAL_DT_AGE_FOR_QTR_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_MONTH_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_DT_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_RTL_MONTH_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_CS_WEEK_ID smallint
> > ,DEFAULT_KYLIN_CAL_DT_YEAR_ID string
> > )
> > ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
> > STORED AS SEQUENCEFILE
> > LOCATION
> >
> '/kylin/kylin_metadata/kylin-d22e7c10-032a-4d22-a802-3b74937e86db/kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db';
> >
> >
> > SET mapreduce.job.split.metainfo.maxsize=-1;
> > SET mapred.compress.map.output=true;
> > SET
> >
> mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> > SET mapred.output.compress=true;
> > SET
> >
> mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> > SET mapred.output.compression.type=BLOCK;
> > SET mapreduce.job.max.split.locations=2000;
> > SET dfs.replication=2;
> > SET hive.merge.mapfiles=true;
> > SET hive.merge.mapredfiles=true;
> > SET hive.merge.size.per.task=268435456;
> > SET hive.support.concurrency=false;
> > SET hive.exec.compress.output=true;
> > SET hive.auto.convert.join.noconditionaltask = true;
> > SET hive.auto.convert.join.noconditionaltask.size = 3;
> > INSERT OVERWRITE TABLE
> >
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> > SELECT
> > KYLIN_CAL_DT.AGE_FOR_QTR_ID
> > ,KYLIN_CAL_DT.AGE_FOR_MONTH_ID
> > ,KYLIN_CAL_DT.AGE_FOR_DT_ID
> > ,KYLIN_CAL_DT.AGE_FOR_RTL_MONTH_ID
> > ,KYLIN_CAL_DT.AGE_FOR_CS_WEEK_ID
> > ,KYLIN_CAL_DT.YEAR_ID
> > FROM DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
> > WHERE (KYLIN_CAL_DT.CAL_DT >= '2013-12-29' AND KYLIN_CAL_DT.CAL_DT <
> > '2016-01-12')
> > ;
> >
> >
> > "
> >
> >
> > Logging initialized using configuration in
> > jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> >
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> [jar:file:/usr/local/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> > OK
> > Time taken: 0.936 seconds
> > OK
> > Time taken: 0.112 seconds
> > OK
> > Time taken: 0.438 seconds
> > Query ID = root_20160105105405_88149f4a-a970-47d0-ba32-9a21ee5afde3
> > Total jobs = 3
> > Launching Job 1 out of 3
> > Number of reduce tasks is set to 0 since there's no reduce operator
> > Starting Job = job_1449731904014_1636, Tracking URL =
> > http://cloud001:8088/proxy/application_1449731904014_1636/
> > Kill Command = /usr/local/hadoop/bin/hadoop job  -kill
> > job_1449731904014_1636
> > Hadoop job information for Stage-1: number of mappers: 1; number of
> > reducers: 0
> > 2016-01-05 10:54:26,177 Stage-1 map = 0%,  reduce = 0%
> > 2016-01-05 10:54:27,236 Stage-1 map = 100%,  reduce = 0%
> > Ended Job = job_1449731904014_1636 with errors
> > Error during job, obtaining debugging information...
> > Examining task ID: task_1449731904014_1636_m_00 (and more) from job
> > job_1449731904014_1636
> >
> >
> > Task with the most failures(1):
> > -
> > Task ID:
> >   task_1449731904014_1636_m_00
> >
> >
> > URL:
> >
> >
> http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1449731904014_1636&tipid=task_1449731904014_1636_m_00
> > -
> > Diagnostic Messages for this Task:
> > java.lang.RuntimeException:

Re: org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread Kiriti Sai
Hi,
This error is caused because there is no Snappy compression codec available
in your setup and Kylin expects it by default.
As a work around, you can disable the use of snappy in the configuration
files of Kylin.
> Comment the compression.codec line in kylin.properties
> comment the properties in the kylin_job_conf.xml which are related to
compression. I guess there are around 4 properties to be commented.

This was the work around I used for a while but its recommended to use
compression techniques to minimize the memory shuffling between reducers.

Thank you.
Sai Kiriti B
On Jan 5, 2016 12:31 PM, "和风" <363938...@qq.com> wrote:

> hi:
>   execution "bulid" cube, jobs exception :
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException
>
>
> logs:
>
>
> OS command error exit with 2 -- hive  -e "USE default;
> DROP TABLE IF EXISTS
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db;
>
>
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> (
> DEFAULT_KYLIN_CAL_DT_AGE_FOR_QTR_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_MONTH_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_DT_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_RTL_MONTH_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_AGE_FOR_CS_WEEK_ID smallint
> ,DEFAULT_KYLIN_CAL_DT_YEAR_ID string
> )
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
> STORED AS SEQUENCEFILE
> LOCATION
> '/kylin/kylin_metadata/kylin-d22e7c10-032a-4d22-a802-3b74937e86db/kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db';
>
>
> SET mapreduce.job.split.metainfo.maxsize=-1;
> SET mapred.compress.map.output=true;
> SET
> mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET mapred.output.compress=true;
> SET
> mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET mapred.output.compression.type=BLOCK;
> SET mapreduce.job.max.split.locations=2000;
> SET dfs.replication=2;
> SET hive.merge.mapfiles=true;
> SET hive.merge.mapredfiles=true;
> SET hive.merge.size.per.task=268435456;
> SET hive.support.concurrency=false;
> SET hive.exec.compress.output=true;
> SET hive.auto.convert.join.noconditionaltask = true;
> SET hive.auto.convert.join.noconditionaltask.size = 3;
> INSERT OVERWRITE TABLE
> kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
> SELECT
> KYLIN_CAL_DT.AGE_FOR_QTR_ID
> ,KYLIN_CAL_DT.AGE_FOR_MONTH_ID
> ,KYLIN_CAL_DT.AGE_FOR_DT_ID
> ,KYLIN_CAL_DT.AGE_FOR_RTL_MONTH_ID
> ,KYLIN_CAL_DT.AGE_FOR_CS_WEEK_ID
> ,KYLIN_CAL_DT.YEAR_ID
> FROM DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
> WHERE (KYLIN_CAL_DT.CAL_DT >= '2013-12-29' AND KYLIN_CAL_DT.CAL_DT <
> '2016-01-12')
> ;
>
>
> "
>
>
> Logging initialized using configuration in
> jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> OK
> Time taken: 0.936 seconds
> OK
> Time taken: 0.112 seconds
> OK
> Time taken: 0.438 seconds
> Query ID = root_20160105105405_88149f4a-a970-47d0-ba32-9a21ee5afde3
> Total jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1449731904014_1636, Tracking URL =
> http://cloud001:8088/proxy/application_1449731904014_1636/
> Kill Command = /usr/local/hadoop/bin/hadoop job  -kill
> job_1449731904014_1636
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> 2016-01-05 10:54:26,177 Stage-1 map = 0%,  reduce = 0%
> 2016-01-05 10:54:27,236 Stage-1 map = 100%,  reduce = 0%
> Ended Job = job_1449731904014_1636 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1449731904014_1636_m_00 (and more) from job
> job_1449731904014_1636
>
>
> Task with the most failures(1):
> -
> Task ID:
>   task_1449731904014_1636_m_00
>
>
> URL:
>
> http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1449731904014_1636&tipid=task_1449731904014_1636_m_00
> -
> Diagnostic Messages for this Task:
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> {"cal_dt":"2013-12-31","year_beg_dt":"2013-01-01","qtr_beg_dt":"2013-10-01","month_beg_dt":"2013-12-01","week_beg_dt":"2013-12-29","age_for_year_id":0,"age_for_qtr_id":0,"age_for_month_id":1,"age_for_week_id":5,"age_for_dt_id":34,"age_for_rt

[jira] [Created] (KYLIN-1286) Clean up license issues on 2.0 branch

2016-01-04 Thread liyang (JIRA)
liyang created KYLIN-1286:
-

 Summary: Clean up license issues on 2.0 branch
 Key: KYLIN-1286
 URL: https://issues.apache.org/jira/browse/KYLIN-1286
 Project: Kylin
  Issue Type: Sub-task
Reporter: liyang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (KYLIN-1285) Release 2.0-rc

2016-01-04 Thread liyang (JIRA)
liyang created KYLIN-1285:
-

 Summary: Release 2.0-rc
 Key: KYLIN-1285
 URL: https://issues.apache.org/jira/browse/KYLIN-1285
 Project: Kylin
  Issue Type: Task
Reporter: liyang
Assignee: liyang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


org.apache.hadoop.hive.ql.metadata.HiveException

2016-01-04 Thread ????
hi:
  execution "bulid" cube, jobs exception : 
org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.hadoop.hive.ql.metadata.HiveException


logs:


OS command error exit with 2 -- hive  -e "USE default;
DROP TABLE IF EXISTS 
kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db;


CREATE EXTERNAL TABLE IF NOT EXISTS 
kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
(
DEFAULT_KYLIN_CAL_DT_AGE_FOR_QTR_ID smallint
,DEFAULT_KYLIN_CAL_DT_AGE_FOR_MONTH_ID smallint
,DEFAULT_KYLIN_CAL_DT_AGE_FOR_DT_ID smallint
,DEFAULT_KYLIN_CAL_DT_AGE_FOR_RTL_MONTH_ID smallint
,DEFAULT_KYLIN_CAL_DT_AGE_FOR_CS_WEEK_ID smallint
,DEFAULT_KYLIN_CAL_DT_YEAR_ID string
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
STORED AS SEQUENCEFILE
LOCATION 
'/kylin/kylin_metadata/kylin-d22e7c10-032a-4d22-a802-3b74937e86db/kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db';


SET mapreduce.job.split.metainfo.maxsize=-1;
SET mapred.compress.map.output=true;
SET 
mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapred.output.compress=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapred.output.compression.type=BLOCK;
SET mapreduce.job.max.split.locations=2000;
SET dfs.replication=2;
SET hive.merge.mapfiles=true;
SET hive.merge.mapredfiles=true;
SET hive.merge.size.per.task=268435456;
SET hive.support.concurrency=false;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join.noconditionaltask = true;
SET hive.auto.convert.join.noconditionaltask.size = 3;
INSERT OVERWRITE TABLE 
kylin_intermediate_learn_kylin_two_2013122900_2016011200_d22e7c10_032a_4d22_a802_3b74937e86db
 SELECT
KYLIN_CAL_DT.AGE_FOR_QTR_ID
,KYLIN_CAL_DT.AGE_FOR_MONTH_ID
,KYLIN_CAL_DT.AGE_FOR_DT_ID
,KYLIN_CAL_DT.AGE_FOR_RTL_MONTH_ID
,KYLIN_CAL_DT.AGE_FOR_CS_WEEK_ID
,KYLIN_CAL_DT.YEAR_ID
FROM DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT 
WHERE (KYLIN_CAL_DT.CAL_DT >= '2013-12-29' AND KYLIN_CAL_DT.CAL_DT < 
'2016-01-12')
;


"


Logging initialized using configuration in 
jar:file:/usr/local/hive/lib/hive-common-1.2.1.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/local/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
OK
Time taken: 0.936 seconds
OK
Time taken: 0.112 seconds
OK
Time taken: 0.438 seconds
Query ID = root_20160105105405_88149f4a-a970-47d0-ba32-9a21ee5afde3
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1449731904014_1636, Tracking URL = 
http://cloud001:8088/proxy/application_1449731904014_1636/
Kill Command = /usr/local/hadoop/bin/hadoop job  -kill job_1449731904014_1636
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2016-01-05 10:54:26,177 Stage-1 map = 0%,  reduce = 0%
2016-01-05 10:54:27,236 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_1449731904014_1636 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1449731904014_1636_m_00 (and more) from job 
job_1449731904014_1636


Task with the most failures(1): 
-
Task ID:
  task_1449731904014_1636_m_00


URL:
  
http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1449731904014_1636&tipid=task_1449731904014_1636_m_00
-
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing row 
{"cal_dt":"2013-12-31","year_beg_dt":"2013-01-01","qtr_beg_dt":"2013-10-01","month_beg_dt":"2013-12-01","week_beg_dt":"2013-12-29","age_for_year_id":0,"age_for_qtr_id":0,"age_for_month_id":1,"age_for_week_id":5,"age_for_dt_id":34,"age_for_rtl_year_id":1,"age_for_rtl_qtr_id":1,"age_for_rtl_month_id":1,"age_for_rtl_week_id":5,"age_for_cs_week_id":5,"day_of_cal_id":41638,"day_of_year_id":365,"day_of_qtr_id":92,"day_of_month_id":31,"day_of_week_id":3,"week_of_year_id":53,"week_of_cal_id":5948,"month_of_qtr_id":3,"month_of_year_id":12,"month_of_cal_id":1368,"qtr_of_year_id":4,"qtr_of_cal_id":456,"year_of_cal_id":114,"year_end_dt":"2013-12-31","qtr_end_dt":"2013-12-31","month_end_dt":"2013-12-31","week_end_dt":"2013-12-31","cal_dt_name":"31-Dec-2013","cal_dt_desc":"Dec
 31st 2013","cal_dt_short_name":"Tue 
12-31-13","ytd_yn_id":0,"qtd_yn_id":0,"mtd_yn_id":0,"wtd_yn_id":0,"season_beg_dt":"2013-12-21","day_in_year_count":365,"day_in_qtr_count":92,"day_in_month_count":31,"day_in_week_count":3,"rtl_year_beg_dt":"2013-12-29","rtl_qtr_beg_dt":"2013-12-29","rtl_month_be

Re: cut size for hbase region

2016-01-04 Thread hongbin ma
btw, what is the version you're using? in theory region count of a single
segment will not exceed 500 by default, we want to check that

On Tue, Jan 5, 2016 at 9:40 AM, hongbin ma  wrote:

> the cutting is based on estimation. due to hbase compression and encoding,
> the estimation might be not very accurate. one recent ticket on this is
> https://issues.apache.org/jira/browse/KYLIN-1237
>
> On Tue, Jan 5, 2016 at 12:30 AM, Zhang, Zhong 
> wrote:
>
>> Hi All,
>>
>> Happy new year!
>>
>> Kylin provides three options for cut size. Please see the following:
>>
>> # The cut size for hbase region, in GB.
>> # E.g, for cube whose capacity be marked as "SMALL", split region per
>> 10GB by default
>> kylin.hbase.region.cut.small=10
>> kylin.hbase.region.cut.medium=20
>> kylin.hbase.region.cut.large=100
>>
>> I choose cube size as small to build the cube and the following is one of
>> the HTable I got.
>> HTable: KYLIN_O03ZWB4DK9
>>
>>   *   Region Count: 979
>>   *   Size: 5.75 TB
>>   *   Start Time: 2011-12-31 00:00:00
>>   *   End Time: 2014-05-01 01:00:00
>> So the size of the HTable is 5.75TB and there are 979 regions in total?
>>
>> Let's do a little bit math. 979*10GB (since split region per 10GB when
>> cube size is
>> marked as small) definitely does not equal 5.75TB. Do I understand
>> correctly?
>>
>> Best regards
>> Zhong
>>
>>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>



-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone


Re: cut size for hbase region

2016-01-04 Thread hongbin ma
the cutting is based on estimation. due to hbase compression and encoding,
the estimation might be not very accurate. one recent ticket on this is
https://issues.apache.org/jira/browse/KYLIN-1237

On Tue, Jan 5, 2016 at 12:30 AM, Zhang, Zhong  wrote:

> Hi All,
>
> Happy new year!
>
> Kylin provides three options for cut size. Please see the following:
>
> # The cut size for hbase region, in GB.
> # E.g, for cube whose capacity be marked as "SMALL", split region per 10GB
> by default
> kylin.hbase.region.cut.small=10
> kylin.hbase.region.cut.medium=20
> kylin.hbase.region.cut.large=100
>
> I choose cube size as small to build the cube and the following is one of
> the HTable I got.
> HTable: KYLIN_O03ZWB4DK9
>
>   *   Region Count: 979
>   *   Size: 5.75 TB
>   *   Start Time: 2011-12-31 00:00:00
>   *   End Time: 2014-05-01 01:00:00
> So the size of the HTable is 5.75TB and there are 979 regions in total?
>
> Let's do a little bit math. 979*10GB (since split region per 10GB when
> cube size is
> marked as small) definitely does not equal 5.75TB. Do I understand
> correctly?
>
> Best regards
> Zhong
>
>


-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone


cut size for hbase region

2016-01-04 Thread Zhang, Zhong
Hi All,

Happy new year!

Kylin provides three options for cut size. Please see the following:

# The cut size for hbase region, in GB.
# E.g, for cube whose capacity be marked as "SMALL", split region per 10GB by 
default
kylin.hbase.region.cut.small=10
kylin.hbase.region.cut.medium=20
kylin.hbase.region.cut.large=100

I choose cube size as small to build the cube and the following is one of the 
HTable I got.
HTable: KYLIN_O03ZWB4DK9

  *   Region Count: 979
  *   Size: 5.75 TB
  *   Start Time: 2011-12-31 00:00:00
  *   End Time: 2014-05-01 01:00:00
So the size of the HTable is 5.75TB and there are 979 regions in total?

Let's do a little bit math. 979*10GB (since split region per 10GB when cube 
size is
marked as small) definitely does not equal 5.75TB. Do I understand correctly?

Best regards
Zhong



Re: aboutthe; parameter 'acceptPartial'

2016-01-04 Thread yu feng
In my opinion, acceptPartial means whether return partial result if your
query result has more rows than limit value , and it will always set to
true no matter query from web UI or jdbc.
However, if you query from web UI, you can set the limit (default
is 5), if you query from jdbc, the default limit will set to 100 if
your sql do not contain 'limit'.

2016-01-04 18:15 GMT+08:00 wangsh...@sinoaudit.cn :

> Hi all:
>  Can anybody tell me what the query parameter 'acceptPartial' means? and I
> wonder how I can setup this parameter in jdbc.
>
>
>
> wangsh...@sinoaudit.cn
>


[jira] [Created] (KYLIN-1284) Restful API Get hive SQL of the cube "cubes/{cubeName}/segs/{segmentName}/sql" returns xml

2016-01-04 Thread Lola Liu (JIRA)
Lola Liu created KYLIN-1284:
---

 Summary: Restful API Get hive SQL of the cube 
"cubes/{cubeName}/segs/{segmentName}/sql" returns xml
 Key: KYLIN-1284
 URL: https://issues.apache.org/jira/browse/KYLIN-1284
 Project: Kylin
  Issue Type: Bug
  Components: REST Service
Affects Versions: v2.0
Reporter: Lola Liu
Assignee: Zhong,Jason
Priority: Minor


STEPS:
Send get request: 
http://kylin-qa2-host:port/kylin/cubes/airline_test_vic/segs/1970010100_2922789940817071255/sql
(Cube name and segment name are from /cubes API response)

RESULT:
Response:











Kylin
























Error Message

{{text}}

Cube Schema

{{schema}}

{{text}}

Error Message

{{text}}

Model Schema

{{schema}}

{{text}}

Error Message

{{text}}

Streaming Schema

{{streamingSchema}}

Kafka Schema

{{kfkSchema}}

[jira] [Created] (KYLIN-1283) Replace GTScanRequest's SerDer form Kryo to manual

2016-01-04 Thread hongbin ma (JIRA)
hongbin ma created KYLIN-1283:
-

 Summary: Replace GTScanRequest's SerDer form Kryo to manual 
 Key: KYLIN-1283
 URL: https://issues.apache.org/jira/browse/KYLIN-1283
 Project: Kylin
  Issue Type: Improvement
Reporter: hongbin ma
Assignee: hongbin ma


Kryo greatly simplifies bject SerDer at the cost of performance. When there're 
tens of segments, such cost accumulates too big to accept. Going to serialize 
GTScanRequest's with manual serialization.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


about the parameter 'acceptPartial'

2016-01-04 Thread wangsh...@sinoaudit.cn
Hi all:
 Can anybody tell me what the query parameter 'acceptPartial' means? and I 
wonder how I can setup this parameter in jdbc.



wangsh...@sinoaudit.cn


[jira] [Created] (KYLIN-1282) Comparison filter on Date/Time column not work for query

2016-01-04 Thread Dong Li (JIRA)
Dong Li created KYLIN-1282:
--

 Summary: Comparison filter on Date/Time column not work for query
 Key: KYLIN-1282
 URL: https://issues.apache.org/jira/browse/KYLIN-1282
 Project: Kylin
  Issue Type: Bug
  Components: Query Engine
Affects Versions: v2.0
Reporter: Dong Li
Assignee: Dong Li


Suppose test_table has column A whose type is 'date', following query will get 
error:

select * from test_table where A > '2012-01-01'

But this query works on 1.x



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)