Re: Welcome new Apache Kylin committer: Luwei Chen

[Chen, Luwei]

Thanks everyone! It is really my honor to become Apache Kylin committer.

I am now a master student in Fudan University major in Software
Engineering, having been working with eBay Kylin team as an intern for 9
months. During my study in Fudan, big data has become the most popular
keyword mentioned by almost every lecturer. And I have taken several
courses about data science. But what I learnt were just theories. Kylin
gives me the chance to participate in a real project, which is really a
valuable experience.

I will continue to work for Kylin and hopefully, my effort can make
contribution to the project.


Luwei


On 12/30/15, 9:47 PM, "Luke Han"  wrote:

>I am very pleased to announce that the Project Management Committee
>(PMC) of Apache Kylin has asked Luwei Chen to become Apache Kylin
>committer
>,
>and she has already accepted.
>
>Luwei has already made many contribution to Kylin community, about
>website,
>documentation, UI and others as well.
>
>Welcome Luwei, our first female committer:)
>Please share with us a little about yourself,
>
>Luke
>
>On behalf of the Apache Kylin PPMC

[jira] [Created] (KYLIN-1291) Make sure thread resources are freed when a query aborts/timeouts

[hongbin ma (JIRA)]

hongbin ma created KYLIN-1291:
-

 Summary: Make sure thread resources are freed when a query 
aborts/timeouts
 Key: KYLIN-1291
 URL: https://issues.apache.org/jira/browse/KYLIN-1291
 Project: Kylin
  Issue Type: Improvement
Reporter: hongbin ma
Assignee: hongbin ma


when user cancels a query or the query is timed out, kylin server should make 
sure the local thread pool resources assigned for the query is freed. 
Meanwhile, it's best also to free all the scanner threads in region servers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Welcome new Apache Kylin committer: Luwei Chen

[Yerui Sun]

Welcome Luwei!

> 在 2016年1月6日，18:05，Chen, Luwei  写道：
> 
> Thanks everyone! It is really my honor to become Apache Kylin committer.
> 
> I am now a master student in Fudan University major in Software
> Engineering, having been working with eBay Kylin team as an intern for 9
> months. During my study in Fudan, big data has become the most popular
> keyword mentioned by almost every lecturer. And I have taken several
> courses about data science. But what I learnt were just theories. Kylin
> gives me the chance to participate in a real project, which is really a
> valuable experience.
> 
> I will continue to work for Kylin and hopefully, my effort can make
> contribution to the project.
> 
> 
> Luwei
> 
> 
> On 12/30/15, 9:47 PM, "Luke Han"  wrote:
> 
>> I am very pleased to announce that the Project Management Committee
>> (PMC) of Apache Kylin has asked Luwei Chen to become Apache Kylin
>> committer
>> ,
>> and she has already accepted.
>> 
>> Luwei has already made many contribution to Kylin community, about
>> website,
>> documentation, UI and others as well.
>> 
>> Welcome Luwei, our first female committer:)
>> Please share with us a little about yourself,
>> 
>> Luke
>> 
>> On behalf of the Apache Kylin PPMC
> 

Re: Error when scan from lower key

[yu feng]

This means your one of hbase scanner execute failed, you can check logs in
regionserver for more detailed  infomation.

2016-01-07 3:46 GMT+08:00 sdangi :

> Hi Kylin Team --
>
>
>
> I'm hitting the below error when executing query with the time dimension in
> the where clause against  a 35GB cube (No issues running otherwise). Same
> exact query works fine against Hive with the WHERE clause.
>
> Any ideas?
>
> Thanks,
> Regards,
>
> Error when scan from lower key  � ;8 to upper key  � on
> table KYLIN_PWNXRLXEZ7. while executing SQL: "SELECT
> CUSTOMER_DIM_T.PRN_CST_NM ,CURRENCY_DIM_T.SHRT_NM ,BRANCH_DIM_T.BR_NM
> ,COUNTRY_DIM_T.CTY_NM ,SUM(AML_TXN_FCT_CUB_T.USD_TXN_AMT)
> ,SUM(AML_TXN_FCT_CUB_T.AC_CCY_TXN_AMT) FROM
> DL_FINSERV_DEMO.AML_TXN_FCT_CUB_T as AML_TXN_FCT_CUB_T INNER JOIN
> DL_FINSERV_DEMO.CUSTOMER_DIM_T as CUSTOMER_DIM_T ON
> AML_TXN_FCT_CUB_T.FIRM_CST_KEY = CUSTOMER_DIM_T.KEY INNER JOIN
> DL_FINSERV_DEMO.BRANCH_DIM_T as BRANCH_DIM_T ON
> AML_TXN_FCT_CUB_T.TXN_BR_KEY
> = BRANCH_DIM_T.BR_KEY INNER JOIN DL_FINSERV_DEMO.COUNTRY_DIM_T as
> COUNTRY_DIM_T ON AML_TXN_FCT_CUB_T.RMTR_CTY_KEY = COUNTRY_DIM_T.CTY_KEY
> INNER JOIN DL_FINSERV_DEMO.CURRENCY_DIM_T as CURRENCY_DIM_T ON
> AML_TXN_FCT_CUB_T.RMTR_CCY_KEY = CURRENCY_DIM_T.KEY INNER JOIN
> DL_FINSERV_DEMO.DATE_DIM_T as DATE_DIM_T ON
> AML_TXN_FCT_CUB_T.TXN_BOOK_DT_KEY = DATE_DIM_T.DT_KEY
>
> WHERE DATE_DIM_T.DT_KEY >= date'2015-04-01'
>
> GROUP BY CUSTOMER_DIM_T.PRN_CST_NM ,CURRENCY_DIM_T.SHRT_NM
> ,BRANCH_DIM_T.BR_NM ,COUNTRY_DIM_T.CTY_NM LIMIT 5000"
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/Error-when-scan-from-lower-key-tp3083.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>

Re: suggest to revise kylin log preamble format

[Li Yang]

Still need month and date for copy and paste lines here and there. Others
are great.

On Thu, Jan 7, 2016 at 10:22 AM, hongbin ma  wrote:

> the reason to remove date info is that our kylin-server-log4j properties
> uses DailyRollingFileAppender, putting date info in such logs is not
> necessary
>
> On Thu, Jan 7, 2016 at 10:19 AM, hongbin ma  wrote:
>
> > As i'm looking at kylin.log i found lots of noise informations: log
> > preamble accounts for nearly half of the space.
> >
> > After a quick study of HBase's default log4j settings, and follow some
> > advice from
> >
> http://www.javacodegeeks.com/2011/01/10-tips-proper-application-logging.html
> .
> > I suggest change our preamble from:
> >
> > [pool-6-thread-1]:[2016-01-06
> >
> 19:03:43,722][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:106)]
> >
> > to
> >
> > 10:05:23,856 INFO  [localhost-startStop-1 CubeManager:837]:
> >
> > which is a much neater version.
> >
> > please comment on this.
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

?????? java.io.IOException: NoSuchObjectException(message:default.kylin_intermediate_learn_kylin

[????]

my show tables ,table is exist;my load tables not success;
logs:
hive> show tables;
OK
kylin_cal_dt
kylin_category_groupings
kylin_intermediate_learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_c8d9323ae277
kylin_sales

 
load tables log:


java.io.IOException: NoSuchObjectException(message:DEFAULT.KYLIN_SALES table 
not found)
at 
org.apache.kylin.metadata.tool.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:97)
at 
org.apache.kylin.metadata.tool.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:75)
at 
org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:514)
at 
org.apache.kylin.rest.service.CubeService$$FastClassByCGLIB$$17a07c0e.invoke()
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at 
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at 
org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at 
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
at 
org.apache.kylin.rest.service.CubeService$$EnhancerByCGLIB$$d50a1b8d.reloadHiveTable()
at 
org.apache.kylin.rest.controller.TableController.loadHiveTable(TableController.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213)
at 
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
at 
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
at 
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at 
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at 
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at 
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at 
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at 
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at 
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at 
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at 

?????? java.io.IOException: NoSuchObjectException(message:default.kylin_intermediate_learn_kylin

[????]

my env hadoop 2.7.1 ,kylin1.2,hive 1.2.1,hbase 0.98.


my hive config :


   
javax.jdo.option.ConnectionURL
jdbc:mysql://10.24.248.196:3306/hive?characterEncoding=UTF-8
JDBC connect string for a JDBC metastore
  
  
   
javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
Driver class name for a JDBC metastore
   
  
   
javax.jdo.option.ConnectionUserName
root
Username to use against metastore database
  


  
javax.jdo.option.ConnectionPassword
root
password to use against metastore database
  
  

  datanucleus.transactionIsolation
  repeatable-read
  


  datanucleus.valuegeneration.transactionIsolation
  repeatable-read
 



  hive.aux.jars.path
  
file:///usr/local/hive/lib/json-serde-1.3.6-jar-with-dependencies.jar,file:///usr/local/hive/lib/gson-2.2.4.jar,file:///usr/local/hive/lib/data-hive-udf.jar
  The location of the plugin jars that contain implementations 
of user defined functions and serdes.
 





--  --
??: "yu feng";;
: 2016??1??7??(??) 11:25
??: "dev"; 

: Re: java.io.IOException: 
NoSuchObjectException(message:default.kylin_intermediate_learn_kylin



I have encountered this problem, it is more likely because your current
hive metastore config is error, could you tell some more detailed
infomation about your env.

2016-01-07 10:23 GMT+08:00  <363938...@qq.com>:

> hi:
>   i build cube have a error.
> logs:
> java.io.IOException:
> NoSuchObjectException(message:default.kylin_intermediate_learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_c8d9323ae277
> table not found)
> at
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
> at
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
> at
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
> at
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> NoSuchObjectException(message:default.kylin_intermediate_learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_c8d9323ae277
> table not found)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1808)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1778)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy47.get_table(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1208)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152)
> at com.sun.proxy.$Proxy48.getTable(Unknown Source)
> at
> org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:180)
> at
> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:105)
> at
> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
> at
> 

Re: Error when scan from lower key

[hongbin ma]

yes, and check the kylin.log before or after, maybe there's some other
clues.

On Thu, Jan 7, 2016 at 9:47 AM, yu feng  wrote:

> This means your one of hbase scanner execute failed, you can check logs in
> regionserver for more detailed  infomation.
>
> 2016-01-07 3:46 GMT+08:00 sdangi :
>
> > Hi Kylin Team --
> >
> >
> >
> > I'm hitting the below error when executing query with the time dimension
> in
> > the where clause against  a 35GB cube (No issues running otherwise). Same
> > exact query works fine against Hive with the WHERE clause.
> >
> > Any ideas?
> >
> > Thanks,
> > Regards,
> >
> > Error when scan from lower key  � ;8 to upper key  � on
> > table KYLIN_PWNXRLXEZ7. while executing SQL: "SELECT
> > CUSTOMER_DIM_T.PRN_CST_NM ,CURRENCY_DIM_T.SHRT_NM ,BRANCH_DIM_T.BR_NM
> > ,COUNTRY_DIM_T.CTY_NM ,SUM(AML_TXN_FCT_CUB_T.USD_TXN_AMT)
> > ,SUM(AML_TXN_FCT_CUB_T.AC_CCY_TXN_AMT) FROM
> > DL_FINSERV_DEMO.AML_TXN_FCT_CUB_T as AML_TXN_FCT_CUB_T INNER JOIN
> > DL_FINSERV_DEMO.CUSTOMER_DIM_T as CUSTOMER_DIM_T ON
> > AML_TXN_FCT_CUB_T.FIRM_CST_KEY = CUSTOMER_DIM_T.KEY INNER JOIN
> > DL_FINSERV_DEMO.BRANCH_DIM_T as BRANCH_DIM_T ON
> > AML_TXN_FCT_CUB_T.TXN_BR_KEY
> > = BRANCH_DIM_T.BR_KEY INNER JOIN DL_FINSERV_DEMO.COUNTRY_DIM_T as
> > COUNTRY_DIM_T ON AML_TXN_FCT_CUB_T.RMTR_CTY_KEY = COUNTRY_DIM_T.CTY_KEY
> > INNER JOIN DL_FINSERV_DEMO.CURRENCY_DIM_T as CURRENCY_DIM_T ON
> > AML_TXN_FCT_CUB_T.RMTR_CCY_KEY = CURRENCY_DIM_T.KEY INNER JOIN
> > DL_FINSERV_DEMO.DATE_DIM_T as DATE_DIM_T ON
> > AML_TXN_FCT_CUB_T.TXN_BOOK_DT_KEY = DATE_DIM_T.DT_KEY
> >
> > WHERE DATE_DIM_T.DT_KEY >= date'2015-04-01'
> >
> > GROUP BY CUSTOMER_DIM_T.PRN_CST_NM ,CURRENCY_DIM_T.SHRT_NM
> > ,BRANCH_DIM_T.BR_NM ,COUNTRY_DIM_T.CTY_NM LIMIT 5000"
> >
> > --
> > View this message in context:
> >
> http://apache-kylin.74782.x6.nabble.com/Error-when-scan-from-lower-key-tp3083.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
>



-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

Re: A problem,maybe a bug, when querying in kylin using not in

[Julian Hyde]

Yes, it looks as if you have a data correctness issue. If I were you I’d look 
at the plan and see if it makes sense.

Your second case, the NullPointerException issue, looks a lot like 
https://issues.apache.org/jira/browse/CALCITE-980 
.

Julian

> On Jan 6, 2016, at 6:28 PM, 曾经 <1032195...@qq.com> wrote:
> 
> Thank you very much for answering. 
> In my result ,not only null in H_CODE is skipped, but also the lines where 
> H_CODE is 'B' and 'O' are all skipped. I think this. 
> I think this is not only a prolem of null.

Re: Re: about apache kylin using problem!

[wenye...@163.com]

Thank you for your reply to my question, but my question is my original data 
has increased, so the amount of data cube how automatically to increase. Also 
you have no on kylin theory that the document is to explain the kylin is how it 
works, how the flow of data, I see the official documents are not Kandao


wenye...@163.com
 
From: Jian Zhong
Date: 2016-01-07 11:04
To: dev
Subject: Re: about apache kylin using problem!
2 when the data in the hive table is increased, how to increase the cube
data, or how to refresh
--  for partition cube, before create the cube you should have a date type
partition column in fact table. and select partition column when create
cube.
when you build the cube, you will be informed to choose your 'end
date','start date' will be the 'end date' of of last build. for refresh,
you can refresh your segment on UI cubes page or through Rest API.
 
On Thu, Jan 7, 2016 at 10:52 AM, wenye...@163.com  wrote:
 
> My environment:
> 1.1.1 HBase version, the use of kylin for
> apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin
> I have two questions:
> Kylin cube build 1.apache after the completion of the data stored in the
> HBase, I view the HBase table data, found that the data value can directly
> view the data
> 2 when the data in the hive table is increased, how to increase the cube
> data, or how to refresh
>
>
>
> thanks!
>
>
>
> wenye...@163.com
>

derived dimension

[Zhang, Zhong]

Hi All,

I'm confused by the derived dimension. The following
two sentences are the source that I found online to guide
me use derived dimension. It's kind of unclear to me.

Dimensions on lookup table that can be derived by PK.
-like User ID derives [Name, Age, Gender]
from [1] at page 10

Given a value in DimA, the value of DimB is determined, so we say
dimB can be derived from DimA. When we build a cube that contains
both DimA and DimB, we simple include DimA, and marking DimB
as Derived.
from [2]

Let us use the sample cube "kylin_sales_cube" as the example to
discuss it. There are two derived dimensions: CAL_DT and CATEGORY.
In CAL_DT, which column derives WEEK_BEG_DT?
In CATEGORY, which column derives 
"USER_DEFINED_FIELD1","USER_DEFINED_FIELD3","UPD_DATE","UPD_USER"?

Is derived dimension used only in lookup table?

[1] 
http://www.slideshare.net/YangLi43/design-cube-in-apache-kylin?next_slideshow=5
[2] 
https://mail-archives.apache.org/mod_mbox/incubator-kylin-dev/201507.mbox/%3c82073af5-ae07-4441-bfdf-e8b9d36ff...@163.com%3E

Best regards,
Zhong

Re: java.io.IOException: NoSuchObjectException(message:default.kylin_intermediate_learn_kylin

[yu feng]

I have encountered this problem, it is more likely because your current
hive metastore config is error, could you tell some more detailed
infomation about your env.

2016-01-07 10:23 GMT+08:00 和风 <363938...@qq.com>:

> hi:
>   i build cube have a error.
> logs:
> java.io.IOException:
> NoSuchObjectException(message:default.kylin_intermediate_learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_c8d9323ae277
> table not found)
> at
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
> at
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
> at
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
> at
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> NoSuchObjectException(message:default.kylin_intermediate_learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_c8d9323ae277
> table not found)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1808)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1778)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy47.get_table(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1208)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152)
> at com.sun.proxy.$Proxy48.getTable(Unknown Source)
> at
> org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:180)
> at
> org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:105)
> at
> org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
> at
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
> ... 13 more

Re: java.io.IOException: NoSuchObjectException(message:default.kylin_intermediate_learn_kylin

[Xiaoyu Wang]

Hi,
I think this exception may cause by the "hive.metastore.uris" property 
is not set in hive-site.xml
Kylin use HCatalog to read Hive table. HCatalog will use 
"hive.metastore.uris" property to create HiveMetaStoreClient and get 
table meta.
if not set . HCatalog use local meta server , so it will throw 
NoSuchObjectException exception.


You can config the "hive.metastore.uris" property in hive-site.xml and 
start hive metastore. so HCatalog can connect to it.



在 2016年01月07日 12:52, yu feng 写道:

you can check whether the table "default.kylin_intermediate_
learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_
c8d9323ae277" exist in your hive and do any other hive-site.xml exist in
your classpath. it is strange because you can load hive table(before create
and build cube).

2016-01-07 11:47 GMT+08:00 和风 <363938...@qq.com>:


my env hadoop 2.7.1 ,kylin1.2,hive 1.2.1,hbase 0.98.


my hive config :



 javax.jdo.option.ConnectionURL
 jdbc:mysql://10.24.248.196:3306/hive?characterEncoding=UTF-8

 JDBC connect string for a JDBC metastore
   


 javax.jdo.option.ConnectionDriverName
 com.mysql.jdbc.Driver
 Driver class name for a JDBC metastore
   


 javax.jdo.option.ConnectionUserName
 root
 Username to use against metastore database
   


   
 javax.jdo.option.ConnectionPassword
 root
 password to use against metastore database
   

 
   datanucleus.transactionIsolation
   repeatable-read
 

 
   datanucleus.valuegeneration.transactionIsolation
   repeatable-read
 


 
   hive.aux.jars.path

file:///usr/local/hive/lib/json-serde-1.3.6-jar-with-dependencies.jar,file:///usr/local/hive/lib/gson-2.2.4.jar,file:///usr/local/hive/lib/data-hive-udf.jar
   The location of the plugin jars that contain
implementations of user defined functions and serdes.
 





-- 原始邮件 --
发件人: "yu feng";;
发送时间: 2016年1月7日(星期四) 中午11:25
收件人: "dev";

主题: Re: java.io.IOException:
NoSuchObjectException(message:default.kylin_intermediate_learn_kylin



I have encountered this problem, it is more likely because your current
hive metastore config is error, could you tell some more detailed
infomation about your env.

2016-01-07 10:23 GMT+08:00 和风 <363938...@qq.com>:


hi:
   i build cube have a error.
logs:
java.io.IOException:


NoSuchObjectException(message:default.kylin_intermediate_learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_c8d9323ae277

table not found)
 at


org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)

 at


org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)

 at


org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)

 at


org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)

 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
 at


org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120)

 at


org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)

 at


org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)

 at


org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)

 at


org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)

 at


java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

 at


java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

 at java.lang.Thread.run(Thread.java:745)
Caused by:


NoSuchObjectException(message:default.kylin_intermediate_learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_c8d9323ae277

table not found)
 at


org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1808)

 at


org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1778)

 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at


sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

 at


sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 at java.lang.reflect.Method.invoke(Method.java:606)
 at


org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)

 at com.sun.proxy.$Proxy47.get_table(Unknown Source)
 at


org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1208)

 at 

A problem,maybe a bug, when querying in kylin using not in

[????]

Thank you very much for answering.In my result ,not only null in H_CODE is 
skipped, but also the lines where H_CODE is 'B' and 'O' are all skipped. I 
think this.
I think this is not only a prolem of null.

Re: about apache kylin using problem!

[Jian Zhong]

2 when the data in the hive table is increased, how to increase the cube
data, or how to refresh
--  for partition cube, before create the cube you should have a date type
partition column in fact table. and select partition column when create
cube.
when you build the cube, you will be informed to choose your 'end
date','start date' will be the 'end date' of of last build. for refresh,
you can refresh your segment on UI cubes page or through Rest API.

On Thu, Jan 7, 2016 at 10:52 AM, wenye...@163.com  wrote:

> My environment:
> 1.1.1 HBase version, the use of kylin for
> apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin
> I have two questions:
> Kylin cube build 1.apache after the completion of the data stored in the
> HBase, I view the HBase table data, found that the data value can directly
> view the data
> 2 when the data in the hive table is increased, how to increase the cube
> data, or how to refresh
>
>
>
> thanks!
>
>
>
> wenye...@163.com
>

java.io.IOException: NoSuchObjectException(message:default.kylin_intermediate_learn_kylin

[????]

hi:
  i build cube have a error.
logs:
java.io.IOException: 
NoSuchObjectException(message:default.kylin_intermediate_learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_c8d9323ae277
 table not found)
at 
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
at 
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
at 
org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:101)
at 
org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:77)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at 
org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at 
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at 
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: 
NoSuchObjectException(message:default.kylin_intermediate_learn_kylin_four_2015020100_2015123000_8d26cc4b_e012_4414_a89b_c8d9323ae277
 table not found)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1808)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1778)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy47.get_table(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1208)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152)
at com.sun.proxy.$Proxy48.getTable(Unknown Source)
at org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:180)
at 
org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:105)
at 
org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
at 
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
... 13 more

Re: suggest to revise kylin log preamble format

[hongbin ma]

the reason to remove date info is that our kylin-server-log4j properties
uses DailyRollingFileAppender, putting date info in such logs is not
necessary

On Thu, Jan 7, 2016 at 10:19 AM, hongbin ma  wrote:

> As i'm looking at kylin.log i found lots of noise informations: log
> preamble accounts for nearly half of the space.
>
> After a quick study of HBase's default log4j settings, and follow some
> advice from
> http://www.javacodegeeks.com/2011/01/10-tips-proper-application-logging.html.
> I suggest change our preamble from:
>
> [pool-6-thread-1]:[2016-01-06
> 19:03:43,722][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:106)]
>
> to
>
> 10:05:23,856 INFO  [localhost-startStop-1 CubeManager:837]:
>
> which is a much neater version.
>
> please comment on this.
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>



-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

about apache kylin using problem!

[wenye...@163.com]

My environment:
1.1.1 HBase version, the use of kylin for 
apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin
I have two questions:
Kylin cube build 1.apache after the completion of the data stored in the HBase, 
I view the HBase table data, found that the data value can directly view the 
data
2 when the data in the hive table is increased, how to increase the cube data, 
or how to refresh



thanks!



wenye...@163.com

Error when scan from lower key

[sdangi]

Hi Kylin Team --



I'm hitting the below error when executing query with the time dimension in
the where clause against  a 35GB cube (No issues running otherwise). Same
exact query works fine against Hive with the WHERE clause.

Any ideas?

Thanks,
Regards,

Error when scan from lower key � ;8 to upper key � on
table KYLIN_PWNXRLXEZ7. while executing SQL: "SELECT
CUSTOMER_DIM_T.PRN_CST_NM ,CURRENCY_DIM_T.SHRT_NM ,BRANCH_DIM_T.BR_NM
,COUNTRY_DIM_T.CTY_NM ,SUM(AML_TXN_FCT_CUB_T.USD_TXN_AMT)
,SUM(AML_TXN_FCT_CUB_T.AC_CCY_TXN_AMT) FROM
DL_FINSERV_DEMO.AML_TXN_FCT_CUB_T as AML_TXN_FCT_CUB_T INNER JOIN
DL_FINSERV_DEMO.CUSTOMER_DIM_T as CUSTOMER_DIM_T ON
AML_TXN_FCT_CUB_T.FIRM_CST_KEY = CUSTOMER_DIM_T.KEY INNER JOIN
DL_FINSERV_DEMO.BRANCH_DIM_T as BRANCH_DIM_T ON AML_TXN_FCT_CUB_T.TXN_BR_KEY
= BRANCH_DIM_T.BR_KEY INNER JOIN DL_FINSERV_DEMO.COUNTRY_DIM_T as
COUNTRY_DIM_T ON AML_TXN_FCT_CUB_T.RMTR_CTY_KEY = COUNTRY_DIM_T.CTY_KEY
INNER JOIN DL_FINSERV_DEMO.CURRENCY_DIM_T as CURRENCY_DIM_T ON
AML_TXN_FCT_CUB_T.RMTR_CCY_KEY = CURRENCY_DIM_T.KEY INNER JOIN
DL_FINSERV_DEMO.DATE_DIM_T as DATE_DIM_T ON
AML_TXN_FCT_CUB_T.TXN_BOOK_DT_KEY = DATE_DIM_T.DT_KEY 

WHERE DATE_DIM_T.DT_KEY >= date'2015-04-01' 

GROUP BY CUSTOMER_DIM_T.PRN_CST_NM ,CURRENCY_DIM_T.SHRT_NM
,BRANCH_DIM_T.BR_NM ,COUNTRY_DIM_T.CTY_NM LIMIT 5000"

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Error-when-scan-from-lower-key-tp3083.html
Sent from the Apache Kylin mailing list archive at Nabble.com.