[jira] [Commented] (KYLIN-1122) Kylin support detail data query from fact table
[ https://issues.apache.org/jira/browse/KYLIN-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095517#comment-15095517 ] hongbin ma commented on KYLIN-1122: --- i think this is will be a critical function for kylin, gonna speed up merging it > Kylin support detail data query from fact table > --- > > Key: KYLIN-1122 > URL: https://issues.apache.org/jira/browse/KYLIN-1122 > Project: Kylin > Issue Type: New Feature > Components: Query Engine >Affects Versions: v1.2 >Reporter: Xiaoyu Wang >Assignee: liyang > Fix For: v2.0, v1.3 > > Attachments: > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact(2.x-staging).patch, > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact(update-v2-1.x-staging).patch, > > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact-new-impl-under-refactoring-2.x-staging.patch > > > Now Kylin does not support query correct detail rows from fact table like: > select column1,column2,column3 from fact_table > The jira KYLIN-1075 add the "SUM" function on the measure column if defined. > But only the column number type is support. > I change some code to support this issue: > Add a "VALUE" measure function : the same value and datatype in the input and > output of this function. > If you want to query detail data from fact table > *require*: > 1.Configure the column which not dimensions to "VALUE" or "SUM" measure.(If > not configure measure function in the column will get NULL value) > 2.The source table must has an unique value column and configure it as > dimension. > If you have the better solution please comment here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-745) Generic Data Reader
[ https://issues.apache.org/jira/browse/KYLIN-745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-745: - Assignee: Dong Li (was: ZhouQianhao) > Generic Data Reader > --- > > Key: KYLIN-745 > URL: https://issues.apache.org/jira/browse/KYLIN-745 > Project: Kylin > Issue Type: Wish > Components: Job Engine, Spark Engine >Reporter: Luke Han >Assignee: Dong Li > > When data be stored on existing DW like Oracle, it's not be able to read > directly through Kylin to build cube. > There are many requirements coming from different teams like Candor about > this. > There are two options: > #1, copy your data to Hive and then build cube through Kylin. There are some > cases are running this model to bring data into Hive from DW and leveraging > Kylin very well. > #2, rewrite data read module to pull data from Oracle directly. Actually, > the first step of cube build is generate Hive Query to read data and generate > one temp table in Hive, so it should be not too complicated to do this (but > it depends network and others, otherwise, #1 will be more efficient one). > Then process cube build as normal. Using generical reader to read data from > any SQL rdbms through JDBC or other protocol will be perfect solution since > cube could be built without ETL process > Scope: > Only ready data directly from existing RDBMs and store jointed result in Hive > (temp table) for further processing, no any other transfomation. > By design, Kylin is OLAP system not ETL one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-741) Read data from SparkSQL
[ https://issues.apache.org/jira/browse/KYLIN-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-741: - Assignee: Dong Li (was: ZhouQianhao) > Read data from SparkSQL > --- > > Key: KYLIN-741 > URL: https://issues.apache.org/jira/browse/KYLIN-741 > Project: Kylin > Issue Type: New Feature > Components: Job Engine, SparkSQL >Reporter: Luke Han >Assignee: Dong Li > Fix For: Backlog > > > Read data from SparkSQL directly. > There are some instances enabled SparkSQL interface for data consuming, it > will be great if Kylin could read data directly from SparkSQL. > This feature does not require Spark Cube Build Engine to be ready. It could > continue to leverage existing MR cube build engine and process data on Hadoop > cluster then persistent cube to HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1057) No email notification on job failure in some rare cases
[ https://issues.apache.org/jira/browse/KYLIN-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-1057: -- Assignee: Dong Li (was: ZhouQianhao) > No email notification on job failure in some rare cases > --- > > Key: KYLIN-1057 > URL: https://issues.apache.org/jira/browse/KYLIN-1057 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Dong Li > > Usually Kylin will send email notification (if configured) on job failure; > while today we found there is no email under two job failures; This downgrade > the admin's response efficiency, need check the root cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-227) Support "Pause" on Kylin Job
[ https://issues.apache.org/jira/browse/KYLIN-227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-227: - Assignee: Dong Li (was: ZhouQianhao) > Support "Pause" on Kylin Job > > > Key: KYLIN-227 > URL: https://issues.apache.org/jira/browse/KYLIN-227 > Project: Kylin > Issue Type: Wish > Components: Job Engine >Reporter: Luke Han >Assignee: Dong Li > Labels: github-import > Fix For: Future > > > Add one action called "Pause" to stop current job, user could resume this job > later. > ![image|https://cloud.githubusercontent.com/assets/1104017/5556023/54ae27e2-8d07-11e4-8efb-a22c041243ba.png] > Imported from GitHub > Url: https://github.com/KylinOLAP/Kylin/issues/278 > Created by: [lukehan|https://github.com/lukehan] > Labels: newfeature, > Created at: Fri Dec 26 13:59:03 CST 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-299) add service to differentiate the deploy env
[ https://issues.apache.org/jira/browse/KYLIN-299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-299: - Assignee: Dong Li (was: ZhouQianhao) > add service to differentiate the deploy env > --- > > Key: KYLIN-299 > URL: https://issues.apache.org/jira/browse/KYLIN-299 > Project: Kylin > Issue Type: Wish > Components: Tools, Build and Test >Reporter: Luke Han >Assignee: Dong Li > Labels: github-import > Fix For: v0.6.5 > > > different deploy env (dev,qa,prod) may need different config, > need to add this feature > Imported from GitHub > Url: https://github.com/KylinOLAP/Kylin/issues/206 > Created by: [janzhongi|https://github.com/janzhongi] > Labels: newfeature, > Assignee: [janzhongi|https://github.com/janzhongi] > Created at: Wed Dec 10 10:46:23 CST 2014 > State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1302) java.lang.NullPointerException
[ https://issues.apache.org/jira/browse/KYLIN-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-1302: -- Assignee: Dong Li (was: ZhouQianhao) > java.lang.NullPointerException > -- > > Key: KYLIN-1302 > URL: https://issues.apache.org/jira/browse/KYLIN-1302 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.2 > Environment: CDH-5.4.8-1,hadoop-2.6.0,hive-1.1.0,hbase-1.0.0 >Reporter: gewenhui >Assignee: Dong Li > Fix For: 1.2 > > > in the UI of 7070,job run here: > #2 Step Name: Extract Fact Table Distinct Columns > 2016-01-10 07:44:35.222 - State of Hadoop job: job_1449729554133_9107:RUNNING > - UNDEFINED > 2016-01-10 07:44:45.236 - State of Hadoop job: > job_1449729554133_9107:FINISHED - SUCCEEDED > org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:228) > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159) > at > org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-744) Spark Cube Build Engine
[ https://issues.apache.org/jira/browse/KYLIN-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-744: - Assignee: Dong Li (was: ZhouQianhao) > Spark Cube Build Engine > --- > > Key: KYLIN-744 > URL: https://issues.apache.org/jira/browse/KYLIN-744 > Project: Kylin > Issue Type: New Feature > Components: Spark Engine >Reporter: Luke Han >Assignee: Dong Li > > Efficiency cube generate engine with Spark instead of MapReduce job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-903) automate metadata cleanup job
[ https://issues.apache.org/jira/browse/KYLIN-903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-903: - Assignee: Dong Li (was: ZhouQianhao) > automate metadata cleanup job > - > > Key: KYLIN-903 > URL: https://issues.apache.org/jira/browse/KYLIN-903 > Project: Kylin > Issue Type: Improvement >Reporter: ZhouQianhao >Assignee: Dong Li > Fix For: v2.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1122) Kylin support detail data query from fact table
[ https://issues.apache.org/jira/browse/KYLIN-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095778#comment-15095778 ] Xiaoyu Wang commented on KYLIN-1122: for the "3)" adjustSqlDigest may be contain the parameter "cubeDesc" will be better, In the raw custom function scenario, will use all columns or dimensions. > Kylin support detail data query from fact table > --- > > Key: KYLIN-1122 > URL: https://issues.apache.org/jira/browse/KYLIN-1122 > Project: Kylin > Issue Type: New Feature > Components: Query Engine >Affects Versions: v1.2 >Reporter: Xiaoyu Wang >Assignee: liyang > Fix For: v2.0, v1.3 > > Attachments: > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact(2.x-staging).patch, > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact(update-v2-1.x-staging).patch, > > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact-new-impl-under-refactoring-2.x-staging.patch > > > Now Kylin does not support query correct detail rows from fact table like: > select column1,column2,column3 from fact_table > The jira KYLIN-1075 add the "SUM" function on the measure column if defined. > But only the column number type is support. > I change some code to support this issue: > Add a "VALUE" measure function : the same value and datatype in the input and > output of this function. > If you want to query detail data from fact table > *require*: > 1.Configure the column which not dimensions to "VALUE" or "SUM" measure.(If > not configure measure function in the column will get NULL value) > 2.The source table must has an unique value column and configure it as > dimension. > If you have the better solution please comment here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1312) Enhance DeployCoprocessorCLI to support Cube level filter
Dong Li created KYLIN-1312: -- Summary: Enhance DeployCoprocessorCLI to support Cube level filter Key: KYLIN-1312 URL: https://issues.apache.org/jira/browse/KYLIN-1312 Project: Kylin Issue Type: Improvement Components: Tools, Build and Test Affects Versions: v2.0 Reporter: Dong Li Assignee: Dong Li Priority: Minor Currently DeployCoprocessorCLI can only filter on Tables, we need to enhance it to support Cube as filter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1307) revisit growing dictionary
hongbin ma created KYLIN-1307: - Summary: revisit growing dictionary Key: KYLIN-1307 URL: https://issues.apache.org/jira/browse/KYLIN-1307 Project: Kylin Issue Type: Improvement Reporter: hongbin ma Assignee: hongbin ma the optimization in KYLIN-871 will affect segment pruning based on dictionary values, which in turn hurts query performance. Will temporally roll back it and await more thorough thinking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093523#comment-15093523 ] wangxianbin commented on KYLIN-1082: OK, I think all we need to do is just merge new patch uploaded by fengyu, maybe a little modification. > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1294) Filter of NOT IN (...) can cause incorrect result or NPE
[
https://issues.apache.org/jira/browse/KYLIN-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093623#comment-15093623
]
hongbin ma commented on KYLIN-1294:
---
the issue blocks at CALCITE-980 before we could even reproduce your problem
> Filter of NOT IN (...) can cause incorrect result or NPE
>
>
> Key: KYLIN-1294
> URL: https://issues.apache.org/jira/browse/KYLIN-1294
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: hongbin ma
>
> The version I use is 1.0.
> Recently I updated to 1.2. The problent is like following:
> Case1:***
> When I use “not in” in my sql, the resultset is less than my
> correct data. Like following:
> When the sql I use not including a “not in” is :
> SELECT "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE" AS "H_CODE",
>COUNT(1) AS "sum_Number_of_Records_ok",
>SUM("KYLIN_VIEW_TVAD_SUMMARY"."OCCU") AS "sum_OCCU_ok"
> FROM "BD_WAREHOUSE"."KYLIN_VIEW_TVAD_SUMMARY" "KYLIN_VIEW_TVAD_SUMMARY"
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_CITY_V_KYLIN"
> "KYLIN_TV_DIM_CITY_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."CITY" =
>"KYLIN_TV_DIM_CITY_V_KYLIN"."CITY")
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> "KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."DT" =
>"KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"."DT1")
> GROUP BY "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE"
> The result is like following:
> (null) 206735 436180
> other169103517 480104320
> 无 25970813161884
> A 42464098 148884448
> B 31945514 121964156
> O 27878957265948
> K 37694048 127243245
> Whiel when I use a sql having “not in”, like following:
> SELECT "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE" AS "H_CODE",
>COUNT(1) AS "sum_Number_of_Records_ok",
>SUM("KYLIN_VIEW_TVAD_SUMMARY"."OCCU") AS "sum_OCCU_ok"
> FROM "BD_WAREHOUSE"."KYLIN_VIEW_TVAD_SUMMARY" "KYLIN_VIEW_TVAD_SUMMARY"
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_CITY_V_KYLIN"
> "KYLIN_TV_DIM_CITY_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."CITY" =
>"KYLIN_TV_DIM_CITY_V_KYLIN"."CITY")
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> "KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."DT" =
>"KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"."DT1")
> WHERE ("KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE" NOT IN ('other'))
> GROUP BY "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE"
> The result I got is, there are only 3 lines, and the results are all less
> than the true value:
> 无 323911 646111
> A 7463670 21674399
> K 15248274 54972183
> Case2:
> Sql1: select channel_id, sum(occu) from kylin_view_tvad_summary
> where channel_id not in (2, 3) group by channel_id.
> Here, 2 is a member of channel_id in kylin_view_tvad_summary, while
> 3 is not a member of kylin_view_tvad_summary, there will be a null pointer
> exception like following:
> java.sql.SQLException: error while executing SQL "select channel_id,
> sum(occu) from kylin_view_tvad_summary where channel_id not in (2, 3) group
> by channel_id
> LIMIT 5": null
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:112)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> at
> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:354)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:268)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:114)
> at
> org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke()
> at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618)
> at
> org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$3a29d57a.query()
> at
> org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:178)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:85)
> at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>
[jira] [Updated] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang updated KYLIN-1082: -- Assignee: Zhong Yanghong > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093507#comment-15093507 ] liyang commented on KYLIN-1082: --- Yes it's agreed that hive jar better get auto submitted. And that's why we have this JIRA. I've invited Zhong, Yanghong to work on this item. > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1302) java.lang.NullPointerException
[ https://issues.apache.org/jira/browse/KYLIN-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093597#comment-15093597 ] Shaofeng SHI commented on KYLIN-1302: - Hi gewen, please provide more info about the issue (hadoop environment, kylin version etc), and how to reproduce; only a stacktrace sometimes is poor for a developer to understand the issue. If you can solve the issue and contribute a patch, that will be welcomed. > java.lang.NullPointerException > -- > > Key: KYLIN-1302 > URL: https://issues.apache.org/jira/browse/KYLIN-1302 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.2 > Environment: CDH-5.4.8-1,hadoop-2.6.0,hive-1.1.0,hbase-1.0.0 >Reporter: gewenhui >Assignee: ZhouQianhao > Fix For: 1.2 > > > in the UI of 7070,job run here: > #2 Step Name: Extract Fact Table Distinct Columns > 2016-01-10 07:44:35.222 - State of Hadoop job: job_1449729554133_9107:RUNNING > - UNDEFINED > 2016-01-10 07:44:45.236 - State of Hadoop job: > job_1449729554133_9107:FINISHED - SUCCEEDED > org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): > java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:228) > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159) > at > org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1294) Filter of NOT IN (...) can cause incorrect result or NPE
[
https://issues.apache.org/jira/browse/KYLIN-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093604#comment-15093604
]
hongbin ma commented on KYLIN-1294:
---
the issue is pending on https://issues.apache.org/jira/browse/CALCITE-980,
which is fixed in calcite 1.6. We'll upgrade to calcite 1.6 as soon as they
release it
> Filter of NOT IN (...) can cause incorrect result or NPE
>
>
> Key: KYLIN-1294
> URL: https://issues.apache.org/jira/browse/KYLIN-1294
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: hongbin ma
>
> The version I use is 1.0.
> Recently I updated to 1.2. The problent is like following:
> Case1:***
> When I use “not in” in my sql, the resultset is less than my
> correct data. Like following:
> When the sql I use not including a “not in” is :
> SELECT "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE" AS "H_CODE",
>COUNT(1) AS "sum_Number_of_Records_ok",
>SUM("KYLIN_VIEW_TVAD_SUMMARY"."OCCU") AS "sum_OCCU_ok"
> FROM "BD_WAREHOUSE"."KYLIN_VIEW_TVAD_SUMMARY" "KYLIN_VIEW_TVAD_SUMMARY"
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_CITY_V_KYLIN"
> "KYLIN_TV_DIM_CITY_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."CITY" =
>"KYLIN_TV_DIM_CITY_V_KYLIN"."CITY")
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> "KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."DT" =
>"KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"."DT1")
> GROUP BY "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE"
> The result is like following:
> (null) 206735 436180
> other169103517 480104320
> 无 25970813161884
> A 42464098 148884448
> B 31945514 121964156
> O 27878957265948
> K 37694048 127243245
> Whiel when I use a sql having “not in”, like following:
> SELECT "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE" AS "H_CODE",
>COUNT(1) AS "sum_Number_of_Records_ok",
>SUM("KYLIN_VIEW_TVAD_SUMMARY"."OCCU") AS "sum_OCCU_ok"
> FROM "BD_WAREHOUSE"."KYLIN_VIEW_TVAD_SUMMARY" "KYLIN_VIEW_TVAD_SUMMARY"
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_CITY_V_KYLIN"
> "KYLIN_TV_DIM_CITY_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."CITY" =
>"KYLIN_TV_DIM_CITY_V_KYLIN"."CITY")
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> "KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."DT" =
>"KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"."DT1")
> WHERE ("KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE" NOT IN ('other'))
> GROUP BY "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE"
> The result I got is, there are only 3 lines, and the results are all less
> than the true value:
> 无 323911 646111
> A 7463670 21674399
> K 15248274 54972183
> Case2:
> Sql1: select channel_id, sum(occu) from kylin_view_tvad_summary
> where channel_id not in (2, 3) group by channel_id.
> Here, 2 is a member of channel_id in kylin_view_tvad_summary, while
> 3 is not a member of kylin_view_tvad_summary, there will be a null pointer
> exception like following:
> java.sql.SQLException: error while executing SQL "select channel_id,
> sum(occu) from kylin_view_tvad_summary where channel_id not in (2, 3) group
> by channel_id
> LIMIT 5": null
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:112)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> at
> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:354)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:268)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:114)
> at
> org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke()
> at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618)
> at
> org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$3a29d57a.query()
> at
> org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:178)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:85)
> at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
[jira] [Commented] (KYLIN-1294) Filter of NOT IN (...) can cause incorrect result or NPE
[
https://issues.apache.org/jira/browse/KYLIN-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093619#comment-15093619
]
huawang commented on KYLIN-1294:
The question is even the sql doesn't throw a NPE, the result set of the sql is
not correct as shown in the but description.
> Filter of NOT IN (...) can cause incorrect result or NPE
>
>
> Key: KYLIN-1294
> URL: https://issues.apache.org/jira/browse/KYLIN-1294
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: hongbin ma
>
> The version I use is 1.0.
> Recently I updated to 1.2. The problent is like following:
> Case1:***
> When I use “not in” in my sql, the resultset is less than my
> correct data. Like following:
> When the sql I use not including a “not in” is :
> SELECT "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE" AS "H_CODE",
>COUNT(1) AS "sum_Number_of_Records_ok",
>SUM("KYLIN_VIEW_TVAD_SUMMARY"."OCCU") AS "sum_OCCU_ok"
> FROM "BD_WAREHOUSE"."KYLIN_VIEW_TVAD_SUMMARY" "KYLIN_VIEW_TVAD_SUMMARY"
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_CITY_V_KYLIN"
> "KYLIN_TV_DIM_CITY_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."CITY" =
>"KYLIN_TV_DIM_CITY_V_KYLIN"."CITY")
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> "KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."DT" =
>"KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"."DT1")
> GROUP BY "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE"
> The result is like following:
> (null) 206735 436180
> other169103517 480104320
> 无 25970813161884
> A 42464098 148884448
> B 31945514 121964156
> O 27878957265948
> K 37694048 127243245
> Whiel when I use a sql having “not in”, like following:
> SELECT "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE" AS "H_CODE",
>COUNT(1) AS "sum_Number_of_Records_ok",
>SUM("KYLIN_VIEW_TVAD_SUMMARY"."OCCU") AS "sum_OCCU_ok"
> FROM "BD_WAREHOUSE"."KYLIN_VIEW_TVAD_SUMMARY" "KYLIN_VIEW_TVAD_SUMMARY"
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_CITY_V_KYLIN"
> "KYLIN_TV_DIM_CITY_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."CITY" =
>"KYLIN_TV_DIM_CITY_V_KYLIN"."CITY")
> LEFT JOIN "BD_WAREHOUSE"."KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> "KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"
> ON ("KYLIN_VIEW_TVAD_SUMMARY"."DT" =
>"KYLIN_TV_DIM_DATE_CUBE_V_KYLIN"."DT1")
> WHERE ("KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE" NOT IN ('other'))
> GROUP BY "KYLIN_TV_DIM_CITY_V_KYLIN"."H_CODE"
> The result I got is, there are only 3 lines, and the results are all less
> than the true value:
> 无 323911 646111
> A 7463670 21674399
> K 15248274 54972183
> Case2:
> Sql1: select channel_id, sum(occu) from kylin_view_tvad_summary
> where channel_id not in (2, 3) group by channel_id.
> Here, 2 is a member of channel_id in kylin_view_tvad_summary, while
> 3 is not a member of kylin_view_tvad_summary, there will be a null pointer
> exception like following:
> java.sql.SQLException: error while executing SQL "select channel_id,
> sum(occu) from kylin_view_tvad_summary where channel_id not in (2, 3) group
> by channel_id
> LIMIT 5": null
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:112)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> at
> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:354)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:268)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:114)
> at
> org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke()
> at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618)
> at
> org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$3a29d57a.query()
> at
> org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:178)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:85)
> at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>
[jira] [Commented] (KYLIN-1220) JsonMappingException: Can not deserialize instance of java.lang.String out of START_ARRAY
[ https://issues.apache.org/jira/browse/KYLIN-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093716#comment-15093716 ] Zhong,Jason commented on KYLIN-1220: it's resolved now > JsonMappingException: Can not deserialize instance of java.lang.String out of > START_ARRAY > - > > Key: KYLIN-1220 > URL: https://issues.apache.org/jira/browse/KYLIN-1220 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0 > Environment: HDP2.3.2 >Reporter: wangxianbin >Assignee: Zhong,Jason > Fix For: 2.0 > > > JsonMappingException when I create cube, I think it cause by class var > "column" in class "DimensionDesc" which is not String array any more, how > ever web didn't follow up this change. > L4J [2015-12-10 > 17:23:50,781][ERROR][org.apache.kylin.rest.controller.CubeController] - The > cube definition is not valid. > com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize > instance of java.lang.String out of START_ARRAY token > at [Source: java.io.StringReader@407d4132; line: 10, column: 22] (through > reference chain: > org.apache.kylin.cube.model.CubeDesc["dimensions"]->org.apache.kylin.cube.model.DimensionDesc["column"]) > at > com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:164) > at > com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:575) > at > com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:46) > at > com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11) > at > com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:464) > at > com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:107) > at > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:295) > at > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:121) > at > com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:230) > at > com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:207) > at > com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:23) > at > com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:464) > at > com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:107) > at > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:295) > at > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:121) > at > com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2888) > at > com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2034) > at org.apache.kylin.common.util.JsonUtil.readValue(JsonUtil.java:54) > at > org.apache.kylin.rest.controller.CubeController.deserializeCubeDesc(CubeController.java:699) > at > org.apache.kylin.rest.controller.CubeController.saveCubeDesc(CubeController.java:327) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1220) JsonMappingException: Can not deserialize instance of java.lang.String out of START_ARRAY
[ https://issues.apache.org/jira/browse/KYLIN-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong,Jason resolved KYLIN-1220. Resolution: Fixed Fix Version/s: 2.0 > JsonMappingException: Can not deserialize instance of java.lang.String out of > START_ARRAY > - > > Key: KYLIN-1220 > URL: https://issues.apache.org/jira/browse/KYLIN-1220 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0 > Environment: HDP2.3.2 >Reporter: wangxianbin >Assignee: Zhong,Jason > Fix For: 2.0 > > > JsonMappingException when I create cube, I think it cause by class var > "column" in class "DimensionDesc" which is not String array any more, how > ever web didn't follow up this change. > L4J [2015-12-10 > 17:23:50,781][ERROR][org.apache.kylin.rest.controller.CubeController] - The > cube definition is not valid. > com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize > instance of java.lang.String out of START_ARRAY token > at [Source: java.io.StringReader@407d4132; line: 10, column: 22] (through > reference chain: > org.apache.kylin.cube.model.CubeDesc["dimensions"]->org.apache.kylin.cube.model.DimensionDesc["column"]) > at > com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:164) > at > com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:575) > at > com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:46) > at > com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11) > at > com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:464) > at > com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:107) > at > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:295) > at > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:121) > at > com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:230) > at > com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:207) > at > com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:23) > at > com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:464) > at > com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:107) > at > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:295) > at > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:121) > at > com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2888) > at > com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2034) > at org.apache.kylin.common.util.JsonUtil.readValue(JsonUtil.java:54) > at > org.apache.kylin.rest.controller.CubeController.deserializeCubeDesc(CubeController.java:699) > at > org.apache.kylin.rest.controller.CubeController.saveCubeDesc(CubeController.java:327) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1122) Kylin support detail data query from fact table
[ https://issues.apache.org/jira/browse/KYLIN-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093740#comment-15093740 ] Xiaoyu Wang commented on KYLIN-1122: I created a new branch "KYLIN-1122" base on 2.x-staging. and submit my patch. > Kylin support detail data query from fact table > --- > > Key: KYLIN-1122 > URL: https://issues.apache.org/jira/browse/KYLIN-1122 > Project: Kylin > Issue Type: New Feature > Components: Query Engine >Affects Versions: v1.2 >Reporter: Xiaoyu Wang >Assignee: liyang > Fix For: v2.0, v1.3 > > Attachments: > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact(2.x-staging).patch, > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact(update-v2-1.x-staging).patch, > > 0001-KYLIN-1122-Kylin-support-detail-data-query-from-fact-new-impl-under-refactoring-2.x-staging.patch > > > Now Kylin does not support query correct detail rows from fact table like: > select column1,column2,column3 from fact_table > The jira KYLIN-1075 add the "SUM" function on the measure column if defined. > But only the column number type is support. > I change some code to support this issue: > Add a "VALUE" measure function : the same value and datatype in the input and > output of this function. > If you want to query detail data from fact table > *require*: > 1.Configure the column which not dimensions to "VALUE" or "SUM" measure.(If > not configure measure function in the column will get NULL value) > 2.The source table must has an unique value column and configure it as > dimension. > If you have the better solution please comment here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
