[
https://issues.apache.org/jira/browse/KYLIN-3636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691765#comment-16691765
]
KANG-SEN LU commented on KYLIN-3636:
------------------------------------
Hi, Shaofeng:
I just created a new cube design, the storage_type was set to 0. I am sending
you the json file of the cube here:
* [Grid|http://192.168.230.62:7070/kylin/]
* [SQL|http://192.168.230.62:7070/kylin/]
* [JSON(Cube)|http://192.168.230.62:7070/kylin/]
* [Notification|http://192.168.230.62:7070/kylin/]
* [Storage|http://192.168.230.62:7070/kylin/]
* [Planner|http://192.168.230.62:7070/kylin/]
{
"uuid": "8b9c51f3-e9b6-45ae-998c-c66fbf87dcad",
"last_modified": 1542636675073,
"version": "2.5.1.20500",
"name": "test",
"is_draft": false,
"model_name": "ma_aggs_model",
"description": "",
"null_string": null,
"dimensions": [
\{
"name": "APPLICATION_NAME",
"table": "A_MA_HOURLY_V",
"column": "APPLICATION_NAME",
"derived": null
},
\{
"name": "BRAND_NAME",
"table": "A_MA_HOURLY_V",
"column": "BRAND_NAME",
"derived": null
}
],
"measures": [
\{
"name": "_COUNT_",
"function": {
"expression": "COUNT",
"parameter": {
"type": "constant",
"value": "1"
},
"returntype": "bigint"
}
}
],
"dictionaries": [],
"rowkey": \{
"rowkey_columns": [
{
"column": "A_MA_HOURLY_V.APPLICATION_NAME",
"encoding": "dict",
"encoding_version": 1,
"isShardBy": false
},
\{
"column": "A_MA_HOURLY_V.BRAND_NAME",
"encoding": "dict",
"encoding_version": 1,
"isShardBy": false
}
]
},
"hbase_mapping": \{
"column_family": [
{
"name": "F1",
"columns": [
{
"qualifier": "M",
"measure_refs": [
"_COUNT_"
]
}
]
}
]
},
"aggregation_groups": [
\{
"includes": [
"A_MA_HOURLY_V.APPLICATION_NAME",
"A_MA_HOURLY_V.BRAND_NAME"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [],
"joint_dims": []
}
}
],
"signature": "iBgKI2sCq9L9zGNctnmryw==",
"notify_list": [],
"status_need_notify": [
"ERROR",
"DISCARDED",
"SUCCEED"
],
"partition_date_start": 0,
"partition_date_end": 3153600000000,
"auto_merge_time_ranges": [
604800000,
2419200000
],
"volatile_range": 0,
"retention_range": 0,
"engine_type": 2,
"storage_type": 0,
"override_kylin_properties": {},
"cuboid_black_list": [],
"parent_forward": 3,
"mandatory_dimension_set_list": [],
"snapshot_table_desc_list": []
}
Here is the java code in our kylin 2.5.1 sandbox:
4af0f33248 core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
(honma 2015-09-09 10:01:55 +0800 180) @JsonProperty("engine_type")
c56c741a92 core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
(shaofengshi 2017-11-05 16:57:47 +0800 181) private int engineType =
IEngineAware.ID_MR_V2;
4af0f33248 core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
(honma 2015-09-09 10:01:55 +0800 182)
@JsonProperty("storage_type")
4af0f33248 core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
(honma 2015-09-09 10:01:55 +0800 183) private int storageType =
IStorageAware.ID_HBASE;
> in kylin 2.4.1 and 2.5.1 topn aggregation query caused exception
> ----------------------------------------------------------------
>
> Key: KYLIN-3636
> URL: https://issues.apache.org/jira/browse/KYLIN-3636
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: v2.4.1
> Reporter: KANG-SEN LU
> Assignee: Shaofeng SHI
> Priority: Major
> Fix For: v2.6.0
>
>
> Hi, ShaoFeng:
>
> Thanks for the reply. I missed this email and not responded earlier, I am
> sorry.
>
> I tried to reproduce this problem with the sample database, and it did not
> happen.
>
> So I am hoping by collecting enough "clue", someone can figure out why this
> problem occurred.
>
> --------------------------
> I issued the following query at the sample project to exercise the topn
> aggregation:
>
> select seller_id, SUM(price) as total from kylin_sales group by seller_id
> order by total limit 5;
>
> With my own added debugging, I saw the following log in the kylin.log: (the
> query worked OK).
>
> 2018-10-16 16:18:19,963 INFO [Query a747f16f-4b12-cc97-08d2-9b45c27a529f-90]
> model.FunctionDesc:59 : KSL22222,
> getRewriteFieldName=_KY_SUM_KYLIN_SALES_PRICE_
> 2018-10-16 16:18:19,963 INFO [kylin-coproc--pool12-t1]
> v2.CubeHBaseEndpointRPC:217 : Query-a747f16f-4b12-cc97-08d2-9b45c27a529f:
> send request to the init region server anovadata4.anovadata.local on table
> ANOVA_KYLIN_25X_K758MEAWJG
> 2018-10-16 16:18:19,963 INFO [Query a747f16f-4b12-cc97-08d2-9b45c27a529f-90]
> topn.TopNMeasureType:399 : KSL888: in TopNMeasureType.java, sumFieldName=
> _KY_SUM_KYLIN_SALES_PRICE_
>
>
> When I was executing my project query, I issued the following select
> statement:
>
> SELECT ZETTICSDW.A_VL_HOURLY_V.IMSIID \"ZETTICSDW_A_VL_HOURLY_V_IMSIID\",
> SUM(ZETTICSDW.A_VL_HOURLY_V.SIG_EVENT_COUNT)
> \"vl_aggs_model___USERS_BY_ERROR_3XX\" FROM ZETTICSDW.A_VL_HOURLY_V inner
> JOIN ZETTICSDW.T_VL_TRANSACTION_RULE_V ON (
> ZETTICSDW.A_VL_HOURLY_V.CAUSE_CODE_KEY =
> ZETTICSDW.T_VL_TRANSACTION_RULE_V.CAUSE_CODE_KEY AND
> ZETTICSDW.A_VL_HOURLY_V.REASON_CODE_KEY =
> ZETTICSDW.T_VL_TRANSACTION_RULE_V.REASON_CODE_KEY AND
> ZETTICSDW.A_VL_HOURLY_V.TRANSACTION_TYPE_KEY =
> ZETTICSDW.T_VL_TRANSACTION_RULE_V.TRANSACTION_TYPE_KEY)
> WHERE ((ZETTICSDW.A_VL_HOURLY_V.THEDATE = '20180209') AND
> ((ZETTICSDW.A_VL_HOURLY_V.THEHOUR >= '02') AND
> (ZETTICSDW.A_VL_HOURLY_V.THEHOUR <= '03'))) AND
> ZETTICSDW.T_VL_TRANSACTION_RULE_V.DISPLAY_STRING LIKE '%+3%'
> GROUP BY ZETTICSDW.A_VL_HOURLY_V.IMSIID
> ORDER BY \"vl_aggs_model___USERS_BY_ERROR_3XX\"
> LIMIT 25
>
> An exception occurred within the method "private ColumnRowType
> buildColumnRowType()" of
> "query/src/main/java/org/apache/kylin/query/relnode/OLAPTableScan.java".
>
> if (columns.size() != rowType.getFieldCount()) {
> throw new IllegalStateException("RowType=" +
> rowType.getFieldCount() + ", ColumnRowType=" + columns.size());
> }
>
> It printed "RowType=133, ColumnRowType=132".
> The RowType list contains one extra column name: "ANY
> _KY_SUM_1_3a1aedef_SIG_EVENT_COUNT_".
>
>
> I think this has something to do with the bug fix "KYLIN-3359 Support
> sum(expression) if possible".
>
> After this bug fix was submitted, I noticed that a lot of column name was
> added into rowType like "_KY_SUM_XXX".
>
> This strange column name "_KY_SUM_1_3a1aedef_SIG_EVENT_COUNT_" is very
> similar.
>
> I also found that this extra column name, only existed in RowType but not in
> ColumnRowType, was added in the method "public void
> implementRewrite(RewriteImplementor implementor)" within "OLAPJoinRel.java".
>
> With my own debug statement, I saw this debug text in kylin.log:
>
> relnode.OLAPJoinRel:362 : KSL54: newField= #132:
> _KY_SUM_1_6735969a_SIG_EVENT_COUNT_ ANY
>
>
> I hope someone with a deeper technical knowledge in kylin query engine can
> figure out what was causing the problem I have seen.
>
> Thanks again.
>
> Kang-sen
>
> *From:* ShaoFeng Shi [[mailto:shaofeng...@apache.org]]
> *Sent:* Friday, October 05, 2018 9:59 PM
> *To:* user <[u...@kylin.apache.org|mailto:u...@kylin.apache.org]>
> *Subject:* Re: any body see topn in kylin 2.5.1 working?
>
> Hi Kang-Sen,
>
> Didn't see this; Can you reproduce the problem with the sample cube?
>
> Kang-Sen Lu <[k...@anovadata.com|mailto:k...@anovadata.com]> 于2018年10月6日周六
> 上午12:21写道：
> I have been running kylin 2.2.1 with topn aggregation OK.
>
> Recently, I upgraded to kylin 2.4.1 and 2.5.1, in both releases I am having
> query problem related to topn support.
>
> The problem is an exception occurred with the following error message in
> kylin.log:
>
> Caused by: java.lang.IllegalStateException: RowType=133, ColumnRowType=132
> at
> org.apache.kylin.query.relnode.OLAPJoinRel.buildColumnRowType(OLAPJoinRel.java:223)
> at
> org.apache.kylin.query.relnode.OLAPJoinRel.implementRewrite(OLAPJoinRel.java:359)
> at
> org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
> at
> org.apache.kylin.query.relnode.OLAPFilterRel.implementRewrite(OLAPFilterRel.java:163)
> at
> org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
> at
> org.apache.kylin.query.relnode.OLAPProjectRel.implementRewrite(OLAPProjectRel.java:228)
> at
> org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
> at
> org.apache.kylin.query.relnode.OLAPAggregateRel.implementRewrite(OLAPAggregateRel.java:402)
> at
> org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
> at
> org.apache.kylin.query.relnode.OLAPSortRel.implementRewrite(OLAPSortRel.java:86)
> at
> org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
> at
> org.apache.kylin.query.relnode.OLAPLimitRel.implementRewrite(OLAPLimitRel.java:109)
> at
> org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
> at
> org.apache.kylin.query.relnode.OLAPToEnumerableConverter.implement(OLAPToEnumerableConverter.java:100)
> at
> org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.implementRoot(EnumerableRelImplementor.java:108)
> at
> org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:92)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1281)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:331)
> at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:228)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:801)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:656)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:618)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:214)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:603)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
> ... 84 more
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
