[jira] [Commented] (KYLIN-4174) job statistics info is incorrect
[
https://issues.apache.org/jira/browse/KYLIN-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939145#comment-16939145
]
ZhouKang commented on KYLIN-4174:
-
[~wangrupeng]
sorry, miss description in issue. you should choose a project first.
> job statistics info is incorrect
>
>
> Key: KYLIN-4174
> URL: https://issues.apache.org/jira/browse/KYLIN-4174
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Assignee: luguosheng
>Priority: Major
> Attachments: image-2019-09-27-12-27-50-336.png,
> image-2019-09-27-12-27-50-393.png, image3.png
>
>
> Kylin: master branch
> The job statistics info is incorrect. The number behind type is not equal to
> the size of job list.
> Step 1: login kylin webui, open the Monitor
> Step 2: refresh the web page
>
> You can see, the query params of RESTful API is wrong, projectName is "_null"
> {code:java}
> // code placeholder
> curl
> 'http://zjy-bigdata-prc-kylin04.bj:7070/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1'
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4174) job statistics info is incorrect
[
https://issues.apache.org/jira/browse/KYLIN-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939144#comment-16939144
]
luguosheng commented on KYLIN-4174:
---
It was fixed. linked to jira number:KYLIN-4100 and code in
https://github.com/apache/kylin/pull/857
but when you full refresh page it will always double request overview api when
project change from null to a certain value because of the project notice
event. if you just change menu to this page. it will work well.
> job statistics info is incorrect
>
>
> Key: KYLIN-4174
> URL: https://issues.apache.org/jira/browse/KYLIN-4174
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Assignee: luguosheng
>Priority: Major
> Attachments: image-2019-09-27-12-27-50-336.png,
> image-2019-09-27-12-27-50-393.png, image3.png
>
>
> Kylin: master branch
> The job statistics info is incorrect. The number behind type is not equal to
> the size of job list.
> Step 1: login kylin webui, open the Monitor
> Step 2: refresh the web page
>
> You can see, the query params of RESTful API is wrong, projectName is "_null"
> {code:java}
> // code placeholder
> curl
> 'http://zjy-bigdata-prc-kylin04.bj:7070/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1'
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KYLIN-4174) job statistics info is incorrect
[
https://issues.apache.org/jira/browse/KYLIN-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luguosheng reassigned KYLIN-4174:
-
Assignee: luguosheng
> job statistics info is incorrect
>
>
> Key: KYLIN-4174
> URL: https://issues.apache.org/jira/browse/KYLIN-4174
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Assignee: luguosheng
>Priority: Major
> Attachments: image-2019-09-27-12-27-50-336.png,
> image-2019-09-27-12-27-50-393.png, image3.png
>
>
> Kylin: master branch
> The job statistics info is incorrect. The number behind type is not equal to
> the size of job list.
> Step 1: login kylin webui, open the Monitor
> Step 2: refresh the web page
>
> You can see, the query params of RESTful API is wrong, projectName is "_null"
> {code:java}
> // code placeholder
> curl
> 'http://zjy-bigdata-prc-kylin04.bj:7070/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1'
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4174) job statistics info is incorrect
[
https://issues.apache.org/jira/browse/KYLIN-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939111#comment-16939111
]
wangrupeng edited comment on KYLIN-4174 at 9/27/19 4:32 AM:
I try to use the latest version of kylin built from master branch and I follow
your steps, but I get the request url like
"http://cdh-client:7919/kylin/api/jobs/overview?jobSearchMode=ALL&timeFilter=2";
. it don't contain the param "projectName".
!image3.png|width=497,height=455!
!image-2019-09-27-12-27-50-393.png|width=897,height=291!
But I use the restful tool to send the request with param "projectName" and the
url is
"http://cdh-client:7919/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1";
, it seems nothing wrong about the job info overview.
!image-2019-09-27-12-27-50-336.png|width=698,height=296!
was (Author: wangrupeng):
I try to use the latest version of kylin built from master branch and I follow
your steps, but I get the request url like
"http://cdh-client:7919/kylin/api/jobs/overview?jobSearchMode=ALL&timeFilter=2";
. it don't contain the param "projectName".
!image3.png|width=497,height=455!
!image-2019-09-27-12-27-50-393.png|width=897,height=291!
But I use the restful tool to send the request with param "projectName" and the
url is
"http://cdh-client:7919/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1";
!image-2019-09-27-12-27-50-336.png|width=698,height=296!
> job statistics info is incorrect
>
>
> Key: KYLIN-4174
> URL: https://issues.apache.org/jira/browse/KYLIN-4174
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
> Attachments: image-2019-09-27-12-27-50-336.png,
> image-2019-09-27-12-27-50-393.png, image3.png
>
>
> Kylin: master branch
> The job statistics info is incorrect. The number behind type is not equal to
> the size of job list.
> Step 1: login kylin webui, open the Monitor
> Step 2: refresh the web page
>
> You can see, the query params of RESTful API is wrong, projectName is "_null"
> {code:java}
> // code placeholder
> curl
> 'http://zjy-bigdata-prc-kylin04.bj:7070/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1'
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4174) job statistics info is incorrect
[
https://issues.apache.org/jira/browse/KYLIN-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939111#comment-16939111
]
wangrupeng commented on KYLIN-4174:
---
I try to use the latest version of kylin built from master branch and I follow
your steps, but I get the request url like
"http://cdh-client:7919/kylin/api/jobs/overview?jobSearchMode=ALL&timeFilter=2";
. it don't contain the param "projectName".
!image3.png|width=497,height=455!
!image-2019-09-27-12-27-50-393.png|width=897,height=291!
But I use the restful tool to send the request with param "projectName" and the
url is
"http://cdh-client:7919/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1";
!image-2019-09-27-12-27-50-336.png|width=698,height=296!
> job statistics info is incorrect
>
>
> Key: KYLIN-4174
> URL: https://issues.apache.org/jira/browse/KYLIN-4174
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
> Attachments: image-2019-09-27-12-27-50-336.png,
> image-2019-09-27-12-27-50-393.png, image3.png
>
>
> Kylin: master branch
> The job statistics info is incorrect. The number behind type is not equal to
> the size of job list.
> Step 1: login kylin webui, open the Monitor
> Step 2: refresh the web page
>
> You can see, the query params of RESTful API is wrong, projectName is "_null"
> {code:java}
> // code placeholder
> curl
> 'http://zjy-bigdata-prc-kylin04.bj:7070/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1'
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4174) job statistics info is incorrect
[
https://issues.apache.org/jira/browse/KYLIN-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangrupeng updated KYLIN-4174:
--
Attachment: image-2019-09-27-12-27-50-393.png
> job statistics info is incorrect
>
>
> Key: KYLIN-4174
> URL: https://issues.apache.org/jira/browse/KYLIN-4174
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
> Attachments: image-2019-09-27-12-27-50-336.png,
> image-2019-09-27-12-27-50-393.png, image3.png
>
>
> Kylin: master branch
> The job statistics info is incorrect. The number behind type is not equal to
> the size of job list.
> Step 1: login kylin webui, open the Monitor
> Step 2: refresh the web page
>
> You can see, the query params of RESTful API is wrong, projectName is "_null"
> {code:java}
> // code placeholder
> curl
> 'http://zjy-bigdata-prc-kylin04.bj:7070/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1'
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4174) job statistics info is incorrect
[
https://issues.apache.org/jira/browse/KYLIN-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangrupeng updated KYLIN-4174:
--
Attachment: image-2019-09-27-12-27-50-336.png
> job statistics info is incorrect
>
>
> Key: KYLIN-4174
> URL: https://issues.apache.org/jira/browse/KYLIN-4174
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
> Attachments: image-2019-09-27-12-27-50-336.png,
> image-2019-09-27-12-27-50-393.png, image3.png
>
>
> Kylin: master branch
> The job statistics info is incorrect. The number behind type is not equal to
> the size of job list.
> Step 1: login kylin webui, open the Monitor
> Step 2: refresh the web page
>
> You can see, the query params of RESTful API is wrong, projectName is "_null"
> {code:java}
> // code placeholder
> curl
> 'http://zjy-bigdata-prc-kylin04.bj:7070/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1'
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4174) job statistics info is incorrect
[
https://issues.apache.org/jira/browse/KYLIN-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangrupeng updated KYLIN-4174:
--
Attachment: image3.png
> job statistics info is incorrect
>
>
> Key: KYLIN-4174
> URL: https://issues.apache.org/jira/browse/KYLIN-4174
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
> Attachments: image3.png
>
>
> Kylin: master branch
> The job statistics info is incorrect. The number behind type is not equal to
> the size of job list.
> Step 1: login kylin webui, open the Monitor
> Step 2: refresh the web page
>
> You can see, the query params of RESTful API is wrong, projectName is "_null"
> {code:java}
> // code placeholder
> curl
> 'http://zjy-bigdata-prc-kylin04.bj:7070/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1'
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] shaofengshi merged pull request #862: Added blog entry describing how to connect e.g. Tableau to Apache Kylin
shaofengshi merged pull request #862: Added blog entry describing how to connect e.g. Tableau to Apache Kylin URL: https://github.com/apache/kylin/pull/862 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] pnaszarkowski commented on issue #862: Added blog entry describing how to connect e.g. Tableau to Apache Kylin
pnaszarkowski commented on issue #862: Added blog entry describing how to connect e.g. Tableau to Apache Kylin URL: https://github.com/apache/kylin/pull/862#issuecomment-535659251 To be reviewed by @shaofengshi ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asf-ci commented on issue #862: Added blog entry 2019-09-22-kylin-tableau.md describing how to connec…
asf-ci commented on issue #862: Added blog entry 2019-09-22-kylin-tableau.md describing how to connec… URL: https://github.com/apache/kylin/pull/862#issuecomment-535657476 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] pnaszarkowski opened a new pull request #862: Added blog entry 2019-09-22-kylin-tableau.md describing how to connec…
pnaszarkowski opened a new pull request #862: Added blog entry 2019-09-22-kylin-tableau.md describing how to connec… URL: https://github.com/apache/kylin/pull/862 …t Tableau with Kylin This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Reopened] (KYLIN-3628) Query with lookup table always use latest snapshot
[ https://issues.apache.org/jira/browse/KYLIN-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vsevolod Ostapenko reopened KYLIN-3628: --- The most recent change will silently swallow legit exceptions in the findLatestSnapshot and may result in an incorrect cube instance to be returned, effectively re-introducing the original bug for cases when there is an exception. > Query with lookup table always use latest snapshot > -- > > Key: KYLIN-3628 > URL: https://issues.apache.org/jira/browse/KYLIN-3628 > Project: Kylin > Issue Type: Improvement >Reporter: Na Zhai >Assignee: Na Zhai >Priority: Major > Fix For: v3.0.0-alpha2, v2.6.4 > > > If user queries a lookup table, Kylin will randomly selects a Cube (which has > the snapshot of this lookup table) to answer it. This causes uncertainty when > there are multiple cubes (share the same lookup): some cubes are newly built, > some not. If Kylin picks an old cube, the query result is old. > To remove this uncertainty, for such queries, either always use latest > snapshot, or use earlist snapshot. We believe the "latest" version is better. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-3628) Query with lookup table always use latest snapshot
[ https://issues.apache.org/jira/browse/KYLIN-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938797#comment-16938797 ] Vsevolod Ostapenko commented on KYLIN-3628: --- [~hit_lacus] The latest change effectively reintroduces the issue that was supposed to be fixed by the prior series of changes. I don't see a good reason for doing try/catch and silently swallowing _any_ exception and then returning the current cube, even when it does not contain lookup table snapshot. > Query with lookup table always use latest snapshot > -- > > Key: KYLIN-3628 > URL: https://issues.apache.org/jira/browse/KYLIN-3628 > Project: Kylin > Issue Type: Improvement >Reporter: Na Zhai >Assignee: Na Zhai >Priority: Major > Fix For: v3.0.0-alpha2, v2.6.4 > > > If user queries a lookup table, Kylin will randomly selects a Cube (which has > the snapshot of this lookup table) to answer it. This causes uncertainty when > there are multiple cubes (share the same lookup): some cubes are newly built, > some not. If Kylin picks an old cube, the query result is old. > To remove this uncertainty, for such queries, either always use latest > snapshot, or use earlist snapshot. We believe the "latest" version is better. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] coveralls edited a comment on issue #860: build dimension dictionary using spark
coveralls edited a comment on issue #860: build dimension dictionary using spark URL: https://github.com/apache/kylin/pull/860#issuecomment-535306368 ## Pull Request Test Coverage Report for [Build 5063](https://coveralls.io/builds/25950247) * **0** of **277** **(0.0%)** changed or added relevant lines in **6** files are covered. * **6** unchanged lines in **5** files lost coverage. * Overall coverage decreased (**-0.09%**) to **27.823%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/25950247/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L1458) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/model/TblColRef.java](https://coveralls.io/builds/25950247/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FTblColRef.java#L284) | 0 | 1 | 0.0% | [engine-mr/src/main/java/org/apache/kylin/engine/mr/JobBuilderSupport.java](https://coveralls.io/builds/25950247/source?filename=engine-mr%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fmr%2FJobBuilderSupport.java#L255) | 0 | 1 | 0.0% | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkExecutable.java](https://coveralls.io/builds/25950247/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkExecutable.java#L371) | 0 | 10 | 0.0% | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java](https://coveralls.io/builds/25950247/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkBatchCubingJobBuilder2.java#L80) | 0 | 18 | 0.0% | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBuildDictionary.java](https://coveralls.io/builds/25950247/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkBuildDictionary.java#L86) | 0 | 246 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/25950247/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L553) | 1 | 78.12% | | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/25950247/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% | | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java](https://coveralls.io/builds/25950247/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkBatchCubingJobBuilder2.java#L86) | 1 | 0.0% | | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkExecutable.java](https://coveralls.io/builds/25950247/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkExecutable.java#L374) | 1 | 0.0% | | [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/25950247/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% | | Totals | [](https://coveralls.io/builds/25950247) | | :-- | --: | | Change from base [Build 5053](https://coveralls.io/builds/25913474): | -0.09% | | Covered Lines: | 23703 | | Relevant Lines: | 85192 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3518) Coprocessor reports NPE when execute a query on HBase 2.0
[
https://issues.apache.org/jira/browse/KYLIN-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938646#comment-16938646
]
ASF subversion and git services commented on KYLIN-3518:
Commit 64fd8083e61bfc2a9a9b15e8548dee4617f176dc in kylin's branch
refs/heads/2.6.x-hadoop3.1 from Cheng Wang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=64fd808 ]
KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT
KYLIN-3518 Fix Coprocessor NPE problem on hbase 2
Signed-off-by: shaofengshi
KYLIN-2565 Add cdh60 profile
Signed-off-by: shaofengshi
> Coprocessor reports NPE when execute a query on HBase 2.0
> -
>
> Key: KYLIN-3518
> URL: https://issues.apache.org/jira/browse/KYLIN-3518
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Reporter: Shao Feng Shi
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.5.0
>
>
> On HDP 3.0, build a cube and then run a simple count query, NPE occurred:
>
> {code:java}
> 2018-08-28 01:30:16,969 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 9 ms, use 81000 ms as timeout
> for coprocessor
> 2018-08-28 01:30:16,983 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:141 : Serialized scanRequestBytes 522 bytes,
> rawScanBytesString 44 bytes
> 2018-08-28 01:30:16,984 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:143 : The scan 67b41fc6 for segment
> kylin_sales_cube_clone[2012010100_2013010100] is as below with 1
> separate raw scans, shard part of start/end key is set to 0
> 2018-08-28 01:30:16,991 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseRPC:288 : Visiting hbase table KYLIN_5Q088VO5I0: cuboid require
> post aggregation, from 0 to 16384 Start:
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00) Stop:
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\xFF\xFF\xFF\x00
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\xFF\xFF\xFF\x00), No Fuzzy Key
> 2018-08-28 01:30:16,991 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:148 : Submitting rpc to 1 shards starting from shard
> 0, scan range count 1
> 2018-08-28 01:30:17,010 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> common.KylinConfig:332 : Loading kylin-defaults.properties from
> file:/root/shaofengshi/apache-kylin-2.5.0-SNAPSHOT-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.5.0-SNAPSHOT.jar!/kylin-defaults.properties
> 2018-08-28 01:30:17,033 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> common.KylinConfig:291 : KYLIN_CONF property was not set, will seek
> KYLIN_HOME env variable
> 2018-08-28 01:30:17,051 INFO [pool-14-thread-1] hbase.HBaseConnection:110 :
> Creating coprocessor thread pool with max of 2048, core of 2048
> 2018-08-28 01:30:17,094 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> gtrecord.SequentialCubeTupleIterator:73 : Using SortedIteratorMergerWithLimit
> to merge segment results
> 2018-08-28 01:30:17,097 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> enumerator.OLAPEnumerator:117 : return TupleIterator...
> 2018-08-28 01:30:21,607 INFO [kylin-coproc--pool9-t1]
> client.RpcRetryingCallerImpl:134 : Call exception, tries=6, retries=6,
> started=4410 ms ago, cancelled=false, msg=java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:468)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:253)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
> at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8032)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2426)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2408)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42010)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> ... 3 more
> , details=row '' on table 'KYLIN_5Q088VO5I0' at
> region=KYLIN_5Q088VO5I0,,1535417272444.27b82cb4702db4557a98b9a7e60b7692.,
> hostname=ignite03.com,16020,1534313612401, seqNum=2
> 2018-08-28 01:30:25,633 INFO [kylin-coproc--pool9-t1]
> client.RpcRetryingCaller
[jira] [Commented] (KYLIN-3537) Use Spark to build Cube on Yarn faild at Setp8 on HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938648#comment-16938648 ] ASF subversion and git services commented on KYLIN-3537: Commit a2d1d047c8ecc401f8e2678f040d9ca8cd212456 in kylin's branch refs/heads/2.6.x-hadoop3.1 from alexandr.sidorchuk [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a2d1d04 ] KYLIN-3537 Use Spark to build Cube on Yarn faild at Setp8 on HDP3.0 add HBase dependicies to avoid java.lang.NoClassDefFoundError Signed-off-by: shaofengshi > Use Spark to build Cube on Yarn faild at Setp8 on HDP3.0 > > > Key: KYLIN-3537 > URL: https://issues.apache.org/jira/browse/KYLIN-3537 > Project: Kylin > Issue Type: Bug > Environment: HDP3.0 >Reporter: Lijun Cao >Assignee: Alexander >Priority: Major > Fix For: v2.6.1 > > Attachments: KYLIN-3537.master.001.patch > > > *The log on Yarn:* > 18/09/05 03:34:00 INFO scheduler.DAGScheduler: Job 0 failed: > saveAsNewAPIHadoopDataset at SparkCubeHFile.java:227, took 13.734642 s > 18/09/05 03:34:00 ERROR yarn.ApplicationMaster: User class threw exception: > java.lang.RuntimeException: error execute > org.apache.kylin.storage.hbase.steps.SparkCubeHFile > java.lang.RuntimeException: error execute > org.apache.kylin.storage.hbase.steps.SparkCubeHFile > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) > at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:636) > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in > stage 1.0 (TID 14, ignite02.com, executor 2): java.lang.NoClassDefFoundError: > org/apache/hadoop/hbase/io/hfile/HFileWriterImpl > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.createRecordWriter(HFileOutputFormat2.java:209) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.getRecordWriter(HFileOutputFormat2.java:181) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1119) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Driver stacktrace: > at > org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802) > at scala.Option.foreach(Option.scala:257) > at > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594) > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) > at > org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1928) > at org.a
[jira] [Commented] (KYLIN-2565) Support Hadoop 3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938645#comment-16938645 ] ASF subversion and git services commented on KYLIN-2565: Commit 64fd8083e61bfc2a9a9b15e8548dee4617f176dc in kylin's branch refs/heads/2.6.x-hadoop3.1 from Cheng Wang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=64fd808 ] KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT KYLIN-3518 Fix Coprocessor NPE problem on hbase 2 Signed-off-by: shaofengshi KYLIN-2565 Add cdh60 profile Signed-off-by: shaofengshi > Support Hadoop 3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: Wang Cheng >Assignee: Shao Feng Shi >Priority: Major > Fix For: v2.5.0 > > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4158) Query failed for GroupBy an expression of column with limit in SQL
[
https://issues.apache.org/jira/browse/KYLIN-4158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938642#comment-16938642
]
ASF subversion and git services commented on KYLIN-4158:
Commit 1a733c997341811e9dd93cd89d2af01e7f2a5f6d in kylin's branch
refs/heads/2.6.x-hadoop3.1 from ZhengshuaiPENG
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=1a733c9 ]
#KYLIN-4158, limit not push down to storage scan level if group by is an
expression
> Query failed for GroupBy an expression of column with limit in SQL
> --
>
> Key: KYLIN-4158
> URL: https://issues.apache.org/jira/browse/KYLIN-4158
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.6.2
>Reporter: PENG Zhengshuai
>Assignee: PENG Zhengshuai
>Priority: Major
> Fix For: v3.0.0-beta, v2.6.4
>
>
> When executing a sql query, the limit should not be pushdown to storage scan
> level if group by clause is an expression of columns.
> For example:
> {code:java}
> SELECT (CASE WHEN ("TEST_KYLIN_FACT"."LSTG_FORMAT_NAME" IN ('Auction',
> 'FP-GTC')) THEN 'Auction' ELSE "TEST_KYLIN_FACT"."LSTG_FORMAT_NAME" END) AS
> "LSTG_FORMAT_NAME__group_",
> SUM("TEST_KYLIN_FACT"."PRICE") AS "sum_PRICE_ok"
> FROM "TEST_KYLIN_FACT" "TEST_KYLIN_FACT"
> GROUP BY (CASE WHEN ("TEST_KYLIN_FACT"."LSTG_FORMAT_NAME" IN ('Auction',
> 'FP-GTC')) THEN 'Auction' ELSE "TEST_KYLIN_FACT"."LSTG_FORMAT_NAME" END)
> limit 4
> {code}
> In this SQL, group by clause is an expression, the limit should NOT be
> push-down to storage scan level, otherwise, the result is incorrect
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-3628) Query with lookup table always use latest snapshot
[ https://issues.apache.org/jira/browse/KYLIN-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938644#comment-16938644 ] ASF subversion and git services commented on KYLIN-3628: Commit c6cfbff3d17687a348f14d612aa5dd119f97ecd0 in kylin's branch refs/heads/2.6.x-hadoop3.1 from XiaoxiangYu [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=c6cfbff ] KYLIN-3628 Fix regression bug Previous code change will cause NPE for "SELECT * from DIMENSION_TABLE" when dimension table name use alias. > Query with lookup table always use latest snapshot > -- > > Key: KYLIN-3628 > URL: https://issues.apache.org/jira/browse/KYLIN-3628 > Project: Kylin > Issue Type: Improvement >Reporter: Na Zhai >Assignee: Na Zhai >Priority: Major > Fix For: v3.0.0-alpha2, v2.6.4 > > > If user queries a lookup table, Kylin will randomly selects a Cube (which has > the snapshot of this lookup table) to answer it. This causes uncertainty when > there are multiple cubes (share the same lookup): some cubes are newly built, > some not. If Kylin picks an old cube, the query result is old. > To remove this uncertainty, for such queries, either always use latest > snapshot, or use earlist snapshot. We believe the "latest" version is better. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-2565) Support Hadoop 3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938647#comment-16938647 ] ASF subversion and git services commented on KYLIN-2565: Commit 64fd8083e61bfc2a9a9b15e8548dee4617f176dc in kylin's branch refs/heads/2.6.x-hadoop3.1 from Cheng Wang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=64fd808 ] KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT KYLIN-3518 Fix Coprocessor NPE problem on hbase 2 Signed-off-by: shaofengshi KYLIN-2565 Add cdh60 profile Signed-off-by: shaofengshi > Support Hadoop 3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: Wang Cheng >Assignee: Shao Feng Shi >Priority: Major > Fix For: v2.5.0 > > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4103) Make the user string in granting operation of project is case insensitive
[ https://issues.apache.org/jira/browse/KYLIN-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938643#comment-16938643 ] ASF subversion and git services commented on KYLIN-4103: Commit 8698f535fc907425e8d2bc9b0769262e27086f42 in kylin's branch refs/heads/2.6.x-hadoop3.1 from Liu Shaohui [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8698f53 ] KYLIN-4103: Make the user string in granting operation of project is case insensitive > Make the user string in granting operation of project is case insensitive > - > > Key: KYLIN-4103 > URL: https://issues.apache.org/jira/browse/KYLIN-4103 > Project: Kylin > Issue Type: Bug >Reporter: Shaohui Liu >Assignee: Shaohui Liu >Priority: Minor > Fix For: v3.0.0-beta, v2.6.4 > > > Currently the user name of login operation is case insensitive. User can > login in kylin with lower case string or upper case string. But it is not > granting operation. > If we use lower case string of user name in project granting operation, there > will be no exception but the user can not set the project. > The reason is that the sid in AccessService/AclService is not case insensitive -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4154) Metadata inconsistency between multi Kylin server caused by Broadcaster closing
[ https://issues.apache.org/jira/browse/KYLIN-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938640#comment-16938640 ] ASF subversion and git services commented on KYLIN-4154: Commit 80834373edee373d25f87d53e4a2530c34c20a4b in kylin's branch refs/heads/2.6.x-hadoop3.1 from chao long [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8083437 ] KYLIN-4154 Sync all events in broadcaster before closing > Metadata inconsistency between multi Kylin server caused by Broadcaster > closing > > > Key: KYLIN-4154 > URL: https://issues.apache.org/jira/browse/KYLIN-4154 > Project: Kylin > Issue Type: Bug > Components: Metadata >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0-beta > > > To avoid Broadcaster memory leak, KYLIN-4131 close the metadata sync thread > after receiving "Sync All" event, but there may be some events in the waiting > queue, which haven't been synced yet. So we should sync all events before > closing sync thread. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-2565) Support Hadoop 3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938649#comment-16938649 ] ASF subversion and git services commented on KYLIN-2565: Commit bf8aa795f4d896eeac02c8bcd09d4ce8d18d3007 in kylin's branch refs/heads/2.6.x-hadoop3.1 from Chunen Ni [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=bf8aa79 ] KYLIN-2565 Fix use of HBaseAdmin > Support Hadoop 3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: Wang Cheng >Assignee: Shao Feng Shi >Priority: Major > Fix For: v2.5.0 > > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4153) Failed to read big resource /dict/xxxx at "Build Dimension Dictionary" Step
[
https://issues.apache.org/jira/browse/KYLIN-4153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938620#comment-16938620
]
ASF subversion and git services commented on KYLIN-4153:
Commit 9349e4adf50e20b79e3675dd798009039041defb in kylin's branch
refs/heads/2.6.x from XiaoxiangYu
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9349e4a ]
KYLIN-4153 Delete marker if real file not exists
> Failed to read big resource /dict/ at "Build Dimension Dictionary" Step
>
>
> Key: KYLIN-4153
> URL: https://issues.apache.org/jira/browse/KYLIN-4153
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.6.0
>Reporter: Xiaoxiang Yu
>Assignee: Xiaoxiang Yu
>Priority: Major
> Fix For: v3.0.0-beta, v2.6.4
>
>
> At the version of *Kylin 2.6.0*, kylin team has introduce an important
> refactor of Kylin's Metadata Store, which add a lot of enhancement such as
> upload/download metadata concurrently, store metadata with JDBC etc. Please
> refer to https://issues.apache.org/jira/browse/KYLIN-3671 for detail.
>
> When kylin want to save a *big resource*(such as dict or snapshot) into
> metadata store, it won't store it into metadata store(HBase or RDBMS)
> directly. Instead, kylin will first {color:red}save it into HDFS(Step
> 1){color}, and then {color:red}write a empty byte array as marker into
> metadata store(Step 2) {color}. If first action succeed and second action
> failed, a rollback method will be called to revert modification for HDFS
> files. We could regard it as a complete and atomic transaction.
>
> {color:#0747A6}Here is part of the source code added in KYLIN-3671.{color}
> Check it at
> https://github.com/apache/kylin/blob/8737bc1f555a2789a67462c8f8420b6ab3be97ce/core-common/src/main/java/org/apache/kylin/common/persistence/PushdownResourceStore.java#L58
> .
> {code:java}
> final void putBigResource(String resPath, ContentWriter content, long newTS)
> throws IOException {
> // pushdown the big resource to DFS file
> RollbackablePushdown pushdown = writePushdown(resPath, content); // Step
> 1: write big resource into HDFS
> try {
> // write a marker in resource store, to indicate the resource is now
> available
> logger.debug("Writing marker for big resource {}", resPath);
> putResourceWithRetry(resPath,
> ContentWriter.create(BytesUtil.EMPTY_BYTE_ARRAY), newTS); // Step 2: write
> marker into HBase/RDBMS
> } catch (Throwable ex) {
> pushdown.rollback();
> throw ex;
> } finally {
> pushdown.close();
> }
> }
> {code}
>
>
>
> But in some case, both step 1 and step 2 succeed but an exception still
> throwed in step 2,{color:red} the rollback won't clear marker written in Step
> 2{color}, which break the atomicity of this put action, thus cause the
> FileNotFoundException when Kylin want to read that dict later.
>
>
>
> {color:#0747A6}Here is part of reporter's kylin.log of incomplete rollback
> action.{color}
>
>
> {noformat}
> 2019-08-29 05:13:51,237 INFO [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] dict.DictionaryManager:388 : Saving
> dictionary at
> /dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict
> 2019-08-29 05:13:51,238 DEBUG [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] persistence.HDFSResourceStore:98 :
> Writing pushdown file
> /kylin/kylin_metadata/resources/dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict.temp.-1798610090
> 2019-08-29 05:13:51,256 DEBUG [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] persistence.HDFSResourceStore:117 :
> Move
> /kylin/kylin_metadata/resources/dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict.temp.-1798610090
> to
> /kylin/kylin_metadata/resources/dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict
> 2019-08-29 05:13:51,258 DEBUG [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] persistence.HDFSResourceStore:65 :
> Writing marker for big resource
> /dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict
> 2019-08-29 05:13:56,263 WARN
> [hconnection-0x56f3258e-shared--pool10944-t54867] client.AsyncProcess:1263 :
> #10545, table=kylin_metadata, attempt=1/1 failed=1ops, last exception:
> java.io.IOException: Call to tx-dn41.data/10.14.243.51:60020 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2662317,
> waitTime=5001, operationTimeout=5000 expired. on
> tx-dn41.data,60020,1565943919204, tracking started Thu Aug 29 05:1
[jira] [Commented] (KYLIN-4158) Query failed for GroupBy an expression of column with limit in SQL
[
https://issues.apache.org/jira/browse/KYLIN-4158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938624#comment-16938624
]
ASF subversion and git services commented on KYLIN-4158:
Commit 1a733c997341811e9dd93cd89d2af01e7f2a5f6d in kylin's branch
refs/heads/2.6.x from ZhengshuaiPENG
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=1a733c9 ]
#KYLIN-4158, limit not push down to storage scan level if group by is an
expression
> Query failed for GroupBy an expression of column with limit in SQL
> --
>
> Key: KYLIN-4158
> URL: https://issues.apache.org/jira/browse/KYLIN-4158
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.6.2
>Reporter: PENG Zhengshuai
>Assignee: PENG Zhengshuai
>Priority: Major
> Fix For: v3.0.0-beta, v2.6.4
>
>
> When executing a sql query, the limit should not be pushdown to storage scan
> level if group by clause is an expression of columns.
> For example:
> {code:java}
> SELECT (CASE WHEN ("TEST_KYLIN_FACT"."LSTG_FORMAT_NAME" IN ('Auction',
> 'FP-GTC')) THEN 'Auction' ELSE "TEST_KYLIN_FACT"."LSTG_FORMAT_NAME" END) AS
> "LSTG_FORMAT_NAME__group_",
> SUM("TEST_KYLIN_FACT"."PRICE") AS "sum_PRICE_ok"
> FROM "TEST_KYLIN_FACT" "TEST_KYLIN_FACT"
> GROUP BY (CASE WHEN ("TEST_KYLIN_FACT"."LSTG_FORMAT_NAME" IN ('Auction',
> 'FP-GTC')) THEN 'Auction' ELSE "TEST_KYLIN_FACT"."LSTG_FORMAT_NAME" END)
> limit 4
> {code}
> In this SQL, group by clause is an expression, the limit should NOT be
> push-down to storage scan level, otherwise, the result is incorrect
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4154) Metadata inconsistency between multi Kylin server caused by Broadcaster closing
[ https://issues.apache.org/jira/browse/KYLIN-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938623#comment-16938623 ] ASF subversion and git services commented on KYLIN-4154: Commit 80834373edee373d25f87d53e4a2530c34c20a4b in kylin's branch refs/heads/2.6.x from chao long [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8083437 ] KYLIN-4154 Sync all events in broadcaster before closing > Metadata inconsistency between multi Kylin server caused by Broadcaster > closing > > > Key: KYLIN-4154 > URL: https://issues.apache.org/jira/browse/KYLIN-4154 > Project: Kylin > Issue Type: Bug > Components: Metadata >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0-beta > > > To avoid Broadcaster memory leak, KYLIN-4131 close the metadata sync thread > after receiving "Sync All" event, but there may be some events in the waiting > queue, which haven't been synced yet. So we should sync all events before > closing sync thread. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4131) Broadcaster memory leak
[
https://issues.apache.org/jira/browse/KYLIN-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938618#comment-16938618
]
ASF subversion and git services commented on KYLIN-4131:
Commit 8bd6bedbc26da9d6acd9bf2232688bfaf8804e92 in kylin's branch
refs/heads/2.6.x from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8bd6bed ]
KYLIN-4131 Fix Broadcaster memory leak
> Broadcaster memory leak
> ---
>
> Key: KYLIN-4131
> URL: https://issues.apache.org/jira/browse/KYLIN-4131
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Critical
> Fix For: v3.0.0, v2.6.4
>
> Attachments: image-2019-08-12-16-39-46-262.png,
> image-2019-08-12-16-39-52-327.png, image-2019-08-12-17-16-05-532.png,
> image-2019-08-12-17-22-15-783.png, image-2019-08-12-17-22-23-477.png
>
>
> h3. *Test evidence*
> Use MAT(Memory Analyzer) to analyze this issue.
> # start Kylin, dump a jvm heap info snapshot and import into MAT. We can
> see there is only one Broadcaster and ProjectManager object.
> !image-2019-08-12-17-22-15-783.png|width=721,height=520!
> # Call 'list cube' api and 'reload metadata' api many times. We can see
> there are many BroadCaster and Manager objects in the memory, which should be
> recycled by GC, but they are not.
> !image-2019-08-12-17-22-23-477.png|width=724,height=498!
>
> *Root cause*
> Broadcaster hold the ref of Listener object defined in Manager, and the
> Listener object hold the ref of Manager, because the impl class of Listener
> is an inner class of Manager.
> Broadcaster.java
> {code:java}
> private Map> listenerMap = Maps.newConcurrentMap();
> {code}
> ProjectManager.java -> ProjectSyncListener
> {code:java}
> private class ProjectSyncListener extends Broadcaster.Listener {
> @Override
> public void onEntityChange(Broadcaster broadcaster, String entity, Event
> event, String cacheKey)
> throws IOException {
> String project = cacheKey;
> if (event == Event.DROP) {
> removeProjectLocal(project);
> return;
> }
> reloadProjectQuietly(project);
> broadcaster.notifyProjectSchemaUpdate(project);
> broadcaster.notifyProjectDataUpdate(project);
> }
> }
> {code}
> And Broadcaster can't be recycled by GC because it holds a running thread,
> which cause the Manager object also not being recycled.
> *After Fix*
> The Broadcaster objects and Manager objects are recycled by GC.
> !image-2019-08-12-17-16-05-532.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4055) cube quey and ad-hoc query return different meta info
[ https://issues.apache.org/jira/browse/KYLIN-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938621#comment-16938621 ] ASF subversion and git services commented on KYLIN-4055: Commit 32ad065b56d3ff2b91a0de585b21480b19d1f853 in kylin's branch refs/heads/2.6.x from javalife0312 [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=32ad065 ] KYLIN-4055 cube quey and ad-hoc query return different meta info > cube quey and ad-hoc query return different meta info > - > > Key: KYLIN-4055 > URL: https://issues.apache.org/jira/browse/KYLIN-4055 > Project: Kylin > Issue Type: Improvement >Reporter: jinguowei >Assignee: Yaqian Zhang >Priority: Major > Fix For: v3.0.0-beta, v2.6.4 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4157) When using PrepareStatement query, functions within WHERE will cause InternalErrorException
[ https://issues.apache.org/jira/browse/KYLIN-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938622#comment-16938622 ] ASF subversion and git services commented on KYLIN-4157: Commit 699158af217821ea902c3db04924241b3521deb7 in kylin's branch refs/heads/2.6.x from mawu2 [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=699158a ] KYLIN-4157 When using PrepareStatement query, functions within WHERE will cause InternalErrorException > When using PrepareStatement query, functions within WHERE will cause > InternalErrorException > --- > > Key: KYLIN-4157 > URL: https://issues.apache.org/jira/browse/KYLIN-4157 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.6.3 >Reporter: Marc Wu >Assignee: Marc Wu >Priority: Major > Fix For: v3.0.0-beta, v2.6.4 > > Attachments: image-2019-09-04-15-39-52-867.png, > image-2019-09-04-15-39-58-276.png, image-2019-09-04-20-57-46-121.png > > > Hi Kylin Team: > I found a bug when I'm using PreparedStatement query. > Let me use table KYLIN_SALES to explain the scenario. > There is a SQL like: > select LSTG_FORMAT_NAME, sum(PRICE) from KYLIN_SALES where > lower(LSTG_FORMAT_NAME) = 'fp-gtc' group by LSTG_FORMAT_NAME > In some cases, user doesn't know the LSTG_FORMAT_NAME is upper case or lower > case, or they just want to query data ignoring cases. > So assume they use lower(LSTG_FORMAT_NAME) = 'fp-gtc', it's a function > within the filter. > When I execute this SQL on Kylin web console, it can get the right result, > but when I tried to execute it by PreparedStatement query on Postman, it > threw InternalErrorException. !image-2019-09-04-15-39-58-276.png! > > !image-2019-09-04-15-39-52-867.png! > > > h2. Root Cause > I debugged Kylin to find the root cause, it turns out when the filter > contains function, compFilter.getColumn is null, and the > compFilter.getFunction contains lower(LSTG_FORMAT_NAME), so that > compFilter.getColumn().getType().isDateTimeFamily() will throw > NullPointerException, that's the root cause for this 500 error. > > h2. !image-2019-09-04-20-57-46-121.png! Solution > I suggest to check nullable first before using compFilter.getColumn. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4103) Make the user string in granting operation of project is case insensitive
[ https://issues.apache.org/jira/browse/KYLIN-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938625#comment-16938625 ] ASF subversion and git services commented on KYLIN-4103: Commit 8698f535fc907425e8d2bc9b0769262e27086f42 in kylin's branch refs/heads/2.6.x from Liu Shaohui [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8698f53 ] KYLIN-4103: Make the user string in granting operation of project is case insensitive > Make the user string in granting operation of project is case insensitive > - > > Key: KYLIN-4103 > URL: https://issues.apache.org/jira/browse/KYLIN-4103 > Project: Kylin > Issue Type: Bug >Reporter: Shaohui Liu >Assignee: Shaohui Liu >Priority: Minor > Fix For: v3.0.0-beta, v2.6.4 > > > Currently the user name of login operation is case insensitive. User can > login in kylin with lower case string or upper case string. But it is not > granting operation. > If we use lower case string of user name in project granting operation, there > will be no exception but the user can not set the project. > The reason is that the sid in AccessService/AclService is not case insensitive -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4034) The table should not display in Insight page when the user has no access to the table
[ https://issues.apache.org/jira/browse/KYLIN-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938619#comment-16938619 ] ASF subversion and git services commented on KYLIN-4034: Commit 560deef12ea6173644e8250958e5f17d0ee0fcab in kylin's branch refs/heads/2.6.x from yaqian.zhang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=560deef ] KYLIN-4034 Table not in Insight when user have no access to the table > The table should not display in Insight page when the user has no access to > the table > - > > Key: KYLIN-4034 > URL: https://issues.apache.org/jira/browse/KYLIN-4034 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.2 >Reporter: Na Zhai >Assignee: Yaqian Zhang >Priority: Major > Fix For: v3.0.0-beta, v2.6.4 > > Attachments: 屏幕快照 2019-09-18 下午11.09.13.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-3628) Query with lookup table always use latest snapshot
[ https://issues.apache.org/jira/browse/KYLIN-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938626#comment-16938626 ] ASF subversion and git services commented on KYLIN-3628: Commit c6cfbff3d17687a348f14d612aa5dd119f97ecd0 in kylin's branch refs/heads/2.6.x from XiaoxiangYu [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=c6cfbff ] KYLIN-3628 Fix regression bug Previous code change will cause NPE for "SELECT * from DIMENSION_TABLE" when dimension table name use alias. > Query with lookup table always use latest snapshot > -- > > Key: KYLIN-3628 > URL: https://issues.apache.org/jira/browse/KYLIN-3628 > Project: Kylin > Issue Type: Improvement >Reporter: Na Zhai >Assignee: Na Zhai >Priority: Major > Fix For: v3.0.0-alpha2, v2.6.4 > > > If user queries a lookup table, Kylin will randomly selects a Cube (which has > the snapshot of this lookup table) to answer it. This causes uncertainty when > there are multiple cubes (share the same lookup): some cubes are newly built, > some not. If Kylin picks an old cube, the query result is old. > To remove this uncertainty, for such queries, either always use latest > snapshot, or use earlist snapshot. We believe the "latest" version is better. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4093) Slow query pages should be open to all users of the project
[ https://issues.apache.org/jira/browse/KYLIN-4093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4093. - Resolution: Fixed > Slow query pages should be open to all users of the project > --- > > Key: KYLIN-4093 > URL: https://issues.apache.org/jira/browse/KYLIN-4093 > Project: Kylin > Issue Type: Improvement >Reporter: Shaohui Liu >Priority: Major > Fix For: v3.0.0-beta, v2.6.4 > > > Currently the show query page only can been seen for kylin admins. It's very > useful for modlers and analysts for this project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4093) Slow query pages should be open to all users of the project
[ https://issues.apache.org/jira/browse/KYLIN-4093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4093: Fix Version/s: v3.0.0-beta > Slow query pages should be open to all users of the project > --- > > Key: KYLIN-4093 > URL: https://issues.apache.org/jira/browse/KYLIN-4093 > Project: Kylin > Issue Type: Improvement >Reporter: Shaohui Liu >Priority: Major > Fix For: v3.0.0-beta, v2.6.4 > > > Currently the show query page only can been seen for kylin admins. It's very > useful for modlers and analysts for this project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4121) Cleanup hive view intermediate tables after job be finished
[ https://issues.apache.org/jira/browse/KYLIN-4121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4121: Fix Version/s: v2.6.4 > Cleanup hive view intermediate tables after job be finished > --- > > Key: KYLIN-4121 > URL: https://issues.apache.org/jira/browse/KYLIN-4121 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shao Feng Shi >Assignee: wangrupeng >Priority: Major > Fix For: v3.0.0-beta, v2.6.4 > > Attachments: image-2019-09-16-19-18-59-663.png, > image-2019-09-16-19-26-31-446.png > > > Reported by community user: > I have a cube with a fact table join a lookup table in hive, and both are > hive view. I submit a job one time per hour. > > Kylin can drop the intermediate fact table , but doesn't drop the > intermediate lookup table . > > I check the source code , and find out that , at '13 step: Hive Cleanup' , > the source code has been commented. > > > It is a legacy issue. Now KYLIN-3515 has fixed that, so the cleanup can be > enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4057) autoMerge job can not stop
[ https://issues.apache.org/jira/browse/KYLIN-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4057: Fix Version/s: v2.6.4 > autoMerge job can not stop > -- > > Key: KYLIN-4057 > URL: https://issues.apache.org/jira/browse/KYLIN-4057 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.2 >Reporter: chenchen >Assignee: wangrupeng >Priority: Major > Fix For: v3.0.0-beta, v2.6.4 > > Attachments: 5031561517158_.pic_hd.jpg > > > In this version，I used DistributedScheduler。 > In my case, A cube config autoMerge, But the job that automatically merges > segments is wrong ，I can't fix this error, I can only discard and drop the > job, but Kylin's monitoring mechanism creates the same new job. > I want to ask if there is any way to fix this problem without banning cube. > The error is reported as follows. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-1856) Kylin shows old error in job step output after resume - specifically in #4 Step Name: Build Dimension Dictionary
[ https://issues.apache.org/jira/browse/KYLIN-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-1856: Fix Version/s: v2.6.4 > Kylin shows old error in job step output after resume - specifically in #4 > Step Name: Build Dimension Dictionary > > > Key: KYLIN-1856 > URL: https://issues.apache.org/jira/browse/KYLIN-1856 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.5.2, v1.5.2.1 >Reporter: Richard Calaba >Assignee: Yaqian Zhang >Priority: Minor > Fix For: v3.0.0-beta, v2.6.4 > > > I have realized that if my job stops with error and I try to recover the > error and resume the job - then the latest step starts again from scratch. > This is fine but in my opinion the log of the Step should clear as well - now > it is showing the error from my previous attempt. > Specifically observed in #4 Step Name: Build Dimension Dictionary - but is > probbaly generic issue. > To correct this: clear the log of the Build Step after the job Step is > resumed. Already when the job step is restarted, not after it is completed. > (if Kylin fails i.e. for out of memory - it silently dies and analyzing the > step log shows wrong error (from previous run) - if it would be empty -> I > would know that most probable cause was that Kylin died) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4047) Use push-down query when division dynamic column cube query is not supported
[
https://issues.apache.org/jira/browse/KYLIN-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4047:
Fix Version/s: (was: v3.0.0-beta)
v3.0.0-alpha2
> Use push-down query when division dynamic column cube query is not supported
>
>
> Key: KYLIN-4047
> URL: https://issues.apache.org/jira/browse/KYLIN-4047
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: Future, v2.6.2, v3.0.0-alpha
>Reporter: wangxiaojing
>Priority: Minor
> Fix For: v3.0.0-alpha2, v2.6.4
>
>
> Hello all ,
> At present, if a table has created cube, the query with division
> dynamic column will throw an exception without pushdown query results, but if
> switching an project, pushdown query will be carried out, which will give
> users inconsistent experience. Can we also use pushdown query when query
> division, multiplication dynamic column throw exceptions ?
> Scene：
> SQL:select round (car_dismatch_cnt*100/answer_order_cnt, 4) from db.table1.
> 1、Executes the SQL on project where has create a cube has db.table1,it will
> throws the right side of the BinaryTuple Expressions columns is not supported
> for / exception and is not queried through pushdown;
> 2、When change a project where not create the table cube , the same query can
> be executed to push down and get the right result.
> 3. Expect that no matter whether a table cube is created or not, when the
> cube query is not success, the result can still be obtained by pushdown
> query, which will give users a consistent experience.
> {code:java}
> // exception
> 2019-06-06 19:04:55,902 ERROR [http-bio-8081-exec-1]
> controller.BasicController:63 :
> org.apache.kylin.rest.exception.InternalErrorException: That the right side
> of the BinaryTupleExpression owns columns is not supported for /
> while executing SQL: "select DT,ORD_AMT/ORD_AMT from db.table1 LIMIT 5"
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:452)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:373)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at
[jira] [Resolved] (KYLIN-4013) Only show the cubes under one model
[ https://issues.apache.org/jira/browse/KYLIN-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4013. - Resolution: Fixed > Only show the cubes under one model > --- > > Key: KYLIN-4013 > URL: https://issues.apache.org/jira/browse/KYLIN-4013 > Project: Kylin > Issue Type: Improvement > Components: Web >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Major > Fix For: v3.0.0-alpha2, v2.6.4 > > > Some improvement for UI。 > User may want to see the cubes under specified model. Add an extra action > 'Cubes' in the drop-down list of model to filter cube list -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4047) Use push-down query when division dynamic column cube query is not supported
[
https://issues.apache.org/jira/browse/KYLIN-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen resolved KYLIN-4047.
-
Resolution: Fixed
> Use push-down query when division dynamic column cube query is not supported
>
>
> Key: KYLIN-4047
> URL: https://issues.apache.org/jira/browse/KYLIN-4047
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: Future, v2.6.2, v3.0.0-alpha
>Reporter: wangxiaojing
>Priority: Minor
> Fix For: v3.0.0-beta, v2.6.4
>
>
> Hello all ,
> At present, if a table has created cube, the query with division
> dynamic column will throw an exception without pushdown query results, but if
> switching an project, pushdown query will be carried out, which will give
> users inconsistent experience. Can we also use pushdown query when query
> division, multiplication dynamic column throw exceptions ?
> Scene：
> SQL:select round (car_dismatch_cnt*100/answer_order_cnt, 4) from db.table1.
> 1、Executes the SQL on project where has create a cube has db.table1,it will
> throws the right side of the BinaryTuple Expressions columns is not supported
> for / exception and is not queried through pushdown;
> 2、When change a project where not create the table cube , the same query can
> be executed to push down and get the right result.
> 3. Expect that no matter whether a table cube is created or not, when the
> cube query is not success, the result can still be obtained by pushdown
> query, which will give users a consistent experience.
> {code:java}
> // exception
> 2019-06-06 19:04:55,902 ERROR [http-bio-8081-exec-1]
> controller.BasicController:63 :
> org.apache.kylin.rest.exception.InternalErrorException: That the right side
> of the BinaryTupleExpression owns columns is not supported for /
> while executing SQL: "select DT,ORD_AMT/ORD_AMT from db.table1 LIMIT 5"
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:452)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:373)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at
> org.springframework.security.web.FilterChainProxy$V
[jira] [Updated] (KYLIN-4047) Use push-down query when division dynamic column cube query is not supported
[
https://issues.apache.org/jira/browse/KYLIN-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4047:
Fix Version/s: v3.0.0-beta
> Use push-down query when division dynamic column cube query is not supported
>
>
> Key: KYLIN-4047
> URL: https://issues.apache.org/jira/browse/KYLIN-4047
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: Future, v2.6.2, v3.0.0-alpha
>Reporter: wangxiaojing
>Priority: Minor
> Fix For: v3.0.0-beta, v2.6.4
>
>
> Hello all ,
> At present, if a table has created cube, the query with division
> dynamic column will throw an exception without pushdown query results, but if
> switching an project, pushdown query will be carried out, which will give
> users inconsistent experience. Can we also use pushdown query when query
> division, multiplication dynamic column throw exceptions ?
> Scene：
> SQL:select round (car_dismatch_cnt*100/answer_order_cnt, 4) from db.table1.
> 1、Executes the SQL on project where has create a cube has db.table1,it will
> throws the right side of the BinaryTuple Expressions columns is not supported
> for / exception and is not queried through pushdown;
> 2、When change a project where not create the table cube , the same query can
> be executed to push down and get the right result.
> 3. Expect that no matter whether a table cube is created or not, when the
> cube query is not success, the result can still be obtained by pushdown
> query, which will give users a consistent experience.
> {code:java}
> // exception
> 2019-06-06 19:04:55,902 ERROR [http-bio-8081-exec-1]
> controller.BasicController:63 :
> org.apache.kylin.rest.exception.InternalErrorException: That the right side
> of the BinaryTupleExpression owns columns is not supported for /
> while executing SQL: "select DT,ORD_AMT/ORD_AMT from db.table1 LIMIT 5"
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:452)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:373)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at
> org.springframework.security.web.FilterChain
[jira] [Resolved] (KYLIN-4046) Refine JDBC Source(source.default=8)
[ https://issues.apache.org/jira/browse/KYLIN-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4046. - Resolution: Fixed > Refine JDBC Source(source.default=8) > > > Key: KYLIN-4046 > URL: https://issues.apache.org/jira/browse/KYLIN-4046 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Critical > Fix For: v3.0.0-alpha2, v2.6.4 > > > Currently, the function of ingest from RDBMS(kylin.source.default=8) to Kylin > has some problem such as: > # case-sensitive > # weak dialect compatibility > # mis-use quote character > We need to fix this problem in kylin.source.default=8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4046) Refine JDBC Source(source.default=8)
[ https://issues.apache.org/jira/browse/KYLIN-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4046: Fix Version/s: v2.6.4 > Refine JDBC Source(source.default=8) > > > Key: KYLIN-4046 > URL: https://issues.apache.org/jira/browse/KYLIN-4046 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Critical > Fix For: v3.0.0-alpha2, v2.6.4 > > > Currently, the function of ingest from RDBMS(kylin.source.default=8) to Kylin > has some problem such as: > # case-sensitive > # weak dialect compatibility > # mis-use quote character > We need to fix this problem in kylin.source.default=8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (KYLIN-4046) Refine JDBC Source(source.default=8)
[ https://issues.apache.org/jira/browse/KYLIN-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen reopened KYLIN-4046: - > Refine JDBC Source(source.default=8) > > > Key: KYLIN-4046 > URL: https://issues.apache.org/jira/browse/KYLIN-4046 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Critical > Fix For: v3.0.0-alpha2 > > > Currently, the function of ingest from RDBMS(kylin.source.default=8) to Kylin > has some problem such as: > # case-sensitive > # weak dialect compatibility > # mis-use quote character > We need to fix this problem in kylin.source.default=8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4167) Refactor streaming coordinator
[ https://issues.apache.org/jira/browse/KYLIN-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4167. - Resolution: Fixed > Refactor streaming coordinator > -- > > Key: KYLIN-4167 > URL: https://issues.apache.org/jira/browse/KYLIN-4167 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Major > Fix For: v3.0.0-beta > > > h2. Summary > # Currently, *coordinator* has too many responsibility, which violate single > responsibility principle, and it not easy for extension, a good separation of > responsibilities is a recommended way. > # Some cluster level operation has no atomicity guarantee, we should > implement then in idempotent way to achieve final consistency > # Resubmit when job was discarded > # Clarify overall design for realtime OLAP > > h4. StreamingCoordinator > Facade of coordinator, will controll BuildJobSummitter/ReceiverClusterMangaer > and delegate operation to them. > h4. BuildJobSubmitter > The main responsibility of BuildJobSubmitter including: > 1. Try to find candidate segment which ready to submit a build job > 2. Trace the status of candidate segment's build job and promote segment if > it is has met requirements > h4. > h4. ReceiverClusterManager > This class manage operation related to multi streaming receivers. They are > often not atomic and maybe idempotent. > h4. ClusterStateChecker > Basic step of this class: > 1. stop/pause coordinator to avoid underlying concurrency issue > 2. check inconsistent state of all receiver cluster > 3. send summary via mail to kylin admin > 4. if need, call ClusterDoctor to repair inconsistent issue > h4. ClusterDoctor > Repair inconsistent state according to result of ClusterStateChecker > > > h3. Candidate Segment > The candidate segments are those segments what can be saw/perceived by > streaming coordinator, > candidate segment could be divided into following state/queue: > 1. segment which data are uploaded *PARTLY* > 2. segment which data are uploaded completely and *WAITING* to build > 3. segment which in *BUILDING* state, job's state should be one of > (NEW/RUNNING/ERROR/DISCARD) > 4. segment which built *succeed* and wait to be delivered to historical part > (and to be deleted in realtime part) > 5. segment which *in historical part*(HBase Ready Segment) > > By design, segment should transfer to next queue in sequential way(shouldn't > jump the queue), do not break this. > h3. Atomicity > In a multi-step transcation, following acepts should be thought twice: > 1. should *fail fast* or continue when exception thrown. > 2. should API(remote call) be *synchronous* or asynchronous > 3. when transcation failed, could *roll back* always succeed > 4. transcation should be *idempotent* so when it failed, it could be fixed by > retry > > How to ensure whole cluster opreation smoothly without blocking problem. I > divided all multi-step transcation into three kinds: > NotAtomicIdempotent > NotAtomicAndNotIdempotent > NonSideEffect -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4010) Auto adjust offset according to query server's timezone for time derived column
[
https://issues.apache.org/jira/browse/KYLIN-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen resolved KYLIN-4010.
-
Resolution: Fixed
> Auto adjust offset according to query server's timezone for time derived
> column
> ---
>
> Key: KYLIN-4010
> URL: https://issues.apache.org/jira/browse/KYLIN-4010
> Project: Kylin
> Issue Type: Improvement
> Components: Others
>Affects Versions: v3.0.0-alpha
>Reporter: zengrui
>Assignee: Xiaoxiang Yu
>Priority: Minor
> Fix For: v3.0.0-beta
>
> Attachments: image-2019-07-15-17-15-31-209.png,
> image-2019-07-15-17-17-04-029.png, image-2019-07-15-17-17-39-568.png,
> image-2019-09-22-16-35-23-663.png, image-2019-09-22-16-43-19-248.png,
> image-2019-09-22-20-28-02-402.png, image-2019-09-22-20-35-19-038.png,
> image-2019-09-22-20-53-34-720.png, image-2019-09-24-17-23-42-451.png,
> image-2019-09-24-17-33-29-824.png, image-2019-09-24-17-35-47-589.png,
> image-2019-09-24-17-36-39-611.png
>
>
> h2. Backgroud
> In realtime OLAP, we index real-time event in streaming receiver. We know
> that each event must contains a timestamp column (we often call it event
> time), that value should represent when this event was produced. Because
> event maybe come from different timezone and use local timezone is always
> *error-prone*, so we recommend to use a {color:#DE350B}GMT+0{color}
> timestamp(System.currentTimeMillis()) to avoid such issue.
> I think this is good by design, it is easy to understand and always correct.
> But the *side effect* is that, the end user(business manager behind a BI
> tools) are unhappy because he have to use GMT+0 with date/time related filter
> in SQL and should understand the result should be *shifted* with his local
> timezone. It is not user-firendly and inconvenient for normal user. Because
> user may compare query result from different data source and compare them and
> summarize, use GMT+0 may trouble them.
> h2. Example
> For example, kylin user work in *GMT+8* (maybe in Shanghai) want to know some
> metrics which occured from {color:#DE350B}2019-09-01 12:00:00{color} to
> {color:#DE350B}2019-09-01 14:00:00{color} in his {color:#DE350B}local
> timezone{color}, so he has to {color:#DE350B}rewrite{color} his query (with
> eight hour offset) to following:
> {code:sql}
> select hour_start, count(*)
> from realtime_table
> where hour_start >= "2019-09-01 04:00:00" and hour_start < "2019-09-01
> 06:00:00"
> group by hour_start
> {code}
> And he will get result like :
> ||hour_start ||count||
> |2019-09-01 04:00:00 |139202|
> |2019-09-01 05:00:00 |89398|
> And he must convert to a more meaningful result in his mind, it is realy
> annoying!
> ||hour_start ||count||
> |2019-09-01 12:00:00 |139202|
> |2019-09-01 13:00:00 |89398|
> h2. Desgin
> We should not change the way receiver index event, event time should be
> stored in UTC timestamp. We should auto rewrite sql's event time related
> filter.
> In kylin, filter condition in where clause will be convert to a
> *TupleFilter*, and it looks like *RelNode* in Apache Calicate.
> For where hour_start >= "2019-09-01 12:00:00" and hour_start < "2019-09-01
> 14:00:00", we will send TupleFilter to streaming receiver or region server
> which looks like this:
> {noformat}
> AND
> GreatThanOrEqual
> hout_start
> CAST
> "2019-09-01 12:00:00"
> timestamp
> LessThanOrEqual
> hout_start
> CAST
> "2019-09-01 14:00:00"
> timestamp
> {noformat}
> But for streaming query, we want to change each ConstantTupleFilter and minus
> value for that timestamp. So the TupleFilter which be sent will be following:
> {noformat}
> AND
> GreatThanOrEqual
> hout_start
> CAST
> "2019-09-01 04:00:00"
> timestamp
> LessThanOrEqual
> hout_start
> CAST
> "2019-09-01 06:00:00"
> timestamp
> {noformat}
> Before query result processed by *OLAPEnumerator*, kylin will plus each
> value of time derived column, thus protect row from be filtered by calcite
> generated code.
> So, user will get what he want in his timezone without any burden.
> h2. How to use
> To enable auto shift by time zone, please set
> {color:#DE350B}kylin.stream.auto.just.by.timezone{color} to true.
> You can specific time zone by {color:#DE350B}kylin.web.timezone{color},
> otherwise, time zone will be auto detected.
> Only *time derived column* will be affected.
> h2. Related Issue
> Originally, the event time can only in the format of a long value (UTC
> timestamp). But in some case, the event time is in a format of "-MM-dd
> HH:mm:ss", we use a new class DateTimeParser(introduced in KYLIN-4001) to
> convert such format into a UTC timestamp.
> h3. Old Describe
> In Real-Time Stre
[GitHub] [kylin] coveralls edited a comment on issue #860: build dimension dictionary using spark
coveralls edited a comment on issue #860: build dimension dictionary using spark URL: https://github.com/apache/kylin/pull/860#issuecomment-535306368 ## Pull Request Test Coverage Report for [Build 5062](https://coveralls.io/builds/25943400) * **0** of **268** **(0.0%)** changed or added relevant lines in **5** files are covered. * **2** unchanged lines in **2** files lost coverage. * Overall coverage decreased (**-0.09%**) to **27.827%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/25943400/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L1458) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/model/TblColRef.java](https://coveralls.io/builds/25943400/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FTblColRef.java#L284) | 0 | 1 | 0.0% | [engine-mr/src/main/java/org/apache/kylin/engine/mr/JobBuilderSupport.java](https://coveralls.io/builds/25943400/source?filename=engine-mr%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fmr%2FJobBuilderSupport.java#L255) | 0 | 1 | 0.0% | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java](https://coveralls.io/builds/25943400/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkBatchCubingJobBuilder2.java#L80) | 0 | 18 | 0.0% | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBuildDictionary.java](https://coveralls.io/builds/25943400/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkBuildDictionary.java#L86) | 0 | 247 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/25943400/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% | | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java](https://coveralls.io/builds/25943400/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkBatchCubingJobBuilder2.java#L86) | 1 | 0.0% | | Totals | [](https://coveralls.io/builds/25943400) | | :-- | --: | | Change from base [Build 5053](https://coveralls.io/builds/25913474): | -0.09% | | Covered Lines: | 23704 | | Relevant Lines: | 85183 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] coveralls commented on issue #861: KYLIN-4180 Prevent abnormal CPU usage by limiting flat filters length
coveralls commented on issue #861: KYLIN-4180 Prevent abnormal CPU usage by limiting flat filters length URL: https://github.com/apache/kylin/pull/861#issuecomment-535420207 ## Pull Request Test Coverage Report for [Build 5060](https://coveralls.io/builds/25943151) * **6** of **6** **(100.0%)** changed or added relevant lines in **1** file are covered. * **7** unchanged lines in **2** files lost coverage. * Overall coverage decreased (**-0.001%**) to **27.915%** --- | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/25943151/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/25943151/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L439) | 5 | 77.81% | | Totals | [](https://coveralls.io/builds/25943151) | | :-- | --: | | Change from base [Build 5053](https://coveralls.io/builds/25913474): | -0.001% | | Covered Lines: | 23705 | | Relevant Lines: | 84919 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4098) Add cube auto merge api
[ https://issues.apache.org/jira/browse/KYLIN-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938431#comment-16938431 ] nichunen commented on KYLIN-4098: - The PR will introduce the problem described in https://issues.apache.org/jira/browse/KYLIN-3718, re-open this issue to fix in 3.0ga release > Add cube auto merge api > --- > > Key: KYLIN-4098 > URL: https://issues.apache.org/jira/browse/KYLIN-4098 > Project: Kylin > Issue Type: New Feature >Reporter: Shaohui Liu >Assignee: Shaohui Liu >Priority: Minor > Fix For: v3.0.0 > > Attachments: image-2019-09-19-17-11-49-733.png > > > Currently the auto merging of cube is triggered by the event of new segment > is ready automatically. When the cluster restart, there may be too many > merging job. > It's better to have a rest api to trigger the merging and make it more > controllable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] codecov-io commented on issue #861: KYLIN-4180 Prevent abnormal CPU usage by limiting flat filters length
codecov-io commented on issue #861: KYLIN-4180 Prevent abnormal CPU usage by limiting flat filters length URL: https://github.com/apache/kylin/pull/861#issuecomment-535417736 # [Codecov](https://codecov.io/gh/apache/kylin/pull/861?src=pr&el=h1) Report > Merging [#861](https://codecov.io/gh/apache/kylin/pull/861?src=pr&el=desc) into [master](https://codecov.io/gh/apache/kylin/commit/75c90df928d921808201b172f14c42cc91eeae02?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/kylin/pull/861?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #861 +/- ## - Coverage 25.46% 25.46% -0.01% - Complexity 6108 6110 +2 Files 1410 1410 Lines 8491684919 +3 Branches 1189611897 +1 Hits 2162321623 - Misses6121461217 +3 Partials 2079 2079 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/861?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../org/apache/kylin/metadata/filter/TupleFilter.java](https://codecov.io/gh/apache/kylin/pull/861/diff?src=pr&el=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvZmlsdGVyL1R1cGxlRmlsdGVyLmphdmE=) | `72.9% <100%> (+0.4%)` | `48 <0> (+1)` | :arrow_up: | | [...he/kylin/job/impl/threadpool/DefaultScheduler.java](https://codecov.io/gh/apache/kylin/pull/861/diff?src=pr&el=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9pbXBsL3RocmVhZHBvb2wvRGVmYXVsdFNjaGVkdWxlci5qYXZh) | `74.41% <0%> (-2.33%)` | `12% <0%> (ø)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/861/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `69.3% <0%> (-0.92%)` | `7% <0%> (ø)` | | | [...org/apache/kylin/rest/util/QueryRequestLimits.java](https://codecov.io/gh/apache/kylin/pull/861/diff?src=pr&el=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3QvdXRpbC9RdWVyeVJlcXVlc3RMaW1pdHMuamF2YQ==) | `40.47% <0%> (+4.76%)` | `6% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/861?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/861?src=pr&el=footer). Last update [75c90df...20b647f](https://codecov.io/gh/apache/kylin/pull/861?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Reopened] (KYLIN-3797) Too many or filters may break Kylin server when flatting filter
[ https://issues.apache.org/jira/browse/KYLIN-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen reopened KYLIN-3797: - > Too many or filters may break Kylin server when flatting filter > --- > > Key: KYLIN-3797 > URL: https://issues.apache.org/jira/browse/KYLIN-3797 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Gang Ma >Assignee: Gang Ma >Priority: Major > Fix For: v3.0.0-alpha > > > Kylin needs to convert filter into a flat format, like: OR(AND(f1,f2), > AND(f3,f4,f5)…) , so that Kylin can build HBase scan/filter accordingly, but > when there are too many or filters in query, like: AND(OR(f1,f2,...,f1000), > OR(g1,g2,...,g1000), OR(h1,h2,...,h1000)), then the generated flat filter > size will be 1000*1000*1000=1 billion, the flatting process will cause Kylin > server OOM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3797) Too many or filters may break Kylin server when flatting filter
[ https://issues.apache.org/jira/browse/KYLIN-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-3797: Fix Version/s: v2.6.4 > Too many or filters may break Kylin server when flatting filter > --- > > Key: KYLIN-3797 > URL: https://issues.apache.org/jira/browse/KYLIN-3797 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Gang Ma >Assignee: Gang Ma >Priority: Major > Fix For: v3.0.0-alpha, v2.6.4 > > > Kylin needs to convert filter into a flat format, like: OR(AND(f1,f2), > AND(f3,f4,f5)…) , so that Kylin can build HBase scan/filter accordingly, but > when there are too many or filters in query, like: AND(OR(f1,f2,...,f1000), > OR(g1,g2,...,g1000), OR(h1,h2,...,h1000)), then the generated flat filter > size will be 1000*1000*1000=1 billion, the flatting process will cause Kylin > server OOM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] coveralls edited a comment on issue #860: build dimension dictionary using spark
coveralls edited a comment on issue #860: build dimension dictionary using spark URL: https://github.com/apache/kylin/pull/860#issuecomment-535306368 ## Pull Request Test Coverage Report for [Build 5059](https://coveralls.io/builds/25942171) * **0** of **268** **(0.0%)** changed or added relevant lines in **5** files are covered. * **2** unchanged lines in **2** files lost coverage. * Overall coverage decreased (**-0.09%**) to **27.829%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/25942171/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L1458) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/model/TblColRef.java](https://coveralls.io/builds/25942171/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FTblColRef.java#L284) | 0 | 1 | 0.0% | [engine-mr/src/main/java/org/apache/kylin/engine/mr/JobBuilderSupport.java](https://coveralls.io/builds/25942171/source?filename=engine-mr%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fmr%2FJobBuilderSupport.java#L255) | 0 | 1 | 0.0% | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java](https://coveralls.io/builds/25942171/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkBatchCubingJobBuilder2.java#L80) | 0 | 18 | 0.0% | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBuildDictionary.java](https://coveralls.io/builds/25942171/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkBuildDictionary.java#L86) | 0 | 247 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/25942171/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% | | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java](https://coveralls.io/builds/25942171/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkBatchCubingJobBuilder2.java#L86) | 1 | 0.0% | | Totals | [](https://coveralls.io/builds/25942171) | | :-- | --: | | Change from base [Build 5053](https://coveralls.io/builds/25913474): | -0.09% | | Covered Lines: | 23706 | | Relevant Lines: | 85183 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4180) Prevent abnormal CPU usage by limiting flat filters length
[
https://issues.apache.org/jira/browse/KYLIN-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938404#comment-16938404
]
ASF GitHub Bot commented on KYLIN-4180:
---
TempleZhou commented on pull request #861: KYLIN-4180 Prevent abnormal CPU
usage by limiting flat filters length
URL: https://github.com/apache/kylin/pull/861
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Prevent abnormal CPU usage by limiting flat filters length
> --
>
> Key: KYLIN-4180
> URL: https://issues.apache.org/jira/browse/KYLIN-4180
> Project: Kylin
> Issue Type: Improvement
> Components: Metadata
>Affects Versions: v2.4.0, v2.5.0, v2.6.0
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Major
>
> The following SQL will lead to the list returned by
> org.apache.kylin.metadata.filter.TupleFilter#cartesianProduct become very
> large, and the process to producing the List will use lots of CPU.
> {code:sql}
> select * from temp.test_filter
> where ((a >= 0 and a <= 1)
> or(a >= 1 and a <= 2)
> or(a >= 2 and a <= 3)
> ...
> or(a >= 300 and a <= 301)
> ) and ((b >= 0 and b <= 1)
> or(b >= 1 and b <= 2)
> or(b >= 2 and b <= 3)
> ...
> or(b >= 300 and b <= 301)
> )
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] asf-ci commented on issue #861: KYLIN-4180 Prevent abnormal CPU usage by limiting flat filters length
asf-ci commented on issue #861: KYLIN-4180 Prevent abnormal CPU usage by limiting flat filters length URL: https://github.com/apache/kylin/pull/861#issuecomment-535399213 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] TempleZhou opened a new pull request #861: KYLIN-4180 Prevent abnormal CPU usage by limiting flat filters length
TempleZhou opened a new pull request #861: KYLIN-4180 Prevent abnormal CPU usage by limiting flat filters length URL: https://github.com/apache/kylin/pull/861 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Reopened] (KYLIN-4180) Prevent abnormal CPU usage by limiting flat filters length
[
https://issues.apache.org/jira/browse/KYLIN-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Temple Zhou reopened KYLIN-4180:
The fix solution can be better.
> Prevent abnormal CPU usage by limiting flat filters length
> --
>
> Key: KYLIN-4180
> URL: https://issues.apache.org/jira/browse/KYLIN-4180
> Project: Kylin
> Issue Type: Improvement
> Components: Metadata
>Affects Versions: v2.4.0, v2.5.0, v2.6.0
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Major
>
> The following SQL will lead to the list returned by
> org.apache.kylin.metadata.filter.TupleFilter#cartesianProduct become very
> large, and the process to producing the List will use lots of CPU.
> {code:sql}
> select * from temp.test_filter
> where ((a >= 0 and a <= 1)
> or(a >= 1 and a <= 2)
> or(a >= 2 and a <= 3)
> ...
> or(a >= 300 and a <= 301)
> ) and ((b >= 0 and b <= 1)
> or(b >= 1 and b <= 2)
> or(b >= 2 and b <= 3)
> ...
> or(b >= 300 and b <= 301)
> )
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (KYLIN-4180) Prevent abnormal CPU usage by limiting flat filters length
[
https://issues.apache.org/jira/browse/KYLIN-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Temple Zhou closed KYLIN-4180.
--
Resolution: Duplicate
> Prevent abnormal CPU usage by limiting flat filters length
> --
>
> Key: KYLIN-4180
> URL: https://issues.apache.org/jira/browse/KYLIN-4180
> Project: Kylin
> Issue Type: Improvement
> Components: Metadata
>Affects Versions: v2.4.0, v2.5.0, v2.6.0
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Major
>
> The following SQL will lead to the list returned by
> org.apache.kylin.metadata.filter.TupleFilter#cartesianProduct become very
> large, and the process to producing the List will use lots of CPU.
> {code:sql}
> select * from temp.test_filter
> where ((a >= 0 and a <= 1)
> or(a >= 1 and a <= 2)
> or(a >= 2 and a <= 3)
> ...
> or(a >= 300 and a <= 301)
> ) and ((b >= 0 and b <= 1)
> or(b >= 1 and b <= 2)
> or(b >= 2 and b <= 3)
> ...
> or(b >= 300 and b <= 301)
> )
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
