[GitHub] [kylin] hit-lacus commented on pull request #1573: KYLIN-4900 real-time stream doesn't support GMT-N timezone and doesn't convert other time columnexcept derived column above day
hit-lacus commented on pull request #1573: URL: https://github.com/apache/kylin/pull/1573#issuecomment-781912579 Firstly, I feel good about this patch. Secondly, last time you said you will verify if **lambda mode** (use data from hive to refresh/overwrite data from Kafka) works as you expected. Since the fact that these two feature are related (on timezone), I worried about that if you have an another patch on **lambda mode**, I have to test timezone related test case twice. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4900) The result of derived time columns are error, when timezone is GMT-1 or GMT-N
[
https://issues.apache.org/jira/browse/KYLIN-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286920#comment-17286920
]
ASF GitHub Bot commented on KYLIN-4900:
---
hit-lacus commented on pull request #1573:
URL: https://github.com/apache/kylin/pull/1573#issuecomment-781912579
Firstly, I feel good about this patch.
Secondly, last time you said you will verify if **lambda mode** (use data
from hive to refresh/overwrite data from Kafka) works as you expected. Since
the fact that these two feature are related (on timezone), I worried about that
if you have an another patch on **lambda mode**, I have to test timezone
related test case twice.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> The result of derived time columns are error, when timezone is GMT-1 or GMT-N
> -
>
> Key: KYLIN-4900
> URL: https://issues.apache.org/jira/browse/KYLIN-4900
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
>Reporter: Kun Liu
>Assignee: Kun Liu
>Priority: Blocker
> Fix For: v3.1.2
>
> Attachments: day_start is error.png, result- day_start.png
>
>
> When set the configuration of `kylin.stream.event.timezone` to GMT-1 or
> GMT-N, the result of DAY_START is error.
>
> The data produced by use `$KYLIN_HOME/bin/kylin.sh
> org.apache.kylin.source.kafka.util.KafkaSampleProducer --topic
> kylin_streaming_topic --broker localhost:9092 --interval 1`
>
> message template is ：
> 2021-02-05 06:32:28,720 INFO [main] util.KafkaSampleProducer:136 : Sending 1
> message:
> \{"country":"US","amount":65.78351439157635,"qty":9,"currency":"USD","order_time":1612506748660,"category":"ELECTRONIC","device":"Windows","user":{"gender":"Male","id":"e1f07f05-9eff-46fa-d401-180d0441df13","first_name":"unknown","age":22}}
>
> The order_time of first message is 1612506748660 which is 2021-02-05 14:32:28
> GMT+8 or 2021-02-05 5:32:28 GMT-0
>
> The query result is in the attachments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] zhangayqian commented on a change in pull request #1585: KYLIN-4906 Support query/job server dynamic register and discovery in kylin4
zhangayqian commented on a change in pull request #1585: URL: https://github.com/apache/kylin/pull/1585#discussion_r578983496 ## File path: core-common/pom.xml ## @@ -49,6 +49,12 @@ com.fasterxml.jackson.core jackson-databind + +org.apache.curator +curator-x-discovery + Review comment: This comment can be deleted. Because kylin4 is removing hbase dependency. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4906) support query/job server dynamic register and discovery in kylin4
[ https://issues.apache.org/jira/browse/KYLIN-4906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286945#comment-17286945 ] ASF GitHub Bot commented on KYLIN-4906: --- zhangayqian commented on a change in pull request #1585: URL: https://github.com/apache/kylin/pull/1585#discussion_r578983496 ## File path: core-common/pom.xml ## @@ -49,6 +49,12 @@ com.fasterxml.jackson.core jackson-databind + +org.apache.curator +curator-x-discovery + Review comment: This comment can be deleted. Because kylin4 is removing hbase dependency. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support query/job server dynamic register and discovery in kylin4 > - > > Key: KYLIN-4906 > URL: https://issues.apache.org/jira/browse/KYLIN-4906 > Project: Kylin > Issue Type: Improvement > Components: Integration >Affects Versions: v4.0.0-beta >Reporter: ShengJun Zheng >Assignee: ShengJun Zheng >Priority: Major > Fix For: v4.0.0-GA > > Attachments: screenshot-1.png, screenshot-2.png > > > Hi，currently we are troubled by a static configuration: > kylin.server.cluster-servers. There tow situations not so friendly to us. > 1. when adding query node to cluster, we forgot to update this > configuration in job server. It caused the newly added server was not > notified of all metadata update events, causing wrong query result. > 2. we plan to deploy query server in k8s , this static configuration is not > cloud native (having to update k8s configmap) > We use kylin 2.x，seems this is still a problem in kylin 3.x (relavent code: > org.apache.kylin.metadata.cachesync.Broadcaster). > I wonder if i was wrong or is there already a solution . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] lgtm-com[bot] commented on pull request #1568: KYLIN-4895: change spark deploy mode of kylin4.0 engine from local to cluster
lgtm-com[bot] commented on pull request #1568: URL: https://github.com/apache/kylin/pull/1568#issuecomment-781958873 This pull request **introduces 2 alerts** when merging b01587624bc3edf34d19548751d5a9737ec42f2d into b428f1e1ab6a48101bec03e515d135c57af32878 - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-cd1533446511e850b609c3597a1229696d86cef7) **new alerts:** * 2 for Potential input resource leak This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4895) change spark deploy mode of kylin4.0 engine from local to cluster
[ https://issues.apache.org/jira/browse/KYLIN-4895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286960#comment-17286960 ] ASF GitHub Bot commented on KYLIN-4895: --- lgtm-com[bot] commented on pull request #1568: URL: https://github.com/apache/kylin/pull/1568#issuecomment-781958873 This pull request **introduces 2 alerts** when merging b01587624bc3edf34d19548751d5a9737ec42f2d into b428f1e1ab6a48101bec03e515d135c57af32878 - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-cd1533446511e850b609c3597a1229696d86cef7) **new alerts:** * 2 for Potential input resource leak This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > change spark deploy mode of kylin4.0 engine from local to cluster > - > > Key: KYLIN-4895 > URL: https://issues.apache.org/jira/browse/KYLIN-4895 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Affects Versions: v4.0.0-alpha >Reporter: tianhui >Priority: Major > Fix For: v4.0.0-GA > > > In cloud native environment, the memory of pod is quite limited. But the > spark Driver can use a huge amount of memory, in job engine pod, which is > quite difficult to manage when there are more and more projects. > So it's better to put spark driver on yarn, and kylin only maintain its > status. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] zhengshengjun commented on a change in pull request #1585: KYLIN-4906 Support query/job server dynamic register and discovery in kylin4
zhengshengjun commented on a change in pull request #1585: URL: https://github.com/apache/kylin/pull/1585#discussion_r579576534 ## File path: core-common/pom.xml ## @@ -49,6 +49,12 @@ com.fasterxml.jackson.core jackson-databind + +org.apache.curator +curator-x-discovery + Review comment: thanks for reviewing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4906) support query/job server dynamic register and discovery in kylin4
[ https://issues.apache.org/jira/browse/KYLIN-4906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17287456#comment-17287456 ] ASF GitHub Bot commented on KYLIN-4906: --- zhengshengjun commented on a change in pull request #1585: URL: https://github.com/apache/kylin/pull/1585#discussion_r579576534 ## File path: core-common/pom.xml ## @@ -49,6 +49,12 @@ com.fasterxml.jackson.core jackson-databind + +org.apache.curator +curator-x-discovery + Review comment: thanks for reviewing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support query/job server dynamic register and discovery in kylin4 > - > > Key: KYLIN-4906 > URL: https://issues.apache.org/jira/browse/KYLIN-4906 > Project: Kylin > Issue Type: Improvement > Components: Integration >Affects Versions: v4.0.0-beta >Reporter: ShengJun Zheng >Assignee: ShengJun Zheng >Priority: Major > Fix For: v4.0.0-GA > > Attachments: screenshot-1.png, screenshot-2.png > > > Hi，currently we are troubled by a static configuration: > kylin.server.cluster-servers. There tow situations not so friendly to us. > 1. when adding query node to cluster, we forgot to update this > configuration in job server. It caused the newly added server was not > notified of all metadata update events, causing wrong query result. > 2. we plan to deploy query server in k8s , this static configuration is not > cloud native (having to update k8s configmap) > We use kylin 2.x，seems this is still a problem in kylin 3.x (relavent code: > org.apache.kylin.metadata.cachesync.Broadcaster). > I wonder if i was wrong or is there already a solution . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] codecov-io edited a comment on pull request #1585: KYLIN-4906 Support query/job server dynamic register and discovery in kylin4
codecov-io edited a comment on pull request #1585: URL: https://github.com/apache/kylin/pull/1585#issuecomment-781874343 # [Codecov](https://codecov.io/gh/apache/kylin/pull/1585?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`kylin-on-parquet-v2@b428f1e`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/kylin/pull/1585?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## kylin-on-parquet-v2#1585 +/- ## == Coverage ? 24.36% Complexity ? 4649 == Files ? 1145 Lines ?65335 Branches ? 9595 == Hits ?15917 Misses ?47754 Partials ? 1664 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1585?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1585?src=pr&el=footer). Last update [b428f1e...3f9dd29](https://codecov.io/gh/apache/kylin/pull/1585?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4906) support query/job server dynamic register and discovery in kylin4
[ https://issues.apache.org/jira/browse/KYLIN-4906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17287462#comment-17287462 ] ASF GitHub Bot commented on KYLIN-4906: --- codecov-io edited a comment on pull request #1585: URL: https://github.com/apache/kylin/pull/1585#issuecomment-781874343 # [Codecov](https://codecov.io/gh/apache/kylin/pull/1585?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`kylin-on-parquet-v2@b428f1e`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/kylin/pull/1585?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## kylin-on-parquet-v2#1585 +/- ## == Coverage ? 24.36% Complexity ? 4649 == Files ? 1145 Lines ?65335 Branches ? 9595 == Hits ?15917 Misses ?47754 Partials ? 1664 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1585?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1585?src=pr&el=footer). Last update [b428f1e...3f9dd29](https://codecov.io/gh/apache/kylin/pull/1585?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support query/job server dynamic register and discovery in kylin4 > - > > Key: KYLIN-4906 > URL: https://issues.apache.org/jira/browse/KYLIN-4906 > Project: Kylin > Issue Type: Improvement > Components: Integration >Affects Versions: v4.0.0-beta >Reporter: ShengJun Zheng >Assignee: ShengJun Zheng >Priority: Major > Fix For: v4.0.0-GA > > Attachments: screenshot-1.png, screenshot-2.png > > > Hi，currently we are troubled by a static configuration: > kylin.server.cluster-servers. There tow situations not so friendly to us. > 1. when adding query node to cluster, we forgot to update this > configuration in job server. It caused the newly added server was not > notified of all metadata update events, causing wrong query result. > 2. we plan to deploy query server in k8s , this static configuration is not > cloud native (having to update k8s configmap) > We use kylin 2.x，seems this is still a problem in kylin 3.x (relavent code: > org.apache.kylin.metadata.cachesync.Broadcaster). > I wonder if i was wrong or is there already a solution . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] hit-lacus merged pull request #1581: HOTFIX Add missing docker file: Dockerfile_hadoop
hit-lacus merged pull request #1581: URL: https://github.com/apache/kylin/pull/1581 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (KYLIN-4905) Support limit .. offset ... in spark query engine
[ https://issues.apache.org/jira/browse/KYLIN-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu reassigned KYLIN-4905: --- Assignee: ShengJun Zheng > Support limit .. offset ... in spark query engine > - > > Key: KYLIN-4905 > URL: https://issues.apache.org/jira/browse/KYLIN-4905 > Project: Kylin > Issue Type: New Feature > Components: Query Engine >Affects Versions: v4.0.0-alpha >Reporter: ShengJun Zheng >Assignee: ShengJun Zheng >Priority: Major > Fix For: v4.0.0-GA > > > when use top-level result offset clause in query expression (ANSI SQL) : > limit xxx offset xxx in spark query engine，limit will not push down into > spark engine, and offset will not take effect. This is incompatible wIth > Kylin 2.x~3.x. > After looking through the code, i found it's because spark dose not support > limit ... offset ... now. There is a spark issue in progress: > https://issues.apache.org/jira/browse/SPARK-28330, which was created in 2019 > but still in progress. > So, should we support this feature temporarily in KYLIN? ： > 1. push down limit to spark > 2. take result from starting offset in KYLIN query server > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4903) cache parent datasource to accelerate next layer's cuboid building
[ https://issues.apache.org/jira/browse/KYLIN-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu reassigned KYLIN-4903: --- Assignee: ShengJun Zheng > cache parent datasource to accelerate next layer's cuboid building > -- > > Key: KYLIN-4903 > URL: https://issues.apache.org/jira/browse/KYLIN-4903 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-beta >Reporter: ShengJun Zheng >Assignee: ShengJun Zheng >Priority: Major > Fix For: v4.0.0-GA > > > In Kylin V4, parent datasource is not cached in next layer's cuboid building, > causing repeated HDFS files read. Cacheing parent datasource in memory will > in enhance 20~30% build performance in our case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4895) change spark deploy mode of kylin4.0 engine from local to cluster
[ https://issues.apache.org/jira/browse/KYLIN-4895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu reassigned KYLIN-4895: --- Assignee: tianhui > change spark deploy mode of kylin4.0 engine from local to cluster > - > > Key: KYLIN-4895 > URL: https://issues.apache.org/jira/browse/KYLIN-4895 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Affects Versions: v4.0.0-alpha >Reporter: tianhui >Assignee: tianhui >Priority: Major > Fix For: v4.0.0-GA > > > In cloud native environment, the memory of pod is quite limited. But the > spark Driver can use a huge amount of memory, in job engine pod, which is > quite difficult to manage when there are more and more projects. > So it's better to put spark driver on yarn, and kylin only maintain its > status. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4891) Set the default value of 'kylin.query.spark-engine.expose-sharding-trait' to false
[ https://issues.apache.org/jira/browse/KYLIN-4891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu closed KYLIN-4891. --- > Set the default value of 'kylin.query.spark-engine.expose-sharding-trait' to > false > -- > > Key: KYLIN-4891 > URL: https://issues.apache.org/jira/browse/KYLIN-4891 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Minor > > Set the default value of 'kylin.query.spark-engine.expose-sharding-trait' to > false -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4891) Set the default value of 'kylin.query.spark-engine.expose-sharding-trait' to false
[ https://issues.apache.org/jira/browse/KYLIN-4891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-4891: Fix Version/s: (was: v4.0.0-GA) > Set the default value of 'kylin.query.spark-engine.expose-sharding-trait' to > false > -- > > Key: KYLIN-4891 > URL: https://issues.apache.org/jira/browse/KYLIN-4891 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Minor > > Set the default value of 'kylin.query.spark-engine.expose-sharding-trait' to > false -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4886) Current rocksdbjni 5.9.2 dependency does not support ARM64 platform.
[
https://issues.apache.org/jira/browse/KYLIN-4886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-4886:
Fix Version/s: (was: v3.1.2)
> Current rocksdbjni 5.9.2 dependency does not support ARM64 platform.
>
>
> Key: KYLIN-4886
> URL: https://issues.apache.org/jira/browse/KYLIN-4886
> Project: Kylin
> Issue Type: Sub-task
>Reporter: liusheng
>Priority: Minor
>
> Similar to the Storm issue: https://issues.apache.org/jira/browse/STORM-3599
> Kylin depend on the rocksdbjni 5.9.2 which cannot support ARM64 arch, we'd
> better to bump the version to at least 5.18.4 which support ARM64 arch.
> {noformat}
> [ERROR] Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 0.345
> s <<< FAILURE! - in org.apache.kylin.dict.lookup.cache.RocksDBLookupTableTest
> [ERROR] testGet(org.apache.kylin.dict.lookup.cache.RocksDBLookupTableTest)
> Time elapsed: 0.27 s <<< ERROR! java.lang.UnsatisfiedLinkError:
> /tmp/librocksdbjni5578787791642743739.so:
> /tmp/librocksdbjni5578787791642743739.so: cannot open shared object file: No
> such file or directory (Possible cause : can't load AMD 64-bit .so on a
> AARCH64-bit platform)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4881) Cube owners/Project Admins cannot see the Cube Planner tab in the web UI
[ https://issues.apache.org/jira/browse/KYLIN-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-4881: Fix Version/s: v3.1.2 > Cube owners/Project Admins cannot see the Cube Planner tab in the web UI > > > Key: KYLIN-4881 > URL: https://issues.apache.org/jira/browse/KYLIN-4881 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v3.1.1 >Reporter: Rafael Felix Correa >Priority: Major > Fix For: v3.1.2 > > > Steps to reproduce: > * On a LDAP+SAML configured kylin instance, create a cube in any project and > grant either cube ownership or project admin privileges to a non-admin > federated user; > * Login in the web UI as the non-admin user; and > * Navigate to the cube details page > What's expected: > * Cube Owners/Project Admins should see the Planner tab, and have access to > the Cube Planner feature > What happens instead: > * Cube Owners/Project Admins can't see the Planner tab in the UI. triggering > via API works and Kylin Admins can see it in the UI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4879) The function of sql to remove comments is not perfect. In some cases, the sql query conditions used will be modified
[
https://issues.apache.org/jira/browse/KYLIN-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-4879:
Fix Version/s: (was: v3.1.0)
v3.1.2
> The function of sql to remove comments is not perfect. In some cases, the sql
> query conditions used will be modified
>
>
> Key: KYLIN-4879
> URL: https://issues.apache.org/jira/browse/KYLIN-4879
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.6.0
>Reporter: wangjie
>Priority: Critical
> Fix For: v3.1.2
>
>
> In the removeCommentInSql method of QueryUtil of the query module, if the
> single quote character of the user's sql contains – or /**/, the regular
> expression will rewrite the sql query condition.
> E.g:
> (1) When the single quotation mark contains --, line break
> {quote}String sql = "select count(*) from test_kylin_fact WHERE column_name
> ='--this is not comment'\n "+ "LIMIT 100 offset 0";
> {quote}
> After the removeCommentInSql method, it will become:
> {quote}select count(*) from test_kylin_fact WHERE column_name = 'LIMIT 100
> offset 0
> {quote}
> (2) Contain multiple lines of comments in single quotes
> {quote}String sql = "select count(*) from test_kylin_fact WHERE column_name
> ='/**--this *is not comment***/'";
> {quote}
> After the removeCommentInSql method, it will become:
> {quote}select count(*) from test_kylin_fact WHERE column_name =''
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4878) Not able to build kylin cube for redshift datasource
[
https://issues.apache.org/jira/browse/KYLIN-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-4878:
Fix Version/s: v3.1.2
> Not able to build kylin cube for redshift datasource
>
>
> Key: KYLIN-4878
> URL: https://issues.apache.org/jira/browse/KYLIN-4878
> Project: Kylin
> Issue Type: Bug
>Reporter: vipul sikka
>Priority: Major
> Fix For: v3.1.2
>
> Attachments: kylin_settings.txt
>
>
> I am using kylin 3.1.0 and try to use redshift as our datasource. When we
> build any cube it keeps hanging and no job created for it too. The logs we
> are gettiing for cube 'Sales_Cube_clone' are:-
> cube.CubeManager:373 : Updating cube instance 'Sales_Cube_clone'
> 2021-01-18 07:12:03,740 WARN [http-bio-7070-exec-6] model.Segments:463 : NEW
> segment start does not fit/connect with other segments:
> Sales_Cube_clone[20181004021000_20190704063000]
> 2021-01-18 07:12:03,741 WARN [http-bio-7070-exec-6] model.Segments:465 : NEW
> segment end does not fit/connect with other segments:
> Sales_Cube_clone[20181004021000_20190704063000]
> 2021-01-18 07:12:03,741 WARN [http-bio-7070-exec-6] model.Segments:463 : NEW
> segment start does not fit/connect with other segments:
> Sales_Cube_clone[20190901010500_20200602052500]
> 2021-01-18 07:12:03,741 WARN [http-bio-7070-exec-6] model.Segments:465 : NEW
> segment end does not fit/connect with other segments:
> Sales_Cube_clone[20190901010500_20200602052500]
> 2021-01-18 07:12:03,742 WARN [http-bio-7070-exec-6] model.Segments:463 : NEW
> segment start does not fit/connect with other segments:
> Sales_Cube_clone[2021010100_20210107073500]
>
> 2021-01-18 07:11:12,783 WARN [http-bio-7070-exec-1]
> annotation.ExceptionHandlerExceptionResolver:136 : Resolved
> [org.apache.kylin.rest.exception.InternalErrorException: Segments overlap:
> Sales_Cube_clone[2021010100_20210107073500] and
> Sales_Cube_clone[20210101031500_20210114105000]]
> 2021-01-18 07:11:30,297 INFO [BadQueryDetector] service.BadQueryDetector:148
> : Detect bad query.
> 2021-01-18 07:11:33,255 INFO [FetcherRunner 1778655433-46]
> threadpool.DefaultFetcherRunner:115 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 6 already succeed, 4 error, 1 discarded, 0 others
> 2021-01-18 07:12:03,255 INFO [FetcherRunner 1778655433-46]
> threadpool.DefaultFetcherRunner:115 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 6 already succeed, 4 error, 1 discarded, 0 others
> 2021-01-18 07:12:03,739 INFO [http-bio-7070-exec-6] cube.CubeManager:373 :
> Updating cube instance 'Sales_Cube_clone'
> 2021-01-18 07:12:03,740 WARN [http-bio-7070-exec-6] model.Segments:463 : NEW
> segment start does not fit/connect with other segments:
> Sales_Cube_clone[20181004021000_20190704063000]
> 2021-01-18 07:12:03,741 WARN [http-bio-7070-exec-6] model.Segments:465 : NEW
> segment end does not fit/connect with other segments:
> Sales_Cube_clone[20181004021000_20190704063000]
> 2021-01-18 07:12:03,741 WARN [http-bio-7070-exec-6] model.Segments:463 : NEW
> segment start does not fit/connect with other segments:
> Sales_Cube_clone[20190901010500_20200602052500]
> 2021-01-18 07:12:03,741 WARN [http-bio-7070-exec-6] model.Segments:465 : NEW
> segment end does not fit/connect with other segments:
> Sales_Cube_clone[20190901010500_20200602052500]
> 2021-01-18 07:12:03,742 WARN [http-bio-7070-exec-6] model.Segments:463 : NEW
> segment start does not fit/connect with other segments:
> Sales_Cube_clone[2021010100_20210107073500]
> 2021-01-18 07:12:03,742 DEBUG [http-bio-7070-exec-6]
> cachesync.CachedCrudAssist:223 : Saving CubeInstance at
> /cube/Sales_Cube_clone.json
> 2021-01-18 07:12:03,754 DEBUG [pool-5-thread-1] cachesync.Broadcaster:119 :
> Servers in the cluster: [localhost:7070]
> 2021-01-18 07:12:03,754 DEBUG [pool-5-thread-1] cachesync.Broadcaster:129 :
> Announcing new broadcast to all: BroadcastEvent{entity=cube,
> event=update, cacheKey=Sales_Cube_clone}
> 2021-01-18 07:12:03,761 DEBUG [http-bio-7070-exec-4]
> cachesync.Broadcaster:267 : Broadcasting UPDATE, cube, Sales_Cube_clone
> 2021-01-18 07:12:03,767 DEBUG [http-bio-7070-exec-4]
> cachesync.Broadcaster:267 : Broadcasting UPDATE, project_data, Demo
> 2021-01-18 07:12:03,768 INFO [http-bio-7070-exec-4] service.CacheService:123
> : cleaning cache for project Demo (currently remove all entries)
> 2021-01-18 07:12:03,769 DEBUG [http-bio-7070-exec-4]
> cachesync.Broadcaster:301 : Done broadcasting UPDATE, project_data, Demo
> 2021-01-18 07:12:03,770 DEBUG [http-bio-7070-exec-4]
> cachesync.Broadcaster:301 : Done broadcasting UPDATE, cube, Sales_Cube_clone
> 2021-01-18 07:12:04,007 DEBUG [http-bio-7070-exec-8] common.KylinCo
[jira] [Updated] (KYLIN-4876) kylin开启下压后,还是报 No realization found for OLAPContext
[ https://issues.apache.org/jira/browse/KYLIN-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-4876: Fix Version/s: v3.1.2 > kylin开启下压后,还是报 No realization found for OLAPContext > --- > > Key: KYLIN-4876 > URL: https://issues.apache.org/jira/browse/KYLIN-4876 > Project: Kylin > Issue Type: Bug > Components: Environment , Query Engine >Affects Versions: v2.5.0, v3.1.1 >Reporter: roger wang >Priority: Blocker > Fix For: v3.1.2 > > > kylin在开启查询下压以后,根据文档的步骤,配置了如下配置: > QUERY PUSH DOWN ### > # > kylin.query.pushdown.runner-class-name=org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl > # > kylin.query.pushdown.update-enabled=false > kylin.query.pushdown.jdbc.url=jdbc:hive2://hadoop102:1/default > kylin.query.pushdown.jdbc.driver=org.apache.hive.jdbc.HiveDriver > kylin.query.pushdown.jdbc.username= > kylin.query.pushdown.jdbc.password= > # > kylin.query.pushdown.jdbc.pool-max-total=8 > kylin.query.pushdown.jdbc.pool-max-idle=8 > kylin.query.pushdown.jdbc.pool-min-idle=0 > > 执行官网的sql: select max(price) as total_sold from kylin_sales > 报错: > Caused by: org.apache.kylin.metadata.realization.NoRealizationFoundException: > No realization found for OLAPContext, > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4869) Can not create global dictionary in UI
[ https://issues.apache.org/jira/browse/KYLIN-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu closed KYLIN-4869. --- > Can not create global dictionary in UI > -- > > Key: KYLIN-4869 > URL: https://issues.apache.org/jira/browse/KYLIN-4869 > Project: Kylin > Issue Type: Bug >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: all > > Attachments: dict1.png > > > I already know that global dict is used for Measure, could some one give me > some help? > !dict1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4866) Streaming from Kafka with nested timestamp column
[
https://issues.apache.org/jira/browse/KYLIN-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu reassigned KYLIN-4866:
---
Assignee: Xiaoxiang Yu
> Streaming from Kafka with nested timestamp column
> -
>
> Key: KYLIN-4866
> URL: https://issues.apache.org/jira/browse/KYLIN-4866
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
>Affects Versions: v3.1.0
> Environment: Docker setup
>Reporter: Hélder Hugo Ferreira
>Assignee: Xiaoxiang Yu
>Priority: Major
> Fix For: v3.1.2
>
> Attachments: image-2021-01-08-16-16-51-615.png
>
>
> We have an IOT Data Platform which saves in Kafka Json messages with the
> following format:
> {code:json}
> {
> "SysProperties": {
> "EventId": "1610120564321723276",
> "EnqueueTime": "2021-01-08T15:42:44.045Z",
> "UserName": "dummyuser"
> },
> "AppProperties": {
> "EventDefinition": "EquipmentData",
> "EventTime": "2021-01-08T15:42:43.26+00:00",
> "ApplicationName": "ConnectIoTSimulator",
> "ApplicationContext": null
> },
> "Data": {
> "Equipment": "DUMMY-01",
> "Material": "DUMMYZA9983884",
> "MeasurementData": {
> "Temp": 57,
> "Pressure": 48.379,
> "Energy": 47,
> "Humidity": 31.727
> }
> }
> {code}
> From these messages, we are trying to build a cube in which the
> "AppProperties.EventTime" shall be the timestamp column (using format
> -MM-dd'T'HH:mm:ss.SSSZZ), "Data.MeasurementData.Temp",
> "Data.MeasurementData.Pressure", "Data.MeasurementData.Energy" and
> "Data.MeasurementData.Humidity" shall be the metrics and all other fields
> dimensions.
> We can setup everything fine in the kylin portal (like in the tutorial),
> however on runtime we have the following exceptions logged by the streaming
> receiver component:
> !image-2021-01-08-16-16-51-615.png!
> After some investigation, we figured out it only works if the timestamp field
> within the Json messages is not nested inside inner objects, like in the
> following example:
> {code:json}
> {
> "timestamp": 1610038228492,
> "nestedData": {
> "Equipment": "EPFTES-01",
> "Material": "ZA9983884",
> "Temp": 10,
> "Pressure": 20.459,
> "Energy": 30,
> "Humidity": 80.967
> }
> }{code}
> Therefore, we believe there is a bug handling the timestamp column when it is
> nested within Json messages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4861) Wrong way to get CubeManager instance in CubeInstance.latestCopyForWrite()
[
https://issues.apache.org/jira/browse/KYLIN-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-4861:
Fix Version/s: v3.1.2
> Wrong way to get CubeManager instance in CubeInstance.latestCopyForWrite()
> --
>
> Key: KYLIN-4861
> URL: https://issues.apache.org/jira/browse/KYLIN-4861
> Project: Kylin
> Issue Type: Bug
>Reporter: Zhong Yanghong
>Assignee: Xiaoxiang Yu
>Priority: Major
> Fix For: v3.1.2
>
>
> Each cube can have its own KylinConfig. Then for the following code:
> {code}
> public CubeInstance latestCopyForWrite() {
> CubeManager mgr = CubeManager.getInstance(config);
> CubeInstance latest = mgr.getCube(name); // in case this object is
> out-of-date
> return mgr.copyForWrite(latest);
> }
> {code}
> Each cube can have a different CubeManager instance, which may easily cause
> map consistency issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
