[jira] [Updated] (KYLIN-4652) Fix data type incompatible issue for sum(case when)
[ https://issues.apache.org/jira/browse/KYLIN-4652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-4652: -- Summary: Fix data type incompatible issue for sum(case when) (was: Fix sum(case when)) > Fix data type incompatible issue for sum(case when) > --- > > Key: KYLIN-4652 > URL: https://issues.apache.org/jira/browse/KYLIN-4652 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4653) Make the capacity for the LinkedBlockingQueue of BlockingReservoir configurable
[ https://issues.apache.org/jira/browse/KYLIN-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-4653: -- Summary: Make the capacity for the LinkedBlockingQueue of BlockingReservoir configurable (was: Don't set capacity for the LinkedBlockingQueue of BlockingReservoir) > Make the capacity for the LinkedBlockingQueue of BlockingReservoir > configurable > --- > > Key: KYLIN-4653 > URL: https://issues.apache.org/jira/browse/KYLIN-4653 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4626) add set kylin home sh
[ https://issues.apache.org/jira/browse/KYLIN-4626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162877#comment-17162877 ] ASF GitHub Bot commented on KYLIN-4626: --- bigxiaochu commented on pull request #1311: URL: https://github.com/apache/kylin/pull/1311#issuecomment-662516076 @xiaowangyu how do you see This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > add set kylin home sh > - > > Key: KYLIN-4626 > URL: https://issues.apache.org/jira/browse/KYLIN-4626 > Project: Kylin > Issue Type: Improvement >Reporter: chuxiao >Assignee: chuxiao >Priority: Major > > KYLIN_HOME 是重要的，几乎每个脚本都离不开它。但随便设置环境变量并不是一个最佳行为，比如安装了多套实例。增加set-kylin-home.sh, > kylin实例可以设置自己的环境变量 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4613) add buildCubeCLi as hadoop main class and jobRestClient
[ https://issues.apache.org/jira/browse/KYLIN-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162881#comment-17162881 ] ASF GitHub Bot commented on KYLIN-4613: --- bigxiaochu commented on pull request #1299: URL: https://github.com/apache/kylin/pull/1299#issuecomment-662517072 @xiaowangyu how do you see. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > add buildCubeCLi as hadoop main class and jobRestClient > - > > Key: KYLIN-4613 > URL: https://issues.apache.org/jira/browse/KYLIN-4613 > Project: Kylin > Issue Type: Improvement > Components: Client - CLI >Reporter: chuxiao >Assignee: chuxiao >Priority: Minor > > support submit job and wait finish. retry 3 times when error -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] bigxiaochu commented on pull request #1292: KYLIN-4607 destributed scheduler kill job instead of fetcherRunner k…
bigxiaochu commented on pull request #1292: URL: https://github.com/apache/kylin/pull/1292#issuecomment-662518142 @RupengWang how do you see. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4607) destributed scheduler kill job instead of fetcherRunner kill job
[ https://issues.apache.org/jira/browse/KYLIN-4607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162883#comment-17162883 ] ASF GitHub Bot commented on KYLIN-4607: --- bigxiaochu commented on pull request #1292: URL: https://github.com/apache/kylin/pull/1292#issuecomment-662518142 @RupengWang how do you see. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > destributed scheduler kill job instead of fetcherRunner kill job > - > > Key: KYLIN-4607 > URL: https://issues.apache.org/jira/browse/KYLIN-4607 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v3.0.0-alpha >Reporter: chuxiao >Assignee: chuxiao >Priority: Major > > see KYLIN-4250 > For DistributedScheduler, even if FetchFailed is true, not in runningJobs, > the status is running, FetchRunner should not kill the job because the job > may be scheduler by another kylin service. So destributed scheduler kill job > when isMetaDataPersistException > 注释掉了一个测试用例，这个用例依赖的逻辑是task状态异常，但metastore有问题作业异常状态无法持久化，于是通过fetcher把运行状态改成异常状态。这个逻辑是有问题的。 > 对DefaultScheduler来说，如果JobRunner抓到持久化异常，会执行强杀作业逻辑，强杀会一直循环直到成功，是不需要fetcher再去强杀的，因为如果持久化异常导致JobRunner强杀未成功，fetcher也不会成功。 > 而对于DistributedScheduler来说，fetcher的强杀逻辑会把其他进程正在running的作业给误杀 > 掉，更不能保留。 > 详见https://zhuanlan.zhihu.com/p/154376900 > 所以这个case可以删掉 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] bigxiaochu commented on pull request #1299: KYLIN-4613 add build cubev2 and JobRestClient
bigxiaochu commented on pull request #1299: URL: https://github.com/apache/kylin/pull/1299#issuecomment-662517072 @xiaowangyu how do you see. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kylin] bigxiaochu commented on pull request #1311: KYLIN-4626 add set kylin home shell
bigxiaochu commented on pull request #1311: URL: https://github.com/apache/kylin/pull/1311#issuecomment-662516076 @xiaowangyu how do you see This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4648) Upgrade Spark to latest 2.3 or 2.4 version
[ https://issues.apache.org/jira/browse/KYLIN-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162889#comment-17162889 ] chuxiao commented on KYLIN-4648: it is 2.3.2 now, how about 2.4.4? > Upgrade Spark to latest 2.3 or 2.4 version > -- > > Key: KYLIN-4648 > URL: https://issues.apache.org/jira/browse/KYLIN-4648 > Project: Kylin > Issue Type: Task > Components: Spark Engine >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.1 > > > To get some bugfix like: > https://issues.apache.org/jira/browse/SPARK-23243 > https://issues.apache.org/jira/browse/SPARK-28699 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (KYLIN-4648) Upgrade Spark to latest 2.3 or 2.4 version
[ https://issues.apache.org/jira/browse/KYLIN-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chuxiao updated KYLIN-4648: --- Comment: was deleted (was: it is 2.3.2 now, how about 2.4.4?) > Upgrade Spark to latest 2.3 or 2.4 version > -- > > Key: KYLIN-4648 > URL: https://issues.apache.org/jira/browse/KYLIN-4648 > Project: Kylin > Issue Type: Task > Components: Spark Engine >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.1 > > > To get some bugfix like: > https://issues.apache.org/jira/browse/SPARK-23243 > https://issues.apache.org/jira/browse/SPARK-28699 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4652) Fix data type incompatible issue for sum(case when)
[ https://issues.apache.org/jira/browse/KYLIN-4652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162892#comment-17162892 ] chuxiao commented on KYLIN-4652: what is problem > Fix data type incompatible issue for sum(case when) > --- > > Key: KYLIN-4652 > URL: https://issues.apache.org/jira/browse/KYLIN-4652 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4648) Upgrade Spark to latest 2.3 or 2.4 version
[ https://issues.apache.org/jira/browse/KYLIN-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162891#comment-17162891 ] chuxiao commented on KYLIN-4648: it is 2.4.5 now > Upgrade Spark to latest 2.3 or 2.4 version > -- > > Key: KYLIN-4648 > URL: https://issues.apache.org/jira/browse/KYLIN-4648 > Project: Kylin > Issue Type: Task > Components: Spark Engine >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.1 > > > To get some bugfix like: > https://issues.apache.org/jira/browse/SPARK-23243 > https://issues.apache.org/jira/browse/SPARK-28699 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4500) Timeout waiting for connection from pool
[
https://issues.apache.org/jira/browse/KYLIN-4500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162898#comment-17162898
]
Gabor Arki commented on KYLIN-4500:
---
[~hit_lacus], the linked issue seems to be unrelated.
We are running Kylin on AWS EMR cluster and use S3 (EMRFS) for data storage
instead of HDFS to make the cluster stateless. However, after some continuous
uptime, we are always facing this issue where both the query server and the
Kylin MR jobs are suddenly failing with the aforementioned Exception. The root
cause of these failures is that the connection pool of the EMR cluster to S3 is
exhausted and new operations fail to acquire a connection and time out while
waiting for an S3 connection.
No matter how much of a pool size we are configuring for the
fs.s3.maxConnections value, this keeps happening. The underlying issue is very
likely a connection leak where some code is not properly closing and returning
a connection to the pool. Given a query server restart is solving the issue, I
suspect the pool is exhausted somewhere in the Kylin query server code.
> Timeout waiting for connection from pool
>
>
> Key: KYLIN-4500
> URL: https://issues.apache.org/jira/browse/KYLIN-4500
> Project: Kylin
> Issue Type: Bug
>Reporter: Gabor Arki
>Priority: Major
> Attachments: kylin-connection-timeout.txt
>
>
> h4. Environment
> * Kylin server 3.0.0
> * EMR 5.28
> h4. Issue
> After an extended uptime, both Kylin query server and jobs running on EMR
> stop working. The root cause in both cases is:
> {noformat}
> Caused by: java.io.IOException:
> com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable
> to execute HTTP request: Timeout waiting for connection from pool
> at
> com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
> ~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
> Based on
> [https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
> increasing the fs.s3.maxConnections setting to 1 is just delaying the
> issue thus the underlying issue is likely a connection leak. It also
> indicates a leak that restarting the kylin service solves the problem.
> A full stack trace from the QueryService is attached.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] bigxiaochu opened a new pull request #1325: KYLIN-4653 Make the capacity for the LinkedBlockingQueue of BlockingR…
bigxiaochu opened a new pull request #1325: URL: https://github.com/apache/kylin/pull/1325 …eservoir configurable ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4653) Make the capacity for the LinkedBlockingQueue of BlockingReservoir configurable
[ https://issues.apache.org/jira/browse/KYLIN-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162905#comment-17162905 ] chuxiao commented on KYLIN-4653: done > Make the capacity for the LinkedBlockingQueue of BlockingReservoir > configurable > --- > > Key: KYLIN-4653 > URL: https://issues.apache.org/jira/browse/KYLIN-4653 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4656) Guava classpath conflict caused by kylin-jdbc 3.1.0 jar
Gabor Arki created KYLIN-4656: - Summary: Guava classpath conflict caused by kylin-jdbc 3.1.0 jar Key: KYLIN-4656 URL: https://issues.apache.org/jira/browse/KYLIN-4656 Project: Kylin Issue Type: Bug Components: Driver - JDBC Affects Versions: v3.1.0 Reporter: Gabor Arki The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged version of the Guava library. This is causing class duplications with the original guava jar if it is also on the classpath which results in non-deterministic, runtime errors depending on which version of a certain guava class has been picked up by the classloader from the 2 versions. Either implement a proper shading with package relocation or rely on transitive dependency, but do not shade non-repackaged versions of libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4653) Make the capacity for the LinkedBlockingQueue of BlockingReservoir configurable
[ https://issues.apache.org/jira/browse/KYLIN-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162904#comment-17162904 ] ASF GitHub Bot commented on KYLIN-4653: --- bigxiaochu opened a new pull request #1325: URL: https://github.com/apache/kylin/pull/1325 …eservoir configurable ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make the capacity for the LinkedBlockingQueue of BlockingReservoir > configurable > --- > > Key: KYLIN-4653 > URL: https://issues.apache.org/jira/browse/KYLIN-4653 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4656) Guava classpath conflict caused by kylin-jdbc 3.1.0 jar
[ https://issues.apache.org/jira/browse/KYLIN-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Arki updated KYLIN-4656: -- Description: The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged version of the Guava library. This is causing class duplication with the original guava jar if it is also on the classpath which results in non-deterministic, runtime errors depending on which version of a certain guava class has been picked up by the class-loader from the 2 versions. Based on the runtime errors of the missing classes and methods, it seems to be a very old version, probably <=14. Either implement a proper shading with package relocation or rely on transitive dependency, but do not shade non-repackaged versions of libraries. was: The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged version of the Guava library. This is causing class duplications with the original guava jar if it is also on the classpath which results in non-deterministic, runtime errors depending on which version of a certain guava class has been picked up by the classloader from the 2 versions. Either implement a proper shading with package relocation or rely on transitive dependency, but do not shade non-repackaged versions of libraries. > Guava classpath conflict caused by kylin-jdbc 3.1.0 jar > --- > > Key: KYLIN-4656 > URL: https://issues.apache.org/jira/browse/KYLIN-4656 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v3.1.0 >Reporter: Gabor Arki >Priority: Critical > > The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged > version of the Guava library. This is causing class duplication with the > original guava jar if it is also on the classpath which results in > non-deterministic, runtime errors depending on which version of a certain > guava class has been picked up by the class-loader from the 2 versions. Based > on the runtime errors of the missing classes and methods, it seems to be a > very old version, probably <=14. > > Either implement a proper shading with package relocation or rely on > transitive dependency, but do not shade non-repackaged versions of libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4656) Guava classpath conflict caused by kylin-jdbc 3.1.0 jar
[ https://issues.apache.org/jira/browse/KYLIN-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Arki updated KYLIN-4656: -- Description: The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged version of the Guava library. This is causing class duplication with the original guava jar if it is also on the classpath which results in non-deterministic, runtime errors depending on which version of a certain guava class has been picked up by the class-loader from the 2 versions. Based on the runtime errors of the missing classes and methods, it seems to be a very old version, probably <=14. Because of the, Either implement a proper shading with package relocation or rely on transitive dependency, but do not shade non-repackaged versions of libraries. was: The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged version of the Guava library. This is causing class duplication with the original guava jar if it is also on the classpath which results in non-deterministic, runtime errors depending on which version of a certain guava class has been picked up by the class-loader from the 2 versions. Based on the runtime errors of the missing classes and methods, it seems to be a very old version, probably <=14. Either implement a proper shading with package relocation or rely on transitive dependency, but do not shade non-repackaged versions of libraries. > Guava classpath conflict caused by kylin-jdbc 3.1.0 jar > --- > > Key: KYLIN-4656 > URL: https://issues.apache.org/jira/browse/KYLIN-4656 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v3.1.0 >Reporter: Gabor Arki >Priority: Critical > > The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged > version of the Guava library. This is causing class duplication with the > original guava jar if it is also on the classpath which results in > non-deterministic, runtime errors depending on which version of a certain > guava class has been picked up by the class-loader from the 2 versions. Based > on the runtime errors of the missing classes and methods, it seems to be a > very old version, probably <=14. Because of the, > > Either implement a proper shading with package relocation or rely on > transitive dependency, but do not shade non-repackaged versions of libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4656) Guava classpath conflict caused by kylin-jdbc 3.1.0 jar
[ https://issues.apache.org/jira/browse/KYLIN-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Arki updated KYLIN-4656: -- Description: The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged version of the Guava library. This is causing class duplication with the original guava jar if it is also on the classpath which results in non-deterministic, runtime errors depending on which version of a certain guava class has been picked up by the class-loader from the 2 versions. Based on the runtime errors of the missing classes and methods, it seems to be a very old version, probably <=14. Either implement a proper shading with package relocation or rely on transitive dependency, but do not shade non-repackaged versions of libraries. was: The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged version of the Guava library. This is causing class duplication with the original guava jar if it is also on the classpath which results in non-deterministic, runtime errors depending on which version of a certain guava class has been picked up by the class-loader from the 2 versions. Based on the runtime errors of the missing classes and methods, it seems to be a very old version, probably <=14. Because of the, Either implement a proper shading with package relocation or rely on transitive dependency, but do not shade non-repackaged versions of libraries. > Guava classpath conflict caused by kylin-jdbc 3.1.0 jar > --- > > Key: KYLIN-4656 > URL: https://issues.apache.org/jira/browse/KYLIN-4656 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v3.1.0 >Reporter: Gabor Arki >Priority: Critical > > The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged > version of the Guava library. This is causing class duplication with the > original guava jar if it is also on the classpath which results in > non-deterministic, runtime errors depending on which version of a certain > guava class has been picked up by the class-loader from the 2 versions. Based > on the runtime errors of the missing classes and methods, it seems to be a > very old version, probably <=14. > > Either implement a proper shading with package relocation or rely on > transitive dependency, but do not shade non-repackaged versions of libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4500) Timeout waiting for connection from pool
[
https://issues.apache.org/jira/browse/KYLIN-4500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162898#comment-17162898
]
Gabor Arki edited comment on KYLIN-4500 at 7/22/20, 4:21 PM:
-
[~hit_lacus], the linked issue seems to be unrelated. But if the dictionary is
stored on the cluster, it could be related, however I do not see any
FileNotFoundException logs when we are hitting this issue. I do see the slow
ramp-up in CLOSE_WAIT connections though on the server.
We are running Kylin on AWS EMR cluster and use S3 (EMRFS) for data storage
instead of HDFS to make the cluster stateless. However, after some continuous
uptime, we are always facing this issue where both the query server and the
Kylin MR jobs are suddenly failing with the aforementioned Exception. The root
cause of these failures is that the connection pool of the EMR cluster to S3 is
exhausted and new operations fail to acquire a connection and time out while
waiting for an S3 connection.
No matter how much of a pool size we are configuring for the
fs.s3.maxConnections value, this keeps happening. The underlying issue is very
likely a connection leak where some code is not properly closing and returning
a connection to the pool. Given a query server restart is solving the issue, I
suspect the pool is exhausted somewhere in the Kylin query server code.
was (Author: arkigabor):
[~hit_lacus], the linked issue seems to be unrelated.
We are running Kylin on AWS EMR cluster and use S3 (EMRFS) for data storage
instead of HDFS to make the cluster stateless. However, after some continuous
uptime, we are always facing this issue where both the query server and the
Kylin MR jobs are suddenly failing with the aforementioned Exception. The root
cause of these failures is that the connection pool of the EMR cluster to S3 is
exhausted and new operations fail to acquire a connection and time out while
waiting for an S3 connection.
No matter how much of a pool size we are configuring for the
fs.s3.maxConnections value, this keeps happening. The underlying issue is very
likely a connection leak where some code is not properly closing and returning
a connection to the pool. Given a query server restart is solving the issue, I
suspect the pool is exhausted somewhere in the Kylin query server code.
> Timeout waiting for connection from pool
>
>
> Key: KYLIN-4500
> URL: https://issues.apache.org/jira/browse/KYLIN-4500
> Project: Kylin
> Issue Type: Bug
>Reporter: Gabor Arki
>Priority: Major
> Attachments: kylin-connection-timeout.txt
>
>
> h4. Environment
> * Kylin server 3.0.0
> * EMR 5.28
> h4. Issue
> After an extended uptime, both Kylin query server and jobs running on EMR
> stop working. The root cause in both cases is:
> {noformat}
> Caused by: java.io.IOException:
> com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable
> to execute HTTP request: Timeout waiting for connection from pool
> at
> com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
> ~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
> Based on
> [https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
> increasing the fs.s3.maxConnections setting to 1 is just delaying the
> issue thus the underlying issue is likely a connection leak. It also
> indicates a leak that restarting the kylin service solves the problem.
> A full stack trace from the QueryService is attached.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4500) Timeout waiting for connection from pool
[
https://issues.apache.org/jira/browse/KYLIN-4500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162925#comment-17162925
]
Gabor Arki commented on KYLIN-4500:
---
For now, I will keep monitoring our server with netstat and try to determine
whether there is any correlation with the S3 pool exhaustion. Also, we will try
to upgrade to 3.1.0 but will probably take some time to tell whether the issue
is still reproducible with that version. I will post an update with our
findings once I have them.
> Timeout waiting for connection from pool
>
>
> Key: KYLIN-4500
> URL: https://issues.apache.org/jira/browse/KYLIN-4500
> Project: Kylin
> Issue Type: Bug
>Reporter: Gabor Arki
>Priority: Major
> Attachments: kylin-connection-timeout.txt
>
>
> h4. Environment
> * Kylin server 3.0.0
> * EMR 5.28
> h4. Issue
> After an extended uptime, both Kylin query server and jobs running on EMR
> stop working. The root cause in both cases is:
> {noformat}
> Caused by: java.io.IOException:
> com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable
> to execute HTTP request: Timeout waiting for connection from pool
> at
> com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
> ~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
> Based on
> [https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
> increasing the fs.s3.maxConnections setting to 1 is just delaying the
> issue thus the underlying issue is likely a connection leak. It also
> indicates a leak that restarting the kylin service solves the problem.
> A full stack trace from the QueryService is attached.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] coveralls commented on pull request #1325: KYLIN-4653 Make the capacity for the LinkedBlockingQueue of BlockingR…
coveralls commented on pull request #1325: URL: https://github.com/apache/kylin/pull/1325#issuecomment-662616156 ## Pull Request Test Coverage Report for [Build 6170](https://coveralls.io/builds/32256498) * **4** of **4** **(100.0%)** changed or added relevant lines in **1** file are covered. * No unchanged relevant lines lost coverage. * Overall coverage increased (+**0.005%**) to **28.051%** --- | Totals | [](https://coveralls.io/builds/32256498) | | :-- | --: | | Change from base [Build 6168](https://coveralls.io/builds/32224888): | 0.005% | | Covered Lines: | 26247 | | Relevant Lines: | 93568 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4653) Make the capacity for the LinkedBlockingQueue of BlockingReservoir configurable
[ https://issues.apache.org/jira/browse/KYLIN-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163003#comment-17163003 ] ASF GitHub Bot commented on KYLIN-4653: --- coveralls commented on pull request #1325: URL: https://github.com/apache/kylin/pull/1325#issuecomment-662616156 ## Pull Request Test Coverage Report for [Build 6170](https://coveralls.io/builds/32256498) * **4** of **4** **(100.0%)** changed or added relevant lines in **1** file are covered. * No unchanged relevant lines lost coverage. * Overall coverage increased (+**0.005%**) to **28.051%** --- | Totals | [](https://coveralls.io/builds/32256498) | | :-- | --: | | Change from base [Build 6168](https://coveralls.io/builds/32224888): | 0.005% | | Covered Lines: | 26247 | | Relevant Lines: | 93568 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make the capacity for the LinkedBlockingQueue of BlockingReservoir > configurable > --- > > Key: KYLIN-4653 > URL: https://issues.apache.org/jira/browse/KYLIN-4653 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4656) Guava classpath conflict caused by kylin-jdbc 3.1.0 jar
[ https://issues.apache.org/jira/browse/KYLIN-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163171#comment-17163171 ] Yaqian Zhang commented on KYLIN-4656: - Hi Gabor： I don't seem to understand what you mean. Is there an error log to let us know the details? kylin-jdbc 3.1.0.jar use a shaded guava replaced guava, and the shaded guava is 28.2-jre. > Guava classpath conflict caused by kylin-jdbc 3.1.0 jar > --- > > Key: KYLIN-4656 > URL: https://issues.apache.org/jira/browse/KYLIN-4656 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v3.1.0 >Reporter: Gabor Arki >Priority: Critical > > The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged > version of the Guava library. This is causing class duplication with the > original guava jar if it is also on the classpath which results in > non-deterministic, runtime errors depending on which version of a certain > guava class has been picked up by the class-loader from the 2 versions. Based > on the runtime errors of the missing classes and methods, it seems to be a > very old version, probably <=14. > > Either implement a proper shading with package relocation or rely on > transitive dependency, but do not shade non-repackaged versions of libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4645) Start kylin3.1.0-hadoop3 failed due to Guava version conflict with query-module
[
https://issues.apache.org/jira/browse/KYLIN-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163173#comment-17163173
]
zhou pengbo commented on KYLIN-4645:
when I copy our cluster guava-28.0-jre.jar to kylin/tool/ and kylin/tomcat/lib
directories，kylin can be started.
> Start kylin3.1.0-hadoop3 failed due to Guava version conflict with
> query-module
> ---
>
> Key: KYLIN-4645
> URL: https://issues.apache.org/jira/browse/KYLIN-4645
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v3.1.0
> Environment: Kylin：apache-kylin-3.1.0-bin-hadoop3
> HDP-3.1.4.0-315 with Hadoop(3.1.1) and Hive(3.1.0) and Spark2(2.3.2) and
> Hbase(2.0.2) .And My cluster guava version is guava-28.0-jre。
>Reporter: zhou pengbo
>Priority: Major
> Attachments: Chat-post.png, error-log.png, kylin-guava.png,
> kylin.log, kylin.out
>
>
> this is my development environment ：
> HDP-3.1.4.0-315 with Hadoop(3.1.1) and Hive(3.1.0) and Spark2(2.3.2) and
> Hbase(2.0.2) .
> our HDP cluster guava version all is guava-28.0-jre.
> when I run bin/kylin.sh start , the error appear ：
> {code:java}
> 2020-07-16 18:07:56,373 DEBUG [main] common.KylinConfig:346 : KYLIN_CONF
> property was not set, will seek KYLIN_HOME env variable2020-07-16
> 18:07:56,373 DEBUG [main] common.KylinConfig:346 : KYLIN_CONF property was
> not set, will seek KYLIN_HOME env variable2020-07-16 18:07:56,375 INFO
> [main] common.KylinConfig:352 : Use
> KYLIN_HOME=/home/kylin/apache-kylin-3.1.0-bin-hadoop32020-07-16 18:07:56,378
> INFO [main] common.KylinConfig:153 : Initialized a new KylinConfig from
> getInstanceFromEnv : 79673072020-07-16 18:07:56,417 INFO [main]
> persistence.ResourceStore:90 : Using metadata url kylin_metadata@hbase for
> resource storeException in thread "main" java.lang.IllegalArgumentException:
> Failed to find metadata store by url: kylin_metadata@hbase at
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:101)
> at
> org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:113)
> at
> org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:99)
> at
> org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:43)Caused
> by: java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:94)
> ... 3 moreCaused by: java.lang.NoSuchMethodError:
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1358) at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1339) at
> org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:77) at
> org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:63)
> at
> org.apache.kylin.storage.hbase.HBaseConnection.newHBaseConfiguration(HBaseConnection.java:170)
> at
> org.apache.kylin.storage.hbase.HBaseConnection.get(HBaseConnection.java:259)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.getConnection(HBaseResourceStore.java:96)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:119)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:89)
> ... 8 more2020-07-16 18:07:56,630 INFO [close-hbase-conn]
> hbase.HBaseConnection:138 : Closing HBase connections...ERROR: Unknown error.
> Please check full log.{code}
> then I chat with shaofeng_shi@kyligence with this issue， confirm the
> problem maybe:
> in kylin-3.1.0, have fix the issue KYLIN-4394 about the problem of guava
> version , and add shade-guava version in kylin-3.1.0. but in the
> query-module ， because of calcite dependent on guava-14.0, so now kylin
> depent on two version of guava . thus, Caused the above problem. but This is
> also a guess.
> Related Links ：
> [https://www.mail-archive.com/search?l=u...@kylin.apache.org&q=subject:%22Re%5C%3A%22&o=newest&f=1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4655) Load HFile to HBase Table step is loading the files over and over
[ https://issues.apache.org/jira/browse/KYLIN-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BKRV updated KYLIN-4655: Attachment: logs.zip > Load HFile to HBase Table step is loading the files over and over > - > > Key: KYLIN-4655 > URL: https://issues.apache.org/jira/browse/KYLIN-4655 > Project: Kylin > Issue Type: Bug > Components: Storage - HBase >Affects Versions: v2.6.5 > Environment: Production >Reporter: BKRV >Priority: Major > Attachments: logs.zip > > > When the cube is created using Spark engine selection load Hfile to Hbase > step is required to copy ~9 GB data. But in turn the job runs in a never > ending loop and the files are getting copied over several times. Even after > discarding the cube the job keeps running in backend. > Hbase master service is required to be rebooted for this stop. > Unable to identify the root cause. > When running using MR engine do not encounter this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4655) Load HFile to HBase Table step is loading the files over and over
[
https://issues.apache.org/jira/browse/KYLIN-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163196#comment-17163196
]
BKRV commented on KYLIN-4655:
-
Have uploaded kylin logs.
Just so I understand it right, if the "{color:#44}Convert Cuboid Data to
HFile{color}" creates 8 GB of data...am I to expect that "{color:#44}Load
HFile to HBase Table{color}" should also have 8 GB output?
> Load HFile to HBase Table step is loading the files over and over
> -
>
> Key: KYLIN-4655
> URL: https://issues.apache.org/jira/browse/KYLIN-4655
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: v2.6.5
> Environment: Production
>Reporter: BKRV
>Priority: Major
> Attachments: logs.zip
>
>
> When the cube is created using Spark engine selection load Hfile to Hbase
> step is required to copy ~9 GB data. But in turn the job runs in a never
> ending loop and the files are getting copied over several times. Even after
> discarding the cube the job keeps running in backend.
> Hbase master service is required to be rebooted for this stop.
> Unable to identify the root cause.
> When running using MR engine do not encounter this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] xiaowangyu commented on pull request #1299: KYLIN-4613 add build cubev2 and JobRestClient
xiaowangyu commented on pull request #1299: URL: https://github.com/apache/kylin/pull/1299#issuecomment-662824201 > @xiaowangyu how do you see. I think the change of JobRestClient and CubeBuildingCLIV2 can be update in original class may be better. Compatible with the original implementation. For users, the interface is unified. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4613) add buildCubeCLi as hadoop main class and jobRestClient
[ https://issues.apache.org/jira/browse/KYLIN-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163225#comment-17163225 ] ASF GitHub Bot commented on KYLIN-4613: --- xiaowangyu commented on pull request #1299: URL: https://github.com/apache/kylin/pull/1299#issuecomment-662824201 > @xiaowangyu how do you see. I think the change of JobRestClient and CubeBuildingCLIV2 can be update in original class may be better. Compatible with the original implementation. For users, the interface is unified. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > add buildCubeCLi as hadoop main class and jobRestClient > - > > Key: KYLIN-4613 > URL: https://issues.apache.org/jira/browse/KYLIN-4613 > Project: Kylin > Issue Type: Improvement > Components: Client - CLI >Reporter: chuxiao >Assignee: chuxiao >Priority: Minor > > support submit job and wait finish. retry 3 times when error -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4656) Guava classpath conflict caused by kylin-jdbc 3.1.0 jar
[ https://issues.apache.org/jira/browse/KYLIN-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Arki updated KYLIN-4656: -- Attachment: image-2020-07-23-07-44-40-675.png > Guava classpath conflict caused by kylin-jdbc 3.1.0 jar > --- > > Key: KYLIN-4656 > URL: https://issues.apache.org/jira/browse/KYLIN-4656 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v3.1.0 >Reporter: Gabor Arki >Priority: Critical > Attachments: image-2020-07-23-07-44-40-675.png > > > The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged > version of the Guava library. This is causing class duplication with the > original guava jar if it is also on the classpath which results in > non-deterministic, runtime errors depending on which version of a certain > guava class has been picked up by the class-loader from the 2 versions. Based > on the runtime errors of the missing classes and methods, it seems to be a > very old version, probably <=14. > > Either implement a proper shading with package relocation or rely on > transitive dependency, but do not shade non-repackaged versions of libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4656) Guava classpath conflict caused by kylin-jdbc 3.1.0 jar
[ https://issues.apache.org/jira/browse/KYLIN-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163235#comment-17163235 ] Gabor Arki commented on KYLIN-4656: --- Download the published 3.1.0 jar: [https://mvnrepository.com/artifact/org.apache.kylin/kylin-jdbc/3.1.0] Then unzip the file. You will see the com.google package inside the jar containing the guava library within the kylin-jdbc jar. !image-2020-07-23-07-44-40-675.png! If you also have the real guava jar on your classpath, you end up having 2 separate and highly incompatible versions of guava classes present, and the class-loader is loading one of them randomly causing numerous runtime issues. > Guava classpath conflict caused by kylin-jdbc 3.1.0 jar > --- > > Key: KYLIN-4656 > URL: https://issues.apache.org/jira/browse/KYLIN-4656 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v3.1.0 >Reporter: Gabor Arki >Priority: Critical > Attachments: image-2020-07-23-07-44-40-675.png > > > The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged > version of the Guava library. This is causing class duplication with the > original guava jar if it is also on the classpath which results in > non-deterministic, runtime errors depending on which version of a certain > guava class has been picked up by the class-loader from the 2 versions. Based > on the runtime errors of the missing classes and methods, it seems to be a > very old version, probably <=14. > > Either implement a proper shading with package relocation or rely on > transitive dependency, but do not shade non-repackaged versions of libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4656) Guava classpath conflict caused by kylin-jdbc 3.1.0 jar
[ https://issues.apache.org/jira/browse/KYLIN-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163235#comment-17163235 ] Gabor Arki edited comment on KYLIN-4656 at 7/23/20, 5:47 AM: - Download the published 3.1.0 jar: [https://mvnrepository.com/artifact/org.apache.kylin/kylin-jdbc/3.1.0] Then unzip the file. You will see the com.google package inside the jar containing the guava library within the kylin-jdbc jar. From my findings, this is likely version 14 of guava. !image-2020-07-23-07-44-40-675.png! If you also have the real guava jar on your classpath, you end up having 2 separate and highly incompatible versions of guava classes present, and the class-loader is loading one of them randomly causing numerous runtime issues. was (Author: arkigabor): Download the published 3.1.0 jar: [https://mvnrepository.com/artifact/org.apache.kylin/kylin-jdbc/3.1.0] Then unzip the file. You will see the com.google package inside the jar containing the guava library within the kylin-jdbc jar. !image-2020-07-23-07-44-40-675.png! If you also have the real guava jar on your classpath, you end up having 2 separate and highly incompatible versions of guava classes present, and the class-loader is loading one of them randomly causing numerous runtime issues. > Guava classpath conflict caused by kylin-jdbc 3.1.0 jar > --- > > Key: KYLIN-4656 > URL: https://issues.apache.org/jira/browse/KYLIN-4656 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v3.1.0 >Reporter: Gabor Arki >Priority: Critical > Attachments: image-2020-07-23-07-44-40-675.png > > > The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged > version of the Guava library. This is causing class duplication with the > original guava jar if it is also on the classpath which results in > non-deterministic, runtime errors depending on which version of a certain > guava class has been picked up by the class-loader from the 2 versions. Based > on the runtime errors of the missing classes and methods, it seems to be a > very old version, probably <=14. > > Either implement a proper shading with package relocation or rely on > transitive dependency, but do not shade non-repackaged versions of libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4656) Guava classpath conflict caused by kylin-jdbc 3.1.0 jar
[ https://issues.apache.org/jira/browse/KYLIN-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163235#comment-17163235 ] Gabor Arki edited comment on KYLIN-4656 at 7/23/20, 5:48 AM: - Download the published 3.1.0 jar: [https://mvnrepository.com/artifact/org.apache.kylin/kylin-jdbc/3.1.0] Then unzip the file. You will see the com.google package inside the jar containing the guava library within the kylin-jdbc jar. From my findings, this is likely version 14 of guava. !image-2020-07-23-07-44-40-675.png! If you also have the real guava jar on your classpath, you end up having 2 separate and highly incompatible versions of guava classes present, and the class-loader is loading one of the competing classes randomly causing numerous runtime issues. was (Author: arkigabor): Download the published 3.1.0 jar: [https://mvnrepository.com/artifact/org.apache.kylin/kylin-jdbc/3.1.0] Then unzip the file. You will see the com.google package inside the jar containing the guava library within the kylin-jdbc jar. From my findings, this is likely version 14 of guava. !image-2020-07-23-07-44-40-675.png! If you also have the real guava jar on your classpath, you end up having 2 separate and highly incompatible versions of guava classes present, and the class-loader is loading one of them randomly causing numerous runtime issues. > Guava classpath conflict caused by kylin-jdbc 3.1.0 jar > --- > > Key: KYLIN-4656 > URL: https://issues.apache.org/jira/browse/KYLIN-4656 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v3.1.0 >Reporter: Gabor Arki >Priority: Critical > Attachments: image-2020-07-23-07-44-40-675.png > > > The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged > version of the Guava library. This is causing class duplication with the > original guava jar if it is also on the classpath which results in > non-deterministic, runtime errors depending on which version of a certain > guava class has been picked up by the class-loader from the 2 versions. Based > on the runtime errors of the missing classes and methods, it seems to be a > very old version, probably <=14. > > Either implement a proper shading with package relocation or rely on > transitive dependency, but do not shade non-repackaged versions of libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4656) Guava classpath conflict caused by kylin-jdbc 3.1.0 jar
[ https://issues.apache.org/jira/browse/KYLIN-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163235#comment-17163235 ] Gabor Arki edited comment on KYLIN-4656 at 7/23/20, 5:52 AM: - Download the published 3.1.0 jar: [https://mvnrepository.com/artifact/org.apache.kylin/kylin-jdbc/3.1.0] Then unzip the file. You will see the com.google package inside the jar containing the guava library within the kylin-jdbc jar. From my findings, this is likely version 14 of guava. !image-2020-07-23-07-44-40-675.png! If you also have the real guava jar on your classpath, you end up having 2 separate and highly incompatible versions of guava classes present, and the class-loader is loading one of the competing classes randomly causing numerous runtime issues. Also, the shaded version you are referring too is also present in the jar and is properly relocated to the org.apache.kylin.shaded.com.google package and is fine as is. was (Author: arkigabor): Download the published 3.1.0 jar: [https://mvnrepository.com/artifact/org.apache.kylin/kylin-jdbc/3.1.0] Then unzip the file. You will see the com.google package inside the jar containing the guava library within the kylin-jdbc jar. From my findings, this is likely version 14 of guava. !image-2020-07-23-07-44-40-675.png! If you also have the real guava jar on your classpath, you end up having 2 separate and highly incompatible versions of guava classes present, and the class-loader is loading one of the competing classes randomly causing numerous runtime issues. > Guava classpath conflict caused by kylin-jdbc 3.1.0 jar > --- > > Key: KYLIN-4656 > URL: https://issues.apache.org/jira/browse/KYLIN-4656 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v3.1.0 >Reporter: Gabor Arki >Priority: Critical > Attachments: image-2020-07-23-07-44-40-675.png > > > The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged > version of the Guava library. This is causing class duplication with the > original guava jar if it is also on the classpath which results in > non-deterministic, runtime errors depending on which version of a certain > guava class has been picked up by the class-loader from the 2 versions. Based > on the runtime errors of the missing classes and methods, it seems to be a > very old version, probably <=14. > > Either implement a proper shading with package relocation or rely on > transitive dependency, but do not shade non-repackaged versions of libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4656) Guava classpath conflict caused by kylin-jdbc 3.1.0 jar
[ https://issues.apache.org/jira/browse/KYLIN-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163249#comment-17163249 ] Yaqian Zhang commented on KYLIN-4656: - Thank you for you reply. I see. This issue should be fix. > Guava classpath conflict caused by kylin-jdbc 3.1.0 jar > --- > > Key: KYLIN-4656 > URL: https://issues.apache.org/jira/browse/KYLIN-4656 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v3.1.0 >Reporter: Gabor Arki >Priority: Critical > Attachments: image-2020-07-23-07-44-40-675.png > > > The newly released kylin-jdbc 3.1.0 jar contains a shaded, non-repackaged > version of the Guava library. This is causing class duplication with the > original guava jar if it is also on the classpath which results in > non-deterministic, runtime errors depending on which version of a certain > guava class has been picked up by the class-loader from the 2 versions. Based > on the runtime errors of the missing classes and methods, it seems to be a > very old version, probably <=14. > > Either implement a proper shading with package relocation or rely on > transitive dependency, but do not shade non-repackaged versions of libraries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
