[jira] [Commented] (KYLIN-4268) build job failed caused by hadoop utils update
[
https://issues.apache.org/jira/browse/KYLIN-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048140#comment-17048140
]
ZhouKang commented on KYLIN-4268:
-
I think it's worked using solution 1.
So close this issue.
> build job failed caused by hadoop utils update
> --
>
> Key: KYLIN-4268
> URL: https://issues.apache.org/jira/browse/KYLIN-4268
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> When we update hadoop package, such as hbase and hive. The job will fill
> cause by:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.io.FileNotFoundException: File
> file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
> at
> org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
> {code}
>
> The root cause is kylin will generate the full command as soon as the job
> summited. And the parameter '--jars' use file's absolute path.
>
> there will be 2 ways to fix this problem.
> # keep the old package in the same path
> # change the generation of '--jars' to the stage of job execution.
>
> which is the better choose?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (KYLIN-4268) build job failed caused by hadoop utils update
[
https://issues.apache.org/jira/browse/KYLIN-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang closed KYLIN-4268.
---
Resolution: Abandoned
> build job failed caused by hadoop utils update
> --
>
> Key: KYLIN-4268
> URL: https://issues.apache.org/jira/browse/KYLIN-4268
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> When we update hadoop package, such as hbase and hive. The job will fill
> cause by:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.io.FileNotFoundException: File
> file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
> at
> org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
> {code}
>
> The root cause is kylin will generate the full command as soon as the job
> summited. And the parameter '--jars' use file's absolute path.
>
> there will be 2 ways to fix this problem.
> # keep the old package in the same path
> # change the generation of '--jars' to the stage of job execution.
>
> which is the better choose?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] shaofengshi opened a new pull request #1126: Docker
shaofengshi opened a new pull request #1126: Docker URL: https://github.com/apache/kylin/pull/1126 Update website with the new slogan, and the new docker document This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4378) Not able to build a cube with Kafka Streaming Source as fact table and Jdbc Source as lookup table.
[ https://issues.apache.org/jira/browse/KYLIN-4378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047822#comment-17047822 ] Sonu Singh commented on KYLIN-4378: --- Just for my understanding, I want to know is any specific reason for not supporting this functionality? > Not able to build a cube with Kafka Streaming Source as fact table and Jdbc > Source as lookup table. > --- > > Key: KYLIN-4378 > URL: https://issues.apache.org/jira/browse/KYLIN-4378 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v3.0.0 >Reporter: Sonu Singh >Priority: Blocker > Fix For: v3.1.0 > > > I am not able to build a cube with Kafka Streaming source and Jdbc Source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4378) Not able to build a cube with Kafka Streaming Source as fact table and Jdbc Source as lookup table.
[ https://issues.apache.org/jira/browse/KYLIN-4378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047820#comment-17047820 ] Sonu Singh commented on KYLIN-4378: --- Hi Shao, Thanks for the update. I will move my lookup table to hive. Thanks :) > Not able to build a cube with Kafka Streaming Source as fact table and Jdbc > Source as lookup table. > --- > > Key: KYLIN-4378 > URL: https://issues.apache.org/jira/browse/KYLIN-4378 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v3.0.0 >Reporter: Sonu Singh >Priority: Blocker > Fix For: v3.1.0 > > > I am not able to build a cube with Kafka Streaming source and Jdbc Source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4207) CubeMetadataValidator exception
[
https://issues.apache.org/jira/browse/KYLIN-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4207:
Fix Version/s: v3.0.0
> CubeMetadataValidator exception
> ---
>
> Key: KYLIN-4207
> URL: https://issues.apache.org/jira/browse/KYLIN-4207
> Project: Kylin
> Issue Type: Improvement
>Reporter: ZhouKang
>Assignee: ZhouKang
>Priority: Major
> Fix For: v3.0.0
>
>
> {code:java}
> // code placeholder
> 2019-10-28 11:19:08,583 ERROR [http-nio-7070-exec-9]
> validation.CubeMetadataValidator:56 : Construct cube metadata validator rule:
> org.apache.kylin.cube.model.validation.rule.CubeInfoRule failed. Ignore this
> rule
> java.lang.UnsupportedOperationException
> at java.util.AbstractList.add(AbstractList.java:148)
> at java.util.AbstractList.add(AbstractList.java:108)
> at
> org.apache.kylin.cube.model.validation.CubeMetadataValidator.(CubeMetadataValidator.java:54)
> {code}
> should not use Arrays.asList() in CubeMetadataValidator.java:49, which will
> cause UnsupportedOperationExcepiton
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KYLIN-4207) CubeMetadataValidator exception
[
https://issues.apache.org/jira/browse/KYLIN-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen resolved KYLIN-4207.
-
Resolution: Fixed
> CubeMetadataValidator exception
> ---
>
> Key: KYLIN-4207
> URL: https://issues.apache.org/jira/browse/KYLIN-4207
> Project: Kylin
> Issue Type: Improvement
>Reporter: ZhouKang
>Assignee: ZhouKang
>Priority: Major
> Fix For: v3.0.0
>
>
> {code:java}
> // code placeholder
> 2019-10-28 11:19:08,583 ERROR [http-nio-7070-exec-9]
> validation.CubeMetadataValidator:56 : Construct cube metadata validator rule:
> org.apache.kylin.cube.model.validation.rule.CubeInfoRule failed. Ignore this
> rule
> java.lang.UnsupportedOperationException
> at java.util.AbstractList.add(AbstractList.java:148)
> at java.util.AbstractList.add(AbstractList.java:108)
> at
> org.apache.kylin.cube.model.validation.CubeMetadataValidator.(CubeMetadataValidator.java:54)
> {code}
> should not use Arrays.asList() in CubeMetadataValidator.java:49, which will
> cause UnsupportedOperationExcepiton
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KYLIN-4341) by-level cuboid intermediate files are left behind and not cleaned up after job is complete
[
https://issues.apache.org/jira/browse/KYLIN-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangrupeng reassigned KYLIN-4341:
-
Assignee: wangrupeng
> by-level cuboid intermediate files are left behind and not cleaned up after
> job is complete
> ---
>
> Key: KYLIN-4341
> URL: https://issues.apache.org/jira/browse/KYLIN-4341
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.6.4
> Environment: Kylin 2.6.4, CenOS 7.6, HDP 2.6.5
>Reporter: Vsevolod Ostapenko
>Assignee: wangrupeng
>Priority: Major
>
> Setup: MR as a cube build engine and by-level cube build strategy (auto
> picked).
> Upon completion of a cube segment build job a number of intermediate files
> are still left behind.
> Namely, output of the MR-jobs that produce the base cuboid, subsequent level
> cuboids, as well as rowkey_stats from the hfile creation step.
> The files in question consume about the same amount of space in HDFS as the
> final hfile.
> This lead to wasted space in HDFS that is not released for as long as the
> corresponding cube segment is online. The only point the leaked space is
> released, is when segment is offlined and cleaned up as part of the segment
> retention.
> Sample output is as follows.
> {quote}$ hadoop fs -ls -R
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/
> drwxr-xr-x - kylin hdfs 0 2020-01-07 04:44
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89
> drwxr-xr-x - kylin hdfs 0 2020-01-07 04:26
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid
> drwxr-xr-x - kylin hdfs 0 2020-01-07 04:36
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid
> -rw-r--r-- 2 kylin hdfs 0 2020-01-07 04:36
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/_SUCCESS
> -rw-r--r-- 2 kylin hdfs 51570048 2020-01-07 04:35
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-0
> -rw-r--r-- 2 kylin hdfs 51477377 2020-01-07 04:36
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-1
> -rw-r--r-- 2 kylin hdfs 51615162 2020-01-07 04:35
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-2
> -rw-r--r-- 2 kylin hdfs 51591031 2020-01-07 04:36
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-3
> -rw-r--r-- 2 kylin hdfs 51648914 2020-01-07 04:35
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-4
> -rw-r--r-- 2 kylin hdfs 51532761 2020-01-07 04:36
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-5
> -rw-r--r-- 2 kylin hdfs 51455652 2020-01-07 04:35
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-6
> -rw-r--r-- 2 kylin hdfs 51552752 2020-01-07 04:36
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-7
> drwxr-xr-x - kylin hdfs 0 2020-01-07 04:25
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid
> -rw-r--r-- 2 kylin hdfs 0 2020-01-07 04:25
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid/_SUCCESS
> -rw-r--r-- 2 kylin hdfs 16293012 2020-01-07 04:25
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid/part-r-0
> -rw-r--r-- 2 kylin hdfs 16283730 2020-01-07 04:25
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid/part-r-1
> -rw-r--r-- 2 kylin hdfs 16288965 2020-01-07 04:25
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid/part-r-2
> -rw-r--r-- 2 kylin hdfs 16270572 2020-01-07 04:25
> /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid/part-r-3
> drwxr-xr-x - kylin hdfs 0 2020-01-07 04:23
> /user/kylin/26x/test_kylin_26x_metadata/kyli
[jira] [Created] (KYLIN-4392) Bump Flink version from 1.9.2 to 1.10.0
vinoyang created KYLIN-4392: --- Summary: Bump Flink version from 1.9.2 to 1.10.0 Key: KYLIN-4392 URL: https://issues.apache.org/jira/browse/KYLIN-4392 Project: Kylin Issue Type: Sub-task Components: Flink Engine Reporter: vinoyang Assignee: vinoyang Recently, Flink has release v1.10.0. It's a major version that contains some great features and big changes. We can tentatively upgrade to v1.10.0 after we upgrade to v1.9.2 firstly. If we find some fatal issues, we can fall back to v1.9.2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4391) Bump Flink version from 1.9.0 to 1.9.2
vinoyang created KYLIN-4391: --- Summary: Bump Flink version from 1.9.0 to 1.9.2 Key: KYLIN-4391 URL: https://issues.apache.org/jira/browse/KYLIN-4391 Project: Kylin Issue Type: Sub-task Components: Flink Engine Reporter: vinoyang Assignee: vinoyang Flink 1.9.2 is a stable version in 1.9 series. We should upgrade to this version for flink engine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4348) Fix distributed concurrency lock bug
[ https://issues.apache.org/jira/browse/KYLIN-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4348: Fix Version/s: v3.1.0 > Fix distributed concurrency lock bug > > > Key: KYLIN-4348 > URL: https://issues.apache.org/jira/browse/KYLIN-4348 > Project: Kylin > Issue Type: Sub-task >Reporter: wangxiaojing >Assignee: wangxiaojing >Priority: Major > Fix For: v3.1.0 > > Attachments: image-2020-02-03-10-54-21-976.png, > image-2020-02-03-10-54-53-468.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3861) No match found for function signature COUNT_DISTINCT()
[ https://issues.apache.org/jira/browse/KYLIN-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3861. Resolution: Incomplete > No match found for function signature COUNT_DISTINCT() > --- > > Key: KYLIN-3861 > URL: https://issues.apache.org/jira/browse/KYLIN-3861 > Project: Kylin > Issue Type: Improvement >Reporter: jinguowei >Assignee: jinguowei >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3758) Flink Cube Build Engine
[ https://issues.apache.org/jira/browse/KYLIN-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi updated KYLIN-3758: - Fix Version/s: v3.1.0 > Flink Cube Build Engine > --- > > Key: KYLIN-3758 > URL: https://issues.apache.org/jira/browse/KYLIN-3758 > Project: Kylin > Issue Type: New Feature > Components: Flink Engine >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Fix For: v3.1.0 > > > Kylin currently supports Apache Spark as the engine for building Cubes. As a > next-generation computing engine, Apache Flink is dedicated to the > convergence of streams and batches, with superior performance and elegant > architecture. Flink and Kylin have many overlapping users in China. If Kylin > can support Flink as the engine for building Cube, then this will give many > users more choices. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3992) easy exceeds 'max-return-row' when gouping by a high-cardinality dimension with a long day span filter
[
https://issues.apache.org/jira/browse/KYLIN-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shao Feng Shi closed KYLIN-3992.
Resolution: Won't Fix
> easy exceeds 'max-return-row' when gouping by a high-cardinality dimension
> with a long day span filter
> --
>
> Key: KYLIN-3992
> URL: https://issues.apache.org/jira/browse/KYLIN-3992
> Project: Kylin
> Issue Type: Bug
>Reporter: hejian
>Priority: Major
>
> In my case, a dimension with 3million cardinality called seller_id,
> {code:java}
> select seller_id, sum(volume) as count_num from tab_orders where
> day_id>='2019-04-07' and day_id<'2019-04-27' group by seller_id order by
> count_num desc offset 0 limit 10{code}
> I think it relates to both the *cardinality(seller_id)* and
> *cardinality(day_id)* ?
> Hard to determine the suitable value to 'kylin.query.max-return-row', any
> good solution to handle this issue?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-3992) easy exceeds 'max-return-row' when gouping by a high-cardinality dimension with a long day span filter
[
https://issues.apache.org/jira/browse/KYLIN-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047672#comment-17047672
]
Shao Feng Shi commented on KYLIN-3992:
--
Add some filtering conditions on the high cardinality dimension; or use TopN to
cut the long tails.
> easy exceeds 'max-return-row' when gouping by a high-cardinality dimension
> with a long day span filter
> --
>
> Key: KYLIN-3992
> URL: https://issues.apache.org/jira/browse/KYLIN-3992
> Project: Kylin
> Issue Type: Bug
>Reporter: hejian
>Priority: Major
>
> In my case, a dimension with 3million cardinality called seller_id,
> {code:java}
> select seller_id, sum(volume) as count_num from tab_orders where
> day_id>='2019-04-07' and day_id<'2019-04-27' group by seller_id order by
> count_num desc offset 0 limit 10{code}
> I think it relates to both the *cardinality(seller_id)* and
> *cardinality(day_id)* ?
> Hard to determine the suitable value to 'kylin.query.max-return-row', any
> good solution to handle this issue?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-3924) Build cube on step 'build Dimension Dictionary ', the contents of non-dimensional columns are also checked and built
[ https://issues.apache.org/jira/browse/KYLIN-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047670#comment-17047670 ] Shao Feng Shi commented on KYLIN-3924: -- [~fxjwind] The lookup table snapshot is shared across cubes/models; So it is captured on all columns. In your case, I suggest to create a hive view, hidding those big columns, and then use the view as the lookup table. Kylin will materialize the view during the build. > Build cube on step 'build Dimension Dictionary ', the contents of > non-dimensional columns are also checked and built > > > Key: KYLIN-3924 > URL: https://issues.apache.org/jira/browse/KYLIN-3924 > Project: Kylin > Issue Type: Bug > Components: Job Engine, Tools, Build and Test >Affects Versions: v2.6.1 >Reporter: wangxiaojing >Priority: Major > Labels: build, patch > Attachments: builderr.log > > > Hi all, > I use version 2.6.1 , when build cube on step 'build Dimension Dictionary' > for lookup tables ,it throws some exceptions,it seems checks the content size > of each field in the table while building Dimension Dictionary, and build all > fields in the table with a dictionary. > Suggestion: > 1. Large content fields may not be dimension columns, in fact, it is not > necessary to check the value size of non-dimension columns，only check > dimension columns. > 2. If the value of some fields in the dimension table is too long, the > exception is not clear, can not be located the problem column quickly. > 3. There is no need to build dictionaries for non-dimensional columns -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3924) Build cube on step 'build Dimension Dictionary ', the contents of non-dimensional columns are also checked and built
[ https://issues.apache.org/jira/browse/KYLIN-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3924. Resolution: Won't Fix > Build cube on step 'build Dimension Dictionary ', the contents of > non-dimensional columns are also checked and built > > > Key: KYLIN-3924 > URL: https://issues.apache.org/jira/browse/KYLIN-3924 > Project: Kylin > Issue Type: Bug > Components: Job Engine, Tools, Build and Test >Affects Versions: v2.6.1 >Reporter: wangxiaojing >Priority: Major > Labels: build, patch > Attachments: builderr.log > > > Hi all, > I use version 2.6.1 , when build cube on step 'build Dimension Dictionary' > for lookup tables ,it throws some exceptions,it seems checks the content size > of each field in the table while building Dimension Dictionary, and build all > fields in the table with a dictionary. > Suggestion: > 1. Large content fields may not be dimension columns, in fact, it is not > necessary to check the value size of non-dimension columns，only check > dimension columns. > 2. If the value of some fields in the dimension table is too long, the > exception is not clear, can not be located the problem column quickly. > 3. There is no need to build dictionaries for non-dimensional columns -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3954) kylin one instance log :executablePO is null
[ https://issues.apache.org/jira/browse/KYLIN-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3954. Resolution: Incomplete > kylin one instance log :executablePO is null > > > Key: KYLIN-3954 > URL: https://issues.apache.org/jira/browse/KYLIN-3954 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 > Environment: cdh57-hadoop >Reporter: leh >Priority: Major > Labels: starter > Original Estimate: 96h > Remaining Estimate: 96h > > in kylin instance, the kylin.log print mang WARN log, like this: > -- > 2019-04-12 11:56:38,636 WARN [http-bio-8070-exec-12] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:38,641 WARN [http-bio-8070-exec-11] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:38,786 WARN [http-bio-8070-exec-10] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:38,787 WARN [http-bio-8070-exec-4] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:38,937 WARN [http-bio-8070-exec-15] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:38,937 WARN [http-bio-8070-exec-14] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,088 WARN [http-bio-8070-exec-9] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,088 WARN [http-bio-8070-exec-13] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,238 WARN [http-bio-8070-exec-5] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,238 WARN [http-bio-8070-exec-6] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,398 WARN [http-bio-8070-exec-8] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,398 WARN [http-bio-8070-exec-3] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,549 WARN [http-bio-8070-exec-2] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,549 WARN [http-bio-8070-exec-7] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,699 WARN [http-bio-8070-exec-1] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,702 WARN [http-bio-8070-exec-12] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,851 WARN [http-bio-8070-exec-11] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:39,852 WARN [http-bio-8070-exec-10] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,001 WARN [http-bio-8070-exec-4] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,003 WARN [http-bio-8070-exec-15] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,152 WARN [http-bio-8070-exec-14] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,157 WARN [http-bio-8070-exec-9] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,305 WARN [http-bio-8070-exec-13] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,308 WARN [http-bio-8070-exec-5] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,455 WARN [http-bio-8070-exec-6] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,458 WARN [http-bio-8070-exec-8] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,605 WARN [http-bio-8070-exec-3] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,609 WARN [http-bio-8070-exec-2] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,754 WARN [http-bio-8070-exec-7] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,759 WARN [http-bio-8070-exec-1] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,903 WARN [http-bio-8070-exec-12] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:40,911 WARN [http-bio-8070-exec-11] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:41,053 WARN [http-bio-8070-exec-10] > execution.ExecutableManager:563 : executablePO is null > 2019-04-12 11:56:41,061 WARN [http-bio-8070-exec-4] > execution.ExecutableManager:563 : executablePO is null > - > what's wrong? how to resolve it? so many WARN infos print frequently. > memo: > cluster is three kylin instance, two instance is "kylin.server.mode=all" ,the > last one is query. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4018) apache kylin 连接powerbi出错
[ https://issues.apache.org/jira/browse/KYLIN-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4018. Resolution: Won't Fix > apache kylin 连接powerbi出错 > > > Key: KYLIN-4018 > URL: https://issues.apache.org/jira/browse/KYLIN-4018 > Project: Kylin > Issue Type: Wish > Components: Driver - ODBC > Environment: apache kylin，powerbi desktop >Reporter: jamie_lee >Priority: Major > > 我在将apache kylin的数据源连接到powebi > desktop时使用了KylinODBCDriver的驱动，好像只支持导入模式，不支持directquery模式，以至于虽然连接成功了，但是导入的数据好像格式有些问题(有些类型为bigint的字段显示为0)，所以现在apache > kylin和powerbi之间的odbc驱动不支持directquery模式了么？或者是不是我现在用的驱动不对？我应该用哪个驱动呢？求解答，谢谢！ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4048) Too long spark cube building time for too many eviction and loading for dict slices
[
https://issues.apache.org/jira/browse/KYLIN-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shao Feng Shi closed KYLIN-4048.
Resolution: Fixed
> Too long spark cube building time for too many eviction and loading for dict
> slices
> ---
>
> Key: KYLIN-4048
> URL: https://issues.apache.org/jira/browse/KYLIN-4048
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v2.5.2
>Reporter: Shaohui Liu
>Priority: Major
>
> In our cluster, a cube building costs too long. In the log for spark, we
> found there are too many eviction and loading for dict slices in
> AppendTrieDictionary.
> {code:java}
> $ grep "read slice from" spark.log | wc -l
> 119721
> $ grep "Evict slice with key" spark.log| wc -l
> 119634
> {code}
> The reason is that the memory of spark executor(4G) is not enough to hold all
> the slices of dict(3.3G in hdfs) in memory, which cause the bad performance
> of cube building.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (KYLIN-3953) cube run many building instances，but dict file locked，failed to acquire lock
[ https://issues.apache.org/jira/browse/KYLIN-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3953. Resolution: Incomplete > cube run many building instances，but dict file locked，failed to acquire lock > > > Key: KYLIN-3953 > URL: https://issues.apache.org/jira/browse/KYLIN-3953 > Project: Kylin > Issue Type: Bug > Components: Storage - HBase >Affects Versions: v2.6.1 > Environment: cdh57-hadoop, kylin 2.6.1 >Reporter: leh >Priority: Major > Labels: build > Original Estimate: 96h > Remaining Estimate: 96h > > when i build the cube history data for many days,and run many instances in > monitor, but hang out the step" #4 Step Name: Build Dimension Dictionary" > kylin.log as follow: > 2019-04-12 11:20:28,519 DEBUG [Scheduler 635400189 Job > b66ccafa-a2ce-621d-b76d-bac2b16bcca3-240] util.ZookeeperDistributedLock:147 : > 240-156...@xxx.host-web1.com trying to lock > /kylin/kylin_metadata/dict/DEFAULT.APP_STRATEGY_ADS_VIEW_BASE_INFO_DI_DURATION_CUID/lock > 2019-04-12 11:20:28,575 DEBUG [Scheduler 635400189 Job > b66ccafa-a2ce-621d-b76d-bac2b16bcca3-240] util.ZookeeperDistributedLock:152 : > 240-156...@xxx.host-web1.com see > /kylin/kylin_metadata/dict/DEFAULT.APP_STRATEGY_ADS_VIEW_BASE_INFO_DI_DURATION_CUID/lock > is already locked > 2019-04-12 11:20:28,609 DEBUG [Scheduler 635400189 Job > b66ccafa-a2ce-621d-b76d-bac2b16bcca3-240] util.ZookeeperDistributedLock:162 : > 240-156...@xxx.host-web1.com failed to acquire lock at > /kylin/kylin_metadata/dict/DEFAULT.APP_STRATEGY_ADS_VIEW_BASE_INFO_DI_DURATION_CUID/lock, > which is held by 145-156...@xxx.host-web1.com > --- > kylin.properties: > ## Max count of concurrent jobs running > kylin.job.max-concurrent-jobs=50 > kylin.cube.max-building-segments=62 > --- > and the same cube ,can I build many instance parallismly? and how to avoid > this question? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4045) ParquetMROutput miss megering cuboid data step
[ https://issues.apache.org/jira/browse/KYLIN-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4045. Resolution: Won't Fix > ParquetMROutput miss megering cuboid data step > -- > > Key: KYLIN-4045 > URL: https://issues.apache.org/jira/browse/KYLIN-4045 > Project: Kylin > Issue Type: Bug > Components: Storage - Parquet >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: Future > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4189) Prevent abnormal CPU usage by limiting lookup table's rows
[
https://issues.apache.org/jira/browse/KYLIN-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047665#comment-17047665
]
Temple Zhou commented on KYLIN-4189:
The "group by" clause may cause hard computing to implement "count dinstinct".
If there are too many rows, CPU utility will skyrocket. :(
> Prevent abnormal CPU usage by limiting lookup table's rows
> --
>
> Key: KYLIN-4189
> URL: https://issues.apache.org/jira/browse/KYLIN-4189
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: v2.4.0, v2.5.0, v2.6.0
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Major
>
> Although lookup table size is limited, snapshot table with too many rows will
> break the server.
> Such as:
> {code:sql}
> select
> dim.b
> FROM
> fact
> INNER JOIN dim ON fact.B = dim.B
> group by
> dim.b
> {code}
> If the number of rows in column dim.B is million, the query will execute for
> dozens seconds and the CPU utilization is extremely high.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4048) Too long spark cube building time for too many eviction and loading for dict slices
[
https://issues.apache.org/jira/browse/KYLIN-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047664#comment-17047664
]
Shao Feng Shi commented on KYLIN-4048:
--
This should be related with the huge global dictionary. And the "shrunk
dictionary" has solved this problem I think. It was enabled by default in Kylin
2.6.
> Too long spark cube building time for too many eviction and loading for dict
> slices
> ---
>
> Key: KYLIN-4048
> URL: https://issues.apache.org/jira/browse/KYLIN-4048
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v2.5.2
>Reporter: Shaohui Liu
>Priority: Major
>
> In our cluster, a cube building costs too long. In the log for spark, we
> found there are too many eviction and loading for dict slices in
> AppendTrieDictionary.
> {code:java}
> $ grep "read slice from" spark.log | wc -l
> 119721
> $ grep "Evict slice with key" spark.log| wc -l
> 119634
> {code}
> The reason is that the memory of spark executor(4G) is not enough to hold all
> the slices of dict(3.3G in hdfs) in memory, which cause the bad performance
> of cube building.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-3679) Fetch Kafka topic with Spark streaming
[ https://issues.apache.org/jira/browse/KYLIN-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047660#comment-17047660 ] Shao Feng Shi commented on KYLIN-3679: -- Hi [~codingforfun] weibin, have you made some progress on this? I'm looking forward to this feature! > Fetch Kafka topic with Spark streaming > -- > > Key: KYLIN-3679 > URL: https://issues.apache.org/jira/browse/KYLIN-3679 > Project: Kylin > Issue Type: New Feature > Components: Spark Engine >Reporter: Shao Feng Shi >Assignee: weibin0516 >Priority: Major > > Now Kylin uses a MR job to fetch Kafka messages in parallel and then persist > to HDFS for subsequent processing. If user selects to use Spark engine, we > can use Spark streaming API to do this. Spark streaming can read the Kafka > message in a given offset range as a RDD, then it would be easy to process; > https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html > With Spark streaming, Kylin can also easily connect with other data source > like Kinesis, Flume, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4052) Improve compatibility with the Tableau 2019.x
[ https://issues.apache.org/jira/browse/KYLIN-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047662#comment-17047662 ] Shao Feng Shi commented on KYLIN-4052: -- Hello Temple, do you have plan on this? [~temple.zhou] > Improve compatibility with the Tableau 2019.x > - > > Key: KYLIN-4052 > URL: https://issues.apache.org/jira/browse/KYLIN-4052 > Project: Kylin > Issue Type: Improvement >Affects Versions: Future >Reporter: Temple Zhou >Assignee: Temple Zhou >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4088) Using metadata url kylin_metadata@hbase for resource store
[ https://issues.apache.org/jira/browse/KYLIN-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4088. Resolution: Not A Problem > Using metadata url kylin_metadata@hbase for resource store > -- > > Key: KYLIN-4088 > URL: https://issues.apache.org/jira/browse/KYLIN-4088 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.3 >Reporter: RandySun >Priority: Major > > I am using CDH 6.2 and trying to install apache-kylin-2.6.3-bin-cdh60.tar.gz > into my server > Env-check passed, however, when I entered the start command, I got the > following error: > persistence.ResourceStore:88 : Using metadata url kylin_metadata@hbase for > resource store > Exception in thread "main" java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > at > org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99) > at > org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:111) > at > org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:99) > at > org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:43) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:92) > ... 3 more > Caused by: java.lang.NoSuchMethodError: > com.ctc.wstx.stax.WstxInputFactory.createSR(Lcom/ctc/wstx/api/ReaderConfig;Lcom/ctc/wstx/io/SystemId;Lcom/ctc/wstx/io/InputBootstrapper;ZZ)Lorg/apache/phoenix/shaded/org/codehaus/stax2/XMLStreamReader2; > at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2803) > at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2787) > at > org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2838) > at > org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2812) > at > org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2689) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1160) > at > org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:69) > at > org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:59) > at > org.apache.kylin.storage.hbase.HBaseConnection.newHBaseConfiguration(HBaseConnection.java:169) > at > org.apache.kylin.storage.hbase.HBaseConnection.get(HBaseConnection.java:249) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.getConnection(HBaseResourceStore.java:95) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:114) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:88) > ... 8 more > 2019-07-16 14:36:57,963 INFO [close-hbase-conn] hbase.HBaseConnection:137 : > Closing HBase connections... > ERROR: Unknown error. Please check full log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4056) Distributed Scheduler support auto add/delete node and do not restart
[ https://issues.apache.org/jira/browse/KYLIN-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4056. Resolution: Duplicate > Distributed Scheduler support auto add/delete node and do not restart > - > > Key: KYLIN-4056 > URL: https://issues.apache.org/jira/browse/KYLIN-4056 > Project: Kylin > Issue Type: Improvement >Reporter: jinguowei >Assignee: jinguowei >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4056) Distributed Scheduler support auto add/delete node and do not restart
[ https://issues.apache.org/jira/browse/KYLIN-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047650#comment-17047650 ] Shao Feng Shi commented on KYLIN-4056: -- I think this has already been implemented with the Curator based scheduler, released in Kylin 3.0 > Distributed Scheduler support auto add/delete node and do not restart > - > > Key: KYLIN-4056 > URL: https://issues.apache.org/jira/browse/KYLIN-4056 > Project: Kylin > Issue Type: Improvement >Reporter: jinguowei >Assignee: jinguowei >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4109) CubeHFileMapperTest failed after commit: f4d2405f6aa978bbc3153c9ca9fa339b9d7e6c30
[
https://issues.apache.org/jira/browse/KYLIN-4109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047649#comment-17047649
]
Shao Feng Shi commented on KYLIN-4109:
--
Shaohui, any updates on this? [~liushaohui]
> CubeHFileMapperTest failed after commit:
> f4d2405f6aa978bbc3153c9ca9fa339b9d7e6c30
> -
>
> Key: KYLIN-4109
> URL: https://issues.apache.org/jira/browse/KYLIN-4109
> Project: Kylin
> Issue Type: Bug
>Reporter: Shaohui Liu
>Priority: Major
>
> CubeHFileMapperTest failed in 2.5.x-hadoop3.1 for following changes
> {code:java}
> - assertEquals("cf1", new String(p1.getSecond().getFamily(),
> StandardCharsets.UTF_8));
> - assertEquals("usd_amt", new String(p1.getSecond().getQualifier(),
> StandardCharsets.UTF_8));
> - assertEquals("35.43", new String(p1.getSecond().getValue(),
> StandardCharsets.UTF_8));
> + assertEquals("cf1", new String(copy(p1.getSecond(;
> + assertEquals("usd_amt", new String(copy(p1.getSecond(;
> + assertEquals("35.43", new String(copy(p1.getSecond(;
>
> assertEquals(key, p2.getFirst());
> - assertEquals("cf1", new String(p2.getSecond().getFamily(),
> StandardCharsets.UTF_8));
> - assertEquals("item_count", new String(p2.getSecond().getQualifier(),
> StandardCharsets.UTF_8));
> - assertEquals("2", new String(p2.getSecond().getValue(),
> StandardCharsets.UTF_8));
> + assertEquals("cf1", new String(copy(p2.getSecond(;
> + assertEquals("item_count", new String(copy(p2.getSecond(;
> + assertEquals("2", new String(copy(p2.getSecond(;
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KYLIN-4132) Kylin needn't use "org.apache.directory.api.util.Strings" to import api-util.jar
[ https://issues.apache.org/jira/browse/KYLIN-4132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangrupeng reassigned KYLIN-4132: - Assignee: wangrupeng > Kylin needn't use "org.apache.directory.api.util.Strings" to import > api-util.jar > > > Key: KYLIN-4132 > URL: https://issues.apache.org/jira/browse/KYLIN-4132 > Project: Kylin > Issue Type: Improvement > Components: REST Service > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: wangrupeng >Priority: Major > Original Estimate: 168h > Remaining Estimate: 168h > > Hi Team: > I read kylin code find that JobService class and ProjectService class import > "org.apache.directory.api.util.Strings" use the function of > "Strings.isEmpty(String)" from "api-util.jar" to judge whether a string is > empty. > This function can be replaced by importing > "org.apache.commons.lang.StringUtils" use "StringUtils.isEmpty(String)" from > “common-lang.jar”. > The "api-util.jar" was dependent by hbase-client. When kylin build&package, > there isn't "api-util.jar" in kylin.war. When start kylin process, it will be > imported from hbase client lib. But if hbase client doesn't dependency > "api-util.jar", kylin running failed and will throw exception. > In fact, I encountered this problem in the latest Fusion-Insignt version. > Kylin start failed because there isn't "api-util.jar" in hbase client lib. > > So I suggest replace "org.apache.directory.api.util.Strings.isEmpty(String)" > to "org.apache.commons.lang.StringUtils.isEmpty(String)" to avoid this > problem. > Best wishes > Yours zjt. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-2517) Upgrade hbase dependency to 1.4.7
[ https://issues.apache.org/jira/browse/KYLIN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-2517. Resolution: Won't Fix > Upgrade hbase dependency to 1.4.7 > - > > Key: KYLIN-2517 > URL: https://issues.apache.org/jira/browse/KYLIN-2517 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: weibin0516 >Priority: Major > > There have been major enhancements / bug fixes since the hbase 1.1.1 release. > This issue is to upgrade to 1.4.7 release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4144) Cannot update user's info because of case-sensitive process
[
https://issues.apache.org/jira/browse/KYLIN-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047641#comment-17047641
]
Shao Feng Shi commented on KYLIN-4144:
--
Have this been fixed?
> Cannot update user's info because of case-sensitive process
> ---
>
> Key: KYLIN-4144
> URL: https://issues.apache.org/jira/browse/KYLIN-4144
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
> Fix For: v2.5.3
>
>
> In https://issues.apache.org/jira/browse/KYLIN-3531 , uppercase of usernames
> will be saved in HBase, which make our case faild.
>
> when users was added useing kylin:v2.4.0 (maybe older), the rawkey in hbase
> is lowercase, when update user's info in Kylin:v2.5.3, there is en error.
> {code:java}
> // code placeholder
> Overwriting conflict /user/, expect old TS xxx, but it is 0, the expected
> new TS: {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4189) Prevent abnormal CPU usage by limiting lookup table's rows
[
https://issues.apache.org/jira/browse/KYLIN-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047625#comment-17047625
]
Shao Feng Shi commented on KYLIN-4189:
--
This is processing in memory I think. Anyway to prevent the high CPU?
> Prevent abnormal CPU usage by limiting lookup table's rows
> --
>
> Key: KYLIN-4189
> URL: https://issues.apache.org/jira/browse/KYLIN-4189
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: v2.4.0, v2.5.0, v2.6.0
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Major
>
> Although lookup table size is limited, snapshot table with too many rows will
> break the server.
> Such as:
> {code:sql}
> select
> dim.b
> FROM
> fact
> INNER JOIN dim ON fact.B = dim.B
> group by
> dim.b
> {code}
> If the number of rows in column dim.B is million, the query will execute for
> dozens seconds and the CPU utilization is extremely high.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4202) Tableau Server failed with "Access violation - no RTTI data!"
[ https://issues.apache.org/jira/browse/KYLIN-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047622#comment-17047622 ] Shao Feng Shi commented on KYLIN-4202: -- Temple, have you found a resolution? > Tableau Server failed with "Access violation - no RTTI data!" > - > > Key: KYLIN-4202 > URL: https://issues.apache.org/jira/browse/KYLIN-4202 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.0, v2.6.0, v3.0.0-alpha2 >Reporter: Temple Zhou >Assignee: Temple Zhou >Priority: Major > > I can open the reports using Tableau Desktop with *JDBC Driver*, but the > reposts will failed if I publish it to Tableau Server. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4230) 使用的过程中会导致死锁的情况
[ https://issues.apache.org/jira/browse/KYLIN-4230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4230. Resolution: Incomplete > 使用的过程中会导致死锁的情况 > -- > > Key: KYLIN-4230 > URL: https://issues.apache.org/jira/browse/KYLIN-4230 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.3.1 > Environment: kylin-2.3.1，centos 7 >Reporter: 谢沛辰 >Priority: Major > Attachments: lock.log > > > 创建模型后，立马创建立方体会导致死锁。 > 查看代码发现创建立方体会锁住cube，然后请求project的锁。而模型创建成功会去更新project缓存，这个会先锁住project，后期拿立方体信息，会去请求cube的锁。这样在一定情况下会导致资源竞争，最后导致整个系统阻塞的情况。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4231) write conflict while add model by different query server parallelly
[
https://issues.apache.org/jira/browse/KYLIN-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047620#comment-17047620
]
Shao Feng Shi commented on KYLIN-4231:
--
If possible, only route all write to one Kylin instance, and query to others
> write conflict while add model by different query server parallelly
> ---
>
> Key: KYLIN-4231
> URL: https://issues.apache.org/jira/browse/KYLIN-4231
> Project: Kylin
> Issue Type: New Feature
>Reporter: ZhouKang
>Priority: Major
>
> a kylin cluster have more than 1 query server, all of them are the backend
> server of nginx.
> when our user use RESTful API to create model in the *same* project
> parallelly, there will be a problem.
> the server returns:
> {code:java}
> // code placeholder
> Overwriting conflict /project/learn_kylin.json, expect old TS 1572596034269,
> but it is 1572596042929
> {code}
> BUT, the model '/model_desc/xxx.json' has already in metastore, so the next
> time our user want
> to retry (create model), he will get:
> {code:java}
> // code placeholder
> Overwriting conflict /model_desc/test_945.json, expect old TS 0, but it is
> 1572596193812
> {code}
> I think the problem is that:
> when server A receive PUT model, it will change the project info, but the
> cache update broadcast cannot be processed by server B before server B
> processing another model creation request.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KYLIN-4231) write conflict while add model by different query server parallelly
[
https://issues.apache.org/jira/browse/KYLIN-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shao Feng Shi resolved KYLIN-4231.
--
Resolution: Won't Fix
> write conflict while add model by different query server parallelly
> ---
>
> Key: KYLIN-4231
> URL: https://issues.apache.org/jira/browse/KYLIN-4231
> Project: Kylin
> Issue Type: New Feature
>Reporter: ZhouKang
>Priority: Major
>
> a kylin cluster have more than 1 query server, all of them are the backend
> server of nginx.
> when our user use RESTful API to create model in the *same* project
> parallelly, there will be a problem.
> the server returns:
> {code:java}
> // code placeholder
> Overwriting conflict /project/learn_kylin.json, expect old TS 1572596034269,
> but it is 1572596042929
> {code}
> BUT, the model '/model_desc/xxx.json' has already in metastore, so the next
> time our user want
> to retry (create model), he will get:
> {code:java}
> // code placeholder
> Overwriting conflict /model_desc/test_945.json, expect old TS 0, but it is
> 1572596193812
> {code}
> I think the problem is that:
> when server A receive PUT model, it will change the project info, but the
> cache update broadcast cannot be processed by server B before server B
> processing another model creation request.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (KYLIN-4016) Garbage Collection on HBase hbase数据存储在s3但是却找的是hdfs
[ https://issues.apache.org/jira/browse/KYLIN-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4016. > Garbage Collection on HBase hbase数据存储在s3但是却找的是hdfs > --- > > Key: KYLIN-4016 > URL: https://issues.apache.org/jira/browse/KYLIN-4016 > Project: Kylin > Issue Type: Bug > Components: Storage - HBase >Affects Versions: v2.5.0 >Reporter: Tom_yj >Priority: Major > > Garbage Collection on HBase > hbase数据存储在s3上，清理数据时却找的是hdfs，报文件不存在异常。 > 需要清理的文件在s3上真实存在，hdfs上不存在 > > kylin.properties > kylin.env.hdfs-working-dir=s3://XXX-hive/kylin > kylin.storage.hbase.cluster-fs=s3://XXX-hive/hbase > > > > log > java.io.FileNotFoundException: File > /kylin/kylin_metadata/kylin-d3926099-21bb-6893-1055-6d52f2fe17b7/XXX does not > exist. > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:114) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:964) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:961) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:971) > at > org.apache.kylin.storage.hbase.steps.HDFSPathGarbageCollectionStep.dropHdfsPathOnCluster(HDFSPathGarbageCollectionStep.java:95) > at > org.apache.kylin.storage.hbase.steps.HDFSPathGarbageCollectionStep.doWork(HDFSPathGarbageCollectionStep.java:65) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4016) Garbage Collection on HBase hbase数据存储在s3但是却找的是hdfs
[ https://issues.apache.org/jira/browse/KYLIN-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi resolved KYLIN-4016. -- Resolution: Duplicate > Garbage Collection on HBase hbase数据存储在s3但是却找的是hdfs > --- > > Key: KYLIN-4016 > URL: https://issues.apache.org/jira/browse/KYLIN-4016 > Project: Kylin > Issue Type: Bug > Components: Storage - HBase >Affects Versions: v2.5.0 >Reporter: Tom_yj >Priority: Major > > Garbage Collection on HBase > hbase数据存储在s3上，清理数据时却找的是hdfs，报文件不存在异常。 > 需要清理的文件在s3上真实存在，hdfs上不存在 > > kylin.properties > kylin.env.hdfs-working-dir=s3://XXX-hive/kylin > kylin.storage.hbase.cluster-fs=s3://XXX-hive/hbase > > > > log > java.io.FileNotFoundException: File > /kylin/kylin_metadata/kylin-d3926099-21bb-6893-1055-6d52f2fe17b7/XXX does not > exist. > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904) > at > org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:114) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:964) > at > org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:961) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:971) > at > org.apache.kylin.storage.hbase.steps.HDFSPathGarbageCollectionStep.dropHdfsPathOnCluster(HDFSPathGarbageCollectionStep.java:95) > at > org.apache.kylin.storage.hbase.steps.HDFSPathGarbageCollectionStep.doWork(HDFSPathGarbageCollectionStep.java:65) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-2206) Comparing two columns causes IllegalStateException
[ https://issues.apache.org/jira/browse/KYLIN-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-2206. Resolution: Cannot Reproduce > Comparing two columns causes IllegalStateException > -- > > Key: KYLIN-2206 > URL: https://issues.apache.org/jira/browse/KYLIN-2206 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Priority: Major > Attachments: image-2019-11-05-10-34-10-675.png > > > Sql is like " ... where fo.pick_due_ts >= fo.last_update_ts ... " > Error message is "Duplicate columns! old is PICK_DUE_TS and new is > LAST_UPDATE_TS", likely thrown from CompareTupleFilter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4011) Kyling grouping function
[
https://issues.apache.org/jira/browse/KYLIN-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047618#comment-17047618
]
Shao Feng Shi commented on KYLIN-4011:
--
[~tagtang] could you provide more information? thx!
> Kyling grouping function
>
>
> Key: KYLIN-4011
> URL: https://issues.apache.org/jira/browse/KYLIN-4011
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.6.2
>Reporter: tag
>Assignee: Yaqian Zhang
>Priority: Major
> Attachments: image-2019-11-05-18-04-46-639.png
>
>
> {{```}}
> {{select dim1, case grouping(dim2) when 1 then 'ALL' else dim2 end, sum(col)
> as metric1 from table group by grouping sets((dim1, dim2), (dim1));}}
> {{```}}
> {{`case grouping(dim2) when 2 then 'All' else dim2 end` can work in version
> v2.5.2, but invalid in version v2.6.2. }}
> {{How to query aggregate by grouping sets?}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (KYLIN-3736) Disable Tableau Probing Queries
[ https://issues.apache.org/jira/browse/KYLIN-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3736. Resolution: Incomplete > Disable Tableau Probing Queries > --- > > Key: KYLIN-3736 > URL: https://issues.apache.org/jira/browse/KYLIN-3736 > Project: Kylin > Issue Type: Bug >Reporter: anil kumar >Assignee: anil kumar >Priority: Major > > Hi Team, > > Is there a configuration property to disable Tableau Probing queries. > Sometimes it is not necessary to probe SQL statements when Tableau publish > the reports. These queries leads confusion and some sort of syntactical > errors / exceptions. > > *sample query :* > CREATE LOCAL TEMPORARY TABLE > "XT01_11B3408A_5074_4E96_B7A9_2D4543569E63_1_Connect" ( "COL" INTEGER ) ON > COMMIT PRESERVE ROWS > DROP TABLE "XT01_11B3408A_5074_4E96_B7A9_2D4543569E63_1_Connect" > SELECT 1 AS "COL" > SELECT TOP 1 "COL" FROM ( SELECT 1 AS "COL" ) AS "CHECKTOP" > SELECT "COL" FROM (SELECT 1 AS "COL") AS "CHECKTOP" LIMIT 1 > SELECT "SUBCOL" AS "COL" FROM ( SELECT 1 AS "SUBCOL" ) "SUBQUERY" GROUP BY 1 > *For example below query returns error like below* > SELECT "SUBCOL" AS "COL" FROM ( SELECT 1 AS "SUBCOL" ) "SUBQUERY" GROUP BY 1 > ERROR [http-bio--exec-7] controller.BasicController:62 > org.apache.kylin.rest.exception.InternalErrorException: group by 1 > > These are *TablueTestQueries* in *QueriesUtil.java* in Utility classes > > Regards, > Anilkumar. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4170) Add cube information and query information when hbase deadline happened
[ https://issues.apache.org/jira/browse/KYLIN-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047617#comment-17047617 ] Shao Feng Shi commented on KYLIN-4170: -- Pan, are you working on this? [~灬kestrel灬] > Add cube information and query information when hbase deadline happened > --- > > Key: KYLIN-4170 > URL: https://issues.apache.org/jira/browse/KYLIN-4170 > Project: Kylin > Issue Type: Improvement > Components: Storage - HBase >Affects Versions: v2.6.2 >Reporter: 柳攀 >Assignee: 柳攀 >Priority: Major > Fix For: Future > > Original Estimate: 168h > Remaining Estimate: 168h > > When there is a large number of cube query and hbase server is overload, > hbase deadline would happened.But there is no information about cube and > query for us to locate the problem.If we could get cube and query information > in query log, we could solve the query problem quickly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3754) Could not initialize class org.apache.calcite.jdbc.Driver
[ https://issues.apache.org/jira/browse/KYLIN-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3754. Resolution: Incomplete > Could not initialize class org.apache.calcite.jdbc.Driver > - > > Key: KYLIN-3754 > URL: https://issues.apache.org/jira/browse/KYLIN-3754 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.5.2 > Environment: hdp 2.4.2.0-258 > hadoop 2.7.1.2.4.2.0-258 > hive 1.2.1000.2.4.2.0-258 >Reporter: wangxianbin >Priority: Major > Attachments: calcite driver error.png > > > after upgrade from 2.5.1 to 2.5.2, fail to initialize class > org.apache.calcite.jdbc.Driver, it seem like related to calcite upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4268) build job failed caused by hadoop utils update
[
https://issues.apache.org/jira/browse/KYLIN-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047612#comment-17047612
]
Shao Feng Shi commented on KYLIN-4268:
--
How often does this happen?
> build job failed caused by hadoop utils update
> --
>
> Key: KYLIN-4268
> URL: https://issues.apache.org/jira/browse/KYLIN-4268
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> When we update hadoop package, such as hbase and hive. The job will fill
> cause by:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.io.FileNotFoundException: File
> file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
> at
> org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
> {code}
>
> The root cause is kylin will generate the full command as soon as the job
> summited. And the parameter '--jars' use file's absolute path.
>
> there will be 2 ways to fix this problem.
> # keep the old package in the same path
> # change the generation of '--jars' to the stage of job execution.
>
> which is the better choose?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4267) Add doc for KYLIN-4178
[ https://issues.apache.org/jira/browse/KYLIN-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi updated KYLIN-4267: - Component/s: Documentation > Add doc for KYLIN-4178 > -- > > Key: KYLIN-4267 > URL: https://issues.apache.org/jira/browse/KYLIN-4267 > Project: Kylin > Issue Type: Task > Components: Documentation >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4267) Add doc for KYLIN-4178
[ https://issues.apache.org/jira/browse/KYLIN-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4267. > Add doc for KYLIN-4178 > -- > > Key: KYLIN-4267 > URL: https://issues.apache.org/jira/browse/KYLIN-4267 > Project: Kylin > Issue Type: Task > Components: Documentation >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4267) Add doc for KYLIN-4178
[ https://issues.apache.org/jira/browse/KYLIN-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi resolved KYLIN-4267. -- Resolution: Fixed > Add doc for KYLIN-4178 > -- > > Key: KYLIN-4267 > URL: https://issues.apache.org/jira/browse/KYLIN-4267 > Project: Kylin > Issue Type: Task > Components: Documentation >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4279) Mapreduce Exception when building cube at step 3
[ https://issues.apache.org/jira/browse/KYLIN-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4279. Resolution: Incomplete > Mapreduce Exception when building cube at step 3 > > > Key: KYLIN-4279 > URL: https://issues.apache.org/jira/browse/KYLIN-4279 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.6.4 >Reporter: Kanivel >Priority: Major > Attachments: Screenshot from 2019-12-05 12-55-22.png, Screenshot from > 2019-12-05 12-55-47.png > > > I have setup hadoop multinode cluster through ambari .ambari running as > master node and hive running in another aws instance.I have installed kylin > in master node and configured hive instance in kylin.properties. I can create > model,can create cube.But exception occur when build cube.Build fails at 3rd > step.When I check logs it shows Mapreduce exception.2nd step also taking more > time.I have 3 tables and each have 10 records.sales,customers,products.sales > is a fact table.customers and products are dimesions table..Please look at > the attached screenshots. > The below error I got, > org.apache.kylin.engine.mr.exception.MapReduceException: Counters: 13 Job > Counters Failed map tasks=4 Killed reduce tasks=3 Launched map tasks=4 Other > local map tasks=3 Data-local map tasks=1 Total time spent by all maps in > occupied slots (ms)=194628 Total time spent by all reduces in occupied slots > (ms)=0 Total time spent by all map tasks (ms)=27804 Total vcore-milliseconds > taken by all map tasks=27804 Total megabyte-milliseconds taken by all map > tasks=199299072 Map-Reduce Framework CPU time spent (ms)=0 Physical memory > (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Job Diagnostics:Task > failed task_1575385745059_0009_m_00 Job failed as tasks failed. > failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task > Diagnostics: [2019-12-04 07:44:23.184]Exception from container-launch. > Container id: container_e03_1575385745059_0009_01_04 Exit code: 255 > [2019-12-04 07:44:23.186]Container exited with a non-zero exit code 255. > Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes > of stderr : [2019-12-04 07:44:23.186]Container exited with a non-zero exit > code 255. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last > 4096 bytes of stderr : at > org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4279) Mapreduce Exception when building cube at step 3
[ https://issues.apache.org/jira/browse/KYLIN-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047598#comment-17047598 ] Shao Feng Shi commented on KYLIN-4279: -- Need to look into MR for error root cause. > Mapreduce Exception when building cube at step 3 > > > Key: KYLIN-4279 > URL: https://issues.apache.org/jira/browse/KYLIN-4279 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.6.4 >Reporter: Kanivel >Priority: Major > Attachments: Screenshot from 2019-12-05 12-55-22.png, Screenshot from > 2019-12-05 12-55-47.png > > > I have setup hadoop multinode cluster through ambari .ambari running as > master node and hive running in another aws instance.I have installed kylin > in master node and configured hive instance in kylin.properties. I can create > model,can create cube.But exception occur when build cube.Build fails at 3rd > step.When I check logs it shows Mapreduce exception.2nd step also taking more > time.I have 3 tables and each have 10 records.sales,customers,products.sales > is a fact table.customers and products are dimesions table..Please look at > the attached screenshots. > The below error I got, > org.apache.kylin.engine.mr.exception.MapReduceException: Counters: 13 Job > Counters Failed map tasks=4 Killed reduce tasks=3 Launched map tasks=4 Other > local map tasks=3 Data-local map tasks=1 Total time spent by all maps in > occupied slots (ms)=194628 Total time spent by all reduces in occupied slots > (ms)=0 Total time spent by all map tasks (ms)=27804 Total vcore-milliseconds > taken by all map tasks=27804 Total megabyte-milliseconds taken by all map > tasks=199299072 Map-Reduce Framework CPU time spent (ms)=0 Physical memory > (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Job Diagnostics:Task > failed task_1575385745059_0009_m_00 Job failed as tasks failed. > failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task > Diagnostics: [2019-12-04 07:44:23.184]Exception from container-launch. > Container id: container_e03_1575385745059_0009_01_04 Exit code: 255 > [2019-12-04 07:44:23.186]Container exited with a non-zero exit code 255. > Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes > of stderr : [2019-12-04 07:44:23.186]Container exited with a non-zero exit > code 255. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last > 4096 bytes of stderr : at > org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4282) support case when in count (distinct)
[ https://issues.apache.org/jira/browse/KYLIN-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047596#comment-17047596 ] Shao Feng Shi commented on KYLIN-4282: -- Yanghong, what's the plan for this? Thx! > support case when in count (distinct) > - > > Key: KYLIN-4282 > URL: https://issues.apache.org/jira/browse/KYLIN-4282 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4281) Precisely set the data type of tuple expression
[ https://issues.apache.org/jira/browse/KYLIN-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047597#comment-17047597 ] Shao Feng Shi commented on KYLIN-4281: -- +1 > Precisely set the data type of tuple expression > --- > > Key: KYLIN-4281 > URL: https://issues.apache.org/jira/browse/KYLIN-4281 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > > Previously to simplify the calculating of sum(case when), all of the binary > calculation is based on BigDecimal, which is not good for all, especially > when dealing with count(distinct case when), whose inner data type may be hll > or bitmap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4349) Close InputStream in RowRecordReader.initReaders()
[ https://issues.apache.org/jira/browse/KYLIN-4349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047595#comment-17047595 ] nichunen commented on KYLIN-4349: - Hi, [~codingforfun] This will fail the integration test. > Close InputStream in RowRecordReader.initReaders() > -- > > Key: KYLIN-4349 > URL: https://issues.apache.org/jira/browse/KYLIN-4349 > Project: Kylin > Issue Type: Bug >Affects Versions: v3.0.0 >Reporter: weibin0516 >Assignee: weibin0516 >Priority: Major > Attachments: image-2020-01-16-10-44-40-119.png > > > Some InputStream not closed properly > !image-2020-01-16-10-44-40-119.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4285) Mapreduce Exception when building cube at step 3
[ https://issues.apache.org/jira/browse/KYLIN-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4285. Resolution: Incomplete Please check Hadoop MR log for details. > Mapreduce Exception when building cube at step 3 > > > Key: KYLIN-4285 > URL: https://issues.apache.org/jira/browse/KYLIN-4285 > Project: Kylin > Issue Type: Bug >Reporter: Kanivel >Priority: Major > > Hi, >Cube fails at step 3.Below is the log. >Kylin Version:2.6.4 > Hadoop Version:3.1 > org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job > job_1575740521851_0009Job Diagnostics:Task failed > task_1575740521851_0009_m_00 > Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 > killedReduces: 0 > Failure task Diagnostics: > [2019-12-09 11:11:48.821]Exception from container-launch. > Container id: container_e07_1575740521851_0009_01_07 > Exit code: 255 > [2019-12-09 11:11:48.823]Container exited with a non-zero exit code 255. > Error file: prelaunch.err. > Last 4096 bytes of prelaunch.err : > Last 4096 bytes of stderr : > [2019-12-09 11:11:48.823]Container exited with a non-zero exit code 255. > Error file: prelaunch.err. > Last 4096 bytes of prelaunch.err : > Last 4096 bytes of stderr : > at > org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4342) Build Global Dict by MR/Hive New Version
[ https://issues.apache.org/jira/browse/KYLIN-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi updated KYLIN-4342: - Fix Version/s: v3.1.0 Affects Version/s: (was: Future) > Build Global Dict by MR/Hive New Version > > > Key: KYLIN-4342 > URL: https://issues.apache.org/jira/browse/KYLIN-4342 > Project: Kylin > Issue Type: Improvement >Reporter: wangxiaojing >Assignee: wangxiaojing >Priority: Major > Fix For: v3.1.0 > > > At present, there are two limitations and some distributed concurrency lock > bugs in the implementation of global dictionary through MR/Hive: > 1. Limited by Hive order by global sorting on the shuffle stage, the memory > and build time becomes uncontrollable with data volume reaching billion > level. We have tested the base of 800 million level to configure 15g memory, > and the build time of build dictionary needs more than 10 hours; > 2. Multi global dictionary columns is calculated serially. > 3. Some distributed concurrency lock bugs. > We have improved the original version.The general idea of the new version is > the same as the previous Mr / Hive implementation, that is, to complete > global dictionary coding through Hive or MR, and then replace the original > value in the flat table with the dictionary encoded value.[Mr /Hive > V1|[http://kylin.apache.org/docs30/howto/howto_use_hive_mr_dict.html]] > However, in the new version, will add "parallel part build" and "parallel > total build" two steps by mr to replace the original "build dict" step, so as > to solve the above two limitations.And use ZK to solve the distributed > concurrency lock bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4363) Query failed with "I failed to find one of the right cookies" error
[
https://issues.apache.org/jira/browse/KYLIN-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shao Feng Shi closed KYLIN-4363.
Resolution: Fixed
> Query failed with "I failed to find one of the right cookies" error
> ---
>
> Key: KYLIN-4363
> URL: https://issues.apache.org/jira/browse/KYLIN-4363
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v3.1.0
>Reporter: Shao Feng Shi
>Assignee: ZhouKang
>Priority: Major
> Fix For: v3.1.0
>
>
> {code:java}
> // code placeholder
> Caused by: java.lang.RuntimeException: I failed to find one of the right
> cookies. 1701223222Caused by: java.lang.RuntimeException: I failed to find
> one of the right cookies. 1701223222 at
> org.roaringbitmap.buffer.ImmutableRoaringArray.(ImmutableRoaringArray.java:46)
> at
> org.roaringbitmap.buffer.ImmutableRoaringBitmap.(ImmutableRoaringBitmap.java:908)
> at
> org.apache.kylin.measure.bitmap.RoaringBitmapCounter.peekLength(RoaringBitmapCounter.java:141)
> at
> org.apache.kylin.measure.bitmap.BitmapSerializer.peekLength(BitmapSerializer.java:69)
> at
> org.apache.kylin.cube.gridtable.CubeCodeSystem.codeLength(CubeCodeSystem.java:100)
> at org.apache.kylin.gridtable.GTRecord.loadColumns(GTRecord.java:279) at
> org.apache.kylin.storage.gtrecord.PartitionResultIterator.next(PartitionResultIterator.java:56)
> at
> org.apache.kylin.storage.gtrecord.PartitionResultIterator.next(PartitionResultIterator.java:35)
> at com.google.common.collect.Iterators$PeekingImpl.next(Iterators.java:1222)
> at
> org.apache.kylin.storage.gtrecord.SortMergedPartitionResultIterator.next(SortMergedPartitionResultIterator.java:93)
> at
> org.apache.kylin.storage.gtrecord.SortMergedPartitionResultIterator.next(SortMergedPartitionResultIterator.java:38)
> at com.google.common.collect.Iterators$PeekingImpl.next(Iterators.java:1222)
> at
> org.apache.kylin.gridtable.GTStreamAggregateScanner$AbstractStreamMergeIterator.next(GTStreamAggregateScanner.java:102)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:162)
> at
> org.apache.kylin.storage.gtrecord.PeekingImpl.hasNext(PeekingImpl.java:45) at
> org.apache.kylin.storage.gtrecord.SortedIteratorMerger.getHeap(SortedIteratorMerger.java:52)
> at
> org.apache.kylin.storage.gtrecord.SortedIteratorMergerWithLimit$MergedIteratorWithLimit.hasNext(SortedIteratorMergerWithLimit.java:81)
> at
> org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:144)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:63)
> at Baz$1$1.moveNext(Unknown Source) at
> org.apache.calcite.linq4j.EnumerableDefaults.aggregate(EnumerableDefaults.java:116)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.aggregate(DefaultEnumerable.java:107)
> at Baz.bind(Unknown Source) at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:365)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:301)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:559)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:550)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:182)
> at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> {code}
> I build a cube with an older version of Kylin, and then query the cube from
> the latest master version, and then got this error. With some analysis, I
> think it is related with KYLIN-4322
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KYLIN-3969) BeelineHiveClient can't parse Hive meta data result set with partitioned table
[
https://issues.apache.org/jira/browse/KYLIN-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen reassigned KYLIN-3969:
---
Assignee: Tuo Zhu
> BeelineHiveClient can't parse Hive meta data result set with partitioned table
> --
>
> Key: KYLIN-3969
> URL: https://issues.apache.org/jira/browse/KYLIN-3969
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.6.1
> Environment: Hive 3.1.0, HiveServer2, Beeline（Along with Ambari
> 2.7.1.0)
> CentOS Linux release 7.6.1810
>Reporter: Tuo Zhu
>Assignee: Tuo Zhu
>Priority: Blocker
>
> When creating a model, you can add tables without partitions, but can't add
> ones with partitions.
> Web UI pops："Oops. cannot get HiveTableMeta".
> I tried same table in Hive2 with cli, it works.
> log prints:
> {code:java}
> java.lang.RuntimeException: cannot get HiveTableMeta
> at
> org.apache.kylin.source.hive.HiveMetadataExplorer.loadTableMetadata(HiveMetadataExplorer.java:68)
> at
> org.apache.kylin.rest.service.TableService.extractHiveTableMeta(TableService.java:205)
> at
> org.apache.kylin.rest.service.TableService.loadHiveTablesToProject(TableService.java:131)
> at
> org.apache.kylin.rest.controller.TableController.loadHiveTables(TableController.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.servletapi.SecurityContextHolderAwareR
[jira] [Resolved] (KYLIN-4160) Auto redirect to host:port/kylin when user only enter host:port in broswer
[ https://issues.apache.org/jira/browse/KYLIN-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi resolved KYLIN-4160. -- Resolution: Fixed > Auto redirect to host:port/kylin when user only enter host:port in broswer > -- > > Key: KYLIN-4160 > URL: https://issues.apache.org/jira/browse/KYLIN-4160 > Project: Kylin > Issue Type: Improvement >Reporter: luguosheng >Assignee: luguosheng >Priority: Major > Fix For: v2.6.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4072) CDH 6.x find-hbase-dependency.sh return with "base-common lib not found"
[ https://issues.apache.org/jira/browse/KYLIN-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi resolved KYLIN-4072. -- Resolution: Fixed > CDH 6.x find-hbase-dependency.sh return with "base-common lib not found" > > > Key: KYLIN-4072 > URL: https://issues.apache.org/jira/browse/KYLIN-4072 > Project: Kylin > Issue Type: Bug >Affects Versions: all >Reporter: Xiaoxiang Yu >Assignee: wangrupeng >Priority: Major > Fix For: v2.6.6 > > Attachments: image-2019-07-04-17-36-50-112.png, > image-2019-07-04-17-38-40-660.png, image-2019-07-04-17-39-56-973.png > > > In CDH 6.x, I find that execute "sh bin/find-hbase-dependency.sh" can not > find hbase dependency as expected. > > !image-2019-07-04-17-38-40-660.png! > !image-2019-07-04-17-36-50-112.png! > > And I compare the output of "hbase classpath" of different CDH version, I > find the root cause should be the change of the "hbase classpath". That is to > say, newer CDH didn't absolute path of all jars, only parent path. > !image-2019-07-04-17-39-56-973.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3769) DATA SOURCE 能自主添加各种数据源
[ https://issues.apache.org/jira/browse/KYLIN-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3769. Resolution: Not A Problem > DATA SOURCE 能自主添加各种数据源 > -- > > Key: KYLIN-3769 > URL: https://issues.apache.org/jira/browse/KYLIN-3769 > Project: Kylin > Issue Type: Wish > Components: RDBMS Source >Affects Versions: v2.5.2 >Reporter: caiyinting >Priority: Critical > Fix For: Future > > > kylin作为展现层的应用层存储非常合适，但是目前数据的来源只能支持一种；实际生产过程中，有可能企业不存在数仓，有可能直接连接的线上多个业务库；也有可能部分数据来自数仓，部分数据来自业务系统等各种需求；目前对数据源的支持太不灵活，需要改进，望支持！ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3332) Support percentile measure in ODBC driver
[ https://issues.apache.org/jira/browse/KYLIN-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3332. Resolution: Won't Fix > Support percentile measure in ODBC driver > - > > Key: KYLIN-3332 > URL: https://issues.apache.org/jira/browse/KYLIN-3332 > Project: Kylin > Issue Type: Wish >Reporter: Le Anh Vu >Assignee: Dong Li >Priority: Critical > > I'm using Tableau to connect to Kylin via ODBC driver and I see percentile > measure option greyed out and requires extract to be able to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3254) Cannot use Kylin Cubes with Excel
[ https://issues.apache.org/jira/browse/KYLIN-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3254. Resolution: Won't Fix > Cannot use Kylin Cubes with Excel > - > > Key: KYLIN-3254 > URL: https://issues.apache.org/jira/browse/KYLIN-3254 > Project: Kylin > Issue Type: Bug > Components: Driver - ODBC >Affects Versions: v2.2.0 >Reporter: Jean-Luc BELLIER >Assignee: Dong Li >Priority: Critical > > Hello > I downloaded the ODBC driver to use Kylin with Excel. My OS is Windows 8.1 > 64-bits > I first clicked on the KylinODBCDriver (x64).exe, but it says that the > program could not be executed on my PC. > So i ran the 32-bit version. I succeeded in configuring the Kylin ODBC > connection. > > Now I would like to use my Kylin cubes with Excel. So I downloaded PowerBI as > mentioned on Kylin site, but the installation program tells me that my > version of Excel was not compatible, because the driver requires Excel2010 > SP1 or Excel 2013. > My version of Excel is 2013 Professional version. > > Are there any limitations ? How can I visualize my Kylin cubes with Excel in > my case ? > > Thank you in advance for your help. > > Best regards, > Jean-Luc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3764) 能否支持多个字段进行concat
[ https://issues.apache.org/jira/browse/KYLIN-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3764. Resolution: Not A Problem > 能否支持多个字段进行concat > > > Key: KYLIN-3764 > URL: https://issues.apache.org/jira/browse/KYLIN-3764 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: 王振强 >Priority: Critical > > 需要查询多个字段进行拼接成的字符串，但是目前只能进行两两进行合并。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-3764) 能否支持多个字段进行concat
[ https://issues.apache.org/jira/browse/KYLIN-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047577#comment-17047577 ] Shao Feng Shi commented on KYLIN-3764: -- Concat that in Hive view, and then use Hive view in Kylin. > 能否支持多个字段进行concat > > > Key: KYLIN-3764 > URL: https://issues.apache.org/jira/browse/KYLIN-3764 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: 王振强 >Priority: Critical > > 需要查询多个字段进行拼接成的字符串，但是目前只能进行两两进行合并。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3420) common sql query fail on kylin web ui
[ https://issues.apache.org/jira/browse/KYLIN-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3420. Resolution: Not A Problem > common sql query fail on kylin web ui > - > > Key: KYLIN-3420 > URL: https://issues.apache.org/jira/browse/KYLIN-3420 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.1 > Environment: debian >Reporter: ZoneYang >Priority: Critical > Labels: bug > Attachments: Clip(06-19-14-57-34).png, Clip(06-19-14-58-36).png > > > kylin version:2.3.1 > sql: > SELECT "COUNTRY", > "LATITUDE", > "LONGITUDE", > "NAME" > FROM DEFAULT.KYLIN_COUNTRY > LIMIT 10 > > when i execute this sql, kylin web ui will tell me fail -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-3937) 从 kafka 流构建 cube ,发生死锁
[ https://issues.apache.org/jira/browse/KYLIN-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047575#comment-17047575 ] Shao Feng Shi commented on KYLIN-3937: -- Didn't see this; More inputs? > 从 kafka 流构建 cube ,发生死锁 > -- > > Key: KYLIN-3937 > URL: https://issues.apache.org/jira/browse/KYLIN-3937 > Project: Kylin > Issue Type: Bug > Components: Job Engine, Metadata, Spark Engine >Affects Versions: v2.6.0 > Environment: kylin: 2.6.0 > hdp: 2.5.0.0-1245 >Reporter: 风语者 >Priority: Critical > Attachments: deadlock_kylin.png, kylin_monitors.png > > > 你好，用 kylin 从 kafka 流构建 cube, 每五分钟 构建一次，从日志看到好像发生了死锁现象。 > 目前为止，发生了再次，第一次我将所有正在进行的构建全部 discard , 然后死锁现象还是存在，目前不知道该怎么处理这个问题。烦请查看下吧。多谢多谢。 > 用 spark 进行构建，到第三步：Build Dimension Dictionary 时，发生了死锁。 > 发生死锁时，没有发生 cube 构建错误现象。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3940) Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
[ https://issues.apache.org/jira/browse/KYLIN-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3940. Resolution: Incomplete > Exception in thread "main" java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > - > > Key: KYLIN-3940 > URL: https://issues.apache.org/jira/browse/KYLIN-3940 > Project: Kylin > Issue Type: Task > Components: Environment >Affects Versions: v2.5.2 > Environment: apache-hive-2.3.4-bin > hbase-1.4.9 > hadoop-3.1.0 > apache-kylin-2.5.2-bin-hadoop3 >Reporter: praveenece >Priority: Critical > > I am getting the below error. This error occurred when use > apache-kylin-2.5.2-bin-hadoop3. if we use apache-kylin-2.5.2-bin-hbase1x > kylin gets started. > apache-hive-2.3.4-bin > hbase-1.4.9 > hadoop-3.1.0 > apache-kylin-2.5.2-bin-hadoop3 > Exception in thread "main" java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > at > org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:98) > at > org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) > at > org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:98) > at > org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:92) > ... 3 more > Caused by: java.lang.NoClassDefFoundError: > org/apache/hadoop/hbase/client/ColumnFamilyDescriptor > at > org.apache.kylin.storage.hbase.HBaseResourceStore.getConnection(HBaseResourceStore.java:86) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:112) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:93) > ... 8 more > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hbase.client.ColumnFamilyDescriptor > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4119) The admin of project can't operate the action of Hybrids
[ https://issues.apache.org/jira/browse/KYLIN-4119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047573#comment-17047573 ] Shao Feng Shi commented on KYLIN-4119: -- Hi Peng [~haven0930] would you like to contribute a pull request? Thx! > The admin of project can't operate the action of Hybrids > > > Key: KYLIN-4119 > URL: https://issues.apache.org/jira/browse/KYLIN-4119 > Project: Kylin > Issue Type: Bug > Components: Integration >Affects Versions: v2.6.1 >Reporter: Peng Huang >Priority: Critical > Attachments: 002b1d4cbb76e315efcdc0308984bec.png > > > The admin of project can't operate the action of Hybrids, only the admin of > system can do it -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4186) Cannot get hivetablemeta
[ https://issues.apache.org/jira/browse/KYLIN-4186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4186. Resolution: Incomplete > Cannot get hivetablemeta > > > Key: KYLIN-4186 > URL: https://issues.apache.org/jira/browse/KYLIN-4186 > Project: Kylin > Issue Type: Test > Components: Environment > Environment: hadoop-3.1.1 > apache-hive-2.3.4-bin > hbase-2.1.1 > apache-kylin-2.5.2-bin-hadoop3 >Reporter: praveenece >Priority: Critical > > apache-kylin-2.6.3-bin-hadoop3 > apache-hive-2.3.2-bin > hbase-2.0.0 > hadoop-3.1.1 > When load data from hive in Kylin getting * Cannot get hivetablemeta* this > error. > Please help me which version of hive i need to use for this environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4200) SparkCubingBylayer is not rubust using CubeStatsReader to estimateLayerPartitionNum
[ https://issues.apache.org/jira/browse/KYLIN-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047569#comment-17047569 ] Shao Feng Shi commented on KYLIN-4200: -- Hi Hailin, so what's the proposal? > SparkCubingBylayer is not rubust using CubeStatsReader to > estimateLayerPartitionNum > --- > > Key: KYLIN-4200 > URL: https://issues.apache.org/jira/browse/KYLIN-4200 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.4 >Reporter: hailin.huang >Priority: Critical > > In prod env, I found some scene as follow: > if user has measure of Bitmap, the Spark tasks often block at job 0,1 > which compute layer 0,1. >after analyze spark log, I found that, spark use CubeStatsReader to > estimateLayerPartition Num, If layer 0 has cuboid 255, and its size is 10M, > according to the default configuration pararm ,the partition will be 1, > however it's quick small for this amout of data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4228) Not a valid ID: xxx while executing SQL
[
https://issues.apache.org/jira/browse/KYLIN-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shao Feng Shi closed KYLIN-4228.
Resolution: Not A Problem
> Not a valid ID: xxx while executing SQL
> ---
>
> Key: KYLIN-4228
> URL: https://issues.apache.org/jira/browse/KYLIN-4228
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v3.0.0-beta
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Critical
>
> Executing the SQL with "case when" maybe cause the following error:
> Not a valid ID: xxx while executing SQL
> null while executing SQL
> Can be fixed by setting "kylin.query.enable-dynamic-column" to Ture
> {code:java}
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227)
> at
> org.apache.kylin.rest.service.QueryService.executeRequest(QueryService.java:969)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:672)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:218)
> at
> org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:479)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:439)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:377)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93)
> at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(Ano
[jira] [Commented] (KYLIN-4370) Spark job failing with JDBC source on 8th step with error : org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table or view not found: `default`.`kylin_int
[ https://issues.apache.org/jira/browse/KYLIN-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047568#comment-17047568 ] ASF subversion and git services commented on KYLIN-4370: Commit 3ffd97175022f771cecda83d1b20eac14b68eab9 in kylin's branch refs/heads/2.6.x from rupengwang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3ffd971 ] KYLIN-4370 Fix "Table or view not found" error when build cube with spark for JDBC source at step "Build cube by Layer" > Spark job failing with JDBC source on 8th step with error : > org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table or view > not found: `default`.`kylin_intermediate table' > > > Key: KYLIN-4370 > URL: https://issues.apache.org/jira/browse/KYLIN-4370 > Project: Kylin > Issue Type: Bug >Affects Versions: v3.0.0 >Reporter: Sonu Singh >Assignee: wangrupeng >Priority: Blocker > Fix For: v3.0.2, v2.6.6 > > Attachments: image-2020-02-19-17-20-20-076.png, > image-2020-02-19-18-48-00-857.png, image-2020-02-21-23-44-53-459.png, > image-2020-02-21-23-47-52-247.png > > > 020-02-03 11:18:45,899 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:45 INFO Client:54 - Application report for > application_1580106368736_0431 (state: RUNNING) > 2020-02-03 11:18:46,901 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46 INFO Client:54 - Application report for > application_1580106368736_0431 (state: FINISHED) > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46 INFO Client:54 - > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > client token: N/A > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > diagnostics: User class threw exception: java.lang.RuntimeException: error > execute org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table > or view not found: > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b`;; > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 'UnresolvedRelation > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b` > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:34) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:36) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > java.lang.reflect.Method.invoke(Method.java:498) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > Caused by: org.apache.spark.sql.AnalysisException: Table or view not found: > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b`;; > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 'UnresolvedRelation > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b` > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:86) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:84) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode
[jira] [Commented] (KYLIN-4329) use nrt streaming build kafka data,but the segment cannot merge
[ https://issues.apache.org/jira/browse/KYLIN-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047559#comment-17047559 ] Shao Feng Shi commented on KYLIN-4329: -- [~kangkingkang] any update? > use nrt streaming build kafka data,but the segment cannot merge > - > > Key: KYLIN-4329 > URL: https://issues.apache.org/jira/browse/KYLIN-4329 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.4 > Environment: kylin 2.6.4 >Reporter: kangkang >Priority: Critical > Labels: build > Fix For: v3.1.0 > > Attachments: 1981578450672_.pic.jpg, 2001578450672_.pic_hd.jpg, > image-2020-02-27-16-21-08-415.png > > Original Estimate: 72h > Remaining Estimate: 72h > > i use near streaming build for kafka data ,build once an hour > , I also set the time for automatic merge > but the segment cannot merge, i want to know why ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-2346) OLAP TableScan MUST NOT be shared by more than one prent
[ https://issues.apache.org/jira/browse/KYLIN-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-2346. Resolution: Incomplete > OLAP TableScan MUST NOT be shared by more than one prent > > > Key: KYLIN-2346 > URL: https://issues.apache.org/jira/browse/KYLIN-2346 > Project: Kylin > Issue Type: Bug >Reporter: james >Priority: Blocker > > i am a new fresher of the apache-kylin, > i build cube,success. > i want to run a sql sentence > the result of the relation-database is right..what i want.. > the result of the cube modile is wrong..the results is as the titles > the sql is > select * from tradectr.dbo.g_og_corpinfo t1 left join > tradectr.dbo.g_og_corpinfo t2 on t1.parentsid = t2.corpsid where > t2.levelscode = 'G011' AND t2.corpname like '%湖北%' > it is a simple sql,just to find the company of the hubei province... > also somebody could tell me the type of the sql which apache-kylin > supports,thanks very much。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-1576) Support of new join type in the Cube Model - Temporal Join
[
https://issues.apache.org/jira/browse/KYLIN-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shao Feng Shi closed KYLIN-1576.
Resolution: Won't Fix
> Support of new join type in the Cube Model - Temporal Join
> --
>
> Key: KYLIN-1576
> URL: https://issues.apache.org/jira/browse/KYLIN-1576
> Project: Kylin
> Issue Type: New Feature
> Components: Others
>Affects Versions: Future
>Reporter: Richard Calaba
>Assignee: Billy Liu
>Priority: Blocker
>
> There is a notion of time-dependent master data in many business scenarios.
> Typically modeled with granularity 1 day (datefrom, dateto fields of type
> DATE defining validity time of one master data record). Occasionally you can
> think of lower granularity so use of TIMESTAMP can be also seen as an valid
> scenario). Example of such master data definition could be:
> Master Data / Dimension Table:
> =
> KEY: PRODUCT_ID, DATE_TO,
> NON-KEY: DATE_FROM, PRODUCT_DESCRIPTION
> - assuming that PRODUCT_DESCRIPTION cannot have 2 values during one day it is
> assumed that DATE_TO <= DATE_TO and also that there are no overlapping
> intervals (DATE_FROM, DATE_TO) for all PRODUCT master data
> - the KEY is then intentionally defined as (PRODUCT_ID, DATE_TO) so the
> statment SELECT * from PRODUCT WHERE ID = 'prod_key_1' AND DATE_TO >=
> today/now and DATE_FROM <= today/now is efficient way to retrieve 'current'
> PRODUCT master data (description). The today/now is also being named as 'key
> date'.
> - now if I have transaction data (FACT table) of product sales, i.e:
> SALES_DATE, PRODUCT_ID, STORE_ID,
> I would like to show the Sold Products at Store at certain date and also show
> the Description of the product at the date of product sale (assuming here
> that there is product catalog which can be updated independently, but for
> auditing purposes the original product description used during sale is needed
> to be displayed/used).
> The SQL for the temporal join would be then:
> SELECT S.PRODUCT_ID, S.SALES_DATE, P.PRODUCT_DESCRIPTION
> FROM SALES as S LEFT OUTER JOIN PRODUCT as P
> ON S.PRODUCT_ID = P.PRODUCT_ID
> AND S.SALES_DATE >= P.DATE_FROM AND
> AND S.SALES_DATE <= P.DATE_TO
> (also INNER TEMPORAL JOIN can be defined and be valid in some scenarios but
> in this case it won't be the proper join - we need to show the product sales
> even the description wasn't maintained in product master data)
> (some more details for temporal joins - see i.e. here -
> http://scn.sap.com/community/hana-in-memory/blog/2014/09/28/using-temporal-join-to-fetch-the-result-set-within-the-time-interval
> )
> This scenario can be supported by Kylin if following enhancement would be
> done:
> 1) The Cube Model allowing to define special variant of LEFT OUTER and INNER
> joins (suggesting name temporal (left outer/inner) join) which forces to
> specify a 'key date' as a expression (column / constant / ...) from the FACT
> table and 2 validity fields ('valid from' and 'valid to') fro the LOOKUP
> table/ Those 2 validity fields are defining master data record validity
> period. Supported types for those fields should be DATE, optionally TIMESTAMP
> is also fine but rarely used in business scenarios.
> Other option rather then defining new join type is to loosen the join
> condition and allowing <= and >= operands to be used as part of the LOOKUP
> join definition.
> 2) The Cube Definition then needs to know the extension of the join type in
> the cube model and needs to force the additional fields (key-date,
> valid-from, valid-to) be part of the whole cube structure. Or alternatively
> cube definition for derived dimensions can be extended to define a
> "time-dependent derived lookup" similar way as described in the step 1) for
> the suggested cube model join type extension.
> 3) Very often the time-partition field of the cube which is being used for
> incremental data loads to cubes will be the 'key-date'. BUT this shouldn't be
> hard-coded this way as this is not true for every scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-3733) kylin can't response correct data when using "in" filter
[
https://issues.apache.org/jira/browse/KYLIN-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047556#comment-17047556
]
Shao Feng Shi commented on KYLIN-3733:
--
Any update?
> kylin can't response correct data when using "in" filter
>
>
> Key: KYLIN-3733
> URL: https://issues.apache.org/jira/browse/KYLIN-3733
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.3.2, v2.5.0
>Reporter: Lianggang
>Assignee: Wu Bin
>Priority: Blocker
> Attachments: cube.json, demo.txt, image-2018-12-28-16-30-50-326.png,
> image-2018-12-28-16-31-33-678.png, image-2018-12-28-16-32-15-776.png,
> image-2018-12-28-16-32-50-531.png, sapshot.docx, with.png, with_out.png
>
>
> Hi guys,
> I used the attachment data to create one cube in kylin. I encounter a strange
> issue.
> when I use sql_1 to query, the data is correct.
> sql_1:
> _select exp_id,os,new_user_flag_
> _,sum(sample_size)_
> _,sum(sum_x)_
> _,sum(sum_square_x)_
> _,sum(act_dev)_
> _from RPT_QUKAN.ABTEST_GRAPH_DETAIL_CUBE_MIDU_VIEW5_
> _where refresh_dt ='2018-12-13'_
> _and exp_id *in* ('8382')_
> _group by exp_id,os,new_user_flag;_
>
> when I use sql_2 to query, the data is *not* correct.
> sql_2:
> _select exp_id,os,new_user_flag_
> _,sum(sample_size)_
> _,sum(sum_x)_
> _,sum(sum_square_x)_
> _,sum(act_dev)_
> _from RPT_QUKAN.ABTEST_GRAPH_DETAIL_CUBE_MIDU_VIEW5_
> _where refresh_dt ='2018-12-13'_
> _and exp_id *in* ('8382','8383')_
> _group by exp_id,os,new_user_flag;_
>
> when I add trim() for exp_id, the data is correct. like sql_3:
> _select exp_id,os,new_user_flag_
> _,sum(sample_size)_
> _,sum(sum_x)_
> _,sum(sum_square_x)_
> _,sum(act_dev)_
> _from RPT_QUKAN.ABTEST_GRAPH_DETAIL_CUBE_MIDU_VIEW5_
> _where refresh_dt ='2018-12-13'_
> _and *trim*(exp_id) in ('8382','8383')_
> _group by exp_id,os,new_user_flag;_
> I am not sure whether the rowkey is too long for this result. please help
> check the issue. thanks!
> the table structure like below:
> CREATE TABLE `RPT_QUKAN.ABTEST_GRAPH_DETAIL_CUBE_MIDU_VIEW5` (
> `exp_id` string
> ,`new_user_flag` string
> ,`os` string
> ,`sample_size` double
> ,`sum_x` double
> ,`sum_square_x` double
> ,`act_dev` bigint
> ,`data_dt` string
> ,`refresh_dt` string)
>
> the dimension columns: exp_id,new_user_flag,os,data_dt,refresh_dt
> the *sum* measure columns: sum_x,sample_size,sum_square_x,act_dev
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (KYLIN-3859) Unknown column 'A0.CATALOG_NAME' in 'where clause'
[ https://issues.apache.org/jira/browse/KYLIN-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3859. Resolution: Incomplete > Unknown column 'A0.CATALOG_NAME' in 'where clause' > --- > > Key: KYLIN-3859 > URL: https://issues.apache.org/jira/browse/KYLIN-3859 > Project: Kylin > Issue Type: Test > Components: Storage - HBase >Reporter: praveenece >Priority: Blocker > > Find below software and its version installed. > 1.apache-kylin-2.6.0-bin-hadoop3 > 2.hadoop-3.1.1 > 3.hbase-2.0.0 > 4.apache-hive-3.1.1-bin > the below exception is being thrown while trying to load data source from > Kylin UI. > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column > 'A0.CATALOG_NAME' in 'where clause'. > after updating columns in DBS and TBLS in Hive metastore i am able to view db > and table names but we tried to do sync i am getting exceptions like : > org.apache.kylin.engine.mr.exception.HadoopShellException: > java.lang.RuntimeException: cannot get HiveTableMeta > at org.apache.kylin.source.hive.HiveTable.(HiveTable.java:50) > at > org.apache.kylin.source.hive.HiveSource.createReadableTable(HiveSource.java:68) > at > org.apache.kylin.source.SourceManager.createReadableTable(SourceManager.java:145) > at > org.apache.kylin.cube.CubeManager$DictionaryAssist.buildSnapshotTable(CubeManager.java:1173) > at > org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:1090) > at > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:97) > at > org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55) > at > org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73) > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93) > at > org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: MetaException(message:java.lang.UnsupportedOperationException: > Storage schema reading not supported) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6935) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_fields_with_environment_context(HiveMetaStore.java:5182) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_fields(HiveMetaStore.java:5144) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) > at com.sun.proxy.$Proxy66.get_fields(Unknown Source) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1845) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1839) > at > org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:78) > at org.apache.kylin.source.hive.HiveTable.(HiveTable.java:48) > ... 16 more > Caused by: java.lang.UnsupportedOperationException: Storage schema reading > not supported > at > org.apache.hadoop.hive.metastore.DefaultStorageSchemaReader.readSchema(DefaultStorageSchemaReader.java:36) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_fields_with_environment_context(HiveMetaStore.java:5173) > ... 28 more > result code:2 > at > org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable
[jira] [Closed] (KYLIN-3762) 构建cube出现error
[ https://issues.apache.org/jira/browse/KYLIN-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3762. Resolution: Incomplete > 构建cube出现error > -- > > Key: KYLIN-3762 > URL: https://issues.apache.org/jira/browse/KYLIN-3762 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: 王振强 >Priority: Blocker > Attachments: mr_error_log.jpg > > > 构建cube出现error，以下是mr的log > 2019-01-09 14:06:43,552 INFO [fetcher#1] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle > output of map attempt_1544413042966_0495_m_02_0 decomp: 59732393 len: > 9306880 to MEMORY > 2019-01-09 14:06:43,595 INFO [fetcher#2] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#2 about to shuffle > output of map attempt_1544413042966_0495_m_03_0 decomp: 61855877 len: > 9709575 to MEMORY > 2019-01-09 14:06:43,625 INFO [fetcher#1] > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 59732393 > bytes from map-output for attempt_1544413042966_0495_m_02_0 > 2019-01-09 14:06:43,625 INFO [fetcher#1] > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile > -> map-output of size: 59732393, inMemoryMapOutputs.size() -> 1, commitMemory > -> 0, usedMemory ->121588270 > 2019-01-09 14:06:43,872 INFO [fetcher#1] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle > output of map attempt_1544413042966_0495_m_01_0 decomp: 60219839 len: > 9341252 to MEMORY > 2019-01-09 14:06:43,889 INFO [fetcher#2] > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 61855877 > bytes from map-output for attempt_1544413042966_0495_m_03_0 > 2019-01-09 14:06:43,889 INFO [fetcher#2] > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile > -> map-output of size: 61855877, inMemoryMapOutputs.size() -> 2, commitMemory > -> 59732393, usedMemory ->181808109 > 2019-01-09 14:06:43,889 INFO [fetcher#2] > org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: ap-cdh04:13562 > freed by fetcher#2 in 579ms > 2019-01-09 14:06:43,938 INFO [fetcher#1] > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 60219839 > bytes from map-output for attempt_1544413042966_0495_m_01_0 > 2019-01-09 14:06:43,938 INFO [fetcher#1] > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile > -> map-output of size: 60219839, inMemoryMapOutputs.size() -> 3, commitMemory > -> 121588270, usedMemory ->181808109 > 2019-01-09 14:06:43,949 INFO [fetcher#1] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle > output of map attempt_1544413042966_0495_m_00_0 decomp: 61659651 len: > 9657307 to MEMORY > 2019-01-09 14:06:43,994 INFO [fetcher#1] > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput: Read 61659651 > bytes from map-output for attempt_1544413042966_0495_m_00_0 > 2019-01-09 14:06:43,995 INFO [fetcher#1] > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: closeInMemoryFile > -> map-output of size: 61659651, inMemoryMapOutputs.size() -> 4, commitMemory > -> 181808109, usedMemory ->243467760 > 2019-01-09 14:06:43,995 INFO [fetcher#1] > org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: ap-cdh03:13562 > freed by fetcher#1 in 685ms > 2019-01-09 14:06:44,001 INFO [EventFetcher for fetching Map Completion > Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is > interrupted.. Returning > 2019-01-09 14:06:44,014 INFO [main] > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: finalMerge called > with 4 in-memory map-outputs and 0 on-disk map-outputs > 2019-01-09 14:06:44,024 INFO [main] org.apache.hadoop.mapred.Merger: Merging > 4 sorted segments > 2019-01-09 14:06:44,026 INFO [main] org.apache.hadoop.mapred.Merger: Down to > the last merge-pass, with 4 segments left of total size: 243467522 bytes > 2019-01-09 14:06:47,997 INFO [main] > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merged 4 segments, > 243467760 bytes to disk to satisfy reduce memory limit > 2019-01-09 14:06:47,998 INFO [main] > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 1 files, > 35816160 bytes from disk > 2019-01-09 14:06:47,998 INFO [main] > org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl: Merging 0 segments, > 0 bytes from memory into reduce > 2019-01-09 14:06:47,999 INFO [main] org.apache.hadoop.mapred.Merger: Merging > 1 sorted segments > 2019-01-09 14:06:48,004 INFO [main] org.apache.hadoop.mapred.Merger: Down to > the last merge-pass, with 1 segments left of total size: 243467668 bytes > ^@^A^@^@^V^@ > > ^@^GVERSION*(^@&container_1544413042966_0495_01_02^H*(^@&container_1544413042966_0495_01_39^D*(^@&container_1544413042966_0495_01_19^F*(^@&container_154
[jira] [Closed] (KYLIN-3319) exceeds threshold 5000000 while executing SQL
[
https://issues.apache.org/jira/browse/KYLIN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shao Feng Shi closed KYLIN-3319.
Resolution: Not A Problem
> exceeds threshold 500 while executing SQL
> -
>
> Key: KYLIN-3319
> URL: https://issues.apache.org/jira/browse/KYLIN-3319
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.3.0
>Reporter: cc
>Priority: Blocker
> Attachments: image.png, sql_limit_result.png
>
>
> {color:#d04437}hello,what is the reason of the errors?{color}
> {color:#d04437}Query returned 5008662 rows exceeds threshold 500 while
> executing SQL: "select * from USER_FREQ_DAY_TEST LIMIT 5"{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (KYLIN-3967) sum along with case expression does not work in query
[ https://issues.apache.org/jira/browse/KYLIN-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3967. Resolution: Not A Problem > sum along with case expression does not work in query > - > > Key: KYLIN-3967 > URL: https://issues.apache.org/jira/browse/KYLIN-3967 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Reporter: Gladson Vas >Priority: Blocker > Attachments: notworkingcase.jpg, workingcase.jpg > > > When i try to run a query with a sum case expression combination, > eg: select sum(case when col1<0 then 0 else col1 end ) from table > i get the following error: > No realization found for OLAPContext, CUBE_UNMATCHED_AGGREGATION[FunctionDesc > [expression=SUM, parameter=CASE(<($8, 0), 0, $8), returnType=null]], > rel#36838:OLAPTableScan.OLAP.[](table=[DEFAULT, table],ctx=,fields=[0, 1, 2, > 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, > 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, > 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, > 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, > 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, > 100, 101, 102, 103, 104, 105, 106]) while executing SQL: "select sum (case > when col1 <0 then 0 else col1 end ) from table LIMIT 5" > > is there any way to support this sum case expression in the query engine? > Also I get the same error when the sum operation is done on a column derived > from a case expression in a subquery. > eg: select sum(a.col1) from (select case when col1<0 then 0 else col1 end as > col1 from table) a > Thanks, > Gladson > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4206) Build kylin on EMR 5.23. The kylin version is 2.6.4. When building the cube, the hive table cannot be found
[ https://issues.apache.org/jira/browse/KYLIN-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-4206: Fix Version/s: v2.6.6 > Build kylin on EMR 5.23. The kylin version is 2.6.4. When building the cube, > the hive table cannot be found > --- > > Key: KYLIN-4206 > URL: https://issues.apache.org/jira/browse/KYLIN-4206 > Project: Kylin > Issue Type: Bug > Components: Environment >Affects Versions: v2.6.4 > Environment: EMR 5.23(hadoop 2.8.5\HBase 1.4.9\hive 2.3.4\Spark > 2.4.0\Tez 0.9.1\HCatalog 2.3.4\Zookeeper 3.4.13) > kylin 2.6.4 >Reporter: rongneng.wei >Assignee: rongneng.wei >Priority: Major > Fix For: v3.1.0, v3.0.2, v2.6.6 > > Attachments: kylin.properties, kylin_hive_conf.xml, kylin_job_conf.xml > > > hi， > I Build kylin on EMR 5.23. The kylin version is 2.6.4.When building the > cube, the hive table cannot be found.The detailed error information is as > follows： > java.lang.RuntimeException: java.io.IOException: > NoSuchObjectException(message:kylin_flat_db_test1.kylin_intermediate_kylin_sales_cube_4e93b31d_3be2_c9e8_55de_a9814f63c4ba > table not found)java.lang.RuntimeException: java.io.IOException: > NoSuchObjectException(message:kylin_flat_db_test1.kylin_intermediate_kylin_sales_cube_4e93b31d_3be2_c9e8_55de_a9814f63c4ba > table not found) at > org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:83) > at > org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:126) > at > org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:104) > at > org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:131) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > On the EMR, hive metadata is shared by glue, and the URL of Metastore is > configured in hive-site.xml. > hive.metastore.uris > thrift://ip-172-40-15-164.ec2.internal:9083 > JDBC connect string for a JDBC metastore > > > hive.metastore.client.factory.class > > com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory > > But when I use hive's own metadata, that is, don't use glue to share > metadata, the above exception will not occur, comment out the following > configuration. > > But since EMR uses shared metadata, if you don't use metadata sharing, then I > can't query other hive tables built by the cluster. > The configuration file is detailed in the attachment. Please help me solve > this problem.Thank you。 > Best regard. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4383) Kylin Integrated Issue with Amazon EMR and AWS Glue in HiveMetaStoreClientFactory.java
[ https://issues.apache.org/jira/browse/KYLIN-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-4383: Fix Version/s: v3.0.2 > Kylin Integrated Issue with Amazon EMR and AWS Glue in > HiveMetaStoreClientFactory.java > -- > > Key: KYLIN-4383 > URL: https://issues.apache.org/jira/browse/KYLIN-4383 > Project: Kylin > Issue Type: Bug > Components: Integration >Affects Versions: v3.0.0 > Environment: Amazon EMR 5.29.0 + AWS Glue >Reporter: Tian Shi >Assignee: Xiaoxiang Yu >Priority: Blocker > Labels: AWS, easyfix > Fix For: v3.1.0, v3.0.2, v2.6.6 > > Original Estimate: 24h > Remaining Estimate: 24h > > Following the official docs with [link Install Kylin on AWS > EMR|http://kylin.apache.org/docs31/install/kylin_aws_emr.html], when choosing > AWS Glue as Hive metastore, it does not take effect with the configuration > "kylin.source.hive.metadata-type=gluecatalog". > Then I clone the master branch and compile Kylin from the source code > (3.1.0-SNAPSHOT) and deploy it on EMR with Glue, when "Load table from the > tree", I encountered with error "cannot create metastore client gluecatalog". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4206) Build kylin on EMR 5.23. The kylin version is 2.6.4. When building the cube, the hive table cannot be found
[ https://issues.apache.org/jira/browse/KYLIN-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-4206: Fix Version/s: v3.0.2 > Build kylin on EMR 5.23. The kylin version is 2.6.4. When building the cube, > the hive table cannot be found > --- > > Key: KYLIN-4206 > URL: https://issues.apache.org/jira/browse/KYLIN-4206 > Project: Kylin > Issue Type: Bug > Components: Environment >Affects Versions: v2.6.4 > Environment: EMR 5.23(hadoop 2.8.5\HBase 1.4.9\hive 2.3.4\Spark > 2.4.0\Tez 0.9.1\HCatalog 2.3.4\Zookeeper 3.4.13) > kylin 2.6.4 >Reporter: rongneng.wei >Assignee: rongneng.wei >Priority: Major > Fix For: v3.1.0, v3.0.2 > > Attachments: kylin.properties, kylin_hive_conf.xml, kylin_job_conf.xml > > > hi， > I Build kylin on EMR 5.23. The kylin version is 2.6.4.When building the > cube, the hive table cannot be found.The detailed error information is as > follows： > java.lang.RuntimeException: java.io.IOException: > NoSuchObjectException(message:kylin_flat_db_test1.kylin_intermediate_kylin_sales_cube_4e93b31d_3be2_c9e8_55de_a9814f63c4ba > table not found)java.lang.RuntimeException: java.io.IOException: > NoSuchObjectException(message:kylin_flat_db_test1.kylin_intermediate_kylin_sales_cube_4e93b31d_3be2_c9e8_55de_a9814f63c4ba > table not found) at > org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:83) > at > org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:126) > at > org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:104) > at > org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:131) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > On the EMR, hive metadata is shared by glue, and the URL of Metastore is > configured in hive-site.xml. > hive.metastore.uris > thrift://ip-172-40-15-164.ec2.internal:9083 > JDBC connect string for a JDBC metastore > > > hive.metastore.client.factory.class > > com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory > > But when I use hive's own metadata, that is, don't use glue to share > metadata, the above exception will not occur, comment out the following > configuration. > > But since EMR uses shared metadata, if you don't use metadata sharing, then I > can't query other hive tables built by the cluster. > The configuration file is detailed in the attachment. Please help me solve > this problem.Thank you。 > Best regard. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3976) Failed to find metadata store by url: kylin_metadata@hbase
[ https://issues.apache.org/jira/browse/KYLIN-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-3976. Resolution: Not A Problem > Failed to find metadata store by url: kylin_metadata@hbase > -- > > Key: KYLIN-3976 > URL: https://issues.apache.org/jira/browse/KYLIN-3976 > Project: Kylin > Issue Type: Bug > Components: Metadata > Environment: > kylin2.3.2+hadoop3.1.2+hive2.3.4+hbase1.2.0+zookeeper3.4.5 >Reporter: ZhangXihao >Priority: Blocker > Original Estimate: 48h > Remaining Estimate: 48h > > 2019-04-24 17:30:01,894 DEBUG [main] hbase.HBaseConnection:337 : Creating > HTable 'kylin_metadata' > 2019-04-24 17:30:01,936 INFO [main] Configuration.deprecation:1174 : > hadoop.native.lib is deprecated. Instead, use io.native.lib.available > 2019-04-24 17:30:02,089 DEBUG [main] util.ZookeeperDistributedLock:223 : > 7116@ambari02 trying to unlock > /kylin/kylin_metadata/create_htable/kylin_metadata/lock > 2019-04-24 17:30:02,099 INFO [main] util.ZookeeperDistributedLock:234 : > 7116@ambari02 released lock at > /kylin/kylin_metadata/create_htable/kylin_metadata/lock > Exception in thread "main" java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > at > org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:96) > at > org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:108) > at > org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94) > at > org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:90) > ... 3 more > Caused by: java.lang.NoSuchMethodError: > org.apache.hadoop.hbase.HTableDescriptor.addFamily(Lorg/apache/hadoop/hbase/HColumnDescriptor;)Lorg/apache/hadoop/hbase/HTableDescriptor; > at > org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:344) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:111) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:92) > ... 8 more > 2019-04-24 17:30:02,105 INFO [close-hbase-conn] hbase.HBaseConnection:137 : > Closing HBase connections... > 2019-04-24 17:30:02,107 INFO [close-hbase-conn] > client.ConnectionManager$HConnectionImplementation:1830 : Closing zookeeper > sessionid=0x16a4c1bd0c80008 > 2019-04-24 17:30:02,111 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : > Session: 0x16a4c1bd0c80008 closed > 2019-04-24 17:30:02,111 INFO [main-EventThread] zookeeper.ClientCnxn:512 : > EventThread shut down > 2019-04-24 17:30:02,114 INFO [Thread-7] zookeeper.ZooKeeper:684 : Session: > 0x16a4c1bd0c80009 closed > 2019-04-24 17:30:02,114 INFO [main-EventThread] zookeeper.ClientCnxn:512 : > EventThread shut down -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4383) Kylin Integrated Issue with Amazon EMR and AWS Glue in HiveMetaStoreClientFactory.java
[ https://issues.apache.org/jira/browse/KYLIN-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-4383: Fix Version/s: v2.6.6 v3.1.0 > Kylin Integrated Issue with Amazon EMR and AWS Glue in > HiveMetaStoreClientFactory.java > -- > > Key: KYLIN-4383 > URL: https://issues.apache.org/jira/browse/KYLIN-4383 > Project: Kylin > Issue Type: Bug > Components: Integration >Affects Versions: v3.0.0 > Environment: Amazon EMR 5.29.0 + AWS Glue >Reporter: Tian Shi >Assignee: Xiaoxiang Yu >Priority: Blocker > Labels: AWS, easyfix > Fix For: v3.1.0, v2.6.6 > > Original Estimate: 24h > Remaining Estimate: 24h > > Following the official docs with [link Install Kylin on AWS > EMR|http://kylin.apache.org/docs31/install/kylin_aws_emr.html], when choosing > AWS Glue as Hive metastore, it does not take effect with the configuration > "kylin.source.hive.metadata-type=gluecatalog". > Then I clone the master branch and compile Kylin from the source code > (3.1.0-SNAPSHOT) and deploy it on EMR with Glue, when "Load table from the > tree", I encountered with error "cannot create metastore client gluecatalog". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4370) Spark job failing with JDBC source on 8th step with error : org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table or view not found: `default`.`kylin_inte
[ https://issues.apache.org/jira/browse/KYLIN-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi resolved KYLIN-4370. -- Resolution: Fixed > Spark job failing with JDBC source on 8th step with error : > org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table or view > not found: `default`.`kylin_intermediate table' > > > Key: KYLIN-4370 > URL: https://issues.apache.org/jira/browse/KYLIN-4370 > Project: Kylin > Issue Type: Bug >Affects Versions: v3.0.0 >Reporter: Sonu Singh >Assignee: wangrupeng >Priority: Blocker > Fix For: v3.0.2, v2.6.6 > > Attachments: image-2020-02-19-17-20-20-076.png, > image-2020-02-19-18-48-00-857.png, image-2020-02-21-23-44-53-459.png, > image-2020-02-21-23-47-52-247.png > > > 020-02-03 11:18:45,899 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:45 INFO Client:54 - Application report for > application_1580106368736_0431 (state: RUNNING) > 2020-02-03 11:18:46,901 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46 INFO Client:54 - Application report for > application_1580106368736_0431 (state: FINISHED) > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46 INFO Client:54 - > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > client token: N/A > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > diagnostics: User class threw exception: java.lang.RuntimeException: error > execute org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table > or view not found: > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b`;; > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 'UnresolvedRelation > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b` > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:34) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:36) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > java.lang.reflect.Method.invoke(Method.java:498) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > Caused by: org.apache.spark.sql.AnalysisException: Table or view not found: > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b`;; > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 'UnresolvedRelation > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b` > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:86) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:84) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127) > 2020-02-03 11:18:46,908 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:84) > 2020-02-03 11:18:46,908 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:92)
[jira] [Commented] (KYLIN-4370) Spark job failing with JDBC source on 8th step with error : org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table or view not found: `default`.`kylin_int
[ https://issues.apache.org/jira/browse/KYLIN-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047544#comment-17047544 ] ASF subversion and git services commented on KYLIN-4370: Commit 18230a978ce70019411958849685a0adb5d2d862 in kylin's branch refs/heads/3.0.x from rupengwang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=18230a9 ] KYLIN-4370 Fix "Table or view not found" error when build cube with spark for JDBC source at step "Build cube by Layer" > Spark job failing with JDBC source on 8th step with error : > org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table or view > not found: `default`.`kylin_intermediate table' > > > Key: KYLIN-4370 > URL: https://issues.apache.org/jira/browse/KYLIN-4370 > Project: Kylin > Issue Type: Bug >Affects Versions: v3.0.0 >Reporter: Sonu Singh >Assignee: wangrupeng >Priority: Blocker > Fix For: v3.0.2, v2.6.6 > > Attachments: image-2020-02-19-17-20-20-076.png, > image-2020-02-19-18-48-00-857.png, image-2020-02-21-23-44-53-459.png, > image-2020-02-21-23-47-52-247.png > > > 020-02-03 11:18:45,899 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:45 INFO Client:54 - Application report for > application_1580106368736_0431 (state: RUNNING) > 2020-02-03 11:18:46,901 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46 INFO Client:54 - Application report for > application_1580106368736_0431 (state: FINISHED) > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46 INFO Client:54 - > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > client token: N/A > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > diagnostics: User class threw exception: java.lang.RuntimeException: error > execute org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table > or view not found: > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b`;; > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 'UnresolvedRelation > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b` > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:34) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:36) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > java.lang.reflect.Method.invoke(Method.java:498) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > Caused by: org.apache.spark.sql.AnalysisException: Table or view not found: > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b`;; > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 'UnresolvedRelation > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b` > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:86) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:84) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode
[jira] [Commented] (KYLIN-4370) Spark job failing with JDBC source on 8th step with error : org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table or view not found: `default`.`kylin_int
[ https://issues.apache.org/jira/browse/KYLIN-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047542#comment-17047542 ] ASF GitHub Bot commented on KYLIN-4370: --- shaofengshi commented on pull request #1112: KYLIN-4370 Fix "Table or view not found" error when using JDBC source URL: https://github.com/apache/kylin/pull/1112 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Spark job failing with JDBC source on 8th step with error : > org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table or view > not found: `default`.`kylin_intermediate table' > > > Key: KYLIN-4370 > URL: https://issues.apache.org/jira/browse/KYLIN-4370 > Project: Kylin > Issue Type: Bug >Affects Versions: v3.0.0 >Reporter: Sonu Singh >Assignee: wangrupeng >Priority: Blocker > Fix For: v3.0.2, v2.6.6 > > Attachments: image-2020-02-19-17-20-20-076.png, > image-2020-02-19-18-48-00-857.png, image-2020-02-21-23-44-53-459.png, > image-2020-02-21-23-47-52-247.png > > > 020-02-03 11:18:45,899 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:45 INFO Client:54 - Application report for > application_1580106368736_0431 (state: RUNNING) > 2020-02-03 11:18:46,901 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46 INFO Client:54 - Application report for > application_1580106368736_0431 (state: FINISHED) > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46 INFO Client:54 - > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > client token: N/A > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > diagnostics: User class threw exception: java.lang.RuntimeException: error > execute org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table > or view not found: > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b`;; > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 'UnresolvedRelation > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b` > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:34) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:36) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > java.lang.reflect.Method.invoke(Method.java:498) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > Caused by: org.apache.spark.sql.AnalysisException: Table or view not found: > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b`;; > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 'UnresolvedRelation > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b` > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:86) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.app
[jira] [Commented] (KYLIN-4370) Spark job failing with JDBC source on 8th step with error : org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table or view not found: `default`.`kylin_int
[ https://issues.apache.org/jira/browse/KYLIN-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047543#comment-17047543 ] ASF subversion and git services commented on KYLIN-4370: Commit ebd49cb4549c74e014a01a05de00e32e87391ed5 in kylin's branch refs/heads/master from rupengwang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=ebd49cb ] KYLIN-4370 Fix "Table or view not found" error when build cube with spark for JDBC source at step "Build cube by Layer" > Spark job failing with JDBC source on 8th step with error : > org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table or view > not found: `default`.`kylin_intermediate table' > > > Key: KYLIN-4370 > URL: https://issues.apache.org/jira/browse/KYLIN-4370 > Project: Kylin > Issue Type: Bug >Affects Versions: v3.0.0 >Reporter: Sonu Singh >Assignee: wangrupeng >Priority: Blocker > Fix For: v3.0.2, v2.6.6 > > Attachments: image-2020-02-19-17-20-20-076.png, > image-2020-02-19-18-48-00-857.png, image-2020-02-21-23-44-53-459.png, > image-2020-02-21-23-47-52-247.png > > > 020-02-03 11:18:45,899 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:45 INFO Client:54 - Application report for > application_1580106368736_0431 (state: RUNNING) > 2020-02-03 11:18:46,901 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46 INFO Client:54 - Application report for > application_1580106368736_0431 (state: FINISHED) > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46 INFO Client:54 - > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > client token: N/A > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > diagnostics: User class threw exception: java.lang.RuntimeException: error > execute org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: Table > or view not found: > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b`;; > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 'UnresolvedRelation > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b` > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46,906 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:34) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:36) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > java.lang.reflect.Method.invoke(Method.java:498) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > Caused by: org.apache.spark.sql.AnalysisException: Table or view not found: > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b`;; > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 'UnresolvedRelation > `default`.`kylin_intermediate_cube_2_30012020_spark_ded81d01_32e9_4f82_ea97_61788fdfd59b` > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:86) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:84) > 2020-02-03 11:18:46,907 INFO [pool-25-thread-1] spark.SparkExecutable:33 : at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNod
[GitHub] [kylin] shaofengshi merged pull request #1112: KYLIN-4370 Fix "Table or view not found" error when using JDBC source
shaofengshi merged pull request #1112: KYLIN-4370 Fix "Table or view not found" error when using JDBC source URL: https://github.com/apache/kylin/pull/1112 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4378) Not able to build a cube with Kafka Streaming Source as fact table and Jdbc Source as lookup table.
[ https://issues.apache.org/jira/browse/KYLIN-4378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047541#comment-17047541 ] Shao Feng Shi commented on KYLIN-4378: -- Hi Sonu, even Kylin 3.0, don't support Kafka join with JDBC data. And no plan to support that so far. Is it possible to port the JDBC lookup table to Hive? > Not able to build a cube with Kafka Streaming Source as fact table and Jdbc > Source as lookup table. > --- > > Key: KYLIN-4378 > URL: https://issues.apache.org/jira/browse/KYLIN-4378 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v3.0.0 >Reporter: Sonu Singh >Priority: Blocker > Fix For: v3.1.0 > > > I am not able to build a cube with Kafka Streaming source and Jdbc Source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4239) Project list is only visible to super administrators and owners
[ https://issues.apache.org/jira/browse/KYLIN-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047537#comment-17047537 ] Shao Feng Shi commented on KYLIN-4239: -- [~kyle0805] xiang, have you worked on it? thx > Project list is only visible to super administrators and owners > --- > > Key: KYLIN-4239 > URL: https://issues.apache.org/jira/browse/KYLIN-4239 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: v2.6.0 >Reporter: xiang zhang >Assignee: xiang zhang >Priority: Critical > Fix For: Future > > > In the current kylin display project list, the user can see all the project > lists and the corresponding models and cubes, It is safer and easier to > manage when only deplays projects whose owner was current user and the > corresponding super administrator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4237) Return error when execute 'explain plan for SQL' to get the execution plan of SQL
[ https://issues.apache.org/jira/browse/KYLIN-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4237. > Return error when execute 'explain plan for SQL' to get the execution plan of > SQL > - > > Key: KYLIN-4237 > URL: https://issues.apache.org/jira/browse/KYLIN-4237 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v3.0.0-alpha2 >Reporter: Yaqian Zhang >Assignee: Yaqian Zhang >Priority: Minor > Fix For: v3.1.0, v3.0.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4235) Failed to load table metadata from JDBC data source
[ https://issues.apache.org/jira/browse/KYLIN-4235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4235. > Failed to load table metadata from JDBC data source > --- > > Key: KYLIN-4235 > URL: https://issues.apache.org/jira/browse/KYLIN-4235 > Project: Kylin > Issue Type: Bug >Affects Versions: v3.0.0-alpha2, v2.6.4 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: v3.1.0, v3.0.1 > > Attachments: image-2019-11-01-23-45-56-319.png > > > After adding the following configurations to the project, the table metadata > still cannot be loaded > !image-2019-11-01-23-45-56-319.png|width=480,height=373! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4234) Add a project extractor for Kylin admin
[ https://issues.apache.org/jira/browse/KYLIN-4234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047536#comment-17047536 ] Shao Feng Shi commented on KYLIN-4234: -- +1 good feature, especially for administrator. > Add a project extractor for Kylin admin > --- > > Key: KYLIN-4234 > URL: https://issues.apache.org/jira/browse/KYLIN-4234 > Project: Kylin > Issue Type: New Feature >Reporter: ZhouKang >Priority: Major > > the data in Kylin metastore is not relational, so when there are many > projects or models in the system, it's hard to get all information. > sometimes we need feature such as: > * project's email list > * cube list and status (last modify time) > > Suggestions are welcomed, thanks~ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4234) Add a project extractor for Kylin admin
[ https://issues.apache.org/jira/browse/KYLIN-4234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi updated KYLIN-4234: - Summary: Add a project extractor for Kylin admin (was: add a project extractor for Kylin admin) > Add a project extractor for Kylin admin > --- > > Key: KYLIN-4234 > URL: https://issues.apache.org/jira/browse/KYLIN-4234 > Project: Kylin > Issue Type: New Feature >Reporter: ZhouKang >Priority: Major > > the data in Kylin metastore is not relational, so when there are many > projects or models in the system, it's hard to get all information. > sometimes we need feature such as: > * project's email list > * cube list and status (last modify time) > > Suggestions are welcomed, thanks~ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4233) simple time schedule module for kylin job
[ https://issues.apache.org/jira/browse/KYLIN-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047535#comment-17047535 ] Shao Feng Shi commented on KYLIN-4233: -- Hi Kang, what's your design for this? We also had this idea before, but abandon it after some investigation. Firstly, the scheduler need be enough robust, stable. Secondly, as Kylin can be deployed as a cluster, while the scheduler can only have one active instance. It need auto fail-over. Please consider these when you design it. Thanks! > simple time schedule module for kylin job > - > > Key: KYLIN-4233 > URL: https://issues.apache.org/jira/browse/KYLIN-4233 > Project: Kylin > Issue Type: New Feature >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > > add a simple time schedule module, so user can use kylin instead of > third-party schedule system -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4236) Can kylin support more than 64 dimensions?
[ https://issues.apache.org/jira/browse/KYLIN-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi closed KYLIN-4236. > Can kylin support more than 64 dimensions? > -- > > Key: KYLIN-4236 > URL: https://issues.apache.org/jira/browse/KYLIN-4236 > Project: Kylin > Issue Type: Wish > Components: Metadata >Affects Versions: v2.6.4 >Reporter: zhao jintao >Priority: Critical > > As we know: > The default maximum size of kylin dimension is 63, which is specified by the > kylin.cube.rowkey.max-size value in the configuration file. > The default maximum cubic combination of kylin cuboids is 32768, which is > specified by the kylin.cube.aggrgroup.max-combination value in the > configuration file; > We can use many various methods to reduce cuboid's combination > In our system, we used more than 80 dimensions, the size combination is just > 1000 combinations, is very fixed. > After setting kylin.cube.rowkey.max-size to 128, if I save a cube which has > more than 64 dimensions, kylin throws exception: "Cuboid -1 out of range > 0-9223372036854775806". > After analysis, the cuboid in kylin is identified by the N-bit binary number, > and the cuboid is saved with long type. If a cube has more than 64 > dimensions, the cuboid value exceeds the maximum value of the long type and > the error is reported. > Can kylin support this scene which has a limited combination of category > dimensions (less than 1000) but has large number of dimensions (more than 80)? > Can kylin save the cuboid value using BigInteger type? What risk if kylin > save cuboid uses BigInteger type? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4236) Can kylin support more than 64 dimensions?
[ https://issues.apache.org/jira/browse/KYLIN-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi resolved KYLIN-4236. -- Resolution: Won't Fix > Can kylin support more than 64 dimensions? > -- > > Key: KYLIN-4236 > URL: https://issues.apache.org/jira/browse/KYLIN-4236 > Project: Kylin > Issue Type: Wish > Components: Metadata >Affects Versions: v2.6.4 >Reporter: zhao jintao >Priority: Critical > > As we know: > The default maximum size of kylin dimension is 63, which is specified by the > kylin.cube.rowkey.max-size value in the configuration file. > The default maximum cubic combination of kylin cuboids is 32768, which is > specified by the kylin.cube.aggrgroup.max-combination value in the > configuration file; > We can use many various methods to reduce cuboid's combination > In our system, we used more than 80 dimensions, the size combination is just > 1000 combinations, is very fixed. > After setting kylin.cube.rowkey.max-size to 128, if I save a cube which has > more than 64 dimensions, kylin throws exception: "Cuboid -1 out of range > 0-9223372036854775806". > After analysis, the cuboid in kylin is identified by the N-bit binary number, > and the cuboid is saved with long type. If a cube has more than 64 > dimensions, the cuboid value exceeds the maximum value of the long type and > the error is reported. > Can kylin support this scene which has a limited combination of category > dimensions (less than 1000) but has large number of dimensions (more than 80)? > Can kylin save the cuboid value using BigInteger type? What risk if kylin > save cuboid uses BigInteger type? -- This message was sent by Atlassian Jira (v8.3.4#803005)
