[GitHub] [kylin] nichunen merged pull request #664: KYLIN-4017 Build engine get zk(zookeeper) lock failed when building job, it causes the whole build engine doesn't work
nichunen merged pull request #664: KYLIN-4017 Build engine get zk(zookeeper) lock failed when building job, it causes the whole build engine doesn't work URL: https://github.com/apache/kylin/pull/664 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4017) Build engine get zk(zookeeper) lock failed when building job, it causes the whole build engine doesn't work.
[
https://issues.apache.org/jira/browse/KYLIN-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862675#comment-16862675
]
ASF subversion and git services commented on KYLIN-4017:
Commit a74dc055a163e6adb0269e0924fbc78e8f997db2 in kylin's branch
refs/heads/master from wangxiaojing
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a74dc05 ]
KYLIN-4017 Build engine get zk(zookeeper) lock failed when building job, this
causes the whole build engine doesn't work.
> Build engine get zk(zookeeper) lock failed when building job, it causes the
> whole build engine doesn't work.
>
>
> Key: KYLIN-4017
> URL: https://issues.apache.org/jira/browse/KYLIN-4017
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Tools, Build and Test
>Affects Versions: Future, v3.0.0, v3.0.0-alpha
>Reporter: wangxiaojing
>Priority: Critical
> Labels: build
> Fix For: Future, v3.0.0-alpha
>
> Attachments: zkinstancestart.png
>
>
> Kylin has ZK acquisition lock exception when it is building job. Only restart
> can solve this problem. Otherwise, it can't build job ,the whole build engine
> doesn't work.This problem will continue to occur one day after restart. Log
> looks like below:
> {code:java}
> 2019-05-15 11:09:43,209 INFO [FetcherRunner 1910115020-57]
> threadpool.FetcherRunner:59 :
> CubingJob{id=878974c4-4c65-88a4-a912-b238fcc33bdc, name=BUILD CUBE -
> es_report_respnse_rate_cube - 2019051300_2019051400 - GMT+08:00
> 2019-05-15 11:03:15, state=READY} prepare to schedule and its priority is 20
> 2019-05-15 11:09:43,209 INFO [FetcherRunner 1910115020-57]
> threadpool.FetcherRunner:63 :
> CubingJob{id=878974c4-4c65-88a4-a912-b238fcc33bdc, name=BUILD CUBE -
> es_report_respnse_rate_cube - 2019051300_2019051400 - GMT+08:00
> 2019-05-15 11:03:15, state=READY} scheduled
> 2019-05-15 11:09:43,209 DEBUG [Scheduler 719764581 Job
> 878974c4-4c65-88a4-a912-b238fcc33bdc-132]
> zookeeper.ZookeeperDistributedLock:92 :
> 18...@bigdata-kylin-build01.gz01.diditaxi.com trying to lock
> /job_engine/lock/878974c4-4c65-88a4-a912-b238fcc33bdc
> 2019-05-15 11:09:43,212 ERROR [pool-12-thread-10]
> threadpool.DistributedScheduler:115 : unknown error execute
> job:878974c4-4c65-88a4-a912-b238fcc33bdc in server:
> 18...@bigdata-kylin-build01.gz01.diditaxi.com
> java.lang.IllegalStateException: Error while
> 18...@bigdata-kylin-build01.gz01.diditaxi.com trying to lock
> /job_engine/lock/878974c4-4c65-88a4-a912-b238fcc33bdc
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:99)
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperJobLock.lock(ZookeeperJobLock.java:41)
> at
> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:105)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: instance must be started before
> calling this method
> at
> org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:176)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.create(CuratorFrameworkImpl.java:351)
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:95)
> ... 5 more{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-4017) Build engine get zk(zookeeper) lock failed when building job, it causes the whole build engine doesn't work.
[
https://issues.apache.org/jira/browse/KYLIN-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862674#comment-16862674
]
ASF GitHub Bot commented on KYLIN-4017:
---
nichunen commented on pull request #664: KYLIN-4017 Build engine get
zk(zookeeper) lock failed when building job, it causes the whole build engine
doesn't work
URL: https://github.com/apache/kylin/pull/664
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Build engine get zk(zookeeper) lock failed when building job, it causes the
> whole build engine doesn't work.
>
>
> Key: KYLIN-4017
> URL: https://issues.apache.org/jira/browse/KYLIN-4017
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Tools, Build and Test
>Affects Versions: Future, v3.0.0, v3.0.0-alpha
>Reporter: wangxiaojing
>Priority: Critical
> Labels: build
> Fix For: Future, v3.0.0-alpha
>
> Attachments: zkinstancestart.png
>
>
> Kylin has ZK acquisition lock exception when it is building job. Only restart
> can solve this problem. Otherwise, it can't build job ,the whole build engine
> doesn't work.This problem will continue to occur one day after restart. Log
> looks like below:
> {code:java}
> 2019-05-15 11:09:43,209 INFO [FetcherRunner 1910115020-57]
> threadpool.FetcherRunner:59 :
> CubingJob{id=878974c4-4c65-88a4-a912-b238fcc33bdc, name=BUILD CUBE -
> es_report_respnse_rate_cube - 2019051300_2019051400 - GMT+08:00
> 2019-05-15 11:03:15, state=READY} prepare to schedule and its priority is 20
> 2019-05-15 11:09:43,209 INFO [FetcherRunner 1910115020-57]
> threadpool.FetcherRunner:63 :
> CubingJob{id=878974c4-4c65-88a4-a912-b238fcc33bdc, name=BUILD CUBE -
> es_report_respnse_rate_cube - 2019051300_2019051400 - GMT+08:00
> 2019-05-15 11:03:15, state=READY} scheduled
> 2019-05-15 11:09:43,209 DEBUG [Scheduler 719764581 Job
> 878974c4-4c65-88a4-a912-b238fcc33bdc-132]
> zookeeper.ZookeeperDistributedLock:92 :
> 18...@bigdata-kylin-build01.gz01.diditaxi.com trying to lock
> /job_engine/lock/878974c4-4c65-88a4-a912-b238fcc33bdc
> 2019-05-15 11:09:43,212 ERROR [pool-12-thread-10]
> threadpool.DistributedScheduler:115 : unknown error execute
> job:878974c4-4c65-88a4-a912-b238fcc33bdc in server:
> 18...@bigdata-kylin-build01.gz01.diditaxi.com
> java.lang.IllegalStateException: Error while
> 18...@bigdata-kylin-build01.gz01.diditaxi.com trying to lock
> /job_engine/lock/878974c4-4c65-88a4-a912-b238fcc33bdc
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:99)
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperJobLock.lock(ZookeeperJobLock.java:41)
> at
> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:105)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: instance must be started before
> calling this method
> at
> org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:176)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.create(CuratorFrameworkImpl.java:351)
> at
> org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:95)
> ... 5 more{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] nichunen commented on a change in pull request #664: KYLIN-4017 Build engine get zk(zookeeper) lock failed when building job, it causes the whole build engine doesn't work
nichunen commented on a change in pull request #664: KYLIN-4017 Build engine
get zk(zookeeper) lock failed when building job, it causes the whole build
engine doesn't work
URL: https://github.com/apache/kylin/pull/664#discussion_r293188955
##
File path: core-common/src/main/java/org/apache/kylin/common/util/ZKUtil.java
##
@@ -84,7 +84,7 @@ public void onRemoval(RemovalNotification notification
logger.error("Error at closing " + curator, ex);
}
}
-}).expireAfterWrite(1, TimeUnit.DAYS).build();
+}).expireAfterWrite(1, TimeUnit.DAYS).build();//never expired
Review comment:
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-4015. - Resolution: Fixed > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862648#comment-16862648 ] ASF subversion and git services commented on KYLIN-4015: Commit fb486745ba7ca024b117ba2cc7ec95f64c41f401 in kylin's branch refs/heads/2.6.x from shqmh [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=fb48674 ] KYLIN-4015 change uhc path filter .dci to FactDistinctColumnsReducer.DIMENSION_COL_INFO_FILE_POSTFIX > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862647#comment-16862647 ] ASF subversion and git services commented on KYLIN-4015: Commit fc3e6e52140b8e724043e7a0134d06aa4c9e34ca in kylin's branch refs/heads/2.6.x from shqmh [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=fc3e6e5 ] KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files to solve the problem that MR engine run failed because the ".dci" file is not a Sequence file. > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862646#comment-16862646 ] ASF subversion and git services commented on KYLIN-4015: Commit f19f16aad334fd65f9946dd51d369497a15cb9e4 in kylin's branch refs/heads/master from shqmh [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=f19f16a ] KYLIN-4015 change uhc path filter .dci to FactDistinctColumnsReducer.DIMENSION_COL_INFO_FILE_POSTFIX > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862644#comment-16862644 ] ASF GitHub Bot commented on KYLIN-4015: --- shaofengshi commented on pull request #661: KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files URL: https://github.com/apache/kylin/pull/661 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862645#comment-16862645 ] ASF subversion and git services commented on KYLIN-4015: Commit b0ae00315e1fe3d9301581387668fde3ec0efd6f in kylin's branch refs/heads/master from shqmh [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=b0ae003 ] KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files to solve the problem that MR engine run failed because the ".dci" file is not a Sequence file. > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] shaofengshi merged pull request #661: KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files
shaofengshi merged pull request #661: KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files URL: https://github.com/apache/kylin/pull/661 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] zhaojintaozhao commented on a change in pull request #661: KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files
zhaojintaozhao commented on a change in pull request #661: KYLIN-4015 "Build
UHC Dictionary" step filter ".dci" files
URL: https://github.com/apache/kylin/pull/661#discussion_r292914479
##
File path:
engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/filter/UHCDictPathFilter.java
##
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+package org.apache.kylin.engine.mr.steps.filter;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Locale;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class UHCDictPathFilter implements PathFilter {
+private static final Logger logger =
LoggerFactory.getLogger(UHCDictPathFilter.class);
+
+private static final String DCIFILE_POSTFIX = ".dci";
Review comment:
Hi Shaofeng:
I have replaced ".dci" to
FactDistinctColumnsReducer.DIMENSION_COL_INFO_FILE_POSTFIX in my latest PR.
Thank you.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] zhaojintaozhao commented on a change in pull request #661: KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files
zhaojintaozhao commented on a change in pull request #661: KYLIN-4015 "Build
UHC Dictionary" step filter ".dci" files
URL: https://github.com/apache/kylin/pull/661#discussion_r292905054
##
File path:
engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/filter/UHCDictPathFilter.java
##
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+package org.apache.kylin.engine.mr.steps.filter;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Locale;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class UHCDictPathFilter implements PathFilter {
+private static final Logger logger =
LoggerFactory.getLogger(UHCDictPathFilter.class);
+
+private static final String DCIFILE_POSTFIX = ".dci";
+
+@Override
+public boolean accept(Path path) {
+
+Pattern pattern = Pattern.compile(DCIFILE_POSTFIX);
+Matcher matcher =
pattern.matcher(path.getName().toLowerCase(Locale.ROOT));
+
+if (matcher.find()) {
+logger.info("filter file: " + path.getName());
+return false;
Review comment:
Hi Shaofeng
How can we implement it with a whitelist way? After "Extract Fact Table
Distinct Columns" step, Kylin generate files like "colName.dci-r-1",
"colName.rldict-r-2" or "-r-2","-r-3". After filtering path,
only file name like ".rldict-r-1" or "-r-2" are retained?
I want to check whether the type of hdfs file is "SequenceFile", but I don't
find any hadoop API can provide this ability. Hadoop does not perceive the
specific type of file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] coveralls edited a comment on issue #661: KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files
coveralls edited a comment on issue #661: KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files URL: https://github.com/apache/kylin/pull/661#issuecomment-496012899 ## Pull Request Test Coverage Report for [Build 4603](https://coveralls.io/builds/23947371) * **0** of **9** **(0.0%)** changed or added relevant lines in **2** files are covered. * **1234** unchanged lines in **36** files lost coverage. * Overall coverage decreased (**-0.08%**) to **28.218%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UHCDictionaryJob.java](https://coveralls.io/builds/23947371/source?filename=engine-mr%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fmr%2Fsteps%2FUHCDictionaryJob.java#L85) | 0 | 1 | 0.0% | [engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/filter/UHCDictPathFilter.java](https://coveralls.io/builds/23947371/source?filename=engine-mr%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fmr%2Fsteps%2Ffilter%2FUHCDictPathFilter.java#L31) | 0 | 8 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-metadata/src/main/java/org/apache/kylin/dimension/TimeDimEnc.java](https://coveralls.io/builds/23947371/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdimension%2FTimeDimEnc.java#L46) | 1 | 55.56% | | [engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UHCDictionaryJob.java](https://coveralls.io/builds/23947371/source?filename=engine-mr%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fmr%2Fsteps%2FUHCDictionaryJob.java#L86) | 1 | 0.0% | | [stream-core/src/main/java/org/apache/kylin/stream/core/storage/columnar/ColumnarStoreCache.java](https://coveralls.io/builds/23947371/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2Fcolumnar%2FColumnarStoreCache.java#L136) | 1 | 55.81% | | [server-base/src/main/java/org/apache/kylin/rest/util/QueryRequestLimits.java](https://coveralls.io/builds/23947371/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Futil%2FQueryRequestLimits.java#L72) | 1 | 47.62% | | [core-common/src/main/java/org/apache/kylin/common/persistence/ResourceStore.java](https://coveralls.io/builds/23947371/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Fpersistence%2FResourceStore.java#L200) | 1 | 76.9% | | [stream-core/src/main/java/org/apache/kylin/stream/core/query/StreamingQueryProfile.java](https://coveralls.io/builds/23947371/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fquery%2FStreamingQueryProfile.java#L192) | 1 | 37.08% | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/23947371/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 3 | 78.42% | | [source-kafka/src/main/java/org/apache/kylin/source/kafka/hadoop/KafkaFlatTableJob.java](https://coveralls.io/builds/23947371/source?filename=source-kafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fkafka%2Fhadoop%2FKafkaFlatTableJob.java#L164) | 3 | 0.0% | | [core-cube/src/main/java/org/apache/kylin/cube/kv/CubeDimEncMap.java](https://coveralls.io/builds/23947371/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fkv%2FCubeDimEncMap.java#L57) | 3 | 71.43% | | [server-base/src/main/java/org/apache/kylin/rest/request/JobOptimizeRequest.java](https://coveralls.io/builds/23947371/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Frequest%2FJobOptimizeRequest.java#L37) | 4 | 0.0% | | Totals | [](https://coveralls.io/builds/23947371) | | :-- | --: | | Change from base [Build 4541](https://coveralls.io/builds/2358): | -0.08% | | Covered Lines: | 23250 | | Relevant Lines: | 82393 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] codecov-io commented on issue #684: KYLIN-4041 CONCAT NULL not working properly
codecov-io commented on issue #684: KYLIN-4041 CONCAT NULL not working properly URL: https://github.com/apache/kylin/pull/684#issuecomment-501174272 # [Codecov](https://codecov.io/gh/apache/kylin/pull/684?src=pr=h1) Report > Merging [#684](https://codecov.io/gh/apache/kylin/pull/684?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/30ff09f350dbeada44baa4ae8735f08ac0b3268c?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/kylin/pull/684?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #684 +/- ## - Coverage 25.73% 25.72% -0.01% Complexity 6010 6010 Files 1384 1384 Lines 8238482388 +4 Branches 1155111552 +1 - Hits 2120221197 -5 - Misses5913959146 +7 - Partials 2043 2045 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/684?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ain/java/org/apache/kylin/query/udf/ConcatUDF.java](https://codecov.io/gh/apache/kylin/pull/684/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L3VkZi9Db25jYXRVREYuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/684/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `63.84% <0%> (-2.31%)` | `0% <0%> (ø)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/684/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `69.6% <0%> (-0.61%)` | `7% <0%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/684?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/684?src=pr=footer). Last update [30ff09f...2dbdefa](https://codecov.io/gh/apache/kylin/pull/684?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] wangxiaojing123 commented on a change in pull request #664: KYLIN-4017 Build engine get zk(zookeeper) lock failed when building job, it causes the whole build engine doesn't work
wangxiaojing123 commented on a change in pull request #664: KYLIN-4017 Build
engine get zk(zookeeper) lock failed when building job, it causes the whole
build engine doesn't work
URL: https://github.com/apache/kylin/pull/664#discussion_r292799574
##
File path: core-common/src/main/java/org/apache/kylin/common/util/ZKUtil.java
##
@@ -84,7 +84,7 @@ public void onRemoval(RemovalNotification notification
logger.error("Error at closing " + curator, ex);
}
}
-}).expireAfterWrite(1, TimeUnit.DAYS).build();
+}).expireAfterWrite(1, TimeUnit.DAYS).build();//never expired
Review comment:
@nichunen Hello nichunen,Thank you reply. "My concern is why after the cache
of curator expired, the newly created zookeeper client is not in state of
CuratorFrameworkState.STARTED?" Besause now the code never start a new
zookeeper client, it use the old one which state is closed。The new code in my
patch I add two changes:1、never expire,2、in case of client is closed, create a
new one (but before ,there is no this feature) .
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] yanghua commented on issue #683: KYLIN-3991 Provide more config options for Flink cube engine
yanghua commented on issue #683: KYLIN-3991 Provide more config options for Flink cube engine URL: https://github.com/apache/kylin/pull/683#issuecomment-501167831 Hi @shaofengshi I opened a PR to fix the parallelism issue. It seems the Travis has an error. So if you have time, pls review this PR. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4041) CONCAT NULL not working properly
[ https://issues.apache.org/jira/browse/KYLIN-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861825#comment-16861825 ] Yifei Wu commented on KYLIN-4041: - try to fix in PR, plz help to review it: https://github.com/apache/kylin/pull/684 > CONCAT NULL not working properly > > > Key: KYLIN-4041 > URL: https://issues.apache.org/jira/browse/KYLIN-4041 > Project: Kylin > Issue Type: Improvement >Reporter: Yifei Wu >Assignee: Yifei Wu >Priority: Major > > to support SQL 'concat' function, Kylin introduces a UDF named ConcatUDF. > but the NULL case cannot deal with correctly, like the below sql: > ``` > select > concat(meta_categ_name, CAST(null as VARCHAR)) as c1, > concat(meta_categ_name, ' Catagory') as c2 > from test_kylin_fact > left JOIN edw.test_cal_dt as test_cal_dt > ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt > left JOIN test_category_groupings > ON test_kylin_fact.leaf_categ_id = test_category_groupings.leaf_categ_id AND > test_kylin_fact.lstg_site_id = test_category_groupings.site_id > left JOIN edw.test_sites as test_sites > ON test_kylin_fact.lstg_site_id = test_sites.site_id > where meta_categ_name = 'Computers' > group by meta_categ_name > order by meta_categ_name > limit 1 > ``` > when CONCAT value with NULL, the result is expected to be still NULL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] vaful opened a new pull request #684: KYLIN-4041 CONCAT NULL not working properly
vaful opened a new pull request #684: KYLIN-4041 CONCAT NULL not working properly URL: https://github.com/apache/kylin/pull/684 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4041) CONCAT NULL not working properly
[ https://issues.apache.org/jira/browse/KYLIN-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861824#comment-16861824 ] ASF GitHub Bot commented on KYLIN-4041: --- vaful commented on pull request #684: KYLIN-4041 CONCAT NULL not working properly URL: https://github.com/apache/kylin/pull/684 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > CONCAT NULL not working properly > > > Key: KYLIN-4041 > URL: https://issues.apache.org/jira/browse/KYLIN-4041 > Project: Kylin > Issue Type: Improvement >Reporter: Yifei Wu >Assignee: Yifei Wu >Priority: Major > > to support SQL 'concat' function, Kylin introduces a UDF named ConcatUDF. > but the NULL case cannot deal with correctly, like the below sql: > ``` > select > concat(meta_categ_name, CAST(null as VARCHAR)) as c1, > concat(meta_categ_name, ' Catagory') as c2 > from test_kylin_fact > left JOIN edw.test_cal_dt as test_cal_dt > ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt > left JOIN test_category_groupings > ON test_kylin_fact.leaf_categ_id = test_category_groupings.leaf_categ_id AND > test_kylin_fact.lstg_site_id = test_category_groupings.site_id > left JOIN edw.test_sites as test_sites > ON test_kylin_fact.lstg_site_id = test_sites.site_id > where meta_categ_name = 'Computers' > group by meta_categ_name > order by meta_categ_name > limit 1 > ``` > when CONCAT value with NULL, the result is expected to be still NULL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] asfgit commented on issue #684: KYLIN-4041 CONCAT NULL not working properly
asfgit commented on issue #684: KYLIN-4041 CONCAT NULL not working properly URL: https://github.com/apache/kylin/pull/684#issuecomment-501155625 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asfgit commented on issue #684: KYLIN-4041 CONCAT NULL not working properly
asfgit commented on issue #684: KYLIN-4041 CONCAT NULL not working properly URL: https://github.com/apache/kylin/pull/684#issuecomment-501155624 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-4041) CONCAT NULL not working properly
[ https://issues.apache.org/jira/browse/KYLIN-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yifei Wu updated KYLIN-4041: Description: to support SQL 'concat' function, Kylin introduces a UDF named ConcatUDF. but the NULL case cannot deal with correctly, like the below sql: ``` select concat(meta_categ_name, CAST(null as VARCHAR)) as c1, concat(meta_categ_name, ' Catagory') as c2 from test_kylin_fact left JOIN edw.test_cal_dt as test_cal_dt ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt left JOIN test_category_groupings ON test_kylin_fact.leaf_categ_id = test_category_groupings.leaf_categ_id AND test_kylin_fact.lstg_site_id = test_category_groupings.site_id left JOIN edw.test_sites as test_sites ON test_kylin_fact.lstg_site_id = test_sites.site_id where meta_categ_name = 'Computers' group by meta_categ_name order by meta_categ_name limit 1 ``` when CONCAT value with NULL, the result is expected to be still NULL. was: to support SQL 'concat' function, Kylin introduces a UDF named ConcatUDF. but the NULL case cannot deal with correctly, like the below sql: ``` select concat(meta_categ_name, CAST(null as VARCHAR)) as c1, concat(meta_categ_name, ' Catagory') as c2 from test_kylin_fact left JOIN edw.test_cal_dt as test_cal_dt ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt left JOIN test_category_groupings ON test_kylin_fact.leaf_categ_id = test_category_groupings.leaf_categ_id AND test_kylin_fact.lstg_site_id = test_category_groupings.site_id left JOIN edw.test_sites as test_sites ON test_kylin_fact.lstg_site_id = test_sites.site_id where meta_categ_name = 'Computers' group by meta_categ_name order by meta_categ_name limit 1 ``` > CONCAT NULL not working properly > > > Key: KYLIN-4041 > URL: https://issues.apache.org/jira/browse/KYLIN-4041 > Project: Kylin > Issue Type: Improvement >Reporter: Yifei Wu >Assignee: Yifei Wu >Priority: Major > > to support SQL 'concat' function, Kylin introduces a UDF named ConcatUDF. > but the NULL case cannot deal with correctly, like the below sql: > ``` > select > concat(meta_categ_name, CAST(null as VARCHAR)) as c1, > concat(meta_categ_name, ' Catagory') as c2 > from test_kylin_fact > left JOIN edw.test_cal_dt as test_cal_dt > ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt > left JOIN test_category_groupings > ON test_kylin_fact.leaf_categ_id = test_category_groupings.leaf_categ_id AND > test_kylin_fact.lstg_site_id = test_category_groupings.site_id > left JOIN edw.test_sites as test_sites > ON test_kylin_fact.lstg_site_id = test_sites.site_id > where meta_categ_name = 'Computers' > group by meta_categ_name > order by meta_categ_name > limit 1 > ``` > when CONCAT value with NULL, the result is expected to be still NULL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4041) CONCAT NULL not working properly
Yifei Wu created KYLIN-4041: --- Summary: CONCAT NULL not working properly Key: KYLIN-4041 URL: https://issues.apache.org/jira/browse/KYLIN-4041 Project: Kylin Issue Type: Improvement Reporter: Yifei Wu Assignee: Yifei Wu to support SQL 'concat' function, Kylin introduces a UDF named ConcatUDF. but the NULL case cannot deal with correctly, like the below sql: ``` select concat(meta_categ_name, CAST(null as VARCHAR)) as c1, concat(meta_categ_name, ' Catagory') as c2 from test_kylin_fact left JOIN edw.test_cal_dt as test_cal_dt ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt left JOIN test_category_groupings ON test_kylin_fact.leaf_categ_id = test_category_groupings.leaf_categ_id AND test_kylin_fact.lstg_site_id = test_category_groupings.site_id left JOIN edw.test_sites as test_sites ON test_kylin_fact.lstg_site_id = test_sites.site_id where meta_categ_name = 'Computers' group by meta_categ_name order by meta_categ_name limit 1 ``` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (KYLIN-4038) Lookup tables is empty when build on ORC tables.
[
https://issues.apache.org/jira/browse/KYLIN-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-4038:
Comment: was deleted
(was: {code}$ watch org.apache.kylin.dict.lookup.SnapshotTable takeSnapshot
"\{params,target,returnObj}" -x 2
Press Q or Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 172 ms.
ts=2019-06-12 02:35:45; [cost=235.194326ms] result=@ArrayList[
@Object[][
@HiveTable[hive: database=[DEFAULT], table=[KYLIN_ACCOUNT_SIDA]],
@TableDesc[TableDesc\{name='KYLIN_ACCOUNT_SIDA', columns=[ColumnDesc{id='1',
name='ACCOUNT_ID', datatype='bigint', comment='null'}, ColumnDesc\{id='2',
name='ACCOUNT_BUYER_LEVEL', datatype='integer', comment='null'},
ColumnDesc\{id='3', name='ACCOUNT_SELLER_LEVEL', datatype='integer',
comment='null'}, ColumnDesc\{id='4', name='ACCOUNT_COUNTRY',
datatype='varchar(256)', comment='null'}, ColumnDesc\{id='5',
name='ACCOUNT_CONTACT', datatype='varchar(256)', comment='null'}],
sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=DEFAULT],
identity='DEFAULT.KYLIN_ACCOUNT_SIDA'}],
],
@SnapshotTable[
STORAGE_TYPE_METASTORE=@String[metaStore],
tableName=@String[DEFAULT.KYLIN_ACCOUNT_SIDA],
signature=@TableSignature[FileSignature
[path=hdfs://hdp30.hortonworks.com:8020/warehouse/tablespace/managed/hive/kylin_account_sida,
size=0, lastModifiedTime=1560234144865]],
useDictionary=@Boolean[true],
lastBuildTime=@Long[0],
rowIndices=@ArrayList[isEmpty=true;size=0],
dict=@TrieDictionary[org.apache.kylin.dict.TrieDictionary@663163f9],
NULL_STR=@String[ ÿ],
],
null,
]
$
$
$ watch org.apache.kylin.dict.lookup.SnapshotTable takeSnapshot
"\{params,target,returnObj}" -x 2
Press Q or Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 123 ms.
ts=2019-06-12 02:40:06; [cost=611.591702ms] result=@ArrayList[
@Object[][
@HiveTable[hive: database=[DEFAULT], table=[KYLIN_CAL_DT]],
@TableDesc[TableDesc\{name='KYLIN_CAL_DT', columns=[ColumnDesc{id='1',
name='CAL_DT', datatype='date', comment='null'}, ColumnDesc\{id='2',
name='YEAR_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='3',
name='QTR_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='4',
name='MONTH_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='5',
name='WEEK_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='6',
name='AGE_FOR_YEAR_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='7', name='AGE_FOR_QTR_ID', datatype='smallint',
comment='null'}, ColumnDesc\{id='8', name='AGE_FOR_MONTH_ID',
datatype='smallint', comment='null'}, ColumnDesc\{id='9',
name='AGE_FOR_WEEK_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='10', name='AGE_FOR_DT_ID', datatype='smallint',
comment='null'}, ColumnDesc\{id='11', name='AGE_FOR_RTL_YEAR_ID',
datatype='smallint', comment='null'}, ColumnDesc\{id='12',
name='AGE_FOR_RTL_QTR_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='13', name='AGE_FOR_RTL_MONTH_ID', datatype='smallint',
comment='null'}, ColumnDesc\{id='14', name='AGE_FOR_RTL_WEEK_ID',
datatype='smallint', comment='null'}, ColumnDesc\{id='15',
name='AGE_FOR_CS_WEEK_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='16', name='DAY_OF_CAL_ID', datatype='integer', comment='null'},
ColumnDesc\{id='17', name='DAY_OF_YEAR_ID', datatype='smallint',
comment='null'}, ColumnDesc\{id='18', name='DAY_OF_QTR_ID',
datatype='smallint', comment='null'}, ColumnDesc\{id='19',
name='DAY_OF_MONTH_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='20', name='DAY_OF_WEEK_ID', datatype='integer',
comment='null'}, ColumnDesc\{id='21', name='WEEK_OF_YEAR_ID',
datatype='tinyint', comment='null'}, ColumnDesc\{id='22',
name='WEEK_OF_CAL_ID', datatype='integer', comment='null'},
ColumnDesc\{id='23', name='MONTH_OF_QTR_ID', datatype='tinyint',
comment='null'}, ColumnDesc\{id='24', name='MONTH_OF_YEAR_ID',
datatype='tinyint', comment='null'}, ColumnDesc\{id='25',
name='MONTH_OF_CAL_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='26', name='QTR_OF_YEAR_ID', datatype='tinyint',
comment='null'}, ColumnDesc\{id='27', name='QTR_OF_CAL_ID',
datatype='smallint', comment='null'}, ColumnDesc\{id='28',
name='YEAR_OF_CAL_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='29', name='YEAR_END_DT', datatype='varchar(256)',
comment='null'}, ColumnDesc\{id='30', name='QTR_END_DT',
datatype='varchar(256)', comment='null'}, ColumnDesc\{id='31',
name='MONTH_END_DT', datatype='varchar(256)', comment='null'},
ColumnDesc\{id='32', name='WEEK_END_DT', datatype='varchar(256)',
comment='null'}, ColumnDesc\{id='33', name='CAL_DT_NAME',
datatype='varchar(256)', comment='null'}, ColumnDesc\{id='34',
name='CAL_DT_DESC', datatype='varchar(256)', comment='null'},
ColumnDesc\{id='35', name='CAL_DT_SHORT_NAME', datatype='varchar(256)',
comment='null'}, ColumnDesc\{id='36', name='YTD_YN_ID', datatype='tinyint',
[jira] [Commented] (KYLIN-4038) Lookup tables is empty when build on ORC tables.
[
https://issues.apache.org/jira/browse/KYLIN-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861804#comment-16861804
]
XiaoXiang Yu commented on KYLIN-4038:
-
{code}$ watch org.apache.kylin.dict.lookup.SnapshotTable takeSnapshot
"\{params,target,returnObj}" -x 2
Press Q or Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 172 ms.
ts=2019-06-12 02:35:45; [cost=235.194326ms] result=@ArrayList[
@Object[][
@HiveTable[hive: database=[DEFAULT], table=[KYLIN_ACCOUNT_SIDA]],
@TableDesc[TableDesc\{name='KYLIN_ACCOUNT_SIDA', columns=[ColumnDesc{id='1',
name='ACCOUNT_ID', datatype='bigint', comment='null'}, ColumnDesc\{id='2',
name='ACCOUNT_BUYER_LEVEL', datatype='integer', comment='null'},
ColumnDesc\{id='3', name='ACCOUNT_SELLER_LEVEL', datatype='integer',
comment='null'}, ColumnDesc\{id='4', name='ACCOUNT_COUNTRY',
datatype='varchar(256)', comment='null'}, ColumnDesc\{id='5',
name='ACCOUNT_CONTACT', datatype='varchar(256)', comment='null'}],
sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=DEFAULT],
identity='DEFAULT.KYLIN_ACCOUNT_SIDA'}],
],
@SnapshotTable[
STORAGE_TYPE_METASTORE=@String[metaStore],
tableName=@String[DEFAULT.KYLIN_ACCOUNT_SIDA],
signature=@TableSignature[FileSignature
[path=hdfs://hdp30.hortonworks.com:8020/warehouse/tablespace/managed/hive/kylin_account_sida,
size=0, lastModifiedTime=1560234144865]],
useDictionary=@Boolean[true],
lastBuildTime=@Long[0],
rowIndices=@ArrayList[isEmpty=true;size=0],
dict=@TrieDictionary[org.apache.kylin.dict.TrieDictionary@663163f9],
NULL_STR=@String[ ÿ],
],
null,
]
$
$
$ watch org.apache.kylin.dict.lookup.SnapshotTable takeSnapshot
"\{params,target,returnObj}" -x 2
Press Q or Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 123 ms.
ts=2019-06-12 02:40:06; [cost=611.591702ms] result=@ArrayList[
@Object[][
@HiveTable[hive: database=[DEFAULT], table=[KYLIN_CAL_DT]],
@TableDesc[TableDesc\{name='KYLIN_CAL_DT', columns=[ColumnDesc{id='1',
name='CAL_DT', datatype='date', comment='null'}, ColumnDesc\{id='2',
name='YEAR_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='3',
name='QTR_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='4',
name='MONTH_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='5',
name='WEEK_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='6',
name='AGE_FOR_YEAR_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='7', name='AGE_FOR_QTR_ID', datatype='smallint',
comment='null'}, ColumnDesc\{id='8', name='AGE_FOR_MONTH_ID',
datatype='smallint', comment='null'}, ColumnDesc\{id='9',
name='AGE_FOR_WEEK_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='10', name='AGE_FOR_DT_ID', datatype='smallint',
comment='null'}, ColumnDesc\{id='11', name='AGE_FOR_RTL_YEAR_ID',
datatype='smallint', comment='null'}, ColumnDesc\{id='12',
name='AGE_FOR_RTL_QTR_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='13', name='AGE_FOR_RTL_MONTH_ID', datatype='smallint',
comment='null'}, ColumnDesc\{id='14', name='AGE_FOR_RTL_WEEK_ID',
datatype='smallint', comment='null'}, ColumnDesc\{id='15',
name='AGE_FOR_CS_WEEK_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='16', name='DAY_OF_CAL_ID', datatype='integer', comment='null'},
ColumnDesc\{id='17', name='DAY_OF_YEAR_ID', datatype='smallint',
comment='null'}, ColumnDesc\{id='18', name='DAY_OF_QTR_ID',
datatype='smallint', comment='null'}, ColumnDesc\{id='19',
name='DAY_OF_MONTH_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='20', name='DAY_OF_WEEK_ID', datatype='integer',
comment='null'}, ColumnDesc\{id='21', name='WEEK_OF_YEAR_ID',
datatype='tinyint', comment='null'}, ColumnDesc\{id='22',
name='WEEK_OF_CAL_ID', datatype='integer', comment='null'},
ColumnDesc\{id='23', name='MONTH_OF_QTR_ID', datatype='tinyint',
comment='null'}, ColumnDesc\{id='24', name='MONTH_OF_YEAR_ID',
datatype='tinyint', comment='null'}, ColumnDesc\{id='25',
name='MONTH_OF_CAL_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='26', name='QTR_OF_YEAR_ID', datatype='tinyint',
comment='null'}, ColumnDesc\{id='27', name='QTR_OF_CAL_ID',
datatype='smallint', comment='null'}, ColumnDesc\{id='28',
name='YEAR_OF_CAL_ID', datatype='smallint', comment='null'},
ColumnDesc\{id='29', name='YEAR_END_DT', datatype='varchar(256)',
comment='null'}, ColumnDesc\{id='30', name='QTR_END_DT',
datatype='varchar(256)', comment='null'}, ColumnDesc\{id='31',
name='MONTH_END_DT', datatype='varchar(256)', comment='null'},
ColumnDesc\{id='32', name='WEEK_END_DT', datatype='varchar(256)',
comment='null'}, ColumnDesc\{id='33', name='CAL_DT_NAME',
datatype='varchar(256)', comment='null'}, ColumnDesc\{id='34',
name='CAL_DT_DESC', datatype='varchar(256)', comment='null'},
ColumnDesc\{id='35', name='CAL_DT_SHORT_NAME', datatype='varchar(256)',
comment='null'}, ColumnDesc\{id='36', name='YTD_YN_ID', datatype='tinyint',
[jira] [Commented] (KYLIN-4038) Lookup tables is empty when build on ORC tables.
[
https://issues.apache.org/jira/browse/KYLIN-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861786#comment-16861786
]
XiaoXiang Yu commented on KYLIN-4038:
-
Hi, have you ever try external table? Could that meet the same error?
> Lookup tables is empty when build on ORC tables.
>
>
> Key: KYLIN-4038
> URL: https://issues.apache.org/jira/browse/KYLIN-4038
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.2
> Environment: HDP 3 cluster.
>Reporter: Alexander
>Assignee: XiaoXiang Yu
>Priority: Blocker
> Attachments: image-2019-06-11-16-12-09-472.png
>
>
> Create tables, simple copy of learn_kylin project.
> {code:java}
> // Create tables
> create table kylin_sales_sida (trans_id bigint,part_dt date,lstg_format_name
> string,leaf_categ_id bigint,lstg_site_id int,slr_segment_cd smallint,price
> decimal(19,4),item_count bigint,seller_id bigint,buyer_id bigint,ops_user_id
> string,ops_region string) stored as orc tblproperties
> ("orc.compress"="SNAPPY");
> insert into table kylin_sales_sida select * from kylin_sales;
> create table kylin_country_sida (country string,latitude double,longitude
> double,name string) stored as orc tblproperties ("orc.compress"="SNAPPY");
> insert into table kylin_country_sida select * from kylin_country;
> create table kylin_category_groupings_sida (leaf_categ_id
> bigint,leaf_categ_name string,site_id int,categ_busn_mgr
> string,categ_busn_unit string,regn_categ string,user_defined_field1
> string,user_defined_field3 string,kylin_groupings_cre_date
> string,kylin_groupings_upd_date string,kylin_groupings_cre_user
> string,kylin_groupings_upd_user string,meta_categ_id
> decimal(10,0),meta_categ_name string,categ_lvl2_id
> decimal(10,0),categ_lvl3_id decimal(10,0),categ_lvl4_id
> decimal(10,0),categ_lvl5_id decimal(10,0),categ_lvl6_id
> decimal(10,0),categ_lvl7_id decimal(10,0),categ_lvl2_name
> string,categ_lvl3_name string,categ_lvl4_name string,categ_lvl5_name
> string,categ_lvl6_name string,categ_lvl7_name string,categ_flags
> decimal(10,0),adult_categ_yn string,domain_id
> decimal(10,0),user_defined_field5 string,vcs_id decimal(10,0),gcs_id
> decimal(10,0),move_to decimal(10,0),sap_category_id decimal(10,0),src_id
> tinyint,bsns_vrtcl_name string) stored as orc tblproperties
> ("orc.compress"="SNAPPY");
> insert into table kylin_category_groupings_sida select * from
> kylin_category_groupings;
> create table kylin_cal_dt_sida (cal_dt date,year_beg_dt date,qtr_beg_dt
> date,month_beg_dt date,week_beg_dt date,age_for_year_id
> smallint,age_for_qtr_id smallint,age_for_month_id smallint,age_for_week_id
> smallint,age_for_dt_id smallint,age_for_rtl_year_id
> smallint,age_for_rtl_qtr_id smallint,age_for_rtl_month_id
> smallint,age_for_rtl_week_id smallint,age_for_cs_week_id
> smallint,day_of_cal_id int,day_of_year_id smallint,day_of_qtr_id
> smallint,day_of_month_id smallint,day_of_week_id int,week_of_year_id
> tinyint,week_of_cal_id int,month_of_qtr_id tinyint,month_of_year_id
> tinyint,month_of_cal_id smallint,qtr_of_year_id tinyint,qtr_of_cal_id
> smallint,year_of_cal_id smallint,year_end_dt string,qtr_end_dt
> string,month_end_dt string,week_end_dt string,cal_dt_name string,cal_dt_desc
> string,cal_dt_short_name string,ytd_yn_id tinyint,qtd_yn_id tinyint,mtd_yn_id
> tinyint,wtd_yn_id tinyint,season_beg_dt string,day_in_year_count
> smallint,day_in_qtr_count tinyint,day_in_month_count
> tinyint,day_in_week_count tinyint,rtl_year_beg_dt string,rtl_qtr_beg_dt
> string,rtl_month_beg_dt string,rtl_week_beg_dt string,cs_week_beg_dt
> string,cal_date string,day_of_week string,month_id string,prd_desc
> string,prd_flag string,prd_id string,prd_ind string,qtr_desc string,qtr_id
> string,qtr_ind string,retail_week string,retail_year string,retail_start_date
> string,retail_wk_end_date string,week_ind string,week_num_desc
> string,week_beg_date string,week_end_date string,week_in_year_id
> string,week_id string,week_beg_end_desc_mdy string,week_beg_end_desc_md
> string,year_id string,year_ind string,cal_dt_mns_1year_dt
> string,cal_dt_mns_2year_dt string,cal_dt_mns_1qtr_dt
> string,cal_dt_mns_2qtr_dt string,cal_dt_mns_1month_dt
> string,cal_dt_mns_2month_dt string,cal_dt_mns_1week_dt
> string,cal_dt_mns_2week_dt string,curr_cal_dt_mns_1year_yn_id
> tinyint,curr_cal_dt_mns_2year_yn_id tinyint,curr_cal_dt_mns_1qtr_yn_id
> tinyint,curr_cal_dt_mns_2qtr_yn_id tinyint,curr_cal_dt_mns_1month_yn_id
> tinyint,curr_cal_dt_mns_2month_yn_id tinyint,curr_cal_dt_mns_1week_yn_ind
> tinyint,curr_cal_dt_mns_2week_yn_ind tinyint,rtl_month_of_rtl_year_id
> string,rtl_qtr_of_rtl_year_id tinyint,rtl_week_of_rtl_year_id
> tinyint,season_of_year_id tinyint,ytm_yn_id tinyint,ytq_yn_id
> tinyint,ytw_yn_id
