[GitHub] [kylin] nichunen commented on a change in pull request #664: KYLIN-4017 Build engine get zk(zookeeper) lock failed when building job, it causes the whole build engine doesn't work
nichunen commented on a change in pull request #664: KYLIN-4017 Build engine
get zk(zookeeper) lock failed when building job, it causes the whole build
engine doesn't work
URL: https://github.com/apache/kylin/pull/664#discussion_r290148917
##
File path: core-common/src/main/java/org/apache/kylin/common/util/ZKUtil.java
##
@@ -84,7 +84,7 @@ public void onRemoval(RemovalNotification notification
logger.error("Error at closing " + curator, ex);
}
}
-}).expireAfterWrite(1, TimeUnit.DAYS).build();
+}).expireAfterWrite(1, TimeUnit.DAYS).build();//never expired
Review comment:
Hi, what's the purpose of making the cache of curator client never expire?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (KYLIN-3845) Kylin build error If the Kafka data source lacks selected dimensions or metrics in the kylin stream build.
[
https://issues.apache.org/jira/browse/KYLIN-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855344#comment-16855344
]
ASF subversion and git services commented on KYLIN-3845:
Commit c3a71101286ba994dec85aa515236b28c98b9e94 in kylin's branch
refs/heads/master from zhaojintaozhao
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=c3a7110 ]
KYLIN-3845 solve the problem that kylin build error in stream build task. (#663)
* KYLIN-3845 solve the problem that kylin build error if Kafka data source
lacks selected dimensions or metrics in a kylin stream build task.
> Kylin build error If the Kafka data source lacks selected dimensions or
> metrics in the kylin stream build.
> --
>
> Key: KYLIN-3845
> URL: https://issues.apache.org/jira/browse/KYLIN-3845
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, NRT Streaming
>Affects Versions: v2.5.2
> Environment: Fusion Insight
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Major
> Labels: easyfix
> Fix For: v3.0.0
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Hi dear team:
> I'm developing OLAP Platform based on Kylin2.5.2. During my work, I build a
> streaming cube from Kafka source using kafka demo.
> In my streaming project, I set country、currency as dimensions and userId as
> metrics. But the cube build failed in 3rd step("Extract Fact Table Distinct
> Columns"). The exception is java.lang.ArrayIndexOutOfBoundsException.
> This is logs:
> 2019-03-02 14:21:01,492 INFO [main] org.apache.kylin.engine.mr.KylinReducer:
> Do cleanup, available memory: 1334m
> 2019-03-02 14:21:01,492 INFO [main] org.apache.kylin.engine.mr.KylinReducer:
> Total rows: 127
> 2019-03-02 14:21:01,492 INFO [main] org.apache.hadoop.mapred.MapTask:
> Finished spill 0
> 2019-03-02 14:21:01,492 INFO [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child: java.lang.ArrayIndexOutOfBoundsException:2
> 2019-03-02 14:21:01,492 INFO [main] org.apache.kylin.engine.mr.KylinReducer:
> Do cleanup, available memory: 1334m
> at
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(FactDistinctColumnsMapper.java:177)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(MapperTask.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:187)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java;180)
>
> Then I find that in Kafka datasource, some streaming data lack the userId
> column. Most of the streaming data(contry, currency,userId) is
> ("China","CNY","843c4d");but a small amount of data lack userId, some data is
> ("China","CNY"). so when run the 3rd step("Extract Fact Table Distinct
> Columns"),MR engine will throw exception if the streaming data lack userId.
> The I check the source of Kylin, FactDistinctColumnsMapper.java:
> public void doMap(KEYIN key, Object record, Context context) throws
> IOException, InterruptedException {
> Collection rowCollection =
> flatTableInputFormat.parseMapperInput(record);
> for (String[] row : rowCollection) {
> context.getCounter(RawDataCounter.BYTES).increment(countSizeInBytes(row));
> for (int i = 0; i < allCols.size(); i++) {
> String fieldValue = row[columnIndex[i]];
> if (fieldValue == null)
> continue;
> final DataType type = allCols.get(i).getType();
> ...
> I find that columnIndex[i] is equal with the size of row if the streaming
> data lack one column. So the row[columnIndex[i]] will throw the
> ArrayIndexOutOfBoundsException. So I change this code, check the
> columnIndex[i] and the size of row. If columnIndex[i] is equal with or larger
> than the size of row, I set fieldValue empty value. And After I change my
> code， the 3rd step("Extract Fact Table Distinct Columns") will run success.
> Those are what I found, which will cause problem for developers.
> How do you think?
> Best regard
> jintao
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] nichunen merged pull request #663: KYLIN-3845 solve the problem that kylin build error in stream build task.
nichunen merged pull request #663: KYLIN-3845 solve the problem that kylin build error in stream build task. URL: https://github.com/apache/kylin/pull/663 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3845) Kylin build error If the Kafka data source lacks selected dimensions or metrics in the kylin stream build.
[
https://issues.apache.org/jira/browse/KYLIN-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855343#comment-16855343
]
ASF subversion and git services commented on KYLIN-3845:
Commit c3a71101286ba994dec85aa515236b28c98b9e94 in kylin's branch
refs/heads/master from zhaojintaozhao
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=c3a7110 ]
KYLIN-3845 solve the problem that kylin build error in stream build task. (#663)
* KYLIN-3845 solve the problem that kylin build error if Kafka data source
lacks selected dimensions or metrics in a kylin stream build task.
> Kylin build error If the Kafka data source lacks selected dimensions or
> metrics in the kylin stream build.
> --
>
> Key: KYLIN-3845
> URL: https://issues.apache.org/jira/browse/KYLIN-3845
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, NRT Streaming
>Affects Versions: v2.5.2
> Environment: Fusion Insight
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Major
> Labels: easyfix
> Fix For: v3.0.0
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Hi dear team:
> I'm developing OLAP Platform based on Kylin2.5.2. During my work, I build a
> streaming cube from Kafka source using kafka demo.
> In my streaming project, I set country、currency as dimensions and userId as
> metrics. But the cube build failed in 3rd step("Extract Fact Table Distinct
> Columns"). The exception is java.lang.ArrayIndexOutOfBoundsException.
> This is logs:
> 2019-03-02 14:21:01,492 INFO [main] org.apache.kylin.engine.mr.KylinReducer:
> Do cleanup, available memory: 1334m
> 2019-03-02 14:21:01,492 INFO [main] org.apache.kylin.engine.mr.KylinReducer:
> Total rows: 127
> 2019-03-02 14:21:01,492 INFO [main] org.apache.hadoop.mapred.MapTask:
> Finished spill 0
> 2019-03-02 14:21:01,492 INFO [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child: java.lang.ArrayIndexOutOfBoundsException:2
> 2019-03-02 14:21:01,492 INFO [main] org.apache.kylin.engine.mr.KylinReducer:
> Do cleanup, available memory: 1334m
> at
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(FactDistinctColumnsMapper.java:177)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(MapperTask.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:187)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java;180)
>
> Then I find that in Kafka datasource, some streaming data lack the userId
> column. Most of the streaming data(contry, currency,userId) is
> ("China","CNY","843c4d");but a small amount of data lack userId, some data is
> ("China","CNY"). so when run the 3rd step("Extract Fact Table Distinct
> Columns"),MR engine will throw exception if the streaming data lack userId.
> The I check the source of Kylin, FactDistinctColumnsMapper.java:
> public void doMap(KEYIN key, Object record, Context context) throws
> IOException, InterruptedException {
> Collection rowCollection =
> flatTableInputFormat.parseMapperInput(record);
> for (String[] row : rowCollection) {
> context.getCounter(RawDataCounter.BYTES).increment(countSizeInBytes(row));
> for (int i = 0; i < allCols.size(); i++) {
> String fieldValue = row[columnIndex[i]];
> if (fieldValue == null)
> continue;
> final DataType type = allCols.get(i).getType();
> ...
> I find that columnIndex[i] is equal with the size of row if the streaming
> data lack one column. So the row[columnIndex[i]] will throw the
> ArrayIndexOutOfBoundsException. So I change this code, check the
> columnIndex[i] and the size of row. If columnIndex[i] is equal with or larger
> than the size of row, I set fieldValue empty value. And After I change my
> code， the 3rd step("Extract Fact Table Distinct Columns") will run success.
> Those are what I found, which will cause problem for developers.
> How do you think?
> Best regard
> jintao
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3845) Kylin build error If the Kafka data source lacks selected dimensions or metrics in the kylin stream build.
[
https://issues.apache.org/jira/browse/KYLIN-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855342#comment-16855342
]
ASF GitHub Bot commented on KYLIN-3845:
---
nichunen commented on pull request #663: KYLIN-3845 solve the problem that
kylin build error in stream build task.
URL: https://github.com/apache/kylin/pull/663
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Kylin build error If the Kafka data source lacks selected dimensions or
> metrics in the kylin stream build.
> --
>
> Key: KYLIN-3845
> URL: https://issues.apache.org/jira/browse/KYLIN-3845
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, NRT Streaming
>Affects Versions: v2.5.2
> Environment: Fusion Insight
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Major
> Labels: easyfix
> Fix For: v3.0.0
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Hi dear team:
> I'm developing OLAP Platform based on Kylin2.5.2. During my work, I build a
> streaming cube from Kafka source using kafka demo.
> In my streaming project, I set country、currency as dimensions and userId as
> metrics. But the cube build failed in 3rd step("Extract Fact Table Distinct
> Columns"). The exception is java.lang.ArrayIndexOutOfBoundsException.
> This is logs:
> 2019-03-02 14:21:01,492 INFO [main] org.apache.kylin.engine.mr.KylinReducer:
> Do cleanup, available memory: 1334m
> 2019-03-02 14:21:01,492 INFO [main] org.apache.kylin.engine.mr.KylinReducer:
> Total rows: 127
> 2019-03-02 14:21:01,492 INFO [main] org.apache.hadoop.mapred.MapTask:
> Finished spill 0
> 2019-03-02 14:21:01,492 INFO [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child: java.lang.ArrayIndexOutOfBoundsException:2
> 2019-03-02 14:21:01,492 INFO [main] org.apache.kylin.engine.mr.KylinReducer:
> Do cleanup, available memory: 1334m
> at
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(FactDistinctColumnsMapper.java:177)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(MapperTask.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:187)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java;180)
>
> Then I find that in Kafka datasource, some streaming data lack the userId
> column. Most of the streaming data(contry, currency,userId) is
> ("China","CNY","843c4d");but a small amount of data lack userId, some data is
> ("China","CNY"). so when run the 3rd step("Extract Fact Table Distinct
> Columns"),MR engine will throw exception if the streaming data lack userId.
> The I check the source of Kylin, FactDistinctColumnsMapper.java:
> public void doMap(KEYIN key, Object record, Context context) throws
> IOException, InterruptedException {
> Collection rowCollection =
> flatTableInputFormat.parseMapperInput(record);
> for (String[] row : rowCollection) {
> context.getCounter(RawDataCounter.BYTES).increment(countSizeInBytes(row));
> for (int i = 0; i < allCols.size(); i++) {
> String fieldValue = row[columnIndex[i]];
> if (fieldValue == null)
> continue;
> final DataType type = allCols.get(i).getType();
> ...
> I find that columnIndex[i] is equal with the size of row if the streaming
> data lack one column. So the row[columnIndex[i]] will throw the
> ArrayIndexOutOfBoundsException. So I change this code, check the
> columnIndex[i] and the size of row. If columnIndex[i] is equal with or larger
> than the size of row, I set fieldValue empty value. And After I change my
> code， the 3rd step("Extract Fact Table Distinct Columns") will run success.
> Those are what I found, which will cause problem for developers.
> How do you think?
> Best regard
> jintao
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] coveralls edited a comment on issue #669: KYLIN-3997: Add a health check job of Kylin
coveralls edited a comment on issue #669: KYLIN-3997: Add a health check job of Kylin URL: https://github.com/apache/kylin/pull/669#issuecomment-497677938 ## Pull Request Test Coverage Report for [Build 4575](https://coveralls.io/builds/23773589) * **5** of **180** **(2.78%)** changed or added relevant lines in **8** files are covered. * **304** unchanged lines in **7** files lost coverage. * Overall coverage decreased (**-0.04%**) to **28.24%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-metadata/src/main/java/org/apache/kylin/metadata/cachesync/CachedCrudAssist.java](https://coveralls.io/builds/23773589/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fcachesync%2FCachedCrudAssist.java#L147) | 3 | 5 | 60.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/model/DataModelManager.java](https://coveralls.io/builds/23773589/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FDataModelManager.java#L109) | 0 | 2 | 0.0% | [core-common/src/main/java/org/apache/kylin/common/util/BufferedLogger.java](https://coveralls.io/builds/23773589/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FBufferedLogger.java#L49) | 0 | 3 | 0.0% | [core-common/src/main/java/org/apache/kylin/common/util/SoutLogger.java](https://coveralls.io/builds/23773589/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FSoutLogger.java#L34) | 0 | 3 | 0.0% | [tool/src/main/java/org/apache/kylin/tool/KylinHealthCheckJob.java](https://coveralls.io/builds/23773589/source?filename=tool%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Ftool%2FKylinHealthCheckJob.java#L23) | 0 | 4 | 0.0% | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/23773589/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L2197) | 0 | 5 | 0.0% | [server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java](https://coveralls.io/builds/23773589/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fjob%2FKylinHealthCheckJob.java#L57) | 0 | 156 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-metadata/src/main/java/org/apache/kylin/metadata/cachesync/CachedCrudAssist.java](https://coveralls.io/builds/23773589/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fcachesync%2FCachedCrudAssist.java#L148) | 1 | 64.52% | | [stream-core/src/main/java/org/apache/kylin/stream/core/query/StreamingQueryProfile.java](https://coveralls.io/builds/23773589/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fquery%2FStreamingQueryProfile.java#L192) | 1 | 37.08% | | [stream-core/src/main/java/org/apache/kylin/stream/core/storage/columnar/RawRecord.java](https://coveralls.io/builds/23773589/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2Fcolumnar%2FRawRecord.java#L43) | 4 | 62.5% | | [engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkExecutable.java](https://coveralls.io/builds/23773589/source?filename=engine-spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fspark%2FSparkExecutable.java#L449) | 6 | 0.0% | | [jdbc/src/main/java/org/apache/kylin/jdbc/KylinMeta.java](https://coveralls.io/builds/23773589/source?filename=jdbc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjdbc%2FKylinMeta.java#L89) | 36 | 56.92% | | [stream-receiver/src/main/java/org/apache/kylin/stream/server/rest/util/QueryUtil.java](https://coveralls.io/builds/23773589/source?filename=stream-receiver%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fserver%2Frest%2Futil%2FQueryUtil.java#L38) | 36 | 0.0% | | [core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java](https://coveralls.io/builds/23773589/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2FCubeManager.java#L204) | 220 | 42.6% | | Totals | [](https://coveralls.io/builds/23773589) | | :-- | --: | | Change from base [Build 4562](https://coveralls.io/builds/23709655): | -0.04% | | Covered Lines: | 23262 | | Relevant Lines: | 82373 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (KYLIN-4009) Build cube failed when kylin.metadata.url=kylin_metadata@hbase
[ https://issues.apache.org/jira/browse/KYLIN-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu closed KYLIN-4009. --- Resolution: Cannot Reproduce Looks like user's problem is caused by other reason. > Build cube failed when kylin.metadata.url=kylin_metadata@hbase > -- > > Key: KYLIN-4009 > URL: https://issues.apache.org/jira/browse/KYLIN-4009 > Project: Kylin > Issue Type: Bug > Components: Others >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Assignee: XiaoXiang Yu >Priority: Major > Attachments: err.bmp > > > Build cube failed when kylin.metadata.url=kylin_metadata@hbase. > If kylin.metadata.url=kylin_metadata@hbase, kylin use PushdownResourceStore > to write file to hdfs，and the dict files will be writeen twice， and the > second time it will be writeen to an empty file。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] lshmouse commented on issue #669: KYLIN-3997: Add a health check job of Kylin
lshmouse commented on issue #669: KYLIN-3997: Add a health check job of Kylin URL: https://github.com/apache/kylin/pull/669#issuecomment-498492582 Thanks for @hit-lacus 's review. I will update the pr asap. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3983) Add extra metadata for measure
[ https://issues.apache.org/jira/browse/KYLIN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855233#comment-16855233 ] ASF GitHub Bot commented on KYLIN-3983: --- nichunen commented on pull request #662: KYLIN-3983 Add extra metadata for measure. Fix the dead lock cause by… URL: https://github.com/apache/kylin/pull/662 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add extra metadata for measure > -- > > Key: KYLIN-3983 > URL: https://issues.apache.org/jira/browse/KYLIN-3983 > Project: Kylin > Issue Type: Sub-task > Components: Metadata >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Minor > > Just like CubeDesc and CubeInstance, we need to add extra metadata for > measure to persist some runtime data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3983) Add extra metadata for measure
[ https://issues.apache.org/jira/browse/KYLIN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855232#comment-16855232 ] ASF subversion and git services commented on KYLIN-3983: Commit fbc5c0d04aa5341e81a02802c652decd71800e36 in kylin's branch refs/heads/dynamic-measure from yuzhang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=fbc5c0d ] KYLIN-3983 Add extra metadata for measure. Fix the dead lock cause by lock order. No need to use different lock in CubeManager and MeasureManager. > Add extra metadata for measure > -- > > Key: KYLIN-3983 > URL: https://issues.apache.org/jira/browse/KYLIN-3983 > Project: Kylin > Issue Type: Sub-task > Components: Metadata >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Minor > > Just like CubeDesc and CubeInstance, we need to add extra metadata for > measure to persist some runtime data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] nichunen merged pull request #662: KYLIN-3983 Add extra metadata for measure. Fix the dead lock cause by…
nichunen merged pull request #662: KYLIN-3983 Add extra metadata for measure. Fix the dead lock cause by… URL: https://github.com/apache/kylin/pull/662 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3983) Add extra metadata for measure
[ https://issues.apache.org/jira/browse/KYLIN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855234#comment-16855234 ] ASF subversion and git services commented on KYLIN-3983: Commit 908e202cc1177c1e479425c7e192572205157abf in kylin's branch refs/heads/dynamic-measure from yuzhang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=908e202 ] KYLIN-3983 cut interface, remove "project" from parameters > Add extra metadata for measure > -- > > Key: KYLIN-3983 > URL: https://issues.apache.org/jira/browse/KYLIN-3983 > Project: Kylin > Issue Type: Sub-task > Components: Metadata >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Minor > > Just like CubeDesc and CubeInstance, we need to add extra metadata for > measure to persist some runtime data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-4024) when ad-hoc Push Down by presto engine Unrecognized column type: INTEGER,TIME,VARBINARY
[ https://issues.apache.org/jira/browse/KYLIN-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu reassigned KYLIN-4024: --- Assignee: XiaoXiang Yu > when ad-hoc Push Down by presto engine Unrecognized column type: > INTEGER,TIME,VARBINARY > > > Key: KYLIN-4024 > URL: https://issues.apache.org/jira/browse/KYLIN-4024 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: all, v3.0.0, v2.6.2 >Reporter: wangxiaojing >Assignee: XiaoXiang Yu >Priority: Major > Fix For: Future > > > Hello , > When do ad-hoc Push Down query by presto engine ,it throws Unrecognized > column type,like INTEGER,TIME,VARBINARY 。 > The field of the table that appears in the hive is defined as an int > type. If the Presto query is used, the INTEGER type will be returned. > At this time, kylin ad-hoc push down will not recognize this type. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4031) RestClient will throw exception with message contains clear-text password
[ https://issues.apache.org/jira/browse/KYLIN-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854688#comment-16854688 ] nichunen commented on KYLIN-4031: - +1 Would you please submit a pr? > RestClient will throw exception with message contains clear-text password > - > > Key: KYLIN-4031 > URL: https://issues.apache.org/jira/browse/KYLIN-4031 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Priority: Major > > Hi dear kylin team: > I found that RestClient:97 will throw IllegalArgumentException with > message contains clear-text password when set error uri with user:pwd. This > may casue some security problem, I think. > How do you think about this? > > Best Regards > >yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4009) Build cube failed when kylin.metadata.url=kylin_metadata@hbase
[ https://issues.apache.org/jira/browse/KYLIN-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854626#comment-16854626 ] XiaoXiang Yu commented on KYLIN-4009: - Hi, as the comment (Writing marker for big resource) say, that empty file is a marker, if this empty file is not exists, getResource method will throw a NPE. > Build cube failed when kylin.metadata.url=kylin_metadata@hbase > -- > > Key: KYLIN-4009 > URL: https://issues.apache.org/jira/browse/KYLIN-4009 > Project: Kylin > Issue Type: Bug > Components: Others >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Assignee: XiaoXiang Yu >Priority: Major > Attachments: err.bmp > > > Build cube failed when kylin.metadata.url=kylin_metadata@hbase. > If kylin.metadata.url=kylin_metadata@hbase, kylin use PushdownResourceStore > to write file to hdfs，and the dict files will be writeen twice， and the > second time it will be writeen to an empty file。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-4009) Build cube failed when kylin.metadata.url=kylin_metadata@hbase
[ https://issues.apache.org/jira/browse/KYLIN-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu reassigned KYLIN-4009: --- Assignee: XiaoXiang Yu > Build cube failed when kylin.metadata.url=kylin_metadata@hbase > -- > > Key: KYLIN-4009 > URL: https://issues.apache.org/jira/browse/KYLIN-4009 > Project: Kylin > Issue Type: Bug > Components: Others >Affects Versions: v3.0.0-alpha >Reporter: zengrui >Assignee: XiaoXiang Yu >Priority: Major > Attachments: err.bmp > > > Build cube failed when kylin.metadata.url=kylin_metadata@hbase. > If kylin.metadata.url=kylin_metadata@hbase, kylin use PushdownResourceStore > to write file to hdfs，and the dict files will be writeen twice， and the > second time it will be writeen to an empty file。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4031) RestClient will throw exception with message contains clear-text password
Yuzhang QIU created KYLIN-4031: -- Summary: RestClient will throw exception with message contains clear-text password Key: KYLIN-4031 URL: https://issues.apache.org/jira/browse/KYLIN-4031 Project: Kylin Issue Type: Improvement Components: REST Service Affects Versions: v2.5.2 Reporter: Yuzhang QIU Hi dear kylin team: I found that RestClient:97 will throw IllegalArgumentException with message contains clear-text password when set error uri with user:pwd. This may casue some security problem, I think. How do you think about this? Best Regards yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] codecov-io commented on issue #673: #KYLIN-4030, ResourceStore deleteResource by comparing timestamp may be failed
codecov-io commented on issue #673: #KYLIN-4030, ResourceStore deleteResource by comparing timestamp may be failed URL: https://github.com/apache/kylin/pull/673#issuecomment-498247059 # [Codecov](https://codecov.io/gh/apache/kylin/pull/673?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@354f6de`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `50%`. [](https://codecov.io/gh/apache/kylin/pull/673?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master #673 +/- ## = Coverage ? 25.79% Complexity? 6009 = Files ? 1382 Lines ?82204 Branches ?11518 = Hits ?21203 Misses?58953 Partials ? 2048 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/673?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...apache/kylin/storage/hbase/HBaseResourceStore.java](https://codecov.io/gh/apache/kylin/pull/673/diff?src=pr&el=tree#diff-c3RvcmFnZS1oYmFzZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc3RvcmFnZS9oYmFzZS9IQmFzZVJlc291cmNlU3RvcmUuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...he/kylin/common/persistence/JDBCResourceStore.java](https://codecov.io/gh/apache/kylin/pull/673/diff?src=pr&el=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9wZXJzaXN0ZW5jZS9KREJDUmVzb3VyY2VTdG9yZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...he/kylin/common/persistence/HDFSResourceStore.java](https://codecov.io/gh/apache/kylin/pull/673/diff?src=pr&el=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9wZXJzaXN0ZW5jZS9IREZTUmVzb3VyY2VTdG9yZS5qYXZh) | `18.51% <0%> (ø)` | `5 <0> (?)` | | | [...he/kylin/common/persistence/FileResourceStore.java](https://codecov.io/gh/apache/kylin/pull/673/diff?src=pr&el=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9wZXJzaXN0ZW5jZS9GaWxlUmVzb3VyY2VTdG9yZS5qYXZh) | `62.62% <100%> (ø)` | `28 <0> (?)` | | | [...apache/kylin/common/persistence/ResourceStore.java](https://codecov.io/gh/apache/kylin/pull/673/diff?src=pr&el=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9wZXJzaXN0ZW5jZS9SZXNvdXJjZVN0b3JlLmphdmE=) | `73.05% <83.33%> (ø)` | `56 <2> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/673?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/673?src=pr&el=footer). Last update [354f6de...2dec22e](https://codecov.io/gh/apache/kylin/pull/673?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-4030) ResourceStore deleteResource by comparing timestamp may be failed to delete caused by time precision loose
[ https://issues.apache.org/jira/browse/KYLIN-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PENG Zhengshuai updated KYLIN-4030: --- Description: In ResourceStore, the interface *deleteResourceImpl(String resPath, long timestamp)* maybe failed to delete the resource because the resource timestamp may lose the precision. For example, if we use *$KYLIN_HOME/bin/metastore.sh* to backup from metastore to Local OS filesystem, and restore from local filesystem to metastore. The resource timestamp will lose precision in Millis Second. An example here: original timestamp: 1559564381 after precision lost: 1559564000 The fix design is to tolerate [0-999] ms difference when comparing the timestamp was: In ResourceStore, the interface *deleteResourceImpl(String resPath, long timestamp)* maybe failed to delete the resource because the resource timestamp may lose the precision. For example, if we use *$KYLIN_HOME/bin/metastore.sh* to backup from metastore to Local OS filesystem, and restore from local filesystem to metastore. The resource timestamp will lose precision in Millis Second. An example here: original timestamp: 1559564381 after precision lost: 1559564000 > ResourceStore deleteResource by comparing timestamp may be failed to delete > caused by time precision loose > -- > > Key: KYLIN-4030 > URL: https://issues.apache.org/jira/browse/KYLIN-4030 > Project: Kylin > Issue Type: Bug >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Minor > > In ResourceStore, the interface *deleteResourceImpl(String resPath, long > timestamp)* maybe failed to delete the resource because the resource > timestamp may lose the precision. > For example, if we use *$KYLIN_HOME/bin/metastore.sh* to backup from > metastore to Local OS filesystem, and restore from local filesystem to > metastore. The resource timestamp will lose precision in Millis Second. > An example here: > original timestamp: 1559564381 > after precision lost: 1559564000 > The fix design is to tolerate [0-999] ms difference when comparing the > timestamp -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] hit-lacus commented on issue #669: KYLIN-3997: Add a health check job of Kylin
hit-lacus commented on issue #669: KYLIN-3997: Add a health check job of Kylin URL: https://github.com/apache/kylin/pull/669#issuecomment-498242577 I think it is a good feature, by this feature, Kylin's administrator needn't check metadata error in kylin.log or find useless cube manually. Some value such as *ExpansionRate* or *Out of date* should be configurable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] hit-lacus commented on a change in pull request #669: KYLIN-3997: Add a health check job of Kylin
hit-lacus commented on a change in pull request #669: KYLIN-3997: Add a health
check job of Kylin
URL: https://github.com/apache/kylin/pull/669#discussion_r289823388
##
File path: core-cube/src/main/java/org/apache/kylin/cube/CubeDescManager.java
##
@@ -164,6 +164,10 @@ public CubeDesc reloadCubeDescQuietly(String name) {
}
}
+public List getErrorCubes() {
Review comment:
And it is never called anywhere.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] hit-lacus commented on a change in pull request #669: KYLIN-3997: Add a health check job of Kylin
hit-lacus commented on a change in pull request #669: KYLIN-3997: Add a health
check job of Kylin
URL: https://github.com/apache/kylin/pull/669#discussion_r289821911
##
File path: core-cube/src/main/java/org/apache/kylin/cube/CubeDescManager.java
##
@@ -164,6 +164,10 @@ public CubeDesc reloadCubeDescQuietly(String name) {
}
}
+public List getErrorCubes() {
Review comment:
This method is duplicated with `getErrorCubes` in `CubeManager.java`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (KYLIN-4030) ResourceStore deleteResource by comparing timestamp may be failed to delete caused by time precision loose
[ https://issues.apache.org/jira/browse/KYLIN-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PENG Zhengshuai updated KYLIN-4030: --- Summary: ResourceStore deleteResource by comparing timestamp may be failed to delete caused by time precision loose (was: ResourceStore deleteResource by comparing timestamp may failed to delete caused by time precision loose) > ResourceStore deleteResource by comparing timestamp may be failed to delete > caused by time precision loose > -- > > Key: KYLIN-4030 > URL: https://issues.apache.org/jira/browse/KYLIN-4030 > Project: Kylin > Issue Type: Bug >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Minor > > In ResourceStore, the interface *deleteResourceImpl(String resPath, long > timestamp)* maybe failed to delete the resource because the resource > timestamp may lose the precision. > For example, if we use *$KYLIN_HOME/bin/metastore.sh* to backup from > metastore to Local OS filesystem, and restore from local filesystem to > metastore. The resource timestamp will lose precision in Millis Second. > An example here: > original timestamp: 1559564381 > after precision lost: 1559564000 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4030) ResourceStore deleteResource by comparing timestamp may failed to delete caused by time precision loose
[ https://issues.apache.org/jira/browse/KYLIN-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PENG Zhengshuai updated KYLIN-4030: --- Summary: ResourceStore deleteResource by comparing timestamp may failed to delete caused by time precision loose (was: Resource deleteResource by comparing timestamp may failed to delete caused by time precision loose) > ResourceStore deleteResource by comparing timestamp may failed to delete > caused by time precision loose > --- > > Key: KYLIN-4030 > URL: https://issues.apache.org/jira/browse/KYLIN-4030 > Project: Kylin > Issue Type: Bug >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Minor > > In ResourceStore, the interface *deleteResourceImpl(String resPath, long > timestamp)* maybe failed to delete the resource because the resource > timestamp may lose the precision. > For example, if we use *$KYLIN_HOME/bin/metastore.sh* to backup from > metastore to Local OS filesystem, and restore from local filesystem to > metastore. The resource timestamp will lose precision in Millis Second. > An example here: > original timestamp: 1559564381 > after precision lost: 1559564000 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] asfgit commented on issue #673: #KYLIN-4030, Resource deleteResource by comparing timestamp may faile…
asfgit commented on issue #673: #KYLIN-4030, Resource deleteResource by comparing timestamp may faile… URL: https://github.com/apache/kylin/pull/673#issuecomment-498235378 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4030) Resource deleteResource by comparing timestamp may failed to delete caused by time precision loose
[ https://issues.apache.org/jira/browse/KYLIN-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854535#comment-16854535 ] ASF GitHub Bot commented on KYLIN-4030: --- ZhengshuaiPENG commented on pull request #673: #KYLIN-4030, Resource deleteResource by comparing timestamp may faile… URL: https://github.com/apache/kylin/pull/673 …d to delete caused by time precision loose This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Resource deleteResource by comparing timestamp may failed to delete caused by > time precision loose > -- > > Key: KYLIN-4030 > URL: https://issues.apache.org/jira/browse/KYLIN-4030 > Project: Kylin > Issue Type: Bug >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Minor > > In ResourceStore, the interface *deleteResourceImpl(String resPath, long > timestamp)* maybe failed to delete the resource because the resource > timestamp may lose the precision. > For example, if we use *$KYLIN_HOME/bin/metastore.sh* to backup from > metastore to Local OS filesystem, and restore from local filesystem to > metastore. The resource timestamp will lose precision in Millis Second. > An example here: > original timestamp: 1559564381 > after precision lost: 1559564000 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] ZhengshuaiPENG opened a new pull request #673: #KYLIN-4030, Resource deleteResource by comparing timestamp may faile…
ZhengshuaiPENG opened a new pull request #673: #KYLIN-4030, Resource deleteResource by comparing timestamp may faile… URL: https://github.com/apache/kylin/pull/673 …d to delete caused by time precision loose This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-4030) Resource deleteResource by comparing timestamp may failed to delete caused by time precision loose
[ https://issues.apache.org/jira/browse/KYLIN-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PENG Zhengshuai updated KYLIN-4030: --- Description: In ResourceStore, the interface *deleteResourceImpl(String resPath, long timestamp)* maybe failed to delete the resource because the resource timestamp may lose the precision. For example, if we use *$KYLIN_HOME/bin/metastore.sh* to backup from metastore to Local OS filesystem, and restore from local filesystem to metastore. The resource timestamp will lose precision in Millis Second. An example here: original timestamp: 1559564381 after precision lost: 1559564000 was: In ResourceStore, the interface `deleteResourceImpl(String resPath, long timestamp)` maybe failed to delete the resource because the resource timestamp may lose the precision. For example, if we use $KYLIN_HOME/bin/metastore.sh to backup from metastore to Local OS filesystem, and restore from local filesystem to metastore. The resource timestamp will lose precision in Millis Second. An example here: original timestamp: 1559564381 after precision lost: 1559564000 > Resource deleteResource by comparing timestamp may failed to delete caused by > time precision loose > -- > > Key: KYLIN-4030 > URL: https://issues.apache.org/jira/browse/KYLIN-4030 > Project: Kylin > Issue Type: Bug >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Minor > > In ResourceStore, the interface *deleteResourceImpl(String resPath, long > timestamp)* maybe failed to delete the resource because the resource > timestamp may lose the precision. > For example, if we use *$KYLIN_HOME/bin/metastore.sh* to backup from > metastore to Local OS filesystem, and restore from local filesystem to > metastore. The resource timestamp will lose precision in Millis Second. > An example here: > original timestamp: 1559564381 > after precision lost: 1559564000 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4030) Resource deleteResource by comparing timestamp may failed to delete caused by time precision loose
PENG Zhengshuai created KYLIN-4030: -- Summary: Resource deleteResource by comparing timestamp may failed to delete caused by time precision loose Key: KYLIN-4030 URL: https://issues.apache.org/jira/browse/KYLIN-4030 Project: Kylin Issue Type: Bug Reporter: PENG Zhengshuai Assignee: PENG Zhengshuai In ResourceStore, the interface `deleteResourceImpl(String resPath, long timestamp)` maybe failed to delete the resource because the resource timestamp may lose the precision. For example, if we use $KYLIN_HOME/bin/metastore.sh to backup from metastore to Local OS filesystem, and restore from local filesystem to metastore. The resource timestamp will lose precision in Millis Second. An example here: original timestamp: 1559564381 after precision lost: 1559564000 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] nichunen merged pull request #670: KYLIN-4025: Add detail exception in kylin http response
nichunen merged pull request #670: KYLIN-4025: Add detail exception in kylin http response URL: https://github.com/apache/kylin/pull/670 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4025) Add detail exception in kylin http response
[ https://issues.apache.org/jira/browse/KYLIN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854445#comment-16854445 ] ASF GitHub Bot commented on KYLIN-4025: --- nichunen commented on pull request #670: KYLIN-4025: Add detail exception in kylin http response URL: https://github.com/apache/kylin/pull/670 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add detail exception in kylin http response > > > Key: KYLIN-4025 > URL: https://issues.apache.org/jira/browse/KYLIN-4025 > Project: Kylin > Issue Type: New Feature > Components: REST Service >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > > Currently there is no detailed in http response when the requests go wrong, > because InternalErrorException in controller wrap the execption and remove > the exception stack. > It's better add the detail exception in kylin http response > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4025) Add detail exception in kylin http response
[ https://issues.apache.org/jira/browse/KYLIN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1685#comment-1685 ] ASF subversion and git services commented on KYLIN-4025: Commit 354f6ded8506fa53b4a78a8614da1e1c2bf1fef2 in kylin's branch refs/heads/master from Liu Shaohui [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=354f6de ] KYLIN-4025: Add detail exception in kylin http response > Add detail exception in kylin http response > > > Key: KYLIN-4025 > URL: https://issues.apache.org/jira/browse/KYLIN-4025 > Project: Kylin > Issue Type: New Feature > Components: REST Service >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > > Currently there is no detailed in http response when the requests go wrong, > because InternalErrorException in controller wrap the execption and remove > the exception stack. > It's better add the detail exception in kylin http response > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] hit-lacus commented on a change in pull request #669: KYLIN-3997: Add a health check job of Kylin
hit-lacus commented on a change in pull request #669: KYLIN-3997: Add a health
check job of Kylin
URL: https://github.com/apache/kylin/pull/669#discussion_r289769697
##
File path:
server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java
##
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.rest.job;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.OptionBuilder;
+import org.apache.commons.cli.Options;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.kylin.common.util.AbstractApplication;
+import org.apache.kylin.common.util.BufferedLogger;
+import org.apache.kylin.common.util.HadoopUtil;
+import org.apache.kylin.common.util.MailService;
+import org.apache.kylin.common.util.OptionsHelper;
+import org.apache.kylin.cube.CubeInstance;
+import org.apache.kylin.cube.CubeManager;
+import org.apache.kylin.cube.CubeSegment;
+import org.apache.kylin.cube.model.CubeDesc;
+import org.apache.kylin.engine.mr.CubingJob;
+import org.apache.kylin.engine.mr.JobBuilderSupport;
+import org.apache.kylin.job.dao.ExecutableDao;
+import org.apache.kylin.job.dao.ExecutablePO;
+import org.apache.kylin.job.execution.CheckpointExecutable;
+import org.apache.kylin.job.execution.ExecutableState;
+import org.apache.kylin.metadata.model.DataModelManager;
+import org.apache.kylin.metadata.model.SegmentStatusEnum;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.List;
+import java.util.Locale;
+
+public class KylinHealthCheckJob extends AbstractApplication {
+ private static final Logger logger =
LoggerFactory.getLogger(KylinHealthCheckJob.class);
+
+ @SuppressWarnings("static-access")
+ private static final Option OPTION_FIX =
OptionBuilder.withArgName("fix").hasArg().isRequired(false)
+ .withDescription("Fix the unhealth cube").create("fix");
+
+ public static void main(String[] args) throws Exception {
+new KylinHealthCheckJob().execute(args);
+ }
+ final KylinConfig config;
+ final BufferedLogger reporter = new BufferedLogger(logger);
+ final CubeManager cubeManager;
+
+ public KylinHealthCheckJob() {
+this(KylinConfig.getInstanceFromEnv());
+ }
+
+ public KylinHealthCheckJob(KylinConfig config) {
+this.config = config;
+this.cubeManager = CubeManager.getInstance(config);
+ }
+
+ @Override
+ protected Options getOptions() {
+Options options = new Options();
+// TODO: Support to fix the unhealthy cube automatically
+options.addOption(OPTION_FIX);
+return options;
+ }
+
+ @Override
+ protected void execute(OptionsHelper optionsHelper) throws Exception {
+logger.info("options: '" + optionsHelper.getOptionsAsString() + "'");
+checkCubeHealth();
+ }
+
+ private void checkCubeHealth() throws Exception {
+CubeManager cubeManager = CubeManager.getInstance(config);
+
+List cubes = cubeManager.listAllCubes();
+// TODO:
+
+checkErrorMeta();
+
+// Check if the cubeid data exist for later cube merge
+checkSegmentHDFSPath(cubes);
+
+// Check if the hbase table exits or online
+checkHBaseTables(cubes);
+
+// Check if there are holes in cube
+// TODO: check if there are overlaps in segments of cube
+checkCubeHoles(cubes);
+
+// Check if there are too many segments
+checkTooManySegments(cubes);
+
+// Check if there are stale metadata
+checkStaleSegments(cubes);
+
+// Disable/Delete the out-of-date cube
+checkOutOfDateCube(cubes);
+
+// Check data expand rate
+checkDataExpansionRate(cubes);
+
+// Check auto merge param
+checkCubeDescParams(cubes);
+
+// ERROR history stopped build job
+checkStoppedJob();
+
+sendMail(reporter.getBufferedLog());
+ }
+
+ private void sendMail(String content) {
+logger.info("Send Kylin cluster report");
+String subject = "Kylin Cluster H
[GitHub] [kylin] hit-lacus commented on a change in pull request #669: KYLIN-3997: Add a health check job of Kylin
hit-lacus commented on a change in pull request #669: KYLIN-3997: Add a health
check job of Kylin
URL: https://github.com/apache/kylin/pull/669#discussion_r289769411
##
File path:
server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java
##
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.rest.job;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.OptionBuilder;
+import org.apache.commons.cli.Options;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.client.HBaseAdmin;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.kylin.common.util.AbstractApplication;
+import org.apache.kylin.common.util.BufferedLogger;
+import org.apache.kylin.common.util.HadoopUtil;
+import org.apache.kylin.common.util.MailService;
+import org.apache.kylin.common.util.OptionsHelper;
+import org.apache.kylin.cube.CubeInstance;
+import org.apache.kylin.cube.CubeManager;
+import org.apache.kylin.cube.CubeSegment;
+import org.apache.kylin.cube.model.CubeDesc;
+import org.apache.kylin.engine.mr.CubingJob;
+import org.apache.kylin.engine.mr.JobBuilderSupport;
+import org.apache.kylin.job.dao.ExecutableDao;
+import org.apache.kylin.job.dao.ExecutablePO;
+import org.apache.kylin.job.execution.CheckpointExecutable;
+import org.apache.kylin.job.execution.ExecutableState;
+import org.apache.kylin.metadata.model.DataModelManager;
+import org.apache.kylin.metadata.model.SegmentStatusEnum;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.List;
+import java.util.Locale;
+
+public class KylinHealthCheckJob extends AbstractApplication {
+ private static final Logger logger =
LoggerFactory.getLogger(KylinHealthCheckJob.class);
+
+ @SuppressWarnings("static-access")
+ private static final Option OPTION_FIX =
OptionBuilder.withArgName("fix").hasArg().isRequired(false)
+ .withDescription("Fix the unhealth cube").create("fix");
+
+ public static void main(String[] args) throws Exception {
+new KylinHealthCheckJob().execute(args);
+ }
+ final KylinConfig config;
+ final BufferedLogger reporter = new BufferedLogger(logger);
+ final CubeManager cubeManager;
+
+ public KylinHealthCheckJob() {
+this(KylinConfig.getInstanceFromEnv());
+ }
+
+ public KylinHealthCheckJob(KylinConfig config) {
+this.config = config;
+this.cubeManager = CubeManager.getInstance(config);
+ }
+
+ @Override
+ protected Options getOptions() {
+Options options = new Options();
+// TODO: Support to fix the unhealthy cube automatically
+options.addOption(OPTION_FIX);
+return options;
+ }
+
+ @Override
+ protected void execute(OptionsHelper optionsHelper) throws Exception {
+logger.info("options: '" + optionsHelper.getOptionsAsString() + "'");
+checkCubeHealth();
+ }
+
+ private void checkCubeHealth() throws Exception {
+CubeManager cubeManager = CubeManager.getInstance(config);
+
+List cubes = cubeManager.listAllCubes();
+// TODO:
+
+checkErrorMeta();
+
+// Check if the cubeid data exist for later cube merge
+checkSegmentHDFSPath(cubes);
+
+// Check if the hbase table exits or online
+checkHBaseTables(cubes);
+
+// Check if there are holes in cube
+// TODO: check if there are overlaps in segments of cube
+checkCubeHoles(cubes);
+
+// Check if there are too many segments
+checkTooManySegments(cubes);
+
+// Check if there are stale metadata
+checkStaleSegments(cubes);
+
+// Disable/Delete the out-of-date cube
+checkOutOfDateCube(cubes);
+
+// Check data expand rate
+checkDataExpansionRate(cubes);
+
+// Check auto merge param
+checkCubeDescParams(cubes);
+
+// ERROR history stopped build job
+checkStoppedJob();
+
+sendMail(reporter.getBufferedLog());
+ }
+
+ private void sendMail(String content) {
+logger.info("Send Kylin cluster report");
+String subject = "Kylin Cluster H
[GitHub] [kylin] hit-lacus commented on a change in pull request #669: KYLIN-3997: Add a health check job of Kylin
hit-lacus commented on a change in pull request #669: KYLIN-3997: Add a health check job of Kylin URL: https://github.com/apache/kylin/pull/669#discussion_r289760620 ## File path: server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java ## @@ -0,0 +1,298 @@ +/* Review comment: Hi, @lshmouse , I found this java file is not formatted as expected. It looks like it is indented with two blank and it is different with other Java file, maybe you should check **Setup IDE code formatter** in http://kylin.apache.org/development/dev_env.html. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] nichunen commented on issue #657: adjust HBase tableName toUpperCase for HFile
nichunen commented on issue #657: adjust HBase tableName toUpperCase for HFile URL: https://github.com/apache/kylin/pull/657#issuecomment-498176603 > ### Description > HBase is case sensitive, tableName.toUpperCase() will lead to the created table can not be found. > > ### Improvement > Just remove the toUppderCase() in CubeHFileJob. @liuzx32 Hi, why the upper case of Hbase table name bother you? Can you describe the case of "table can not be found" in detail? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] nichunen commented on issue #658: add overwrite method getKafkaConsumer in KafkaClient
nichunen commented on issue #658: add overwrite method getKafkaConsumer in KafkaClient URL: https://github.com/apache/kylin/pull/658#issuecomment-498178611 @liuzx32 Hi, could you please open a jira issue for this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4004) Multi user groups table permission error
[ https://issues.apache.org/jira/browse/KYLIN-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854398#comment-16854398 ] ASF GitHub Bot commented on KYLIN-4004: --- yuexingri commented on pull request #644: KYLIN-4004, fix table acl when use groups URL: https://github.com/apache/kylin/pull/644 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Multi user groups table permission error > > > Key: KYLIN-4004 > URL: https://issues.apache.org/jira/browse/KYLIN-4004 > Project: Kylin > Issue Type: Bug > Components: Others >Affects Versions: all >Reporter: haifeng wang >Priority: Major > Fix For: v2.6.3 > > > When a user has multiple user groups, When collecting the blacklist of the > table, Using the union of the blacklists of each group -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] yuexingri closed pull request #644: KYLIN-4004, fix table acl when use groups
yuexingri closed pull request #644: KYLIN-4004, fix table acl when use groups URL: https://github.com/apache/kylin/pull/644 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-4029) Overwriting conflict when create a new data model
[
https://issues.apache.org/jira/browse/KYLIN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-4029:
---
Issue Type: Bug (was: Improvement)
> Overwriting conflict when create a new data model
> --
>
> Key: KYLIN-4029
> URL: https://issues.apache.org/jira/browse/KYLIN-4029
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Priority: Minor
>
> A creating model request failed for Overwriting conflicts when saving project
> metadata.
> It left a stale state in meta and user can not delete it or creating a one
> with same name.
> {code:java}
> 2019-05-31 16:35:11,668 ERROR [http-bio-7070-exec-57]
> controller.BasicController:63 :
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /project/BigBI_Hive.json, expect old TS 1559291698212, but it is
> 1559291711327
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:326)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:327)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:309)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:288)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> at
> org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:373)
> at
> org.apache.kylin.metadata.project.ProjectManager.addModelToProject(ProjectManager.java:251)
> at
> org.apache.kylin.metadata.model.DataModelManager.createDataModelDesc(DataModelManager.java:256)
> at
> org.apache.kylin.rest.service.ModelService.createModelDesc(ModelService.java:148)
> at
> org.apache.kylin.rest.controller.ModelController.saveModelDesc(ModelController.java:128){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-4029) Overwriting conflict when create a new data model
[
https://issues.apache.org/jira/browse/KYLIN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-4029:
---
Affects Version/s: v2.5.2
> Overwriting conflict when create a new data model
> --
>
> Key: KYLIN-4029
> URL: https://issues.apache.org/jira/browse/KYLIN-4029
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.5.2
>Reporter: Liu Shaohui
>Priority: Minor
>
> A creating model request failed for Overwriting conflicts when saving project
> metadata.
> It left a stale state in meta and user can not delete it or creating a one
> with same name.
> {code:java}
> 2019-05-31 16:35:11,668 ERROR [http-bio-7070-exec-57]
> controller.BasicController:63 :
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /project/BigBI_Hive.json, expect old TS 1559291698212, but it is
> 1559291711327
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:326)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:327)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:309)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:288)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> at
> org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:373)
> at
> org.apache.kylin.metadata.project.ProjectManager.addModelToProject(ProjectManager.java:251)
> at
> org.apache.kylin.metadata.model.DataModelManager.createDataModelDesc(DataModelManager.java:256)
> at
> org.apache.kylin.rest.service.ModelService.createModelDesc(ModelService.java:148)
> at
> org.apache.kylin.rest.controller.ModelController.saveModelDesc(ModelController.java:128){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-4029) Overwriting conflict when create a new data model
[
https://issues.apache.org/jira/browse/KYLIN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-4029:
---
Component/s: Metadata
> Overwriting conflict when create a new data model
> --
>
> Key: KYLIN-4029
> URL: https://issues.apache.org/jira/browse/KYLIN-4029
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.2
>Reporter: Liu Shaohui
>Priority: Minor
>
> A creating model request failed for Overwriting conflicts when saving project
> metadata.
> It left a stale state in meta and user can not delete it or creating a one
> with same name.
> {code:java}
> 2019-05-31 16:35:11,668 ERROR [http-bio-7070-exec-57]
> controller.BasicController:63 :
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /project/BigBI_Hive.json, expect old TS 1559291698212, but it is
> 1559291711327
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:326)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:327)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:309)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:288)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> at
> org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:373)
> at
> org.apache.kylin.metadata.project.ProjectManager.addModelToProject(ProjectManager.java:251)
> at
> org.apache.kylin.metadata.model.DataModelManager.createDataModelDesc(DataModelManager.java:256)
> at
> org.apache.kylin.rest.service.ModelService.createModelDesc(ModelService.java:148)
> at
> org.apache.kylin.rest.controller.ModelController.saveModelDesc(ModelController.java:128){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-4029) Overwriting conflict when create a new data model
Liu Shaohui created KYLIN-4029:
--
Summary: Overwriting conflict when create a new data model
Key: KYLIN-4029
URL: https://issues.apache.org/jira/browse/KYLIN-4029
Project: Kylin
Issue Type: Improvement
Reporter: Liu Shaohui
A creating model request failed for Overwriting conflicts when saving project
metadata.
It left a stale state in meta and user can not delete it or creating a one with
same name.
{code:java}
2019-05-31 16:35:11,668 ERROR [http-bio-7070-exec-57]
controller.BasicController:63 :
org.apache.kylin.common.persistence.WriteConflictException: Overwriting
conflict /project/BigBI_Hive.json, expect old TS 1559291698212, but it is
1559291711327
at
org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:326)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:327)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:309)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:288)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
at
org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:373)
at
org.apache.kylin.metadata.project.ProjectManager.addModelToProject(ProjectManager.java:251)
at
org.apache.kylin.metadata.model.DataModelManager.createDataModelDesc(DataModelManager.java:256)
at
org.apache.kylin.rest.service.ModelService.createModelDesc(ModelService.java:148)
at
org.apache.kylin.rest.controller.ModelController.saveModelDesc(ModelController.java:128){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] coveralls commented on issue #672: KYLIN-4028 Speed up startup progress using cached dependency
coveralls commented on issue #672: KYLIN-4028 Speed up startup progress using cached dependency URL: https://github.com/apache/kylin/pull/672#issuecomment-498160906 ## Pull Request Test Coverage Report for [Build 4572](https://coveralls.io/builds/23752496) * **0** of **0** changed or added relevant lines in **0** files are covered. * **2** unchanged lines in **1** file lost coverage. * Overall coverage remained the same at **28.282%** --- | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [stream-core/src/main/java/org/apache/kylin/stream/core/storage/CheckPointStore.java](https://coveralls.io/builds/23752496/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2FCheckPointStore.java#L98) | 2 | 73.74% | | Totals | [](https://coveralls.io/builds/23752496) | | :-- | --: | | Change from base [Build 4569](https://coveralls.io/builds/23731284): | 0.0% | | Covered Lines: | 23246 | | Relevant Lines: | 82194 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] codecov-io commented on issue #672: KYLIN-4028 Speed up startup progress using cached dependency
codecov-io commented on issue #672: KYLIN-4028 Speed up startup progress using cached dependency URL: https://github.com/apache/kylin/pull/672#issuecomment-498160363 # [Codecov](https://codecov.io/gh/apache/kylin/pull/672?src=pr&el=h1) Report > Merging [#672](https://codecov.io/gh/apache/kylin/pull/672?src=pr&el=desc) into [master](https://codecov.io/gh/apache/kylin/commit/e3ed8b29a78f69ca629303ac517a669df7ae86ca?src=pr&el=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/kylin/pull/672?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #672 +/- ## - Coverage 25.8% 25.78% -0.02% + Complexity 6011 6008 -3 Files 1382 1382 Lines 8219482194 Branches 1151611516 - Hits 2120821195 -13 - Misses5894258951 +9 - Partials 2044 2048 +4 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/672?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/672/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `72.97% <0%> (-5.41%)` | `6% <0%> (-1%)` | | | [...che/kylin/stream/core/storage/CheckPointStore.java](https://codecov.io/gh/apache/kylin/pull/672/diff?src=pr&el=tree#diff-c3RyZWFtLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3N0cmVhbS9jb3JlL3N0b3JhZ2UvQ2hlY2tQb2ludFN0b3JlLmphdmE=) | `63.63% <0%> (-3.04%)` | `21% <0%> (-1%)` | | | [...he/kylin/job/impl/threadpool/DefaultScheduler.java](https://codecov.io/gh/apache/kylin/pull/672/diff?src=pr&el=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9pbXBsL3RocmVhZHBvb2wvRGVmYXVsdFNjaGVkdWxlci5qYXZh) | `74.41% <0%> (-2.33%)` | `12% <0%> (ø)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/672/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `69.6% <0%> (-1.52%)` | `7% <0%> (ø)` | | | [...a/org/apache/kylin/dict/Number2BytesConverter.java](https://codecov.io/gh/apache/kylin/pull/672/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlcjJCeXRlc0NvbnZlcnRlci5qYXZh) | `81.74% <0%> (-0.8%)` | `17% <0%> (-1%)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/672?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/672?src=pr&el=footer). Last update [e3ed8b2...37bfe8b](https://codecov.io/gh/apache/kylin/pull/672?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4028) Speed up startup progress using cached dependency
[
https://issues.apache.org/jira/browse/KYLIN-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854349#comment-16854349
]
Temple Zhou commented on KYLIN-4028:
{code:bash}
..
if [[ -z $reload_dependency && `ls -1 ${dir}/cached-* 2>/dev/null | wc -l`
-eq 5 ]]
then
echo "Using cached dependency..."
source ${dir}/cached-hive-dependency.sh
source ${dir}/cached-hbase-dependency.sh
source ${dir}/cached-hadoop-conf-dir.sh
source ${dir}/cached-kafka-dependency.sh
source ${dir}/cached-spark-dependency.sh
else
source ${dir}/find-hive-dependency.sh
source ${dir}/find-hbase-dependency.sh
source ${dir}/find-hadoop-conf-dir.sh
source ${dir}/find-kafka-dependency.sh
source ${dir}/find-spark-dependency.sh
fi
..
{code}
Just set the second parameter to "--reload-dependency", which will reload the
dependency.
> Speed up startup progress using cached dependency
> -
>
> Key: KYLIN-4028
> URL: https://issues.apache.org/jira/browse/KYLIN-4028
> Project: Kylin
> Issue Type: Improvement
> Components: Others
>Affects Versions: all
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Minor
>
> The hive/hadoop/hbase dependencies are not volatile, and finding the
> dependencies every time I start the Kylin server will slow down the startup
> speed.
> So, if there are dependencies generated by previous running, we can use it to
> start the server without finding the dependencies again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] shaofengshi commented on a change in pull request #661: KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files
shaofengshi commented on a change in pull request #661: KYLIN-4015 "Build UHC
Dictionary" step filter ".dci" files
URL: https://github.com/apache/kylin/pull/661#discussion_r288366396
##
File path:
engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/filter/UHCDictPathFilter.java
##
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+package org.apache.kylin.engine.mr.steps.filter;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Locale;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class UHCDictPathFilter implements PathFilter {
+private static final Logger logger =
LoggerFactory.getLogger(UHCDictPathFilter.class);
+
+private static final String DCIFILE_POSTFIX = ".dci";
+
+@Override
+public boolean accept(Path path) {
+
+Pattern pattern = Pattern.compile(DCIFILE_POSTFIX);
+Matcher matcher =
pattern.matcher(path.getName().toLowerCase(Locale.ROOT));
+
+if (matcher.find()) {
+logger.info("filter file: " + path.getName());
+return false;
Review comment:
Hello Jintao, this filter is doing in a blacklist way, can we implement it
in a whitelist way? Say, ".dci" is an exceptional case, there might be another
exceptional case in the future, if the developer doesn't know this list, the
same error will occur again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] shaofengshi commented on a change in pull request #661: KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files
shaofengshi commented on a change in pull request #661: KYLIN-4015 "Build UHC
Dictionary" step filter ".dci" files
URL: https://github.com/apache/kylin/pull/661#discussion_r288366515
##
File path:
engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/filter/UHCDictPathFilter.java
##
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+package org.apache.kylin.engine.mr.steps.filter;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Locale;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class UHCDictPathFilter implements PathFilter {
+private static final Logger logger =
LoggerFactory.getLogger(UHCDictPathFilter.class);
+
+private static final String DCIFILE_POSTFIX = ".dci";
Review comment:
Please reuse the DIMENSION_COL_INFO_FILE_POSTFIX contant in
FactDistinctColumnsReducer, or abstract it to a common place like
BatchConstants.java.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (KYLIN-4028) Speed up startup progress using cached dependency
[ https://issues.apache.org/jira/browse/KYLIN-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854322#comment-16854322 ] Shaofeng SHI commented on KYLIN-4028: - This is a good idea! It will accelerate the startup, and also minimal the impace when some services like Hive/HBase are inactive. Temple, is there a way to clear the cache easily? In case some times the classpath or path got changed in the client side. > Speed up startup progress using cached dependency > - > > Key: KYLIN-4028 > URL: https://issues.apache.org/jira/browse/KYLIN-4028 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: all >Reporter: Temple Zhou >Assignee: Temple Zhou >Priority: Minor > > The hive/hadoop/hbase dependencies are not volatile, and finding the > dependencies every time I start the Kylin server will slow down the startup > speed. > So, if there are dependencies generated by previous running, we can use it to > start the server without finding the dependencies again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4028) Speed up startup progress using cached dependency
[ https://issues.apache.org/jira/browse/KYLIN-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854308#comment-16854308 ] ASF GitHub Bot commented on KYLIN-4028: --- TempleZhou commented on pull request #672: KYLIN-4028 Speed up startup progress using cached dependency URL: https://github.com/apache/kylin/pull/672 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Speed up startup progress using cached dependency > - > > Key: KYLIN-4028 > URL: https://issues.apache.org/jira/browse/KYLIN-4028 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: all >Reporter: Temple Zhou >Assignee: Temple Zhou >Priority: Minor > > The hive/hadoop/hbase dependencies are not volatile, and finding the > dependencies every time I start the Kylin server will slow down the startup > speed. > So, if there are dependencies generated by previous running, we can use it to > start the server without finding the dependencies again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] asfgit commented on issue #672: KYLIN-4028 Speed up startup progress using cached dependency
asfgit commented on issue #672: KYLIN-4028 Speed up startup progress using cached dependency URL: https://github.com/apache/kylin/pull/672#issuecomment-498149179 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] TempleZhou opened a new pull request #672: KYLIN-4028 Speed up startup progress using cached dependency
TempleZhou opened a new pull request #672: KYLIN-4028 Speed up startup progress using cached dependency URL: https://github.com/apache/kylin/pull/672 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (KYLIN-4028) Speed up startup progress using cached dependency
Temple Zhou created KYLIN-4028: -- Summary: Speed up startup progress using cached dependency Key: KYLIN-4028 URL: https://issues.apache.org/jira/browse/KYLIN-4028 Project: Kylin Issue Type: Improvement Components: Others Affects Versions: all Reporter: Temple Zhou Assignee: Temple Zhou The hive/hadoop/hbase dependencies are not volatile, and finding the dependencies every time I start the Kylin server will slow down the startup speed. So, if there are dependencies generated by previous running, we can use it to start the server without finding the dependencies again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
