[jira] [Updated] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[ https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang chen updated KYLIN-3562: -- Attachment: image-2018-09-17-16-40-56-212.png > TS conflict when kylin update metadata in HBase > --- > > Key: KYLIN-3562 > URL: https://issues.apache.org/jira/browse/KYLIN-3562 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.4.0 >Reporter: Lingang Deng >Priority: Major > Attachments: image-2018-09-17-16-40-56-212.png > > > Error log was as follows, > {code:java} > org.apache.kylin.common.persistence.WriteConflictException: Overwriting > conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207 > at > org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325) > at > org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282) > at > org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code} > > what disturbs me the most was that the error was happened several hours, > then my all build job and query job failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[ https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617247#comment-16617247 ] Chang chen commented on KYLIN-3562: --- [~Shaofengshi] Please look at this issue. we will meet it regularly every day since last week. It looks like a concurrency issue, i.e. two threads simultaneously update /user/admin. Obviously, one thread get oldTS before another thread. !image-2018-09-17-16-40-56-212.png! Above is result not reason, because we find that kylin update user very frequently! Taking a look at *KylinAuthenticationProvider.authenticate*, if the user is in userCache, then we will avoid call *KylinUserService.updateUser*, but unfortunately *updateUser* will set {color:#FF}_evictCacheFlag_{color} to true which makes userCache is invalidated at the next *authenticate* call. > TS conflict when kylin update metadata in HBase > --- > > Key: KYLIN-3562 > URL: https://issues.apache.org/jira/browse/KYLIN-3562 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.4.0 >Reporter: Lingang Deng >Priority: Major > Attachments: image-2018-09-17-16-40-56-212.png > > > Error log was as follows, > {code:java} > org.apache.kylin.common.persistence.WriteConflictException: Overwriting > conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207 > at > org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325) > at > org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282) > at > org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code} > > what disturbs me the most was that the error was happened several hours, > then my all build job and query job failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[ https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617247#comment-16617247 ] Chang chen edited comment on KYLIN-3562 at 9/17/18 8:57 AM: [~Shaofengshi] Please look at this issue. we will meet it regularly every day since last week. It looks like a concurrency issue, i.e. two threads simultaneously update /user/admin. Obviously, one thread get oldTS before another thread updates key. !image-2018-09-17-16-40-56-212.png! Above is result not reason, because we find that kylin update user very frequently! Taking a look at *KylinAuthenticationProvider.authenticate*, if the user is in userCache, then we will avoid call *KylinUserService.updateUser*, but unfortunately *updateUser* will set {color:#ff}_evictCacheFlag_{color} to true which makes userCache is invalidated at the next *authenticate* call. was (Author: baibaichen): [~Shaofengshi] Please look at this issue. we will meet it regularly every day since last week. It looks like a concurrency issue, i.e. two threads simultaneously update /user/admin. Obviously, one thread get oldTS before another thread. !image-2018-09-17-16-40-56-212.png! Above is result not reason, because we find that kylin update user very frequently! Taking a look at *KylinAuthenticationProvider.authenticate*, if the user is in userCache, then we will avoid call *KylinUserService.updateUser*, but unfortunately *updateUser* will set {color:#FF}_evictCacheFlag_{color} to true which makes userCache is invalidated at the next *authenticate* call. > TS conflict when kylin update metadata in HBase > --- > > Key: KYLIN-3562 > URL: https://issues.apache.org/jira/browse/KYLIN-3562 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.4.0 >Reporter: Lingang Deng >Priority: Major > Attachments: image-2018-09-17-16-40-56-212.png > > > Error log was as follows, > {code:java} > org.apache.kylin.common.persistence.WriteConflictException: Overwriting > conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207 > at > org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325) > at > org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282) > at > org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code} > > what disturbs me the most was that the error was happened several hours, > then my all build job and query job failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3566) Concating two dims in where clause is wrong
Chang chen created KYLIN-3566: - Summary: Concating two dims in where clause is wrong Key: KYLIN-3566 URL: https://issues.apache.org/jira/browse/KYLIN-3566 Project: Kylin Issue Type: Bug Affects Versions: v2.4.0 Reporter: Chang chen Attachments: image-2018-09-17-20-27-04-025.png The following SQL returns less data: {quote}SELECT concat(concat("DATE",' '), minutes) AS "__timestamp", count(*) as pv FROM kylin."MODEL_MODELSERVER" WHERE concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00' GROUP BY concat(concat("DATE",' '), minutes) order by "__timestamp" {quote} If I rewrite where as {quote} ("DATE" = '2018-09-04' and minutes >= '01:00') or "DATE" >= '2018-09-05' {quote} The Result sets are correct. This is because *BuiltInFunctionTransformer.translateFunctionTupleFilter* only pass the _minutes_ to *concat*. see the codes in red rectangle. !image-2018-09-17-20-27-04-025.png! And hence for _concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00'_, it actually evals _minutes >=_ _'2018-09-04 00:00:00'_ , which gets true when minutes >= "20:00" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3566) Concating two dims in where clause is wrong
[ https://issues.apache.org/jira/browse/KYLIN-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang chen updated KYLIN-3566: -- Description: The following SQL returns less data: {quote}SELECT concat(concat("DATE",' '), minutes) AS "__timestamp", count( * ) as pv FROM kylin."MODEL_MODELSERVER" WHERE concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00' GROUP BY concat(concat("DATE",' '), minutes) order by "__timestamp" {quote} If I rewrite where as {quote} ("DATE" = '2018-09-04' and minutes >= '01:00') or "DATE" >= '2018-09-05' {quote} The Result sets are correct. This is because *BuiltInFunctionTransformer.translateFunctionTupleFilter* only pass the _minutes_ to *concat*. see the codes in red rectangle. !image-2018-09-17-20-27-04-025.png! And hence for _concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00'_, it actually evals _minutes >=_ _'2018-09-04 00:00:00'_ , which gets true when minutes >= "20:00" was: The following SQL returns less data: {quote}SELECT concat(concat("DATE",' '), minutes) AS "__timestamp", count(*) as pv FROM kylin."MODEL_MODELSERVER" WHERE concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00' GROUP BY concat(concat("DATE",' '), minutes) order by "__timestamp" {quote} If I rewrite where as {quote} ("DATE" = '2018-09-04' and minutes >= '01:00') or "DATE" >= '2018-09-05' {quote} The Result sets are correct. This is because *BuiltInFunctionTransformer.translateFunctionTupleFilter* only pass the _minutes_ to *concat*. see the codes in red rectangle. !image-2018-09-17-20-27-04-025.png! And hence for _concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00'_, it actually evals _minutes >=_ _'2018-09-04 00:00:00'_ , which gets true when minutes >= "20:00" > Concating two dims in where clause is wrong > --- > > Key: KYLIN-3566 > URL: https://issues.apache.org/jira/browse/KYLIN-3566 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.4.0 >Reporter: Chang chen >Priority: Major > Attachments: image-2018-09-17-20-27-04-025.png > > > The following SQL returns less data: > {quote}SELECT > concat(concat("DATE",' '), minutes) AS "__timestamp", > count( * ) as pv > FROM kylin."MODEL_MODELSERVER" > WHERE concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00' > GROUP BY concat(concat("DATE",' '), minutes) > order by "__timestamp" > {quote} > If I rewrite where as > {quote} ("DATE" = '2018-09-04' and minutes >= '01:00') or "DATE" >= > '2018-09-05' > {quote} > The Result sets are correct. This is because > *BuiltInFunctionTransformer.translateFunctionTupleFilter* only pass the > _minutes_ to *concat*. see the codes in red rectangle. > !image-2018-09-17-20-27-04-025.png! > And hence for _concat(concat("DATE",' '), minutes) >= '2018-09-04 > 00:00:00'_, it actually evals _minutes >=_ _'2018-09-04 00:00:00'_ , which > gets true when minutes >= "20:00" > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[ https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16620380#comment-16620380 ] Chang chen commented on KYLIN-3562: --- [~yichen.zhou] this issue is caused by update meta concurrently, I don't think it is related with KYLIN-3531 > TS conflict when kylin update metadata in HBase > --- > > Key: KYLIN-3562 > URL: https://issues.apache.org/jira/browse/KYLIN-3562 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.4.0 >Reporter: Lingang Deng >Priority: Major > Attachments: image-2018-09-17-16-40-56-212.png > > > Error log was as follows, > {code:java} > org.apache.kylin.common.persistence.WriteConflictException: Overwriting > conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207 > at > org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325) > at > org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282) > at > org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code} > > what disturbs me the most was that the error was happened several hours, > then my all build job and query job failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[ https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang chen updated KYLIN-3562: -- Attachment: image-2018-09-25-15-03-51-009.png > TS conflict when kylin update metadata in HBase > --- > > Key: KYLIN-3562 > URL: https://issues.apache.org/jira/browse/KYLIN-3562 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.4.0 >Reporter: Lingang Deng >Priority: Major > Attachments: image-2018-09-17-16-40-56-212.png, > image-2018-09-25-15-03-51-009.png > > > Error log was as follows, > {code:java} > org.apache.kylin.common.persistence.WriteConflictException: Overwriting > conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207 > at > org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325) > at > org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282) > at > org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code} > > what disturbs me the most was that the error was happened several hours, > then my all build job and query job failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[ https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626890#comment-16626890 ] Chang chen commented on KYLIN-3562: --- Is there anyone looking into this issue? we fequently meet this issue !image-2018-09-25-15-03-51-009.png! > TS conflict when kylin update metadata in HBase > --- > > Key: KYLIN-3562 > URL: https://issues.apache.org/jira/browse/KYLIN-3562 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.4.0 >Reporter: Lingang Deng >Priority: Major > Attachments: image-2018-09-17-16-40-56-212.png, > image-2018-09-25-15-03-51-009.png > > > Error log was as follows, > {code:java} > org.apache.kylin.common.persistence.WriteConflictException: Overwriting > conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207 > at > org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325) > at > org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282) > at > org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code} > > what disturbs me the most was that the error was happened several hours, > then my all build job and query job failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[ https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626890#comment-16626890 ] Chang chen edited comment on KYLIN-3562 at 9/25/18 8:43 AM: [~Shaofengshi] please take a look at this issue? we fequently meet it I think it's caused by KYLIN-3246 ([git|https://github.com/apache/kylin/commit/7c9892b2f8163d63536d4b5e76759186584c3d60?diff=split#diff-73841391c29e982f15bf5f5f6d1fe8a1]), before this pathc, when was (Author: baibaichen): Is there anyone looking into this issue? we fequently meet this issue !image-2018-09-25-15-03-51-009.png! > TS conflict when kylin update metadata in HBase > --- > > Key: KYLIN-3562 > URL: https://issues.apache.org/jira/browse/KYLIN-3562 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.4.0 >Reporter: Lingang Deng >Priority: Major > Attachments: image-2018-09-17-16-40-56-212.png, > image-2018-09-25-15-03-51-009.png > > > Error log was as follows, > {code:java} > org.apache.kylin.common.persistence.WriteConflictException: Overwriting > conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207 > at > org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325) > at > org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282) > at > org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code} > > what disturbs me the most was that the error was happened several hours, > then my all build job and query job failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[ https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626890#comment-16626890 ] Chang chen edited comment on KYLIN-3562 at 9/25/18 8:46 AM: [~Shaofengshi] please take a look at this issue? we fequently meet it I think it's caused by KYLIN-3246 ([git|https://github.com/apache/kylin/commit/7c9892b2f8163d63536d4b5e76759186584c3d60?diff=split#diff-73841391c29e982f15bf5f5f6d1fe8a1]), before this patch, KylinUserService.updateUser will call _ResourceStore.putResourceWithoutCheck_, and after this patch, this function will eventually call _ResourceStore.checkAndPutResourceCheckpoint_. !image-2018-09-25-16-43-50-277.png! was (Author: baibaichen): [~Shaofengshi] please take a look at this issue? we fequently meet it I think it's caused by KYLIN-3246 ([git|https://github.com/apache/kylin/commit/7c9892b2f8163d63536d4b5e76759186584c3d60?diff=split#diff-73841391c29e982f15bf5f5f6d1fe8a1]), before this pathc, when > TS conflict when kylin update metadata in HBase > --- > > Key: KYLIN-3562 > URL: https://issues.apache.org/jira/browse/KYLIN-3562 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.4.0 >Reporter: Lingang Deng >Priority: Major > Attachments: image-2018-09-17-16-40-56-212.png, > image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png > > > Error log was as follows, > {code:java} > org.apache.kylin.common.persistence.WriteConflictException: Overwriting > conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207 > at > org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325) > at > org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282) > at > org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code} > > what disturbs me the most was that the error was happened several hours, > then my all build job and query job failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[ https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626890#comment-16626890 ] Chang chen edited comment on KYLIN-3562 at 9/25/18 8:51 AM: [~Shaofengshi] please take a look at this issue? we fequently meet it I think it's caused by KYLIN-3246 ([git|https://github.com/apache/kylin/commit/7c9892b2f8163d63536d4b5e76759186584c3d60?diff=split#diff-73841391c29e982f15bf5f5f6d1fe8a1]), before this patch, KylinUserService.updateUser will call _ResourceStore.putResourceWithoutCheck_, and after this patch, this function will eventually call _ResourceStore.checkAndPutResourceCheckpoint_. !image-2018-09-25-16-43-50-277.png! [~Aron.tao], could you please see this issue? I see your are the first author of KylinUserService.java, and using putResourceWithoutCheck at the first place Thanks Chang was (Author: baibaichen): [~Shaofengshi] please take a look at this issue? we fequently meet it I think it's caused by KYLIN-3246 ([git|https://github.com/apache/kylin/commit/7c9892b2f8163d63536d4b5e76759186584c3d60?diff=split#diff-73841391c29e982f15bf5f5f6d1fe8a1]), before this patch, KylinUserService.updateUser will call _ResourceStore.putResourceWithoutCheck_, and after this patch, this function will eventually call _ResourceStore.checkAndPutResourceCheckpoint_. !image-2018-09-25-16-43-50-277.png! > TS conflict when kylin update metadata in HBase > --- > > Key: KYLIN-3562 > URL: https://issues.apache.org/jira/browse/KYLIN-3562 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.4.0 >Reporter: Lingang Deng >Priority: Major > Attachments: image-2018-09-17-16-40-56-212.png, > image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png > > > Error log was as follows, > {code:java} > org.apache.kylin.common.persistence.WriteConflictException: Overwriting > conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207 > at > org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325) > at > org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282) > at > org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code} > > what disturbs me the most was that the error was happened several hours, > then my all build job and query job failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[ https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628482#comment-16628482 ] Chang chen commented on KYLIN-3562: --- The root cause is same as KYLIN-3499, however we can still remove unnecessary update. > TS conflict when kylin update metadata in HBase > --- > > Key: KYLIN-3562 > URL: https://issues.apache.org/jira/browse/KYLIN-3562 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.4.0 >Reporter: Lingang Deng >Priority: Major > Attachments: image-2018-09-17-16-40-56-212.png, > image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png > > > Error log was as follows, > {code:java} > org.apache.kylin.common.persistence.WriteConflictException: Overwriting > conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207 > at > org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325) > at > org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303) > at > org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282) > at > org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code} > > what disturbs me the most was that the error was happened several hours, > then my all build job and query job failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3656) Improve HLLCounter performance
Chang chen created KYLIN-3656: - Summary: Improve HLLCounter performance Key: KYLIN-3656 URL: https://issues.apache.org/jira/browse/KYLIN-3656 Project: Kylin Issue Type: Improvement Affects Versions: all Reporter: Chang chen The current HLLCounter implementation has some room to improve performance, as we find in our product environment. # Create HLLCounter from another HLLCounter, we can copy register instead of merge # To compute harmonic mean in the HLLCSnapshot, we could ## using table to cache all 1/2^r without computing on the fly ## remove floating addition by using integer addition in the bigger loop ## remove branch, e.g. needn't checking whether registers[i] is zero or not, although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3656) Improve HLLCounter performance
[ https://issues.apache.org/jira/browse/KYLIN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang chen updated KYLIN-3656: -- Attachment: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch > Improve HLLCounter performance > -- > > Key: KYLIN-3656 > URL: https://issues.apache.org/jira/browse/KYLIN-3656 > Project: Kylin > Issue Type: Improvement >Affects Versions: all >Reporter: Chang chen >Priority: Major > Attachments: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch > > > The current HLLCounter implementation has some room to improve performance, > as we find in our product environment. > # Create HLLCounter from another HLLCounter, we can copy register instead > of merge > # To compute harmonic mean in the HLLCSnapshot, we could > ## using table to cache all 1/2^r without computing on the fly > ## remove floating addition by using integer addition in the bigger loop > ## remove branch, e.g. needn't checking whether registers[i] is zero or not, > although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3656) Improve HLLCounter performance
[ https://issues.apache.org/jira/browse/KYLIN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671478#comment-16671478 ] Chang chen commented on KYLIN-3656: --- To get better result, we disable compress by following settings: kylin.storage.hbase.endpoint-compress-result=false > Improve HLLCounter performance > -- > > Key: KYLIN-3656 > URL: https://issues.apache.org/jira/browse/KYLIN-3656 > Project: Kylin > Issue Type: Improvement >Affects Versions: all >Reporter: Chang chen >Assignee: Chang chen >Priority: Major > Fix For: v2.6.0 > > Attachments: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch > > > The current HLLCounter implementation has some room to improve performance, > as we find in our product environment. > # Create HLLCounter from another HLLCounter, we can copy register instead > of merge > # To compute harmonic mean in the HLLCSnapshot, we could > ## using table to cache all 1/2^r without computing on the fly > ## remove floating addition by using integer addition in the bigger loop > ## remove branch, e.g. needn't checking whether registers[i] is zero or not, > although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3656) Improve HLLCounter performance
[ https://issues.apache.org/jira/browse/KYLIN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang chen updated KYLIN-3656: -- Attachment: image-2018-11-05-18-15-36-463.png > Improve HLLCounter performance > -- > > Key: KYLIN-3656 > URL: https://issues.apache.org/jira/browse/KYLIN-3656 > Project: Kylin > Issue Type: Improvement >Affects Versions: all >Reporter: Chang chen >Assignee: Chang chen >Priority: Major > Fix For: v2.6.0 > > Attachments: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch, > image-2018-11-05-18-15-36-463.png > > > The current HLLCounter implementation has some room to improve performance, > as we find in our product environment. > # Create HLLCounter from another HLLCounter, we can copy register instead > of merge > # To compute harmonic mean in the HLLCSnapshot, we could > ## using table to cache all 1/2^r without computing on the fly > ## remove floating addition by using integer addition in the bigger loop > ## remove branch, e.g. needn't checking whether registers[i] is zero or not, > although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3656) Improve HLLCounter performance
[ https://issues.apache.org/jira/browse/KYLIN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674917#comment-16674917 ] Chang chen commented on KYLIN-3656: --- [~yimingliu] you can try the UT for the result. The following is my Test : !image-2018-11-05-18-15-36-463.png! Removing reflection isn't include in this patch. > Improve HLLCounter performance > -- > > Key: KYLIN-3656 > URL: https://issues.apache.org/jira/browse/KYLIN-3656 > Project: Kylin > Issue Type: Improvement >Affects Versions: all >Reporter: Chang chen >Assignee: Chang chen >Priority: Major > Fix For: v2.6.0 > > Attachments: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch, > image-2018-11-05-18-15-36-463.png > > > The current HLLCounter implementation has some room to improve performance, > as we find in our product environment. > # Create HLLCounter from another HLLCounter, we can copy register instead > of merge > # To compute harmonic mean in the HLLCSnapshot, we could > ## using table to cache all 1/2^r without computing on the fly > ## remove floating addition by using integer addition in the bigger loop > ## remove branch, e.g. needn't checking whether registers[i] is zero or not, > although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)