[jira] [Updated] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chang chen updated KYLIN-3562:
--
Attachment: image-2018-09-17-16-40-56-212.png
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Priority: Major
> Attachments: image-2018-09-17-16-40-56-212.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617247#comment-16617247
]
Chang chen commented on KYLIN-3562:
---
[~Shaofengshi] Please look at this issue. we will meet it regularly every day
since last week. It looks like a concurrency issue, i.e. two threads
simultaneously update /user/admin. Obviously, one thread get oldTS before
another thread.
!image-2018-09-17-16-40-56-212.png!
Above is result not reason, because we find that kylin update user very
frequently!
Taking a look at *KylinAuthenticationProvider.authenticate*, if the user is in
userCache, then we will avoid call *KylinUserService.updateUser*, but
unfortunately *updateUser* will set {color:#FF}_evictCacheFlag_{color} to
true which makes userCache is invalidated at the next *authenticate* call.
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Priority: Major
> Attachments: image-2018-09-17-16-40-56-212.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617247#comment-16617247
]
Chang chen edited comment on KYLIN-3562 at 9/17/18 8:57 AM:
[~Shaofengshi] Please look at this issue. we will meet it regularly every day
since last week. It looks like a concurrency issue, i.e. two threads
simultaneously update /user/admin. Obviously, one thread get oldTS before
another thread updates key.
!image-2018-09-17-16-40-56-212.png!
Above is result not reason, because we find that kylin update user very
frequently!
Taking a look at *KylinAuthenticationProvider.authenticate*, if the user is in
userCache, then we will avoid call *KylinUserService.updateUser*, but
unfortunately *updateUser* will set {color:#ff}_evictCacheFlag_{color} to
true which makes userCache is invalidated at the next *authenticate* call.
was (Author: baibaichen):
[~Shaofengshi] Please look at this issue. we will meet it regularly every day
since last week. It looks like a concurrency issue, i.e. two threads
simultaneously update /user/admin. Obviously, one thread get oldTS before
another thread.
!image-2018-09-17-16-40-56-212.png!
Above is result not reason, because we find that kylin update user very
frequently!
Taking a look at *KylinAuthenticationProvider.authenticate*, if the user is in
userCache, then we will avoid call *KylinUserService.updateUser*, but
unfortunately *updateUser* will set {color:#FF}_evictCacheFlag_{color} to
true which makes userCache is invalidated at the next *authenticate* call.
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Priority: Major
> Attachments: image-2018-09-17-16-40-56-212.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3566) Concating two dims in where clause is wrong
Chang chen created KYLIN-3566:
-
Summary: Concating two dims in where clause is wrong
Key: KYLIN-3566
URL: https://issues.apache.org/jira/browse/KYLIN-3566
Project: Kylin
Issue Type: Bug
Affects Versions: v2.4.0
Reporter: Chang chen
Attachments: image-2018-09-17-20-27-04-025.png
The following SQL returns less data:
{quote}SELECT
concat(concat("DATE",' '), minutes) AS "__timestamp",
count(*) as pv
FROM kylin."MODEL_MODELSERVER"
WHERE concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00'
GROUP BY concat(concat("DATE",' '), minutes)
order by "__timestamp"
{quote}
If I rewrite where as
{quote} ("DATE" = '2018-09-04' and minutes >= '01:00') or "DATE" >=
'2018-09-05'
{quote}
The Result sets are correct. This is because
*BuiltInFunctionTransformer.translateFunctionTupleFilter* only pass the
_minutes_ to *concat*. see the codes in red rectangle.
!image-2018-09-17-20-27-04-025.png!
And hence for _concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00'_,
it actually evals _minutes >=_ _'2018-09-04 00:00:00'_ , which gets true when
minutes >= "20:00"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3566) Concating two dims in where clause is wrong
[
https://issues.apache.org/jira/browse/KYLIN-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chang chen updated KYLIN-3566:
--
Description:
The following SQL returns less data:
{quote}SELECT
concat(concat("DATE",' '), minutes) AS "__timestamp",
count( * ) as pv
FROM kylin."MODEL_MODELSERVER"
WHERE concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00'
GROUP BY concat(concat("DATE",' '), minutes)
order by "__timestamp"
{quote}
If I rewrite where as
{quote} ("DATE" = '2018-09-04' and minutes >= '01:00') or "DATE" >=
'2018-09-05'
{quote}
The Result sets are correct. This is because
*BuiltInFunctionTransformer.translateFunctionTupleFilter* only pass the
_minutes_ to *concat*. see the codes in red rectangle.
!image-2018-09-17-20-27-04-025.png!
And hence for _concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00'_,
it actually evals _minutes >=_ _'2018-09-04 00:00:00'_ , which gets true when
minutes >= "20:00"
was:
The following SQL returns less data:
{quote}SELECT
concat(concat("DATE",' '), minutes) AS "__timestamp",
count(*) as pv
FROM kylin."MODEL_MODELSERVER"
WHERE concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00'
GROUP BY concat(concat("DATE",' '), minutes)
order by "__timestamp"
{quote}
If I rewrite where as
{quote} ("DATE" = '2018-09-04' and minutes >= '01:00') or "DATE" >=
'2018-09-05'
{quote}
The Result sets are correct. This is because
*BuiltInFunctionTransformer.translateFunctionTupleFilter* only pass the
_minutes_ to *concat*. see the codes in red rectangle.
!image-2018-09-17-20-27-04-025.png!
And hence for _concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00'_,
it actually evals _minutes >=_ _'2018-09-04 00:00:00'_ , which gets true when
minutes >= "20:00"
> Concating two dims in where clause is wrong
> ---
>
> Key: KYLIN-3566
> URL: https://issues.apache.org/jira/browse/KYLIN-3566
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Chang chen
>Priority: Major
> Attachments: image-2018-09-17-20-27-04-025.png
>
>
> The following SQL returns less data:
> {quote}SELECT
> concat(concat("DATE",' '), minutes) AS "__timestamp",
> count( * ) as pv
> FROM kylin."MODEL_MODELSERVER"
> WHERE concat(concat("DATE",' '), minutes) >= '2018-09-04 00:00:00'
> GROUP BY concat(concat("DATE",' '), minutes)
> order by "__timestamp"
> {quote}
> If I rewrite where as
> {quote} ("DATE" = '2018-09-04' and minutes >= '01:00') or "DATE" >=
> '2018-09-05'
> {quote}
> The Result sets are correct. This is because
> *BuiltInFunctionTransformer.translateFunctionTupleFilter* only pass the
> _minutes_ to *concat*. see the codes in red rectangle.
> !image-2018-09-17-20-27-04-025.png!
> And hence for _concat(concat("DATE",' '), minutes) >= '2018-09-04
> 00:00:00'_, it actually evals _minutes >=_ _'2018-09-04 00:00:00'_ , which
> gets true when minutes >= "20:00"
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16620380#comment-16620380
]
Chang chen commented on KYLIN-3562:
---
[~yichen.zhou] this issue is caused by update meta concurrently, I don't think
it is related with KYLIN-3531
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Priority: Major
> Attachments: image-2018-09-17-16-40-56-212.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chang chen updated KYLIN-3562:
--
Attachment: image-2018-09-25-15-03-51-009.png
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Priority: Major
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626890#comment-16626890
]
Chang chen commented on KYLIN-3562:
---
Is there anyone looking into this issue? we fequently meet this issue
!image-2018-09-25-15-03-51-009.png!
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Priority: Major
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626890#comment-16626890
]
Chang chen edited comment on KYLIN-3562 at 9/25/18 8:43 AM:
[~Shaofengshi] please take a look at this issue? we fequently meet it
I think it's caused by KYLIN-3246
([git|https://github.com/apache/kylin/commit/7c9892b2f8163d63536d4b5e76759186584c3d60?diff=split#diff-73841391c29e982f15bf5f5f6d1fe8a1]),
before this pathc, when
was (Author: baibaichen):
Is there anyone looking into this issue? we fequently meet this issue
!image-2018-09-25-15-03-51-009.png!
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Priority: Major
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626890#comment-16626890
]
Chang chen edited comment on KYLIN-3562 at 9/25/18 8:46 AM:
[~Shaofengshi] please take a look at this issue? we fequently meet it
I think it's caused by KYLIN-3246
([git|https://github.com/apache/kylin/commit/7c9892b2f8163d63536d4b5e76759186584c3d60?diff=split#diff-73841391c29e982f15bf5f5f6d1fe8a1]),
before this patch, KylinUserService.updateUser will call
_ResourceStore.putResourceWithoutCheck_, and after this patch, this function
will eventually call _ResourceStore.checkAndPutResourceCheckpoint_.
!image-2018-09-25-16-43-50-277.png!
was (Author: baibaichen):
[~Shaofengshi] please take a look at this issue? we fequently meet it
I think it's caused by KYLIN-3246
([git|https://github.com/apache/kylin/commit/7c9892b2f8163d63536d4b5e76759186584c3d60?diff=split#diff-73841391c29e982f15bf5f5f6d1fe8a1]),
before this pathc, when
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Priority: Major
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626890#comment-16626890
]
Chang chen edited comment on KYLIN-3562 at 9/25/18 8:51 AM:
[~Shaofengshi] please take a look at this issue? we fequently meet it
I think it's caused by KYLIN-3246
([git|https://github.com/apache/kylin/commit/7c9892b2f8163d63536d4b5e76759186584c3d60?diff=split#diff-73841391c29e982f15bf5f5f6d1fe8a1]),
before this patch, KylinUserService.updateUser will call
_ResourceStore.putResourceWithoutCheck_, and after this patch, this function
will eventually call _ResourceStore.checkAndPutResourceCheckpoint_.
!image-2018-09-25-16-43-50-277.png!
[~Aron.tao], could you please see this issue? I see your are the first author
of KylinUserService.java, and using putResourceWithoutCheck at the first place
Thanks
Chang
was (Author: baibaichen):
[~Shaofengshi] please take a look at this issue? we fequently meet it
I think it's caused by KYLIN-3246
([git|https://github.com/apache/kylin/commit/7c9892b2f8163d63536d4b5e76759186584c3d60?diff=split#diff-73841391c29e982f15bf5f5f6d1fe8a1]),
before this patch, KylinUserService.updateUser will call
_ResourceStore.putResourceWithoutCheck_, and after this patch, this function
will eventually call _ResourceStore.checkAndPutResourceCheckpoint_.
!image-2018-09-25-16-43-50-277.png!
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Priority: Major
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628482#comment-16628482
]
Chang chen commented on KYLIN-3562:
---
The root cause is same as KYLIN-3499, however we can still remove unnecessary
update.
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Priority: Major
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3656) Improve HLLCounter performance
Chang chen created KYLIN-3656: - Summary: Improve HLLCounter performance Key: KYLIN-3656 URL: https://issues.apache.org/jira/browse/KYLIN-3656 Project: Kylin Issue Type: Improvement Affects Versions: all Reporter: Chang chen The current HLLCounter implementation has some room to improve performance, as we find in our product environment. # Create HLLCounter from another HLLCounter, we can copy register instead of merge # To compute harmonic mean in the HLLCSnapshot, we could ## using table to cache all 1/2^r without computing on the fly ## remove floating addition by using integer addition in the bigger loop ## remove branch, e.g. needn't checking whether registers[i] is zero or not, although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3656) Improve HLLCounter performance
[ https://issues.apache.org/jira/browse/KYLIN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang chen updated KYLIN-3656: -- Attachment: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch > Improve HLLCounter performance > -- > > Key: KYLIN-3656 > URL: https://issues.apache.org/jira/browse/KYLIN-3656 > Project: Kylin > Issue Type: Improvement >Affects Versions: all >Reporter: Chang chen >Priority: Major > Attachments: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch > > > The current HLLCounter implementation has some room to improve performance, > as we find in our product environment. > # Create HLLCounter from another HLLCounter, we can copy register instead > of merge > # To compute harmonic mean in the HLLCSnapshot, we could > ## using table to cache all 1/2^r without computing on the fly > ## remove floating addition by using integer addition in the bigger loop > ## remove branch, e.g. needn't checking whether registers[i] is zero or not, > although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3656) Improve HLLCounter performance
[ https://issues.apache.org/jira/browse/KYLIN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671478#comment-16671478 ] Chang chen commented on KYLIN-3656: --- To get better result, we disable compress by following settings: kylin.storage.hbase.endpoint-compress-result=false > Improve HLLCounter performance > -- > > Key: KYLIN-3656 > URL: https://issues.apache.org/jira/browse/KYLIN-3656 > Project: Kylin > Issue Type: Improvement >Affects Versions: all >Reporter: Chang chen >Assignee: Chang chen >Priority: Major > Fix For: v2.6.0 > > Attachments: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch > > > The current HLLCounter implementation has some room to improve performance, > as we find in our product environment. > # Create HLLCounter from another HLLCounter, we can copy register instead > of merge > # To compute harmonic mean in the HLLCSnapshot, we could > ## using table to cache all 1/2^r without computing on the fly > ## remove floating addition by using integer addition in the bigger loop > ## remove branch, e.g. needn't checking whether registers[i] is zero or not, > although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3656) Improve HLLCounter performance
[ https://issues.apache.org/jira/browse/KYLIN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang chen updated KYLIN-3656: -- Attachment: image-2018-11-05-18-15-36-463.png > Improve HLLCounter performance > -- > > Key: KYLIN-3656 > URL: https://issues.apache.org/jira/browse/KYLIN-3656 > Project: Kylin > Issue Type: Improvement >Affects Versions: all >Reporter: Chang chen >Assignee: Chang chen >Priority: Major > Fix For: v2.6.0 > > Attachments: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch, > image-2018-11-05-18-15-36-463.png > > > The current HLLCounter implementation has some room to improve performance, > as we find in our product environment. > # Create HLLCounter from another HLLCounter, we can copy register instead > of merge > # To compute harmonic mean in the HLLCSnapshot, we could > ## using table to cache all 1/2^r without computing on the fly > ## remove floating addition by using integer addition in the bigger loop > ## remove branch, e.g. needn't checking whether registers[i] is zero or not, > although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3656) Improve HLLCounter performance
[ https://issues.apache.org/jira/browse/KYLIN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674917#comment-16674917 ] Chang chen commented on KYLIN-3656: --- [~yimingliu] you can try the UT for the result. The following is my Test : !image-2018-11-05-18-15-36-463.png! Removing reflection isn't include in this patch. > Improve HLLCounter performance > -- > > Key: KYLIN-3656 > URL: https://issues.apache.org/jira/browse/KYLIN-3656 > Project: Kylin > Issue Type: Improvement >Affects Versions: all >Reporter: Chang chen >Assignee: Chang chen >Priority: Major > Fix For: v2.6.0 > > Attachments: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch, > image-2018-11-05-18-15-36-463.png > > > The current HLLCounter implementation has some room to improve performance, > as we find in our product environment. > # Create HLLCounter from another HLLCounter, we can copy register instead > of merge > # To compute harmonic mean in the HLLCSnapshot, we could > ## using table to cache all 1/2^r without computing on the fly > ## remove floating addition by using integer addition in the bigger loop > ## remove branch, e.g. needn't checking whether registers[i] is zero or not, > although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
