[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[
https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833729#comment-17833729
]
Hudson commented on HBASE-28458:
Results for branch branch-3
[build #176 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/176/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/176/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/176/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/176/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully
> cached
> ---
>
> Key: HBASE-28458
> URL: https://issues.apache.org/jira/browse/HBASE-28458
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
>
> Noticed that
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was
> flakey, failing whenever the block eviction happened while prefetch was still
> ongoing.
> In the test, we pass an instance of BucketCache directly to the cache config,
> so the test is actually placing both data and meta blocks in the bucket
> cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted
> after the it has already evicted two blocks.
> Inside BucketCache.notifyFileCachingCompleted, we iterate through the
> backingMap entry set, counting number of blocks for the given file. Then, to
> consider whether the file is fully cached or not, we do the following
> validation:
> {noformat}
> if (dataBlockCount == count.getValue() || totalBlockCount ==
> count.getValue()) {
> LOG.debug("File {} has now been fully cached.", fileName);
> fileCacheCompleted(fileName, size);
> } {noformat}
> But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts
> two blocks and asserts that the file hasn't been considered fully cached.
> When these evictions happen while prefetch is still going, we'll pass that
> check, as the the number of blocks for the file in the backingMap would still
> be 55, which is what we pass as dataBlockCount.
> As BucketCache is intended for storing data blocks only, I believe we should
> make sure BucketCache.notifyFileCachingCompleted only accounts for data
> blocks. Also, the
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should
> be updated to consistently reproduce the eviction concurrent to the prefetch.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[
https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833733#comment-17833733
]
Hudson commented on HBASE-28458:
Results for branch master
[build #1042 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1042/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1042/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1042/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1042/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully
> cached
> ---
>
> Key: HBASE-28458
> URL: https://issues.apache.org/jira/browse/HBASE-28458
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
>
> Noticed that
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was
> flakey, failing whenever the block eviction happened while prefetch was still
> ongoing.
> In the test, we pass an instance of BucketCache directly to the cache config,
> so the test is actually placing both data and meta blocks in the bucket
> cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted
> after the it has already evicted two blocks.
> Inside BucketCache.notifyFileCachingCompleted, we iterate through the
> backingMap entry set, counting number of blocks for the given file. Then, to
> consider whether the file is fully cached or not, we do the following
> validation:
> {noformat}
> if (dataBlockCount == count.getValue() || totalBlockCount ==
> count.getValue()) {
> LOG.debug("File {} has now been fully cached.", fileName);
> fileCacheCompleted(fileName, size);
> } {noformat}
> But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts
> two blocks and asserts that the file hasn't been considered fully cached.
> When these evictions happen while prefetch is still going, we'll pass that
> check, as the the number of blocks for the file in the backingMap would still
> be 55, which is what we pass as dataBlockCount.
> As BucketCache is intended for storing data blocks only, I believe we should
> make sure BucketCache.notifyFileCachingCompleted only accounts for data
> blocks. Also, the
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should
> be updated to consistently reproduce the eviction concurrent to the prefetch.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[
https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17834486#comment-17834486
]
Hudson commented on HBASE-28458:
Results for branch branch-2.6
[build #89 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/89/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/89/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/89/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/89/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/89/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully
> cached
> ---
>
> Key: HBASE-28458
> URL: https://issues.apache.org/jira/browse/HBASE-28458
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0, 4.0.0-alpha-1, 2.7.0
>
>
> Noticed that
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was
> flakey, failing whenever the block eviction happened while prefetch was still
> ongoing.
> In the test, we pass an instance of BucketCache directly to the cache config,
> so the test is actually placing both data and meta blocks in the bucket
> cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted
> after the it has already evicted two blocks.
> Inside BucketCache.notifyFileCachingCompleted, we iterate through the
> backingMap entry set, counting number of blocks for the given file. Then, to
> consider whether the file is fully cached or not, we do the following
> validation:
> {noformat}
> if (dataBlockCount == count.getValue() || totalBlockCount ==
> count.getValue()) {
> LOG.debug("File {} has now been fully cached.", fileName);
> fileCacheCompleted(fileName, size);
> } {noformat}
> But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts
> two blocks and asserts that the file hasn't been considered fully cached.
> When these evictions happen while prefetch is still going, we'll pass that
> check, as the the number of blocks for the file in the backingMap would still
> be 55, which is what we pass as dataBlockCount.
> As BucketCache is intended for storing data blocks only, I believe we should
> make sure BucketCache.notifyFileCachingCompleted only accounts for data
> blocks. Also, the
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should
> be updated to consistently reproduce the eviction concurrent to the prefetch.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[
https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17834494#comment-17834494
]
Hudson commented on HBASE-28458:
Results for branch branch-2
[build #1025 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1025/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1025/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1025/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1025/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1025/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully
> cached
> ---
>
> Key: HBASE-28458
> URL: https://issues.apache.org/jira/browse/HBASE-28458
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0, 4.0.0-alpha-1, 2.7.0
>
>
> Noticed that
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was
> flakey, failing whenever the block eviction happened while prefetch was still
> ongoing.
> In the test, we pass an instance of BucketCache directly to the cache config,
> so the test is actually placing both data and meta blocks in the bucket
> cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted
> after the it has already evicted two blocks.
> Inside BucketCache.notifyFileCachingCompleted, we iterate through the
> backingMap entry set, counting number of blocks for the given file. Then, to
> consider whether the file is fully cached or not, we do the following
> validation:
> {noformat}
> if (dataBlockCount == count.getValue() || totalBlockCount ==
> count.getValue()) {
> LOG.debug("File {} has now been fully cached.", fileName);
> fileCacheCompleted(fileName, size);
> } {noformat}
> But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts
> two blocks and asserts that the file hasn't been considered fully cached.
> When these evictions happen while prefetch is still going, we'll pass that
> check, as the the number of blocks for the file in the backingMap would still
> be 55, which is what we pass as dataBlockCount.
> As BucketCache is intended for storing data blocks only, I believe we should
> make sure BucketCache.notifyFileCachingCompleted only accounts for data
> blocks. Also, the
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should
> be updated to consistently reproduce the eviction concurrent to the prefetch.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[
https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17834558#comment-17834558
]
Hudson commented on HBASE-28458:
Results for branch master
[build #1045 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1045/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1045/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1045/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1045/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully
> cached
> ---
>
> Key: HBASE-28458
> URL: https://issues.apache.org/jira/browse/HBASE-28458
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0, 4.0.0-alpha-1, 2.7.0
>
>
> Noticed that
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was
> flakey, failing whenever the block eviction happened while prefetch was still
> ongoing.
> In the test, we pass an instance of BucketCache directly to the cache config,
> so the test is actually placing both data and meta blocks in the bucket
> cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted
> after the it has already evicted two blocks.
> Inside BucketCache.notifyFileCachingCompleted, we iterate through the
> backingMap entry set, counting number of blocks for the given file. Then, to
> consider whether the file is fully cached or not, we do the following
> validation:
> {noformat}
> if (dataBlockCount == count.getValue() || totalBlockCount ==
> count.getValue()) {
> LOG.debug("File {} has now been fully cached.", fileName);
> fileCacheCompleted(fileName, size);
> } {noformat}
> But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts
> two blocks and asserts that the file hasn't been considered fully cached.
> When these evictions happen while prefetch is still going, we'll pass that
> check, as the the number of blocks for the file in the backingMap would still
> be 55, which is what we pass as dataBlockCount.
> As BucketCache is intended for storing data blocks only, I believe we should
> make sure BucketCache.notifyFileCachingCompleted only accounts for data
> blocks. Also, the
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should
> be updated to consistently reproduce the eviction concurrent to the prefetch.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[
https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17834559#comment-17834559
]
Hudson commented on HBASE-28458:
Results for branch branch-3
[build #179 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/179/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/179/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/179/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/179/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully
> cached
> ---
>
> Key: HBASE-28458
> URL: https://issues.apache.org/jira/browse/HBASE-28458
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0, 4.0.0-alpha-1, 2.7.0
>
>
> Noticed that
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was
> flakey, failing whenever the block eviction happened while prefetch was still
> ongoing.
> In the test, we pass an instance of BucketCache directly to the cache config,
> so the test is actually placing both data and meta blocks in the bucket
> cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted
> after the it has already evicted two blocks.
> Inside BucketCache.notifyFileCachingCompleted, we iterate through the
> backingMap entry set, counting number of blocks for the given file. Then, to
> consider whether the file is fully cached or not, we do the following
> validation:
> {noformat}
> if (dataBlockCount == count.getValue() || totalBlockCount ==
> count.getValue()) {
> LOG.debug("File {} has now been fully cached.", fileName);
> fileCacheCompleted(fileName, size);
> } {noformat}
> But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts
> two blocks and asserts that the file hasn't been considered fully cached.
> When these evictions happen while prefetch is still going, we'll pass that
> check, as the the number of blocks for the file in the backingMap would still
> be 55, which is what we pass as dataBlockCount.
> As BucketCache is intended for storing data blocks only, I believe we should
> make sure BucketCache.notifyFileCachingCompleted only accounts for data
> blocks. Also, the
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should
> be updated to consistently reproduce the eviction concurrent to the prefetch.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
