[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[ https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833729#comment-17833729 ] Hudson commented on HBASE-28458: Results for branch branch-3 [build #176 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/176/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/176/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/176/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/176/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully > cached > --- > > Key: HBASE-28458 > URL: https://issues.apache.org/jira/browse/HBASE-28458 > Project: HBase > Issue Type: Bug >Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > > Noticed that > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was > flakey, failing whenever the block eviction happened while prefetch was still > ongoing. > In the test, we pass an instance of BucketCache directly to the cache config, > so the test is actually placing both data and meta blocks in the bucket > cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted > after the it has already evicted two blocks. > Inside BucketCache.notifyFileCachingCompleted, we iterate through the > backingMap entry set, counting number of blocks for the given file. Then, to > consider whether the file is fully cached or not, we do the following > validation: > {noformat} > if (dataBlockCount == count.getValue() || totalBlockCount == > count.getValue()) { > LOG.debug("File {} has now been fully cached.", fileName); > fileCacheCompleted(fileName, size); > } {noformat} > But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts > two blocks and asserts that the file hasn't been considered fully cached. > When these evictions happen while prefetch is still going, we'll pass that > check, as the the number of blocks for the file in the backingMap would still > be 55, which is what we pass as dataBlockCount. > As BucketCache is intended for storing data blocks only, I believe we should > make sure BucketCache.notifyFileCachingCompleted only accounts for data > blocks. Also, the > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should > be updated to consistently reproduce the eviction concurrent to the prefetch. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[ https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833733#comment-17833733 ] Hudson commented on HBASE-28458: Results for branch master [build #1042 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1042/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1042/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1042/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1042/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully > cached > --- > > Key: HBASE-28458 > URL: https://issues.apache.org/jira/browse/HBASE-28458 > Project: HBase > Issue Type: Bug >Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > > Noticed that > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was > flakey, failing whenever the block eviction happened while prefetch was still > ongoing. > In the test, we pass an instance of BucketCache directly to the cache config, > so the test is actually placing both data and meta blocks in the bucket > cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted > after the it has already evicted two blocks. > Inside BucketCache.notifyFileCachingCompleted, we iterate through the > backingMap entry set, counting number of blocks for the given file. Then, to > consider whether the file is fully cached or not, we do the following > validation: > {noformat} > if (dataBlockCount == count.getValue() || totalBlockCount == > count.getValue()) { > LOG.debug("File {} has now been fully cached.", fileName); > fileCacheCompleted(fileName, size); > } {noformat} > But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts > two blocks and asserts that the file hasn't been considered fully cached. > When these evictions happen while prefetch is still going, we'll pass that > check, as the the number of blocks for the file in the backingMap would still > be 55, which is what we pass as dataBlockCount. > As BucketCache is intended for storing data blocks only, I believe we should > make sure BucketCache.notifyFileCachingCompleted only accounts for data > blocks. Also, the > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should > be updated to consistently reproduce the eviction concurrent to the prefetch. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[ https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17834486#comment-17834486 ] Hudson commented on HBASE-28458: Results for branch branch-2.6 [build #89 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/89/]: (x) *{color:red}-1 overall{color}* details (if available): (x) {color:red}-1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/89/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/89/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/89/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/89/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully > cached > --- > > Key: HBASE-28458 > URL: https://issues.apache.org/jira/browse/HBASE-28458 > Project: HBase > Issue Type: Bug >Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0, 4.0.0-alpha-1, 2.7.0 > > > Noticed that > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was > flakey, failing whenever the block eviction happened while prefetch was still > ongoing. > In the test, we pass an instance of BucketCache directly to the cache config, > so the test is actually placing both data and meta blocks in the bucket > cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted > after the it has already evicted two blocks. > Inside BucketCache.notifyFileCachingCompleted, we iterate through the > backingMap entry set, counting number of blocks for the given file. Then, to > consider whether the file is fully cached or not, we do the following > validation: > {noformat} > if (dataBlockCount == count.getValue() || totalBlockCount == > count.getValue()) { > LOG.debug("File {} has now been fully cached.", fileName); > fileCacheCompleted(fileName, size); > } {noformat} > But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts > two blocks and asserts that the file hasn't been considered fully cached. > When these evictions happen while prefetch is still going, we'll pass that > check, as the the number of blocks for the file in the backingMap would still > be 55, which is what we pass as dataBlockCount. > As BucketCache is intended for storing data blocks only, I believe we should > make sure BucketCache.notifyFileCachingCompleted only accounts for data > blocks. Also, the > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should > be updated to consistently reproduce the eviction concurrent to the prefetch. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[ https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17834494#comment-17834494 ] Hudson commented on HBASE-28458: Results for branch branch-2 [build #1025 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1025/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1025/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1025/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1025/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1025/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully > cached > --- > > Key: HBASE-28458 > URL: https://issues.apache.org/jira/browse/HBASE-28458 > Project: HBase > Issue Type: Bug >Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0, 4.0.0-alpha-1, 2.7.0 > > > Noticed that > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was > flakey, failing whenever the block eviction happened while prefetch was still > ongoing. > In the test, we pass an instance of BucketCache directly to the cache config, > so the test is actually placing both data and meta blocks in the bucket > cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted > after the it has already evicted two blocks. > Inside BucketCache.notifyFileCachingCompleted, we iterate through the > backingMap entry set, counting number of blocks for the given file. Then, to > consider whether the file is fully cached or not, we do the following > validation: > {noformat} > if (dataBlockCount == count.getValue() || totalBlockCount == > count.getValue()) { > LOG.debug("File {} has now been fully cached.", fileName); > fileCacheCompleted(fileName, size); > } {noformat} > But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts > two blocks and asserts that the file hasn't been considered fully cached. > When these evictions happen while prefetch is still going, we'll pass that > check, as the the number of blocks for the file in the backingMap would still > be 55, which is what we pass as dataBlockCount. > As BucketCache is intended for storing data blocks only, I believe we should > make sure BucketCache.notifyFileCachingCompleted only accounts for data > blocks. Also, the > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should > be updated to consistently reproduce the eviction concurrent to the prefetch. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[ https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17834558#comment-17834558 ] Hudson commented on HBASE-28458: Results for branch master [build #1045 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1045/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1045/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1045/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1045/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully > cached > --- > > Key: HBASE-28458 > URL: https://issues.apache.org/jira/browse/HBASE-28458 > Project: HBase > Issue Type: Bug >Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0, 4.0.0-alpha-1, 2.7.0 > > > Noticed that > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was > flakey, failing whenever the block eviction happened while prefetch was still > ongoing. > In the test, we pass an instance of BucketCache directly to the cache config, > so the test is actually placing both data and meta blocks in the bucket > cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted > after the it has already evicted two blocks. > Inside BucketCache.notifyFileCachingCompleted, we iterate through the > backingMap entry set, counting number of blocks for the given file. Then, to > consider whether the file is fully cached or not, we do the following > validation: > {noformat} > if (dataBlockCount == count.getValue() || totalBlockCount == > count.getValue()) { > LOG.debug("File {} has now been fully cached.", fileName); > fileCacheCompleted(fileName, size); > } {noformat} > But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts > two blocks and asserts that the file hasn't been considered fully cached. > When these evictions happen while prefetch is still going, we'll pass that > check, as the the number of blocks for the file in the backingMap would still > be 55, which is what we pass as dataBlockCount. > As BucketCache is intended for storing data blocks only, I believe we should > make sure BucketCache.notifyFileCachingCompleted only accounts for data > blocks. Also, the > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should > be updated to consistently reproduce the eviction concurrent to the prefetch. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[ https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17834559#comment-17834559 ] Hudson commented on HBASE-28458: Results for branch branch-3 [build #179 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/179/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/179/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/179/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/179/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully > cached > --- > > Key: HBASE-28458 > URL: https://issues.apache.org/jira/browse/HBASE-28458 > Project: HBase > Issue Type: Bug >Affects Versions: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0, 4.0.0-alpha-1, 2.7.0 > > > Noticed that > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was > flakey, failing whenever the block eviction happened while prefetch was still > ongoing. > In the test, we pass an instance of BucketCache directly to the cache config, > so the test is actually placing both data and meta blocks in the bucket > cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted > after the it has already evicted two blocks. > Inside BucketCache.notifyFileCachingCompleted, we iterate through the > backingMap entry set, counting number of blocks for the given file. Then, to > consider whether the file is fully cached or not, we do the following > validation: > {noformat} > if (dataBlockCount == count.getValue() || totalBlockCount == > count.getValue()) { > LOG.debug("File {} has now been fully cached.", fileName); > fileCacheCompleted(fileName, size); > } {noformat} > But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts > two blocks and asserts that the file hasn't been considered fully cached. > When these evictions happen while prefetch is still going, we'll pass that > check, as the the number of blocks for the file in the backingMap would still > be 55, which is what we pass as dataBlockCount. > As BucketCache is intended for storing data blocks only, I believe we should > make sure BucketCache.notifyFileCachingCompleted only accounts for data > blocks. Also, the > TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should > be updated to consistently reproduce the eviction concurrent to the prefetch. > -- This message was sent by Atlassian Jira (v8.20.10#820010)