[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035956#comment-15035956
]
Heng Chen commented on HBASE-14895:
---
It seems we could add some code in StoreScanner.updateReaders, right?
{code: title=StoreScanner.updateReaders}
this.heap = null;
+StoreFile recentFlush = getRecentFlushStoreFile();
+if (recentFlush != null) {
+ seek(recentFlush.getFirstKey());
+}
{code}
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15036091#comment-15036091 ] ramkrishna.s.vasudevan commented on HBASE-14895: Yes something like that only. We need to clearly identify the latest files and just seek to that. With HBASE-13082 in place we just need to do for flushes only. > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037292#comment-15037292 ] Heng Chen commented on HBASE-14895: --- I see.. It seems StoreScanner has a great change in HBASE-13082 > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037735#comment-15037735 ] ramkrishna.s.vasudevan commented on HBASE-14895: I have patch ready for this. Once HBASE-13082 is checked in will rebase the patch on top of that. Found some interesting things while doing this wrt to the new shipped() call that we make. > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041527#comment-15041527 ] Anoop Sam John commented on HBASE-14895: After HBASE-13082, we will notify ChangedReadersObserver only after a flush operation. After bulk load and compaction we dont need inform the readers. After a flush, which results in exactly one file, the notify mechanism can pass that file's detail also? So on that we can do the reset. We will remove the scanner on the snapshot also (?).. So this can be achieved with lesser code change? Rather than maintain new Lists and all? > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041680#comment-15041680
]
Hadoop QA commented on HBASE-14895:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12775763/HBASE-14895.patch
against master branch at commit 8b3d1f144408e4a7a014c5ac46418c9e91b9b0db.
ATTACHMENT ID: 12775763
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16766//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16766//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16766//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16766//console
This message is automatically generated.
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14895.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044397#comment-15044397 ] ramkrishna.s.vasudevan commented on HBASE-14895: bq.After bulk load and compaction we dont need inform the readers. After a flush, which results in exactly one file, the notify mechanism can pass that file's detail also? True. Doing that way one thing that we can avoid is that though we notify the readers on flush only still when a new scanner is created and that time if a file was already comapcted the new heap will have the flushed file and the older file that was compacted will be removed and we will have the new comapcted file in the heap. Hence we need the above logic to determine the exact new files so that we can seek on all of them. Passing the file name on flush is a good suggestion where we will not end up in a situation where there is a new file in the resetted heap that has come out of compaction or bulk load. > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044451#comment-15044451
]
Heng Chen commented on HBASE-14895:
---
{code}
-List scanners = getScannersNoCompaction();
+final boolean isCompaction = false;
+boolean usePread = get || scanUsePread;
+List scanners = store.getScanners(cacheBlocks, get,
usePread, isCompaction,
+ matcher, scan.getStartRow(), scan.getStopRow(), this.readPt);
+scanners = updatedStoreFiles(scanners, store.getComparator());
+scanners = selectScannersFrom(scanners);
{code}
Should we move selectScannersFrom out from getScannersNoCompaction, and we
could use getScannersNoCompaction directly here?
{code}
+ protected Set existingStoreFileInfos = new
HashSet();
{code}
What it is used for? It seems there is no use on it?
{code}
* @param seekKey
* @param isLazy true if using lazy seek
* @param isParallelSeek true if using parallel seek
+ * @param updatedStoreFile
* @throws IOException
*/
{code}
I did not see param updatedStoreFile in seekScanners.
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14895.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044452#comment-15044452 ] Anoop Sam John commented on HBASE-14895: Ya adding the new compacted file into heap and seek to that is a real unwanted op. Bulk load case it would have been an issue. We don't want to include in between bulk loaded files into our scanners. But a flush in middle might have caused an inconsistency wrt behavior. So we need change. You will do a new patch Ram? I believe that patch will be much smaller in size also. :-) > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044462#comment-15044462 ] ramkrishna.s.vasudevan commented on HBASE-14895: bq.We don't want to include in between bulk loaded files into our scanners. But a flush in middle might have caused an inconsistency wrt behavior. So we need change Ya sometimes it will create issues because bulk loaded files will get included after a flush and some times not. I was working on the patch before HBASE-13082 and so all those were needed. Ya will update the patch and thanks for the good point. > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15044922#comment-15044922
]
Hadoop QA commented on HBASE-14895:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12776069/HBASE-14895_1.patch
against master branch at commit 8bf70144e40650ef972f005e2465bd0e2a087c40.
ATTACHMENT ID: 12776069
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16781//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16781//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16781//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16781//console
This message is automatically generated.
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14895.patch, HBASE-14895_1.patch,
> HBASE-14895_1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046742#comment-15046742 ] ramkrishna.s.vasudevan commented on HBASE-14895: Ping for reviews. Know the reason for the checkstyle complaint - one unused import. Will remove it in next patch after reviews. > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch, HBASE-14895_1.patch, > HBASE-14895_1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15047098#comment-15047098
]
Ted Yu commented on HBASE-14895:
Tests were not actually run:
{code}
Printing hanging tests
Hanging test : org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
{code}
Publishing on reviewboard would facilitate review.
Thanks
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14895.patch, HBASE-14895_1.patch,
> HBASE-14895_1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15047168#comment-15047168 ] ramkrishna.s.vasudevan commented on HBASE-14895: https://reviews.apache.org/r/41093/ - RB link. > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch, HBASE-14895_1.patch, > HBASE-14895_1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1504#comment-1504
]
Hadoop QA commented on HBASE-14895:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12776530/HBASE-14895_2.patch
against master branch at commit 7bfbb6a3c9af4b0e2853b5ea2580a05bb471211b.
ATTACHMENT ID: 12776530
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16809//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16809//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16809//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16809//console
This message is automatically generated.
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14895.patch, HBASE-14895_1.patch,
> HBASE-14895_1.patch, HBASE-14895_2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052479#comment-15052479
]
Anoop Sam John commented on HBASE-14895:
one issue found wrt maintain the scanners in a new List in StoreScanner (These
are memstore scanner and store file scanners at the begin of the scan)
Over the scan time, some of the scanners would have reached its end and we
would have eventually closed them off. And the PriorityQ in KVHeap would have
removed them as well. But we are still holding those refs in this new List in
Store Scanner.
Now after a flush we will use this new List to recreate new Heap. We will
remove old MemstoreScaner and add new MemstoreScanner as well as scanner to the
new flushed file. And use all of them to create new KVHeap. (This might include
already closed StoreFileScanners as well).
In KVHeap constructor
{code}
KeyValueHeap(List scanners,
KVScannerComparator comparator) throws IOException {
this.comparator = comparator;
if (!scanners.isEmpty()) {
this.heap = new PriorityQueue(scanners.size(),
this.comparator);
for (KeyValueScanner scanner : scanners) {
if (scanner.peek() != null) {
this.heap.add(scanner);
} else {
this.scannersForDelayedClose.add(scanner);
}
}
this.current = pollRealKV();
}
}
{code}
So the closed scanners wil have peek Cell as null we wont add them to heap.
Good. But we will add them to delayed close and after one shipped call we will
try close them again
StoreFileScanner#close()
{code}
public void close() {
cur = null;
this.hfs.close();
if (this.reader != null) {
this.reader.decrementRefCount();
}
}
{code}
No impact because of hfs close. But the decr read ref count will be a problem
and can cause issue.
In a simple way we can we can have closed flag in StoreFileScanner which can be
used to check when close call on second time or close call nullifying the
reader and hfs on StoreFileScanner.
Or else if we dont need to maintain a new List of scanners in StoreScanner and
make use of the current scanners in the KVHeap for recreating the new heap,
that would have looked better. No need to maintain same scanners in 2 data
structures.
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14895.patch, HBASE-14895_1.patch,
> HBASE-14895_1.patch, HBASE-14895_2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052489#comment-15052489
]
ramkrishna.s.vasudevan commented on HBASE-14895:
Without maintaining two sepreate lists it would not be possible to reset
theheap with a StorefileScanner that just got removed from the heap and that is
the 'current' scanner and then when a reset happens we cannot add it back to
the heap since it is already removed from the current heap so iterating that
would not give you the 'current' scanner.
So maintaining two DS will help to avoid this problem and hence went with 2 DS.
{code}
public void close() {
cur = null;
this.hfs.close();
if (this.reader != null) {
this.reader.decrementRefCount();
}
}
{code}
This problem of close() can happen in cases when the same completed scanner is
doing a close() once again. I can update the patch with that change. Good point
and nice catch.
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14895.patch, HBASE-14895_1.patch,
> HBASE-14895_1.patch, HBASE-14895_2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052562#comment-15052562 ] Anoop Sam John commented on HBASE-14895: +1 once QA passes. > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch, HBASE-14895_1.patch, > HBASE-14895_1.patch, HBASE-14895_2.patch, HBASE-14895_3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053464#comment-15053464
]
Hadoop QA commented on HBASE-14895:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12777006/HBASE-14895_3.patch
against master branch at commit 22b95aebcd7fc742412ab514520008fda5e327de.
ATTACHMENT ID: 12777006
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16832//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16832//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16832//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16832//console
This message is automatically generated.
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14895.patch, HBASE-14895_1.patch,
> HBASE-14895_1.patch, HBASE-14895_2.patch, HBASE-14895_3.patch,
> HBASE-14895_3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[
https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053776#comment-15053776
]
Hadoop QA commented on HBASE-14895:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12777077/HBASE-14895_3.patch
against master branch at commit 22b95aebcd7fc742412ab514520008fda5e327de.
ATTACHMENT ID: 12777077
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16837//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16837//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16837//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16837//console
This message is automatically generated.
> Seek only to the newly flushed file on scanner reset on flush
> -
>
> Key: HBASE-14895
> URL: https://issues.apache.org/jira/browse/HBASE-14895
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14895.patch, HBASE-14895_1.patch,
> HBASE-14895_1.patch, HBASE-14895_2.patch, HBASE-14895_3.patch,
> HBASE-14895_3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055896#comment-15055896 ] Hudson commented on HBASE-14895: FAILURE: Integrated in HBase-Trunk_matrix #549 (See [https://builds.apache.org/job/HBase-Trunk_matrix/549/]) HBASE-14895 Seek only to the newly flushed file on scanner reset on (ramkrishna: rev 555d9b70bd650a0df0ed9e382de449c337274493) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestBlockEvictionFromClient.java > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch, HBASE-14895_1.patch, > HBASE-14895_1.patch, HBASE-14895_2.patch, HBASE-14895_3.patch, > HBASE-14895_3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056273#comment-15056273 ] stack commented on HBASE-14895: --- You need review on the addendum [~ram_krish] ? Why you need to remove the flush check? > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch, HBASE-14895_1.patch, > HBASE-14895_1.patch, HBASE-14895_2.patch, HBASE-14895_3.patch, > HBASE-14895_3.patch, HBASE-14895_addendum.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056270#comment-15056270 ] stack commented on HBASE-14895: --- bq. saw in this in the code today and interestingly the same thing failed in the build today after the commit. https://builds.apache.org/job/HBase-Trunk_matrix/549/jdk=latest1.8,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.regionserver/TestHRegion/testFlushCacheWhileScanning/ Hurray for a CI that catches issues!!! > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch, HBASE-14895_1.patch, > HBASE-14895_1.patch, HBASE-14895_2.patch, HBASE-14895_3.patch, > HBASE-14895_3.patch, HBASE-14895_addendum.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056401#comment-15056401 ] ramkrishna.s.vasudevan commented on HBASE-14895: After this new patch we don't reset the heap to null when a flush happens. The only time we do is when there is close that happens when the heap.peek() does not return any element. That is what has happened here. The next() call did not yield any cell so the heap was closed and so the heap is null and when we try to do a peek on that heap we got the NPE. > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch, HBASE-14895_1.patch, > HBASE-14895_1.patch, HBASE-14895_2.patch, HBASE-14895_3.patch, > HBASE-14895_3.patch, HBASE-14895_addendum.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057313#comment-15057313 ] ramkrishna.s.vasudevan commented on HBASE-14895: Ping [~anoopsamjohn] and [~stack] for reviews before commit. > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch, HBASE-14895_1.patch, > HBASE-14895_1.patch, HBASE-14895_2.patch, HBASE-14895_3.patch, > HBASE-14895_3.patch, HBASE-14895_addendum.patch, HBASE-14895_addendum_1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057329#comment-15057329 ] Anoop Sam John commented on HBASE-14895: +1 Any other similar mistakes we have still? Will do one more read on the code in StoreScanner. Good to see tests reveal this. :-) > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch, HBASE-14895_1.patch, > HBASE-14895_1.patch, HBASE-14895_2.patch, HBASE-14895_3.patch, > HBASE-14895_3.patch, HBASE-14895_addendum.patch, HBASE-14895_addendum_1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14895) Seek only to the newly flushed file on scanner reset on flush
[ https://issues.apache.org/jira/browse/HBASE-14895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058015#comment-15058015 ] Hudson commented on HBASE-14895: FAILURE: Integrated in HBase-Trunk_matrix #554 (See [https://builds.apache.org/job/HBase-Trunk_matrix/554/]) HBASE-14895 Seek only to the newly flushed file on scanner reset on (ramkrishna: rev 1822bb5d9578e3e650115a220132ab85e7db732a) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java > Seek only to the newly flushed file on scanner reset on flush > - > > Key: HBASE-14895 > URL: https://issues.apache.org/jira/browse/HBASE-14895 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14895.patch, HBASE-14895_1.patch, > HBASE-14895_1.patch, HBASE-14895_2.patch, HBASE-14895_3.patch, > HBASE-14895_3.patch, HBASE-14895_addendum.patch, HBASE-14895_addendum_1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
