[jira] [Commented] (HBASE-13926) Close the scanner only after Call#setResponse
[ https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595085#comment-14595085 ] Hudson commented on HBASE-13926: FAILURE: Integrated in HBase-TRUNK #6586 (See [https://builds.apache.org/job/HBase-TRUNK/6586/]) HBASE-13926 Close the scanner only after Call#setResponse. (anoopsamjohn: rev e4d8fab104d6c4edc8721ee14238fcc59bd203d0) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcCallback.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Shipper.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java * hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcCallContext.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java Close the scanner only after Call#setResponse - Key: HBASE-13926 URL: https://issues.apache.org/jira/browse/HBASE-13926 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13926.patch, HBASE-13926_V2.patch, HBASE-13926_V2.patch, HBASE-13926_V3.patch This is for HBASE-12295. We will be delivering cells directly from shared cache memory. HBASE-12295 adds a ref count based prevention mechanism to avoid block eviction, when that memory area in use by scanners. We will decrement ref count at scanner close. The memory area will be in use till a cellblock is created or PB scan reply message is built. So we can delay the close of the scanner until the response is set for the scanner Call. This is done via a RpcCallback way. The callback is set on Call at scan time. Once the setResponse is done, the Call will execute the RpcCallback. This jira also adds a facility to do some cleanup/close during the course of scan. Scan from client makes many RPCs fetching N rows each time. Only at the end the scanner close will happen. We will add a new batchClose() facility with which we can do any cleanup after every rpc call is executed and rows fetched for return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13926) Close the scanner only after Call#setResponse
[
https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594650#comment-14594650
]
Hadoop QA commented on HBASE-13926:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12740797/HBASE-13926_V2.patch
against master branch at commit db08013ebeeaa85802d9795cc72b4c29c5338a47.
ATTACHMENT ID: 12740797
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14482//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14482//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14482//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14482//console
This message is automatically generated.
Close the scanner only after Call#setResponse
-
Key: HBASE-13926
URL: https://issues.apache.org/jira/browse/HBASE-13926
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0
Attachments: HBASE-13926.patch, HBASE-13926_V2.patch,
HBASE-13926_V2.patch
This is for HBASE-12295. We will be delivering cells directly from shared
cache memory. HBASE-12295 adds a ref count based prevention mechanism to
avoid block eviction, when that memory area in use by scanners. We will
decrement ref count at scanner close. The memory area will be in use till a
cellblock is created or PB scan reply message is built. So we can delay the
close of the scanner until the response is set for the scanner Call. This is
done via a RpcCallback way. The callback is set on Call at scan time. Once
the setResponse is done, the Call will execute the RpcCallback.
This jira also adds a facility to do some cleanup/close during the course of
scan. Scan from client makes many RPCs fetching N rows each time. Only at the
end the scanner close will happen. We will add a new batchClose() facility
with which we can do any cleanup after every rpc call is executed and rows
fetched for return.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13926) Close the scanner only after Call#setResponse
[
https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594748#comment-14594748
]
Hadoop QA commented on HBASE-13926:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12740868/HBASE-13926_V3.patch
against master branch at commit 04c25e0f355aaa6ded37b0477ce126a693756b81.
ATTACHMENT ID: 12740868
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.camel.component.jetty.jettyproducer.HttpJettyProducerRecipientListCustomThreadPoolTest.testRecipientList(HttpJettyProducerRecipientListCustomThreadPoolTest.java:40)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14486//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14486//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14486//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14486//console
This message is automatically generated.
Close the scanner only after Call#setResponse
-
Key: HBASE-13926
URL: https://issues.apache.org/jira/browse/HBASE-13926
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0
Attachments: HBASE-13926.patch, HBASE-13926_V2.patch,
HBASE-13926_V2.patch, HBASE-13926_V3.patch
This is for HBASE-12295. We will be delivering cells directly from shared
cache memory. HBASE-12295 adds a ref count based prevention mechanism to
avoid block eviction, when that memory area in use by scanners. We will
decrement ref count at scanner close. The memory area will be in use till a
cellblock is created or PB scan reply message is built. So we can delay the
close of the scanner until the response is set for the scanner Call. This is
done via a RpcCallback way. The callback is set on Call at scan time. Once
the setResponse is done, the Call will execute the RpcCallback.
This jira also adds a facility to do some cleanup/close during the course of
scan. Scan from client makes many RPCs fetching N rows each time. Only at the
end the scanner close will happen. We will add a new batchClose() facility
with which we can do any cleanup after every rpc call is executed and rows
fetched for return.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13926) Close the scanner only after Call#setResponse
[ https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593479#comment-14593479 ] stack commented on HBASE-13926: --- I think its pretty clean now [~anoopsamjohn] Thanks for entertaining my feedback. Close the scanner only after Call#setResponse - Key: HBASE-13926 URL: https://issues.apache.org/jira/browse/HBASE-13926 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13926.patch, HBASE-13926_V2.patch This is for HBASE-12295. We will be delivering cells directly from shared cache memory. HBASE-12295 adds a ref count based prevention mechanism to avoid block eviction, when that memory area in use by scanners. We will decrement ref count at scanner close. The memory area will be in use till a cellblock is created or PB scan reply message is built. So we can delay the close of the scanner until the response is set for the scanner Call. This is done via a RpcCallback way. The callback is set on Call at scan time. Once the setResponse is done, the Call will execute the RpcCallback. This jira also adds a facility to do some cleanup/close during the course of scan. Scan from client makes many RPCs fetching N rows each time. Only at the end the scanner close will happen. We will add a new batchClose() facility with which we can do any cleanup after every rpc call is executed and rows fetched for return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13926) Close the scanner only after Call#setResponse
[ https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593494#comment-14593494 ] Anoop Sam John commented on HBASE-13926: What happened with QA builds? Its been hours now that I have attached a patch. What we need to do? Close the scanner only after Call#setResponse - Key: HBASE-13926 URL: https://issues.apache.org/jira/browse/HBASE-13926 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13926.patch, HBASE-13926_V2.patch This is for HBASE-12295. We will be delivering cells directly from shared cache memory. HBASE-12295 adds a ref count based prevention mechanism to avoid block eviction, when that memory area in use by scanners. We will decrement ref count at scanner close. The memory area will be in use till a cellblock is created or PB scan reply message is built. So we can delay the close of the scanner until the response is set for the scanner Call. This is done via a RpcCallback way. The callback is set on Call at scan time. Once the setResponse is done, the Call will execute the RpcCallback. This jira also adds a facility to do some cleanup/close during the course of scan. Scan from client makes many RPCs fetching N rows each time. Only at the end the scanner close will happen. We will add a new batchClose() facility with which we can do any cleanup after every rpc call is executed and rows fetched for return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13926) Close the scanner only after Call#setResponse
[ https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593645#comment-14593645 ] stack commented on HBASE-13926: --- Dunno. Says Jenkins is going to shut down on home page of jenkins. Close the scanner only after Call#setResponse - Key: HBASE-13926 URL: https://issues.apache.org/jira/browse/HBASE-13926 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13926.patch, HBASE-13926_V2.patch This is for HBASE-12295. We will be delivering cells directly from shared cache memory. HBASE-12295 adds a ref count based prevention mechanism to avoid block eviction, when that memory area in use by scanners. We will decrement ref count at scanner close. The memory area will be in use till a cellblock is created or PB scan reply message is built. So we can delay the close of the scanner until the response is set for the scanner Call. This is done via a RpcCallback way. The callback is set on Call at scan time. Once the setResponse is done, the Call will execute the RpcCallback. This jira also adds a facility to do some cleanup/close during the course of scan. Scan from client makes many RPCs fetching N rows each time. Only at the end the scanner close will happen. We will add a new batchClose() facility with which we can do any cleanup after every rpc call is executed and rows fetched for return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13926) Close the scanner only after Call#setResponse
[ https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1459#comment-1459 ] stack commented on HBASE-13926: --- Patch LGTM So less confusion, add to batchClose in the interface something like: Calls to batchClose do NOT close the scanner. It is a signal to the scanner that it is safe to release resources creating recent results. On BatchCloseable, anything with 'close' in it seems so final and liable to confuse as to what is going on What about a shipping or delivered concept? So Interface would be Shipped and method would be 'shipped' or 'Delivered' and 'delivered'? Close the scanner only after Call#setResponse - Key: HBASE-13926 URL: https://issues.apache.org/jira/browse/HBASE-13926 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13926.patch This is for HBASE-12295. We will be delivering cells directly from shared cache memory. HBASE-12295 adds a ref count based prevention mechanism to avoid block eviction, when that memory area in use by scanners. We will decrement ref count at scanner close. The memory area will be in use till a cellblock is created or PB scan reply message is built. So we can delay the close of the scanner until the response is set for the scanner Call. This is done via a RpcCallback way. The callback is set on Call at scan time. Once the setResponse is done, the Call will execute the RpcCallback. This jira also adds a facility to do some cleanup/close during the course of scan. Scan from client makes many RPCs fetching N rows each time. Only at the end the scanner close will happen. We will add a new batchClose() facility with which we can do any cleanup after every rpc call is executed and rows fetched for return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13926) Close the scanner only after Call#setResponse
[
https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590524#comment-14590524
]
Hadoop QA commented on HBASE-13926:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12740150/HBASE-13926.patch
against master branch at commit 623fd63827b2953c150597f24c7205737119bebe.
ATTACHMENT ID: 12740150
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.camel.component.jetty.jettyproducer.HttpJettyProducerRecipientListCustomThreadPoolTest.testRecipientList(HttpJettyProducerRecipientListCustomThreadPoolTest.java:40)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14446//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14446//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14446//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14446//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14446//console
This message is automatically generated.
Close the scanner only after Call#setResponse
-
Key: HBASE-13926
URL: https://issues.apache.org/jira/browse/HBASE-13926
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0
Attachments: HBASE-13926.patch
This is for HBASE-12295. We will be delivering cells directly from shared
cache memory. HBASE-12295 adds a ref count based prevention mechanism to
avoid block eviction, when that memory area in use by scanners. We will
decrement ref count at scanner close. The memory area will be in use till a
cellblock is created or PB scan reply message is built. So we can delay the
close of the scanner until the response is set for the scanner Call. This is
done via a RpcCallback way. The callback is set on Call at scan time. Once
the setResponse is done, the Call will execute the RpcCallback.
This jira also adds a facility to do some cleanup/close during the course of
scan. Scan from client makes many RPCs fetching N rows each time. Only at the
end the scanner close will happen. We will add a new batchClose() facility
with which we can do any cleanup after every rpc call is executed and rows
fetched for return.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13926) Close the scanner only after Call#setResponse
[ https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591214#comment-14591214 ] ramkrishna.s.vasudevan commented on HBASE-13926: Implement the noop batchClose() in NonLazyKeyValueScanner and remove from DefaultMemstore? Close the scanner only after Call#setResponse - Key: HBASE-13926 URL: https://issues.apache.org/jira/browse/HBASE-13926 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13926.patch This is for HBASE-12295. We will be delivering cells directly from shared cache memory. HBASE-12295 adds a ref count based prevention mechanism to avoid block eviction, when that memory area in use by scanners. We will decrement ref count at scanner close. The memory area will be in use till a cellblock is created or PB scan reply message is built. So we can delay the close of the scanner until the response is set for the scanner Call. This is done via a RpcCallback way. The callback is set on Call at scan time. Once the setResponse is done, the Call will execute the RpcCallback. This jira also adds a facility to do some cleanup/close during the course of scan. Scan from client makes many RPCs fetching N rows each time. Only at the end the scanner close will happen. We will add a new batchClose() facility with which we can do any cleanup after every rpc call is executed and rows fetched for return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13926) Close the scanner only after Call#setResponse
[ https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591234#comment-14591234 ] Anoop Sam John commented on HBASE-13926: Sure Ram. Thanks. Close the scanner only after Call#setResponse - Key: HBASE-13926 URL: https://issues.apache.org/jira/browse/HBASE-13926 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13926.patch This is for HBASE-12295. We will be delivering cells directly from shared cache memory. HBASE-12295 adds a ref count based prevention mechanism to avoid block eviction, when that memory area in use by scanners. We will decrement ref count at scanner close. The memory area will be in use till a cellblock is created or PB scan reply message is built. So we can delay the close of the scanner until the response is set for the scanner Call. This is done via a RpcCallback way. The callback is set on Call at scan time. Once the setResponse is done, the Call will execute the RpcCallback. This jira also adds a facility to do some cleanup/close during the course of scan. Scan from client makes many RPCs fetching N rows each time. Only at the end the scanner close will happen. We will add a new batchClose() facility with which we can do any cleanup after every rpc call is executed and rows fetched for return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
