[ https://issues.apache.org/jira/browse/HBASE-19215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250033#comment-16250033 ]
Hadoop QA commented on HBASE-19215: ----------------------------------- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m 49s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} branch-1 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 29s{color} | {color:green} branch-1 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s{color} | {color:green} branch-1 passed with JDK v1.8.0_141 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} branch-1 passed with JDK v1.7.0_151 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green} branch-1 passed {color} | | {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 2m 41s{color} | {color:red} branch has 12 errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s{color} | {color:green} branch-1 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} branch-1 passed with JDK v1.8.0_141 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} branch-1 passed with JDK v1.7.0_151 {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s{color} | {color:green} the patch passed with JDK v1.8.0_141 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 2m 10s{color} | {color:red} patch has 12 errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 25m 9s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} the patch passed with JDK v1.8.0_141 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 12s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 64m 13s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:4bf71c3 | | JIRA Issue | HBASE-19215 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12897310/HBASE-19215.branch-1.001.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 417c2da71f02 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh | | git revision | branch-1 / da8e615 | | maven | version: Apache Maven 3.0.5 | | Default Java | 1.7.0_151 | | Multi-JDK versions | /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_141 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/9766/testReport/ | | modules | C: hbase-client U: hbase-client | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/9766/console | | Powered by | Apache Yetus 0.6.0 http://yetus.apache.org | This message was automatically generated. > Incorrect exception handling on the client causes incorrect call timeouts and > byte buffer allocations on the server > ------------------------------------------------------------------------------------------------------------------- > > Key: HBASE-19215 > URL: https://issues.apache.org/jira/browse/HBASE-19215 > Project: HBase > Issue Type: Bug > Affects Versions: 1.3.1 > Reporter: Abhishek Singh Chouhan > Assignee: Abhishek Singh Chouhan > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-19215.branch-1.001.patch > > > Ran into the situation of oome on the client : java.lang.OutOfMemoryError: > Direct buffer memory. > When we encounter an unhandled exception during channel write at RpcClientImpl > {noformat} > checkIsOpen(); // Now we're checking that it didn't became idle in between. > try { > call.callStats.setRequestSizeBytes(IPCUtil.write(this.out, header, > call.param, > cellBlock)); > } catch (IOException e) { > {noformat} > we end up leaving the connection open. This becomes especially problematic > when we get an unhandled exception between writing the length of our request > on the channel and subsequently writing the params and cellblocks > {noformat} > *dos.write(Bytes.toBytes(totalSize));* > // This allocates a buffer that is the size of the message internally. > header.writeDelimitedTo(dos); > if (param != null) param.writeDelimitedTo(dos); > if (cellBlock != null) dos.write(cellBlock.array(), 0, > cellBlock.remaining()); > dos.flush(); > return totalSize; > {noformat} > After reading the length rs allocates a bb and expects data to be filled. > However when we encounter an exception during param write we release the > writelock in rpcclientimpl and do not close the connection, the exception is > handled at AbstractRpcClient.callBlockingMethod and retried. Now the next > client request to the same rs writes to the channel however the server > interprets this as part of the previous request and errors out during proto > conversion when processing the request since its considered malformed(in the > worst case this might be misinterpreted as wrong data?). Now the remaining > data of the current request is read(the current request's size > prev > request's allocated partially filled bytebuffer) and is misinterpreted as the > size of new request, in my case this was in gbs. All the client requests time > out since this bytebuffer is never completely filled. We should close the > connection for any Throwable and not just ioexception. -- This message was sent by Atlassian JIRA (v6.4.14#64029)