[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990422#comment-15990422 ] Hudson commented on HBASE-17872: ABORTED: Integrated in Jenkins build HBase-HBASE-14614 #190 (See [https://builds.apache.org/job/HBase-HBASE-14614/190/]) HBASE-17872 The MSLABImpl generates the invaild cells when unsafe is not (chia7712: rev df96d328fb9fa11f04f84607e9a23f254f513202) * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFromClientSide3WoUnsafe.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch, HBASE-17872.v3.patch, HBASE-17872.v4.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15962246#comment-15962246 ] Hudson commented on HBASE-17872: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2829 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2829/]) HBASE-17872 The MSLABImpl generates the invaild cells when unsafe is not (chia7712: rev df96d328fb9fa11f04f84607e9a23f254f513202) * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFromClientSide3WoUnsafe.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch, HBASE-17872.v3.patch, HBASE-17872.v4.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961887#comment-15961887 ] Chia-Ping Tsai commented on HBASE-17872: bq. It is a pain doing all the reflection code but thank you for taking the pain to do this at test time – and for moving all test baggage to test-time classes – so runtime has no friction. Thanks for the suggestion. [~stack] Will commit it tomorrow if no objection. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch, HBASE-17872.v3.patch, HBASE-17872.v4.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961858#comment-15961858 ] stack commented on HBASE-17872: --- That is a very nice improvement [~chia7712]. +1. It is a pain doing all the reflection code but thank you for taking the pain to do this at test time -- and for moving all test baggage to test-time classes -- so runtime has no friction. I was worried that what was there previous would not always do the right thing. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch, HBASE-17872.v3.patch, HBASE-17872.v4.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961799#comment-15961799 ] Hadoop QA commented on HBASE-17872: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 6s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 25s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 18s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 2s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 29m 10s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 54s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 108m 16s {color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 156m 9s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12862571/HBASE-17872.v4.patch | | JIRA Issue | HBASE-17872 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 0286813ed434 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 59e8b8e | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/6367/testReport/ | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/6367/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > The MSLABImpl generates the invaild
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961642#comment-15961642 ] Chia-Ping Tsai commented on HBASE-17872: bq. Are the test failures related? Seems unlikely. They are unrelated to this jira and they pass locally. bq. On patch, will the change reliably deliver on/off semantic? I can't say for certain. Would you explain more for me? bq. Why we still have test-time convenience methods in the running code The disableUnsafe and detectAvailabilityOfUnsafe are declared package private. Why we move them into test class? Thanks for the more info. [~stack] > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch, HBASE-17872.v3.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961312#comment-15961312 ] stack commented on HBASE-17872: --- Are the test failures related? Seems unlikely. On patch, will the change reliably deliver on/off semantic? I can't say for certain. Why we still have test-time convenience methods in the running code: i.e. disableUnsafe and detectAvailabilityOfUnsafe? > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch, HBASE-17872.v3.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960268#comment-15960268 ] ramkrishna.s.vasudevan commented on HBASE-17872: +1 on v3. Sorry for missing out on the volatile change. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch, HBASE-17872.v3.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960262#comment-15960262 ] Hadoop QA commented on HBASE-17872: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 32s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 51s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 14s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 42s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 4s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 56m 54s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 43s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 184m 20s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 40s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 277m 43s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 | | | hadoop.hbase.client.TestAsyncBalancerAdminApi | | | hadoop.hbase.client.TestAsyncTableAdminApi | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12862402/HBASE-17872.v3.patch | | JIRA Issue | HBASE-17872 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 5d608f7472f8 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh | | git revision | master / 48b2502 | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/6357/artifact/patchprocess/patch-unit-hbase-server.txt | | unit test logs | https://builds.apache.org/j
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959990#comment-15959990 ] Chia-Ping Tsai commented on HBASE-17872: bq. Then it becomes how to change a static final setting at test time The issue is not that complicated. We change the UNSAFE_AVAIL/UNSAFE_UNALIGNED before starting up a mini cluster when testing, so they shouldn't be declared volatile. Please see the v3 patch. Thanks. [~anoop.hbase] [~stack] > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959485#comment-15959485 ] Hadoop QA commented on HBASE-17872: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 32m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 16s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 23s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 22s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 7s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 39s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 58s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 58s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 61m 53s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 18s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 194m 30s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 42s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 329m 16s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 | | | hadoop.hbase.client.TestAsyncTableAdminApi | | Timed out junit tests | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12862277/HBASE-17872.v2.patch | | JIRA Issue | HBASE-17872 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 164342f47bc1 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / d7e3116 | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/6353/artifact/patchprocess/patch-unit-hbase-server.txt | | unit test logs
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959307#comment-15959307 ] Chia-Ping Tsai commented on HBASE-17872: bq. Volatile reads are way more expensive than a local cache read. The UNSAFE_AVAIL check is called frequently. My bad. I overlook the Perf issue. bq. We can not have volatile. copy that. I will fix it tomorrow. Thanks for the feedback. [~stack] and [~anoop.hbase] > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959268#comment-15959268 ] Anoop Sam John commented on HBASE-17872: Thanks Stack.. My bad.. I did not really realize that the boolean is now volatile. Only thought the change is private.. changing my +1. We can not have volatile. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959250#comment-15959250 ] stack commented on HBASE-17872: --- Seems a pity making runtime pay the price of test-time convenience. Volatile reads are way more expensive than a local cache read. The UNSAFE_AVAIL check is called frequently. Make UNSAFE_AVAIL a final static? Then it becomes how to change a static final setting at test time. Reflection? A delegating class that makes use of a subclass of ByteBufferUtils ? Could move the test clutter of detectAvailabilityOfUnsafe and disableUnsafe out of this vital class and into this new test class. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958968#comment-15958968 ] Ted Yu commented on HBASE-17872: +1, pending QA. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958849#comment-15958849 ] Chia-Ping Tsai commented on HBASE-17872: bq. I suggest changing the above method to detectAvailabilityOfUnsafe(). Good point. rename it. bq. Can TestFromClientSide3WoUnsafe be structured in such a way that the two helper methods can be declared package private ? Copy that. I make they in the same package. See v2 patch. Thanks for the feedback. [~yuzhih...@gmail.com] > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch, > HBASE-17872.v2.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958750#comment-15958750 ] Ted Yu commented on HBASE-17872: Can TestFromClientSide3WoUnsafe be structured in such a way that the two helper methods can be declared package private ? > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958713#comment-15958713 ] Ted Yu commented on HBASE-17872: {code} 59public static void tryEnableUnsafe() { {code} Since whether Unsafe is available depends on the return value of a method call, I suggest changing the above method to detectAvailabilityOfUnsafe(). > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958620#comment-15958620 ] Chia-Ping Tsai commented on HBASE-17872: TestZKSecretWatcher pass locally. It is unrelated to this jira. I will commit it tomorrow with @visibileForTesting if no objection. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958598#comment-15958598 ] Hadoop QA commented on HBASE-17872: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 9s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 23s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 24s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 59s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 30m 14s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 52s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 107m 59s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 156m 48s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.security.token.TestZKSecretWatcher | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12862224/HBASE-17872.v1.patch | | JIRA Issue | HBASE-17872 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux ddde29bb616d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 17737b2 | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/6346/artifact/patchprocess/patch-unit-hbase-server.txt | | unit test logs | https://builds.apache.org/job/PreCommit-HBASE-Build/6346/artifact/patchprocess/patch-unit-hbase-server.txt | | Test Results | https://
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958432#comment-15958432 ] Chia-Ping Tsai commented on HBASE-17872: bq. No need to add new patch. Just fix that on ur commit. copy that. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958297#comment-15958297 ] Anoop Sam John commented on HBASE-17872: No need to add new patch. Just fix that on ur commit. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958296#comment-15958296 ] Chia-Ping Tsai commented on HBASE-17872: I will add the @visibileForTesting in next patch. Thanks for the feedback. [~anoop.hbase] and [~ram_krish] > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958292#comment-15958292 ] ramkrishna.s.vasudevan commented on HBASE-17872: Seeing 2nd patch. Looks good. @visibileForTesting tag may be good to be added. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958286#comment-15958286 ] Anoop Sam John commented on HBASE-17872: disableUnsafe -> Pls add @VisibleForTesting. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958281#comment-15958281 ] ramkrishna.s.vasudevan commented on HBASE-17872: +1. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch, HBASE-17872.v1.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958275#comment-15958275 ] Chia-Ping Tsai commented on HBASE-17872: bq. Consider introducing trivial change in hbase-server module to run tests. It is better to run the Test*FromClient without unsafe. But it will introduce a large test and we must expose the BBU#unsafe API. see v1 patch. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957143#comment-15957143 ] Ted Yu commented on HBASE-17872: lgtm Consider introducing trivial change in hbase-server module to run tests. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957001#comment-15957001 ] Chia-Ping Tsai commented on HBASE-17872: Will commit it tomorrow if no objection. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956816#comment-15956816 ] Hadoop QA commented on HBASE-17872: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 59s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 32s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 27m 5s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 7s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 35m 56s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12862068/HBASE-17872.v0.patch | | JIRA Issue | HBASE-17872 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux de453e54346c 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / cbcbcf4 | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/6323/testReport/ | | modules | C: hbase-common U: hbase-common | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/6323/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-1
[jira] [Commented] (HBASE-17872) The MSLABImpl generates the invaild cells when unsafe is not availble
[ https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956781#comment-15956781 ] Anoop Sam John commented on HBASE-17872: LGTM > The MSLABImpl generates the invaild cells when unsafe is not availble > - > > Key: HBASE-17872 > URL: https://issues.apache.org/jira/browse/HBASE-17872 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17872.v0.patch > > > We will get the wrong position of buffer in multithreaded environment, so the > method makes the invalid cell in MSLAB. > {noformat} > public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int > sourceOffset, > int destinationOffset, int length) { > if (in.hasArray() && out.hasArray()) { > // ... > } else if (UNSAFE_AVAIL) { > // ... > } else { > int outOldPos = out.position(); > out.position(destinationOffset); > ByteBuffer inDup = in.duplicate(); > inDup.position(sourceOffset).limit(sourceOffset + length); > out.put(inDup); > out.position(outOldPos); > } > return destinationOffset + length; > } > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)