[jira] [Updated] (HBASE-17278) [C++] Cell Scanner Implementation to be used by ResultScanner
[ https://issues.apache.org/jira/browse/HBASE-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudeep Sunthankar updated HBASE-17278: -- Attachment: HBASE-17278.HBASE-14850.v4.patch This patch addresses the points based on the last feedback. 1) Unnecessary methods removed from CellScanner. 2) Added Decoder() method to KeyValueCodec() which will enable decoding of cell block through base class methods of Advance() and Current() Thanks > [C++] Cell Scanner Implementation to be used by ResultScanner > - > > Key: HBASE-17278 > URL: https://issues.apache.org/jira/browse/HBASE-17278 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Attachments: HBASE-17278.HBASE-14850.v1.patch, > HBASE-17278.HBASE-14850.v2.patch, HBASE-17278.HBASE-14850.v3.patch, > HBASE-17278.HBASE-14850.v4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834019#comment-15834019
]
Hadoop QA commented on HBASE-17471:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 9s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 104m 26s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 152m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.wal.TestSecureWAL |
| | hadoop.hbase.coprocessor.TestWALObserver |
| | hadoop.hbase.wal.TestWALFactory |
| | hadoop.hbase.regionserver.TestWALLockup |
| Timed out junit tests | org.apache.hadoop.hbase.regionserver.wal.TestFSHLog |
| | org.apache.hadoop.hbase.regionserver.wal.TestAsyncFSWAL |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848831/HBASE-17471-duo-v1.patch
|
| JIRA Issue | HBASE-17471 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 43721d7fe479 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7754a96 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5400/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5400/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5400/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit
[jira] [Updated] (HBASE-17500) Implement getTable/creatTable/deleteTable/truncateTable methods
[ https://issues.apache.org/jira/browse/HBASE-17500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17500: --- Status: Patch Available (was: Open) > Implement getTable/creatTable/deleteTable/truncateTable methods > --- > > Key: HBASE-17500 > URL: https://issues.apache.org/jira/browse/HBASE-17500 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17500-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17462) Use sliding window for read/write request costs in StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834015#comment-15834015 ] Hudson commented on HBASE-17462: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2369 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2369/]) HBASE-17462 Use sliding window for read/write request costs in (tedyu: rev 7754a9620eff44d1d570fda534f9159a756310cd) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java > Use sliding window for read/write request costs in StochasticLoadBalancer > - > > Key: HBASE-17462 > URL: https://issues.apache.org/jira/browse/HBASE-17462 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Tim Brown > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: after_changes.png, before_changes.png, > HBASE-17462.patch, HBASE-17462-v2.patch > > > In the thread, http://search-hadoop.com/m/HBase/YGbbyUZKXWALkX1, Timothy was > asking whether the read/write request costs in StochasticLoadBalancer should > be calculated as rates. > This makes sense since read / write load on region server tends to fluctuate > over time. Using sliding window would reflect more recent trend in read / > write load. > Some factors to consider: > The data structure used by StochasticLoadBalancer should be concise. The > number of regions in a cluster can be expected to approach 1 million. We > cannot afford to store long history of read / write requests in master. > Efficiency of cost calculation should be high - there're many cost > functions the balancer goes through, it is expected for each cost function > to return quickly. Otherwise we would not come up with proper region > movement plan(s) in time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17500) Implement getTable/creatTable/deleteTable/truncateTable methods
[ https://issues.apache.org/jira/browse/HBASE-17500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17500: --- Attachment: HBASE-17500-v1.patch Add a initial version. Create/delete/truncate table will use Procedure v2. And the ut was copied from TestAdmin1. > Implement getTable/creatTable/deleteTable/truncateTable methods > --- > > Key: HBASE-17500 > URL: https://issues.apache.org/jira/browse/HBASE-17500 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17500-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[
https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834012#comment-15834012
]
Jingcheng Du commented on HBASE-16981:
--
Thanks [~huaxiang].
Have you updated the patch to address Ted's and Esteban's comments in RB?
{code}
public void updateLatestDate(final String latestDate) {
if ((this.latestDate == null) || (this.latestDate.compareTo(latestDate) <
0)) {
this.latestDate = latestDate;
}
}
{code}
The latestDate has been initialized as empty string. Do we need to check the
null?
Overall I am +1 on the patch if the comments from others are addressed.
Hi [~anoop.hbase], do you want to take a look at the patch? Thanks.
> Expand Mob Compaction Partition policy from daily to weekly, monthly and
> beyond
> ---
>
> Key: HBASE-16981
> URL: https://issues.apache.org/jira/browse/HBASE-16981
> Project: HBase
> Issue Type: New Feature
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-16981.master.001.patch,
> HBASE-16981.master.002.patch, HBASE-16981.master.003.patch,
> HBASE-16981.master.004.patch,
> Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf
>
>
> Today the mob region holds all mob files for all regions. With daily
> partition mob compaction policy, after major mob compaction, there is still
> one file per region daily. Given there is 365 days in one year, at least 365
> files per region. Since HDFS has limitation for number of files under one
> folder, this is not going to scale if there are lots of regions. To reduce
> mob file number, we want to introduce other partition policies such as
> weekly, monthly to compact mob files within one week or month into one file.
> This jira is create to track this effort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17367) Make HTable#getBufferedMutator thread safe
[ https://issues.apache.org/jira/browse/HBASE-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17367: -- Attachment: HBASE-17367.v2.patch Rebase patch after HBASE-17491 goes in and now we could have a thread safe HTable. > Make HTable#getBufferedMutator thread safe > -- > > Key: HBASE-17367 > URL: https://issues.apache.org/jira/browse/HBASE-17367 > Project: HBase > Issue Type: Sub-task >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-17367.patch, HBASE-17367.v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-17491) Remove all setters from HTable interface and introduce a TableBuilder to build Table instance
[
https://issues.apache.org/jira/browse/HBASE-17491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17491:
--
Comment: was deleted
(was: | (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-17491 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848834/HBASE-17491.v8.patch |
| JIRA Issue | HBASE-17491 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5402/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
)
> Remove all setters from HTable interface and introduce a TableBuilder to
> build Table instance
> -
>
> Key: HBASE-17491
> URL: https://issues.apache.org/jira/browse/HBASE-17491
> Project: HBase
> Issue Type: Sub-task
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17491.patch, HBASE-17491.v2.patch,
> HBASE-17491.v3.patch, HBASE-17491.v4.patch, HBASE-17491.v5.patch,
> HBASE-17491.v6.patch, HBASE-17491.v7.patch, HBASE-17491.v8.patch
>
>
> As titled, we will remove all setters in HTable for master branch and
> deprecate them for branch-1 to make HTable thread-safe. And a new
> {{TableBuilder}} interface will be introduced to build Table instance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17491) Remove all setters from HTable interface and introduce a TableBuilder to build Table instance
[
https://issues.apache.org/jira/browse/HBASE-17491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833961#comment-15833961
]
Hadoop QA commented on HBASE-17491:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-17491 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848834/HBASE-17491.v8.patch |
| JIRA Issue | HBASE-17491 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5402/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Remove all setters from HTable interface and introduce a TableBuilder to
> build Table instance
> -
>
> Key: HBASE-17491
> URL: https://issues.apache.org/jira/browse/HBASE-17491
> Project: HBase
> Issue Type: Sub-task
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17491.patch, HBASE-17491.v2.patch,
> HBASE-17491.v3.patch, HBASE-17491.v4.patch, HBASE-17491.v5.patch,
> HBASE-17491.v6.patch, HBASE-17491.v7.patch, HBASE-17491.v8.patch
>
>
> As titled, we will remove all setters in HTable for master branch and
> deprecate them for branch-1 to make HTable thread-safe. And a new
> {{TableBuilder}} interface will be introduced to build Table instance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17491) Remove all setters from HTable interface and introduce a TableBuilder to build Table instance
[
https://issues.apache.org/jira/browse/HBASE-17491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833943#comment-15833943
]
Yu Li commented on HBASE-17491:
---
Pushed to master branch. Thanks all for review [~stack] [~Apache9] [~enis]
[~tedyu]
Left this JIRA open for a while to see whether any more comments, will close
tomorrow if nothing else to do.
> Remove all setters from HTable interface and introduce a TableBuilder to
> build Table instance
> -
>
> Key: HBASE-17491
> URL: https://issues.apache.org/jira/browse/HBASE-17491
> Project: HBase
> Issue Type: Sub-task
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17491.patch, HBASE-17491.v2.patch,
> HBASE-17491.v3.patch, HBASE-17491.v4.patch, HBASE-17491.v5.patch,
> HBASE-17491.v6.patch, HBASE-17491.v7.patch, HBASE-17491.v8.patch
>
>
> As titled, we will remove all setters in HTable for master branch and
> deprecate them for branch-1 to make HTable thread-safe. And a new
> {{TableBuilder}} interface will be introduced to build Table instance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833941#comment-15833941
]
Hadoop QA commented on HBASE-17471:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 50s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 20s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.wal.TestSecureAsyncWALReplay |
| | hadoop.hbase.wal.TestFSHLogProvider |
| | hadoop.hbase.regionserver.wal.TestAsyncWALReplay |
| | hadoop.hbase.wal.TestWALFactory |
| | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| | hadoop.hbase.mapreduce.TestWALRecordReader |
| | hadoop.hbase.regionserver.wal.TestAsyncLogRollPeriod |
| | hadoop.hbase.regionserver.TestWalAndCompactingMemStoreFlush |
| | hadoop.hbase.regionserver.wal.TestAsyncLogRolling |
| | hadoop.hbase.master.TestGetLastFlushedSequenceId |
| | hadoop.hbase.regionserver.wal.TestLogRolling |
| | hadoop.hbase.mapreduce.TestWALPlayer |
| | hadoop.hbase.wal.TestBoundedRegionGroupingStrategy |
| | hadoop.hbase.regionserver.wal.TestAsyncWALReplayCompressed |
| | hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager |
| Timed out junit tests | org.apache.hadoop.hbase.regionserver.wal.TestFSHLog |
| |
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
|
| | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss |
| | org.apache.hadoop.hbase.regionserver.wal.TestAsyncFSWAL |
| | org.apache.hadoop.hbase.TestAcidGuarantees |
| | org.apache.hadoop.hbase.replication.TestSerialReplication |
| | org.apache.hadoop.hbase.regionserver.wal.TestDurability |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:y
[jira] [Updated] (HBASE-17491) Remove all setters from HTable interface and introduce a TableBuilder to build Table instance
[
https://issues.apache.org/jira/browse/HBASE-17491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17491:
--
Attachment: HBASE-17491.v8.patch
Ok, will commit with "since 2.0.0" added in javadoc first, as shown in v8
patch. We can make addendum if needed.
Will commit soon if no objections.
> Remove all setters from HTable interface and introduce a TableBuilder to
> build Table instance
> -
>
> Key: HBASE-17491
> URL: https://issues.apache.org/jira/browse/HBASE-17491
> Project: HBase
> Issue Type: Sub-task
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17491.patch, HBASE-17491.v2.patch,
> HBASE-17491.v3.patch, HBASE-17491.v4.patch, HBASE-17491.v5.patch,
> HBASE-17491.v6.patch, HBASE-17491.v7.patch, HBASE-17491.v8.patch
>
>
> As titled, we will remove all setters in HTable for master branch and
> deprecate them for branch-1 to make HTable thread-safe. And a new
> {{TableBuilder}} interface will be introduced to build Table instance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17067) Procedure v2 - remove zklock/tryLock and use wait/wake
[ https://issues.apache.org/jira/browse/HBASE-17067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833938#comment-15833938 ] stack commented on HBASE-17067: --- .005 addresses @appy review over in rb. > Procedure v2 - remove zklock/tryLock and use wait/wake > -- > > Key: HBASE-17067 > URL: https://issues.apache.org/jira/browse/HBASE-17067 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-17067.master.001.patch, > HBASE-17067.master.002.patch, HBASE-17067.master.003.patch, > HBASE-17067.master.004.patch, HBASE-17067.master.004.patch, > HBASE-17067.master.004.patch, HBASE-17067.master.004.patch, > HBASE-17067.master.004.patch, HBASE-17067.master.004.patch, > HBASE-17067.master.005.patch > > > Once we have HBASE-16744, HBASE-16786, HBASE-16831. we can remove the > tryLock() methods and replace them with the wait/wake methods that are using > the framework events instead of spinning until we can start the proc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17067) Procedure v2 - remove zklock/tryLock and use wait/wake
[ https://issues.apache.org/jira/browse/HBASE-17067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17067: -- Attachment: HBASE-17067.master.005.patch > Procedure v2 - remove zklock/tryLock and use wait/wake > -- > > Key: HBASE-17067 > URL: https://issues.apache.org/jira/browse/HBASE-17067 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-17067.master.001.patch, > HBASE-17067.master.002.patch, HBASE-17067.master.003.patch, > HBASE-17067.master.004.patch, HBASE-17067.master.004.patch, > HBASE-17067.master.004.patch, HBASE-17067.master.004.patch, > HBASE-17067.master.004.patch, HBASE-17067.master.004.patch, > HBASE-17067.master.005.patch > > > Once we have HBASE-16744, HBASE-16786, HBASE-16831. we can remove the > tryLock() methods and replace them with the wait/wake methods that are using > the framework events instead of spinning until we can start the proc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833930#comment-15833930
]
Yu Li commented on HBASE-17471:
---
bq. MVCC and region seqid are totally different concept, combing then mixed up
wal append and memstore insertion.
Yes, it's true from the semantic view, but not strong enough to take the
rollback (re-separate) efforts IMO (and could you help introduce the historical
reason of separating them [~stack]? Thanks). From performance aspect
HBASE-14460 and HBASE-16698 are known issues, and please let us know If you
observed any more (real ones instead of conceptional) [~allan163]. If no more,
I think focusing on improving the mvcc preassign feature is enough. If quite
more, I think it worth a discussion on whether to re-separate mvcc and sequence
id.
bq. As I noticed in our tests, even with preAssign, some handler still need to
wait for the advance of mvcc numbers. Since we now begin a mvcc transition
before append, and complete them after sync. You can treat mvcc queue as a
lock, for now, this lock is holding by a single handler for a long time.
Actually this is exactly what I was concerned about the new design and the
reason asking for a carefully testing. I don't think we could remove all
locking stuff on mvcc, so I care more about the comparison data, say what's the
throughput of the implementation based on new design and the old one. Will wait
for more detailed data. Thanks.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471-duo.patch, HBASE-17471-duo-v1.patch,
> HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch, HBASE-17471.v3.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-17471:
--
Attachment: HBASE-17471-duo-v1.patch
Fix TestWALActionsListener.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471-duo.patch, HBASE-17471-duo-v1.patch,
> HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch, HBASE-17471.v3.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833917#comment-15833917
]
Hadoop QA commented on HBASE-17471:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 37s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 19s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.wal.TestWALActionsListener |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848824/HBASE-17471-duo.patch
|
| JIRA Issue | HBASE-17471 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 0724571561bd 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7754a96 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5399/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5399/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5399/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5399/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833886#comment-15833886
]
Allan Yang commented on HBASE-17471:
When I say 'a lot of problems'， I mean performance problem & simplicity, and
many other issues to repair the performance problem(and many issues to fix bugs
brought by these issues...). MVCC and region seqid are totally different
concept, combing then mixed up wal append and memstore insertion. Separating
then will draw a clean line between this two actions. As I noticed in our
tests, even with preAssign, some handler still need to wait for the advance of
mvcc numbers. Since we now begin a mvcc transition before append, and complete
them after sync. You can treat mvcc queue as a lock, for now, this lock is
holding by a single handler for a long time.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471-duo.patch, HBASE-17471.patch,
> HBASE-17471.tmp, HBASE-17471.v2.patch, HBASE-17471.v3.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-17471:
--
Attachment: HBASE-17471-duo.patch
Try a pre commit check. Remove pre assign related code. StampSequenceId before
publishing the FSWALEntry to ring buffer.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471-duo.patch, HBASE-17471.patch,
> HBASE-17471.tmp, HBASE-17471.v2.patch, HBASE-17471.v3.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833874#comment-15833874
]
Yu Li commented on HBASE-17471:
---
bq. I'd like to say, separating them can truly resolve a lot of problems.
Does this mean we need to fix more problems if stick on current
mvcc-seqid-merge way? Please tell us more about the problems. I think we need
to compare a) the advantage of merging mvcc and sequence-id and b) the efforts
to fix all the problems introduced.
Ping [~stack], for your special notice sir. [~allan163] works in [~zjushch]'s
team and use HBase as a *common* service for Alibaba group, while my team using
HBase as a *special* service (focusing on search and recommendation) for
Alibaba search, so they're experiencing more common workloads/use-cases and may
encounter more issues than us (such as in our scenario we don't have
append/increment requests but they do) in product env.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp,
> HBASE-17471.v2.patch, HBASE-17471.v3.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17471:
---
Attachment: HBASE-17471.v3.patch
rerun Hudson
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp,
> HBASE-17471.v2.patch, HBASE-17471.v3.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17407) Correct update of maxFlushedSeqId in HRegion
[ https://issues.apache.org/jira/browse/HBASE-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833868#comment-15833868 ] Hudson commented on HBASE-17407: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2368 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2368/]) HBASE-17407: Correct update of maxFlushedSeqId in HRegion (zhangduo: rev f254e278ece751e67c92570aef4b15fddab22a94) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceIdAccounting.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DisabledWALProvider.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java > Correct update of maxFlushedSeqId in HRegion > > > Key: HBASE-17407 > URL: https://issues.apache.org/jira/browse/HBASE-17407 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-17407-V01.patch, HBASE-17407-V01.patch, > HBASE-17407-V02.patch, HBASE-17407-V02.patch > > > The attribute maxFlushedSeqId in HRegion is used to track the max sequence id > in the store files and is reported to HMaster. When flushing only part of the > memstore content this value might be incorrect and may cause data loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833862#comment-15833862
]
Allan Yang edited comment on HBASE-17471 at 1/23/17 3:31 AM:
-
Actually, we notice the performance regression after merging mvcc and sequence
id, too. I even prepared a patch which separate mvcc and seqid as before
HBASE-8763. This patch is based on our custom branch-1.1.2. It works well, and
have the same performance(maybe a little better) as preAssignMVCC. But consider
of many features are based on the fact that mvcc and seqid is combined, I gave
up the patch. If anyone interested in separate mvcc and sequence, you are
welcomed to discuss with me. I'd like to say, separating them can truly resolve
a lot of problems. Meanwhile, let's focus on this patch.
was (Author: allan163):
Actually, we notice the performance regression after merging mvcc and sequence
id, too. I even prepared a patch which separate mvcc and seqid as before
HBASE-8763. This patch is based on our custom branch-1.1.2. It works well, and
have the same performance(maybe a little better) as preAssignMVCC. But consider
of many features are based on the fact that mvcc and seqid is combined, I gave
up the patch. If anyone interested in separate mvcc and sequence, you are
welcomed to discuss with me. I'd like to say, separating them can truly resolve
a lot of problems. Meanwhile, let's forces on this patch.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833865#comment-15833865
]
Hadoop QA commented on HBASE-17471:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 39s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 17s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestCheckTestClasses |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848809/HBASE-17471.v2.patch |
| JIRA Issue | HBASE-17471 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux fe4da3ebb376 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7754a96 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5397/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5397/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5397/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5397/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
>
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833862#comment-15833862
]
Allan Yang commented on HBASE-17471:
Actually, we notice the performance regression after merging mvcc and sequence
id, too. I even prepared a patch which separate mvcc and seqid as before
HBASE-8763. This patch is based on our custom branch-1.1.2. It works well, and
have the same performance(maybe a little better) as preAssignMVCC. But consider
of many features are based on the fact that mvcc and seqid is combined, I gave
up the patch. If anyone interested in separate mvcc and sequence, you are
welcomed to discuss with me. I'd like to say, separating them can truly resolve
a lot of problems. Meanwhile, let's forces on this patch.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833858#comment-15833858
]
Yu Li commented on HBASE-17471:
---
bq. I still do not think we need a configurable preAssign...
Ok, maybe I'm kind of too conservative (smile). Let's remove the configuration
if the performance data looks good for all common scenarios, but all depending
on the perf result. :-)
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833852#comment-15833852
]
Yu Li commented on HBASE-17471:
---
bq. Yu Li Could you please help testing the performance of the change as you
already have a good workload to expose the performance issue for the old
Sure, will follow up offline with Allan.
Actually we have observed more than one issue after merging mvcc and sequence
id, such as HBASE-14460 and HBASE-16698. Let's take the chance to look at the
whole thing again and do clean up once for all (hopefully). :-)
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833844#comment-15833844
]
Allan Yang commented on HBASE-17471:
Thank you for your opinion [~carp84], will rebase the code and making it more
clean. Soon I will upload the performance number w/&wo/ the patch
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833842#comment-15833842
]
Hadoop QA commented on HBASE-17471:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 13s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 18s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestCheckTestClasses |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848069/HBASE-17471.patch |
| JIRA Issue | HBASE-17471 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3e2252e849e5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f254e27 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5395/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5395/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5395/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5395/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
>
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833840#comment-15833840
]
Duo Zhang commented on HBASE-17471:
---
{quote}
I think we need only synchronize the acquirement of seqid/mvcc and ringbuffer's
sequence id.
{quote}
Sounds reasonable. I could also prepare a patch here as we also need to modify
the implementation WAL I think. [~carp84] Could you please help testing the
performance of the change as you already have a good workload to expose the
performance issue for the old implementation?
Thanks.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-17509) Remove the mvcc preassign lock and take usage of MVCC internal lock instead
[ https://issues.apache.org/jira/browse/HBASE-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li resolved HBASE-17509. --- Resolution: Duplicate Since [~allan163] already integrate the new design into patch of HBASE-17471, let's focus the work there. Closing this one. > Remove the mvcc preassign lock and take usage of MVCC internal lock instead > --- > > Key: HBASE-17509 > URL: https://issues.apache.org/jira/browse/HBASE-17509 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > > This is an improvement on current mvcc preassign implementation introduced by > HBASE-16698, as well as an easier way to unify the logic of put, append and > increment. Thanks [~Apache9] for contributing the initial idea, will try this > out in this JIRA to see how it works. > Could also be regarded as a first try of HBASE-16873 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833836#comment-15833836
]
Yu Li commented on HBASE-17471:
---
Ok, so you've already take the new design, quick action (smile).
I proposed to do this one and HBASE-17509 in parallel basically because the new
design need more testing to prove no performance regression. But since you've
already chosen to take the new design directly, let's go down this way. An
early reminder that we will need to supply performance number before commit.
Regarding the patch itself, it still looks in a draft style and could not apply
to master branch, but the main logic looks good and follows the new design.
Please rebase it on the latest code base and let's see how the UT works before
further review. Please also remove those commented out lines and upload a patch
on review board for easier review.
Let me close HBASE-17509 and focus here, but I'd suggest to change the title to
something similar to HBASE-17509 if we finally take the new design, not in a
hurry though.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17495) TestHRegionWithInMemoryFlush#testFlushCacheWhileScanning intermittently fails due to assertion error
[
https://issues.apache.org/jira/browse/HBASE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17495:
---
Attachment: testHRegionWithInMemoryFlush-output.0123
> TestHRegionWithInMemoryFlush#testFlushCacheWhileScanning intermittently fails
> due to assertion error
>
>
> Key: HBASE-17495
> URL: https://issues.apache.org/jira/browse/HBASE-17495
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
> Attachments: testHRegionWithInMemoryFlush-output.0119,
> testHRegionWithInMemoryFlush-output.0123
>
>
> Looping through the test (based on commit
> 76dc957f64fa38ce88694054db7dbf590f368ae7), I saw the following test failure:
> {code}
> testFlushCacheWhileScanning(org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush)
> Time elapsed: 0.53 sec <<< FAILURE!
> java.lang.AssertionError: toggle=false i=940 ts=1484852861597 expected:<94>
> but was:<92>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.hbase.regionserver.TestHRegion.testFlushCacheWhileScanning(TestHRegion.java:3533)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> {code}
> See test output for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17495) TestHRegionWithInMemoryFlush#testFlushCacheWhileScanning intermittently fails due to assertion error
[
https://issues.apache.org/jira/browse/HBASE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833828#comment-15833828
]
Ted Yu commented on HBASE-17495:
With HBASE-17407 in place, the test still fails occasionally:
{code}
org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush Time
elapsed: 600.016 sec <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 10 minutes
at java.lang.Object.wait(Native Method)
at
org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.waitForRead(MultiVersionConcurrencyControl.java:218)
at
org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.completeAndWait(MultiVersionConcurrencyControl.java:149)
at
org.apache.hadoop.hbase.regionserver.HRegion.getNextSequenceId(HRegion.java:2729)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2447)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2342)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2313)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2303)
at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1600)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1505)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1455)
at
org.apache.hadoop.hbase.HBaseTestingUtility.closeRegionAndWAL(HBaseTestingUtility.java:374)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileScanning(TestHRegion.java:3693)
{code}
> TestHRegionWithInMemoryFlush#testFlushCacheWhileScanning intermittently fails
> due to assertion error
>
>
> Key: HBASE-17495
> URL: https://issues.apache.org/jira/browse/HBASE-17495
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
> Attachments: testHRegionWithInMemoryFlush-output.0119
>
>
> Looping through the test (based on commit
> 76dc957f64fa38ce88694054db7dbf590f368ae7), I saw the following test failure:
> {code}
> testFlushCacheWhileScanning(org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush)
> Time elapsed: 0.53 sec <<< FAILURE!
> java.lang.AssertionError: toggle=false i=940 ts=1484852861597 expected:<94>
> but was:<92>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.hbase.regionserver.TestHRegion.testFlushCacheWhileScanning(TestHRegion.java:3533)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> {code}
> See test output for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833822#comment-15833822
]
Allan Yang commented on HBASE-17471:
I think we need only synchronize the acquirement of seqid/mvcc and ringbuffer's
sequence id. We need keep the logic under lock as simple as possible. Appending
WAL in MVCC.begin() may be too complicated.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17462) Use sliding window for read/write request costs in StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833818#comment-15833818 ] Ted Yu commented on HBASE-17462: Mind attaching patch for branch-1 ? Thanks > Use sliding window for read/write request costs in StochasticLoadBalancer > - > > Key: HBASE-17462 > URL: https://issues.apache.org/jira/browse/HBASE-17462 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Tim Brown > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: after_changes.png, before_changes.png, > HBASE-17462.patch, HBASE-17462-v2.patch > > > In the thread, http://search-hadoop.com/m/HBase/YGbbyUZKXWALkX1, Timothy was > asking whether the read/write request costs in StochasticLoadBalancer should > be calculated as rates. > This makes sense since read / write load on region server tends to fluctuate > over time. Using sliding window would reflect more recent trend in read / > write load. > Some factors to consider: > The data structure used by StochasticLoadBalancer should be concise. The > number of regions in a cluster can be expected to approach 1 million. We > cannot afford to store long history of read / write requests in master. > Efficiency of cost calculation should be high - there're many cost > functions the balancer goes through, it is expected for each cost function > to return quickly. Otherwise we would not come up with proper region > movement plan(s) in time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17462) Use sliding window for read/write request costs in StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17462: --- Summary: Use sliding window for read/write request costs in StochasticLoadBalancer (was: Investigate using sliding window for read/write request costs in StochasticLoadBalancer) > Use sliding window for read/write request costs in StochasticLoadBalancer > - > > Key: HBASE-17462 > URL: https://issues.apache.org/jira/browse/HBASE-17462 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Tim Brown > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: after_changes.png, before_changes.png, > HBASE-17462.patch, HBASE-17462-v2.patch > > > In the thread, http://search-hadoop.com/m/HBase/YGbbyUZKXWALkX1, Timothy was > asking whether the read/write request costs in StochasticLoadBalancer should > be calculated as rates. > This makes sense since read / write load on region server tends to fluctuate > over time. Using sliding window would reflect more recent trend in read / > write load. > Some factors to consider: > The data structure used by StochasticLoadBalancer should be concise. The > number of regions in a cluster can be expected to approach 1 million. We > cannot afford to store long history of read / write requests in master. > Efficiency of cost calculation should be high - there're many cost > functions the balancer goes through, it is expected for each cost function > to return quickly. Otherwise we would not come up with proper region > movement plan(s) in time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17471:
---
Attachment: (was: HBASE-17471.v2.patch)
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17471:
---
Attachment: HBASE-17471.v2.patch
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833816#comment-15833816
]
Duo Zhang commented on HBASE-17471:
---
I still do not think we need a configurable preAssign...
We can declare the MVCC.begin method like
MVCC.begin(Consumer action)
And in that action we create WALEdit and append it to WAL. Yeah we should not
throw IOException for WAL.append then but I think it is possible, just remove
the closed check in WAL.append, it does not matter.
And then we can remove the stampRegionSequenceId method and mark
WALKey.writeEntry as final. This can simplify our logic a lot.
Thanks.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833811#comment-15833811
]
Hadoop QA commented on HBASE-17471:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-17471 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848807/HBASE-17471.v2.patch |
| JIRA Issue | HBASE-17471 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5396/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17471:
---
Attachment: HBASE-17471.v2.patch
Add a init patch to clarify my thought and run through the Hundson.
Thanks for [~Apache9] 's idea of removing the additional lock.
In this patch, I bring the mvcc preassign to the wal.append(), all acquirement
of seqid and ringbuffer's txid is synchronized under mvcc.writeQueue, making
sure the seqid in WAL is increasing monotonically. I also moved the action of
stamping seqid to cell from ringbuffer's single consume thread to increase
throughput. (According to one of my test, acquire mvcc under lock and stamping
them to cell takes 25% time of the append action)
One thing to do is making preAssign configurable.
[~carp84] (the original author of preAssignMVCC) and the others, please give me
some advice.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17471) Region Seqid will be out of order in WAL if using mvccPreAssign
[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17471:
---
Status: Patch Available (was: Open)
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Attachments: HBASE-17471.patch, HBASE-17471.tmp
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17462) Investigate using sliding window for read/write request costs in StochasticLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833803#comment-15833803
]
Hadoop QA commented on HBASE-17462:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 15s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 81m 27s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 121m 41s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848800/HBASE-17462-v2.patch |
| JIRA Issue | HBASE-17462 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 0636f562bb8e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3abd13d |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5394/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5394/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Investigate using sliding window for read/write request costs in
> StochasticLoadBalancer
> ---
>
> Key: HBASE-17462
> URL: https://issues.apache.org/jira/browse/HBASE-17462
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Tim Brown
> Labels: patch
> Fix For: 2.0.
[jira] [Updated] (HBASE-17407) Correct update of maxFlushedSeqId in HRegion
[ https://issues.apache.org/jira/browse/HBASE-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17407: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~eshcar] for the contribution. > Correct update of maxFlushedSeqId in HRegion > > > Key: HBASE-17407 > URL: https://issues.apache.org/jira/browse/HBASE-17407 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-17407-V01.patch, HBASE-17407-V01.patch, > HBASE-17407-V02.patch, HBASE-17407-V02.patch > > > The attribute maxFlushedSeqId in HRegion is used to track the max sequence id > in the store files and is reported to HMaster. When flushing only part of the > memstore content this value might be incorrect and may cause data loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17407) Correct update of maxFlushedSeqId in HRegion
[ https://issues.apache.org/jira/browse/HBASE-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17407: -- Affects Version/s: 2.0.0 Fix Version/s: 2.0.0 Component/s: wal > Correct update of maxFlushedSeqId in HRegion > > > Key: HBASE-17407 > URL: https://issues.apache.org/jira/browse/HBASE-17407 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-17407-V01.patch, HBASE-17407-V01.patch, > HBASE-17407-V02.patch, HBASE-17407-V02.patch > > > The attribute maxFlushedSeqId in HRegion is used to track the max sequence id > in the store files and is reported to HMaster. When flushing only part of the > memstore content this value might be incorrect and may cause data loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17407) Correct update of maxFlushedSeqId in HRegion
[ https://issues.apache.org/jira/browse/HBASE-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833788#comment-15833788 ] Duo Zhang commented on HBASE-17407: --- Will commit shortly. > Correct update of maxFlushedSeqId in HRegion > > > Key: HBASE-17407 > URL: https://issues.apache.org/jira/browse/HBASE-17407 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17407-V01.patch, HBASE-17407-V01.patch, > HBASE-17407-V02.patch, HBASE-17407-V02.patch > > > The attribute maxFlushedSeqId in HRegion is used to track the max sequence id > in the store files and is reported to HMaster. When flushing only part of the > memstore content this value might be incorrect and may cause data loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17462) Investigate using sliding window for read/write request costs in StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833766#comment-15833766 ] Ted Yu commented on HBASE-17462: +1 on v2. Pending QA. > Investigate using sliding window for read/write request costs in > StochasticLoadBalancer > --- > > Key: HBASE-17462 > URL: https://issues.apache.org/jira/browse/HBASE-17462 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Tim Brown > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: after_changes.png, before_changes.png, > HBASE-17462.patch, HBASE-17462-v2.patch > > > In the thread, http://search-hadoop.com/m/HBase/YGbbyUZKXWALkX1, Timothy was > asking whether the read/write request costs in StochasticLoadBalancer should > be calculated as rates. > This makes sense since read / write load on region server tends to fluctuate > over time. Using sliding window would reflect more recent trend in read / > write load. > Some factors to consider: > The data structure used by StochasticLoadBalancer should be concise. The > number of regions in a cluster can be expected to approach 1 million. We > cannot afford to store long history of read / write requests in master. > Efficiency of cost calculation should be high - there're many cost > functions the balancer goes through, it is expected for each cost function > to return quickly. Otherwise we would not come up with proper region > movement plan(s) in time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17462) Investigate using sliding window for read/write request costs in StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-17462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Brown updated HBASE-17462: -- Attachment: HBASE-17462-v2.patch > Investigate using sliding window for read/write request costs in > StochasticLoadBalancer > --- > > Key: HBASE-17462 > URL: https://issues.apache.org/jira/browse/HBASE-17462 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Tim Brown > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: after_changes.png, before_changes.png, > HBASE-17462.patch, HBASE-17462-v2.patch > > > In the thread, http://search-hadoop.com/m/HBase/YGbbyUZKXWALkX1, Timothy was > asking whether the read/write request costs in StochasticLoadBalancer should > be calculated as rates. > This makes sense since read / write load on region server tends to fluctuate > over time. Using sliding window would reflect more recent trend in read / > write load. > Some factors to consider: > The data structure used by StochasticLoadBalancer should be concise. The > number of regions in a cluster can be expected to approach 1 million. We > cannot afford to store long history of read / write requests in master. > Efficiency of cost calculation should be high - there're many cost > functions the balancer goes through, it is expected for each cost function > to return quickly. Otherwise we would not come up with proper region > movement plan(s) in time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12894) Upgrade Jetty to 9.2.6
[ https://issues.apache.org/jira/browse/HBASE-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833748#comment-15833748 ] Guang Yang commented on HBASE-12894: Thanks [~stack], I will take a look at the failure. > Upgrade Jetty to 9.2.6 > -- > > Key: HBASE-12894 > URL: https://issues.apache.org/jira/browse/HBASE-12894 > Project: HBase > Issue Type: Improvement > Components: REST, UI >Affects Versions: 0.98.0 >Reporter: Rick Hallihan >Assignee: Guang Yang >Priority: Critical > Labels: MicrosoftSupport > Fix For: 2.0.0 > > Attachments: dependency_list_after, dependency_list_before, > HBASE-12894_Jetty9_v0.patch, HBASE-12894_Jetty9_v10.patch, > HBASE-12894_Jetty9_v1.patch, HBASE-12894_Jetty9_v1.patch, > HBASE-12894_Jetty9_v2.patch, HBASE-12894_Jetty9_v3.patch, > HBASE-12894_Jetty9_v4.patch, HBASE-12894_Jetty9_v5.patch, > HBASE-12894_Jetty9_v6.patch, HBASE-12894_Jetty9_v7.patch, > HBASE-12894_Jetty9_v8.patch, HBASE-12894.master.001.patch, > HBASE-12894.master.002.patch, HBASE-12894.master.003.patch > > > The Jetty component that is used for the HBase Stargate REST endpoint is > version 6.1.26 and is fairly outdated. We recently had a customer inquire > about enabling cross-origin resource sharing (CORS) for the REST endpoint and > found that this older version does not include the necessary filter or > configuration options, highlighted at: > http://wiki.eclipse.org/Jetty/Feature/Cross_Origin_Filter > The Jetty project has had significant updates through versions 7, 8 and 9, > including a transition to be an Eclipse subproject, so updating to the latest > version may be non-trivial. The last update to the Jetty component in > https://issues.apache.org/jira/browse/HBASE-3377 was a minor version update > and did not require significant work. This update will include a package > namespace update so there will likely be a larger number of required changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17407) Correct update of maxFlushedSeqId in HRegion
[
https://issues.apache.org/jira/browse/HBASE-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833741#comment-15833741
]
Hadoop QA commented on HBASE-17407:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 81m 54s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 119m 13s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848792/HBASE-17407-V02.patch
|
| JIRA Issue | HBASE-17407 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 1d2f8fe857f0 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3abd13d |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5393/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5393/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Correct update of maxFlushedSeqId in HRegion
>
>
> Key: HBASE-17407
> URL: https://issues.apache.org/jira/browse/HBASE-17407
> Project: HBase
> Issue Type: Bug
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> At
[jira] [Commented] (HBASE-17504) The passed durability of Increment is ignored when syncing WAL
[
https://issues.apache.org/jira/browse/HBASE-17504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833717#comment-15833717
]
Hudson commented on HBASE-17504:
FAILURE: Integrated in Jenkins build HBase-1.4 #600 (See
[https://builds.apache.org/job/HBase-1.4/600/])
HBASE-17504 The passed durability of Increment is ignored when syncing (tedyu:
rev 6e0f3f5bbc3183c8e768a72dbacde56be51442a1)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> The passed durability of Increment is ignored when syncing WAL
> --
>
> Key: HBASE-17504
> URL: https://issues.apache.org/jira/browse/HBASE-17504
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 1.4.0
>
> Attachments: HBASE-17504.branch-1.v0.patch
>
>
> {code:title=HRegion.java|borderStyle=solid}
> private Result doIncrement(Increment increment, long nonceGroup, long nonce)
> throws IOException {
> Durability effectiveDurability =
> getEffectiveDurability(increment.getDurability());
> ...
> if(txid != 0) {
> syncOrDefer(txid, durability);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17407) Correct update of maxFlushedSeqId in HRegion
[ https://issues.apache.org/jira/browse/HBASE-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-17407: -- Attachment: HBASE-17407-V02.patch > Correct update of maxFlushedSeqId in HRegion > > > Key: HBASE-17407 > URL: https://issues.apache.org/jira/browse/HBASE-17407 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17407-V01.patch, HBASE-17407-V01.patch, > HBASE-17407-V02.patch, HBASE-17407-V02.patch > > > The attribute maxFlushedSeqId in HRegion is used to track the max sequence id > in the store files and is reported to HMaster. When flushing only part of the > memstore content this value might be incorrect and may cause data loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17487) Throw exception when pushing pipeline to snapshot fails twice
[
https://issues.apache.org/jira/browse/HBASE-17487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833627#comment-15833627
]
Hadoop QA commented on HBASE-17487:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 18s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 44s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 127m 24s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848782/17487.v3.txt |
| JIRA Issue | HBASE-17487 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6647302c06c7 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3abd13d |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5392/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5392/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Throw exception when pushing pipeline to snapshot fails twice
> -
[jira] [Commented] (HBASE-17509) Remove the mvcc preassign lock and take usage of MVCC internal lock instead
[ https://issues.apache.org/jira/browse/HBASE-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833616#comment-15833616 ] Yu Li commented on HBASE-17509: --- Probably overlaps HBASE-17471, but it's ok to do both in parallel I guess, since HBASE-17471 is a fix of existing problem, and the JIRA here is an improvement of the mechanism. [~allan163] and [~stack], FYI. > Remove the mvcc preassign lock and take usage of MVCC internal lock instead > --- > > Key: HBASE-17509 > URL: https://issues.apache.org/jira/browse/HBASE-17509 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > > This is an improvement on current mvcc preassign implementation introduced by > HBASE-16698, as well as an easier way to unify the logic of put, append and > increment. Thanks [~Apache9] for contributing the initial idea, will try this > out in this JIRA to see how it works. > Could also be regarded as a first try of HBASE-16873 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17509) Remove the mvcc preassign lock and take usage of MVCC internal lock instead
Yu Li created HBASE-17509: - Summary: Remove the mvcc preassign lock and take usage of MVCC internal lock instead Key: HBASE-17509 URL: https://issues.apache.org/jira/browse/HBASE-17509 Project: HBase Issue Type: Bug Reporter: Yu Li Assignee: Yu Li This is an improvement on current mvcc preassign implementation introduced by HBASE-16698, as well as an easier way to unify the logic of put, append and increment. Thanks [~Apache9] for contributing the initial idea, will try this out in this JIRA to see how it works. Could also be regarded as a first try of HBASE-16873 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[ https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833612#comment-15833612 ] Yu Li commented on HBASE-17507: --- bq. Can we reuse the lock?... The action will be executed under the lock in MVCC so can keep the same order with mvcc. This is a good idea and will make it easier to unify the logic of put and append/increment (doMiniBatchMutation and doDelta). Let me open another JIRA to give it a try. Thanks. > Add document for the mvcc preassign feature introduced in HBASE-16698 > - > > Key: HBASE-17507 > URL: https://issues.apache.org/jira/browse/HBASE-17507 > Project: HBase > Issue Type: Task >Reporter: Yu Li >Assignee: Yu Li > > In this JIRA we will supply a design doc of the mvcc preassign mechanism > introduced in HBASE-16698, as well as more detailed performance data of it > for better reference. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17504) The passed durability of Increment is ignored when syncing WAL
[
https://issues.apache.org/jira/browse/HBASE-17504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17504:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> The passed durability of Increment is ignored when syncing WAL
> --
>
> Key: HBASE-17504
> URL: https://issues.apache.org/jira/browse/HBASE-17504
> Project: HBase
> Issue Type: Bug
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 1.4.0
>
> Attachments: HBASE-17504.branch-1.v0.patch
>
>
> {code:title=HRegion.java|borderStyle=solid}
> private Result doIncrement(Increment increment, long nonceGroup, long nonce)
> throws IOException {
> Durability effectiveDurability =
> getEffectiveDurability(increment.getDurability());
> ...
> if(txid != 0) {
> syncOrDefer(txid, durability);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17045) Unify the implementation of small scan and regular scan
[
https://issues.apache.org/jira/browse/HBASE-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833570#comment-15833570
]
Hadoop QA commented on HBASE-17045:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 39s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 3s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 2m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 43s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 30s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 105m 30s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
3s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 192m 4s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848780/HBASE-17045-v7.patch |
| JIRA Issue | HBASE-17045 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux caa23ecd4a8f 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86
[jira] [Updated] (HBASE-17487) Throw exception when pushing pipeline to snapshot fails twice
[ https://issues.apache.org/jira/browse/HBASE-17487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17487: --- Attachment: 17487.v3.txt > Throw exception when pushing pipeline to snapshot fails twice > - > > Key: HBASE-17487 > URL: https://issues.apache.org/jira/browse/HBASE-17487 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: 17487.v2.txt, 17487.v3.txt > > > In CompactingMemStore#pushPipelineToSnapshot() , there is limit of 3 > iterations of pipeline.swap() call after which an empty ImmutableSegment is > used as snapshot. > However, there should be at most two iterations in pushPipelineToSnapshot() > since during the second iteration there is no concurrent write to memstore. > We should throw exception in the 3rd iteration to signify that this scenario > should never happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[ https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833520#comment-15833520 ] Hudson commented on HBASE-17489: SUCCESS: Integrated in Jenkins build HBase-1.4 #599 (See [https://builds.apache.org/job/HBase-1.4/599/]) HBASE-17489 ClientScanner may send a next request to a RegionScanner (zhangduo: rev 57409371a063ed4d93292ca6e0b7d5b8866e1ffc) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedClientScanner.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientScanner.java > ClientScanner may send a next request to a RegionScanner which has been > exhausted > - > > Key: HBASE-17489 > URL: https://issues.apache.org/jira/browse/HBASE-17489 > Project: HBase > Issue Type: Bug > Components: Client, scan >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-17489-branch-1.3.patch, > HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch, > HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch, > HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch > > > Found it when implementing HBASE-17045. Seems the final result of the scan is > correct but no doubt the logic is broken. We need to fix it to stop things > get worse in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[ https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833515#comment-15833515 ] Hudson commented on HBASE-17489: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #99 (See [https://builds.apache.org/job/HBase-1.3-JDK8/99/]) HBASE-17489 ClientScanner may send a next request to a RegionScanner (zhangduo: rev 94ade6a514e9935a8c283befb31f29cd8d3a2045) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientScanner.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedClientScanner.java > ClientScanner may send a next request to a RegionScanner which has been > exhausted > - > > Key: HBASE-17489 > URL: https://issues.apache.org/jira/browse/HBASE-17489 > Project: HBase > Issue Type: Bug > Components: Client, scan >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-17489-branch-1.3.patch, > HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch, > HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch, > HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch > > > Found it when implementing HBASE-17045. Seems the final result of the scan is > correct but no doubt the logic is broken. We need to fix it to stop things > get worse in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17407) Correct update of maxFlushedSeqId in HRegion
[
https://issues.apache.org/jira/browse/HBASE-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833513#comment-15833513
]
Duo Zhang commented on HBASE-17407:
---
{quote}
I think removing the startCacheFlush method may have consequences which are
beyond the scope of this Jira. I'm not sure it removing it would be the right
thing, but if you feel strongly about it then Let's discuss it in a dedicated
issue.
{quote}
Fine. Can do it in another issue.
And for the '<=' to '<' change, I read the code again, it is only a check in
abortCacheFlush so not very critical.
+1 on patch v2. You can try another pre commit run to see if it can pass all
the UTs.
Thanks.
> Correct update of maxFlushedSeqId in HRegion
>
>
> Key: HBASE-17407
> URL: https://issues.apache.org/jira/browse/HBASE-17407
> Project: HBase
> Issue Type: Bug
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-17407-V01.patch, HBASE-17407-V01.patch,
> HBASE-17407-V02.patch
>
>
> The attribute maxFlushedSeqId in HRegion is used to track the max sequence id
> in the store files and is reported to HMaster. When flushing only part of the
> memstore content this value might be incorrect and may cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17407) Correct update of maxFlushedSeqId in HRegion
[ https://issues.apache.org/jira/browse/HBASE-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17407: -- Assignee: Eshcar Hillel > Correct update of maxFlushedSeqId in HRegion > > > Key: HBASE-17407 > URL: https://issues.apache.org/jira/browse/HBASE-17407 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17407-V01.patch, HBASE-17407-V01.patch, > HBASE-17407-V02.patch > > > The attribute maxFlushedSeqId in HRegion is used to track the max sequence id > in the store files and is reported to HMaster. When flushing only part of the > memstore content this value might be incorrect and may cause data loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17045) Unify the implementation of small scan and regular scan
[ https://issues.apache.org/jira/browse/HBASE-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833509#comment-15833509 ] Duo Zhang commented on HBASE-17045: --- Ready for review sir. [~stack] [~enis]. > Unify the implementation of small scan and regular scan > --- > > Key: HBASE-17045 > URL: https://issues.apache.org/jira/browse/HBASE-17045 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17045.patch, HBASE-17045-v1.patch, > HBASE-17045-v2.patch, HBASE-17045-v3.patch, HBASE-17045-v4.patch, > HBASE-17045-v5.patch, HBASE-17045-v7.patch, HBASE-17405-v6.patch > > > See [~enis]'s comment in HBASE-16838 > https://issues.apache.org/jira/browse/HBASE-16838?focusedCommentId=15637803&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15637803 > But there is another scenario that we need small scan is that, we do not know > the stop row but we only want a small set of results. For example, in the > implementation of region locator, we will use small scan and set caching to 1 > as we only need one row. > So I think we need to add a new option(maybe called limit?) for the scan > object, and deprecate the small option. And the server side modification > should also be committed to branch-1 to simplify the logic of async client in > 2.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17508) Unify the implementation of small scan and regular scan for sync client
[ https://issues.apache.org/jira/browse/HBASE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17508: -- Issue Type: Task (was: Bug) > Unify the implementation of small scan and regular scan for sync client > --- > > Key: HBASE-17508 > URL: https://issues.apache.org/jira/browse/HBASE-17508 > Project: HBase > Issue Type: Task > Components: Client, scan >Affects Versions: 2.0.0, 1.4.0 >Reporter: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > > Implement the same logic with HBASE-17045 for sync client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17508) Unify the implementation of small scan and regular scan for sync client
Duo Zhang created HBASE-17508: - Summary: Unify the implementation of small scan and regular scan for sync client Key: HBASE-17508 URL: https://issues.apache.org/jira/browse/HBASE-17508 Project: HBase Issue Type: Bug Components: Client, scan Affects Versions: 2.0.0, 1.4.0 Reporter: Duo Zhang Fix For: 2.0.0, 1.4.0 Implement the same logic with HBASE-17045 for sync client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17045) Unify the implementation of small scan and regular scan
[ https://issues.apache.org/jira/browse/HBASE-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17045: -- Attachment: HBASE-17045-v7.patch Add annotation for Scan.ReadType. > Unify the implementation of small scan and regular scan > --- > > Key: HBASE-17045 > URL: https://issues.apache.org/jira/browse/HBASE-17045 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17045.patch, HBASE-17045-v1.patch, > HBASE-17045-v2.patch, HBASE-17045-v3.patch, HBASE-17045-v4.patch, > HBASE-17045-v5.patch, HBASE-17045-v7.patch, HBASE-17405-v6.patch > > > See [~enis]'s comment in HBASE-16838 > https://issues.apache.org/jira/browse/HBASE-16838?focusedCommentId=15637803&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15637803 > But there is another scenario that we need small scan is that, we do not know > the stop row but we only want a small set of results. For example, in the > implementation of region locator, we will use small scan and set caching to 1 > as we only need one row. > So I think we need to add a new option(maybe called limit?) for the scan > object, and deprecate the small option. And the server side modification > should also be committed to branch-1 to simplify the logic of async client in > 2.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17045) Unify the implementation of small scan and regular scan
[
https://issues.apache.org/jira/browse/HBASE-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833488#comment-15833488
]
Hadoop QA commented on HBASE-17045:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 39s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 35s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 13s {color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 102m 56s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 186m 27s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestInterfaceAudienceAnnotations |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848759/HBASE-17405-v6.patch |
| JIRA Issue | HBASE-17045 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | L
[jira] [Commented] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[ https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833465#comment-15833465 ] Hudson commented on HBASE-17489: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #86 (See [https://builds.apache.org/job/HBase-1.3-JDK7/86/]) HBASE-17489 ClientScanner may send a next request to a RegionScanner (zhangduo: rev 94ade6a514e9935a8c283befb31f29cd8d3a2045) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientScanner.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedClientScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java > ClientScanner may send a next request to a RegionScanner which has been > exhausted > - > > Key: HBASE-17489 > URL: https://issues.apache.org/jira/browse/HBASE-17489 > Project: HBase > Issue Type: Bug > Components: Client, scan >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-17489-branch-1.3.patch, > HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch, > HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch, > HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch > > > Found it when implementing HBASE-17045. Seems the final result of the scan is > correct but no doubt the logic is broken. We need to fix it to stop things > get worse in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17045) Unify the implementation of small scan and regular scan
[
https://issues.apache.org/jira/browse/HBASE-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833463#comment-15833463
]
Hadoop QA commented on HBASE-17045:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 58s {color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 81m 7s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestInterfaceAudienceAnnotations |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848758/HBASE-17405-v6.patch |
| JIRA Issue | HBASE-17045 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Lin
[jira] [Commented] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[ https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833461#comment-15833461 ] Hudson commented on HBASE-17489: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2364 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2364/]) HBASE-17489 ClientScanner may send a next request to a RegionScanner (zhangduo: rev 3abd13dacb57927bd44a47632f4bd0c2e2bb87ea) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedClientScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientScanner.java > ClientScanner may send a next request to a RegionScanner which has been > exhausted > - > > Key: HBASE-17489 > URL: https://issues.apache.org/jira/browse/HBASE-17489 > Project: HBase > Issue Type: Bug > Components: Client, scan >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-17489-branch-1.3.patch, > HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch, > HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch, > HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch > > > Found it when implementing HBASE-17045. Seems the final result of the scan is > correct but no doubt the logic is broken. We need to fix it to stop things > get worse in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[ https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833454#comment-15833454 ] Duo Zhang commented on HBASE-17507: --- And I think we already have a lock in MultiVersionConcurrencyControl. Can we reuse the lock? Maybe something like MultiVersionConcurrencyControl.begin(Runnable action) The action will be executed under the lock in MVCC so can keep the same order with mvcc. That means, in the old time we do mvcc.begin while append WAL, and now we append WAL while acquire mvcc write number. Is it possible? This can save on extra lock which you mentioned above that may have bad impact on performance for an underloaded region. Thanks. > Add document for the mvcc preassign feature introduced in HBASE-16698 > - > > Key: HBASE-17507 > URL: https://issues.apache.org/jira/browse/HBASE-17507 > Project: HBase > Issue Type: Task >Reporter: Yu Li >Assignee: Yu Li > > In this JIRA we will supply a design doc of the mvcc preassign mechanism > introduced in HBASE-16698, as well as more detailed performance data of it > for better reference. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[ https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833446#comment-15833446 ] Yu Li commented on HBASE-17507: --- bq. Yes, A switch can be added, but will increase the complicity of the patch. WAL is not only appended in put/delete/Append/Increment, but also when writing open/close/compaction/flush markers The mvcc preassign feature introduced a per-region level lock, and this is some additional cost for RS with few regions or cluster with light workload. And as I mentioned, we also observed some limitations in other case. I'm glad that the design is well recognized, but I'd say be cautious to make it non-configurable. [~allan163] bq. Have you compared with multiwal? Yes, in our online product cluster we're using 4 wals. With more wals could lower the contention caused by CountDownLatch, but still cannot resolve our problem, which forced us to make this fix. [~Apache9] P.S. I'm wondering why the "Reply" function is removed from JIRA, I think "Reply" makes discussion looks more compacted... [~stack] [~busbey] > Add document for the mvcc preassign feature introduced in HBASE-16698 > - > > Key: HBASE-17507 > URL: https://issues.apache.org/jira/browse/HBASE-17507 > Project: HBase > Issue Type: Task >Reporter: Yu Li >Assignee: Yu Li > > In this JIRA we will supply a design doc of the mvcc preassign mechanism > introduced in HBASE-16698, as well as more detailed performance data of it > for better reference. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17407) Correct update of maxFlushedSeqId in HRegion
[ https://issues.apache.org/jira/browse/HBASE-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833445#comment-15833445 ] Eshcar Hillel commented on HBASE-17407: --- bq. Let's just remove the old startCacheFlush method? I think removing the startCacheFlush method may have consequences which are beyond the scope of this Jira. I'm not sure it removing it would be the right thing, but if you feel strongly about it then Let's discuss it in a dedicated issue. bq. Why changing '<=' to '<' ? We would like to prevent the WAL from going backward, but no harm is done if it stays the same. Specifically, with the current fix in case of EAGER compaction policy the returned sequence number is a lower bound estimation -- it just returns the current minimum sequence number. Most of the time the minimum sequence number will be larger than the previous one. But there might be some cases (like in the first flush to disk) where it is the same and therefore the change. > Correct update of maxFlushedSeqId in HRegion > > > Key: HBASE-17407 > URL: https://issues.apache.org/jira/browse/HBASE-17407 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel > Attachments: HBASE-17407-V01.patch, HBASE-17407-V01.patch, > HBASE-17407-V02.patch > > > The attribute maxFlushedSeqId in HRegion is used to track the max sequence id > in the store files and is reported to HMaster. When flushing only part of the > memstore content this value might be incorrect and may cause data loss. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[
https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833438#comment-15833438
]
Duo Zhang commented on HBASE-17507:
---
{quote}
The preAssignMvccLock is per-region level while w/o mvcc preassign the locking
is per-RS level due to the sequential handling of ringbuffer
{quote}
Have you compared with multiwal?
> Add document for the mvcc preassign feature introduced in HBASE-16698
> -
>
> Key: HBASE-17507
> URL: https://issues.apache.org/jira/browse/HBASE-17507
> Project: HBase
> Issue Type: Task
>Reporter: Yu Li
>Assignee: Yu Li
>
> In this JIRA we will supply a design doc of the mvcc preassign mechanism
> introduced in HBASE-16698, as well as more detailed performance data of it
> for better reference.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[ https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833435#comment-15833435 ] Allan Yang commented on HBASE-17507: Yes, A switch can be added, but will increase the complicity of the patch. WAL is not only appended in put/delete/Append/Increment, but also when writing open/close/compaction/flush markers > Add document for the mvcc preassign feature introduced in HBASE-16698 > - > > Key: HBASE-17507 > URL: https://issues.apache.org/jira/browse/HBASE-17507 > Project: HBase > Issue Type: Task >Reporter: Yu Li >Assignee: Yu Li > > In this JIRA we will supply a design doc of the mvcc preassign mechanism > introduced in HBASE-16698, as well as more detailed performance data of it > for better reference. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[ https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833434#comment-15833434 ] Allan Yang commented on HBASE-17507: I'm working on HBASE-17471 's fix, in the patch, I will always use preAssign > Add document for the mvcc preassign feature introduced in HBASE-16698 > - > > Key: HBASE-17507 > URL: https://issues.apache.org/jira/browse/HBASE-17507 > Project: HBase > Issue Type: Task >Reporter: Yu Li >Assignee: Yu Li > > In this JIRA we will supply a design doc of the mvcc preassign mechanism > introduced in HBASE-16698, as well as more detailed performance data of it > for better reference. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[
https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833433#comment-15833433
]
Yu Li commented on HBASE-17507:
---
The {{preAssignMvccLock}} is per-region level while w/o mvcc preassign the
locking is per-RS level due to the sequential handling of ringbuffer, will talk
more details in the document for better understanding.
> Add document for the mvcc preassign feature introduced in HBASE-16698
> -
>
> Key: HBASE-17507
> URL: https://issues.apache.org/jira/browse/HBASE-17507
> Project: HBase
> Issue Type: Task
>Reporter: Yu Li
>Assignee: Yu Li
>
> In this JIRA we will supply a design doc of the mvcc preassign mechanism
> introduced in HBASE-16698, as well as more detailed performance data of it
> for better reference.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[
https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833430#comment-15833430
]
Yu Li commented on HBASE-17507:
---
bq. what if the order of mvcc differs from the order of wal?
A {{preAssignMvccLock}} was introduced and made {{mvcc.begin}} and
{{WAL.append}} a transaction, so this won't happen.
> Add document for the mvcc preassign feature introduced in HBASE-16698
> -
>
> Key: HBASE-17507
> URL: https://issues.apache.org/jira/browse/HBASE-17507
> Project: HBase
> Issue Type: Task
>Reporter: Yu Li
>Assignee: Yu Li
>
> In this JIRA we will supply a design doc of the mvcc preassign mechanism
> introduced in HBASE-16698, as well as more detailed performance data of it
> for better reference.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[ https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833421#comment-15833421 ] Duo Zhang commented on HBASE-17507: --- Consider WAL, what if the order of mvcc differs from the order of wal? Sorry I haven't read the patch in HBASE-16698 carefully. > Add document for the mvcc preassign feature introduced in HBASE-16698 > - > > Key: HBASE-17507 > URL: https://issues.apache.org/jira/browse/HBASE-17507 > Project: HBase > Issue Type: Task >Reporter: Yu Li >Assignee: Yu Li > > In this JIRA we will supply a design doc of the mvcc preassign mechanism > introduced in HBASE-16698, as well as more detailed performance data of it > for better reference. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17491) Remove all setters from HTable interface and introduce a TableBuilder to build Table instance
[
https://issues.apache.org/jira/browse/HBASE-17491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833419#comment-15833419
]
Yu Li commented on HBASE-17491:
---
bq. On commit, change the class comment so it says since 2.0.0 rather than
since '...after HBASE-17471'
I remember that we also plan to commit this to branch-1, refer to [this comment
|
https://issues.apache.org/jira/browse/HBASE-17361?focusedCommentId=15825775&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15825775].
Plan changes?
Will fix all other javadoc when commit. Thanks for review sir [~stack]
> Remove all setters from HTable interface and introduce a TableBuilder to
> build Table instance
> -
>
> Key: HBASE-17491
> URL: https://issues.apache.org/jira/browse/HBASE-17491
> Project: HBase
> Issue Type: Sub-task
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17491.patch, HBASE-17491.v2.patch,
> HBASE-17491.v3.patch, HBASE-17491.v4.patch, HBASE-17491.v5.patch,
> HBASE-17491.v6.patch, HBASE-17491.v7.patch
>
>
> As titled, we will remove all setters in HTable for master branch and
> deprecate them for branch-1 to make HTable thread-safe. And a new
> {{TableBuilder}} interface will be introduced to build Table instance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[ https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833414#comment-15833414 ] Yu Li edited comment on HBASE-17507 at 1/22/17 9:35 AM: According to our study and prototype here, mvcc preassign will be some kind of limitation when we try to parallelize WAL sync (IO bound) and mvcc assigning (CPU bound) on write path (SEDA), so let's just keep it configurable for a little bit longer (smile). And since SEDA will increase throughput (PROS) but also longer RT (CONS), it doesn't mean mvcc preassign will become useless after SEDA is done on write path, just case by case. was (Author: carp84): According to our study and prototype here, mvcc preassign will be some kind of limitation when we try to parallelize WAL sync (IO bound) and mvcc assigning (CPU bound) on write path, so let's just keep it configurable for a little bit longer (smile). > Add document for the mvcc preassign feature introduced in HBASE-16698 > - > > Key: HBASE-17507 > URL: https://issues.apache.org/jira/browse/HBASE-17507 > Project: HBase > Issue Type: Task >Reporter: Yu Li >Assignee: Yu Li > > In this JIRA we will supply a design doc of the mvcc preassign mechanism > introduced in HBASE-16698, as well as more detailed performance data of it > for better reference. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[ https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833414#comment-15833414 ] Yu Li commented on HBASE-17507: --- According to our study and prototype here, mvcc preassign will be some kind of limitation when we try to parallelize WAL sync (IO bound) and mvcc assigning (CPU bound) on write path, so let's just keep it configurable for a little bit longer (smile). > Add document for the mvcc preassign feature introduced in HBASE-16698 > - > > Key: HBASE-17507 > URL: https://issues.apache.org/jira/browse/HBASE-17507 > Project: HBase > Issue Type: Task >Reporter: Yu Li >Assignee: Yu Li > > In this JIRA we will supply a design doc of the mvcc preassign mechanism > introduced in HBASE-16698, as well as more detailed performance data of it > for better reference. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-16698:
--
Resolution: Fixed
Release Note:
Assign sequenceid to an edit before we go on the ringbuffer; undoes contention
on WALKey latch. Adds a new config "hbase.hregion.mvcc.preassign" which
defaults to true: i.e. this speedup is enabled.
User could set this per-table level, like:
create
'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hregion.mvcc.preassign'=>'false'}}
was:Assign sequenceid to an edit before we go on the ringbuffer; undoes
contention on WALKey latch. Adds a new config "hbase.hregion.mvcc.preassign"
which defaults to true: i.e. this speedup is enabled.
Status: Resolved (was: Patch Available)
Closing this JIRA per above comments.
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: hadoop0495.et2.jstack, HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
[ https://issues.apache.org/jira/browse/HBASE-17507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833403#comment-15833403 ] Duo Zhang commented on HBASE-17507: --- Can we always do preassign? I do not think the document can help end users to choose whether to use this feature... > Add document for the mvcc preassign feature introduced in HBASE-16698 > - > > Key: HBASE-17507 > URL: https://issues.apache.org/jira/browse/HBASE-17507 > Project: HBase > Issue Type: Task >Reporter: Yu Li >Assignee: Yu Li > > In this JIRA we will supply a design doc of the mvcc preassign mechanism > introduced in HBASE-16698, as well as more detailed performance data of it > for better reference. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17507) Add document for the mvcc preassign feature introduced in HBASE-16698
Yu Li created HBASE-17507: - Summary: Add document for the mvcc preassign feature introduced in HBASE-16698 Key: HBASE-17507 URL: https://issues.apache.org/jira/browse/HBASE-17507 Project: HBase Issue Type: Task Reporter: Yu Li Assignee: Yu Li In this JIRA we will supply a design doc of the mvcc preassign mechanism introduced in HBASE-16698, as well as more detailed performance data of it for better reference. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17487) Throw exception when pushing pipeline to snapshot fails twice
[ https://issues.apache.org/jira/browse/HBASE-17487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833402#comment-15833402 ] Anastasia Braginsky commented on HBASE-17487: - The snapshot() method's algorithm is as following: If snapshot is empty (previous flush to disk finished) a. Stop ongoing in-memory compaction (the stopping request may be missed if compaction is just about to end) b. Push active segment to compaction pipeline (as in-memory flush) c. Push pipeline to snapshot: # Push references to all pipeline segments to snapshot (the segments are now referenced both in pipeline and in snapshot) # Try to swap the pipeline, i.e. remove the references to the segments in snapshot if no concurrent activity is done now # If this is not successful (due to concurrent activity) reread the pipeline and redo the steps 1-2 again. # *Here is what we are talking about:* if for the third time the swap is unsuccessful (due to some unknown problem) the snapshot is nullified and we return with empty snapshot from the pushPipelineToSnapshot() method. All segments remain as they were in pipeline. d. The snapshot() method creates a MemStoreSnapshot object as an output. It can be also an empty MemStoreSnapshot object, this is a valid output. [~anoop.hbase], as you can see there is no buggy way of taking care in case when (unexpectedly) the swap fails in third time. The snapshot() method call fails as if the snapshot was not empty (previous flush to disk didn't finish). [~ted_yu], I agree to change the LOG from warm to error. Indeed a more correct to deal with it. > Throw exception when pushing pipeline to snapshot fails twice > - > > Key: HBASE-17487 > URL: https://issues.apache.org/jira/browse/HBASE-17487 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: 17487.v2.txt > > > In CompactingMemStore#pushPipelineToSnapshot() , there is limit of 3 > iterations of pipeline.swap() call after which an empty ImmutableSegment is > used as snapshot. > However, there should be at most two iterations in pushPipelineToSnapshot() > since during the second iteration there is no concurrent write to memstore. > We should throw exception in the 3rd iteration to signify that this scenario > should never happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833400#comment-15833400
]
Yu Li commented on HBASE-16698:
---
The JIRA here introduced mvcc preassign feature, but only implement for Put and
has some problems for Append/Increment. HBASE-17471 is a JIRA locating this
issue and targeting at fixing it (by [~allan163])
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: hadoop0495.et2.jstack, HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17045) Unify the implementation of small scan and regular scan
[ https://issues.apache.org/jira/browse/HBASE-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17045: -- Attachment: HBASE-17405-v6.patch With HBASE-17489 we do not need to modify TestPartialResultsFromClientSide anymore. > Unify the implementation of small scan and regular scan > --- > > Key: HBASE-17045 > URL: https://issues.apache.org/jira/browse/HBASE-17045 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17045.patch, HBASE-17045-v1.patch, > HBASE-17045-v2.patch, HBASE-17045-v3.patch, HBASE-17045-v4.patch, > HBASE-17045-v5.patch, HBASE-17405-v6.patch > > > See [~enis]'s comment in HBASE-16838 > https://issues.apache.org/jira/browse/HBASE-16838?focusedCommentId=15637803&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15637803 > But there is another scenario that we need small scan is that, we do not know > the stop row but we only want a small set of results. For example, in the > implementation of region locator, we will use small scan and set caching to 1 > as we only need one row. > So I think we need to add a new option(maybe called limit?) for the scan > object, and deprecate the small option. And the server side modification > should also be committed to branch-1 to simplify the logic of async client in > 2.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16873) WAL: SequenceId assign with less friction
[ https://issues.apache.org/jira/browse/HBASE-16873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833387#comment-15833387 ] Yu Li commented on HBASE-16873: --- Converting this JIRA from subtask of HBASE-16698 to an improvement, should be a follow up of HBASE-16698 instead of part of it. Just let me know if you have different opinion boss [~stack], thanks. > WAL: SequenceId assign with less friction > - > > Key: HBASE-16873 > URL: https://issues.apache.org/jira/browse/HBASE-16873 > Project: HBase > Issue Type: Improvement > Components: Performance, wal >Reporter: stack > > This is an issue to improve our current sequence id assignment. It has become > complex with lots of friction. > In the old days, a simple notion that the single consumer thread pulling from > the ringbuffer should assign all sequenceids seemed to make sense. It > probably had provenance in the old days when there was a single sequenceid > for a regionserver but seemed like a fine choice even after the move to > region-scoped sequenceids -- rather than regionserver scopce -- and then > beyond that, when region-scoped sequenceids were unified with mvcc. The > rationale ran, a single thread appending to the WAL can run without locks and > this single thread being the arbiter of order, seemed like the natural owner > of the sequenceid increment. > Along comes large-scale production deploy, HBASE-16698. It highlights an > oversight in the above reasoning; i.e. that the single RB consumer thread > must pass a synchronize block per region to do the sequence id update and the > spread between the call to append and actual assign of the sequence id on > other side of the RB is forcing a severe serialization when there is > opportunity for parallellism. > This issue is about taking this finding and doing better than the expedient > fix done on HBASE-16698. Can we do without the lock on the region getting the > sequenceid as we call append? Can we exploit the fact that the ringbuffer > txid is always incrementing as is the region mvcc/sequenceid? Can we use this > fact to do region sequenceid w/o taking a lock? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17045) Unify the implementation of small scan and regular scan
[ https://issues.apache.org/jira/browse/HBASE-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17045: -- Attachment: (was: HBASE-17405-v6.patch) > Unify the implementation of small scan and regular scan > --- > > Key: HBASE-17045 > URL: https://issues.apache.org/jira/browse/HBASE-17045 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17045.patch, HBASE-17045-v1.patch, > HBASE-17045-v2.patch, HBASE-17045-v3.patch, HBASE-17045-v4.patch, > HBASE-17045-v5.patch > > > See [~enis]'s comment in HBASE-16838 > https://issues.apache.org/jira/browse/HBASE-16838?focusedCommentId=15637803&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15637803 > But there is another scenario that we need small scan is that, we do not know > the stop row but we only want a small set of results. For example, in the > implementation of region locator, we will use small scan and set caching to 1 > as we only need one row. > So I think we need to add a new option(maybe called limit?) for the scan > object, and deprecate the small option. And the server side modification > should also be committed to branch-1 to simplify the logic of async client in > 2.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16873) WAL: SequenceId assign with less friction
[ https://issues.apache.org/jira/browse/HBASE-16873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-16873: -- Issue Type: Improvement (was: Sub-task) Parent: (was: HBASE-16698) > WAL: SequenceId assign with less friction > - > > Key: HBASE-16873 > URL: https://issues.apache.org/jira/browse/HBASE-16873 > Project: HBase > Issue Type: Improvement > Components: Performance, wal >Reporter: stack > > This is an issue to improve our current sequence id assignment. It has become > complex with lots of friction. > In the old days, a simple notion that the single consumer thread pulling from > the ringbuffer should assign all sequenceids seemed to make sense. It > probably had provenance in the old days when there was a single sequenceid > for a regionserver but seemed like a fine choice even after the move to > region-scoped sequenceids -- rather than regionserver scopce -- and then > beyond that, when region-scoped sequenceids were unified with mvcc. The > rationale ran, a single thread appending to the WAL can run without locks and > this single thread being the arbiter of order, seemed like the natural owner > of the sequenceid increment. > Along comes large-scale production deploy, HBASE-16698. It highlights an > oversight in the above reasoning; i.e. that the single RB consumer thread > must pass a synchronize block per region to do the sequence id update and the > spread between the call to append and actual assign of the sequence id on > other side of the RB is forcing a severe serialization when there is > opportunity for parallellism. > This issue is about taking this finding and doing better than the expedient > fix done on HBASE-16698. Can we do without the lock on the region getting the > sequenceid as we call append? Can we exploit the fact that the ringbuffer > txid is always incrementing as is the region mvcc/sequenceid? Can we use this > fact to do region sequenceid w/o taking a lock? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833383#comment-15833383
]
Yu Li commented on HBASE-16698:
---
HBASE-17482 is a fix for master branch with the same way as stated in [this
comment |
https://issues.apache.org/jira/browse/HBASE-16698?focusedCommentId=15576721&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15576721]
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: hadoop0495.et2.jstack, HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17045) Unify the implementation of small scan and regular scan
[ https://issues.apache.org/jira/browse/HBASE-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17045: -- Attachment: HBASE-17405-v6.patch Rebase after HBASE-17489. > Unify the implementation of small scan and regular scan > --- > > Key: HBASE-17045 > URL: https://issues.apache.org/jira/browse/HBASE-17045 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17045.patch, HBASE-17045-v1.patch, > HBASE-17045-v2.patch, HBASE-17045-v3.patch, HBASE-17045-v4.patch, > HBASE-17045-v5.patch, HBASE-17405-v6.patch > > > See [~enis]'s comment in HBASE-16838 > https://issues.apache.org/jira/browse/HBASE-16838?focusedCommentId=15637803&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15637803 > But there is another scenario that we need small scan is that, we do not know > the stop row but we only want a small set of results. For example, in the > implementation of region locator, we will use small scan and set caching to 1 > as we only need one row. > So I think we need to add a new option(maybe called limit?) for the scan > object, and deprecate the small option. And the server side modification > should also be committed to branch-1 to simplify the logic of async client in > 2.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833381#comment-15833381
]
Yu Li commented on HBASE-16698:
---
Will do below actions soon:
1. Close this JIRA since all codes already gone into master and branch-1
2. Open another JIRA to supply document mvcc preassign feature, including
* A design doc of the mvcc preassign mechanism and attach here
* A performance report for SYNC_WAL/ASYNC_WAL/SKIP_WAL with mvcc preassign.
(And we could open new JIRA if found any issue later)
3. Link newly found issue related with mvcc preassign with JIRA here.
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: hadoop0495.et2.jstack, HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[ https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17489: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-1.3+. Thanks all for reviewing. > ClientScanner may send a next request to a RegionScanner which has been > exhausted > - > > Key: HBASE-17489 > URL: https://issues.apache.org/jira/browse/HBASE-17489 > Project: HBase > Issue Type: Bug > Components: Client, scan >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-17489-branch-1.3.patch, > HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch, > HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch, > HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch > > > Found it when implementing HBASE-17045. Seems the final result of the scan is > correct but no doubt the logic is broken. We need to fix it to stop things > get worse in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833378#comment-15833378
]
Yu Li commented on HBASE-16698:
---
After some review of the mechanism, let me further explain about the
performance improvement for SYNC_WAL in theory for branch-1:
First, let me echo current process of doMiniBatchMutation for branch-1:
1. acquire locks
2. update timestamps
3. build WAL edit
4. append edit to WAL
5. write to memstore
6. sync WAL or defer
Just before step #5, we have below codes before patch:
{code}
//
// Acquire the latest mvcc number
// --
if (!isInReplay) {
writeEntry = walKey.getWriteEntry();
mvccNum = writeEntry.getWriteNumber();
} else {
mvccNum = batchOp.getReplaySequenceId();
}
//
// STEP 5. Write back to memstore
// Write to memstore. It is ok to write to memstore
// first without syncing the WAL because we do not roll
// forward the memstore MVCC. The MVCC will be moved up when
// the complete operation is done. These changes are not yet
// visible to scanners till we update the MVCC. The MVCC is
// moved only when the sync is complete.
// --
{code}
Where in {{walKey#getWriteEntry}} we will wait for the CountDownLatch, which
will be released by {{RingBufferEventHandler#append}} or say
{{FSWALEntry#stampRegionSequenceId}}. And since ringbuffer is handled in
sequence, this makes a contention for puts on all regions on the same
regionserver.
Notice that under high put overload this will make step #5 waiting, and in some
case this wait may be longer than the sync IO time, then when it arrives at
step #6, {{FSHLog#blockOnSync}} will return directly. This mainly answers below
question [~allan163] asked in previous comment.
bq. But, my question is, even if you solved this problem, the handlers still
have to waitting for syncOrDefer to complete
Let me prepare some doc and upload here to make it easier for understanding w/o
reading the long comment list (smile).
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: hadoop0495.et2.jstack, HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17506) started mvcc transaction is not completed in branch-1
[
https://issues.apache.org/jira/browse/HBASE-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17506:
---
Description:
In {{doMiniBatchMutation}}, if it is in replay and if the the nonce of the
mutation is different, we append them to a different wal. But, after
HBASE-14465, we start a mvcc transition in the ringbuffer's append thread. So,
every time we append a wal entry, we started a mvcc transition, but we didn't
complete the mvcc transition anywhere. This can block other transition of this
region.
{code}
// txid should always increase, so having the one from the last call is ok.
// we use HLogKey here instead of WALKey directly to support legacy
coprocessors.
walKey = new
ReplayHLogKey(this.getRegionInfo().getEncodedNameAsBytes(),
this.htableDescriptor.getTableName(), now, m.getClusterIds(),
currentNonceGroup, currentNonce, mvcc);
txid = this.wal.append(this.htableDescriptor,
this.getRegionInfo(), walKey,
walEdit, true);
walEdit = new WALEdit(cellCount, isInReplay);
walKey = null;
{code}
Looked at master branch, there is no such problem. It has a method
named{{appendCurrentNonces}} :
{code}
private void appendCurrentNonces(final Mutation mutation, final boolean replay,
final WALEdit walEdit, final long now, final long currentNonceGroup,
final long currentNonce)
throws IOException {
if (walEdit.isEmpty()) return;
if (!replay) throw new IOException("Multiple nonces per batch and not in
replay");
WALKey walKey = new WALKey(this.getRegionInfo().getEncodedNameAsBytes(),
this.htableDescriptor.getTableName(), now, mutation.getClusterIds(),
currentNonceGroup, currentNonce, mvcc, this.getReplicationScope());
this.wal.append(this.getRegionInfo(), walKey, walEdit, true);
// Complete the mvcc transaction started down in append else it will block
others
this.mvcc.complete(walKey.getWriteEntry());
}
{code}
Yes, the easiest way to fix branch-1 is to complete the writeEntry like master
branch do. But is it really fine to do this?
1. :
complete the mvcc transition before waiting sync will create a disturbance of
data visibility.
2.:
In what circumstance will there be different nonce and nonce group in a single
wal entry? Nonce are used in append/increment. But in {{batchMuate}} ,we treat
them differently and append one wal entry for each of them. So I think no test
can reach this code path, that maybe why no one has found this bug(Please tell
me if I'm wrong).
was:
In {{doMiniBatchMutation}}, if it is in replay and if the the nonce of the
mutation is different, we append them to a different wal. But, after
HBASE-14465, we start a mvcc transition in the ringbuffer's append thread. So,
every time we append a wal entry, we started a mvcc transition, but we didn't
complete the mvcc transition anywhere. This can block other transition of this
region.
{code}
// txid should always increase, so having the one from the last call is ok.
// we use HLogKey here instead of WALKey directly to support legacy
coprocessors.
walKey = new
ReplayHLogKey(this.getRegionInfo().getEncodedNameAsBytes(),
this.htableDescriptor.getTableName(), now, m.getClusterIds(),
currentNonceGroup, currentNonce, mvcc);
txid = this.wal.append(this.htableDescriptor,
this.getRegionInfo(), walKey,
walEdit, true);
walEdit = new WALEdit(cellCount, isInReplay);
walKey = null;
{code}
Looked at master branch, there is no such problem. It has a method
named{{appendCurrentNonces}} :
{code}
private void appendCurrentNonces(final Mutation mutation, final boolean replay,
final WALEdit walEdit, final long now, final long currentNonceGroup,
final long currentNonce)
throws IOException {
if (walEdit.isEmpty()) return;
if (!replay) throw new IOException("Multiple nonces per batch and not in
replay");
WALKey walKey = new WALKey(this.getRegionInfo().getEncodedNameAsBytes(),
this.htableDescriptor.getTableName(), now, mutation.getClusterIds(),
currentNonceGroup, currentNonce, mvcc, this.getReplicationScope());
this.wal.append(this.getRegionInfo(), walKey, walEdit, true);
// Complete the mvcc transaction started down in append else it will block
others
this.mvcc.complete(walKey.getWriteEntry());
}
{code}
Yes, the easiest way to fix branch-1 is to complete the writeEntry like master
branch do. But is it really fine to do this?
1. Question 1:
complete the mvcc transition before waiting sync will create a disturbance of
data visibility.
2.Question 2:
In what circumstance will there be different nonce and nonce group in a single
wal entry? Nonce are used in append/increment. But in {{batchMuate}} ,we treat
them differently and append one wal ent
[jira] [Commented] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[
https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833363#comment-15833363
]
Duo Zhang commented on HBASE-17489:
---
And for branch-1.3, {{TestMetaWithReplicas}} fails without the patch either.
Will commit shortly.
And for branch-1.2-, I do not think we will introduce new features on these
branches, so it is not a big deal to not apply the patch as the bug does not
impact correctness.
Thanks.
> ClientScanner may send a next request to a RegionScanner which has been
> exhausted
> -
>
> Key: HBASE-17489
> URL: https://issues.apache.org/jira/browse/HBASE-17489
> Project: HBase
> Issue Type: Bug
> Components: Client, scan
>Affects Versions: 2.0.0, 1.3.0, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0, 1.4.0, 1.3.1
>
> Attachments: HBASE-17489-branch-1.3.patch,
> HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch,
> HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch,
> HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch
>
>
> Found it when implementing HBASE-17045. Seems the final result of the scan is
> correct but no doubt the logic is broken. We need to fix it to stop things
> get worse in the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
