[jira] [Commented] (HBASE-17441) precommit test "hadoopcheck" not properly testing Hadoop 3 profile
[
https://issues.apache.org/jira/browse/HBASE-17441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814196#comment-15814196
]
Hadoop QA commented on HBASE-17441:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 42s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 78m 8s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 112m 43s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846494/HBASE-17441.0.patch |
| JIRA Issue | HBASE-17441 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4ee55b56af1a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ac3b1c9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5207/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5207/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> precommit test "hadoopcheck" not properly testing Hadoop 3 profile
> --
>
> Key: HBASE-17441
> URL: https://issues.apache.org/jira/browse/HBASE-17441
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versio
[jira] [Commented] (HBASE-17441) precommit test "hadoopcheck" not properly testing Hadoop 3 profile
[
https://issues.apache.org/jira/browse/HBASE-17441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814186#comment-15814186
]
Hadoop QA commented on HBASE-17441:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 33s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 82m 25s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 119m 46s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846494/HBASE-17441.0.patch |
| JIRA Issue | HBASE-17441 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 35ddb0002d73 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ac3b1c9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5206/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5206/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> precommit test "hadoopcheck" not properly testing Hadoop 3 profile
> --
>
> Key: HBASE-17441
> URL: https://issues.apache.org/jira/browse/HBASE-17441
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Ver
[jira] [Commented] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[ https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814115#comment-15814115 ] Appy commented on HBASE-16744: -- [~stack] This is the base patch and will integrate well on its own. HBASE-16786 is the work to move uses from zk locks to proc locks. If there is any conflict, it should be with this one. HBASE-16831 is to delete zk lock implementation which'll become obsolete once above two patches go in. HBASE-16867 is to handle security aspects, so that not just anyone can get lock through rpc. > Procedure V2 - Lock procedures to allow clients to acquire locks on > tables/namespaces/regions > - > > Key: HBASE-16744 > URL: https://issues.apache.org/jira/browse/HBASE-16744 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Matteo Bertozzi > Attachments: HBASE-16744.master.001.patch, > HBASE-16744.master.002.patch, HBASE-16744.master.003.patch, > HBASE-16744.master.004.patch, HBASE-16744.master.005.patch, > HBASE-16744.master.006.patch, HBASE-16744.master.007.patch, > HBASE-16744.master.008.patch, HBASE-16744.master.009.patch, > HBASE-16744.master.010.patch, HBASE-16744.master.011.patch, > HBASE-16744.master.012.patch, HBASE-16744.master.013.patch, > HBASE-16744.master.014.patch, HBASE-16744.master.015.patch > > > Will help us get rid of ZK locks. > Will be useful for external tools like hbck, future backup manager, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17045) Unify the implementation of small scan and regular scan
[ https://issues.apache.org/jira/browse/HBASE-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814091#comment-15814091 ] Duo Zhang commented on HBASE-17045: --- [~stack] [~enis] PTAL. Thanks. > Unify the implementation of small scan and regular scan > --- > > Key: HBASE-17045 > URL: https://issues.apache.org/jira/browse/HBASE-17045 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17045.patch > > > See [~enis]'s comment in HBASE-16838 > https://issues.apache.org/jira/browse/HBASE-16838?focusedCommentId=15637803&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15637803 > But there is another scenario that we need small scan is that, we do not know > the stop row but we only want a small set of results. For example, in the > implementation of region locator, we will use small scan and set caching to 1 > as we only need one row. > So I think we need to add a new option(maybe called limit?) for the scan > object, and deprecate the small option. And the server side modification > should also be committed to branch-1 to simplify the logic of async client in > 2.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17045) Unify the implementation of small scan and regular scan
[ https://issues.apache.org/jira/browse/HBASE-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17045: -- Assignee: Duo Zhang Affects Version/s: (was: 1.4.0) Status: Patch Available (was: Open) > Unify the implementation of small scan and regular scan > --- > > Key: HBASE-17045 > URL: https://issues.apache.org/jira/browse/HBASE-17045 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17045.patch > > > See [~enis]'s comment in HBASE-16838 > https://issues.apache.org/jira/browse/HBASE-16838?focusedCommentId=15637803&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15637803 > But there is another scenario that we need small scan is that, we do not know > the stop row but we only want a small set of results. For example, in the > implementation of region locator, we will use small scan and set caching to 1 > as we only need one row. > So I think we need to add a new option(maybe called limit?) for the scan > object, and deprecate the small option. And the server side modification > should also be committed to branch-1 to simplify the logic of async client in > 2.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17045) Unify the implementation of small scan and regular scan
[ https://issues.apache.org/jira/browse/HBASE-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17045: -- Attachment: HBASE-17045.patch Add a limit option to Scan which means the limit of rows for this scan operation. RS will close the scanner and set moreResults to false when the number of rows reaches the limit. Also did some refactoring when implementing the feature in RSRpcServices. In AsyncClientScanner, we will also fetch some cells when open scanner. Remove smallScan methods in AsyncTableBase and remove all small scan related code for asynchronous implementation. Add a scanAll method. Deprecated setSmall and isSmall methods for Scan. > Unify the implementation of small scan and regular scan > --- > > Key: HBASE-17045 > URL: https://issues.apache.org/jira/browse/HBASE-17045 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Affects Versions: 2.0.0, 1.4.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17045.patch > > > See [~enis]'s comment in HBASE-16838 > https://issues.apache.org/jira/browse/HBASE-16838?focusedCommentId=15637803&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15637803 > But there is another scenario that we need small scan is that, we do not know > the stop row but we only want a small set of results. For example, in the > implementation of region locator, we will use small scan and set caching to 1 > as we only need one row. > So I think we need to add a new option(maybe called limit?) for the scan > object, and deprecate the small option. And the server side modification > should also be committed to branch-1 to simplify the logic of async client in > 2.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17396) Add first async admin impl and implement balance methods
[ https://issues.apache.org/jira/browse/HBASE-17396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17396: --- Attachment: HBASE-17396-v5.patch > Add first async admin impl and implement balance methods > > > Key: HBASE-17396 > URL: https://issues.apache.org/jira/browse/HBASE-17396 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17396-v1.patch, HBASE-17396-v2.patch, > HBASE-17396-v3.patch, HBASE-17396-v4.patch, HBASE-17396-v5.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16786) Procedure V2 - Move ZK-lock's uses to Procedure framework locks (LockProcedure)
[ https://issues.apache.org/jira/browse/HBASE-16786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814042#comment-15814042 ] stack commented on HBASE-16786: --- Whats the downside [~jingcheng.du]? Delete markers are small. > Procedure V2 - Move ZK-lock's uses to Procedure framework locks > (LockProcedure) > --- > > Key: HBASE-16786 > URL: https://issues.apache.org/jira/browse/HBASE-16786 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Matteo Bertozzi > Attachments: HBASE-16786.master.001.patch, > HBASE-16786.master.002.patch, HBASE-16786.master.003.patch, > HBASE-16786.master.004.patch, HBASE-16786.master.005.patch, > HBASE-16786.master.006.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814027#comment-15814027 ] Jerry He commented on HBASE-17437: -- The hbase.rootdir has a couple of other things, like the version file and clusterId file. Do these need to be duplicated in the new wal root dir? Any relevant checking that needs to be done? For example, this hbase.rootdir and this wal root dir is not a mismatch. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17429) HBase bulkload cannot support HDFS viewFs
[
https://issues.apache.org/jira/browse/HBASE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814025#comment-15814025
]
Hudson commented on HBASE-17429:
FAILURE: Integrated in Jenkins build HBase-1.4 #590 (See
[https://builds.apache.org/job/HBase-1.4/590/])
HBASE-17429 HBase bulkload cannot support HDFS viewFs (shenxianqiang) (tedyu:
rev 2f7ce65b81cd137ba840351c14281c50fb682876)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
> HBase bulkload cannot support HDFS viewFs
> -
>
> Key: HBASE-17429
> URL: https://issues.apache.org/jira/browse/HBASE-17429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6.1, 1.2.4
> Environment: CDH5.7.0 hbase0.98.6
>Reporter: shenxianqiang
>Assignee: shenxianqiang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17429.patch
>
>
> Since hadoop cluster suppor fedration,hbase bulkload performance degrades
> dramatically. Even if hbase directory and bulkload directory in the same
> nameservice.
> {quote}
> 2017-01-04 21:58:40,919 INFO
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem: use bulkload file
> name is : hdfs://CloudTestNameNode2:8020
> 2017-01-04 21:58:40,924 ERROR org.apache.hadoop.ipc.RpcServer: Unexpected
> throwable object
> java.lang.IllegalArgumentException: Wrong FS:
> viewfs://nsX/user/test/I/_tmp/9cde5dde60374b1483b7d09b65258304.top, expected:
> hdfs://CloudTestNameNode2:8020
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:373)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:496)
> at
> org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:708)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3658)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3564)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3378)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29589)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16812) Fix mob locks and correctly override HMobStore.compact()
[ https://issues.apache.org/jira/browse/HBASE-16812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814023#comment-15814023 ] stack commented on HBASE-16812: --- [~jingcheng.du] Any time to work on this one sir? Thanks. > Fix mob locks and correctly override HMobStore.compact() > - > > Key: HBASE-16812 > URL: https://issues.apache.org/jira/browse/HBASE-16812 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Jingcheng Du >Priority: Minor > Attachments: HBASE-16812.master.001.patch > > > compact(CompactionContext compaction, CompactionThroughputController > throughputController) is [deprecated in 1.2.0 > release|https://github.com/apache/hbase/blob/rel/1.2.0/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java#L222]. > Store.java is also marked limited private. > Context: I was cleaning up zk table lock which is also used in that method's > [override|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L460] > in HMobStore. > This method isn't being called from anywhere except CompactionTool (which > creates HStore object, not HMobStore object). > [~jingcheng...@intel.com] Can you PTAL and help me understand what's going on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[ https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16744: -- Attachment: HBASE-16744.master.015.patch > Procedure V2 - Lock procedures to allow clients to acquire locks on > tables/namespaces/regions > - > > Key: HBASE-16744 > URL: https://issues.apache.org/jira/browse/HBASE-16744 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Matteo Bertozzi > Attachments: HBASE-16744.master.001.patch, > HBASE-16744.master.002.patch, HBASE-16744.master.003.patch, > HBASE-16744.master.004.patch, HBASE-16744.master.005.patch, > HBASE-16744.master.006.patch, HBASE-16744.master.007.patch, > HBASE-16744.master.008.patch, HBASE-16744.master.009.patch, > HBASE-16744.master.010.patch, HBASE-16744.master.011.patch, > HBASE-16744.master.012.patch, HBASE-16744.master.013.patch, > HBASE-16744.master.014.patch, HBASE-16744.master.015.patch > > > Will help us get rid of ZK locks. > Will be useful for external tools like hbck, future backup manager, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13652) Case-insensitivity of file system affects table creation
[
https://issues.apache.org/jira/browse/HBASE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813968#comment-15813968
]
Xuesen Liang commented on HBASE-13652:
--
hi [~larsgeorge],
can you review HBASE-13652.master.004.patch?
thanks!
> Case-insensitivity of file system affects table creation
>
>
> Key: HBASE-13652
> URL: https://issues.apache.org/jira/browse/HBASE-13652
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 1.0.0
> Environment: HBase standalone mode on Mac OS X.
>Reporter: Lars George
>Assignee: Xuesen Liang
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-13652.master.001.patch,
> HBASE-13652.master.002.patch, HBASE-13652.master.003.patch,
> HBASE-13652.master.004.patch
>
>
> I noticed this on my Mac OS machine:
> {noformat}
> hbase(main):003:0> list
> TABLE
>
>
> 0 row(s) in 0.0260 seconds
> => []
> hbase(main):004:0> create 'TestTable', 'info'
> 0 row(s) in 0.2830 seconds
> => Hbase::Table - TestTable
> hbase(main):005:0> create 'testtable', 'colfam1'
> 0 row(s) in 0.1750 seconds
> => Hbase::Table - testtable
> hbase(main):006:0> list
> TABLE
>
>
> TestTable
>
>
> TestTable
>
>
> 2 row(s) in 0.0170 seconds
> => ["TestTable", "TestTable"]
> hbase(main):007:0> status 'detailed'
> version 1.0.0
> 0 regionsInTransition
> master coprocessors: []
> 1 live servers
> de1-app-mba-1.internal.larsgeorge.com:58824 1431124680152
> requestsPerSecond=0.0, numberOfOnlineRegions=4, usedHeapMB=47,
> maxHeapMB=4062, numberOfStores=4, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=16, writeRequestsCount=8,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> coprocessors=[]
> "TestTable,,1431124762491.cb30b003b192aa1d429e5a8e908a2f7d."
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=0,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> completeSequenceId=-1, dataLocality=0.0
> "hbase:meta,,1"
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=10, writeRequestsCount=6,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> completeSequenceId=-1, dataLocality=0.0
> "hbase:namespace,,1431124686536.42728f3a3411b0d8ff21c7a2622d9378."
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=6, writeRequestsCount=2,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> completeSequenceId=-1, dataLocality=0.0
> "testtable,,1431124773352.8d461d5069d1c88bc3b26562ab30e333."
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=0,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> completeSequenceId=-1, dataLocality=0.0
> 0 dead servers
> {noformat}
> and on the file system there is
> {noformat}
> larsgeorge:~$ ls -l hbase/data/default/
> total 0
> drwxr-xr-x 7 larsgeorge staff 238 May 8 15:39 TestTable
> {noformat}
> This should not happen on a case sensitive file system (I presume), so is of
> marginal importance. But a simple extra check during directory creation could
> fix this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13652) Case-insensitivity of file system affects table creation
[
https://issues.apache.org/jira/browse/HBASE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813959#comment-15813959
]
Xuesen Liang commented on HBASE-13652:
--
test in hbase shell on MAC:
{code}
liangxuesen$ ./bin/hbase shell
hbase(main):001:0> list
TABLE
0 row(s)
Took 0.4970 seconds
hbase(main):002:0> create 'test_table', 'cf'
Created table test_table
Took 0.3050 seconds
hbase(main):003:0> create 'TEST_table', 'cf'
ERROR: Table already exists: TEST_table! (on case-insensitive filesystem)
... skipped ...
Took 0.2490 seconds
hbase(main):004:0> create 'test_table', 'cf'
ERROR: Table already exists: test_table!
... skipped ...
Took 0.2260 seconds
hbase(main):005:0> list
TABLE
test_table
1 row(s)
Took 0.0120 seconds
{code}
> Case-insensitivity of file system affects table creation
>
>
> Key: HBASE-13652
> URL: https://issues.apache.org/jira/browse/HBASE-13652
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 1.0.0
> Environment: HBase standalone mode on Mac OS X.
>Reporter: Lars George
>Assignee: Xuesen Liang
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-13652.master.001.patch,
> HBASE-13652.master.002.patch, HBASE-13652.master.003.patch,
> HBASE-13652.master.004.patch
>
>
> I noticed this on my Mac OS machine:
> {noformat}
> hbase(main):003:0> list
> TABLE
>
>
> 0 row(s) in 0.0260 seconds
> => []
> hbase(main):004:0> create 'TestTable', 'info'
> 0 row(s) in 0.2830 seconds
> => Hbase::Table - TestTable
> hbase(main):005:0> create 'testtable', 'colfam1'
> 0 row(s) in 0.1750 seconds
> => Hbase::Table - testtable
> hbase(main):006:0> list
> TABLE
>
>
> TestTable
>
>
> TestTable
>
>
> 2 row(s) in 0.0170 seconds
> => ["TestTable", "TestTable"]
> hbase(main):007:0> status 'detailed'
> version 1.0.0
> 0 regionsInTransition
> master coprocessors: []
> 1 live servers
> de1-app-mba-1.internal.larsgeorge.com:58824 1431124680152
> requestsPerSecond=0.0, numberOfOnlineRegions=4, usedHeapMB=47,
> maxHeapMB=4062, numberOfStores=4, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=16, writeRequestsCount=8,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> coprocessors=[]
> "TestTable,,1431124762491.cb30b003b192aa1d429e5a8e908a2f7d."
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=0,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> completeSequenceId=-1, dataLocality=0.0
> "hbase:meta,,1"
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=10, writeRequestsCount=6,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> completeSequenceId=-1, dataLocality=0.0
> "hbase:namespace,,1431124686536.42728f3a3411b0d8ff21c7a2622d9378."
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=6, writeRequestsCount=2,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> completeSequenceId=-1, dataLocality=0.0
> "testtable,,1431124773352.8d461d5069d1c88bc3b26562ab30e333."
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=0,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> complet
[jira] [Updated] (HBASE-17441) precommit test "hadoopcheck" not properly testing Hadoop 3 profile
[ https://issues.apache.org/jira/browse/HBASE-17441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-17441: Attachment: HBASE-17441.0.patch -00 - noop patch. since hbase-server in master is broken against hadoop 3, should fail hadoopcheck. > precommit test "hadoopcheck" not properly testing Hadoop 3 profile > -- > > Key: HBASE-17441 > URL: https://issues.apache.org/jira/browse/HBASE-17441 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17441.0.patch > > > HBASE-14061 made a change that caused building against hadoop 3 to fail, but > the hadoopcheck precommit test gave the change a +1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17441) precommit test "hadoopcheck" not properly testing Hadoop 3 profile
[ https://issues.apache.org/jira/browse/HBASE-17441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-17441: Status: Patch Available (was: Open) > precommit test "hadoopcheck" not properly testing Hadoop 3 profile > -- > > Key: HBASE-17441 > URL: https://issues.apache.org/jira/browse/HBASE-17441 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-17441.0.patch > > > HBASE-14061 made a change that caused building against hadoop 3 to fail, but > the hadoopcheck precommit test gave the change a +1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813885#comment-15813885 ] stack commented on HBASE-17434: --- [~eshcar] You have a bunch of +1s but there are a few nits. You want to do them here? Here is the list: * [~manju_hadoop] says "... With the current patch I would have liked to see either a synchronized(this) or make the pipeline variable as final which would prevent inadvertent resetting of pipeline to new LinkedList in future. This is again a very minor observation so we should be good even without these." * [~anoop.hbase] asks about version++ (see above). * [~Apache9] suggests volatile readOnlyCopy * I have a nit over on RB about use of a synchronized pipeline when it doesn't seem necessary. Answer the above items and can commit. Thanks boss. > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813878#comment-15813878 ] stack commented on HBASE-17434: --- Thank you [~manju_hadoop] for the review. There are a few nits on this patch. I'll add yours to the list on the end here... Thanks. > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17409) Re-fix XSS request issue in JMXJsonServlet
[ https://issues.apache.org/jira/browse/HBASE-17409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813872#comment-15813872 ] Nick Dimiduk commented on HBASE-17409: -- ::thumbsup:: > Re-fix XSS request issue in JMXJsonServlet > -- > > Key: HBASE-17409 > URL: https://issues.apache.org/jira/browse/HBASE-17409 > Project: HBase > Issue Type: Sub-task > Components: security, UI >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8 > > Attachments: HBASE-17409.001.patch, HBASE-17409.002.patch > > > I have a patch here which should mitigate the XSS issue in this servlet > without the use of owasp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813841#comment-15813841 ] ramkrishna.s.vasudevan commented on HBASE-17434: Overall +1 for this patch. > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13652) Case-insensitivity of file system affects table creation
[
https://issues.apache.org/jira/browse/HBASE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813789#comment-15813789
]
Hadoop QA commented on HBASE-13652:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m 52s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 51s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846470/HBASE-13652.master.004.patch
|
| JIRA Issue | HBASE-13652 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile rubocop ruby_lint |
| uname | Linux 2d73f949f761 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ac3b1c9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-
[jira] [Commented] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[ https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813785#comment-15813785 ] Ted Yu commented on HBASE-17436: Looks like JMX is the most accessible choice for users. > Add facility to provide more information for Other Regions seen on Master UI > > > Key: HBASE-17436 > URL: https://issues.apache.org/jira/browse/HBASE-17436 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu > > [~rpednekar] and I were looking at a case where the count displayed under > Other Regions was high (~1200). > Since the table page just maintains a count instead of List of region names, > it is very difficult for user to determine which regions belong to this > category. > We should provide facility to provide more details for this category (LOG, > JMX, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15995) Separate replication WAL reading from shipping
[ https://issues.apache.org/jira/browse/HBASE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813775#comment-15813775 ] Phil Yang commented on HBASE-15995: --- Yes, the overflow only depends size of one Entry in each thread and number of threads, which is safe unless the they are too large. > Separate replication WAL reading from shipping > -- > > Key: HBASE-15995 > URL: https://issues.apache.org/jira/browse/HBASE-15995 > Project: HBase > Issue Type: Sub-task > Components: Replication >Affects Versions: 2.0.0 >Reporter: Vincent Poon >Assignee: Vincent Poon > Fix For: 2.0.0 > > Attachments: HBASE-15995.master.v1.patch, > HBASE-15995.master.v2.patch, HBASE-15995.master.v3.patch, > replicationV1_100ms_delay.png, replicationV2_100ms_delay.png > > > Currently ReplicationSource reads edits from the WAL and ships them in the > same thread. > By breaking out the reading from the shipping, we can introduce greater > parallelism and lay the foundation for further refactoring to a pipelined, > streaming model. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[
https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813763#comment-15813763
]
Anoop Sam John edited comment on HBASE-17434 at 1/10/17 3:49 AM:
-
In RB, I asked one Q regarding doing a version++ newly in the patch.. Pls
correct me if I read it wrong.
{code}
swapSuffix(suffix, segment, closeSuffix);
133 readOnlyCopy = new LinkedList<>(pipeline);
134 version++;
{code}
This one where swapSuffix is doing a version++ within it and we newly added it
again here..
was (Author: anoop.hbase):
In RB, I asked one Q regarding doing a version++ newly in the patch.. Pls
correct me if I read it wrong.
> New Synchronization Scheme for Compaction Pipeline
> --
>
> Key: HBASE-17434
> URL: https://issues.apache.org/jira/browse/HBASE-17434
> Project: HBase
> Issue Type: Bug
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch,
> HBASE-17434-V03.patch, HBASE-17434.master.001.patch
>
>
> A new copyOnWrite synchronization scheme is introduced for the compaction
> pipeline.
> The new scheme is better since it removes the lock from getSegments() which
> is invoked in every get and scan operation, and it reduces the number of
> LinkedList objects that are created at runtime, thus can reduce GC (not by
> much, but still...).
> In addition, it fixes the method getTailSize() in compaction pipeline. This
> method creates a MemstoreSize object which comprises the data size and the
> overhead size of the segment and needs to be atomic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813763#comment-15813763 ] Anoop Sam John commented on HBASE-17434: In RB, I asked one Q regarding doing a version++ newly in the patch.. Pls correct me if I read it wrong. > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17315) [C++] HBase Client and Table Implementation
[
https://issues.apache.org/jira/browse/HBASE-17315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813759#comment-15813759
]
Sudeep Sunthankar commented on HBASE-17315:
---
Thanks [~enis]
{quote}
{code}
hbase::TestUtil *test_util = new hbase::TestUtil();
..
+ delete test_util;
{code}
{quote}
There was a feedback in one of the earlier issues, not to include smart
pointers in test cases, hence I refrained from it.
{quote}
{code}
// ASSERT_TRUE(table != nullptr) << "Unable to get connection to Table.";
{code}
{quote}
Since we moved, it from unique_ptr to return by value, I had commented out the
part.
I will be uncomment it in the next patch and also include changes for
RequestConverter and ResponseConverter classes.
> [C++] HBase Client and Table Implementation
> ---
>
> Key: HBASE-17315
> URL: https://issues.apache.org/jira/browse/HBASE-17315
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-17315.HBASE-14850.v1.patch,
> HBASE-17315.HBASE-14850.v2.patch, HBASE-17315.HBASE-14850.v3.patch,
> HBASE-17315.HBASE-14850.v4.patch, HBASE-17315.HBASE-14850.v5.patch
>
>
> Consists of Client and Table implementation which will be used to call the
> corresponding client methods i.e Get, Gets, Scan etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[
https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813757#comment-15813757
]
Hadoop QA commented on HBASE-16744:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 59s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 18m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
59s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 19s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 28s
{color} | {color:red} hbase-protocol-shaded in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 16s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 24s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 14s
{color} | {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 29s
{color} | {color:red} hbase-protocol-shaded in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 19s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 26s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 14s
{color} | {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 29s {color} |
{color:red} hbase-protocol-shaded in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 19s {color} |
{color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 26s {color} |
{color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 14s {color} |
{color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 29s {color}
| {color:red} hbase-protocol-shaded in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 19s {color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 26s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 14s {color}
| {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 19m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 43 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 43s
{color} | {color:red} The patch causes 44 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{
[jira] [Commented] (HBASE-17431) Incorrect precheck condition in RoundRobinPool#get()
[
https://issues.apache.org/jira/browse/HBASE-17431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813753#comment-15813753
]
Ted Yu commented on HBASE-17431:
Jan:
More investigation is needed.
Let me know if you need help.
> Incorrect precheck condition in RoundRobinPool#get()
>
>
> Key: HBASE-17431
> URL: https://issues.apache.org/jira/browse/HBASE-17431
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Jan Hentschel
>Priority: Minor
> Attachments: HBASE-17431.master.001.patch,
> HBASE-17431.master.002.patch
>
>
> Here is related code:
> {code}
> public R get() {
> if (super.size() < maxSize) {
> return null;
> }
> nextResource %= super.size();
> {code}
> Since super.size() is involved in modulo operation after the check, it seems
> the check should compare against 0 instead of maxSize.
> Looks like a copy-paste error from put() method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813746#comment-15813746 ] Manjunath Anand commented on HBASE-17434: - Hi [~stack] thanks for adding me as contributor and I accept that ReentrantReadWriteLock idiom is complex in this case based on the below comment regarding the read patterns from [~eshcar] . With the current patch I would have liked to see either a synchronized(this) or make the pipeline variable as final which would prevent inadvertent resetting of pipeline to new LinkedList in future. This is again a very minor observation so we should be good even without these. > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17408) Introduce per request limit by number of mutations
[ https://issues.apache.org/jira/browse/HBASE-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17408: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Introduce per request limit by number of mutations > -- > > Key: HBASE-17408 > URL: https://issues.apache.org/jira/browse/HBASE-17408 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Ted Yu >Assignee: ChiaPing Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17408.v0.patch, HBASE-17408.v1.patch, > HBASE-17408.v2.patch > > > HBASE-16224 introduced hbase.client.max.perrequest.heapsize to limit the > amount of data sent from client. > We should consider adding per request limit through the number of mutations > in a batch. > In recent troubleshooting sessions, customer had to do this in their > application code to avoid OOME on the server side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813715#comment-15813715 ] Hudson commented on HBASE-17434: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2290 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2290/]) Revert "HBASE-17434 New Synchronization Scheme for Compaction Pipeline (stack: rev bd157ffe9a72cd723f238f1685608da573bb0df7) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17337) list replication peers request should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813714#comment-15813714 ] Hudson commented on HBASE-17337: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2290 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2290/]) HBASE-17337 list replication peers request should be routed through (zghao: rev ac3b1c9aa9d3ce5514afec0c2e0e34f9ac12d772) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeerDescription.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationSerDeHelper.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/MasterProtos.java * (edit) hbase-protocol-shaded/src/main/protobuf/Replication.proto * (edit) src/main/asciidoc/_chapters/appendix_acl_matrix.adoc * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/ReplicationProtos.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/replication/ReplicationManager.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/RequestConverter.java * (edit) hbase-protocol-shaded/src/main/protobuf/Master.proto * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java > list replication peers request should be routed through master > -- > > Key: HBASE-17337 > URL: https://issues.apache.org/jira/browse/HBASE-17337 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17337-v1.patch, HBASE-17337-v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17429) HBase bulkload cannot support HDFS viewFs
[
https://issues.apache.org/jira/browse/HBASE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813713#comment-15813713
]
Hudson commented on HBASE-17429:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2290 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2290/])
HBASE-17429 HBase bulkload cannot support HDFS viewFs (shenxianqiang) (tedyu:
rev 8dd35631cab14e9ae8ac64b4cd034712f7e6d8e9)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
> HBase bulkload cannot support HDFS viewFs
> -
>
> Key: HBASE-17429
> URL: https://issues.apache.org/jira/browse/HBASE-17429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6.1, 1.2.4
> Environment: CDH5.7.0 hbase0.98.6
>Reporter: shenxianqiang
>Assignee: shenxianqiang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17429.patch
>
>
> Since hadoop cluster suppor fedration,hbase bulkload performance degrades
> dramatically. Even if hbase directory and bulkload directory in the same
> nameservice.
> {quote}
> 2017-01-04 21:58:40,919 INFO
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem: use bulkload file
> name is : hdfs://CloudTestNameNode2:8020
> 2017-01-04 21:58:40,924 ERROR org.apache.hadoop.ipc.RpcServer: Unexpected
> throwable object
> java.lang.IllegalArgumentException: Wrong FS:
> viewfs://nsX/user/test/I/_tmp/9cde5dde60374b1483b7d09b65258304.top, expected:
> hdfs://CloudTestNameNode2:8020
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:373)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:496)
> at
> org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:708)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3658)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3564)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3378)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29589)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-17442) Move most of the replication related classes to hbase-server package
[ https://issues.apache.org/jira/browse/HBASE-17442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-17442: -- Assignee: Guanghao Zhang > Move most of the replication related classes to hbase-server package > > > Key: HBASE-17442 > URL: https://issues.apache.org/jira/browse/HBASE-17442 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > > After the replication requests are routed through master, replication > implementation details didn't need be exposed to client. We should move most > of the replication related classes to hbase-server package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17429) HBase bulkload cannot support HDFS viewFs
[
https://issues.apache.org/jira/browse/HBASE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17429:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
2.0.0
Status: Resolved (was: Patch Available)
> HBase bulkload cannot support HDFS viewFs
> -
>
> Key: HBASE-17429
> URL: https://issues.apache.org/jira/browse/HBASE-17429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6.1, 1.2.4
> Environment: CDH5.7.0 hbase0.98.6
>Reporter: shenxianqiang
>Assignee: shenxianqiang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17429.patch
>
>
> Since hadoop cluster suppor fedration,hbase bulkload performance degrades
> dramatically. Even if hbase directory and bulkload directory in the same
> nameservice.
> {quote}
> 2017-01-04 21:58:40,919 INFO
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem: use bulkload file
> name is : hdfs://CloudTestNameNode2:8020
> 2017-01-04 21:58:40,924 ERROR org.apache.hadoop.ipc.RpcServer: Unexpected
> throwable object
> java.lang.IllegalArgumentException: Wrong FS:
> viewfs://nsX/user/test/I/_tmp/9cde5dde60374b1483b7d09b65258304.top, expected:
> hdfs://CloudTestNameNode2:8020
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:373)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:496)
> at
> org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:708)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3658)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3564)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3378)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29589)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13788) Shell commands do not support column qualifiers containing colon (:)
[
https://issues.apache.org/jira/browse/HBASE-13788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813639#comment-15813639
]
Pankaj Kumar commented on HBASE-13788:
--
My bad, almost forgot this. Please feel free to take this.
> Shell commands do not support column qualifiers containing colon (:)
>
>
> Key: HBASE-13788
> URL: https://issues.apache.org/jira/browse/HBASE-13788
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 0.98.0, 0.96.0, 1.0.0, 1.1.0
>Reporter: Dave Latham
>Assignee: Pankaj Kumar
>
> The shell interprets the colon within the qualifier as a delimiter to a
> FORMATTER instead of part of the qualifier itself.
> Example from the mailing list:
> Hmph, I may have spoken too soon. I know I tested this at one point and
> it worked, but now I'm getting different results:
> On the new cluster, I created a duplicate test table:
> hbase(main):043:0> create 'content3', {NAME => 'x', BLOOMFILTER =>
> 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION =>
> 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', BLOCKSIZE => '65536',
> IN_MEMORY => 'false', BLOCKCACHE => 'true'}
> Then I pull some data from the imported table:
> hbase(main):045:0> scan 'content', {LIMIT=>1,
> STARTROW=>'A:9223370612089311807:twtr:57013379'}
> ROW COLUMN+CELL
>
> A:9223370612089311807:twtr:570133798827921408
> column=x:twitter:username, timestamp=1424775595345, value=BERITA &
> INFORMASI!
> Then put it:
> hbase(main):046:0> put
> 'content3','A:9223370612089311807:twtr:570133798827921408',
> 'x:twitter:username', 'BERITA & INFORMASI!'
> But then when I query it, I see that I've lost the column qualifier
> ":username":
> hbase(main):046:0> scan 'content3'
> ROW COLUMN+CELL
> A:9223370612089311807:twtr:570133798827921408 column=x:twitter,
> timestamp=1432745301788, value=BERITA & INFORMASI!
> Even though I'm missing one of the qualifiers, I can at least filter on
> columns in this sample table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[ https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813596#comment-15813596 ] stack commented on HBASE-16744: --- -014 rebases your patch [~appy] Is it not going to work w/o including other patches? We'll see. I need to finish locking. Current zkbased locks don't play w/ procedures needing to be idempotent... > Procedure V2 - Lock procedures to allow clients to acquire locks on > tables/namespaces/regions > - > > Key: HBASE-16744 > URL: https://issues.apache.org/jira/browse/HBASE-16744 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Matteo Bertozzi > Attachments: HBASE-16744.master.001.patch, > HBASE-16744.master.002.patch, HBASE-16744.master.003.patch, > HBASE-16744.master.004.patch, HBASE-16744.master.005.patch, > HBASE-16744.master.006.patch, HBASE-16744.master.007.patch, > HBASE-16744.master.008.patch, HBASE-16744.master.009.patch, > HBASE-16744.master.010.patch, HBASE-16744.master.011.patch, > HBASE-16744.master.012.patch, HBASE-16744.master.013.patch, > HBASE-16744.master.014.patch > > > Will help us get rid of ZK locks. > Will be useful for external tools like hbck, future backup manager, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[ https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16744: -- Attachment: HBASE-16744.master.014.patch > Procedure V2 - Lock procedures to allow clients to acquire locks on > tables/namespaces/regions > - > > Key: HBASE-16744 > URL: https://issues.apache.org/jira/browse/HBASE-16744 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Matteo Bertozzi > Attachments: HBASE-16744.master.001.patch, > HBASE-16744.master.002.patch, HBASE-16744.master.003.patch, > HBASE-16744.master.004.patch, HBASE-16744.master.005.patch, > HBASE-16744.master.006.patch, HBASE-16744.master.007.patch, > HBASE-16744.master.008.patch, HBASE-16744.master.009.patch, > HBASE-16744.master.010.patch, HBASE-16744.master.011.patch, > HBASE-16744.master.012.patch, HBASE-16744.master.013.patch, > HBASE-16744.master.014.patch > > > Will help us get rid of ZK locks. > Will be useful for external tools like hbck, future backup manager, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813558#comment-15813558 ] Hudson commented on HBASE-12148: FAILURE: Integrated in Jenkins build HBase-1.4 #589 (See [https://builds.apache.org/job/HBase-1.4/589/]) HBASE-12148 Remove TimeRangeTracker as point of contention when many (stack: rev 6130ea4d54d1a904de220ebad57ddea7b45797f0) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTimeRangeTracker.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/ClassSize.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: huaxiang sun > Fix For: 2.0.0, 1.4.0 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.min_and_max_run_independent.patch, 12148.txt, > 12148.txt, 12148v2.txt, 12148v2.txt, 12148v4.patch, HBASE-12148-V3.patch, > HBASE-12148-V3.patch, HBASE-12148-master-v6.patch, > HBASE-12148-master-v7.patch, HBASE-12148-master-v7.patch, > HBASE-12148.branch-1.v5.patch, HBASE-12148.branch-1.v5.patch, > HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot 2014-10-01 at 3.39.46 PM.png, > Screen Shot 2014-10-01 at 3.41.07 PM.png, Screen Shot 2016-04-13 at 1.49.30 > PM.png, Screen Shot 2016-04-13 at 2.02.22 PM.png, Screen Shot 2016-05-18 at > 10.21.53 PM.png, TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17315) [C++] HBase Client and Table Implementation
[
https://issues.apache.org/jira/browse/HBASE-17315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813544#comment-15813544
]
Enis Soztutar commented on HBASE-17315:
---
Thanks Sudeep for the updated patch.
No need for optional here:
{code}
+ std::shared_ptr> conf_;
{code}
Change these two to false for now (since they won't be implemented in the
initial cut):
{code}
+ pb_msg->set_client_handles_partials(true);
+ pb_msg->set_client_handles_heartbeats(true);
{code}
Maybe move this class
{code}
/hbase-native-client/core/protobuf_request_builder.cc
{code}
to the {{serde}} directory and name it request-converter. Also extract out this
logic:
{code}
+ for (auto cell : get_resp->result().cell()) {
+std::shared_ptr pcell =
+std::make_shared(cell.row(), cell.family(), cell.qualifier(),
cell.timestamp(),
+ cell.value(),
static_cast(cell.cell_type()));
+vcells.push_back(pcell);
+ }
+
+ hbase::Result result(vcells, get_resp->result().exists(),
get_resp->result().stale(),
+ get_resp->result().partial());
{code}
into a class called response-converter.
Also we have discussed internally that retuning via unique_ptr's is better for
Table::Get() and Client::Table methods for now. We can always revisit later.
Why this?
{code}
hbase::TestUtil *test_util = new hbase::TestUtil();
..
+ delete test_util;
{code}
Why not unique_ptr, and release()?
Looking at the way you use configuration for tests, maybe we should do
Conf.set(), etc methods, and maybe do a TestConfigurationLoader or something
which is not XML-file based. This way it will be much easier for future tests.
We can do this in a later patch though, no need to change this now.
Did you want to enable this assertion?
{code}
+ // ASSERT_TRUE(table != nullptr) << "Unable to get connection to Table.";
{code}
Otherwise looks pretty good.
> [C++] HBase Client and Table Implementation
> ---
>
> Key: HBASE-17315
> URL: https://issues.apache.org/jira/browse/HBASE-17315
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-17315.HBASE-14850.v1.patch,
> HBASE-17315.HBASE-14850.v2.patch, HBASE-17315.HBASE-14850.v3.patch,
> HBASE-17315.HBASE-14850.v4.patch, HBASE-17315.HBASE-14850.v5.patch
>
>
> Consists of Client and Table implementation which will be used to call the
> corresponding client methods i.e Get, Gets, Scan etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13652) Case-insensitivity of file system affects table creation
[
https://issues.apache.org/jira/browse/HBASE-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuesen Liang updated HBASE-13652:
-
Attachment: HBASE-13652.master.004.patch
Add shell error message.
> Case-insensitivity of file system affects table creation
>
>
> Key: HBASE-13652
> URL: https://issues.apache.org/jira/browse/HBASE-13652
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 1.0.0
> Environment: HBase standalone mode on Mac OS X.
>Reporter: Lars George
>Assignee: Xuesen Liang
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-13652.master.001.patch,
> HBASE-13652.master.002.patch, HBASE-13652.master.003.patch,
> HBASE-13652.master.004.patch
>
>
> I noticed this on my Mac OS machine:
> {noformat}
> hbase(main):003:0> list
> TABLE
>
>
> 0 row(s) in 0.0260 seconds
> => []
> hbase(main):004:0> create 'TestTable', 'info'
> 0 row(s) in 0.2830 seconds
> => Hbase::Table - TestTable
> hbase(main):005:0> create 'testtable', 'colfam1'
> 0 row(s) in 0.1750 seconds
> => Hbase::Table - testtable
> hbase(main):006:0> list
> TABLE
>
>
> TestTable
>
>
> TestTable
>
>
> 2 row(s) in 0.0170 seconds
> => ["TestTable", "TestTable"]
> hbase(main):007:0> status 'detailed'
> version 1.0.0
> 0 regionsInTransition
> master coprocessors: []
> 1 live servers
> de1-app-mba-1.internal.larsgeorge.com:58824 1431124680152
> requestsPerSecond=0.0, numberOfOnlineRegions=4, usedHeapMB=47,
> maxHeapMB=4062, numberOfStores=4, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=16, writeRequestsCount=8,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> coprocessors=[]
> "TestTable,,1431124762491.cb30b003b192aa1d429e5a8e908a2f7d."
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=0,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> completeSequenceId=-1, dataLocality=0.0
> "hbase:meta,,1"
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=10, writeRequestsCount=6,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> completeSequenceId=-1, dataLocality=0.0
> "hbase:namespace,,1431124686536.42728f3a3411b0d8ff21c7a2622d9378."
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=6, writeRequestsCount=2,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> completeSequenceId=-1, dataLocality=0.0
> "testtable,,1431124773352.8d461d5069d1c88bc3b26562ab30e333."
> numberOfStores=1, numberOfStorefiles=0,
> storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=0,
> rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0,
> totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
> completeSequenceId=-1, dataLocality=0.0
> 0 dead servers
> {noformat}
> and on the file system there is
> {noformat}
> larsgeorge:~$ ls -l hbase/data/default/
> total 0
> drwxr-xr-x 7 larsgeorge staff 238 May 8 15:39 TestTable
> {noformat}
> This should not happen on a case sensitive file system (I presume), so is of
> marginal importance. But a simple extra check during directory creation could
> fix this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813502#comment-15813502 ] Duo Zhang commented on HBASE-17434: --- Consider declaring readOnlyCopy as volatile if you want to access it without lock in some methods. Looks good overall. +1. > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[
https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16744:
--
Release Note:
Locks for table/namespace/regions. Use {@link LockServiceClient} to build
instances.
These are remote locks which live on master, and need periodic heartbeats to
keep them alive. (Once we requested the lock, internally an heartbeat thread
will be started). If master doesn't receive the heartbeat in time, it'll
release the lock and make it available to other users.
{@link #requestLock} will contact master to queue the lock and start the
heartbeat thread which'll check lock's status periodically and once the lock is
acquired, it will send the heartbeats to the master.
Use {@link #await} or {@link #await(long, TimeUnit)} to wait for the lock to be
acquired. Always call {@link #unlock()} irrespective of whether lock was
acquired or not. If the lock was acquired, it'll be released. If it was not
acquired, it's possible that master grants the lock in future and the heartbeat
thread keeps it alive forever by sending heartbeats. Calling {@link #unlock()}
will stop the heartbeat thread and cancel the lock queued on master.
There are 4 ways in which these remote locks may be released/can be lost:
* Call {@link #unlock}.
* Lock times out on master: Can happen because of network issues, GC pauses,
etc.
* Worker thread will call the given abortable as soon as it detects such a
situation.
* Fail to contact master: If worker thread can not contact mater and thus
fails to send heartbeat before the timeout expires, it assumes that lock is
lost and calls the abortable.
* Worker thread is interrupted.
Use example:
EntityLock lock = lockServiceClient.*Lock(, "exampled lock", abortable);
lock.requestLock();
can do other initializations here since lock is 'asynchronous'...
if (lock.await(timeout)) {
logic requiring mutual exclusion
}
lock.unlock();
was:
Locks for table/namespace/regions. Use {@link LockServiceClient} to build
instances.
These are remote locks which live on master, and need periodic heartbeats to
keep them alive. (Once we requested the lock, internally an heartbeat thread
will be started). If master doesn't receive the heartbeat in time, it'll
release the lock and make it available to other users.
{@link #requestLock} will contact master to queue the lock and start the
heartbeat thread which'll check lock's status periodically and once the lock is
acquired, it will send the heartbeats to the master.
Use {@link #await} or {@link #await(long, TimeUnit)} to wait for the lock to be
acquired. Always call {@link #unlock()} irrespective of whether lock was
acquired or not. If the lock was acquired, it'll be released. If it was not
acquired, it's possible that master grants the lock in future and the heartbeat
thread keeps it alive forever by sending heartbeats. Calling {@link #unlock()}
will stop the heartbeat thread and cancel the lock queued on master.
There are 4 ways in which these remote locks may be released/can be lost:
* Call {@link #unlock}.
* Lock times out on master: Can happen because of network issues, GC pauses,
etc.
* Worker thread will call the given abortable as soon as it detects such a
situation.
* Fail to contact master: If worker thread can not contact mater and thus
fails to send heartbeat before the timeout expires, it assumes that lock is
lost and calls the abortable.
* Worker thread is interrupted.
Use example:
EntityLock lock = lockServiceClient.*Lock(, "exampled lock", abortable);
lock.requestLock();
can do other initializations here since lock is 'asynchronous'...
if (lock.await(timeout)) {
logic requiring mutual exclusion
}
lock.unlock();
> Procedure V2 - Lock procedures to allow clients to acquire locks on
> tables/namespaces/regions
> -
>
> Key: HBASE-16744
> URL: https://issues.apache.org/jira/browse/HBASE-16744
> Project: HBase
> Issue Type: Sub-task
>Reporter: Appy
>Assignee: Matteo Bertozzi
> Attachments: HBASE-16744.master.001.patch,
> HBASE-16744.master.002.patch, HBASE-16744.master.003.patch,
> HBASE-16744.master.004.patch, HBASE-16744.master.005.patch,
> HBASE-16744.master.006.patch, HBASE-16744.master.007.patch,
> HBASE-16744.master.008.patch, HBASE-16744.master.009.patch,
> HBASE-16744.master.010.patch, HBASE-16744.master.011.patch,
> HBASE-16744.master.012.patch, HBASE-16744.master.013.patch
>
>
> Will help us get rid of ZK locks.
> Will be useful for external tools like hbck, future backup manager, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[
https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813478#comment-15813478
]
Hadoop QA commented on HBASE-16744:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-16744 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834775/HBASE-16744.master.013.patch
|
| JIRA Issue | HBASE-16744 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5203/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Procedure V2 - Lock procedures to allow clients to acquire locks on
> tables/namespaces/regions
> -
>
> Key: HBASE-16744
> URL: https://issues.apache.org/jira/browse/HBASE-16744
> Project: HBase
> Issue Type: Sub-task
>Reporter: Appy
>Assignee: Matteo Bertozzi
> Attachments: HBASE-16744.master.001.patch,
> HBASE-16744.master.002.patch, HBASE-16744.master.003.patch,
> HBASE-16744.master.004.patch, HBASE-16744.master.005.patch,
> HBASE-16744.master.006.patch, HBASE-16744.master.007.patch,
> HBASE-16744.master.008.patch, HBASE-16744.master.009.patch,
> HBASE-16744.master.010.patch, HBASE-16744.master.011.patch,
> HBASE-16744.master.012.patch, HBASE-16744.master.013.patch
>
>
> Will help us get rid of ZK locks.
> Will be useful for external tools like hbck, future backup manager, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[
https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16744:
--
Release Note:
Locks for table/namespace/regions. Use {@link LockServiceClient} to build
instances.
These are remote locks which live on master, and need periodic heartbeats to
keep them alive. (Once we requested the lock, internally an heartbeat thread
will be started). If master doesn't receive the heartbeat in time, it'll
release the lock and make it available to other users.
{@link #requestLock} will contact master to queue the lock and start the
heartbeat thread which'll check lock's status periodically and once the lock is
acquired, it will send the heartbeats to the master.
Use {@link #await} or {@link #await(long, TimeUnit)} to wait for the lock to be
acquired. Always call {@link #unlock()} irrespective of whether lock was
acquired or not. If the lock was acquired, it'll be released. If it was not
acquired, it's possible that master grants the lock in future and the heartbeat
thread keeps it alive forever by sending heartbeats. Calling {@link #unlock()}
will stop the heartbeat thread and cancel the lock queued on master.
There are 4 ways in which these remote locks may be released/can be lost:
* Call {@link #unlock}.
* Lock times out on master: Can happen because of network issues, GC pauses,
etc.
* Worker thread will call the given abortable as soon as it detects such a
situation.
* Fail to contact master: If worker thread can not contact mater and thus
fails to send heartbeat before the timeout expires, it assumes that lock is
lost and calls the abortable.
* Worker thread is interrupted.
Use example:
EntityLock lock = lockServiceClient.*Lock(, "exampled lock", abortable);
lock.requestLock();
can do other initializations here since lock is 'asynchronous'...
if (lock.await(timeout)) {
logic requiring mutual exclusion
}
lock.unlock();
> Procedure V2 - Lock procedures to allow clients to acquire locks on
> tables/namespaces/regions
> -
>
> Key: HBASE-16744
> URL: https://issues.apache.org/jira/browse/HBASE-16744
> Project: HBase
> Issue Type: Sub-task
>Reporter: Appy
>Assignee: Matteo Bertozzi
> Attachments: HBASE-16744.master.001.patch,
> HBASE-16744.master.002.patch, HBASE-16744.master.003.patch,
> HBASE-16744.master.004.patch, HBASE-16744.master.005.patch,
> HBASE-16744.master.006.patch, HBASE-16744.master.007.patch,
> HBASE-16744.master.008.patch, HBASE-16744.master.009.patch,
> HBASE-16744.master.010.patch, HBASE-16744.master.011.patch,
> HBASE-16744.master.012.patch, HBASE-16744.master.013.patch
>
>
> Will help us get rid of ZK locks.
> Will be useful for external tools like hbck, future backup manager, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17442) Move most of the replication related classes to hbase-server package
Guanghao Zhang created HBASE-17442: -- Summary: Move most of the replication related classes to hbase-server package Key: HBASE-17442 URL: https://issues.apache.org/jira/browse/HBASE-17442 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Guanghao Zhang After the replication requests are routed through master, replication implementation details didn't need be exposed to client. We should move most of the replication related classes to hbase-server package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17337) list replication peers request should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17337: --- Resolution: Fixed Release Note: List replication peers request will be roughed through master. Status: Resolved (was: Patch Available) Pushed to master. Thanks all for review. > list replication peers request should be routed through master > -- > > Key: HBASE-17337 > URL: https://issues.apache.org/jira/browse/HBASE-17337 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17337-v1.patch, HBASE-17337-v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17337) list replication peers request should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17337: --- Affects Version/s: 2.0.0 > list replication peers request should be routed through master > -- > > Key: HBASE-17337 > URL: https://issues.apache.org/jira/browse/HBASE-17337 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17337-v1.patch, HBASE-17337-v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
[ https://issues.apache.org/jira/browse/HBASE-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-17440. --- Resolution: Fixed Fix Version/s: 0.98.25 Done. > [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is > closed > > > Key: HBASE-17440 > URL: https://issues.apache.org/jira/browse/HBASE-17440 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.24 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.25 > > Attachments: 17440.txt > > > We're seeing many issue with run-away ZK client connection in long running > app servers. 10k or more send or event threads are happening frequently. > While I looked around in the code I noticed that DelayedClosing closing is > not immediately ended when an HConnection is closed, when there's an issue > with HBase or ZK and client reconnect in a tight loop, this can lead > temporarily to very many threads running. These will all get cleaned out > after at most 60s, but during that time a lot of threads can be created. > The fix is a one-liner. We'll likely file other issues soon. > Interestingly branch-1 and beyond do not have this chore anymore, although - > at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16661) Add last major compaction age to per-region metrics
[
https://issues.apache.org/jira/browse/HBASE-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813412#comment-15813412
]
Gary Helmling commented on HBASE-16661:
---
It's there:
{noformat}
commit 59ca4dad70cee46314c992766fd9303d1e41ee2c
Author: Dustin Pho
Date: Sat Sep 24 14:58:37 2016 -0700
HBASE-16661 Add last major compaction age to per-region metrics
Signed-off-by: Gary Helmling
{noformat}
> Add last major compaction age to per-region metrics
> ---
>
> Key: HBASE-16661
> URL: https://issues.apache.org/jira/browse/HBASE-16661
> Project: HBase
> Issue Type: Improvement
>Reporter: Gary Helmling
>Assignee: Dustin Pho
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16661.001.patch, HBASE-16661.002.patch,
> HBASE-16661.003.patch
>
>
> After HBASE-12859, we can now track the last major compaction timestamp for
> each region. However, this is only exposed through cluster status reporting
> and the admin API.
> We have similar per-region metrics around storefile age, but none that
> filters on major compaction specifically.
> Let's add a metric for last major compaction age.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
[ https://issues.apache.org/jira/browse/HBASE-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813406#comment-15813406 ] Lars Hofhansl commented on HBASE-17440: --- OK... Going to commit in a few. (since there won't be another 0.98 release likely, it's just for completeness) > [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is > closed > > > Key: HBASE-17440 > URL: https://issues.apache.org/jira/browse/HBASE-17440 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.24 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 17440.txt > > > We're seeing many issue with run-away ZK client connection in long running > app servers. 10k or more send or event threads are happening frequently. > While I looked around in the code I noticed that DelayedClosing closing is > not immediately ended when an HConnection is closed, when there's an issue > with HBase or ZK and client reconnect in a tight loop, this can lead > temporarily to very many threads running. These will all get cleaned out > after at most 60s, but during that time a lot of threads can be created. > The fix is a one-liner. We'll likely file other issues soon. > Interestingly branch-1 and beyond do not have this chore anymore, although - > at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813229#comment-15813229 ] stack commented on HBASE-17434: --- Oops. Premature commit. Reverted. Waiting on at least another +1 since we've been having so much fun around this issue. > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813200#comment-15813200 ] Hudson commented on HBASE-17434: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2289 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2289/]) HBASE-17434 New Synchronization Scheme for Compaction Pipeline (Eshcar (stack: rev 1576269123f18c9eb21b04a800e81952ec52c04d) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813199#comment-15813199 ] Hudson commented on HBASE-12148: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2289 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2289/]) HBASE-12148 Remove TimeRangeTracker as point of contention when many (stack: rev dd1ae37148c13e79ec37817f3953a79dd40e8e87) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/ClassSize.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTimeRangeTracker.java > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: huaxiang sun > Fix For: 2.0.0, 1.4.0 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.min_and_max_run_independent.patch, 12148.txt, > 12148.txt, 12148v2.txt, 12148v2.txt, 12148v4.patch, HBASE-12148-V3.patch, > HBASE-12148-V3.patch, HBASE-12148-master-v6.patch, > HBASE-12148-master-v7.patch, HBASE-12148-master-v7.patch, > HBASE-12148.branch-1.v5.patch, HBASE-12148.branch-1.v5.patch, > HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot 2014-10-01 at 3.39.46 PM.png, > Screen Shot 2014-10-01 at 3.41.07 PM.png, Screen Shot 2016-04-13 at 1.49.30 > PM.png, Screen Shot 2016-04-13 at 2.02.22 PM.png, Screen Shot 2016-05-18 at > 10.21.53 PM.png, TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813146#comment-15813146 ] huaxiang sun commented on HBASE-12148: -- Thanks [~stack]. > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: huaxiang sun > Fix For: 2.0.0, 1.4.0 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.min_and_max_run_independent.patch, 12148.txt, > 12148.txt, 12148v2.txt, 12148v2.txt, 12148v4.patch, HBASE-12148-V3.patch, > HBASE-12148-V3.patch, HBASE-12148-master-v6.patch, > HBASE-12148-master-v7.patch, HBASE-12148-master-v7.patch, > HBASE-12148.branch-1.v5.patch, HBASE-12148.branch-1.v5.patch, > HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot 2014-10-01 at 3.39.46 PM.png, > Screen Shot 2014-10-01 at 3.41.07 PM.png, Screen Shot 2016-04-13 at 1.49.30 > PM.png, Screen Shot 2016-04-13 at 2.02.22 PM.png, Screen Shot 2016-05-18 at > 10.21.53 PM.png, TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14061) Support CF-level Storage Policy
[
https://issues.apache.org/jira/browse/HBASE-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reopened HBASE-14061:
-
This causes building against the Hadoop 3 profile to fail, and will cause
building against Hadoop 2.8 to fail once that release happens, due to
conflicting method signatures in HFileSystem here and FileSystem /
FilterFileSystem in Hadoop as of HADOOP-12161.
The problem methods are a part of Hadoop's Public/Stable interface so they're
unlikely to remove them.
Please post an addendum ASAP.
I have filed HBASE-17441 for the failure of hadoopcheck to catch this in
precommit.
> Support CF-level Storage Policy
> ---
>
> Key: HBASE-14061
> URL: https://issues.apache.org/jira/browse/HBASE-14061
> Project: HBase
> Issue Type: Sub-task
> Components: HFile, regionserver
> Environment: hadoop-2.6.0
>Reporter: Victor Xu
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14061-master-v1.patch, HBASE-14061.v2.patch,
> HBASE-14061.v3.patch, HBASE-14061.v4.patch
>
>
> After reading [HBASE-12848|https://issues.apache.org/jira/browse/HBASE-12848]
> and [HBASE-12934|https://issues.apache.org/jira/browse/HBASE-12934], I wrote
> a patch to implement cf-level storage policy.
> My main purpose is to improve random-read performance for some really hot
> data, which usually locates in certain column family of a big table.
> Usage:
> $ hbase shell
> > alter 'TABLE_NAME', METADATA => {'hbase.hstore.block.storage.policy' =>
> > 'POLICY_NAME'}
> > alter 'TABLE_NAME', {NAME=>'CF_NAME', METADATA =>
> > {'hbase.hstore.block.storage.policy' => 'POLICY_NAME'}}
> HDFS's setStoragePolicy can only take effect when new hfile is created in a
> configured directory, so I had to make sub directories(for each cf) in
> region's .tmp directory and set storage policy for them.
> Besides, I had to upgrade hadoop version to 2.6.0 because
> dfs.getStoragePolicy cannot be easily written in reflection, and I needed
> this api to finish my unit test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17441) precommit test "hadoopcheck" not properly testing Hadoop 3 profile
Sean Busbey created HBASE-17441: --- Summary: precommit test "hadoopcheck" not properly testing Hadoop 3 profile Key: HBASE-17441 URL: https://issues.apache.org/jira/browse/HBASE-17441 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Sean Busbey Priority: Blocker Fix For: 2.0.0 HBASE-14061 made a change that caused building against hadoop 3 to fail, but the hadoopcheck precommit test gave the change a +1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-17429) HBase bulkload cannot support HDFS viewFs
[
https://issues.apache.org/jira/browse/HBASE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-17429:
--
Assignee: shenxianqiang
> HBase bulkload cannot support HDFS viewFs
> -
>
> Key: HBASE-17429
> URL: https://issues.apache.org/jira/browse/HBASE-17429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6.1, 1.2.4
> Environment: CDH5.7.0 hbase0.98.6
>Reporter: shenxianqiang
>Assignee: shenxianqiang
> Attachments: HBASE-17429.patch
>
>
> Since hadoop cluster suppor fedration,hbase bulkload performance degrades
> dramatically. Even if hbase directory and bulkload directory in the same
> nameservice.
> {quote}
> 2017-01-04 21:58:40,919 INFO
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem: use bulkload file
> name is : hdfs://CloudTestNameNode2:8020
> 2017-01-04 21:58:40,924 ERROR org.apache.hadoop.ipc.RpcServer: Unexpected
> throwable object
> java.lang.IllegalArgumentException: Wrong FS:
> viewfs://nsX/user/test/I/_tmp/9cde5dde60374b1483b7d09b65258304.top, expected:
> hdfs://CloudTestNameNode2:8020
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:373)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:496)
> at
> org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:708)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3658)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3564)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3378)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29589)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813071#comment-15813071 ] Umesh Agashe edited comment on HBASE-17437 at 1/9/17 10:43 PM: --- +1. This will help in FS layout redo (HBASE-14439) work as well. Overall changes look good to me. was (Author: uagashe): +1. This will help in FS redo work as well. Overall changes look good to me. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813071#comment-15813071 ] Umesh Agashe commented on HBASE-17437: -- +1. This will help in FS redo work as well. Overall changes look good to me. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
[ https://issues.apache.org/jira/browse/HBASE-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813006#comment-15813006 ] Andrew Purtell commented on HBASE-17440: bq. (as said, this is not the main issue we've been seeing, but something I noticed on the way) Right, this is not the issue we are seeing, unless the DelayedChores are leaked and never fire, leading to indefinitely lingering ZK connections. > [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is > closed > > > Key: HBASE-17440 > URL: https://issues.apache.org/jira/browse/HBASE-17440 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.24 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 17440.txt > > > We're seeing many issue with run-away ZK client connection in long running > app servers. 10k or more send or event threads are happening frequently. > While I looked around in the code I noticed that DelayedClosing closing is > not immediately ended when an HConnection is closed, when there's an issue > with HBase or ZK and client reconnect in a tight loop, this can lead > temporarily to very many threads running. These will all get cleaned out > after at most 60s, but during that time a lot of threads can be created. > The fix is a one-liner. We'll likely file other issues soon. > Interestingly branch-1 and beyond do not have this chore anymore, although - > at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
[ https://issues.apache.org/jira/browse/HBASE-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813006#comment-15813006 ] Andrew Purtell edited comment on HBASE-17440 at 1/9/17 10:11 PM: - bq. (as said, this is not the main issue we've been seeing, but something I noticed on the way) Right, this is not the issue we are seeing, unless the DelayedClosing are leaked and never fire, leading to indefinitely lingering ZK connections. was (Author: apurtell): bq. (as said, this is not the main issue we've been seeing, but something I noticed on the way) Right, this is not the issue we are seeing, unless the DelayedChores are leaked and never fire, leading to indefinitely lingering ZK connections. > [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is > closed > > > Key: HBASE-17440 > URL: https://issues.apache.org/jira/browse/HBASE-17440 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.24 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 17440.txt > > > We're seeing many issue with run-away ZK client connection in long running > app servers. 10k or more send or event threads are happening frequently. > While I looked around in the code I noticed that DelayedClosing closing is > not immediately ended when an HConnection is closed, when there's an issue > with HBase or ZK and client reconnect in a tight loop, this can lead > temporarily to very many threads running. These will all get cleaned out > after at most 60s, but during that time a lot of threads can be created. > The fix is a one-liner. We'll likely file other issues soon. > Interestingly branch-1 and beyond do not have this chore anymore, although - > at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
[ https://issues.apache.org/jira/browse/HBASE-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813004#comment-15813004 ] Andrew Purtell commented on HBASE-17440: lgtm [~lhofhansl] > [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is > closed > > > Key: HBASE-17440 > URL: https://issues.apache.org/jira/browse/HBASE-17440 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.24 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 17440.txt > > > We're seeing many issue with run-away ZK client connection in long running > app servers. 10k or more send or event threads are happening frequently. > While I looked around in the code I noticed that DelayedClosing closing is > not immediately ended when an HConnection is closed, when there's an issue > with HBase or ZK and client reconnect in a tight loop, this can lead > temporarily to very many threads running. These will all get cleaned out > after at most 60s, but during that time a lot of threads can be created. > The fix is a one-liner. We'll likely file other issues soon. > Interestingly branch-1 and beyond do not have this chore anymore, although - > at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17435) Call to preCommitStoreFile() hook encounters SaslException in secure deployment
[
https://issues.apache.org/jira/browse/HBASE-17435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812941#comment-15812941
]
Hudson commented on HBASE-17435:
FAILURE: Integrated in Jenkins build HBase-1.4 #588 (See
[https://builds.apache.org/job/HBase-1.4/588/])
HBASE-17435 Call to preCommitStoreFile() hook encounters SaslException (tedyu:
rev 9b26c9ff37f344e996fe93c027d9ce4b5e61cf31)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
> Call to preCommitStoreFile() hook encounters SaslException in secure
> deployment
> ---
>
> Key: HBASE-17435
> URL: https://issues.apache.org/jira/browse/HBASE-17435
> Project: HBase
> Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17435.branch-1.v2.txt, 17435.branch-1.v3.txt,
> 17435.v1.txt, 17435.v2.txt, 17435.v3.txt
>
>
> [~romil.choksi] was testing bulk load in secure cluster where
> LoadIncrementalHFiles failed.
> Looking at region server log, we saw the following:
> {code}
> at
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1257)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1163)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:162)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
> at
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupContexts(BackupSystemTable.java:540)
> at
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupHistory(BackupSystemTable.java:517)
> at
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getTablesForBackupType(BackupSystemTable.java:589)
> at
> org.apache.hadoop.hbase.backup.BackupObserver.preCommitStoreFile(BackupObserver.java:89)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$61.call(RegionCoprocessorHost.java:1494)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preCommitStoreFile(RegionCoprocessorHost.java:1490)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5512)
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:293)
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:276)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1704)
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.secureBulkLoadHFiles(SecureBulkLoadEndpoint.java:276)
> at
> org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4721)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most
> likely cause is missing or invalid credentials. Consider 'kinit'.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:679)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserG
[jira] [Updated] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
[ https://issues.apache.org/jira/browse/HBASE-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-17440: -- Attachment: 17440.txt Here's the trivial fix, just trigger the chore immediately if the connection is stopped, the chore will then determine that its connection is shutdown and end itself. I tested this, I was not able to build up 1000's of DelayedClosing threads in a tight loop. (as said, this is not the main issue we've been seeing, but something I noticed on the way) Please have a look. [~apurtell], fyi. > [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is > closed > > > Key: HBASE-17440 > URL: https://issues.apache.org/jira/browse/HBASE-17440 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.24 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 17440.txt > > > We're seeing many issue with run-away ZK client connection in long running > app servers. 10k or more send or event threads are happening frequently. > While I looked around in the code I noticed that DelayedClosing closing is > not immediately ended when an HConnection is closed, when there's an issue > with HBase or ZK and client reconnect in a tight loop, this can lead > temporarily to very many threads running. These will all get cleaned out > after at most 60s, but during that time a lot of threads can be created. > The fix is a one-liner. We'll likely file other issues soon. > Interestingly branch-1 and beyond do not have this chore anymore, although - > at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
[ https://issues.apache.org/jira/browse/HBASE-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-17440: -- Affects Version/s: 0.98.24 > [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is > closed > > > Key: HBASE-17440 > URL: https://issues.apache.org/jira/browse/HBASE-17440 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.24 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > > We're seeing many issue with run-away ZK client connection in long running > app servers. 10k or more send or event threads are happening frequently. > While I looked around in the code I noticed that DelayedClosing closing is > not immediately ended when an HConnection is closed, when there's an issue > with HBase or ZK and client reconnect in a tight loop, this can lead > temporarily to very many threads running. These will all get cleaned out > after at most 60s, but during that time a lot of threads can be created. > The fix is a one-liner. We'll likely file other issues soon. > Interestingly branch-1 and beyond do not have this chore anymore, although - > at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12148: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master and branch-1. Nice fixup [~huaxiang] > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: huaxiang sun > Fix For: 2.0.0, 1.4.0 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.min_and_max_run_independent.patch, 12148.txt, > 12148.txt, 12148v2.txt, 12148v2.txt, 12148v4.patch, HBASE-12148-V3.patch, > HBASE-12148-V3.patch, HBASE-12148-master-v6.patch, > HBASE-12148-master-v7.patch, HBASE-12148-master-v7.patch, > HBASE-12148.branch-1.v5.patch, HBASE-12148.branch-1.v5.patch, > HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot 2014-10-01 at 3.39.46 PM.png, > Screen Shot 2014-10-01 at 3.41.07 PM.png, Screen Shot 2016-04-13 at 1.49.30 > PM.png, Screen Shot 2016-04-13 at 2.02.22 PM.png, Screen Shot 2016-05-18 at > 10.21.53 PM.png, TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17440) [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed
Lars Hofhansl created HBASE-17440: - Summary: [0.98] Make sure DelayedClosing chore is stopped as soon as an HConnection is closed Key: HBASE-17440 URL: https://issues.apache.org/jira/browse/HBASE-17440 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl We're seeing many issue with run-away ZK client connection in long running app servers. 10k or more send or event threads are happening frequently. While I looked around in the code I noticed that DelayedClosing closing is not immediately ended when an HConnection is closed, when there's an issue with HBase or ZK and client reconnect in a tight loop, this can lead temporarily to very many threads running. These will all get cleaned out after at most 60s, but during that time a lot of threads can be created. The fix is a one-liner. We'll likely file other issues soon. Interestingly branch-1 and beyond do not have this chore anymore, although - at least in branch-1 and later - I still see the ZooKeeperAliveConnection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15995) Separate replication WAL reading from shipping
[ https://issues.apache.org/jira/browse/HBASE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812888#comment-15812888 ] Ted Yu commented on HBASE-15995: Patch v3 looks good overall. Please fix javadoc warnings. > Separate replication WAL reading from shipping > -- > > Key: HBASE-15995 > URL: https://issues.apache.org/jira/browse/HBASE-15995 > Project: HBase > Issue Type: Sub-task > Components: Replication >Affects Versions: 2.0.0 >Reporter: Vincent Poon >Assignee: Vincent Poon > Fix For: 2.0.0 > > Attachments: HBASE-15995.master.v1.patch, > HBASE-15995.master.v2.patch, HBASE-15995.master.v3.patch, > replicationV1_100ms_delay.png, replicationV2_100ms_delay.png > > > Currently ReplicationSource reads edits from the WAL and ships them in the > same thread. > By breaking out the reading from the shipping, we can introduce greater > parallelism and lay the foundation for further refactoring to a pipelined, > streaming model. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15995) Separate replication WAL reading from shipping
[
https://issues.apache.org/jira/browse/HBASE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812877#comment-15812877
]
Hadoop QA commented on HBASE-15995:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 24s
{color} | {color:red} hbase-server generated 2 new + 1 unchanged - 0 fixed = 3
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 77m 7s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 111m 40s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846392/HBASE-15995.master.v3.patch
|
| JIRA Issue | HBASE-15995 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux eb12650a85e5 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 9cbeba6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| javadoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/5202/artifact/patchprocess/diff-javadoc-javadoc-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5202/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5202/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Separate replication WAL reading from shipping
> --
>
> Key: HBASE-15995
> URL: https://issues.apache.org/jira/browse/HBASE-15995
> Project: HBase
> Issue Type: Sub-task
> Components:
[jira] [Commented] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[
https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812869#comment-15812869
]
Hadoop QA commented on HBASE-12148:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 39s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 45s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 5s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 126m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.client.TestAsyncNonMetaRegionLocatorConcurrenyLimit |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846389/HBASE-12148-master-v7.patch
|
| JIRA Issue | HBASE-12148 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 1f68ed08f4c9 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 9cbeba6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5201/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5201/artifact/patchprocess/patch-unit-hbase-server.txt
|
|
[jira] [Commented] (HBASE-17429) HBase bulkload cannot support HDFS viewFs
[
https://issues.apache.org/jira/browse/HBASE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812866#comment-15812866
]
Ted Yu commented on HBASE-17429:
+1
> HBase bulkload cannot support HDFS viewFs
> -
>
> Key: HBASE-17429
> URL: https://issues.apache.org/jira/browse/HBASE-17429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6.1, 1.2.4
> Environment: CDH5.7.0 hbase0.98.6
>Reporter: shenxianqiang
> Attachments: HBASE-17429.patch
>
>
> Since hadoop cluster suppor fedration,hbase bulkload performance degrades
> dramatically. Even if hbase directory and bulkload directory in the same
> nameservice.
> {quote}
> 2017-01-04 21:58:40,919 INFO
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem: use bulkload file
> name is : hdfs://CloudTestNameNode2:8020
> 2017-01-04 21:58:40,924 ERROR org.apache.hadoop.ipc.RpcServer: Unexpected
> throwable object
> java.lang.IllegalArgumentException: Wrong FS:
> viewfs://nsX/user/test/I/_tmp/9cde5dde60374b1483b7d09b65258304.top, expected:
> hdfs://CloudTestNameNode2:8020
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:373)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:496)
> at
> org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:708)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3658)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3564)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3378)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29589)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[
https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812842#comment-15812842
]
Hadoop QA commented on HBASE-17434:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 36s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 54s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 136m 30s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846381/HBASE-17434.master.001.patch
|
| JIRA Issue | HBASE-17434 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f96e4a9f2e93 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 9cbeba6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5200/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5200/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> New Synchronization Scheme for Compaction Pipeline
> --
>
> Key: HBASE-17434
> URL: https://issues.apache.org/jira/browse/HBASE-17434
> Project: HBase
> Issue Type: Bug
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch,
> HBASE-17434-V03.patch, HBASE-17434.master.001.p
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: 14417-tbl-ext.v17.txt > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v17.txt, 14417-tbl-ext.v9.txt, > 14417.v1.txt, 14417.v11.txt, 14417.v13.txt, 14417.v2.txt, 14417.v21.txt, > 14417.v23.txt, 14417.v24.txt, 14417.v25.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board: > https://reviews.apache.org/r/54258/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: (was: 14417-tbl-ext.v17.txt) > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v9.txt, 14417.v1.txt, 14417.v11.txt, > 14417.v13.txt, 14417.v2.txt, 14417.v21.txt, 14417.v23.txt, 14417.v24.txt, > 14417.v25.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board: > https://reviews.apache.org/r/54258/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812759#comment-15812759 ] Aaron Tokhy commented on HBASE-15314: - Sorry I was out for a couple of weeks. Here is the description: The following patch adds a couple of features without making any big changes to the existing BucketCache allocator. This approach introduces the notion of 'segmentation' (or in other words, the underlying IOEngine can be made up of non-contiguous segments. Two methods are added to expose this information to the BucketCache allocator. boolean IOEngine#isSegmented() boolean IOEngine#doesAllocationCrossSegments(long offset, long len) BucketCache calls these methods to determine if a 'contiguous' allocation of a particular block size can occur. It does this by checking if doesAllocationCrossSegments(offset, len) is true. If an allocation crosses a segment, another call to allocate is made for the same block. The first block is wasted (it is marked allocated). The worst case is 1 'largest block' per file. If an allocation fails for any reason, all/any allocated blocks (including wasted ones) are freed again for subsequent allocation requests. This is very similar to a 'JBOD' configuration (there is no striping of any kind). There are also some additional fixes: 1) The 'total size' aligns with the 'total aggregate file size'. This is done by doing a ceiling division, and rounding up the 'totalSize' so that each segment is equally sized. segmentSize = ceil(totalSize / numFiles) totalSize = segmentSize * numFiles 2) All failed allocations, including extra ones made due to crossing segments, are cleaned up > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, > HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812740#comment-15812740 ] Enis Soztutar commented on HBASE-17437: --- +1 for this. We need it for other cloud deployments as well. Let me look at the patch. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16661) Add last major compaction age to per-region metrics
[ https://issues.apache.org/jira/browse/HBASE-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812692#comment-15812692 ] Mikhail Antonov commented on HBASE-16661: - that commit appear to be missing on branch-1 (1.4)? > Add last major compaction age to per-region metrics > --- > > Key: HBASE-16661 > URL: https://issues.apache.org/jira/browse/HBASE-16661 > Project: HBase > Issue Type: Improvement >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16661.001.patch, HBASE-16661.002.patch, > HBASE-16661.003.patch > > > After HBASE-12859, we can now track the last major compaction timestamp for > each region. However, this is only exposed through cluster status reporting > and the admin API. > We have similar per-region metrics around storefile age, but none that > filters on major compaction specifically. > Let's add a metric for last major compaction age. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812682#comment-15812682 ] Yishan Yang commented on HBASE-17437: - Post patch to review board, https://reviews.apache.org/r/55352/ > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-17394) Correct TimeRangeTracker#init() logic
[ https://issues.apache.org/jira/browse/HBASE-17394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun resolved HBASE-17394. -- Resolution: Invalid The logic is correct, the definition is [min, max], I added comments through HBASE-12148 to make it clear and hopefully, it will help code reading a bit easier. > Correct TimeRangeTracker#init() logic > - > > Key: HBASE-17394 > URL: https://issues.apache.org/jira/browse/HBASE-17394 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > > It is supposed to be [minimumTimestamp, maximumTimestamp), the following > logic suggests [minimumTimestamp, maximumTimestamp] > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java#L81 > needs to be modified to set(l, l + 1) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17337) list replication peers request should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812658#comment-15812658 ] Enis Soztutar commented on HBASE-17337: --- +1. > list replication peers request should be routed through master > -- > > Key: HBASE-17337 > URL: https://issues.apache.org/jira/browse/HBASE-17337 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17337-v1.patch, HBASE-17337-v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17435) Call to preCommitStoreFile() hook encounters SaslException in secure deployment
[
https://issues.apache.org/jira/browse/HBASE-17435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812654#comment-15812654
]
Hudson commented on HBASE-17435:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2288 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2288/])
HBASE-17435 Call to preCommitStoreFile() hook encounters SaslException (tedyu:
rev 9cbeba6c3de3ac7dd9c031792c70abe32ba8a62b)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java
> Call to preCommitStoreFile() hook encounters SaslException in secure
> deployment
> ---
>
> Key: HBASE-17435
> URL: https://issues.apache.org/jira/browse/HBASE-17435
> Project: HBase
> Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17435.branch-1.v2.txt, 17435.branch-1.v3.txt,
> 17435.v1.txt, 17435.v2.txt, 17435.v3.txt
>
>
> [~romil.choksi] was testing bulk load in secure cluster where
> LoadIncrementalHFiles failed.
> Looking at region server log, we saw the following:
> {code}
> at
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1257)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1163)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:162)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
> at
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupContexts(BackupSystemTable.java:540)
> at
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupHistory(BackupSystemTable.java:517)
> at
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getTablesForBackupType(BackupSystemTable.java:589)
> at
> org.apache.hadoop.hbase.backup.BackupObserver.preCommitStoreFile(BackupObserver.java:89)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$61.call(RegionCoprocessorHost.java:1494)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preCommitStoreFile(RegionCoprocessorHost.java:1490)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5512)
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:293)
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:276)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1704)
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.secureBulkLoadHFiles(SecureBulkLoadEndpoint.java:276)
> at
> org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4721)
> ...
> Caused by: java.lang.RuntimeException: SASL authentication failed. The most
> likely cause is missing or invalid credentials. Consider 'kinit'.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:679)
> at java.security.AccessController.doPrivileged(Native Method)
>
[jira] [Commented] (HBASE-15995) Separate replication WAL reading from shipping
[ https://issues.apache.org/jira/browse/HBASE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812628#comment-15812628 ] Vincent Poon commented on HBASE-15995: -- I added a check against the totalQuota before reading the next entry. So now TestGlobalThrottler passes without modifications. There's still a race condition since I don't use a lock: multiple threads could do the check and all pass the check, then all read an entry which cumulatively goes above the global quota. But I think this problem even exists in current code - since we currently check after reading an entry, the global quota can still be exceeded. That's why in TestGlobalThrottler we check against a threshold of 600, even though we set the total global buffer to 200, right? And if we added a peer, we would have to change the threshold to 800? So the potential OOM problem exists even currently, though checking the global quota reduces that risk significantly. > Separate replication WAL reading from shipping > -- > > Key: HBASE-15995 > URL: https://issues.apache.org/jira/browse/HBASE-15995 > Project: HBase > Issue Type: Sub-task > Components: Replication >Affects Versions: 2.0.0 >Reporter: Vincent Poon >Assignee: Vincent Poon > Fix For: 2.0.0 > > Attachments: HBASE-15995.master.v1.patch, > HBASE-15995.master.v2.patch, HBASE-15995.master.v3.patch, > replicationV1_100ms_delay.png, replicationV2_100ms_delay.png > > > Currently ReplicationSource reads edits from the WAL and ships them in the > same thread. > By breaking out the reading from the shipping, we can introduce greater > parallelism and lay the foundation for further refactoring to a pipelined, > streaming model. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: 14417-tbl-ext.v17.txt > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v17.txt, 14417-tbl-ext.v9.txt, > 14417.v1.txt, 14417.v11.txt, 14417.v13.txt, 14417.v2.txt, 14417.v21.txt, > 14417.v23.txt, 14417.v24.txt, 14417.v25.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board: > https://reviews.apache.org/r/54258/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: (was: 14417-tbl-ext.v16.txt) > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v17.txt, 14417-tbl-ext.v9.txt, > 14417.v1.txt, 14417.v11.txt, 14417.v13.txt, 14417.v2.txt, 14417.v21.txt, > 14417.v23.txt, 14417.v24.txt, 14417.v25.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board: > https://reviews.apache.org/r/54258/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: (was: 14417-tbl-ext.v15.txt) > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v17.txt, 14417-tbl-ext.v9.txt, > 14417.v1.txt, 14417.v11.txt, 14417.v13.txt, 14417.v2.txt, 14417.v21.txt, > 14417.v23.txt, 14417.v24.txt, 14417.v25.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board: > https://reviews.apache.org/r/54258/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812611#comment-15812611 ] Edward Bortnikov commented on HBASE-17434: -- Suggest to commit. This patch has been well discussed and verified. Would be much more convenient to fix and re-submit HBASE-17081 with a solid synchronization scheme in place. We are trying to solve a bunch of issues that piled up in the CompactingMemstore implementation. This one is a roadblock. Once again, thanks to all who contributed to improving the solution's quality. > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15995) Separate replication WAL reading from shipping
[ https://issues.apache.org/jira/browse/HBASE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated HBASE-15995: - Attachment: HBASE-15995.master.v3.patch > Separate replication WAL reading from shipping > -- > > Key: HBASE-15995 > URL: https://issues.apache.org/jira/browse/HBASE-15995 > Project: HBase > Issue Type: Sub-task > Components: Replication >Affects Versions: 2.0.0 >Reporter: Vincent Poon >Assignee: Vincent Poon > Fix For: 2.0.0 > > Attachments: HBASE-15995.master.v1.patch, > HBASE-15995.master.v2.patch, HBASE-15995.master.v3.patch, > replicationV1_100ms_delay.png, replicationV2_100ms_delay.png > > > Currently ReplicationSource reads edits from the WAL and ships them in the > same thread. > By breaking out the reading from the shipping, we can introduce greater > parallelism and lay the foundation for further refactoring to a pipelined, > streaming model. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812588#comment-15812588 ] Yishan Yang commented on HBASE-17437: - About hbase.regionserver.hlog.dir.perms, since we differentiate logDir out of rootDir, in hbase, master used to set hbase.rootdir.perms by master, here, we just add one more perm parameter for logDir. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812559#comment-15812559 ] huaxiang sun commented on HBASE-12148: -- Thanks [~stack]. I was looking at the test failure. Search the web and found the exact the failure, Ted filed a jira for that. https://issues.apache.org/jira/browse/HBASE-17384 > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: huaxiang sun > Fix For: 2.0.0, 1.4.0 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.min_and_max_run_independent.patch, 12148.txt, > 12148.txt, 12148v2.txt, 12148v2.txt, 12148v4.patch, HBASE-12148-V3.patch, > HBASE-12148-V3.patch, HBASE-12148-master-v6.patch, > HBASE-12148-master-v7.patch, HBASE-12148-master-v7.patch, > HBASE-12148.branch-1.v5.patch, HBASE-12148.branch-1.v5.patch, > HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot 2014-10-01 at 3.39.46 PM.png, > Screen Shot 2014-10-01 at 3.41.07 PM.png, Screen Shot 2016-04-13 at 1.49.30 > PM.png, Screen Shot 2016-04-13 at 2.02.22 PM.png, Screen Shot 2016-05-18 at > 10.21.53 PM.png, TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12148: -- Attachment: HBASE-12148-master-v7.patch Retry. Failure seems unrelated, would you agree [~huaxiang]? Lets see what hadoopqa says second time through. > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: huaxiang sun > Fix For: 2.0.0, 1.4.0 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.min_and_max_run_independent.patch, 12148.txt, > 12148.txt, 12148v2.txt, 12148v2.txt, 12148v4.patch, HBASE-12148-V3.patch, > HBASE-12148-V3.patch, HBASE-12148-master-v6.patch, > HBASE-12148-master-v7.patch, HBASE-12148-master-v7.patch, > HBASE-12148.branch-1.v5.patch, HBASE-12148.branch-1.v5.patch, > HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot 2014-10-01 at 3.39.46 PM.png, > Screen Shot 2014-10-01 at 3.41.07 PM.png, Screen Shot 2016-04-13 at 1.49.30 > PM.png, Screen Shot 2016-04-13 at 2.02.22 PM.png, Screen Shot 2016-05-18 at > 10.21.53 PM.png, TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812537#comment-15812537 ] stack commented on HBASE-17434: --- -001 is V03 but with annotation for the findbugs issue so we get a clean run. +1 on patch (any other +1s?) > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17434: -- Attachment: HBASE-17434.master.001.patch > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch, HBASE-17434.master.001.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812496#comment-15812496 ] Yishan Yang commented on HBASE-17437: - Will address comments and put new patch into review board, thanks! > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812457#comment-15812457 ] stack commented on HBASE-17434: --- Opinions are fine. Telling folks what they 'should' do is not. > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17434) New Synchronization Scheme for Compaction Pipeline
[ https://issues.apache.org/jira/browse/HBASE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812455#comment-15812455 ] stack commented on HBASE-17434: --- Thanks for the narrative [~anastas]. Sounds right to me (You are missing the bit that we are also here because because HBASE-17379 had become an 'occupied' issue with an 'interloper' in the way...) > New Synchronization Scheme for Compaction Pipeline > -- > > Key: HBASE-17434 > URL: https://issues.apache.org/jira/browse/HBASE-17434 > Project: HBase > Issue Type: Bug >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-17434-V01.patch, HBASE-17434-V02.patch, > HBASE-17434-V03.patch > > > A new copyOnWrite synchronization scheme is introduced for the compaction > pipeline. > The new scheme is better since it removes the lock from getSegments() which > is invoked in every get and scan operation, and it reduces the number of > LinkedList objects that are created at runtime, thus can reduce GC (not by > much, but still...). > In addition, it fixes the method getTailSize() in compaction pipeline. This > method creates a MemstoreSize object which comprises the data size and the > overhead size of the segment and needs to be atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812436#comment-15812436 ] Mikhail Antonov commented on HBASE-17437: - I'm with [~busbey] on his assessment that it should be in 1.4+. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812424#comment-15812424 ] Yishan Yang commented on HBASE-17437: - Yeah, we used to use WAL, but from doc, default.xml, etc. On code side, it always use WAL, but from user side, hlog has much more occurrence. (Like 100 to 16). We already deployed it into our production and a lot of customer are using it right now. We passed integration tests , performance tests, unit tests. Up till now, there is no complain about this deployment yet. Thanks for your feed back! > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812416#comment-15812416 ] Yishan Yang commented on HBASE-17437: - We already put this patch in production, integration test, unit test, performance tests are all passed. For replication, it is working. Master branch has multiple commits related to "HBASE-17132 Cleanup deprecated code for WAL", so that part of the classes and unit tests need not be changed. That's the reason branch-1.2 is larger than master branch.. Will put it into review board, thanks! > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812400#comment-15812400 ] Sean Busbey commented on HBASE-17437: - I changed the target version to 1.4+ since this reads strongly to me as an improvement rather than a bug fix. I'm not sure if [~mantonov] feels the same way about branch-1.3. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
