[jira] [Updated] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-17475:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to master and branch-1.
Thanks [~allan163] for the contribution. Thanks [~tedyu] for reviewing.
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API, Client
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Labels: trivial
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475-branch-1.v3.patch,
> HBASE-17475.patch, HBASE-17475.v2.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
>
[jira] [Updated] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-17475:
--
Labels: trivial (was: )
Fix Version/s: 1.4.0
2.0.0
Component/s: Client
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API, Client
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Labels: trivial
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475-branch-1.v3.patch,
> HBASE-17475.patch, HBASE-17475.v2.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
>
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825595#comment-15825595 ] chunhui shen commented on HBASE-16630: -- [~stack] Our change about this which is mentioned in the previous comment has been included in the last patch. The patch is fine and would be harmless. Thanks sir. > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: 16630-v2-suggest.patch, 16630-v3-suggest.patch, > HBASE-16630.patch, HBASE-16630-v2.patch, HBASE-16630-v3.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16119) Procedure v2 - Reimplement merge
[ https://issues.apache.org/jira/browse/HBASE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825593#comment-15825593 ] Stephen Yuan Jiang commented on HBASE-16119: [~stack], release notes are added and also RB question is answered. > Procedure v2 - Reimplement merge > > > Key: HBASE-16119 > URL: https://issues.apache.org/jira/browse/HBASE-16119 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Region Assignment >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-16119.v1-master.patch, HBASE-16119.v2-master.patch > > > use the proc-v2 state machine for merge. also update the logic to have a > single meta-writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16119) Procedure v2 - Reimplement merge
[ https://issues.apache.org/jira/browse/HBASE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16119: --- Hadoop Flags: Incompatible change,Reviewed (was: Reviewed) Release Note: The merge region logic is controlled by master in 2.0.0 (in 1.x, the core merge region logic is in the region server side). The coprocessors related to merge region in RS-side would be no-op in 2.0.0 and later release. Therefore, this is an incompatible change. Users needs to move the CP logic to new master CP and registers them. A new mergeRegionsAsync() API is added in client. The existing mergeRegions() API will call the new API so client does not have to change its code. > Procedure v2 - Reimplement merge > > > Key: HBASE-16119 > URL: https://issues.apache.org/jira/browse/HBASE-16119 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Region Assignment >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-16119.v1-master.patch, HBASE-16119.v2-master.patch > > > use the proc-v2 state machine for merge. also update the logic to have a > single meta-writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825583#comment-15825583
]
Duo Zhang commented on HBASE-17475:
---
Will commit shortly if no objection.
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475-branch-1.v3.patch,
> HBASE-17475.patch, HBASE-17475.v2.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
>
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825581#comment-15825581
]
Hadoop QA commented on HBASE-17475:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 41s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 0s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12847741/HBASE-17475.v2.patch |
| JIRA Issue | HBASE-17475 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 10f12e4c319e 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4ab95eb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5291/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5291/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>
[jira] [Commented] (HBASE-16119) Procedure v2 - Reimplement merge
[ https://issues.apache.org/jira/browse/HBASE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825573#comment-15825573 ] stack commented on HBASE-16119: --- I wanted a release note so I could save reading the patch... Read the patch. There is a question up on rb wondering if a hole in merge as is. Or maybe there is more to do in a follow-on? Thanks [~syuanjiang]. > Procedure v2 - Reimplement merge > > > Key: HBASE-16119 > URL: https://issues.apache.org/jira/browse/HBASE-16119 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Region Assignment >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-16119.v1-master.patch, HBASE-16119.v2-master.patch > > > use the proc-v2 state machine for merge. also update the logic to have a > single meta-writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16786) Procedure V2 - Move ZK-lock's uses to Procedure framework locks (LockProcedure)
[ https://issues.apache.org/jira/browse/HBASE-16786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825568#comment-15825568 ] stack commented on HBASE-16786: --- Review when you have a chance [~appy] or [~syuanjiang]. Thanks. > Procedure V2 - Move ZK-lock's uses to Procedure framework locks > (LockProcedure) > --- > > Key: HBASE-16786 > URL: https://issues.apache.org/jira/browse/HBASE-16786 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Matteo Bertozzi > Attachments: HBASE-16786.master.001.patch, > HBASE-16786.master.002.patch, HBASE-16786.master.003.patch, > HBASE-16786.master.004.patch, HBASE-16786.master.005.patch, > HBASE-16786.master.006.patch, HBASE-16786.master.007.patch, > HBASE-16786.master.008.patch, HBASE-16786.master.009.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825562#comment-15825562
]
Hadoop QA commented on HBASE-17475:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 32s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 37s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 37s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12847737/HBASE-17475.v2.patch |
| JIRA Issue | HBASE-17475 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7b2c852c2edc 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 4cb09a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5289/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5289/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
[jira] [Updated] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17475:
---
Attachment: HBASE-17475.v2.patch
Have no idea why Hudson failed, resubmit the patch to re-run.
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475-branch-1.v3.patch,
> HBASE-17475.patch, HBASE-17475.v2.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
>
[jira] [Updated] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17475:
---
Attachment: (was: HBASE-17475.v2.patch)
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475-branch-1.v3.patch,
> HBASE-17475.patch, HBASE-17475.v2.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
>
[jira] [Updated] (HBASE-17361) Make HTable thread safe
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-17361:
--
Priority: Critical (was: Major)
The AsyncTable is thread safe now. I think we need to make HTable also thread
safe ASAP. [~carp84]
> Make HTable thread safe
> ---
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Currently HTable is marked as NOT thread safe, and this JIRA target at
> improving this to take better usage of the thread-safe BufferedMutator.
> Some findings/work done:
> If we try to do put to the same HTable instance in parallel, there'll be
> problem, since now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> After fixing this, we will find quite some contention on
> {{BufferedMutatorImpl#flush}}, so more efforts required to make HTable thread
> safe but with good performance meanwhile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17372) Make AsyncTable thread safe
[ https://issues.apache.org/jira/browse/HBASE-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17372: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks all for reviewing. > Make AsyncTable thread safe > --- > > Key: HBASE-17372 > URL: https://issues.apache.org/jira/browse/HBASE-17372 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17372.patch, HBASE-17372-v1.patch, > HBASE-17372-v2.patch, HBASE-17372-v3.patch, HBASE-17372-v4.patch, > HBASE-17372-v5.patch, HBASE-17372-v6.patch, HBASE-17372-v7.patch > > > The most methods are already thread safe. The problem is that we have some > methods that used to set timeout, we need to remove these methods and add a > parameter for each call to specific timeout settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17372) Make AsyncTable thread safe
[ https://issues.apache.org/jira/browse/HBASE-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825536#comment-15825536 ] Duo Zhang commented on HBASE-17372: --- The failed UT is known to be flaky I think. Will commit shortly > Make AsyncTable thread safe > --- > > Key: HBASE-17372 > URL: https://issues.apache.org/jira/browse/HBASE-17372 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17372.patch, HBASE-17372-v1.patch, > HBASE-17372-v2.patch, HBASE-17372-v3.patch, HBASE-17372-v4.patch, > HBASE-17372-v5.patch, HBASE-17372-v6.patch, HBASE-17372-v7.patch > > > The most methods are already thread safe. The problem is that we have some > methods that used to set timeout, we need to remove these methods and add a > parameter for each call to specific timeout settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16786) Procedure V2 - Move ZK-lock's uses to Procedure framework locks (LockProcedure)
[ https://issues.apache.org/jira/browse/HBASE-16786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16786: -- Attachment: HBASE-16786.master.009.patch > Procedure V2 - Move ZK-lock's uses to Procedure framework locks > (LockProcedure) > --- > > Key: HBASE-16786 > URL: https://issues.apache.org/jira/browse/HBASE-16786 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Matteo Bertozzi > Attachments: HBASE-16786.master.001.patch, > HBASE-16786.master.002.patch, HBASE-16786.master.003.patch, > HBASE-16786.master.004.patch, HBASE-16786.master.005.patch, > HBASE-16786.master.006.patch, HBASE-16786.master.007.patch, > HBASE-16786.master.008.patch, HBASE-16786.master.009.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825521#comment-15825521
]
Hadoop QA commented on HBASE-17475:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 8s
{color} | {color:red} root in master failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} master passed with JDK v1.8.0_111 {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 6s
{color} | {color:red} hbase-client in master failed with JDK v1.7.0_80. {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m 8s
{color} | {color:red} hbase-client in master failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 5s
{color} | {color:red} hbase-client in master failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} master passed with JDK v1.8.0_111 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 7s
{color} | {color:red} hbase-client in master failed with JDK v1.7.0_80. {color}
|
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 6s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 7s
{color} | {color:red} hbase-client in the patch failed with JDK v1.7.0_80.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 7s {color}
| {color:red} hbase-client in the patch failed with JDK v1.7.0_80. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m 6s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 6s
{color} | {color:red} The patch causes 7 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 12s
{color} | {color:red} The patch causes 7 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 17s
{color} | {color:red} The patch causes 7 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 23s
{color} | {color:red} The patch causes 7 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 29s
{color} | {color:red} The patch causes 7 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 35s
{color} | {color:red} The patch causes 7 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 40s
{color} | {color:red} The patch causes 7 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 46s
{color} | {color:red} The patch causes 7 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 52s
{color} | {color:red} The patch causes 7 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 6s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 6s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc
[jira] [Updated] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17475:
---
Attachment: HBASE-17475.v2.patch
modified master patch attached
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475-branch-1.v3.patch,
> HBASE-17475.patch, HBASE-17475.v2.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
>
[jira] [Commented] (HBASE-16119) Procedure v2 - Reimplement merge
[ https://issues.apache.org/jira/browse/HBASE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825511#comment-15825511 ] stack commented on HBASE-16119: --- Thanks [~ashish singhi] > Procedure v2 - Reimplement merge > > > Key: HBASE-16119 > URL: https://issues.apache.org/jira/browse/HBASE-16119 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Region Assignment >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-16119.v1-master.patch, HBASE-16119.v2-master.patch > > > use the proc-v2 state machine for merge. also update the logic to have a > single meta-writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16119) Procedure v2 - Reimplement merge
[ https://issues.apache.org/jira/browse/HBASE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825510#comment-15825510 ] Ashish Singhi commented on HBASE-16119: --- bq. Adds a method to Admin. Is this an incompat change? Allowed but should we tick the incompat box? In master branch we will be supporting only java8 for now, so we can add default methods in the interface and avoid the binary incompatibility. https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html > Procedure v2 - Reimplement merge > > > Key: HBASE-16119 > URL: https://issues.apache.org/jira/browse/HBASE-16119 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Region Assignment >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-16119.v1-master.patch, HBASE-16119.v2-master.patch > > > use the proc-v2 state machine for merge. also update the logic to have a > single meta-writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16786) Procedure V2 - Move ZK-lock's uses to Procedure framework locks (LockProcedure)
[
https://issues.apache.org/jira/browse/HBASE-16786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825507#comment-15825507
]
Hadoop QA commented on HBASE-16786:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 11s
{color} | {color:red} hbase-procedure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 22s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 10s
{color} | {color:red} hbase-procedure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 22s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 10s {color}
| {color:red} hbase-procedure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 22s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 43s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 26s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 10s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 53s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 35s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 18s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 0s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.7.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 44s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.7.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 26s
{color} | {color:red} The patch causes 16 errors with Hadoop v3.0.0-alpha1.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 8s
{color} | {color:red} hbase-procedure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 19s
{color} | {color:red} hbase-server in the patch
[jira] [Commented] (HBASE-17474) Reduce frequency of NoSuchMethodException when calling setStoragePolicy()
[
https://issues.apache.org/jira/browse/HBASE-17474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825506#comment-15825506
]
Ted Yu commented on HBASE-17474:
Sounds good.
> Reduce frequency of NoSuchMethodException when calling setStoragePolicy()
> -
>
> Key: HBASE-17474
> URL: https://issues.apache.org/jira/browse/HBASE-17474
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Yu Li
>Priority: Minor
>
> Saw the following in test output:
> {code}
> 2017-01-15 09:37:53,383 WARN
> [StoreOpener-98f4c892b5bcd9188f06331426e710f9-1]
> regionserver.HRegionFileSystem(199): Failed to set storage policy of
> [/Users/tyu/trunk/hbase-
> server/target/test-data/e052b6ad-02e2-41bf-a806-d617900a8c22/TestStoreFileRefresherChore/data/default/testIsStale/98f4c892b5bcd9188f06331426e710f9/cf]
> to [HOT]
> java.lang.UnsupportedOperationException: Cannot find specified method
> setStoragePolicy
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:209)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:197)
> at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:244)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5254)
> at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:988)
> at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:985)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy(org.apache.hadoop.fs.Path,
> java.lang.String)
> at java.lang.Class.getMethod(Class.java:1786)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:205)
> ... 11 more
> {code}
> setStoragePolicy() is not implemented by LocalFileSystem.
> We should reduce the frequency of the above log since it clutters test output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825508#comment-15825508 ] stack commented on HBASE-16630: --- [~dvdreddy] And thanks for trying my suggestion though it sounds like it was of no use. [~zjushch] Good to hear from you. You running w/ this patch or do you see this in your deploy? Thanks sir. > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: 16630-v2-suggest.patch, 16630-v3-suggest.patch, > HBASE-16630.patch, HBASE-16630-v2.patch, HBASE-16630-v3.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825504#comment-15825504
]
Duo Zhang commented on HBASE-17475:
---
Can you attach a patch for master? Just rename the patch to
HBASE-17475.v2.patch. Let's run a pre commit job on master. Will commit if we
can pass the pre commit check.
Thanks.
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475-branch-1.v3.patch,
> HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
>
[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825501#comment-15825501 ] Ashish Singhi commented on HBASE-9393: -- Will get back to this very soon with patches for all the versions. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, > HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, > HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, > HBASE-9393.v1.patch, HBASE-9393.v2.patch, HBASE-9393.v3.patch, > HBASE-9393.v4.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v7.patch, HBASE-9393.v8.patch, > HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17468) unread messages in TCP connections - possible connection leak
[ https://issues.apache.org/jira/browse/HBASE-17468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825498#comment-15825498 ] Ashish Singhi commented on HBASE-17468: --- We have been using that solution internally for around a year now. bq. Would it work with 1.2.0? Yes. We have back ported it to 1.0.2 version FYI bq. but would it cause any side effects? We didn't see any. Note: HBASE-9393 patch will work only if your using one of the 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 Hadoop version. > unread messages in TCP connections - possible connection leak > - > > Key: HBASE-17468 > URL: https://issues.apache.org/jira/browse/HBASE-17468 > Project: HBase > Issue Type: Bug >Reporter: Shridhar Sahukar >Priority: Critical > > We are running HBase 1.2.0-cdh5.7.1 (Cloudera distribution). > On our Hadoop cluster, we are seeing that each HBase region server has large > number of TCP connections to all the HDFS data nodes and all these > connections have unread data in socket buffers. Some of these connections are > also in CLOSE_WAIT or FIN_WAIT1 state while the rest are in ESTABLISHED state. > Looks like HBase is creating some connections requesting data from HDFS, but > its forgetting about those connections before it could read the data. Thus > the connections are left lingering around with large data stuck in their > receive buffers. Also, it seems HDFS closes these connections after a while, > but since there is data in receive buffer the connection is left in > CLOSE_WAIT/FIN_WAIT1 states. > Below is a snapshot from one of the region servers: > ## Total number of connections to HDFS (pid of region server is 143722) > [bda@md-bdadev-42 hbase]$ sudo netstat -anp|grep 143722 | wc -l > 827 > ## Connections that are not in ESTABLISHED state > [bda@md-bdadev-42 hbase]$ sudo netstat -anp|grep 143722 | grep -v ESTABLISHED > | wc -l > 344 > ##Snapshot of some of these connections: > tcp 133887 0 146.1.180.43:48533 146.1.180.40:50010 > ESTABLISHED 143722/java > tcp82934 0 146.1.180.43:59647 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp0 0 146.1.180.43:50761 146.1.180.27:2181 > ESTABLISHED 143722/java > tcp 234084 0 146.1.180.43:58335 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp 967667 0 146.1.180.43:56136 146.1.180.68:50010 > ESTABLISHED 143722/java > tcp 156037 0 146.1.180.43:59659 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp 212488 0 146.1.180.43:56810 146.1.180.48:50010 > ESTABLISHED 143722/java > tcp61871 0 146.1.180.43:53593 146.1.180.35:50010 > ESTABLISHED 143722/java > tcp 121216 0 146.1.180.43:35324 146.1.180.38:50010 > ESTABLISHED 143722/java > tcp1 0 146.1.180.43:32982 146.1.180.42:50010 > CLOSE_WAIT 143722/java > tcp82934 0 146.1.180.43:42359 146.1.180.54:50010 > ESTABLISHED 143722/java > tcp 159422 0 146.1.180.43:59731 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp 134573 0 146.1.180.43:60210 146.1.180.76:50010 > ESTABLISHED 143722/java > tcp82934 0 146.1.180.43:59713 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp 135765 0 146.1.180.43:44412 146.1.180.29:50010 > ESTABLISHED 143722/java > tcp 161655 0 146.1.180.43:43117 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp75990 0 146.1.180.43:59729 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp78583 0 146.1.180.43:59971 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp1 0 146.1.180.43:39893 146.1.180.67:50010 > CLOSE_WAIT 143722/java > tcp1 0 146.1.180.43:38834 146.1.180.47:50010 > CLOSE_WAIT 143722/java > tcp1 0 146.1.180.43:40707 146.1.180.50:50010 > CLOSE_WAIT 143722/java > tcp 106102 0 146.1.180.43:48208 146.1.180.75:50010 > ESTABLISHED 143722/java > tcp 332013 0 146.1.180.43:34795 146.1.180.37:50010 > ESTABLISHED 143722/java > tcp1 0 146.1.180.43:57644 146.1.180.67:50010 > CLOSE_WAIT 143722/java > tcp79119 0 146.1.180.43:54438 146.1.180.70:50010 > ESTABLISHED 143722/java > tcp77438 0 146.1.180.43:35259 146.1.180.38:50010 > ESTABLISHED 143722/java > tcp1 0 146.1.180.43:57579 146.1.180.41:50010 > CLOSE_WAIT 143722/java > tcp 318091 0 146.1.180.43:60124
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825495#comment-15825495
]
Allan Yang commented on HBASE-17475:
comment added in V3
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475-branch-1.v3.patch,
> HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
>
[jira] [Updated] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17475:
---
Attachment: HBASE-17475-branch-1.v3.patch
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475-branch-1.v3.patch,
> HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
>
[jira] [Assigned] (HBASE-17474) Reduce frequency of NoSuchMethodException when calling setStoragePolicy()
[
https://issues.apache.org/jira/browse/HBASE-17474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li reassigned HBASE-17474:
-
Assignee: Yu Li
> Reduce frequency of NoSuchMethodException when calling setStoragePolicy()
> -
>
> Key: HBASE-17474
> URL: https://issues.apache.org/jira/browse/HBASE-17474
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Yu Li
>Priority: Minor
>
> Saw the following in test output:
> {code}
> 2017-01-15 09:37:53,383 WARN
> [StoreOpener-98f4c892b5bcd9188f06331426e710f9-1]
> regionserver.HRegionFileSystem(199): Failed to set storage policy of
> [/Users/tyu/trunk/hbase-
> server/target/test-data/e052b6ad-02e2-41bf-a806-d617900a8c22/TestStoreFileRefresherChore/data/default/testIsStale/98f4c892b5bcd9188f06331426e710f9/cf]
> to [HOT]
> java.lang.UnsupportedOperationException: Cannot find specified method
> setStoragePolicy
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:209)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:197)
> at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:244)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5254)
> at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:988)
> at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:985)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy(org.apache.hadoop.fs.Path,
> java.lang.String)
> at java.lang.Class.getMethod(Class.java:1786)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:205)
> ... 11 more
> {code}
> setStoragePolicy() is not implemented by LocalFileSystem.
> We should reduce the frequency of the above log since it clutters test output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16119) Procedure v2 - Reimplement merge
[ https://issues.apache.org/jira/browse/HBASE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825489#comment-15825489 ] stack commented on HBASE-16119: --- See comments above [~syuanjiang] > Procedure v2 - Reimplement merge > > > Key: HBASE-16119 > URL: https://issues.apache.org/jira/browse/HBASE-16119 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Region Assignment >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-16119.v1-master.patch, HBASE-16119.v2-master.patch > > > use the proc-v2 state machine for merge. also update the logic to have a > single meta-writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17474) Reduce frequency of NoSuchMethodException when calling setStoragePolicy()
[
https://issues.apache.org/jira/browse/HBASE-17474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825486#comment-15825486
]
Yu Li commented on HBASE-17474:
---
Maybe simply add an if clause to exempt LocalFileSystem could fix this, like
below:
{code}
if (!(this.fs instanceof LocalFileSystem)) {
LOG.warn("Failed to set storage policy of [" + storeDir + "] to [" +
policyName + "]", e);
}
{code}
Sounds good [~tedyu]? If so, will make a straight-forward patch. Thanks.
> Reduce frequency of NoSuchMethodException when calling setStoragePolicy()
> -
>
> Key: HBASE-17474
> URL: https://issues.apache.org/jira/browse/HBASE-17474
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> Saw the following in test output:
> {code}
> 2017-01-15 09:37:53,383 WARN
> [StoreOpener-98f4c892b5bcd9188f06331426e710f9-1]
> regionserver.HRegionFileSystem(199): Failed to set storage policy of
> [/Users/tyu/trunk/hbase-
> server/target/test-data/e052b6ad-02e2-41bf-a806-d617900a8c22/TestStoreFileRefresherChore/data/default/testIsStale/98f4c892b5bcd9188f06331426e710f9/cf]
> to [HOT]
> java.lang.UnsupportedOperationException: Cannot find specified method
> setStoragePolicy
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:209)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:197)
> at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:244)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5254)
> at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:988)
> at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:985)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy(org.apache.hadoop.fs.Path,
> java.lang.String)
> at java.lang.Class.getMethod(Class.java:1786)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:205)
> ... 11 more
> {code}
> setStoragePolicy() is not implemented by LocalFileSystem.
> We should reduce the frequency of the above log since it clutters test output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825483#comment-15825483
]
Duo Zhang commented on HBASE-17475:
---
+1.
I think it could work but I suggest that we add a comment to tell why we do
this. It is a little confusing.
Thanks.
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825479#comment-15825479 ] stack commented on HBASE-16630: --- [~dvdreddy] You think we should commit this last patch? Thanks. > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: 16630-v2-suggest.patch, 16630-v3-suggest.patch, > HBASE-16630.patch, HBASE-16630-v2.patch, HBASE-16630-v3.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825473#comment-15825473
]
Allan Yang commented on HBASE-17475:
Yes, it is intended to control the stack depth, also prevent the problem which
Duo pointed out: after HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER, thread
pool is always used.
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
>
[jira] [Updated] (HBASE-16786) Procedure V2 - Move ZK-lock's uses to Procedure framework locks (LockProcedure)
[ https://issues.apache.org/jira/browse/HBASE-16786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16786: -- Status: Patch Available (was: Open) > Procedure V2 - Move ZK-lock's uses to Procedure framework locks > (LockProcedure) > --- > > Key: HBASE-16786 > URL: https://issues.apache.org/jira/browse/HBASE-16786 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Matteo Bertozzi > Attachments: HBASE-16786.master.001.patch, > HBASE-16786.master.002.patch, HBASE-16786.master.003.patch, > HBASE-16786.master.004.patch, HBASE-16786.master.005.patch, > HBASE-16786.master.006.patch, HBASE-16786.master.007.patch, > HBASE-16786.master.008.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825393#comment-15825393
]
Ted Yu commented on HBASE-17475:
Is this correct ?
{code}
&& numAttempt %
HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER != 0) {
{code}
Only multiples of DEFAULT_HBASE_CLIENT_RETRIES_NUMBER would be submitted to
pool. Is that intended ?
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
>
[jira] [Updated] (HBASE-17443) Move listReplicated/enableTableRep/disableTableRep methods from ReplicationAdmin to Admin
[ https://issues.apache.org/jira/browse/HBASE-17443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17443: --- Attachment: HBASE-17443-v5.patch > Move listReplicated/enableTableRep/disableTableRep methods from > ReplicationAdmin to Admin > - > > Key: HBASE-17443 > URL: https://issues.apache.org/jira/browse/HBASE-17443 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17443-v1.patch, HBASE-17443-v2.patch, > HBASE-17443-v2.patch, HBASE-17443-v3.patch, HBASE-17443-v3.patch, > HBASE-17443-v4.patch, HBASE-17443-v5.patch > > > We have moved other replication requests to Admin and mark ReplicationAdmin > as Deprecated, so listReplicated/enableTableRep/disableTableRep methods need > move to Admin, too. > Review board: https://reviews.apache.org/r/55534/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10607) [JDK8] NoSuchMethodError involving ConcurrentHashMap.keySet if running on JRE 7
[
https://issues.apache.org/jira/browse/HBASE-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825053#comment-15825053
]
stack commented on HBASE-10607:
---
Ugh. Ran into this just now in spite if bit added to refguide. Dang.
> [JDK8] NoSuchMethodError involving ConcurrentHashMap.keySet if running on JRE
> 7
> ---
>
> Key: HBASE-10607
> URL: https://issues.apache.org/jira/browse/HBASE-10607
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
>Priority: Trivial
> Attachments: 10607.txt
>
>
> If you try running HBase built with Java 8 on a Java 7 JRE, then processes
> will fail early with:
> {noformat}
> java.lang.NoSuchMethodError:
> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
> {noformat}
> I came across this incidentally. I suppose we should consider documenting
> this in the manual in the Troubleshooting section.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824999#comment-15824999
]
Hadoop QA commented on HBASE-17475:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 42s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 6s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12847729/HBASE-17475-branch-1.v2.patch
|
| JIRA Issue | HBASE-17475 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 60cba01763b3 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 /
[jira] [Commented] (HBASE-17469) Properly handle empty TableName in TablePermission#readFields and #write
[
https://issues.apache.org/jira/browse/HBASE-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824983#comment-15824983
]
Ted Yu commented on HBASE-17469:
You were talking about line 215, right ?
{code}
if (!this.table.equals(table)) {
{code}
Please fix in this JIRA.
> Properly handle empty TableName in TablePermission#readFields and #write
>
>
> Key: HBASE-17469
> URL: https://issues.apache.org/jira/browse/HBASE-17469
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Manjunath Anand
> Attachments: HBASE-17469.master.001.patch
>
>
> HBASE-17450 handles the empty table name in equals().
> This JIRA is to properly handle empty TableName in
> TablePermission#readFields() and TablePermission#write() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17469) Properly handle empty TableName in TablePermission#readFields and #write
[ https://issues.apache.org/jira/browse/HBASE-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824978#comment-15824978 ] Manjunath Anand commented on HBASE-17469: - Sure [~yuzhih...@gmail.com] but before that I would like to know if I can add null check for table in TablePermission #implies and #matchesFamily methods and fix the toString() method (if-elseif-else condition seems to be incorrect) as well as part of this JIRA and come up with new patch. > Properly handle empty TableName in TablePermission#readFields and #write > > > Key: HBASE-17469 > URL: https://issues.apache.org/jira/browse/HBASE-17469 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Manjunath Anand > Attachments: HBASE-17469.master.001.patch > > > HBASE-17450 handles the empty table name in equals(). > This JIRA is to properly handle empty TableName in > TablePermission#readFields() and TablePermission#write() methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17469) Properly handle empty TableName in TablePermission#readFields and #write
[ https://issues.apache.org/jira/browse/HBASE-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824969#comment-15824969 ] Ted Yu commented on HBASE-17469: Good. Can you submit the patch for QA run ? > Properly handle empty TableName in TablePermission#readFields and #write > > > Key: HBASE-17469 > URL: https://issues.apache.org/jira/browse/HBASE-17469 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Manjunath Anand > Attachments: HBASE-17469.master.001.patch > > > HBASE-17450 handles the empty table name in equals(). > This JIRA is to properly handle empty TableName in > TablePermission#readFields() and TablePermission#write() methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17469) Properly handle empty TableName in TablePermission#readFields and #write
[
https://issues.apache.org/jira/browse/HBASE-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824959#comment-15824959
]
Manjunath Anand edited comment on HBASE-17469 at 1/17/17 4:35 AM:
--
I get the below error for testMigration testcase without the patch fix.
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.security.access.TablePermission.write(TablePermission.java:405)
at
org.apache.hadoop.hbase.security.access.HbaseObjectWritableFor96Migration.writeObject(HbaseObjectWritableFor96Migration.java:515)
at
org.apache.hadoop.hbase.security.access.HbaseObjectWritableFor96Migration.writeObject(HbaseObjectWritableFor96Migration.java:474)
at
org.apache.hadoop.hbase.security.access.TestTablePermissions.writePermissions(TestTablePermissions.java:172)
at
org.apache.hadoop.hbase.security.access.TestTablePermissions.writePermissionsAsBytes(TestTablePermissions.java:150)
at
org.apache.hadoop.hbase.security.access.TestTablePermissions.testMigration(TestTablePermissions.java:137)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
{code}
Thanks
was (Author: manju_hadoop):
I get the below error for testMigration testcase.
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.security.access.TablePermission.write(TablePermission.java:405)
at
org.apache.hadoop.hbase.security.access.HbaseObjectWritableFor96Migration.writeObject(HbaseObjectWritableFor96Migration.java:515)
at
org.apache.hadoop.hbase.security.access.HbaseObjectWritableFor96Migration.writeObject(HbaseObjectWritableFor96Migration.java:474)
at
org.apache.hadoop.hbase.security.access.TestTablePermissions.writePermissions(TestTablePermissions.java:172)
at
org.apache.hadoop.hbase.security.access.TestTablePermissions.writePermissionsAsBytes(TestTablePermissions.java:150)
at
org.apache.hadoop.hbase.security.access.TestTablePermissions.testMigration(TestTablePermissions.java:137)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
[jira] [Commented] (HBASE-17469) Properly handle empty TableName in TablePermission#readFields and #write
[
https://issues.apache.org/jira/browse/HBASE-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824959#comment-15824959
]
Manjunath Anand commented on HBASE-17469:
-
I get the below error for testMigration testcase.
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.security.access.TablePermission.write(TablePermission.java:405)
at
org.apache.hadoop.hbase.security.access.HbaseObjectWritableFor96Migration.writeObject(HbaseObjectWritableFor96Migration.java:515)
at
org.apache.hadoop.hbase.security.access.HbaseObjectWritableFor96Migration.writeObject(HbaseObjectWritableFor96Migration.java:474)
at
org.apache.hadoop.hbase.security.access.TestTablePermissions.writePermissions(TestTablePermissions.java:172)
at
org.apache.hadoop.hbase.security.access.TestTablePermissions.writePermissionsAsBytes(TestTablePermissions.java:150)
at
org.apache.hadoop.hbase.security.access.TestTablePermissions.testMigration(TestTablePermissions.java:137)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
{code}
Thanks
> Properly handle empty TableName in TablePermission#readFields and #write
>
>
> Key: HBASE-17469
> URL: https://issues.apache.org/jira/browse/HBASE-17469
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Manjunath Anand
> Attachments: HBASE-17469.master.001.patch
>
>
> HBASE-17450 handles the empty table name in equals().
> This JIRA is to properly handle empty TableName in
> TablePermission#readFields() and TablePermission#write() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824947#comment-15824947
]
Allan Yang commented on HBASE-17475:
Thanks, Duo, modified the patch as you pointed out, but use a more simple way.
Keeping the deepest stack depth to a certain number.
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
>
[jira] [Updated] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17475:
---
Attachment: HBASE-17475-branch-1.v2.patch
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch,
> HBASE-17475-branch-1.v2.patch, HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
>
[jira] [Commented] (HBASE-17372) Make AsyncTable thread safe
[
https://issues.apache.org/jira/browse/HBASE-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824941#comment-15824941
]
Hadoop QA commented on HBASE-17372:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 1s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 31s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 55s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12847716/HBASE-17372-v7.patch |
| JIRA Issue | HBASE-17372 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux aafdb4237162 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4cb09a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5283/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
[jira] [Commented] (HBASE-17469) Properly handle empty TableName in TablePermission#readFields and #write
[ https://issues.apache.org/jira/browse/HBASE-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824934#comment-15824934 ] Ted Yu commented on HBASE-17469: For the permission keyed by "bruce" in TestTablePermissions, what exception is thrown without the fix ? Thanks > Properly handle empty TableName in TablePermission#readFields and #write > > > Key: HBASE-17469 > URL: https://issues.apache.org/jira/browse/HBASE-17469 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Manjunath Anand > Attachments: HBASE-17469.master.001.patch > > > HBASE-17450 handles the empty table name in equals(). > This JIRA is to properly handle empty TableName in > TablePermission#readFields() and TablePermission#write() methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824930#comment-15824930
]
Duo Zhang commented on HBASE-17475:
---
This is means we will always resubmit using the thread pool if the numAttempt
is greater than HConstants.DEFAULT_HBASE_CLIENT_RETRIES_NUMBER? Yeah it works
but I think there are better solutions.
See the code of netty
https://github.com/netty/netty/blob/b2379e62f4ae50b55bce7c43e5bd008cf9aacfd1/transport/src/main/java/io/netty/channel/local/LocalChannel.java#L310
Where they use a thread local variable to record the depth of stack, if it
reaches the limit then submit using a thread pool, otherwise just call the
runnable directly.
Thanks.
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch, HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
>
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824914#comment-15824914
]
Hadoop QA commented on HBASE-17475:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 44s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
0s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 41s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 8s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12847724/HBASE-17475-branch-1.patch
|
| JIRA Issue | HBASE-17475 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a96ad7d9ca6f 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 /
[jira] [Commented] (HBASE-17469) Properly handle empty TableName in TablePermission#readFields and #write
[ https://issues.apache.org/jira/browse/HBASE-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824895#comment-15824895 ] Manjunath Anand commented on HBASE-17469: - Hi [~yuzhih...@gmail.com] I have come up with a initial patch which addresses the issue with TablePermission#readFields and #write in a backward compatible way as you mentioned. I found the toString() method if-elseif-else condition wrong and also wanted to know if I can add null check for table in implies and matchesFamily methods and fix the toString() method as well as part of this JIRA. I found a potential bug in AccessControlUtil#toTablePermission during testing and will raise a seperate JIRA soon. > Properly handle empty TableName in TablePermission#readFields and #write > > > Key: HBASE-17469 > URL: https://issues.apache.org/jira/browse/HBASE-17469 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Manjunath Anand > Attachments: HBASE-17469.master.001.patch > > > HBASE-17450 handles the empty table name in equals(). > This JIRA is to properly handle empty TableName in > TablePermission#readFields() and TablePermission#write() methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17469) Properly handle empty TableName in TablePermission#readFields and #write
[ https://issues.apache.org/jira/browse/HBASE-17469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manjunath Anand updated HBASE-17469: Attachment: HBASE-17469.master.001.patch > Properly handle empty TableName in TablePermission#readFields and #write > > > Key: HBASE-17469 > URL: https://issues.apache.org/jira/browse/HBASE-17469 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Manjunath Anand > Attachments: HBASE-17469.master.001.patch > > > HBASE-17450 handles the empty table name in equals(). > This JIRA is to properly handle empty TableName in > TablePermission#readFields() and TablePermission#write() methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824883#comment-15824883
]
Ted Yu commented on HBASE-17475:
lgtm
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch, HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
>
[jira] [Updated] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17475:
---
Attachment: HBASE-17475-branch-1.patch
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475-branch-1.patch, HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
>
[jira] [Updated] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17475:
---
Attachment: HBASE-17475.patch
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17475.patch
>
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
>
[jira] [Updated] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
[
https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17475:
---
Status: Patch Available (was: Open)
> Stack overflow in AsyncProcess if retry too much
>
>
> Key: HBASE-17475
> URL: https://issues.apache.org/jira/browse/HBASE-17475
> Project: HBase
> Issue Type: Bug
> Components: API
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>
> In AsyncProcess, we resubmit the retry task in the same thread
> {code}
> // run all the runnables
> for (Runnable runnable : runnables) {
> if ((--actionsRemaining == 0) && reuseThread) {
> runnable.run();
> } else {
> try {
> pool.submit(runnable);
> }
> ..
> {code}
> But, if we retry too much time. soon, stack overflow will occur. This is very
> common in clusters with Phoenix. Phoenix need to write index table in the
> normal write path, retry will cause stack overflow exception.
> {noformat}
> "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
> nid=0x4a96 waiting on condition [0x7fe3f6301000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
> at
>
[jira] [Created] (HBASE-17475) Stack overflow in AsyncProcess if retry too much
Allan Yang created HBASE-17475:
--
Summary: Stack overflow in AsyncProcess if retry too much
Key: HBASE-17475
URL: https://issues.apache.org/jira/browse/HBASE-17475
Project: HBase
Issue Type: Bug
Components: API
Affects Versions: 2.0.0, 1.4.0
Reporter: Allan Yang
Assignee: Allan Yang
In AsyncProcess, we resubmit the retry task in the same thread
{code}
// run all the runnables
for (Runnable runnable : runnables) {
if ((--actionsRemaining == 0) && reuseThread) {
runnable.run();
} else {
try {
pool.submit(runnable);
}
..
{code}
But, if we retry too much time. soon, stack overflow will occur. This is very
common in clusters with Phoenix. Phoenix need to write index table in the
normal write path, retry will cause stack overflow exception.
{noformat}
"htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x02687800
nid=0x4a96 waiting on condition [0x7fe3f6301000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181)
at
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824847#comment-15824847 ] Ted Yu commented on HBASE-16630: Planning to commit tomorrow morning if there is no more review comment. > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: 16630-v2-suggest.patch, 16630-v3-suggest.patch, > HBASE-16630.patch, HBASE-16630-v2.patch, HBASE-16630-v3.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17372) Make AsyncTable thread safe
[ https://issues.apache.org/jira/browse/HBASE-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17372: -- Attachment: HBASE-17372-v7.patch > Make AsyncTable thread safe > --- > > Key: HBASE-17372 > URL: https://issues.apache.org/jira/browse/HBASE-17372 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17372.patch, HBASE-17372-v1.patch, > HBASE-17372-v2.patch, HBASE-17372-v3.patch, HBASE-17372-v4.patch, > HBASE-17372-v5.patch, HBASE-17372-v6.patch, HBASE-17372-v7.patch > > > The most methods are already thread safe. The problem is that we have some > methods that used to set timeout, we need to remove these methods and add a > parameter for each call to specific timeout settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-17437: -- Status: Patch Available (was: In Progress) > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-17437: -- Status: In Progress (was: Patch Available) > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824806#comment-15824806 ] Zach York commented on HBASE-17437: --- Updated patch and RB for feedback from [~enis] > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-17437: -- Status: Patch Available (was: In Progress) > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-17437: -- Status: In Progress (was: Patch Available) > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-17437: -- Attachment: HBASE-17437.master.004.patch > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, HBASE-17437.master.004.patch, > hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824679#comment-15824679
]
Hadoop QA commented on HBASE-9393:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 47s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 101m 31s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 28s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12794484/HBASE-9393.v15.patch |
| JIRA Issue | HBASE-9393 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 97ef674eb251 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4cb09a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5281/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5281/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Hbase does not closing a closed socket resulting in many CLOSE_WAIT
>
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2
>
[jira] [Commented] (HBASE-17470) Remove merge region code from region server
[ https://issues.apache.org/jira/browse/HBASE-17470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824665#comment-15824665 ] Stephen Yuan Jiang commented on HBASE-17470: The test failure has nothing to do with this patch. It is due to flakiness of the test. HBASE-17474 will address that. > Remove merge region code from region server > --- > > Key: HBASE-17470 > URL: https://issues.apache.org/jira/browse/HBASE-17470 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-17470.v1-master.patch, HBASE-17470.v2-master.patch > > > HBASE-16119 moves the merge region to the master-side. There is no need to > keep region_server-side merge region code to remove logic duplication. > util.Merge and HMerge tools depends on RS-side merge region logic. However, > now we can merge regions using shell command. It is dangerous to do offline > merge. For 2.0, it is a good time to remove those out-of-date tools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17372) Make AsyncTable thread safe
[ https://issues.apache.org/jira/browse/HBASE-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824649#comment-15824649 ] Enis Soztutar commented on HBASE-17372: --- Sorry missed the notifications. Some minor comments in RB, but looks good. We should do HBASE-17361 as well. > Make AsyncTable thread safe > --- > > Key: HBASE-17372 > URL: https://issues.apache.org/jira/browse/HBASE-17372 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17372.patch, HBASE-17372-v1.patch, > HBASE-17372-v2.patch, HBASE-17372-v3.patch, HBASE-17372-v4.patch, > HBASE-17372-v5.patch, HBASE-17372-v6.patch > > > The most methods are already thread safe. The problem is that we have some > methods that used to set timeout, we need to remove these methods and add a > parameter for each call to specific timeout settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17474) Reduce frequency of NoSuchMethodException when calling setStoragePolicy()
Ted Yu created HBASE-17474:
--
Summary: Reduce frequency of NoSuchMethodException when calling
setStoragePolicy()
Key: HBASE-17474
URL: https://issues.apache.org/jira/browse/HBASE-17474
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
Saw the following in test output:
{code}
2017-01-15 09:37:53,383 WARN [StoreOpener-98f4c892b5bcd9188f06331426e710f9-1]
regionserver.HRegionFileSystem(199): Failed to set storage policy of
[/Users/tyu/trunk/hbase-
server/target/test-data/e052b6ad-02e2-41bf-a806-d617900a8c22/TestStoreFileRefresherChore/data/default/testIsStale/98f4c892b5bcd9188f06331426e710f9/cf]
to [HOT]
java.lang.UnsupportedOperationException: Cannot find specified method
setStoragePolicy
at
org.apache.hadoop.hbase.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:209)
at
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:197)
at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:244)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5254)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:988)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:985)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy(org.apache.hadoop.fs.Path,
java.lang.String)
at java.lang.Class.getMethod(Class.java:1786)
at
org.apache.hadoop.hbase.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:205)
... 11 more
{code}
setStoragePolicy() is not implemented by LocalFileSystem.
We should reduce the frequency of the above log since it clutters test output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824530#comment-15824530 ] Ted Yu commented on HBASE-9393: --- [~ashish singhi]: Can you attach patch backported to branch-1.2 ? Thanks > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, > HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, > HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, > HBASE-9393.v1.patch, HBASE-9393.v2.patch, HBASE-9393.v3.patch, > HBASE-9393.v4.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v7.patch, HBASE-9393.v8.patch, > HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17468) unread messages in TCP connections - possible connection leak
[ https://issues.apache.org/jira/browse/HBASE-17468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824410#comment-15824410 ] Shridhar Sahukar commented on HBASE-17468: -- Thanks Anoop/Ashish. We can try the patch available at HBASE-9393. Is HBASE-9393-v15.patch the latest? The fix is marked against 2.0.0 release. Would it work with 1.2.0? I can try to backport it, but would it cause any side effects? Thanks > unread messages in TCP connections - possible connection leak > - > > Key: HBASE-17468 > URL: https://issues.apache.org/jira/browse/HBASE-17468 > Project: HBase > Issue Type: Bug >Reporter: Shridhar Sahukar >Priority: Critical > > We are running HBase 1.2.0-cdh5.7.1 (Cloudera distribution). > On our Hadoop cluster, we are seeing that each HBase region server has large > number of TCP connections to all the HDFS data nodes and all these > connections have unread data in socket buffers. Some of these connections are > also in CLOSE_WAIT or FIN_WAIT1 state while the rest are in ESTABLISHED state. > Looks like HBase is creating some connections requesting data from HDFS, but > its forgetting about those connections before it could read the data. Thus > the connections are left lingering around with large data stuck in their > receive buffers. Also, it seems HDFS closes these connections after a while, > but since there is data in receive buffer the connection is left in > CLOSE_WAIT/FIN_WAIT1 states. > Below is a snapshot from one of the region servers: > ## Total number of connections to HDFS (pid of region server is 143722) > [bda@md-bdadev-42 hbase]$ sudo netstat -anp|grep 143722 | wc -l > 827 > ## Connections that are not in ESTABLISHED state > [bda@md-bdadev-42 hbase]$ sudo netstat -anp|grep 143722 | grep -v ESTABLISHED > | wc -l > 344 > ##Snapshot of some of these connections: > tcp 133887 0 146.1.180.43:48533 146.1.180.40:50010 > ESTABLISHED 143722/java > tcp82934 0 146.1.180.43:59647 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp0 0 146.1.180.43:50761 146.1.180.27:2181 > ESTABLISHED 143722/java > tcp 234084 0 146.1.180.43:58335 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp 967667 0 146.1.180.43:56136 146.1.180.68:50010 > ESTABLISHED 143722/java > tcp 156037 0 146.1.180.43:59659 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp 212488 0 146.1.180.43:56810 146.1.180.48:50010 > ESTABLISHED 143722/java > tcp61871 0 146.1.180.43:53593 146.1.180.35:50010 > ESTABLISHED 143722/java > tcp 121216 0 146.1.180.43:35324 146.1.180.38:50010 > ESTABLISHED 143722/java > tcp1 0 146.1.180.43:32982 146.1.180.42:50010 > CLOSE_WAIT 143722/java > tcp82934 0 146.1.180.43:42359 146.1.180.54:50010 > ESTABLISHED 143722/java > tcp 159422 0 146.1.180.43:59731 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp 134573 0 146.1.180.43:60210 146.1.180.76:50010 > ESTABLISHED 143722/java > tcp82934 0 146.1.180.43:59713 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp 135765 0 146.1.180.43:44412 146.1.180.29:50010 > ESTABLISHED 143722/java > tcp 161655 0 146.1.180.43:43117 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp75990 0 146.1.180.43:59729 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp78583 0 146.1.180.43:59971 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp1 0 146.1.180.43:39893 146.1.180.67:50010 > CLOSE_WAIT 143722/java > tcp1 0 146.1.180.43:38834 146.1.180.47:50010 > CLOSE_WAIT 143722/java > tcp1 0 146.1.180.43:40707 146.1.180.50:50010 > CLOSE_WAIT 143722/java > tcp 106102 0 146.1.180.43:48208 146.1.180.75:50010 > ESTABLISHED 143722/java > tcp 332013 0 146.1.180.43:34795 146.1.180.37:50010 > ESTABLISHED 143722/java > tcp1 0 146.1.180.43:57644 146.1.180.67:50010 > CLOSE_WAIT 143722/java > tcp79119 0 146.1.180.43:54438 146.1.180.70:50010 > ESTABLISHED 143722/java > tcp77438 0 146.1.180.43:35259 146.1.180.38:50010 > ESTABLISHED 143722/java > tcp1 0 146.1.180.43:57579 146.1.180.41:50010 > CLOSE_WAIT 143722/java > tcp 318091 0 146.1.180.43:60124 146.1.180.42:50010 > ESTABLISHED 143722/java > tcp1
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824388#comment-15824388 ] Ted Yu commented on HBASE-16179: Thanks Jerry. Looking forward to your review. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, > 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, > 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, > 16179.v1.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, > 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8865) HBase shell split command acts incorrectly with hex split keys.
[ https://issues.apache.org/jira/browse/HBASE-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824384#comment-15824384 ] Wellington Chevreuil commented on HBASE-8865: - I believe this issue was originally due wrong conversion from ruby bytes to java bytes. This would already be sorted by the changes introduced on HBASE-14767. > HBase shell split command acts incorrectly with hex split keys. > --- > > Key: HBASE-8865 > URL: https://issues.apache.org/jira/browse/HBASE-8865 > Project: HBase > Issue Type: Bug > Components: shell, Usability >Affects Versions: 0.94.8 > Environment: Linux >Reporter: Ding Haifeng > Attachments: 8865.txt > > > When I tried to do a manual region split from HBase shell, I found that split > command acts incorrectly with hex split keys. > Here is an example. > I execute hbase(main):003:0> split 'tsdb', "\x00\x00\xC3" . > While I expect it to split at the 3-byte key "\x00\x00\xC3" , it actually > split at a 5-byte key "\x00\x00\xEF\xBF\xBD". > I test with more split keys and find some patterns: > * If the all bytes in the split key represented in hexadecimal are between > "\x00" and "\x7F" , it works as expected and split at exactly the key > specified. > * If there are any bytes between "\x80" and "xFF", it works incorrectly. No > matter the byte is, it is interpreted as "\xEF\xBF\xBD". Here is another > example. Specifying split key "\x00\xA0\x00\xB0" actually splits at > "\x00\xEF\xBF\xBD\x00\xEF\xBF\xBD". > I'm running Hbase 0.94.8, r1485407, both server-side and client-side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17461) HBase shell "major_compact" command should properly convert "table_or_region_name" parameter to java byte array properly before simply calling "HBaseAdmin.majorCompact
[
https://issues.apache.org/jira/browse/HBASE-17461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824383#comment-15824383
]
Wellington Chevreuil commented on HBASE-17461:
--
These issues are actually fixed already by changes applied on HBASE-14767. It
turned out I was not testing latest Master branch. Apologies for the noise.
> HBase shell "major_compact" command should properly convert
> "table_or_region_name" parameter to java byte array properly before simply
> calling "HBaseAdmin.majorCompact" method
> ---
>
> Key: HBASE-17461
> URL: https://issues.apache.org/jira/browse/HBASE-17461
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>
> On HBase shell, *major_compact* command simply passes the received
> *table_or_region_name* parameter straight to java *HBaseAdmin.majorCompact*
> method.
> On some corner cases, HBase tables row keys may have special characters.
> Then, if a region is split in such a way that row keys with special
> characters are now part of the region name, calling *major_compact* on this
> regions will fail, if the special character ASCII code is higher than 127.
> This happens because Java byte type is signed, while ruby byte type isn't,
> causing the region name to be converted to a wrong string at Java side.
> For example, considering a region named as below:
> {noformat}
> test,\xF8\xB9B2!$\x9C\x0A\xFEG\xC0\xE3\x8B\x1B\xFF\x15,1481745228583.b4bc69356d89018bfad3ee106b717285.
> {noformat}
> Calling major_compat on it fails as follows:
> {noformat}
> hbase(main):008:0* major_compact
> "test,\xF8\xB9B2!$\x9C\x0A\xFEG\xC0\xE3\x8B\x1B\xFF\x15,1484177359169.8128fa75ae0cd4eba38da2667ac8ec98."
> ERROR: Illegal character code:44, <,> at 4. User-space table qualifiers can
> only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9-.]: test,�B2!$�
> �G���1484177359169.8128fa75ae0cd4eba38da2667ac8ec98.
> {noformat}
> An easy solution is to convert *table_or_region_name* parameter properly,
> prior to calling *HBaseAdmin.majorCompact* in the same way as it's already
> done on some other shell commands, such as *get*:
> {noformat}
> admin.major_compact(table_or_region_name.to_s.to_java_bytes, family)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-17461) HBase shell "major_compact" command should properly convert "table_or_region_name" parameter to java byte array properly before simply calling "HBaseAdmin.majorCompact"
[
https://issues.apache.org/jira/browse/HBASE-17461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil resolved HBASE-17461.
--
Resolution: Invalid
> HBase shell "major_compact" command should properly convert
> "table_or_region_name" parameter to java byte array properly before simply
> calling "HBaseAdmin.majorCompact" method
> ---
>
> Key: HBASE-17461
> URL: https://issues.apache.org/jira/browse/HBASE-17461
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>
> On HBase shell, *major_compact* command simply passes the received
> *table_or_region_name* parameter straight to java *HBaseAdmin.majorCompact*
> method.
> On some corner cases, HBase tables row keys may have special characters.
> Then, if a region is split in such a way that row keys with special
> characters are now part of the region name, calling *major_compact* on this
> regions will fail, if the special character ASCII code is higher than 127.
> This happens because Java byte type is signed, while ruby byte type isn't,
> causing the region name to be converted to a wrong string at Java side.
> For example, considering a region named as below:
> {noformat}
> test,\xF8\xB9B2!$\x9C\x0A\xFEG\xC0\xE3\x8B\x1B\xFF\x15,1481745228583.b4bc69356d89018bfad3ee106b717285.
> {noformat}
> Calling major_compat on it fails as follows:
> {noformat}
> hbase(main):008:0* major_compact
> "test,\xF8\xB9B2!$\x9C\x0A\xFEG\xC0\xE3\x8B\x1B\xFF\x15,1484177359169.8128fa75ae0cd4eba38da2667ac8ec98."
> ERROR: Illegal character code:44, <,> at 4. User-space table qualifiers can
> only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9-.]: test,�B2!$�
> �G���1484177359169.8128fa75ae0cd4eba38da2667ac8ec98.
> {noformat}
> An easy solution is to convert *table_or_region_name* parameter properly,
> prior to calling *HBaseAdmin.majorCompact* in the same way as it's already
> done on some other shell commands, such as *get*:
> {noformat}
> admin.major_compact(table_or_region_name.to_s.to_java_bytes, family)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824369#comment-15824369 ] Zach York commented on HBASE-17437: --- [~stack][~enis][~busbey][~tedyu] Can I get someone to review the latest patch. I tried to include all the feedback, but I could have missed something. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Zach York > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, > HBASE-17437.master.001.patch, HBASE-17437.master.002.patch, > HBASE-17437.master.003.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17473) Set capacity on ArrayList in hbase-client
[
https://issues.apache.org/jira/browse/HBASE-17473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824342#comment-15824342
]
Hadoop QA commented on HBASE-17473:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 18s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 5s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 12s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12847652/HBASE-17473.master.002.patch
|
| JIRA Issue | HBASE-17473 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a73dc51dbdb5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4cb09a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5280/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5280/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Set capacity on ArrayList in hbase-client
> -
>
> Key: HBASE-17473
> URL: https://issues.apache.org/jira/browse/HBASE-17473
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Minor
> Attachments: HBASE-17473.master.001.patch,
> HBASE-17473.master.002.patch
>
>
> Set
[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk
[ https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824307#comment-15824307 ] Edward Bortnikov commented on HBASE-17081: -- _v13 was out of sync with trunk (QA ran 1 day after submission). Rebase solved the problem. > Flush the entire CompactingMemStore content to disk > --- > > Key: HBASE-17081 > URL: https://issues.apache.org/jira/browse/HBASE-17081 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, > HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, > HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, > HBASE-17081-V07.patch, HBASE-17081-V10.patch, HBASE-17081-V13.patch, > HBASE-17081-V14.patch, HBaseMeetupDecember2016-V02.pptx, > Pipelinememstore_fortrunk_3.patch > > > Part of CompactingMemStore's memory is held by an active segment, and another > part is divided between immutable segments in the compacting pipeline. Upon > flush-to-disk request we want to flush all of it to disk, in contrast to > flushing only tail of the compacting pipeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17473) Set capacity on ArrayList in hbase-client
[ https://issues.apache.org/jira/browse/HBASE-17473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Hentschel updated HBASE-17473: -- Attachment: HBASE-17473.master.002.patch > Set capacity on ArrayList in hbase-client > - > > Key: HBASE-17473 > URL: https://issues.apache.org/jira/browse/HBASE-17473 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Attachments: HBASE-17473.master.001.patch, > HBASE-17473.master.002.patch > > > Set the capacity on an ArrayList when possible in the *hbase-client* module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk
[ https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824284#comment-15824284 ] ramkrishna.s.vasudevan commented on HBASE-17081: What is there in the updated patch now. Was about to check it and saw _v13 had caused a test case issue. > Flush the entire CompactingMemStore content to disk > --- > > Key: HBASE-17081 > URL: https://issues.apache.org/jira/browse/HBASE-17081 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, > HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, > HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, > HBASE-17081-V07.patch, HBASE-17081-V10.patch, HBASE-17081-V13.patch, > HBASE-17081-V14.patch, HBaseMeetupDecember2016-V02.pptx, > Pipelinememstore_fortrunk_3.patch > > > Part of CompactingMemStore's memory is held by an active segment, and another > part is divided between immutable segments in the compacting pipeline. Upon > flush-to-disk request we want to flush all of it to disk, in contrast to > flushing only tail of the compacting pipeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17473) Set capacity on ArrayList in hbase-client
[
https://issues.apache.org/jira/browse/HBASE-17473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824259#comment-15824259
]
Hadoop QA commented on HBASE-17473:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
7s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 24s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 7s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 0s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12847647/HBASE-17473.master.001.patch
|
| JIRA Issue | HBASE-17473 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a210f1e790f8 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4cb09a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/5279/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5279/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5279/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Set capacity on ArrayList in hbase-client
> -
>
> Key: HBASE-17473
> URL: https://issues.apache.org/jira/browse/HBASE-17473
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Jan Hentschel
>
[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk
[
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824218#comment-15824218
]
Hadoop QA commented on HBASE-17081:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 46s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 83m 11s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 120m 58s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12847625/HBASE-17081-V14.patch
|
| JIRA Issue | HBASE-17081 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux cb66ce12bc1f 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4cb09a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5278/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5278/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824206#comment-15824206 ] Ted Yu commented on HBASE-16630: +1 from me as well. > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: 16630-v2-suggest.patch, 16630-v3-suggest.patch, > HBASE-16630.patch, HBASE-16630-v2.patch, HBASE-16630-v3.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17473) Set capacity on ArrayList in hbase-client
[ https://issues.apache.org/jira/browse/HBASE-17473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Hentschel updated HBASE-17473: -- Status: Patch Available (was: In Progress) > Set capacity on ArrayList in hbase-client > - > > Key: HBASE-17473 > URL: https://issues.apache.org/jira/browse/HBASE-17473 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Attachments: HBASE-17473.master.001.patch > > > Set the capacity on an ArrayList when possible in the *hbase-client* module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17473) Set capacity on ArrayList in hbase-client
[ https://issues.apache.org/jira/browse/HBASE-17473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Hentschel updated HBASE-17473: -- Attachment: HBASE-17473.master.001.patch > Set capacity on ArrayList in hbase-client > - > > Key: HBASE-17473 > URL: https://issues.apache.org/jira/browse/HBASE-17473 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Attachments: HBASE-17473.master.001.patch > > > Set the capacity on an ArrayList when possible in the *hbase-client* module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824150#comment-15824150 ] Ted Yu commented on HBASE-16630: Ran TestAsyncTableBatch with patch locally and didn't reproduce test failure. TestAsyncTableBatch doesn't use bucket cache so it was not related. > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: 16630-v2-suggest.patch, 16630-v3-suggest.patch, > HBASE-16630.patch, HBASE-16630-v2.patch, HBASE-16630-v3.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17443) Move listReplicated/enableTableRep/disableTableRep methods from ReplicationAdmin to Admin
[
https://issues.apache.org/jira/browse/HBASE-17443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824062#comment-15824062
]
Hadoop QA commented on HBASE-17443:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 81m 50s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 44s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12847616/HBASE-17443-v4.patch |
| JIRA Issue | HBASE-17443 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3726c2fa493b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4cb09a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5277/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5277/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Move listReplicated/enableTableRep/disableTableRep
[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk
[ https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824012#comment-15824012 ] Anastasia Braginsky commented on HBASE-17081: - Created one more patch after another re-base and fixing all review comments. > Flush the entire CompactingMemStore content to disk > --- > > Key: HBASE-17081 > URL: https://issues.apache.org/jira/browse/HBASE-17081 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, > HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, > HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, > HBASE-17081-V07.patch, HBASE-17081-V10.patch, HBASE-17081-V13.patch, > HBASE-17081-V14.patch, HBaseMeetupDecember2016-V02.pptx, > Pipelinememstore_fortrunk_3.patch > > > Part of CompactingMemStore's memory is held by an active segment, and another > part is divided between immutable segments in the compacting pipeline. Upon > flush-to-disk request we want to flush all of it to disk, in contrast to > flushing only tail of the compacting pipeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17081) Flush the entire CompactingMemStore content to disk
[ https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anastasia Braginsky updated HBASE-17081: Attachment: HBASE-17081-V14.patch > Flush the entire CompactingMemStore content to disk > --- > > Key: HBASE-17081 > URL: https://issues.apache.org/jira/browse/HBASE-17081 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, > HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, > HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, > HBASE-17081-V07.patch, HBASE-17081-V10.patch, HBASE-17081-V13.patch, > HBASE-17081-V14.patch, HBaseMeetupDecember2016-V02.pptx, > Pipelinememstore_fortrunk_3.patch > > > Part of CompactingMemStore's memory is held by an active segment, and another > part is divided between immutable segments in the compacting pipeline. Upon > flush-to-disk request we want to flush all of it to disk, in contrast to > flushing only tail of the compacting pipeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17443) Move listReplicated/enableTableRep/disableTableRep methods from ReplicationAdmin to Admin
[ https://issues.apache.org/jira/browse/HBASE-17443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823922#comment-15823922 ] Ashish Singhi commented on HBASE-17443: --- +1 on v4 > Move listReplicated/enableTableRep/disableTableRep methods from > ReplicationAdmin to Admin > - > > Key: HBASE-17443 > URL: https://issues.apache.org/jira/browse/HBASE-17443 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17443-v1.patch, HBASE-17443-v2.patch, > HBASE-17443-v2.patch, HBASE-17443-v3.patch, HBASE-17443-v3.patch, > HBASE-17443-v4.patch > > > We have moved other replication requests to Admin and mark ReplicationAdmin > as Deprecated, so listReplicated/enableTableRep/disableTableRep methods need > move to Admin, too. > Review board: https://reviews.apache.org/r/55534/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17263) Netty based rpc server impl
[
https://issues.apache.org/jira/browse/HBASE-17263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823872#comment-15823872
]
Hadoop QA commented on HBASE-17263:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 38s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 35s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 5s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestReplicasClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12847602/HBASE-17263_v6.patch |
| JIRA Issue | HBASE-17263 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7863064f320a 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4cb09a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5276/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5276/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5276/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5276/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Netty based rpc server impl
> -
>
> Key: HBASE-17263
> URL:
[jira] [Updated] (HBASE-17443) Move listReplicated/enableTableRep/disableTableRep methods from ReplicationAdmin to Admin
[ https://issues.apache.org/jira/browse/HBASE-17443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17443: --- Attachment: HBASE-17443-v4.patch > Move listReplicated/enableTableRep/disableTableRep methods from > ReplicationAdmin to Admin > - > > Key: HBASE-17443 > URL: https://issues.apache.org/jira/browse/HBASE-17443 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17443-v1.patch, HBASE-17443-v2.patch, > HBASE-17443-v2.patch, HBASE-17443-v3.patch, HBASE-17443-v3.patch, > HBASE-17443-v4.patch > > > We have moved other replication requests to Admin and mark ReplicationAdmin > as Deprecated, so listReplicated/enableTableRep/disableTableRep methods need > move to Admin, too. > Review board: https://reviews.apache.org/r/55534/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15705) Add on meta cache.
[ https://issues.apache.org/jira/browse/HBASE-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15705: Assignee: (was: Mikhail Antonov) > Add on meta cache. > -- > > Key: HBASE-15705 > URL: https://issues.apache.org/jira/browse/HBASE-15705 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark > Attachments: native-client-meta-cache-v2.patch > > > We need to cache this stuff, and it needs to be fast. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15705) Add on meta cache.
[ https://issues.apache.org/jira/browse/HBASE-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15705: Attachment: native-client-meta-cache-v2.patch re-attaching latest patch from reviews.fb.net > Add on meta cache. > -- > > Key: HBASE-15705 > URL: https://issues.apache.org/jira/browse/HBASE-15705 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Mikhail Antonov > Attachments: native-client-meta-cache-v2.patch > > > We need to cache this stuff, and it needs to be fast. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17443) Move listReplicated/enableTableRep/disableTableRep methods from ReplicationAdmin to Admin
[ https://issues.apache.org/jira/browse/HBASE-17443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823806#comment-15823806 ] Ashish Singhi commented on HBASE-17443: --- I have left some minor comments on the RB. Overall looks good. > Move listReplicated/enableTableRep/disableTableRep methods from > ReplicationAdmin to Admin > - > > Key: HBASE-17443 > URL: https://issues.apache.org/jira/browse/HBASE-17443 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17443-v1.patch, HBASE-17443-v2.patch, > HBASE-17443-v2.patch, HBASE-17443-v3.patch, HBASE-17443-v3.patch > > > We have moved other replication requests to Admin and mark ReplicationAdmin > as Deprecated, so listReplicated/enableTableRep/disableTableRep methods need > move to Admin, too. > Review board: https://reviews.apache.org/r/55534/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk
[
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823755#comment-15823755
]
Hadoop QA commented on HBASE-17081:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 43s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 1s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12847534/HBASE-17081-V13.patch
|
| JIRA Issue | HBASE-17081 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7fd17a6fc698 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4cb09a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5275/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5275/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5275/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5275/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Flush the
[jira] [Created] (HBASE-17473) Set capacity on ArrayList in hbase-client
Jan Hentschel created HBASE-17473: - Summary: Set capacity on ArrayList in hbase-client Key: HBASE-17473 URL: https://issues.apache.org/jira/browse/HBASE-17473 Project: HBase Issue Type: Sub-task Components: Client Reporter: Jan Hentschel Assignee: Jan Hentschel Priority: Minor Set the capacity on an ArrayList when possible in the *hbase-client* module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HBASE-17473) Set capacity on ArrayList in hbase-client
[ https://issues.apache.org/jira/browse/HBASE-17473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-17473 started by Jan Hentschel. - > Set capacity on ArrayList in hbase-client > - > > Key: HBASE-17473 > URL: https://issues.apache.org/jira/browse/HBASE-17473 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > > Set the capacity on an ArrayList when possible in the *hbase-client* module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17472) Correct the semantic of permission grant
[
https://issues.apache.org/jira/browse/HBASE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823708#comment-15823708
]
huzheng commented on HBASE-17472:
-
[~Apache9][~ashish singhi] [~Weizhan Zeng] thanks for your advise, I'll upload
a patch :)
> Correct the semantic of permission grant
> -
>
> Key: HBASE-17472
> URL: https://issues.apache.org/jira/browse/HBASE-17472
> Project: HBase
> Issue Type: Improvement
> Components: Admin
>Reporter: huzheng
>Assignee: huzheng
>
> Currently, HBase grant operation has following semantic:
> {code}
> hbase(main):019:0> grant 'hbase_tst', 'RW', 'ycsb'
> 0 row(s) in 0.0960 seconds
> hbase(main):020:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
>
> hbase_tst default,ycsb,,:
> [Permission:actions=READ,WRITE]
>
>
> 1 row(s) in 0.0550 seconds
> hbase(main):021:0> grant 'hbase_tst', 'CA', 'ycsb'
> 0 row(s) in 0.0820 seconds
> hbase(main):022:0> user_permission 'ycsb'
> User
> Namespace,Table,Family,Qualifier:Permission
>
>
> hbase_tst default,ycsb,,:
> [Permission: actions=CREATE,ADMIN]
>
>
> 1 row(s) in 0.0490 seconds
> {code}
> Later permission will replace previous granted permissions, which confused
> most of HBase administrator.
> It's seems more reasonable that HBase merge multiple granted permission.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17263) Netty based rpc server impl
[ https://issues.apache.org/jira/browse/HBASE-17263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-17263: - Attachment: HBASE-17263_v6.patch > Netty based rpc server impl > - > > Key: HBASE-17263 > URL: https://issues.apache.org/jira/browse/HBASE-17263 > Project: HBase > Issue Type: Sub-task > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0 > > Attachments: HBASE-17263.patch, HBASE-17263_v2.patch, > HBASE-17263_v3.patch, HBASE-17263_v4.patch, HBASE-17263_v5.patch, > HBASE-17263_v6.patch > > > An RPC server with Netty4 implementation, which provide better performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
