[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1612#comment-1612
]
Guanghao Zhang commented on HBASE-18500:
bq. Could you leave above comment in the right place? Or file a issue.
Let me dig it more.
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the
> almost same latency with the blocking write. But for master branch, the
> result shows the async write has better latency than the blocking client.
> Take a look about the code, I thought the difference is the BufferedMutator.
> For master branch, HTable don't have a write buffer and all write request
> will be flushed directly. And user can use BufferedMutator when user want to
> perform client-side buffering of writes. For the performance issue
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121102#comment-16121102
]
Chia-Ping Tsai commented on HBASE-18500:
{quote}
When grant user permission, it will put a record to ACL. The old implementation
use BufferMutator, so it will send a multi request. And use
AccessController.preBatchMutate to check permission when put a record. When
HTable doesn't use BufferMutator, it will send a mutate request and use
AccessController.prePut to check permission. So I change to use put(List),
this will send multi request, too. Then it will same with before. I thought
AccessController may have some bugs which need to dig more. But it is not
related to this issue?
{quote}
Could you leave above comment in the right place? Or file a issue.
{quote}
The default retries number is 35. So it will wait a long time when there are
exception in ut. I thought we should set this to the default
HBaseTestingUtility config in the future.
{quote}
Got it.
+1
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the
> almost same latency with the blocking write. But for master branch, the
> result shows the async write has better latency than the blocking client.
> Take a look about the code, I thought the difference is the BufferedMutator.
> For master branch, HTable don't have a write buffer and all write request
> will be flushed directly. And user can use BufferedMutator when user want to
> perform client-side buffering of writes. For the performance issue
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121087#comment-16121087
]
Guanghao Zhang edited comment on HBASE-18500 at 8/10/17 5:44 AM:
-
bq. t.put(Collections.singletonList(p));
When grant user permission, it will put a record to ACL. The old implementation
use BufferMutator, so it will send a multi request. And use
AccessController.preBatchMutate to check permission when put a record. When
HTable doesn't use BufferMutator, it will send a mutate request and use
AccessController.prePut to check permission. So I change to use put(List),
this will send multi request, too. Then it will same with before. I thought
AccessController may have some bugs which need to dig more. But it is not
related to this issue?
bq. UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 2);
The default retries number is 35. So it will wait a long time when there are
exception in ut. I thought we should set this to the default
HBaseTestingUtility config in the future.
was (Author: zghaobac):
bq. t.put(Collections.singletonList(p));
When grant user permission, it will put a record to ACL. The old implementation
use BufferMutator, so it will send a multi request. And use
AccessControlLists.preBatchMutate to check permission when put a record. When
HTable doesn't use BufferMutator, it will send a mutate request and use
AccessControlLists.prePut to check permission. So I change to use
put(List), this will send multi request, too. Then it will same with
before. I thought AccessControlLists may have some bugs which need to dig
more. But it is not related to this issue?
bq. UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 2);
The default retries number is 35. So it will wait a long time when there are
exception in ut. I thought we should set this to the default
HBaseTestingUtility config in the future.
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the
> almost same latency with the blocking write. But for master branch, the
> result shows the async write has better latency than the blocking client.
> Take a look about the code, I thought the difference is the BufferedMutator.
> For master branch, HTable don't have a write buffer and all write request
> will be flushed directly. And user can use BufferedMutator when user want to
> perform client-side buffering of writes. For the performance issue
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121093#comment-16121093
]
Hadoop QA commented on HBASE-18546:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 46s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
40s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 44s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.protobuf.TestProtobufUtil |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881134/HBASE-18546.v0.patch |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux dd9eea926f1d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6246523 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/8007/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8007/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https
[jira] [Commented] (HBASE-18551) [AMv2] UnassignProcedure and crashed regionservers
[ https://issues.apache.org/jira/browse/HBASE-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121088#comment-16121088 ] stack commented on HBASE-18551: --- Hmm. #3 seems to work. Let me clean up the patch. > [AMv2] UnassignProcedure and crashed regionservers > -- > > Key: HBASE-18551 > URL: https://issues.apache.org/jira/browse/HBASE-18551 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: stack > > This has been [~uagashe] and my obsession over the last few days, what should > an UnassignProcedure do when it dispatches a CLOSE but the CLOSE fails > because of ConnectException or SocketTimeout. > + We used to let UnassignProcedure continue presuming the Region would be > closed since the server is dead. BUT, if the unassign was part of a > MoveProcedure, the unassign would proceed and the Move would then run WITHOUT > first splitting logs. Bad. > + So, we made it so UnassignProcedure failed; let the upper layers take care > of the failure. See HBASE-18491 that enabled this behavior. BUT, we are since > figuring that even if the UP completes as a failure, since it gives up the > Region lock on completion, another procedure -- say an AssignProcedure -- > could cut in before the ServerCrashProcedure had finished and again there > could be dataloss. > + Now we are thinking the UP should hold on to the Region lock until we are > signalled by a ServerCrashProcedure; only then let go of the region. The UP > has context that is hard to pass another. Waiting on a SCP has the UP living > on for what could be a good amount of time. It might be ok if we can suspend > the procedure. > There is a good sample scenario that came up doing the no-regions-on-master > issue, HBASE-18511. When meta is not on master, TestSplitTransactionOnCluster > is failing. It fails because though the test completes, the tests commonly > kill a RegionServer. The teardown for the test runs before we've noticed the > aborted RS. So, the disable of the table in the teardown prepartory to our > deleting the test table as part of clean up, goes to unassign regions but the > unassign fails against the aborted server. > Good stuff. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121087#comment-16121087
]
Guanghao Zhang commented on HBASE-18500:
bq. t.put(Collections.singletonList(p));
When grant user permission, it will put a record to ACL. The old implementation
use BufferMutator, so it will send a multi request. And use
AccessControlLists.preBatchMutate to check permission when put a record. When
HTable doesn't use BufferMutator, it will send a mutate request and use
AccessControlLists.prePut to check permission. So I change to use
put(List), this will send multi request, too. Then it will same with
before. I thought AccessControlLists may have some bugs which need to dig
more. But it is not related to this issue?
bq. UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 2);
The default retries number is 35. So it will wait a long time when there are
exception in ut. I thought we should set this to the default
HBaseTestingUtility config in the future.
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the
> almost same latency with the blocking write. But for master branch, the
> result shows the async write has better latency than the blocking client.
> Take a look about the code, I thought the difference is the BufferedMutator.
> For master branch, HTable don't have a write buffer and all write request
> will be flushed directly. And user can use BufferedMutator when user want to
> perform client-side buffering of writes. For the performance issue
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18112) Write RequestTooLargeException back to client for NettyRpcServer
[ https://issues.apache.org/jira/browse/HBASE-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121085#comment-16121085 ] Duo Zhang commented on HBASE-18112: --- Just go ahead. > Write RequestTooLargeException back to client for NettyRpcServer > > > Key: HBASE-18112 > URL: https://issues.apache.org/jira/browse/HBASE-18112 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: Duo Zhang > > For now we just close the connection so NettyRpcServer can not pass TestIPC. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) Added support for refreshing HFiles through API and shell
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121081#comment-16121081 ] ramkrishna.s.vasudevan commented on HBASE-18448: 2nd approach is best. Rest what ever [~anoop.hbase] says. I think the End point API can be exposed accepting Table Name and since anyway this is to be explicitly invoked the API can be invoked with different table names. > Added support for refreshing HFiles through API and shell > - > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121080#comment-16121080
]
ramkrishna.s.vasudevan commented on HBASE-18500:
LGTM but for the above comments from [~chia7712]. If he is fine then we can go
for commit. Release note is needed I believe?
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the
> almost same latency with the blocking write. But for master branch, the
> result shows the async write has better latency than the blocking client.
> Take a look about the code, I thought the difference is the BufferedMutator.
> For master branch, HTable don't have a write buffer and all write request
> will be flushed directly. And user can use BufferedMutator when user want to
> perform client-side buffering of writes. For the performance issue
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18112) Write RequestTooLargeException back to client for NettyRpcServer
[ https://issues.apache.org/jira/browse/HBASE-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121078#comment-16121078 ] Toshihiro Suzuki commented on HBASE-18112: -- Can I try this? [~Apache9] > Write RequestTooLargeException back to client for NettyRpcServer > > > Key: HBASE-18112 > URL: https://issues.apache.org/jira/browse/HBASE-18112 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: Duo Zhang > > For now we just close the connection so NettyRpcServer can not pass TestIPC. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18528) DON'T allow user to modify the passed table/column descriptor
[ https://issues.apache.org/jira/browse/HBASE-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121075#comment-16121075 ] Anoop Sam John commented on HBASE-18528: bq.The impl of desc passed to MasterCoprocessorHost should be ModifyableTableDescriptor. The ImmutableHTableDescriptor, which is deprecated , will be removed in 3.0 so we should prevent to use it in code base. Which method in MasterCoprocessorHost need/takes ModifyableTableDescriptor? We should be passing TableDescriptor interface type into the MasterCoprocessorHost and MasterObserver right? Sorry I may be missing some thing. Pls correct if so > DON'T allow user to modify the passed table/column descriptor > - > > Key: HBASE-18528 > URL: https://issues.apache.org/jira/browse/HBASE-18528 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, master >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18528.v0.patch > > > We are replacing the HTableDescriptor by TableDescriptor from code base. The > TableDescriptor is designed to be a read-only object so user can't modifiy it > through MasterObserver. HBASE-18502 change many methods of MasterObserver to > use TableDescriptor but some deprecated methods still accept the > HTableDescriptor. User may be confused by why some methods can't modify the > table descriptor. > In short, Should we allow user to modify the passed table descriptor? > # if yes, we should introduce a mechanism that user can return a modified > table descripror > # if no, we should pass ImmutableHTableDescriptor to user. Or we just remove > all methods accepting the HTableDescriptor > Ditto for HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18398) Snapshot operation fails with FileNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-18398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121067#comment-16121067 ] Anoop Sam John commented on HBASE-18398: +1 > Snapshot operation fails with FileNotFoundException > --- > > Key: HBASE-18398 > URL: https://issues.apache.org/jira/browse/HBASE-18398 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.3.2 > > Attachments: HBASE-18398.branch-1.3.001.patch, > HBASE-18398.branch-1.3.002.patch, HBASE-18398.master.001.patch, > HBASE-18398.master.002.patch, HBASE-18398.master.003.patch, > HBASE-18398.master.004.patch > > > Failing to take snapshot due to FileNotFoundException > * FlushSnapshotSubprocedure.RegionSnapshotTask takes a region level read > lock > * Call to HRegion#addRegionToSnapshot. > * Call to SnapshotManifest#addRegion. This gets the current list of store > files. > * RACE → File is marked as compacted away and HFileArchiver moves the > file to archive under store level lock. > * SnapshotManifest#addRegion visits the stale list of store files one by > one. It does a file.getStatus() call to get length of each file. Since the > file object still points to the original file, file.getStatus() fails with > FileNotFoundException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) Added support for refreshing HFiles through API and shell
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121061#comment-16121061 ] Anoop Sam John commented on HBASE-18448: Ya the 2nd approach. I just want to correct one of ur understanding. One EP execution from client side can NOT get executed in all region irrespective of the table. The CPEP call itself has to happen on a table. You can specify a row range though. So what happens is the client side will find all the regions of this table coming under the specified row range and do RPC to all such regions. As u said the execution across regions will happen in parallel way. You can pass the start and end keys as empty byte[] (HConstants.EMPTY_START_ROW) when we need the execution to happen across ALL the regions of the table. So there is no need to check on a region whether this belongs to specified table or not. In fact passing the table name to server is not needed. When the system do execute the CPEP for a region it is sure that this region belongs to the table that u have given. Hope u are fully clear now :-) Ya if u can expose the CPEP client side API for ur user, there is no need for HBase side changes. On the Public Table interface better we do NOT expose such very special APIs. In ur CPEP impl (server side), u will get handle to current Region and from that u can get Stores and on Store interface u can call the refresh method. > Added support for refreshing HFiles through API and shell > - > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18533) Expose BucketCache values to be configured
[ https://issues.apache.org/jira/browse/HBASE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-18533: --- Fix Version/s: 2.0.0-alpha-2 1.5.0 1.4.0 3.0.0 > Expose BucketCache values to be configured > -- > > Key: HBASE-18533 > URL: https://issues.apache.org/jira/browse/HBASE-18533 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Zach York >Assignee: Zach York > Fix For: 3.0.0, 1.4.0, 1.5.0, 2.0.0-alpha-2 > > Attachments: HBASE-18533.branch-1.001.patch, > HBASE-18533.branch-1.002.patch, HBASE-18533.master.001.patch, > HBASE-18533.master.002.patch > > > BucketCache always uses the default values for all cache configuration. > However, this doesn't work for all use cases. In particular, users want to be > able to configure the percentage of the cache that is single access, multi > access, and in-memory access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18533) Expose BucketCache values to be configured
[ https://issues.apache.org/jira/browse/HBASE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121056#comment-16121056 ] Anoop Sam John edited comment on HBASE-18533 at 8/10/17 4:57 AM: - +1 for branches 1.4 + was (Author: anoop.hbase): +1 for branches 1.4+ > Expose BucketCache values to be configured > -- > > Key: HBASE-18533 > URL: https://issues.apache.org/jira/browse/HBASE-18533 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18533.branch-1.001.patch, > HBASE-18533.branch-1.002.patch, HBASE-18533.master.001.patch, > HBASE-18533.master.002.patch > > > BucketCache always uses the default values for all cache configuration. > However, this doesn't work for all use cases. In particular, users want to be > able to configure the percentage of the cache that is single access, multi > access, and in-memory access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18533) Expose BucketCache values to be configured
[ https://issues.apache.org/jira/browse/HBASE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121057#comment-16121057 ] Anoop Sam John commented on HBASE-18533: bq.How can I trigger the QA again? Pls attach the latest patch once more. That will trigger the QA > Expose BucketCache values to be configured > -- > > Key: HBASE-18533 > URL: https://issues.apache.org/jira/browse/HBASE-18533 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18533.branch-1.001.patch, > HBASE-18533.branch-1.002.patch, HBASE-18533.master.001.patch, > HBASE-18533.master.002.patch > > > BucketCache always uses the default values for all cache configuration. > However, this doesn't work for all use cases. In particular, users want to be > able to configure the percentage of the cache that is single access, multi > access, and in-memory access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18533) Expose BucketCache values to be configured
[ https://issues.apache.org/jira/browse/HBASE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121056#comment-16121056 ] Anoop Sam John commented on HBASE-18533: +1 for branches 1.4+ > Expose BucketCache values to be configured > -- > > Key: HBASE-18533 > URL: https://issues.apache.org/jira/browse/HBASE-18533 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18533.branch-1.001.patch, > HBASE-18533.branch-1.002.patch, HBASE-18533.master.001.patch, > HBASE-18533.master.002.patch > > > BucketCache always uses the default values for all cache configuration. > However, this doesn't work for all use cases. In particular, users want to be > able to configure the percentage of the cache that is single access, multi > access, and in-memory access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18398) Snapshot operation fails with FileNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-18398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121045#comment-16121045 ] ramkrishna.s.vasudevan commented on HBASE-18398: +1 for commit. > Snapshot operation fails with FileNotFoundException > --- > > Key: HBASE-18398 > URL: https://issues.apache.org/jira/browse/HBASE-18398 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.3.2 > > Attachments: HBASE-18398.branch-1.3.001.patch, > HBASE-18398.branch-1.3.002.patch, HBASE-18398.master.001.patch, > HBASE-18398.master.002.patch, HBASE-18398.master.003.patch, > HBASE-18398.master.004.patch > > > Failing to take snapshot due to FileNotFoundException > * FlushSnapshotSubprocedure.RegionSnapshotTask takes a region level read > lock > * Call to HRegion#addRegionToSnapshot. > * Call to SnapshotManifest#addRegion. This gets the current list of store > files. > * RACE → File is marked as compacted away and HFileArchiver moves the > file to archive under store level lock. > * SnapshotManifest#addRegion visits the stale list of store files one by > one. It does a file.getStatus() call to get length of each file. Since the > file object still points to the original file, file.getStatus() fails with > FileNotFoundException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18248) Warn if monitored RPC task has been tied up beyond a configurable threshold
[ https://issues.apache.org/jira/browse/HBASE-18248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121043#comment-16121043 ] Hudson commented on HBASE-18248: SUCCESS: Integrated in Jenkins build HBase-2.0 #301 (See [https://builds.apache.org/job/HBase-2.0/301/]) HBASE-18248 Warn if monitored RPC task has been tied up beyond a (apurtell: rev a90217555346bb0768bfb410bb0d1fa02d6d562d) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTaskImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTask.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java > Warn if monitored RPC task has been tied up beyond a configurable threshold > --- > > Key: HBASE-18248 > URL: https://issues.apache.org/jira/browse/HBASE-18248 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18248-branch-1.patch, HBASE-18248-branch-1.patch, > HBASE-18248.patch, HBASE-18248.patch > > > Warn if monitored task has been tied up beyond a configurable threshold. We > especially want to do this for RPC tasks. Use a separate threshold for > warning about stuck RPC tasks versus other types of tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18550) Move sanity checks for BucketCache configs inside BucketCache
[ https://issues.apache.org/jira/browse/HBASE-18550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-18550: --- Priority: Minor (was: Major) > Move sanity checks for BucketCache configs inside BucketCache > - > > Key: HBASE-18550 > URL: https://issues.apache.org/jira/browse/HBASE-18550 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Zach York >Assignee: Zach York >Priority: Minor > > In https://reviews.apache.org/r/61485/, Ramkrishna asked if I would move the > sanity checking of all configs into the BucketCache constructor. This is a > follow-up JIRA to move the sanity checking into the constructor. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18548) Move sources of important Jenkins jobs into source control
[ https://issues.apache.org/jira/browse/HBASE-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121042#comment-16121042 ] stack commented on HBASE-18548: --- +1 for the beautiful patch. Another +1 for updating the doc to match. > Move sources of important Jenkins jobs into source control > -- > > Key: HBASE-18548 > URL: https://issues.apache.org/jira/browse/HBASE-18548 > Project: HBase > Issue Type: Bug > Components: documentation, scripts >Affects Versions: 2.0.0-alpha-1 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18548.patch > > > Move sources of the following Jenkins jobs into source control: > https://builds.apache.org/job/hbase_generate_website/ > https://builds.apache.org/view/All/job/HBase%20Website%20Link%20Checker/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18248) Warn if monitored RPC task has been tied up beyond a configurable threshold
[ https://issues.apache.org/jira/browse/HBASE-18248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121037#comment-16121037 ] Hudson commented on HBASE-18248: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3504 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3504/]) HBASE-18248 Warn if monitored RPC task has been tied up beyond a (apurtell: rev d0941127d424bc76688aec7952388f858d917b14) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTask.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTaskImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java > Warn if monitored RPC task has been tied up beyond a configurable threshold > --- > > Key: HBASE-18248 > URL: https://issues.apache.org/jira/browse/HBASE-18248 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18248-branch-1.patch, HBASE-18248-branch-1.patch, > HBASE-18248.patch, HBASE-18248.patch > > > Warn if monitored task has been tied up beyond a configurable threshold. We > especially want to do this for RPC tasks. Use a separate threshold for > warning about stuck RPC tasks versus other types of tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121036#comment-16121036
]
Sean Busbey commented on HBASE-16179:
-
bq. I am back in US. There should be no ambiguity who would be doing the work
for this JIRA.
Good to hear.
{quote}
Looks like we are going to support Scala 2.10 and 2.11 with Spark 1.6 and 2.x
combinations.
How many artifacts would be produced by the build process ?
{quote}
{quote}
we support Scala 2.10 and 2.11 with and spark 2.0 to spark 2.2 combinations, no
more spark1.6 needed
{quote}
We don't need to work with Spark 1.6, agreed. Agreed we need to run on Spark
2.z. currently the [spark dist
area|https://dist.apache.org/repos/dist/release/spark/] has a 2.0, 2.1, and 2.2
version. (presuming we get HBase 2.0 out the door before there's a Spark 2.3)
We definitely need a build process that makes appropriate artifacts for runtime
of the set of scala x spark versions. I don't know if we need a different
compilation for the different spark minor versions. Presuming worse case
(needing a different artifact for each minor spark version), that'd be 6
artifacts.
bq. Maybe for this jira, we do not need to consider artifacts, and open an
other jira to talk about the artifacts? I am not so sure about this part.
I think Ted's already on track to generate multiple artifacts, presuming we
need them. That said, if he'd rather not deal with the Spark 2.z minor version
runtime compatibility, I'm fine with just making the current module work with
Spark 2 for this jira so long as we have some jira that tracks getting
everything done for the set of scala x spark versions.
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Labels: build
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt,
> 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt,
> 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt,
> 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt,
> 16179.v25.txt, 16179.v26.txt, 16179.v27.txt, 16179.v4.txt, 16179.v5.txt,
> 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-18546:
---
Attachment: HBASE-18546.v0.patch
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: New Feature
> Components: API, Client
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Labels: incompatibleChange
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18546.v0.patch
>
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-18546:
---
Status: Patch Available (was: Open)
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: New Feature
> Components: API, Client
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Labels: incompatibleChange
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18546.v0.patch
>
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18522) Add RowMutations support to Batch
[ https://issues.apache.org/jira/browse/HBASE-18522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121028#comment-16121028 ] Ted Yu commented on HBASE-18522: lgtm > Add RowMutations support to Batch > - > > Key: HBASE-18522 > URL: https://issues.apache.org/jira/browse/HBASE-18522 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.6 >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18522-branch-1.patch, > HBASE-18522-branch-1-v2.patch, HBASE-18522-master-v2.patch > > > RowMutations is multiple Puts and/or Deletes atomically on a single row. > Current Batch call does not support RowMutations as part of the batch. > We should add this missing part. We should be able to batch RowMutations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18020) Update API Compliance Checker to Incorporate Improvements Done in Hadoop
[ https://issues.apache.org/jira/browse/HBASE-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121025#comment-16121025 ] Sean Busbey commented on HBASE-18020: - ping [~awleblang] > Update API Compliance Checker to Incorporate Improvements Done in Hadoop > > > Key: HBASE-18020 > URL: https://issues.apache.org/jira/browse/HBASE-18020 > Project: HBase > Issue Type: Improvement > Components: API, community >Reporter: Alex Leblang >Assignee: Alex Leblang > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7, 1.1.12 > > Attachments: HBASE-18020.0.patch, HBASE-18020.branch-1.2.001.patch, > HBASE-18020.branch-1.2.002.patch, HBASE-18020.branch-1.2.003.patch, > HBASE-18020.branch-1.2.004.patch > > > Recently the Hadoop community has made a number of improvements in their api > compliance checker based on feedback from the hbase and kudu community. We > should adopt these changes ourselves. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18522) Add RowMutations support to Batch
[
https://issues.apache.org/jira/browse/HBASE-18522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121017#comment-16121017
]
Hadoop QA commented on HBASE-18522:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 34s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
41s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}108m
0s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18522 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881116/HBASE-18522-master-v2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 642fb7f0cd9e 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 794a3b1 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8006/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8006/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Add RowMutations supp
[jira] [Commented] (HBASE-18248) Warn if monitored RPC task has been tied up beyond a configurable threshold
[ https://issues.apache.org/jira/browse/HBASE-18248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121000#comment-16121000 ] Hudson commented on HBASE-18248: FAILURE: Integrated in Jenkins build HBase-1.4 #843 (See [https://builds.apache.org/job/HBase-1.4/843/]) HBASE-18248 Warn if monitored RPC task has been tied up beyond a (apurtell: rev 685ab19064e3a83fcb9a9af2f6ebbf7e2f85d6ab) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTask.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTaskImpl.java > Warn if monitored RPC task has been tied up beyond a configurable threshold > --- > > Key: HBASE-18248 > URL: https://issues.apache.org/jira/browse/HBASE-18248 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18248-branch-1.patch, HBASE-18248-branch-1.patch, > HBASE-18248.patch, HBASE-18248.patch > > > Warn if monitored task has been tied up beyond a configurable threshold. We > especially want to do this for RPC tasks. Use a separate threshold for > warning about stuck RPC tasks versus other types of tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120975#comment-16120975
]
Chia-Ping Tsai commented on HBASE-18500:
{code}
try {
- t.put(p);
+ t.put(Collections.singletonList(p));
} finally {
t.close();
}
{code}
{code}
+UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 2);
{code}
Why do we need these changes?
> Performance issue: Don't use BufferedMutator for HTable's put method
>
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch,
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch,
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the
> almost same latency with the blocking write. But for master branch, the
> result shows the async write has better latency than the blocking client.
> Take a look about the code, I thought the difference is the BufferedMutator.
> For master branch, HTable don't have a write buffer and all write request
> will be flushed directly. And user can use BufferedMutator when user want to
> perform client-side buffering of writes. For the performance issue
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-18546:
---
Labels: incompatibleChange (was: )
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: New Feature
> Components: API, Client
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Labels: incompatibleChange
> Fix For: 3.0.0, 2.0.0-alpha-2
>
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-18546:
---
Component/s: Client
API
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: New Feature
> Components: API, Client
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Labels: incompatibleChange
> Fix For: 3.0.0, 2.0.0-alpha-2
>
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120967#comment-16120967
]
Chia-Ping Tsai commented on HBASE-18546:
bq. IIRC the server side implementation will not take care about the timestamp.
The custom timestamp will be adopted if server doesn't find the related cell.
As i see it, it is more like a bug.
{code}
CellUtil.updateLatestStamp(delta, now); // the custom timestamp won't
be rewrited
newCell = delta;
tags = TagUtil.carryForwardTTLTag(tags, mutation.getTTL());
if (tags != null) {
newCell = CellUtil.createCell(delta, tags);
}
{code}
bq. I think this should be a new feature?
We allow user to use custom timestamp for PUT/DELETE/APPEND. Ya, this should be
an new feature even if there is no use case at present.
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: Bug
> Components: API, Client
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Labels: incompatibleChange
> Fix For: 3.0.0, 2.0.0-alpha-2
>
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-18546:
---
Issue Type: New Feature (was: Bug)
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: New Feature
> Components: API, Client
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Labels: incompatibleChange
> Fix For: 3.0.0, 2.0.0-alpha-2
>
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18398) Snapshot operation fails with FileNotFoundException
[
https://issues.apache.org/jira/browse/HBASE-18398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120963#comment-16120963
]
Hadoop QA commented on HBASE-18398:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 14s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}114m
39s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18398 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/1288/HBASE-18398.master.004.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4126635332ac 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 794a3b1 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8005/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8005/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Snapshot operation fails with FileNotFoundException
> ---
>
> Key: HBASE-18398
> URL: https://issues.apache.org/jira/browse/HBASE-18398
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 1.3.2
>
> Attachments: HBASE-18398.branch-1
[jira] [Created] (HBASE-18553) Expose scan cursor for asynchronous scanner
Duo Zhang created HBASE-18553: - Summary: Expose scan cursor for asynchronous scanner Key: HBASE-18553 URL: https://issues.apache.org/jira/browse/HBASE-18553 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0-alpha-1, 3.0.0 Reporter: Duo Zhang Fix For: 3.0.0, 2.0.0-alpha-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18552) Backport the server side change in HBASE-18489 to branch-1
Duo Zhang created HBASE-18552: - Summary: Backport the server side change in HBASE-18489 to branch-1 Key: HBASE-18552 URL: https://issues.apache.org/jira/browse/HBASE-18552 Project: HBase Issue Type: Sub-task Affects Versions: 1.4.0, 1.5.0 Reporter: Duo Zhang Fix For: 1.4.0, 1.5.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18489) Expose scan cursor in RawScanResultConsumer
[ https://issues.apache.org/jira/browse/HBASE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18489: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master and branch-2. Thanks [~tedyu] for reviewing. > Expose scan cursor in RawScanResultConsumer > --- > > Key: HBASE-18489 > URL: https://issues.apache.org/jira/browse/HBASE-18489 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, scan >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18489.patch, HBASE-18489-v1.patch, > HBASE-18489-v2.patch, HBASE-18489-v2.patch > > > The first step of supporting scan cursor for async client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18489) Expose scan cursor in RawScanResultConsumer
[ https://issues.apache.org/jira/browse/HBASE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18489: -- Affects Version/s: 3.0.0 Fix Version/s: 3.0.0 Release Note: Add a 'cursor' method which returns an 'Optional' in 'RawScanResultConsumer.ScanController'. You can use this method to obtain the scan cursor if available. > Expose scan cursor in RawScanResultConsumer > --- > > Key: HBASE-18489 > URL: https://issues.apache.org/jira/browse/HBASE-18489 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client, scan >Affects Versions: 3.0.0, 2.0.0-alpha-1 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18489.patch, HBASE-18489-v1.patch, > HBASE-18489-v2.patch, HBASE-18489-v2.patch > > > The first step of supporting scan cursor for async client. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18248) Warn if monitored RPC task has been tied up beyond a configurable threshold
[ https://issues.apache.org/jira/browse/HBASE-18248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18248: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.5.0 Status: Resolved (was: Patch Available) Pushed to branch-1.4, branch-1, branch-2, and master > Warn if monitored RPC task has been tied up beyond a configurable threshold > --- > > Key: HBASE-18248 > URL: https://issues.apache.org/jira/browse/HBASE-18248 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18248-branch-1.patch, HBASE-18248-branch-1.patch, > HBASE-18248.patch, HBASE-18248.patch > > > Warn if monitored task has been tied up beyond a configurable threshold. We > especially want to do this for RPC tasks. Use a separate threshold for > warning about stuck RPC tasks versus other types of tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18522) Add RowMutations support to Batch
[ https://issues.apache.org/jira/browse/HBASE-18522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-18522: - Attachment: HBASE-18522-master-v2.patch > Add RowMutations support to Batch > - > > Key: HBASE-18522 > URL: https://issues.apache.org/jira/browse/HBASE-18522 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.6 >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18522-branch-1.patch, > HBASE-18522-branch-1-v2.patch, HBASE-18522-master-v2.patch > > > RowMutations is multiple Puts and/or Deletes atomically on a single row. > Current Batch call does not support RowMutations as part of the batch. > We should add this missing part. We should be able to batch RowMutations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18522) Add RowMutations support to Batch
[ https://issues.apache.org/jira/browse/HBASE-18522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-18522: - Attachment: (was: HBASE-18522-master-v2.patch) > Add RowMutations support to Batch > - > > Key: HBASE-18522 > URL: https://issues.apache.org/jira/browse/HBASE-18522 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.6 >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0 > > Attachments: HBASE-18522-branch-1.patch, > HBASE-18522-branch-1-v2.patch, HBASE-18522-master-v2.patch > > > RowMutations is multiple Puts and/or Deletes atomically on a single row. > Current Batch call does not support RowMutations as part of the batch. > We should add this missing part. We should be able to batch RowMutations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18548) Move sources of important Jenkins jobs into source control
[
https://issues.apache.org/jira/browse/HBASE-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120899#comment-16120899
]
Hadoop QA commented on HBASE-18548:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
4s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
5s{color} | {color:red} The patch generated 2 new + 498 unchanged - 0 fixed =
500 total (was 498) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 2s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}143m
14s{color} | {color:green} root in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}199m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18548 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881091/HBASE-18548.patch |
| Optional Tests | asflicense shellcheck shelldocs javac javadoc unit
xml compile |
| uname | Linux c8b9b76bb641 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 794a3b1 |
| Default Java | 1.8.0_131 |
| shellcheck | v0.4.6 |
| shellcheck |
https://builds.apache.org/job/PreCommit-HBASE-Build/8004/artifact/patchprocess/diff-patch-shellcheck.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8004/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HBASE-Build/8004/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8004/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> Move sources of important Jenkins jobs into source control
> --
>
> Key: HBASE-18548
> URL: https://issues.apache.org/jira/browse/HBASE-18548
> Project: HBase
> Issue Type: Bug
> Components: documentation, scripts
>Affects Versions: 2.0.0-alpha-1
[jira] [Commented] (HBASE-18545) Fix broke site build that fails in rsgroups
[
https://issues.apache.org/jira/browse/HBASE-18545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120889#comment-16120889
]
Hudson commented on HBASE-18545:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3503 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3503/])
HBASE-18545 Fix broke site build that fails in rsgroups (stack: rev
794a3b104bf62d028fc8ae27596c3b98ef87e507)
* (edit) hbase-rsgroup/pom.xml
> Fix broke site build that fails in rsgroups
> ---
>
> Key: HBASE-18545
> URL: https://issues.apache.org/jira/browse/HBASE-18545
> Project: HBase
> Issue Type: Bug
> Components: site
>Reporter: stack
>Assignee: stack
> Fix For: 3.0.0
>
>
> Site is broke because of a failure finding .proto files (was pointed out on
> list and in closed proto issue...). Failure looks like this:
> {code}
> [INFO] --- protobuf-maven-plugin:0.5.0:compile (compile-protoc) @
> hbase-rsgroup ---
> [INFO] Compiling 2 proto file(s) to
> /home/jenkins/jenkins-slave/workspace/hbase_generate_website/hbase/hbase-rsgroup/target/generated-sources/protobuf/java
> [ERROR] PROTOC FAILED: HBase.proto: File not found.
> RSGroup.proto: Import "HBase.proto" was not found or had errors.
> RSGroup.proto:31:12: "ServerName" is not defined.
> RSGroup.proto:32:12: "TableName" is not defined.
> [ERROR]
> /home/jenkins/jenkins-slave/workspace/hbase_generate_website/hbase/hbase-rsgroup/src/main/protobuf/RSGroup.proto
> [0:0]: HBase.proto: File not found.
> RSGroup.proto: Import "HBase.proto" was not found or had errors.
> RSGroup.proto:31:12: "ServerName" is not defined.
> RSGroup.proto:32:12: "TableName" is not defined.
> [ERROR]
> /home/jenkins/jenkins-slave/workspace/hbase_generate_website/hbase/hbase-rsgroup/src/main/protobuf/RSGroupAdmin.proto
> [0:0]: HBase.proto: File not found.
> RSGroup.proto: Import "HBase.proto" was not found or had errors.
> RSGroup.proto:31:12: "ServerName" is not defined.
> RSGroup.proto:32:12: "TableName" is not defined.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120880#comment-16120880
]
Duo Zhang commented on HBASE-18546:
---
I think this should be a new feature? IIRC the server side implementation will
not take care about the timestamp.
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18353) Enable TestCorruptedRegionStoreFile that were disabled by Proc-V2 AM in HBASE-14614
[
https://issues.apache.org/jira/browse/HBASE-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120878#comment-16120878
]
Duo Zhang commented on HBASE-18353:
---
The 'move' method need an explicit destination RS which you want it to hold the
region, but unassign does not, it will select a RS randomly. This is exactly
what I want in HBASE-17712.
And again, what is the scenario you want to leave a region in a CLOSED state?
You can have the state internally, but I do not think we should expose this to
user. It is confusing.
{quote}
Also note that we only can deprecate Admin method for 2.0, we cannot remove it.
{quote}
Fair enough. Then you can not change the behavior of 'unassign' method as well.
Thanks.
> Enable TestCorruptedRegionStoreFile that were disabled by Proc-V2 AM in
> HBASE-14614
> ---
>
> Key: HBASE-18353
> URL: https://issues.apache.org/jira/browse/HBASE-18353
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0-alpha-1
>Reporter: Stephen Yuan Jiang
>Assignee: Vladimir Rodionov
> Attachments: HBASE-18353-v1.patch, HBASE-18353-v2.patch
>
>
> HBASE-14614 disabled TestCorruptedRegionStoreFile, as it depends on a
> half-implemented reopen of a region when a store file goes missing.
> This JIRA tracks the work to fix/enable the test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18448) Added support for refreshing HFiles through API and shell
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120871#comment-16120871 ] Ajay Jadhav commented on HBASE-18448: - [~ram_krish] and [~anoop.hbase]: Sorry I got sidetracked last week. I spent some time today looking through the coprocessor EP and doing some background reading. The advantage of using CP EP I see is that it can process the request across regions parallelly which looks promising. Currently, the refreshHFiles API is granular at the table level. It figures out all the regions for the table and issues RPC call to RS. This will update each region sequentially. Now with CP EP, there are 2 ways to go about implementing: 1. Refresh the HFile in memory handle list for all regions irrespective of the table. This way users will be able to keep the replica consistent with all updates in primary without the need of individually issuing refresh for each table but this adds a huge performance overhead in case of slower filesystems like S3. 2. In case we limit it to the table, then the EP will accept the table as input and if the current region belongs to the input table, then do the refreshHFiles else skip. I'm more inclined towards the second approach, let me know your opinion. > Added support for refreshing HFiles through API and shell > - > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120870#comment-16120870 ] Yi Liang commented on HBASE-16179: -- we support Scala 2.10 and 2.11 with and spark 2.0 to spark 2.2 combinations, no more spark1.6 needed And for the artifacts, if I remember correctly in my last patch, the source code is same for different scala(2.10, 2.11) and spark (2.0 to 2.2) versions, same source codes are compatible to all the current versions we support. Maybe for this jira, we do not need to consider artifacts, and open an other jira to talk about the artifacts? I am not so sure about this part. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Labels: build > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, > 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, > 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, > 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt, > 16179.v25.txt, 16179.v26.txt, 16179.v27.txt, 16179.v4.txt, 16179.v5.txt, > 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18534) [C++] Support timeout in Rpc
[
https://issues.apache.org/jira/browse/HBASE-18534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120856#comment-16120856
]
Enis Soztutar commented on HBASE-18534:
---
Thanks Xiaobing for the patch.
- This is 10 seconds, please update the comment.
{code}
+/* sleeping one hour */
+std::this_thread::sleep_for(std::chrono::seconds(10));
{code}
- Let's save this in a class field rather than reading it for every RPC.
{code}
+ int read_rpc_timeout_ms =
conf_->GetInt(ConnectionConfiguration::kRpcReadTimeout,
+
ConnectionConfiguration::kDefaultRpcTimeout);
{code}
The patch looks like a good start. However, you need a TimeoutManager, or an
actual EventBase thread to execute the timeouts, no?
> [C++] Support timeout in Rpc
>
>
> Key: HBASE-18534
> URL: https://issues.apache.org/jira/browse/HBASE-18534
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-18534.000.patch, HBASE-18534.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18398) Snapshot operation fails with FileNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-18398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-18398: -- Attachment: HBASE-18398.master.004.patch HBASE-18398.master.004.patch: Addressing [~anoop.hbase] 's comments > Snapshot operation fails with FileNotFoundException > --- > > Key: HBASE-18398 > URL: https://issues.apache.org/jira/browse/HBASE-18398 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.3.2 > > Attachments: HBASE-18398.branch-1.3.001.patch, > HBASE-18398.branch-1.3.002.patch, HBASE-18398.master.001.patch, > HBASE-18398.master.002.patch, HBASE-18398.master.003.patch, > HBASE-18398.master.004.patch > > > Failing to take snapshot due to FileNotFoundException > * FlushSnapshotSubprocedure.RegionSnapshotTask takes a region level read > lock > * Call to HRegion#addRegionToSnapshot. > * Call to SnapshotManifest#addRegion. This gets the current list of store > files. > * RACE → File is marked as compacted away and HFileArchiver moves the > file to archive under store level lock. > * SnapshotManifest#addRegion visits the stale list of store files one by > one. It does a file.getStatus() call to get length of each file. Since the > file object still points to the original file, file.getStatus() fails with > FileNotFoundException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18551) [AMv2] UnassignProcedure and crashed regionservers
[ https://issues.apache.org/jira/browse/HBASE-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120844#comment-16120844 ] Umesh Agashe commented on HBASE-18551: -- Thanks [~stack], nice description about the set of problems we are working on recently. > [AMv2] UnassignProcedure and crashed regionservers > -- > > Key: HBASE-18551 > URL: https://issues.apache.org/jira/browse/HBASE-18551 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: stack > > This has been [~uagashe] and my obsession over the last few days, what should > an UnassignProcedure do when it dispatches a CLOSE but the CLOSE fails > because of ConnectException or SocketTimeout. > + We used to let UnassignProcedure continue presuming the Region would be > closed since the server is dead. BUT, if the unassign was part of a > MoveProcedure, the unassign would proceed and the Move would then run WITHOUT > first splitting logs. Bad. > + So, we made it so UnassignProcedure failed; let the upper layers take care > of the failure. See HBASE-18491 that enabled this behavior. BUT, we are since > figuring that even if the UP completes as a failure, since it gives up the > Region lock on completion, another procedure -- say an AssignProcedure -- > could cut in before the ServerCrashProcedure had finished and again there > could be dataloss. > + Now we are thinking the UP should hold on to the Region lock until we are > signalled by a ServerCrashProcedure; only then let go of the region. The UP > has context that is hard to pass another. Waiting on a SCP has the UP living > on for what could be a good amount of time. It might be ok if we can suspend > the procedure. > There is a good sample scenario that came up doing the no-regions-on-master > issue, HBASE-18511. When meta is not on master, TestSplitTransactionOnCluster > is failing. It fails because though the test completes, the tests commonly > kill a RegionServer. The teardown for the test runs before we've noticed the > aborted RS. So, the disable of the table in the teardown prepartory to our > deleting the test table as part of clean up, goes to unassign regions but the > unassign fails against the aborted server. > Good stuff. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18520) Add jmx value to determine true Master Start time
[ https://issues.apache.org/jira/browse/HBASE-18520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-18520: -- Release Note: This JIRA adds a JMX value to track when the Master has finished initializing. The jmx config is 'masterFinishedInitializationTime' and details the time in millis that the Master is fully usable and ready to serve requests. > Add jmx value to determine true Master Start time > - > > Key: HBASE-18520 > URL: https://issues.apache.org/jira/browse/HBASE-18520 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Zach York >Assignee: Zach York >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-2 > > Attachments: HBASE-18520.branch-1.001.patch, > HBASE-18520.branch-1.002.patch, HBASE-18520.master.001.patch, > HBASE-18520.master.002.patch, HBASE-18520.master.003.patch > > > The masterActiveTime is being set before regions are assigned. This patch > adds a new jmx metric to expose the final time when the master has become the > active master (All regions are assigned, etc.). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18520) Add jmx value to determine true Master Start time
[ https://issues.apache.org/jira/browse/HBASE-18520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120846#comment-16120846 ] Zach York commented on HBASE-18520: --- Added. > Add jmx value to determine true Master Start time > - > > Key: HBASE-18520 > URL: https://issues.apache.org/jira/browse/HBASE-18520 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Zach York >Assignee: Zach York >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-2 > > Attachments: HBASE-18520.branch-1.001.patch, > HBASE-18520.branch-1.002.patch, HBASE-18520.master.001.patch, > HBASE-18520.master.002.patch, HBASE-18520.master.003.patch > > > The masterActiveTime is being set before regions are assigned. This patch > adds a new jmx metric to expose the final time when the master has become the > active master (All regions are assigned, etc.). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18551) [AMv2] UnassignProcedure and crashed regionservers
stack created HBASE-18551: - Summary: [AMv2] UnassignProcedure and crashed regionservers Key: HBASE-18551 URL: https://issues.apache.org/jira/browse/HBASE-18551 Project: HBase Issue Type: Bug Components: amv2 Reporter: stack This has been [~uagashe] and my obsession over the last few days, what should an UnassignProcedure do when it dispatches a CLOSE but the CLOSE fails because of ConnectException or SocketTimeout. + We used to let UnassignProcedure continue presuming the Region would be closed since the server is dead. BUT, if the unassign was part of a MoveProcedure, the unassign would proceed and the Move would then run WITHOUT first splitting logs. Bad. + So, we made it so UnassignProcedure failed; let the upper layers take care of the failure. See HBASE-18491 that enabled this behavior. BUT, we are since figuring that even if the UP completes as a failure, since it gives up the Region lock on completion, another procedure -- say an AssignProcedure -- could cut in before the ServerCrashProcedure had finished and again there could be dataloss. + Now we are thinking the UP should hold on to the Region lock until we are signalled by a ServerCrashProcedure; only then let go of the region. The UP has context that is hard to pass another. Waiting on a SCP has the UP living on for what could be a good amount of time. It might be ok if we can suspend the procedure. There is a good sample scenario that came up doing the no-regions-on-master issue, HBASE-18511. When meta is not on master, TestSplitTransactionOnCluster is failing. It fails because though the test completes, the tests commonly kill a RegionServer. The teardown for the test runs before we've noticed the aborted RS. So, the disable of the table in the teardown prepartory to our deleting the test table as part of clean up, goes to unassign regions but the unassign fails against the aborted server. Good stuff. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18534) [C++] Support timeout in Rpc
[ https://issues.apache.org/jira/browse/HBASE-18534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18534: -- Attachment: HBASE-18534.001.patch v1: # did some refactoring. # fix some compile issues. > [C++] Support timeout in Rpc > > > Key: HBASE-18534 > URL: https://issues.apache.org/jira/browse/HBASE-18534 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18534.000.patch, HBASE-18534.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120817#comment-16120817
]
Hadoop QA commented on HBASE-14135:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 42s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 57s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay |
| | org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestDeleteTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure |
| | org.apache.hadoop.hbase.regionserver.TestRowTooBig |
| | org.apache.hadoop.hbase.regionserver.wal.TestAsyncWALReplay |
| | org.apache.hadoop.hbase.regionserver.TestSplitLogWorker |
| | org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy
|
| | org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
| | org.apache.hadoop.hbase.regionserver.wal.TestFSHLog |
| | org.apache.hadoop.hbase.regionserver.TestCompaction |
| | org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| | org.apache.hadoop.hbase.regionserver.TestRemoveRegionMetrics |
| | org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries |
| | org.apache.hadoop.hbase.backup.TestRemoteBackup |
| | org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics |
| | org.apache.hadoop.hbase.regionserver.TestTags |
| | org.apache.hadoop.hbase.regionserver.TestTimestampFilterSeekHint |
| | org.apache.hadoop.hbase.TestHBaseTestingUtility |
| | org.apache.hadoop.hbase.TestFullLogReconstruction |
| |
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
|
| | org.apache.hadoop.hbase.wal.TestWALFilter
[jira] [Updated] (HBASE-18533) Expose BucketCache values to be configured
[ https://issues.apache.org/jira/browse/HBASE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-18533: -- Release Note: This patch exposes configuration for Bucketcache. These configs are very similar to those for the LRU cache, but are described below: "hbase.bucketcache.single.factor"; /** Single access bucket size */ "hbase.bucketcache.multi.factor"; /** Multiple access bucket size */ "hbase.bucketcache.memory.factor"; /** In-memory bucket size */ "hbase.bucketcache.extrafreefactor"; /** Free this floating point factor of extra blocks when evicting. For example free the number of blocks requested * (1 + extraFreeFactor) */ "hbase.bucketcache.acceptfactor"; /** Acceptable size of cache (no evictions if size < acceptable) */ "hbase.bucketcache.minfactor"; /** Minimum threshold of cache (when evicting, evict until size < min) */ was: This patch exposes configuration for Bucketcache. These configs are very similar to those for the LRU cache, but are described below: "hbase.bucketcache.single.percentage"; /** Single access bucket size */ "hbase.bucketcache.multi.percentage"; /** Multiple access bucket size */ "hbase.bucketcache.memory.percentage"; /** In-memory bucket size */ "hbase.bucketcache.extrafreefactor"; /** Free this floating point factor of extra blocks when evicting. For example free the number of blocks requested * (1 + extraFreeFactor) */ "hbase.bucketcache.acceptfactor"; /** Acceptable size of cache (no evictions if size < acceptable) */ "hbase.bucketcache.minfactor"; /** Minimum threshold of cache (when evicting, evict until size < min) */ > Expose BucketCache values to be configured > -- > > Key: HBASE-18533 > URL: https://issues.apache.org/jira/browse/HBASE-18533 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18533.branch-1.001.patch, > HBASE-18533.branch-1.002.patch, HBASE-18533.master.001.patch, > HBASE-18533.master.002.patch > > > BucketCache always uses the default values for all cache configuration. > However, this doesn't work for all use cases. In particular, users want to be > able to configure the percentage of the cache that is single access, multi > access, and in-memory access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18533) Expose BucketCache values to be configured
[ https://issues.apache.org/jira/browse/HBASE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-18533: -- Release Note: This patch exposes configuration for Bucketcache. These configs are very similar to those for the LRU cache, but are described below: "hbase.bucketcache.single.percentage"; /** Single access bucket size */ "hbase.bucketcache.multi.percentage"; /** Multiple access bucket size */ "hbase.bucketcache.memory.percentage"; /** In-memory bucket size */ "hbase.bucketcache.extrafreefactor"; /** Free this floating point factor of extra blocks when evicting. For example free the number of blocks requested * (1 + extraFreeFactor) */ "hbase.bucketcache.acceptfactor"; /** Acceptable size of cache (no evictions if size < acceptable) */ "hbase.bucketcache.minfactor"; /** Minimum threshold of cache (when evicting, evict until size < min) */ > Expose BucketCache values to be configured > -- > > Key: HBASE-18533 > URL: https://issues.apache.org/jira/browse/HBASE-18533 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18533.branch-1.001.patch, > HBASE-18533.branch-1.002.patch, HBASE-18533.master.001.patch, > HBASE-18533.master.002.patch > > > BucketCache always uses the default values for all cache configuration. > However, this doesn't work for all use cases. In particular, users want to be > able to configure the percentage of the cache that is single access, multi > access, and in-memory access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18534) [C++] Support timeout in Rpc
[ https://issues.apache.org/jira/browse/HBASE-18534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120731#comment-16120731 ] Xiaobing Zhou edited comment on HBASE-18534 at 8/9/17 10:35 PM: Posted v0: # added per request read RPC timeout Need to also handle write RPC timeout for next. was (Author: xiaobingo): Posted v0: # added per request timeout > [C++] Support timeout in Rpc > > > Key: HBASE-18534 > URL: https://issues.apache.org/jira/browse/HBASE-18534 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18534.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18533) Expose BucketCache values to be configured
[ https://issues.apache.org/jira/browse/HBASE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120762#comment-16120762 ] Andrew Purtell commented on HBASE-18533: Anything good for branch-1 is still good for branch-1.4, thanks > Expose BucketCache values to be configured > -- > > Key: HBASE-18533 > URL: https://issues.apache.org/jira/browse/HBASE-18533 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18533.branch-1.001.patch, > HBASE-18533.branch-1.002.patch, HBASE-18533.master.001.patch, > HBASE-18533.master.002.patch > > > BucketCache always uses the default values for all cache configuration. > However, this doesn't work for all use cases. In particular, users want to be > able to configure the percentage of the cache that is single access, multi > access, and in-memory access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18533) Expose BucketCache values to be configured
[ https://issues.apache.org/jira/browse/HBASE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120757#comment-16120757 ] Zach York commented on HBASE-18533: --- How can I trigger the QA again? > Expose BucketCache values to be configured > -- > > Key: HBASE-18533 > URL: https://issues.apache.org/jira/browse/HBASE-18533 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18533.branch-1.001.patch, > HBASE-18533.branch-1.002.patch, HBASE-18533.master.001.patch, > HBASE-18533.master.002.patch > > > BucketCache always uses the default values for all cache configuration. > However, this doesn't work for all use cases. In particular, users want to be > able to configure the percentage of the cache that is single access, multi > access, and in-memory access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18520) Add jmx value to determine true Master Start time
[ https://issues.apache.org/jira/browse/HBASE-18520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120753#comment-16120753 ] Ted Yu commented on HBASE-18520: Fill out release notes, please. > Add jmx value to determine true Master Start time > - > > Key: HBASE-18520 > URL: https://issues.apache.org/jira/browse/HBASE-18520 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Zach York >Assignee: Zach York >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-2 > > Attachments: HBASE-18520.branch-1.001.patch, > HBASE-18520.branch-1.002.patch, HBASE-18520.master.001.patch, > HBASE-18520.master.002.patch, HBASE-18520.master.003.patch > > > The masterActiveTime is being set before regions are assigned. This patch > adds a new jmx metric to expose the final time when the master has become the > active master (All regions are assigned, etc.). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18533) Expose BucketCache values to be configured
[ https://issues.apache.org/jira/browse/HBASE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120751#comment-16120751 ] Ted Yu commented on HBASE-18533: Can you get a clean QA run ? Fill out release notes. > Expose BucketCache values to be configured > -- > > Key: HBASE-18533 > URL: https://issues.apache.org/jira/browse/HBASE-18533 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18533.branch-1.001.patch, > HBASE-18533.branch-1.002.patch, HBASE-18533.master.001.patch, > HBASE-18533.master.002.patch > > > BucketCache always uses the default values for all cache configuration. > However, this doesn't work for all use cases. In particular, users want to be > able to configure the percentage of the cache that is single access, multi > access, and in-memory access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18533) Expose BucketCache values to be configured
[ https://issues.apache.org/jira/browse/HBASE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120747#comment-16120747 ] Zach York commented on HBASE-18533: --- [~tedyu] Do you have any more comments? I uploaded a new diff to the reviewboard and updated both patches. Also [~stack] and [~apurtell] are you guys okay with this going on branch-2 and branch-1.4 respectively? > Expose BucketCache values to be configured > -- > > Key: HBASE-18533 > URL: https://issues.apache.org/jira/browse/HBASE-18533 > Project: HBase > Issue Type: Improvement > Components: BucketCache >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18533.branch-1.001.patch, > HBASE-18533.branch-1.002.patch, HBASE-18533.master.001.patch, > HBASE-18533.master.002.patch > > > BucketCache always uses the default values for all cache configuration. > However, this doesn't work for all use cases. In particular, users want to be > able to configure the percentage of the cache that is single access, multi > access, and in-memory access. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18550) Move sanity checks for BucketCache configs inside BucketCache
Zach York created HBASE-18550: - Summary: Move sanity checks for BucketCache configs inside BucketCache Key: HBASE-18550 URL: https://issues.apache.org/jira/browse/HBASE-18550 Project: HBase Issue Type: Improvement Components: BucketCache Reporter: Zach York Assignee: Zach York In https://reviews.apache.org/r/61485/, Ramkrishna asked if I would move the sanity checking of all configs into the BucketCache constructor. This is a follow-up JIRA to move the sanity checking into the constructor. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120741#comment-16120741
]
Josh Elser commented on HBASE-14135:
bq. We do have repair tool. It takes care of most failure scenarios (not all of
them probably) and does housekeeping appropriately.
Ah! I'm just showing some ignorance then. That sounds good to me.
bq. v10. Some minor fixes based on a recent RB comments
I will try to find some time to verify that, but I think this is essentially
ready to go.
> HBase Backup/Restore Phase 3: Merge backup images
> -
>
> Key: HBASE-14135
> URL: https://issues.apache.org/jira/browse/HBASE-14135
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14135-v10.patch, HBASE-14135-v3.patch,
> HBASE-14135-v5.patch, HBASE-14135-v6.patch, HBASE-14135-v7.patch,
> HBASE-14135-v8.patch, HBASE-14135-v9.patch
>
>
> User can merge incremental backup images into single incremental backup image.
> # Merge supports only incremental images
> # Merge supports only images for the same backup destinations
> Command:
> {code}
> hbase backup merge image1,image2,..imageK
> {code}
> Example:
> {code}
> hbase backup merge backup_143126764557,backup_143126764456
> {code}
> When operation is complete, only the most recent backup image will be kept
> (in above example - backup_143126764557) as a merged backup image, all other
> images will be deleted from both: file system and backup system tables,
> corresponding backup manifest for the merged backup image will be updated to
> remove dependencies from deleted images. Merged backup image will contains
> all the data from original image and from deleted images.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18534) [C++] Support timeout in Rpc
[ https://issues.apache.org/jira/browse/HBASE-18534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120731#comment-16120731 ] Xiaobing Zhou commented on HBASE-18534: --- Posted v0: # added per request timeout > [C++] Support timeout in Rpc > > > Key: HBASE-18534 > URL: https://issues.apache.org/jira/browse/HBASE-18534 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18534.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18534) [C++] Support timeout in Rpc
[ https://issues.apache.org/jira/browse/HBASE-18534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-18534: -- Attachment: HBASE-18534.000.patch > [C++] Support timeout in Rpc > > > Key: HBASE-18534 > URL: https://issues.apache.org/jira/browse/HBASE-18534 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-18534.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18548) Move sources of important Jenkins jobs into source control
[ https://issues.apache.org/jira/browse/HBASE-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-18548: Status: Patch Available (was: Open) > Move sources of important Jenkins jobs into source control > -- > > Key: HBASE-18548 > URL: https://issues.apache.org/jira/browse/HBASE-18548 > Project: HBase > Issue Type: Bug > Components: documentation, scripts >Affects Versions: 2.0.0-alpha-1 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18548.patch > > > Move sources of the following Jenkins jobs into source control: > https://builds.apache.org/job/hbase_generate_website/ > https://builds.apache.org/view/All/job/HBase%20Website%20Link%20Checker/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18548) Move sources of important Jenkins jobs into source control
[ https://issues.apache.org/jira/browse/HBASE-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120695#comment-16120695 ] Misty Stanley-Jones commented on HBASE-18548: - Assuming this gets approved and merged, I'll update the Jenkins jobs to use the scripts directly. > Move sources of important Jenkins jobs into source control > -- > > Key: HBASE-18548 > URL: https://issues.apache.org/jira/browse/HBASE-18548 > Project: HBase > Issue Type: Bug > Components: documentation, scripts >Affects Versions: 2.0.0-alpha-1 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18548.patch > > > Move sources of the following Jenkins jobs into source control: > https://builds.apache.org/job/hbase_generate_website/ > https://builds.apache.org/view/All/job/HBase%20Website%20Link%20Checker/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18548) Move sources of important Jenkins jobs into source control
[ https://issues.apache.org/jira/browse/HBASE-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-18548: Attachment: HBASE-18548.patch Patch includes the two scripts and some updated docs about how to generate the site. > Move sources of important Jenkins jobs into source control > -- > > Key: HBASE-18548 > URL: https://issues.apache.org/jira/browse/HBASE-18548 > Project: HBase > Issue Type: Bug > Components: documentation, scripts >Affects Versions: 2.0.0-alpha-1 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18548.patch > > > Move sources of the following Jenkins jobs into source control: > https://builds.apache.org/job/hbase_generate_website/ > https://builds.apache.org/view/All/job/HBase%20Website%20Link%20Checker/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18541) [C++] Segfaults from JNI
[
https://issues.apache.org/jira/browse/HBASE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120683#comment-16120683
]
Ted Yu commented on HBASE-18541:
The stack trace involved AsyncRpcClient.createRpcChannel().
Looking at the master branch, there isn't AsyncRpcClient.java.
Can we sync up HBASE-14850 branch with the master branch ?
> [C++] Segfaults from JNI
>
>
> Key: HBASE-18541
> URL: https://issues.apache.org/jira/browse/HBASE-18541
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Ted Yu
>
> retry-test and multi-retry-test fails flakily when run with
> {code}
> buck test --all --no-results-cache
> {code}
> or when run in a loop:
> {code}
> for i in `seq 1 10`; do buck test --no-results-cache core:retry-test || break
> 1; done
> {code}
> The problem seems to be within the JNI internals and usually happens at the
> create table method call. I was not able to inspect much, but the comments in
> our mini-cluster indicate that we may need to use global references instead
> of local ones. I suspect the problem happens when there is a GC run for the
> test since the failure happens usually after some time (but almost always in
> create table method).
> [~ted_yu] do you mind taking a look at this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18549) Unclaimed replication queues can go undetected
Ashu Pachauri created HBASE-18549: - Summary: Unclaimed replication queues can go undetected Key: HBASE-18549 URL: https://issues.apache.org/jira/browse/HBASE-18549 Project: HBase Issue Type: Bug Components: Replication Reporter: Ashu Pachauri Priority: Critical Fix For: 1.3.2 We have come across this situation multiple times where a zookeeper issues can cause NodeFailoverWorker to fail picking up replication queue for a dead region server silently. One example is when the znode size for a particular queue exceed jute.maxBuffer value. There can be other situations that may lead to this and just go undetected. We need to have a metric for number of unclaimed replication queues. This will help in mitigating the problem through alerting on the metric and identifying underlying issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-14135:
--
Attachment: HBASE-14135-v10.patch
v10. Some minor fixes based on a recent RB comments
> HBase Backup/Restore Phase 3: Merge backup images
> -
>
> Key: HBASE-14135
> URL: https://issues.apache.org/jira/browse/HBASE-14135
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14135-v10.patch, HBASE-14135-v3.patch,
> HBASE-14135-v5.patch, HBASE-14135-v6.patch, HBASE-14135-v7.patch,
> HBASE-14135-v8.patch, HBASE-14135-v9.patch
>
>
> User can merge incremental backup images into single incremental backup image.
> # Merge supports only incremental images
> # Merge supports only images for the same backup destinations
> Command:
> {code}
> hbase backup merge image1,image2,..imageK
> {code}
> Example:
> {code}
> hbase backup merge backup_143126764557,backup_143126764456
> {code}
> When operation is complete, only the most recent backup image will be kept
> (in above example - backup_143126764557) as a merged backup image, all other
> images will be deleted from both: file system and backup system tables,
> corresponding backup manifest for the merged backup image will be updated to
> remove dependencies from deleted images. Merged backup image will contains
> all the data from original image and from deleted images.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120641#comment-16120641
]
Vladimir Rodionov commented on HBASE-14135:
---
{quote}
If it would be too difficult (which is what my gut-reaction was), a
tool/utility to summarize these (expected) transient data (files in HDFS and
hbase:backup records) would be really nice to have. If/when we would have to
debug some kind of issue WRT backups or just HDFS use by hbase, such a tool
could give us a definitive yes/no as to whether these transient files are to
blame or not.
{quote}
We do have repair tool. It takes care of most failure scenarios (not all of
them probably) and does housekeeping appropriately.
> HBase Backup/Restore Phase 3: Merge backup images
> -
>
> Key: HBASE-14135
> URL: https://issues.apache.org/jira/browse/HBASE-14135
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14135-v3.patch, HBASE-14135-v5.patch,
> HBASE-14135-v6.patch, HBASE-14135-v7.patch, HBASE-14135-v8.patch,
> HBASE-14135-v9.patch
>
>
> User can merge incremental backup images into single incremental backup image.
> # Merge supports only incremental images
> # Merge supports only images for the same backup destinations
> Command:
> {code}
> hbase backup merge image1,image2,..imageK
> {code}
> Example:
> {code}
> hbase backup merge backup_143126764557,backup_143126764456
> {code}
> When operation is complete, only the most recent backup image will be kept
> (in above example - backup_143126764557) as a merged backup image, all other
> images will be deleted from both: file system and backup system tables,
> corresponding backup manifest for the merged backup image will be updated to
> remove dependencies from deleted images. Merged backup image will contains
> all the data from original image and from deleted images.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18525) TestAssignmentManager#testSocketTimeout fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-18525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120640#comment-16120640
]
Hudson commented on HBASE-18525:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3502 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3502/])
HBASE-18525 [AMv2] Fixed test TestAssignmentManager#testSocketTimeout on
(stack: rev 67eddf5874f6a008fe341fa6ce6b34e42a945118)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestAssignmentManager.java
* (edit) pom.xml
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
> TestAssignmentManager#testSocketTimeout fails in master branch
> --
>
> Key: HBASE-18525
> URL: https://issues.apache.org/jira/browse/HBASE-18525
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Umesh Agashe
> Fix For: 2.0.0-alpha-2
>
> Attachments: 18525.v1.txt, 18525.v2.txt, hbase-18525.master.003.patch
>
>
> Toward the end of the test output, I saw:
> {code}
> 2017-08-05 03:30:16,591 INFO [Time-limited test]
> assignment.TestAssignmentManager(446): ExecutionException
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hbase.master.procedure.ServerCrashException:
> ServerCrashProcedure pid=3, server=localhost,103,1
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait$ProcedureFuture.get(ProcedureSyncWait.java:104)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait$ProcedureFuture.get(ProcedureSyncWait.java:62)
> at
> org.apache.hadoop.hbase.master.assignment.TestAssignmentManager.waitOnFuture(TestAssignmentManager.java:444)
> at
> org.apache.hadoop.hbase.master.assignment.TestAssignmentManager.testSocketTimeout(TestAssignmentManager.java:255)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.master.procedure.ServerCrashException:
> ServerCrashProcedure pid=3, server=localhost,103,1
> at
> org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:169)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:274)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:847)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1440)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1209)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:79)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1719)
> {code}
> This test failure seems to happen after HBASE-18491 was checked in.
> Looking at the change in UnassignProcedure, it seems we should handle the two
> conditions differently:
> {code}
> if (serverCrashed.get() || !isServerOnline(env, regionNode)) {
> {code}
> With attached patch, TestAssignmentManager#testSocketTimeout and
> TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta pass.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18528) DON'T allow user to modify the passed table/column descriptor
[
https://issues.apache.org/jira/browse/HBASE-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120613#comment-16120613
]
Hadoop QA commented on HBASE-18528:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 57s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
32s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}105m
48s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18528 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881052/HBASE-18528.v0.patch |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux b1359772f3e4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 67eddf5 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8002/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8002/console |
| Powered by |
[jira] [Created] (HBASE-18548) Move sources of important Jenkins jobs into source control
Misty Stanley-Jones created HBASE-18548: --- Summary: Move sources of important Jenkins jobs into source control Key: HBASE-18548 URL: https://issues.apache.org/jira/browse/HBASE-18548 Project: HBase Issue Type: Bug Components: documentation, scripts Affects Versions: 2.0.0-alpha-1 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Priority: Critical Fix For: 2.0.0 Move sources of the following Jenkins jobs into source control: https://builds.apache.org/job/hbase_generate_website/ https://builds.apache.org/view/All/job/HBase%20Website%20Link%20Checker/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18525) TestAssignmentManager#testSocketTimeout fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-18525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120597#comment-16120597
]
Hudson commented on HBASE-18525:
FAILURE: Integrated in Jenkins build HBase-2.0 #299 (See
[https://builds.apache.org/job/HBase-2.0/299/])
HBASE-18525 [AMv2] Fixed test TestAssignmentManager#testSocketTimeout on
(stack: rev f855b51650f2c8e3b2baac6132c79a5e460285e0)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestAssignmentManager.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
* (edit) pom.xml
> TestAssignmentManager#testSocketTimeout fails in master branch
> --
>
> Key: HBASE-18525
> URL: https://issues.apache.org/jira/browse/HBASE-18525
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Umesh Agashe
> Fix For: 2.0.0-alpha-2
>
> Attachments: 18525.v1.txt, 18525.v2.txt, hbase-18525.master.003.patch
>
>
> Toward the end of the test output, I saw:
> {code}
> 2017-08-05 03:30:16,591 INFO [Time-limited test]
> assignment.TestAssignmentManager(446): ExecutionException
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hbase.master.procedure.ServerCrashException:
> ServerCrashProcedure pid=3, server=localhost,103,1
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait$ProcedureFuture.get(ProcedureSyncWait.java:104)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait$ProcedureFuture.get(ProcedureSyncWait.java:62)
> at
> org.apache.hadoop.hbase.master.assignment.TestAssignmentManager.waitOnFuture(TestAssignmentManager.java:444)
> at
> org.apache.hadoop.hbase.master.assignment.TestAssignmentManager.testSocketTimeout(TestAssignmentManager.java:255)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.master.procedure.ServerCrashException:
> ServerCrashProcedure pid=3, server=localhost,103,1
> at
> org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:169)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:274)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:847)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1440)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1209)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:79)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1719)
> {code}
> This test failure seems to happen after HBASE-18491 was checked in.
> Looking at the change in UnassignProcedure, it seems we should handle the two
> conditions differently:
> {code}
> if (serverCrashed.get() || !isServerOnline(env, regionNode)) {
> {code}
> With attached patch, TestAssignmentManager#testSocketTimeout and
> TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta pass.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18529) Do not delete the tmp jars dir when load the coprocessor jar
[
https://issues.apache.org/jira/browse/HBASE-18529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120567#comment-16120567
]
Ted Yu commented on HBASE-18529:
We also need to consider compatibility issue if hbase + ServerName is used for
directory name
> Do not delete the tmp jars dir when load the coprocessor jar
>
>
> Key: HBASE-18529
> URL: https://issues.apache.org/jira/browse/HBASE-18529
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Reporter: Yun Zhao
>Assignee: Yun Zhao
> Attachments: HBASE-18529.master.001.patch,
> HBASE-18529.master.002.patch
>
>
> When multi regionserver is deployed on a single server, used default
> hbase.local.dir . The tmp jars dir will deleted when one of them is restarted.
> Also when multi regionserver start at the same time, the jar in the
> copyToLocalFile process may be deleted, causing the coprocessor load failed.
> {code}
> 2017-08-06 20:02:15,326 ERROR [RS_OPEN_REGION--2]
> regionserver.RegionCoprocessorHost: Failed to load coprocessor
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native
> Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:226)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:629)
> at
> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:467)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1968)
> at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1937)
> at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1913)
> at
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:168)
> at
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:250)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18541) [C++] Segfaults from JNI
[
https://issues.apache.org/jira/browse/HBASE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120556#comment-16120556
]
Ted Yu commented on HBASE-18541:
I used the 'for' command which reproduced the test failure.
However, there was not much information in the test output:
{code}
-rw-r--r-- 1 root root 10 Aug 9 17:59 exitCode
-rw-r--r-- 1 root root 15 Aug 9 17:59 output
-rw-r--r-- 1 root root 0 Aug 9 17:59 results
{code}
> [C++] Segfaults from JNI
>
>
> Key: HBASE-18541
> URL: https://issues.apache.org/jira/browse/HBASE-18541
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Ted Yu
>
> retry-test and multi-retry-test fails flakily when run with
> {code}
> buck test --all --no-results-cache
> {code}
> or when run in a loop:
> {code}
> for i in `seq 1 10`; do buck test --no-results-cache core:retry-test || break
> 1; done
> {code}
> The problem seems to be within the JNI internals and usually happens at the
> create table method call. I was not able to inspect much, but the comments in
> our mini-cluster indicate that we may need to use global references instead
> of local ones. I suspect the problem happens when there is a GC run for the
> test since the failure happens usually after some time (but almost always in
> create table method).
> [~ted_yu] do you mind taking a look at this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18547) Provide better cleanup of WAL related entries in backup table
Ted Yu created HBASE-18547: -- Summary: Provide better cleanup of WAL related entries in backup table Key: HBASE-18547 URL: https://issues.apache.org/jira/browse/HBASE-18547 Project: HBase Issue Type: Bug Reporter: Ted Yu I was discussing the design around incremental backup with Vladimir. Currently if WAL file is recorded in backup table, BackupLogCleaner would signal that the file can be cleaned up. However, cleaner chore doesn't notify BackupLogCleaner whether the file is actually deleted. This means that potentially large number of WAL files would stay in backup table. This issue is to investigate better cleanup strategy for the WAL entries. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18547) Provide better cleanup of WAL related entries in backup table
[ https://issues.apache.org/jira/browse/HBASE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18547: --- Description: I was discussing the design around incremental backup with Vladimir. Currently if WAL file is recorded in backup table, BackupLogCleaner would signal that the file can be cleaned up. However, cleaner chore doesn't notify BackupLogCleaner whether the file is actually deleted. This means that potentially large number of WAL file entries would stay in backup table. This issue is to investigate better cleanup strategy for the WAL entries. was: I was discussing the design around incremental backup with Vladimir. Currently if WAL file is recorded in backup table, BackupLogCleaner would signal that the file can be cleaned up. However, cleaner chore doesn't notify BackupLogCleaner whether the file is actually deleted. This means that potentially large number of WAL files would stay in backup table. This issue is to investigate better cleanup strategy for the WAL entries. > Provide better cleanup of WAL related entries in backup table > - > > Key: HBASE-18547 > URL: https://issues.apache.org/jira/browse/HBASE-18547 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > > I was discussing the design around incremental backup with Vladimir. > Currently if WAL file is recorded in backup table, BackupLogCleaner would > signal that the file can be cleaned up. > However, cleaner chore doesn't notify BackupLogCleaner whether the file is > actually deleted. > This means that potentially large number of WAL file entries would stay in > backup table. > This issue is to investigate better cleanup strategy for the WAL entries. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15511) ClusterStatus should be able to return responses by scope
[
https://issues.apache.org/jira/browse/HBASE-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120489#comment-16120489
]
Hadoop QA commented on HBASE-15511:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 41s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
55s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}119m
33s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
53s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}183m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.13.1 Server=1.13.1 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-15511 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881038/HBASE-15511.master.011.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux 19e569b33346 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool
[jira] [Commented] (HBASE-18528) DON'T allow user to modify the passed table/column descriptor
[ https://issues.apache.org/jira/browse/HBASE-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120488#comment-16120488 ] Stephen Yuan Jiang commented on HBASE-18528: +1 > DON'T allow user to modify the passed table/column descriptor > - > > Key: HBASE-18528 > URL: https://issues.apache.org/jira/browse/HBASE-18528 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, master >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18528.v0.patch > > > We are replacing the HTableDescriptor by TableDescriptor from code base. The > TableDescriptor is designed to be a read-only object so user can't modifiy it > through MasterObserver. HBASE-18502 change many methods of MasterObserver to > use TableDescriptor but some deprecated methods still accept the > HTableDescriptor. User may be confused by why some methods can't modify the > table descriptor. > In short, Should we allow user to modify the passed table descriptor? > # if yes, we should introduce a mechanism that user can return a modified > table descripror > # if no, we should pass ImmutableHTableDescriptor to user. Or we just remove > all methods accepting the HTableDescriptor > Ditto for HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120464#comment-16120464
]
Chia-Ping Tsai commented on HBASE-18546:
Ditto for Increment#setTimeRange
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-18546:
---
Priority: Critical (was: Major)
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-18546:
---
Fix Version/s: 2.0.0-alpha-2
3.0.0
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 3.0.0, 2.0.0-alpha-2
>
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18546) Append/Increment a cell with custom timestamp
[
https://issues.apache.org/jira/browse/HBASE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120459#comment-16120459
]
Chia-Ping Tsai commented on HBASE-18546:
For operational compatibility, we should fix the Increment. But i have no use
cases..
> Append/Increment a cell with custom timestamp
> -
>
> Key: HBASE-18546
> URL: https://issues.apache.org/jira/browse/HBASE-18546
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>
> We don't pass the custom timestamp for Increment, and the increment's
> timestamp always be rewrite. Hence, user can't increment a cell with custom
> timestamp.
> {code:title=ProtobufUtil.java}
> if (values != null && values.size() > 0) {
> for (Cell cell: values) {
> valueBuilder.clear();
> valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
> cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength()));
> valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
> cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength()));
> if (cell.getTagsLength() > 0) {
>
> valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
> cell.getTagsOffset(), cell.getTagsLength()));
> }
> columnBuilder.addQualifierValue(valueBuilder.build());
> }
> }
> {code}
> In contrast to Increment, user can append the cell with custom timestamp. It
> would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18546) Append/Increment a cell with custom timestamp
Chia-Ping Tsai created HBASE-18546:
--
Summary: Append/Increment a cell with custom timestamp
Key: HBASE-18546
URL: https://issues.apache.org/jira/browse/HBASE-18546
Project: HBase
Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai
We don't pass the custom timestamp for Increment, and the increment's timestamp
always be rewrite. Hence, user can't increment a cell with custom timestamp.
{code:title=ProtobufUtil.java}
if (values != null && values.size() > 0) {
for (Cell cell: values) {
valueBuilder.clear();
valueBuilder.setQualifier(UnsafeByteOperations.unsafeWrap(
cell.getQualifierArray(), cell.getQualifierOffset(),
cell.getQualifierLength()));
valueBuilder.setValue(UnsafeByteOperations.unsafeWrap(
cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength()));
if (cell.getTagsLength() > 0) {
valueBuilder.setTags(UnsafeByteOperations.unsafeWrap(cell.getTagsArray(),
cell.getTagsOffset(), cell.getTagsLength()));
}
columnBuilder.addQualifierValue(valueBuilder.build());
}
}
{code}
In contrast to Increment, user can append the cell with custom timestamp. It
would be better that make their behavior consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18024) HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists
[
https://issues.apache.org/jira/browse/HBASE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120428#comment-16120428
]
Hadoop QA commented on HBASE-18024:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 39s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}111m
22s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18024 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880969/HBASE-18024.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 0040094b56df 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f314b59 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8000/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8000/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> HRegion#initializeRegionInternals should not re-create .hregioninfo file when
> the region directory no longer exists
> ---
>
> Key: HBASE-18024
> URL: https://issues.apache.org/jira/browse/HBASE-18024
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Af
[jira] [Commented] (HBASE-18537) [C++] Improvements to load-client
[ https://issues.apache.org/jira/browse/HBASE-18537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120413#comment-16120413 ] Enis Soztutar commented on HBASE-18537: --- [~ted_yu] do you mind taking a look. > [C++] Improvements to load-client > - > > Key: HBASE-18537 > URL: https://issues.apache.org/jira/browse/HBASE-18537 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: HBASE-14850 > > Attachments: hbase-18537.patch > > > A couple of improvements to the load-client after spending some time with > testing: > - better log messages > - support for progress > - minor bug fixes -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18545) Fix broke site build that fails in rsgroups
[
https://issues.apache.org/jira/browse/HBASE-18545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-18545.
---
Resolution: Fixed
Fix Version/s: 3.0.0
I pushed this. It seems to fix it:
{code}
commit 794a3b104bf62d028fc8ae27596c3b98ef87e507
Author: Michael Stack
Date: Wed Aug 9 11:24:51 2017 -0700
HBASE-18545 Fix broke site build that fails in rsgroups
diff --git a/hbase-rsgroup/pom.xml b/hbase-rsgroup/pom.xml
index 0bbabe93ff..a3e710cac2 100644
--- a/hbase-rsgroup/pom.xml
+++ b/hbase-rsgroup/pom.xml
@@ -60,6 +60,11 @@
compile
+
+
+
${basedir}/../hbase-protocol/src/main/protobuf
+
+
{code}
Repro'd the issue by running commands found here:
https://builds.apache.org/view/H-L/view/HBase/job/hbase_generate_website/configure
> Fix broke site build that fails in rsgroups
> ---
>
> Key: HBASE-18545
> URL: https://issues.apache.org/jira/browse/HBASE-18545
> Project: HBase
> Issue Type: Bug
> Components: site
>Reporter: stack
>Assignee: stack
> Fix For: 3.0.0
>
>
> Site is broke because of a failure finding .proto files (was pointed out on
> list and in closed proto issue...). Failure looks like this:
> {code}
> [INFO] --- protobuf-maven-plugin:0.5.0:compile (compile-protoc) @
> hbase-rsgroup ---
> [INFO] Compiling 2 proto file(s) to
> /home/jenkins/jenkins-slave/workspace/hbase_generate_website/hbase/hbase-rsgroup/target/generated-sources/protobuf/java
> [ERROR] PROTOC FAILED: HBase.proto: File not found.
> RSGroup.proto: Import "HBase.proto" was not found or had errors.
> RSGroup.proto:31:12: "ServerName" is not defined.
> RSGroup.proto:32:12: "TableName" is not defined.
> [ERROR]
> /home/jenkins/jenkins-slave/workspace/hbase_generate_website/hbase/hbase-rsgroup/src/main/protobuf/RSGroup.proto
> [0:0]: HBase.proto: File not found.
> RSGroup.proto: Import "HBase.proto" was not found or had errors.
> RSGroup.proto:31:12: "ServerName" is not defined.
> RSGroup.proto:32:12: "TableName" is not defined.
> [ERROR]
> /home/jenkins/jenkins-slave/workspace/hbase_generate_website/hbase/hbase-rsgroup/src/main/protobuf/RSGroupAdmin.proto
> [0:0]: HBase.proto: File not found.
> RSGroup.proto: Import "HBase.proto" was not found or had errors.
> RSGroup.proto:31:12: "ServerName" is not defined.
> RSGroup.proto:32:12: "TableName" is not defined.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18545) Fix broke site build that fails in rsgroups
stack created HBASE-18545:
-
Summary: Fix broke site build that fails in rsgroups
Key: HBASE-18545
URL: https://issues.apache.org/jira/browse/HBASE-18545
Project: HBase
Issue Type: Bug
Components: site
Reporter: stack
Assignee: stack
Site is broke because of a failure finding .proto files (was pointed out on
list and in closed proto issue...). Failure looks like this:
{code}
[INFO] --- protobuf-maven-plugin:0.5.0:compile (compile-protoc) @ hbase-rsgroup
---
[INFO] Compiling 2 proto file(s) to
/home/jenkins/jenkins-slave/workspace/hbase_generate_website/hbase/hbase-rsgroup/target/generated-sources/protobuf/java
[ERROR] PROTOC FAILED: HBase.proto: File not found.
RSGroup.proto: Import "HBase.proto" was not found or had errors.
RSGroup.proto:31:12: "ServerName" is not defined.
RSGroup.proto:32:12: "TableName" is not defined.
[ERROR]
/home/jenkins/jenkins-slave/workspace/hbase_generate_website/hbase/hbase-rsgroup/src/main/protobuf/RSGroup.proto
[0:0]: HBase.proto: File not found.
RSGroup.proto: Import "HBase.proto" was not found or had errors.
RSGroup.proto:31:12: "ServerName" is not defined.
RSGroup.proto:32:12: "TableName" is not defined.
[ERROR]
/home/jenkins/jenkins-slave/workspace/hbase_generate_website/hbase/hbase-rsgroup/src/main/protobuf/RSGroupAdmin.proto
[0:0]: HBase.proto: File not found.
RSGroup.proto: Import "HBase.proto" was not found or had errors.
RSGroup.proto:31:12: "ServerName" is not defined.
RSGroup.proto:32:12: "TableName" is not defined.
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18528) DON'T allow user to modify the passed table/column descriptor
[ https://issues.apache.org/jira/browse/HBASE-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120406#comment-16120406 ] Chia-Ping Tsai commented on HBASE-18528: bq. Why 2 way indirection? The impl of desc passed to MasterCoprocessorHost should be ModifyableTableDescriptor. The ImmutableHTableDescriptor, which is deprecated , will be removed in 3.0 so we should prevent to use it in code base. > DON'T allow user to modify the passed table/column descriptor > - > > Key: HBASE-18528 > URL: https://issues.apache.org/jira/browse/HBASE-18528 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, master >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18528.v0.patch > > > We are replacing the HTableDescriptor by TableDescriptor from code base. The > TableDescriptor is designed to be a read-only object so user can't modifiy it > through MasterObserver. HBASE-18502 change many methods of MasterObserver to > use TableDescriptor but some deprecated methods still accept the > HTableDescriptor. User may be confused by why some methods can't modify the > table descriptor. > In short, Should we allow user to modify the passed table descriptor? > # if yes, we should introduce a mechanism that user can return a modified > table descripror > # if no, we should pass ImmutableHTableDescriptor to user. Or we just remove > all methods accepting the HTableDescriptor > Ditto for HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18528) DON'T allow user to modify the passed table/column descriptor
[ https://issues.apache.org/jira/browse/HBASE-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120365#comment-16120365 ] Anoop Sam John commented on HBASE-18528: bq. return new ImmutableHTableDescriptor(new HTableDescriptor(desc)); Why 2 way indirection? Why can not make ImmutableHTableDescriptor directly from desc? > DON'T allow user to modify the passed table/column descriptor > - > > Key: HBASE-18528 > URL: https://issues.apache.org/jira/browse/HBASE-18528 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, master >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18528.v0.patch > > > We are replacing the HTableDescriptor by TableDescriptor from code base. The > TableDescriptor is designed to be a read-only object so user can't modifiy it > through MasterObserver. HBASE-18502 change many methods of MasterObserver to > use TableDescriptor but some deprecated methods still accept the > HTableDescriptor. User may be confused by why some methods can't modify the > table descriptor. > In short, Should we allow user to modify the passed table descriptor? > # if yes, we should introduce a mechanism that user can return a modified > table descripror > # if no, we should pass ImmutableHTableDescriptor to user. Or we just remove > all methods accepting the HTableDescriptor > Ditto for HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18525) TestAssignmentManager#testSocketTimeout fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-18525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120351#comment-16120351
]
Umesh Agashe commented on HBASE-18525:
--
Thanks for reviewing and fixing the Javadoc [~stack]!
> TestAssignmentManager#testSocketTimeout fails in master branch
> --
>
> Key: HBASE-18525
> URL: https://issues.apache.org/jira/browse/HBASE-18525
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Umesh Agashe
> Fix For: 2.0.0-alpha-2
>
> Attachments: 18525.v1.txt, 18525.v2.txt, hbase-18525.master.003.patch
>
>
> Toward the end of the test output, I saw:
> {code}
> 2017-08-05 03:30:16,591 INFO [Time-limited test]
> assignment.TestAssignmentManager(446): ExecutionException
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hbase.master.procedure.ServerCrashException:
> ServerCrashProcedure pid=3, server=localhost,103,1
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait$ProcedureFuture.get(ProcedureSyncWait.java:104)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait$ProcedureFuture.get(ProcedureSyncWait.java:62)
> at
> org.apache.hadoop.hbase.master.assignment.TestAssignmentManager.waitOnFuture(TestAssignmentManager.java:444)
> at
> org.apache.hadoop.hbase.master.assignment.TestAssignmentManager.testSocketTimeout(TestAssignmentManager.java:255)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.master.procedure.ServerCrashException:
> ServerCrashProcedure pid=3, server=localhost,103,1
> at
> org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:169)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:274)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:847)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1440)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1209)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:79)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1719)
> {code}
> This test failure seems to happen after HBASE-18491 was checked in.
> Looking at the change in UnassignProcedure, it seems we should handle the two
> conditions differently:
> {code}
> if (serverCrashed.get() || !isServerOnline(env, regionNode)) {
> {code}
> With attached patch, TestAssignmentManager#testSocketTimeout and
> TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta pass.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18528) DON'T allow user to modify the passed table/column descriptor
[ https://issues.apache.org/jira/browse/HBASE-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18528: --- Attachment: HBASE-18528.v0.patch > DON'T allow user to modify the passed table/column descriptor > - > > Key: HBASE-18528 > URL: https://issues.apache.org/jira/browse/HBASE-18528 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, master >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18528.v0.patch > > > We are replacing the HTableDescriptor by TableDescriptor from code base. The > TableDescriptor is designed to be a read-only object so user can't modifiy it > through MasterObserver. HBASE-18502 change many methods of MasterObserver to > use TableDescriptor but some deprecated methods still accept the > HTableDescriptor. User may be confused by why some methods can't modify the > table descriptor. > In short, Should we allow user to modify the passed table descriptor? > # if yes, we should introduce a mechanism that user can return a modified > table descripror > # if no, we should pass ImmutableHTableDescriptor to user. Or we just remove > all methods accepting the HTableDescriptor > Ditto for HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18528) DON'T allow user to modify the passed table/column descriptor
[ https://issues.apache.org/jira/browse/HBASE-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18528: --- Status: Patch Available (was: Open) > DON'T allow user to modify the passed table/column descriptor > - > > Key: HBASE-18528 > URL: https://issues.apache.org/jira/browse/HBASE-18528 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, master >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18528.v0.patch > > > We are replacing the HTableDescriptor by TableDescriptor from code base. The > TableDescriptor is designed to be a read-only object so user can't modifiy it > through MasterObserver. HBASE-18502 change many methods of MasterObserver to > use TableDescriptor but some deprecated methods still accept the > HTableDescriptor. User may be confused by why some methods can't modify the > table descriptor. > In short, Should we allow user to modify the passed table descriptor? > # if yes, we should introduce a mechanism that user can return a modified > table descripror > # if no, we should pass ImmutableHTableDescriptor to user. Or we just remove > all methods accepting the HTableDescriptor > Ditto for HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18517) limit max log message width in log4j
[
https://issues.apache.org/jira/browse/HBASE-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120312#comment-16120312
]
Josh Elser commented on HBASE-18517:
Excellent. Thanks much!
> limit max log message width in log4j
>
>
> Key: HBASE-18517
> URL: https://issues.apache.org/jira/browse/HBASE-18517
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0, 2.0.0-alpha-2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
> Fix For: 1.5.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18517.branch-1.001.patch,
> HBASE-18517.master.001.patch
>
>
> We had two cases now in our prod / pilot setups which is leading to humongous
> log lines in RegionServer logs.
> In first case, one of the phoenix user had constructed a query with a really
> large list of Id filters (61 MB) that translated into HBase scan that was
> running slow which lead to responseTooSlow messages in the logs with the
> entire filter list being printed in the logs, example
> ipc.RpcServer - (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1501457864417,"responsesize":11,"method":"Scan","param":"region
> { type: REGION_NAME value: .
> org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter\\022\351\\200\\036\\n(org.apache.phoenix.filter.SkipScanFilter
> ... ...
> There was another case where a use case had created a table with really large
> key/region names. This was causing humongous log lines for flush and
> compaction on these regions filling up the RS logs
> These large logs usually cause issues with disk I/O load, loading the splunk
> servers, even machine perf degradations. With 61 MB log lines basic log
> processing commands like vim, scrolling the logs, wc -l , etc were getting
> stuck. High GC activity was also noted on this cluster although not 100% sure
> if it was related to above issue.
> We should consider limiting the message size in logs which can be easily done
> by adding a maximum width format modifier on the message conversion character
> in log4j.properties
> log4j.appender.console.layout.ConversionPattern=...: %m%n
> to
> log4j.appender.console.layout.ConversionPattern=...: %.1m%n
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
