[jira] [Updated] (HBASE-18846) Accommodate the hbase-indexer/lily/SEP consumer deploy-type
[
https://issues.apache.org/jira/browse/HBASE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-18846:
--
Description:
This is a follow-on from HBASE-10504, Define a Replication Interface. There we
defined a new, flexible replication endpoint for others to implement but it did
little to help the case of the lily hbase-indexer. This issue takes up the case
of the hbase-indexer.
The hbase-indexer poses to hbase as a 'fake' peer cluster (For why
hbase-indexer is implemented so, the advantage to having the indexing done in a
separate process set that can be independently scaled, can participate in the
same security realm, etc., see discussion in HBASE-10504). The hbase-indexer
will start up a cut-down "RegionServer" processes that are just an instance of
hbase RpcServer hosting an AdminProtos Service. They make themselves 'appear'
to the Replication Source by hoisting up an ephemeral znode 'registering' as a
RegionServer. The source cluster then streams WALEdits to the Admin Protos
method:
{code}
public ReplicateWALEntryResponse replicateWALEntry(final RpcController
controller,
final ReplicateWALEntryRequest request) throws ServiceException {
{code}
The hbase-indexer relies on other hbase internals like Server so it can get a
ZooKeeperWatcher instance and know the 'name' to use for this cut-down server.
Thoughts on how to proceed include:
* Better formalize its current digestion of hbase internals; make it so
rpcserver is allowed to be used by others, etc. This would be hard to do given
they use basics like Server, Protobuf serdes for WAL types, and AdminProtos
Service. Any change in this wide API breaks (again) hbase-indexer. We have made
a 'channel' for Coprocessor Endpoints so they continue to work though they use
'internal' types. They can use protos in hbase-protocol. hbase-protocol protos
are in a limbo currently where they are sort-of 'public'; a TODO. Perhaps the
hbase-indexer could do similar relying on the hbase-protocol (pb2.5) content
and we could do something to reveal rpcserver and zk for hbase-indexer safe use.
* Start an actual RegionServer only have it register the AdminProtos Service
only -- not ClientProtos and the Service that does Master interaction, etc.
Then have the hbase-indexer implement an AdminCoprocessor to override the
replicateWALEntry method (the Admin CP implementation may need work). This
would narrow the hbase-indexer exposure to that of the Admin Coprocessor
Interface
Other crazy notions occur including the setup of an Admin Interface Coprocessor
Endpoint. A new ReplicationEndpoint would feed the replication stream to the
remote cluster via the CPEP registered channel.
But time is short. Hopefully we can figure something that will work in 2.0
timeframe w/o too much code movement.
was:
This is a follow-on from HBASE-10504, Define a Replication Interface. There we
defined a new, flexible replication endpoint for others to implement but it did
little to help the case of the lily hbase-indexer. This issue takes up the case
of the hbase-indexer.
The hbase-indexer poses to hbase as a 'fake' peer cluster. The hbase-indexer
will start up cut-down "RegionServer" processes that are nought but an hbase
RpcServer hosting an AdminProtos Service. They make themselves 'appear' to the
Replication Source by hoisting up an ephemeral znode 'registering' as a
RegionServer. The source cluster then streams WALEdits to the Admin Protos
method:
{code}
public ReplicateWALEntryResponse replicateWALEntry(final RpcController
controller,
final ReplicateWALEntryRequest request) throws ServiceException {
{code}
The hbase-indexer relies on other hbase internals like Server so it can get a
ZooKeeperWatcher instance and know the 'name' to use for this cut-down server.
Thoughts on how to proceed include:
* Better formalize its current digestion of hbase internals; make it so
rpcserver is allowed to be used by others, etc.
* Start an actual RegionServer only have it register the AdminProtos Service
only -- not AdminProtos and ClientProtos, etc. Then have the hbase-indexer
implement an AdminCoprocessor to override the replicateWALEntry method (the
Admin CP implementation may need work).
I'll be back
> Accommodate the hbase-indexer/lily/SEP consumer deploy-type
> ---
>
> Key: HBASE-18846
> URL: https://issues.apache.org/jira/browse/HBASE-18846
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>
> This is a follow-on from HBASE-10504, Define a Replication Interface. There
> we defined a new, flexible replication endpoint for others to implement but
> it did little to help the case of the lily hbase-indexer. This issue takes up
> the case of the hbase-indexer.
> The hbase-indexer poses to hbase as a 'fake' peer cluster (For
[jira] [Commented] (HBASE-18795) Expose KeyValue.getBuffer() for tests alone
[ https://issues.apache.org/jira/browse/HBASE-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171144#comment-16171144 ] Anoop Sam John commented on HBASE-18795: I would be we can leave it. Fix from 2.0 onwards > Expose KeyValue.getBuffer() for tests alone > --- > > Key: HBASE-18795 > URL: https://issues.apache.org/jira/browse/HBASE-18795 > Project: HBase > Issue Type: Improvement >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18795.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18807) Remove PB references from Observers for Quotas
[
https://issues.apache.org/jira/browse/HBASE-18807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171141#comment-16171141
]
Anoop Sam John commented on HBASE-18807:
Had a quick look at the patch. Seems fine approach.
{code}
default void preSetUserQuota(
1100 final ObserverContext ctx, final
String userName,
1101 final TableName tableName, final QuotaSettings quotas) throws
IOException {}
{code}
So for every CP hook, we pass the super type QuotaSettings. How easy/difficult
it will be the CP user to know which kind of Quota setting this is and get the
details? The implementations of this super type are not exposed! That means
they should not be using the specific methods in them. Is that enough?
Should we be exposing them and pass specific object types to corresponding CPs?
Or other ways?
> Remove PB references from Observers for Quotas
> --
>
> Key: HBASE-18807
> URL: https://issues.apache.org/jira/browse/HBASE-18807
> Project: HBase
> Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-18807.001.branch-2.patch
>
>
> Break-out from the parent:
> Same idea, just applied to the Observer methods for pre/post quota
> operations. Requires changes to MasterQuotaManager and the QuotaSettings
> implementations as some business logic is written on the PB objects directly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18795) Expose KeyValue.getBuffer() for tests alone
[ https://issues.apache.org/jira/browse/HBASE-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171144#comment-16171144 ] Anoop Sam John edited comment on HBASE-18795 at 9/19/17 5:42 AM: - I would say, we can leave it. Fix from 2.0 onwards was (Author: anoop.hbase): I would be we can leave it. Fix from 2.0 onwards > Expose KeyValue.getBuffer() for tests alone > --- > > Key: HBASE-18795 > URL: https://issues.apache.org/jira/browse/HBASE-18795 > Project: HBase > Issue Type: Improvement >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18795.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-10504) Define Replication Interface
[ https://issues.apache.org/jira/browse/HBASE-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171139#comment-16171139 ] stack edited comment on HBASE-10504 at 9/19/17 5:41 AM: Marking as 'done' in 2.0.0. This work was finished in 2.0.0 but also a bunch of extra work was done under the umbrella of this JIRA by [~zghaobac], etc. HBASE-18846 is the follow-on for hbase-indexer was (Author: stack): Marking as 'done' in 2.0.0. HBASE-18846 is the follow-on for hbase-indexer > Define Replication Interface > > > Key: HBASE-10504 > URL: https://issues.apache.org/jira/browse/HBASE-10504 > Project: HBase > Issue Type: Task > Components: Replication >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 2.0.0 > > Attachments: hbase-10504_v1.patch, hbase-10504_wip1.patch > > > HBase has replication. Fellas have been hijacking the replication apis to do > all kinds of perverse stuff like indexing hbase content (hbase-indexer > https://github.com/NGDATA/hbase-indexer) and our [~toffer] just showed up w/ > overrides that replicate via an alternate channel (over a secure thrift > channel between dcs over on HBASE-9360). This issue is about surfacing these > APIs as public with guarantees to downstreamers similar to those we have on > our public client-facing APIs (and so we don't break them for downstreamers). > Any input [~phunt] or [~gabriel.reid] or [~toffer]? > Thanks. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-10504) Define Replication Interface
[ https://issues.apache.org/jira/browse/HBASE-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-10504. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Marking as 'done' in 2.0.0. HBASE-18846 is the follow-on for hbase-indexer > Define Replication Interface > > > Key: HBASE-10504 > URL: https://issues.apache.org/jira/browse/HBASE-10504 > Project: HBase > Issue Type: Task > Components: Replication >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 2.0.0 > > Attachments: hbase-10504_v1.patch, hbase-10504_wip1.patch > > > HBase has replication. Fellas have been hijacking the replication apis to do > all kinds of perverse stuff like indexing hbase content (hbase-indexer > https://github.com/NGDATA/hbase-indexer) and our [~toffer] just showed up w/ > overrides that replicate via an alternate channel (over a secure thrift > channel between dcs over on HBASE-9360). This issue is about surfacing these > APIs as public with guarantees to downstreamers similar to those we have on > our public client-facing APIs (and so we don't break them for downstreamers). > Any input [~phunt] or [~gabriel.reid] or [~toffer]? > Thanks. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18846) Accommodate the hbase-indexer/lily/SEP consumer deploy-type
stack created HBASE-18846:
-
Summary: Accommodate the hbase-indexer/lily/SEP consumer
deploy-type
Key: HBASE-18846
URL: https://issues.apache.org/jira/browse/HBASE-18846
Project: HBase
Issue Type: Bug
Reporter: stack
This is a follow-on from HBASE-10504, Define a Replication Interface. There we
defined a new, flexible replication endpoint for others to implement but it did
little to help the case of the lily hbase-indexer. This issue takes up the case
of the hbase-indexer.
The hbase-indexer poses to hbase as a 'fake' peer cluster. The hbase-indexer
will start up cut-down "RegionServer" processes that are nought but an hbase
RpcServer hosting an AdminProtos Service. They make themselves 'appear' to the
Replication Source by hoisting up an ephemeral znode 'registering' as a
RegionServer. The source cluster then streams WALEdits to the Admin Protos
method:
{code}
public ReplicateWALEntryResponse replicateWALEntry(final RpcController
controller,
final ReplicateWALEntryRequest request) throws ServiceException {
{code}
The hbase-indexer relies on other hbase internals like Server so it can get a
ZooKeeperWatcher instance and know the 'name' to use for this cut-down server.
Thoughts on how to proceed include:
* Better formalize its current digestion of hbase internals; make it so
rpcserver is allowed to be used by others, etc.
* Start an actual RegionServer only have it register the AdminProtos Service
only -- not AdminProtos and ClientProtos, etc. Then have the hbase-indexer
implement an AdminCoprocessor to override the replicateWALEntry method (the
Admin CP implementation may need work).
I'll be back
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171136#comment-16171136
]
Hadoop QA commented on HBASE-14247:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
19s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
54s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 3s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
9s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m
58s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.13.1 Server=1.13.1 Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-14247 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12887812/HBASE-14247.master.002.patch
|
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 44eb49bf645b 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-10504) Define Replication Interface
[ https://issues.apache.org/jira/browse/HBASE-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171133#comment-16171133 ] stack commented on HBASE-10504: --- All subtasks in this issue are done. A Replication Interface has been implemented. Resolving. But first, on rereading the clean discussion above, this implementation has little for the lily/SEP/hbase-indexer "fake cluster" case. The lily-folks reasonably want to index in a separate process. They want to be able to scale up and down this consumer independent of RegionServer count and they don't want to have to implement our rpc, security or serde. In "Gabriel Reid added a comment - 11/Feb/14 23:59", he states that hbase-indexer makes use of o.a.h.h.replication.regionserver.ReplicationSource and o.a.h.h.Server, as well as the generated AdminProtos. He continued: " The one thing I can think of right now that would be nice is a better hook into o.a.h.h.replication.regionserver.ReplicationSource#removeNonReplicableEdits. Right now we override that method to do additional filtering on HLog entries that we know we don't want do send to the indexer" This latter asked-for-functionality, a sender-side filtering mechanism, was added in a subtask of this issue, HBASE-11367, where you can define implementations of WALEntryFilter when you define a ReplicationEndpoint (The current SEP has actually stopped filtering source-side because breakage was common). That leaves all the rest: the standing up of an hbase rpcserver instance that floats an Admin Service doing a hokey register of the RegionServer ephemeral znode up in zk. As suggested above, lets open a distinct issue for helping out the hbase-indexer case. > Define Replication Interface > > > Key: HBASE-10504 > URL: https://issues.apache.org/jira/browse/HBASE-10504 > Project: HBase > Issue Type: Task > Components: Replication >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Attachments: hbase-10504_v1.patch, hbase-10504_wip1.patch > > > HBase has replication. Fellas have been hijacking the replication apis to do > all kinds of perverse stuff like indexing hbase content (hbase-indexer > https://github.com/NGDATA/hbase-indexer) and our [~toffer] just showed up w/ > overrides that replicate via an alternate channel (over a secure thrift > channel between dcs over on HBASE-9360). This issue is about surfacing these > APIs as public with guarantees to downstreamers similar to those we have on > our public client-facing APIs (and so we don't break them for downstreamers). > Any input [~phunt] or [~gabriel.reid] or [~toffer]? > Thanks. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171125#comment-16171125
]
Hudson commented on HBASE-15931:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #289 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/289/])
HBASE-15931 Add log for long-running tasks in AsyncProcess (addendum) (liyu:
rev 025c3f47da5a297838a00e673b2bc6f4f4ff1e16)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-10504) Define Replication Interface
[ https://issues.apache.org/jira/browse/HBASE-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-10504: -- Release Note: Adds a new plugin point ReplicationEndpoint. ReplicationSource, internal to hbase, tails the WAL and calls registered ReplicationEndpoints. ReplicationEndpoint implementations are responsible for actually shipping the edits to the other (hbase or non-hbase) cluster. ReplicationEndpoint can be defined per peer. Default inter-cluster replication works without any changes (lily etc should still work). ReplicationEndpoints have various facility including means for filtering out WAL edits source-side before they can be shipped to remote peers. > Define Replication Interface > > > Key: HBASE-10504 > URL: https://issues.apache.org/jira/browse/HBASE-10504 > Project: HBase > Issue Type: Task > Components: Replication >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Attachments: hbase-10504_v1.patch, hbase-10504_wip1.patch > > > HBase has replication. Fellas have been hijacking the replication apis to do > all kinds of perverse stuff like indexing hbase content (hbase-indexer > https://github.com/NGDATA/hbase-indexer) and our [~toffer] just showed up w/ > overrides that replicate via an alternate channel (over a secure thrift > channel between dcs over on HBASE-9360). This issue is about surfacing these > APIs as public with guarantees to downstreamers similar to those we have on > our public client-facing APIs (and so we don't break them for downstreamers). > Any input [~phunt] or [~gabriel.reid] or [~toffer]? > Thanks. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171121#comment-16171121
]
Hudson commented on HBASE-15931:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #219 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/219/])
HBASE-15931 Add log for long-running tasks in AsyncProcess (addendum) (liyu:
rev da25def91b517ecc279444d8ab4dd943c117f61c)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171119#comment-16171119
]
Hudson commented on HBASE-15931:
SUCCESS: Integrated in Jenkins build HBase-1.4 #924 (See
[https://builds.apache.org/job/HBase-1.4/924/])
HBASE-15931 Add log for long-running tasks in AsyncProcess (addendum) (liyu:
rev a1d0bb97ac0a3384dab14e343c4394f09ac09eee)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171113#comment-16171113
]
Hudson commented on HBASE-15931:
FAILURE: Integrated in Jenkins build HBase-1.5 #69 (See
[https://builds.apache.org/job/HBase-1.5/69/])
HBASE-15931 Add log for long-running tasks in AsyncProcess (addendum) (liyu:
rev 539fce344048514606dd0c350985875774ec5e95)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171109#comment-16171109
]
Hudson commented on HBASE-15931:
SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1996 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1996/])
HBASE-15931 Add log for long-running tasks in AsyncProcess (addendum) (liyu:
rev 68cd18176e197c08294af622b849f2e7f9ea1fc7)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18795) Expose KeyValue.getBuffer() for tests alone
[ https://issues.apache.org/jira/browse/HBASE-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171108#comment-16171108 ] ramkrishna.s.vasudevan commented on HBASE-18795: I pushed this to master and branch-2 but I have not yet removed the @Deprecated tag in branch-1.4. If needed we can do it or leave it for now. > Expose KeyValue.getBuffer() for tests alone > --- > > Key: HBASE-18795 > URL: https://issues.apache.org/jira/browse/HBASE-18795 > Project: HBase > Issue Type: Improvement >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18795.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171106#comment-16171106
]
Hudson commented on HBASE-15931:
SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1912 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1912/])
HBASE-15931 Add log for long-running tasks in AsyncProcess (addendum) (liyu:
rev 68cd18176e197c08294af622b849f2e7f9ea1fc7)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[
https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171095#comment-16171095
]
Hadoop QA commented on HBASE-18090:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 8m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
7s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
36m 56s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 7s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
14s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.client.TestScanWithoutFetchingData |
| | org.apache.hadoop.hbase.coprocessor.TestHTableWrapper |
| | org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence |
| | org.apache.hadoop.hbase.client.TestAsyncTableScanAll |
| | org.apache.hadoop.hbase.client.TestAsyncReplicationAdminApi |
| | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestMultipleTimestamps |
| | org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries |
| |
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171085#comment-16171085
]
Yu Li commented on HBASE-15931:
---
Oh yes, didn't create new JIRA since the one-line change is minor... but will
notice in later ones for sure. Thanks for the reminder boss. [~ndimiduk]
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18829) Consider reverting HBASE-14893 Race between mutation on region and region closing operation
[ https://issues.apache.org/jira/browse/HBASE-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171080#comment-16171080 ] Vincent Poon commented on HBASE-18829: -- The lock is a non-fair mode lock, so in theory there are no guarantees as to when the writelock will be obtained when there are many readers contending for the lock as well. Setting this.closing(true) at least prevents other readers from trying to obtain the read lock, giving the writer eventual access to the lock. At least, that's my understanding of the intention from looking at the code. > Consider reverting HBASE-14893 Race between mutation on region and region > closing operation > --- > > Key: HBASE-18829 > URL: https://issues.apache.org/jira/browse/HBASE-18829 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 18829.v1.txt, HBASE-18829.patch > > > HBASE-14893 was brought to attention by [~rajeshbabu] over PHOENIX-3111. > This issue is to consider reverting the fix from HBASE-14893 based on the > following observations: > * The closing boolean was intended to be acquired before taking the lock > ([~enis]) > * Phoenix local index has evolved over the years, the situation leading to > NotServingRegionException may not exist from Phoenix side > * Even if the situation still exists, downstream project (Phoenix) should > properly handle NotServingRegionException without change in locking scheme in > hbase -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171075#comment-16171075
]
Nick Dimiduk commented on HBASE-15931:
--
[~carp84]: FYI, please don't commit patches for long-ago release JIRA numbers.
Anything that's gone out in a release is supposed to be considered archived,
permanently frozen in time, a part of the historical record. Addendums to such
tickets make it difficult to parse that record. This one is minor, so fine to
let it be. Just for future reference...
We used to have a policy about closing all associated tickets once a release
ships. Seems a RM or two have been remiss in this practice, including yours
truly. Worth a revival thread on the dev list.
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18807) Remove PB references from Observers for Quotas
[
https://issues.apache.org/jira/browse/HBASE-18807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171073#comment-16171073
]
Hadoop QA commented on HBASE-18807:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
2s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
27s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
27s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 27s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m
20s{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m
42s{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.2. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m
21s{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.3. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m
59s{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m
32s{color} | {color:red} The patch causes 22 errors with Hadoop v2.6.5. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m
1s{color} | {color:red} The patch causes 22 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m
30s{color} | {color:red} The patch causes 22 errors with Hadoop v2.7.2. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m
15s{color} | {color:red} The patch causes 22 errors with Hadoop v2.7.3. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 14m
0s{color} | {color:red} The patch causes 22 errors with Hadoop v3.0.0-alpha2.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
24s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
51s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 34s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} |
[jira] [Commented] (HBASE-18651) Let ChaosMonkeyRunner expose the chaos monkey runner it creates
[ https://issues.apache.org/jira/browse/HBASE-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171060#comment-16171060 ] Ted Yu commented on HBASE-18651: lgtm [~mdrob]: Can you take a look. > Let ChaosMonkeyRunner expose the chaos monkey runner it creates > --- > > Key: HBASE-18651 > URL: https://issues.apache.org/jira/browse/HBASE-18651 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Reid Chan > Attachments: HBASE-18651.master.001.patch, > HBASE-18651.master.002.patch, HBASE-18651.master.003.patch, > HBASE-18651.master.004.patch, HBASE-18651.master.005.patch > > > Currently ChaosMonkeyRunner#main() instantiates ChaosMonkeyRunner without > keeping track of the instance. > This poses some challenge when ChaosMonkeyRunner is used programmatically > because the caller cannot get hold of the runner. > As [~mdrob] suggested, we should expose the chaos monkey runner. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18610) Provide capability to activate chaos monkey
[
https://issues.apache.org/jira/browse/HBASE-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171054#comment-16171054
]
Ted Yu edited comment on HBASE-18610 at 9/19/17 3:15 AM:
-
Patch v8 handles the case where monkey is not used.
The second part from your comment indicated that non-remote hbase-site.xml was
used.
{code}
ChaosMonkeyRunner must run againt a distributed cluster
{code}
Can you check ?
was (Author: yuzhih...@gmail.com):
Patch v8 handles the case where monkey is not used.
The second part from your comment indicated that non-remote hbase-site.xml was
used.
Can you check ?
> Provide capability to activate chaos monkey
> ---
>
> Key: HBASE-18610
> URL: https://issues.apache.org/jira/browse/HBASE-18610
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
> Attachments: 18610.v1.txt, 18610.v4.txt, 18610.v5.txt, 18610.v6.txt,
> 18610.v7.txt, 18610.v8.txt
>
>
> Currently load-client runs against a cluster where region servers are stable.
> We need to introduce chaos monkey so that wider coverage for read path is
> exercised.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18651) Let ChaosMonkeyRunner expose the chaos monkey runner it creates
[ https://issues.apache.org/jira/browse/HBASE-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171056#comment-16171056 ] Reid Chan commented on HBASE-18651: --- ping [~tedyu], what do think about the latest patch. > Let ChaosMonkeyRunner expose the chaos monkey runner it creates > --- > > Key: HBASE-18651 > URL: https://issues.apache.org/jira/browse/HBASE-18651 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Reid Chan > Attachments: HBASE-18651.master.001.patch, > HBASE-18651.master.002.patch, HBASE-18651.master.003.patch, > HBASE-18651.master.004.patch, HBASE-18651.master.005.patch > > > Currently ChaosMonkeyRunner#main() instantiates ChaosMonkeyRunner without > keeping track of the instance. > This poses some challenge when ChaosMonkeyRunner is used programmatically > because the caller cannot get hold of the runner. > As [~mdrob] suggested, we should expose the chaos monkey runner. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18610) Provide capability to activate chaos monkey
[ https://issues.apache.org/jira/browse/HBASE-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171054#comment-16171054 ] Ted Yu commented on HBASE-18610: Patch v8 handles the case where monkey is not used. The second part from your comment indicated that non-remote hbase-site.xml was used. Can you check ? > Provide capability to activate chaos monkey > --- > > Key: HBASE-18610 > URL: https://issues.apache.org/jira/browse/HBASE-18610 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu > Attachments: 18610.v1.txt, 18610.v4.txt, 18610.v5.txt, 18610.v6.txt, > 18610.v7.txt, 18610.v8.txt > > > Currently load-client runs against a cluster where region servers are stable. > We need to introduce chaos monkey so that wider coverage for read path is > exercised. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18806) VerifyRep by snapshot need not to restore snapshot for each mapper
[
https://issues.apache.org/jira/browse/HBASE-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171049#comment-16171049
]
Ted Yu commented on HBASE-18806:
Please attach patch for QA run.
Have you tried it (or backport) on cluster ?
> VerifyRep by snapshot need not to restore snapshot for each mapper
> --
>
> Key: HBASE-18806
> URL: https://issues.apache.org/jira/browse/HBASE-18806
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0-alpha-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-18806.v1.patch, HBASE-18806.v2.patch
>
>
> In following method stack, seems like each mapper task will restore the
> snapshot. If we verify replication by a snapshot which has many hfiles,
> then we will take long time to restore snapshot. In our cluster, we took
> ~30min for the snapshot restoring when verify a big table.
> {code}
> Verifier.map
> |> replicatedScanner = new TableSnapshotScanner(...)
> |>
> TableSnapshotScanner.init()
>
> |-> RestoreSnapshotHelper.copySnapshotForScanner
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18807) Remove PB references from Observers for Quotas
[ https://issues.apache.org/jira/browse/HBASE-18807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18807: --- Attachment: HBASE-18807.001.branch-2.patch .001 Preliminary patch (wouldn't suggest reviewing unless interested in double-checking approach). Wanted to get an earlier QA run. Most of the quota-related tests are passing now, but there are some laggards. Need to clean up the code as a whole as well. The general approach is the same, but this turned out to be a rather large undertaking to untangle. > Remove PB references from Observers for Quotas > -- > > Key: HBASE-18807 > URL: https://issues.apache.org/jira/browse/HBASE-18807 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18807.001.branch-2.patch > > > Break-out from the parent: > Same idea, just applied to the Observer methods for pre/post quota > operations. Requires changes to MasterQuotaManager and the QuotaSettings > implementations as some business logic is written on the PB objects directly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18807) Remove PB references from Observers for Quotas
[ https://issues.apache.org/jira/browse/HBASE-18807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-18807: --- Status: Patch Available (was: Open) > Remove PB references from Observers for Quotas > -- > > Key: HBASE-18807 > URL: https://issues.apache.org/jira/browse/HBASE-18807 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18807.001.branch-2.patch > > > Break-out from the parent: > Same idea, just applied to the Observer methods for pre/post quota > operations. Requires changes to MasterQuotaManager and the QuotaSettings > implementations as some business logic is written on the PB objects directly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18829) Consider reverting HBASE-14893 Race between mutation on region and region closing operation
[ https://issues.apache.org/jira/browse/HBASE-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171044#comment-16171044 ] Allan Yang commented on HBASE-18829: I am still not get the point why should we revert? Isn't better we acquire the close lock first before we set the closing to true? > Consider reverting HBASE-14893 Race between mutation on region and region > closing operation > --- > > Key: HBASE-18829 > URL: https://issues.apache.org/jira/browse/HBASE-18829 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 18829.v1.txt, HBASE-18829.patch > > > HBASE-14893 was brought to attention by [~rajeshbabu] over PHOENIX-3111. > This issue is to consider reverting the fix from HBASE-14893 based on the > following observations: > * The closing boolean was intended to be acquired before taking the lock > ([~enis]) > * Phoenix local index has evolved over the years, the situation leading to > NotServingRegionException may not exist from Phoenix side > * Even if the situation still exists, downstream project (Phoenix) should > properly handle NotServingRegionException without change in locking scheme in > hbase -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18609) Apply ClusterStatus#getClusterStatus(EnumSet) in code base
[ https://issues.apache.org/jira/browse/HBASE-18609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171034#comment-16171034 ] Reid Chan commented on HBASE-18609: --- ping [~chia7712] if you have time.. > Apply ClusterStatus#getClusterStatus(EnumSet) in code base > -- > > Key: HBASE-18609 > URL: https://issues.apache.org/jira/browse/HBASE-18609 > Project: HBase > Issue Type: Improvement >Reporter: Chia-Ping Tsai >Assignee: Reid Chan > Fix For: 2.0.0 > > Attachments: HBASE-18609.master.001.patch, > HBASE-18609.master.002.patch, HBASE-18609.master.003.patch, > HBASE-18609.master.004.patch > > > HBASE-15511 enable us to get the cluster status by scope, and after > refactoring in HBASE-18621. We should apply it in code base so as to prevent > the useless information. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171033#comment-16171033 ] Xiang Li commented on HBASE-15482: -- Thanks [~jerryhe]. Got it. Just un-assign this JIRA. Joined the new company last week, still learning everything, hard to manage my time. Should be back soon. > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Priority: Minor > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171032#comment-16171032
]
Hudson commented on HBASE-15931:
SUCCESS: Integrated in Jenkins build HBase-1.2-IT #956 (See
[https://builds.apache.org/job/HBase-1.2-IT/956/])
HBASE-15931 Add log for long-running tasks in AsyncProcess (addendum) (liyu:
rev da25def91b517ecc279444d8ab4dd943c117f61c)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Li reassigned HBASE-15482: Assignee: (was: Xiang Li) > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Priority: Minor > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-14247:
---
Attachment: HBASE-14247.master.002.patch
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247.master.001.patch,
> HBASE-14247.master.002.patch, HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171022#comment-16171022
]
Hudson commented on HBASE-15931:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #223 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/223/])
HBASE-15931 Add log for long-running tasks in AsyncProcess (addendum) (liyu:
rev da25def91b517ecc279444d8ab4dd943c117f61c)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171017#comment-16171017
]
Hudson commented on HBASE-15931:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #208 (See
[https://builds.apache.org/job/HBase-1.3-IT/208/])
HBASE-15931 Add log for long-running tasks in AsyncProcess (addendum) (liyu:
rev 025c3f47da5a297838a00e673b2bc6f4f4ff1e16)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171016#comment-16171016
]
Hudson commented on HBASE-15931:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #280 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/280/])
HBASE-15931 Add log for long-running tasks in AsyncProcess (addendum) (liyu:
rev 025c3f47da5a297838a00e673b2bc6f4f4ff1e16)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18806) VerifyRep by snapshot need not to restore snapshot for each mapper
[
https://issues.apache.org/jira/browse/HBASE-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171015#comment-16171015
]
Zheng Hu commented on HBASE-18806:
--
Hi [~tedyu], Thanks for the reviewing .. Any other concerns for patch v2 ?
> VerifyRep by snapshot need not to restore snapshot for each mapper
> --
>
> Key: HBASE-18806
> URL: https://issues.apache.org/jira/browse/HBASE-18806
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.0.0-alpha-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-18806.v1.patch, HBASE-18806.v2.patch
>
>
> In following method stack, seems like each mapper task will restore the
> snapshot. If we verify replication by a snapshot which has many hfiles,
> then we will take long time to restore snapshot. In our cluster, we took
> ~30min for the snapshot restoring when verify a big table.
> {code}
> Verifier.map
> |> replicatedScanner = new TableSnapshotScanner(...)
> |>
> TableSnapshotScanner.init()
>
> |-> RestoreSnapshotHelper.copySnapshotForScanner
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171010#comment-16171010
]
Yu Li commented on HBASE-15931:
---
Pushed addendum to all 1.1+ branches.
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15931) Add log for long-running tasks in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171005#comment-16171005
]
Yu Li commented on HBASE-15931:
---
Sorry for the late response, just back from vacation. Thanks for pointing this
out [~Lijian Shen], will make an addendum and push to all relative branches
soon.
> Add log for long-running tasks in AsyncProcess
> --
>
> Key: HBASE-15931
> URL: https://issues.apache.org/jira/browse/HBASE-15931
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20, 1.1.6, 1.2.3
>
> Attachments: HBASE-15931.patch
>
>
> Currently if there's any long-tail tasks in a multi-action request like
> triggered by {{BufferedMutatorImpl#backgroundFlushCommits}}, we could see
> logging message like below
> {noformat}
> 2016-05-31 09:36:55,461 INFO [Thread-16]
> org.apache.hadoop.hbase.client.AsyncProcess: #28, waiting for some tasks to
> finish. Expected max=0, tasksInProgress=1
> {noformat}
> but there's no way to know detail of this long-tail, such as which
> table/region it's accessing. This JIRA aims at adding such log for easier
> debugging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18825) Use HStoreFile instead of StoreFile in our own code base and remove unnecessary methods in StoreFile interface
[ https://issues.apache.org/jira/browse/HBASE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171003#comment-16171003 ] Duo Zhang commented on HBASE-18825: --- [~anoopsamjohn] [~stack] PTAL. Thanks. > Use HStoreFile instead of StoreFile in our own code base and remove > unnecessary methods in StoreFile interface > -- > > Key: HBASE-18825 > URL: https://issues.apache.org/jira/browse/HBASE-18825 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18825.patch, HBASE-18825-v1.patch, > HBASE-18825-v2.patch, HBASE-18825-v3.patch > > > Use generic types to avoid too many casts. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18825) Use HStoreFile instead of StoreFile in our own code base and remove unnecessary methods in StoreFile interface
[ https://issues.apache.org/jira/browse/HBASE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18825: -- Attachment: HBASE-18825-v3.patch Fix failed UTs and javadoc warnings. > Use HStoreFile instead of StoreFile in our own code base and remove > unnecessary methods in StoreFile interface > -- > > Key: HBASE-18825 > URL: https://issues.apache.org/jira/browse/HBASE-18825 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18825.patch, HBASE-18825-v1.patch, > HBASE-18825-v2.patch, HBASE-18825-v3.patch > > > Use generic types to avoid too many casts. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170991#comment-16170991 ] xinxin fan commented on HBASE-18090: Thanks for review, new patch with some tests for the new methods in the RegionSplitter has uploaded. > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch, > HBASE-18090-branch-1.3-v2.patch, HBASE-18090-V3-master.patch, > HBASE-18090-V4-master.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xinxin fan updated HBASE-18090: --- Attachment: HBASE-18090-V4-master.patch > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov >Assignee: xinxin fan > Attachments: HBASE-18090-branch-1.3-v1.patch, > HBASE-18090-branch-1.3-v2.patch, HBASE-18090-V3-master.patch, > HBASE-18090-V4-master.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18809) [C++] Templatize AsyncBatchRpcRetryingCaller Class
[
https://issues.apache.org/jira/browse/HBASE-18809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170989#comment-16170989
]
Enis Soztutar commented on HBASE-18809:
---
Thanks Sudeep.
Instead of {{std::shared_ptr}}, can we use {{folly::Unit}}? Is
shared_ptr wrapper necessary for types elsewhere?
{code}
+ return this->Batch(
{code}
Why did {{ResponseConverter::GetResults}} move to the .h file? Also there is
code duplication here. Can we do abstract the part that is common, and only do
the changed parts?
> [C++] Templatize AsyncBatchRpcRetryingCaller Class
> --
>
> Key: HBASE-18809
> URL: https://issues.apache.org/jira/browse/HBASE-18809
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-18809.HBASE-14850.v1.patch
>
>
> Objective is to templatize AsyncBatchRpcRetryingCaller class to support Get,
> Put, Delete etc request types and Result, Void response types.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18845) TestReplicationSmallTests fails after HBASE-14004
Duo Zhang created HBASE-18845: - Summary: TestReplicationSmallTests fails after HBASE-14004 Key: HBASE-18845 URL: https://issues.apache.org/jira/browse/HBASE-18845 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 2.0.0-alpha-3, 3.0.0 Reporter: Duo Zhang Fix For: 3.0.0, 2.0.0-alpha-4 testEmptyWALRecovery and testVerifyRepJob -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18786) FileNotFoundException should not be silently handled for primary region replicas
[ https://issues.apache.org/jira/browse/HBASE-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18786: --- Attachment: HBASE-18786-branch-1.patch Local tests seem ok, at least branch-1 > FileNotFoundException should not be silently handled for primary region > replicas > > > Key: HBASE-18786 > URL: https://issues.apache.org/jira/browse/HBASE-18786 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Ashu Pachauri >Assignee: Andrew Purtell > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.3.2, 1.5.0 > > Attachments: HBASE-18786-branch-1.patch, HBASE-18786.patch > > > This is a follow up for HBASE-18186. > FileNotFoundException while scanning from a primary region replica can be > indicative of a more severe problem. Handling them silently can cause many > underlying issues go undetected. We should either > 1. Hard fail the regionserver if there is a FNFE on a primary region replica, > OR > 2. Report these exceptions as some region / server level metric so that these > can be proactively investigated. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18786) FileNotFoundException should not be silently handled for primary region replicas
[ https://issues.apache.org/jira/browse/HBASE-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18786: --- Fix Version/s: 1.5.0 1.4.0 3.0.0 2.0.0 Status: Patch Available (was: Open) > FileNotFoundException should not be silently handled for primary region > replicas > > > Key: HBASE-18786 > URL: https://issues.apache.org/jira/browse/HBASE-18786 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Ashu Pachauri >Assignee: Andrew Purtell > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.3.2, 1.5.0 > > Attachments: HBASE-18786-branch-1.patch, HBASE-18786.patch > > > This is a follow up for HBASE-18186. > FileNotFoundException while scanning from a primary region replica can be > indicative of a more severe problem. Handling them silently can cause many > underlying issues go undetected. We should either > 1. Hard fail the regionserver if there is a FNFE on a primary region replica, > OR > 2. Report these exceptions as some region / server level metric so that these > can be proactively investigated. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-14004) [Replication] Inconsistency between Memstore and WAL may result in data in remote cluster that is not in the origin
[ https://issues.apache.org/jira/browse/HBASE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170982#comment-16170982 ] Duo Zhang commented on HBASE-14004: --- OK, will check later. > [Replication] Inconsistency between Memstore and WAL may result in data in > remote cluster that is not in the origin > --- > > Key: HBASE-14004 > URL: https://issues.apache.org/jira/browse/HBASE-14004 > Project: HBase > Issue Type: Bug > Components: regionserver, Replication >Reporter: He Liangliang >Assignee: Duo Zhang >Priority: Critical > Labels: replication, wal > Fix For: 3.0.0, 2.0.0-alpha-4 > > Attachments: HBASE-14004.patch, HBASE-14004-v1.patch, > HBASE-14004-v2.patch, HBASE-14004-v2.patch, HBASE-14004-v3.patch > > > Looks like the current write path can cause inconsistency between > memstore/hfile and WAL which cause the slave cluster has more data than the > master cluster. > The simplified write path looks like: > 1. insert record into Memstore > 2. write record to WAL > 3. sync WAL > 4. rollback Memstore if 3 fails > It's possible that the HDFS sync RPC call fails, but the data is already > (may partially) transported to the DNs which finally get persisted. As a > result, the handler will rollback the Memstore and the later flushed HFile > will also skip this record. > == > This is a long lived issue. The above problem is solved by write path > reorder, as now we will sync wal first before modifying memstore. But the > problem may still exists as replication thread may read the new data before > we return from hflush. See this document for more details: > https://docs.google.com/document/d/11AyWtGhItQs6vsLRIx32PwTxmBY3libXwGXI25obVEY/edit# > So we need to keep a sync length in WAL and tell replication wal reader this > is limit when you read this wal file. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18610) Provide capability to activate chaos monkey
[ https://issues.apache.org/jira/browse/HBASE-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170980#comment-16170980 ] Enis Soztutar commented on HBASE-18610: --- Ted, are you gonna update the patch? > Provide capability to activate chaos monkey > --- > > Key: HBASE-18610 > URL: https://issues.apache.org/jira/browse/HBASE-18610 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu > Attachments: 18610.v1.txt, 18610.v4.txt, 18610.v5.txt, 18610.v6.txt, > 18610.v7.txt, 18610.v8.txt > > > Currently load-client runs against a cluster where region servers are stable. > We need to introduce chaos monkey so that wider coverage for read path is > exercised. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-13346) Clean up Filter package for post 1.0 s/KeyValue/Cell/g
[
https://issues.apache.org/jira/browse/HBASE-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170973#comment-16170973
]
Hadoop QA commented on HBASE-13346:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
44s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 27s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
17s{color} | {color:red} hbase-client generated 2 new + 2 unchanged - 0 fixed =
4 total (was 2) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
41s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 11s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
13s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
32s{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.security.access.TestCellACLWithMultipleVersions |
| | hadoop.hbase.security.access.TestScanEarlyTermination |
| |
hadoop.hbase.security.visibility.TestVisibilityLabelsWithCustomVisLabService |
| | hadoop.hbase.security.visibility.TestEnforcingScanLabelGenerator |
| | hadoop.hbase.client.TestRawAsyncScanCursor |
[jira] [Commented] (HBASE-18713) Fix build through make command
[
https://issues.apache.org/jira/browse/HBASE-18713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170972#comment-16170972
]
Enis Soztutar commented on HBASE-18713:
---
bq. Updated to commit 21c08df9079b8cc9dd1c8c2208a7604fc12fca12 .
I was not talking about updating the source. You should update the libraries
folly and wangle. If you are using docker, that means that you need to rebuild
the image.
> Fix build through make command
> --
>
> Key: HBASE-18713
> URL: https://issues.apache.org/jira/browse/HBASE-18713
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>
> make all' resulted in build error using docker VM in HBASE-14850 branch:
> {code}
> connection/rpc-client.cc: In member function
> 'folly::Future
> hbase::RpcClient::SendRequest(std::shared_ptr,
> std::unique_ptr)':
> connection/rpc-client.cc:95:21: error: 'class folly::exception_wrapper' has
> no member named 'get_exception'
> CHECK(e.cause().get_exception() != nullptr);
> ^
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18832) LTT fails with casting exception for HColumnDescriptor
[
https://issues.apache.org/jira/browse/HBASE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170938#comment-16170938
]
Hudson commented on HBASE-18832:
FAILURE: Integrated in Jenkins build HBase-2.0 #536 (See
[https://builds.apache.org/job/HBase-2.0/536/])
HBASE-18832 Fixed LTT, removed references to deprecated (stack: rev
00d9e38479890bb58d2aafd9b373458e0650e4f4)
* (edit)
hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
> LTT fails with casting exception for HColumnDescriptor
> --
>
> Key: HBASE-18832
> URL: https://issues.apache.org/jira/browse/HBASE-18832
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha-4
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Fix For: 2.0.0-alpha-4
>
> Attachments: hbase-18832.master.001.patch
>
>
> Here is the stack trace:
> {code}
> 2017-09-15 12:38:38,158 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> java.lang.ClassCastException:
> org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder$ModifyableColumnFamilyDescriptor
> cannot be cast to org.apache.hadoop.hbase.HColumnDescriptor
> at
> org.apache.hadoop.hbase.util.LoadTestTool.applyColumnFamilyOptions(LoadTestTool.java:265)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.initTestTable(LoadTestTool.java:540)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.loadTable(LoadTestTool.java:567)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:548)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:262)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:793)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18803) Mapreduce job get failed caused by NoClassDefFoundError: org/apache/commons/lang3/ArrayUtils
[ https://issues.apache.org/jira/browse/HBASE-18803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170933#comment-16170933 ] Hudson commented on HBASE-18803: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3737 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3737/]) HBASE-18803: Mapreduce job get failed caused by NoClassDefFoundError: (stack: rev e10d7836ed420754fcf13e3ab7ea1c1c2f1c2098) * (edit) hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java > Mapreduce job get failed caused by NoClassDefFoundError: > org/apache/commons/lang3/ArrayUtils > > > Key: HBASE-18803 > URL: https://issues.apache.org/jira/browse/HBASE-18803 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-2 >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18803-v1-master.patch > > > 2017-09-12 17:44:13,020 INFO [main] mapreduce.Job: Task Id : > attempt_1505170797673_0010_m_00_0, Status : FAILED > Error: java.io.IOException: java.lang.reflect.InvocationTargetException > at > org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:221) > at > org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:114) > at > org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:311) > at > org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:264) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:219) > ... 11 more > Caused by: java.lang.NoClassDefFoundError: org/apache/commons/lang3/ArrayUtils > at org.apache.hadoop.hbase.HConstants.(HConstants.java:1144) > at > org.apache.hadoop.hbase.client.ConnectionUtils.getPauseTime(ConnectionUtils.java:84) > at > org.apache.hadoop.hbase.client.AsyncProcess.(AsyncProcess.java:209) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:266) > ... 16 more -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18832) LTT fails with casting exception for HColumnDescriptor
[
https://issues.apache.org/jira/browse/HBASE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170934#comment-16170934
]
Hudson commented on HBASE-18832:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3737 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3737/])
HBASE-18832 Fixed LTT, removed references to deprecated (stack: rev
34d648808d262b48bd4986e99a712ceeb8252cfa)
* (edit)
hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
> LTT fails with casting exception for HColumnDescriptor
> --
>
> Key: HBASE-18832
> URL: https://issues.apache.org/jira/browse/HBASE-18832
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha-4
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Fix For: 2.0.0-alpha-4
>
> Attachments: hbase-18832.master.001.patch
>
>
> Here is the stack trace:
> {code}
> 2017-09-15 12:38:38,158 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> java.lang.ClassCastException:
> org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder$ModifyableColumnFamilyDescriptor
> cannot be cast to org.apache.hadoop.hbase.HColumnDescriptor
> at
> org.apache.hadoop.hbase.util.LoadTestTool.applyColumnFamilyOptions(LoadTestTool.java:265)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.initTestTable(LoadTestTool.java:540)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.loadTable(LoadTestTool.java:567)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:548)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:262)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:793)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18844) Release hbase-2.0.0-alpha-4; Theme "Coprocessor API Cleanup"
[ https://issues.apache.org/jira/browse/HBASE-18844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170928#comment-16170928 ] stack commented on HBASE-18844: --- Review the alpha-3 release. It could do w/ a bit of scripting and fixup of the refguide RC section. > Release hbase-2.0.0-alpha-4; Theme "Coprocessor API Cleanup" > > > Key: HBASE-18844 > URL: https://issues.apache.org/jira/browse/HBASE-18844 > Project: HBase > Issue Type: Bug >Reporter: stack > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-18767) Release hbase-2.0.0-alpha-3; Theme "Scrubbed API"
[ https://issues.apache.org/jira/browse/HBASE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-18767. --- Resolution: Fixed Assignee: stack Fix Version/s: (was: 2.0.0-alpha-4) 2.0.0-alpha-3 Release 2.0.0-alpha-3. > Release hbase-2.0.0-alpha-3; Theme "Scrubbed API" > - > > Key: HBASE-18767 > URL: https://issues.apache.org/jira/browse/HBASE-18767 > Project: HBase > Issue Type: Task >Reporter: stack >Assignee: stack > Fix For: 2.0.0-alpha-3 > > > From the dev mail thread: "Moving 2.0 forward", the theme is solidifying API. > I listed a bunch of API JIRAs to address. [~mdrob] nicely tagged them all w/ > the 2.0.0-alpha-3 fix version. This issue is for pushing out alpha-3. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18844) Release hbase-2.0.0-alpha-4; Theme "Coprocessor API Cleanup"
stack created HBASE-18844: - Summary: Release hbase-2.0.0-alpha-4; Theme "Coprocessor API Cleanup" Key: HBASE-18844 URL: https://issues.apache.org/jira/browse/HBASE-18844 Project: HBase Issue Type: Bug Reporter: stack -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18652) Expose individual cache stats in a CombinedCache through JMX
[ https://issues.apache.org/jira/browse/HBASE-18652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18652: -- Fix Version/s: (was: 2.0.0-alpha-3) 2.0.0-alpha-4 > Expose individual cache stats in a CombinedCache through JMX > > > Key: HBASE-18652 > URL: https://issues.apache.org/jira/browse/HBASE-18652 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Biju Nair >Assignee: Biju Nair > Fix For: 1.4.0, 1.5.0, 2.0.0-alpha-4 > > Attachments: 18652-branch-1-V3.0.PATCH, HBASE-18652-addendum.patch, > HBASE-18652-BRANCH-1.PATCH, HBASE-18652-branch-1-v1.0.patch, > HBASE-18652-branch-1-V2.0.PATCH, HBASE-18652-branch-1-V3.0.PATCH, > HBASE-18652-branch-1-V3.0.PATCH, HBASE-18652.PATCH, HBASE-18652-V1.0.PATCH, > HBASE-18652-V2.0.PATCH, HBASE-18652-WIP.PATCH > > > With offHeap cache being used to store data blocks and on-heap for index and > bloom filters, exposing the stats from the individual caches through JMX will > help understand the cache usage trend. Currently the combined cache stats is > available through JMX which may not provide insight into the individual cache > usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18410) FilterList Improvement.
[ https://issues.apache.org/jira/browse/HBASE-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18410: -- Fix Version/s: (was: 2.0.0-alpha-3) 2.0.0-alpha-4 > FilterList Improvement. > - > > Key: HBASE-18410 > URL: https://issues.apache.org/jira/browse/HBASE-18410 > Project: HBase > Issue Type: Umbrella >Reporter: Zheng Hu >Assignee: Zheng Hu > Fix For: 2.0.0-alpha-4 > > > FilterList.java is complex now, and we have found some improvements for it. > So create an issue to address it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-15666) shaded dependencies for hbase-testing-util
[ https://issues.apache.org/jira/browse/HBASE-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15666: -- Fix Version/s: (was: 2.0.0-alpha-3) 2.0.0-alpha-4 > shaded dependencies for hbase-testing-util > -- > > Key: HBASE-15666 > URL: https://issues.apache.org/jira/browse/HBASE-15666 > Project: HBase > Issue Type: New Feature > Components: test >Affects Versions: 1.1.0, 1.2.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 1.5.0, 2.0.0-alpha-4 > > > Folks that make use of our shaded client but then want to test things using > the hbase-testing-util end up getting all of our dependencies again in the > test scope. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18596) A hbase1 cluster should be able to replicate to a hbase2 cluster; verify
[ https://issues.apache.org/jira/browse/HBASE-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18596: -- Fix Version/s: (was: 2.0.0-alpha-3) 2.0.0-alpha-4 > A hbase1 cluster should be able to replicate to a hbase2 cluster; verify > > > Key: HBASE-18596 > URL: https://issues.apache.org/jira/browse/HBASE-18596 > Project: HBase > Issue Type: Task >Reporter: stack >Assignee: Esteban Gutierrez >Priority: Blocker > Fix For: 2.0.0-alpha-4 > > > From the mailing list thread "[DISCUSS] hbase-2.0.0 compatibility > expectations", [~esteban] asks: > bq. Should we add additional details around replication as well? for > instance, shall we consider a hbase-1.x cluster as a client for a hbase-2.x > cluster? > The latter should be a blocker. Verify it works. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18797) Deprecate Filter#filterKeyValue and add Filter#filterCell
[ https://issues.apache.org/jira/browse/HBASE-18797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18797: -- Fix Version/s: (was: 2.0.0-alpha-3) 2.0.0-alpha-4 > Deprecate Filter#filterKeyValue and add Filter#filterCell > - > > Key: HBASE-18797 > URL: https://issues.apache.org/jira/browse/HBASE-18797 > Project: HBase > Issue Type: Sub-task > Components: API, Filters >Reporter: Abhishek Singh Chouhan >Assignee: Tamas Penzes >Priority: Critical > Fix For: 2.0.0-alpha-4 > > > Part of filter package cleanup. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18767) Release hbase-2.0.0-alpha-3; Theme "Scrubbed API"
[ https://issues.apache.org/jira/browse/HBASE-18767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18767: -- Fix Version/s: (was: 2.0.0-alpha-3) 2.0.0-alpha-4 > Release hbase-2.0.0-alpha-3; Theme "Scrubbed API" > - > > Key: HBASE-18767 > URL: https://issues.apache.org/jira/browse/HBASE-18767 > Project: HBase > Issue Type: Task >Reporter: stack > Fix For: 2.0.0-alpha-4 > > > From the dev mail thread: "Moving 2.0 forward", the theme is solidifying API. > I listed a bunch of API JIRAs to address. [~mdrob] nicely tagged them all w/ > the 2.0.0-alpha-3 fix version. This issue is for pushing out alpha-3. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18731) [compat 1-2] QuotaSettings#setupSetQuotaRequest param changed
[ https://issues.apache.org/jira/browse/HBASE-18731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18731: -- Fix Version/s: (was: 2.0.0-alpha-3) 2.0.0-alpha-4 > [compat 1-2] QuotaSettings#setupSetQuotaRequest param changed > - > > Key: HBASE-18731 > URL: https://issues.apache.org/jira/browse/HBASE-18731 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: stack >Assignee: Sean Busbey > Fix For: 2.0.0-alpha-4 > > > QuotaSettings is Audience public. The param for setupSetQuotaRequest moved to > package > org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.SetQuotaRequest. > QuotaSettings was added in 1.1. > Is QuotaSettings for use by anyone but our CPEP? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18763) quotas.TestQuotaThrottle failing on all nightly runs
[
https://issues.apache.org/jira/browse/HBASE-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-18763:
--
Fix Version/s: (was: 2.0.0-alpha-3)
2.0.0-alpha-4
> quotas.TestQuotaThrottle failing on all nightly runs
>
>
> Key: HBASE-18763
> URL: https://issues.apache.org/jira/browse/HBASE-18763
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-4
>
>
> Recent flaky analysis has this test at 100% failure over 30 runs:
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/10803/artifact/dashboard.html
> {code}
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testUserAndTableThrottle
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testNamespaceGlobalReadAndWriteThrottle
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testUserGlobalThrottle
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testUserTableThrottle
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testTableGlobalThrottle
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testTableGlobalReadAndWriteThrottle
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testUserGlobalReadAndWriteThrottle
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testUserGlobalBypassThrottle
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testUserNamespaceThrottle
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testNamespaceGlobalThrottle
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testUserTableReadAndWriteThrottle
> org.apache.hadoop.hbase.quotas.TestQuotaThrottle.testUserNamespaceReadAndWriteThrottle
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18795) Expose KeyValue.getBuffer() for tests alone
[ https://issues.apache.org/jira/browse/HBASE-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18795: -- Fix Version/s: (was: 2.0.0-alpha-3) 2.0.0-alpha-4 > Expose KeyValue.getBuffer() for tests alone > --- > > Key: HBASE-18795 > URL: https://issues.apache.org/jira/browse/HBASE-18795 > Project: HBase > Issue Type: Improvement >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18795.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18601) Update Htrace to 4.2
[ https://issues.apache.org/jira/browse/HBASE-18601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18601: -- Fix Version/s: (was: 2.0.0-alpha-3) 2.0.0-alpha-4 > Update Htrace to 4.2 > > > Key: HBASE-18601 > URL: https://issues.apache.org/jira/browse/HBASE-18601 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0, 3.0.0 >Reporter: Tamas Penzes >Assignee: Tamas Penzes > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18601.master.001.patch, > HBASE-18601.master.002.patch, HBASE-18601.master.003 (3).patch, > HBASE-18601.master.003.patch, HBASE-18601.master.004.patch, > HBASE-18601.master.004.patch > > > HTrace is not perfectly integrated into HBase, the version 3.2.0 is buggy, > the upgrade to 4.x is not trivial and would take time. It might not worth to > keep it in this state, so would be better to remove it. > Of course it doesn't mean tracing would be useless, just that in this form > the use of HTrace 3.2 might not add any value to the project and fixing it > would be far too much effort. > - > Based on the decision of the community we keep htrace now and update version -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18758) [compat 1-2] Test delegation tokens continue to work when hbase1 going against hbase2 cluster
[ https://issues.apache.org/jira/browse/HBASE-18758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18758: -- Fix Version/s: (was: 2.0.0-alpha-3) 2.0.0-alpha-4 > [compat 1-2] Test delegation tokens continue to work when hbase1 going > against hbase2 cluster > - > > Key: HBASE-18758 > URL: https://issues.apache.org/jira/browse/HBASE-18758 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: stack >Priority: Critical > Fix For: 2.0.0-alpha-4 > > > Suggested by [~toffer] Need to test. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18601) Update Htrace to 4.2
[ https://issues.apache.org/jira/browse/HBASE-18601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18601: -- Attachment: HBASE-18601.master.004.patch Retry > Update Htrace to 4.2 > > > Key: HBASE-18601 > URL: https://issues.apache.org/jira/browse/HBASE-18601 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0, 3.0.0 >Reporter: Tamas Penzes >Assignee: Tamas Penzes > Fix For: 2.0.0-alpha-3 > > Attachments: HBASE-18601.master.001.patch, > HBASE-18601.master.002.patch, HBASE-18601.master.003 (3).patch, > HBASE-18601.master.003.patch, HBASE-18601.master.004.patch, > HBASE-18601.master.004.patch > > > HTrace is not perfectly integrated into HBase, the version 3.2.0 is buggy, > the upgrade to 4.x is not trivial and would take time. It might not worth to > keep it in this state, so would be better to remove it. > Of course it doesn't mean tracing would be useless, just that in this form > the use of HTrace 3.2 might not add any value to the project and fixing it > would be far too much effort. > - > Based on the decision of the community we keep htrace now and update version -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18601) Update Htrace to 4.2
[
https://issues.apache.org/jira/browse/HBASE-18601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170915#comment-16170915
]
Hadoop QA commented on HBASE-18601:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 8m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
44s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hbase-testing-util . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 15m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
14s{color} | {color:red} hbase-rest in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
36s{color} | {color:red} hbase-spark in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} rubocop {color} | {color:green} 0m
5s{color} | {color:green} There were no new rubocop issues. {color} |
| {color:green}+1{color} | {color:green} ruby-lint {color} | {color:green} 0m
1s{color} | {color:green} There were no new ruby-lint issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
25s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
23s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m
30s{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
31s{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.2. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m
35s{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.3. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m
36s{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m
38s{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.5. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hbase-testing-util . {color} |
| {color:red}-1{color} |
[jira] [Updated] (HBASE-15410) Utilize the max seek value when all Filters in MUST_PASS_ALL FilterList return SEEK_NEXT_USING_HINT
[ https://issues.apache.org/jira/browse/HBASE-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15410: --- Fix Version/s: 1.5.0 > Utilize the max seek value when all Filters in MUST_PASS_ALL FilterList > return SEEK_NEXT_USING_HINT > --- > > Key: HBASE-15410 > URL: https://issues.apache.org/jira/browse/HBASE-15410 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Labels: filter, perfomance > Fix For: 1.5.0, 2.0.0-alpha-3 > > Attachments: 15410.branch-1.patch, 15410.v1.patch, 15410.v2.patch, > 15410-wip.patch > > > As Preston mentioned in the comment in HBASE-15243: > https://issues.apache.org/jira/browse/HBASE-15243?focusedCommentId=15143557=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15143557 > An optimization for filters returning a SEEK_NEXT_USING_HINT would be to seek > to the highest hint (Any previous/lower row won't be accepted by the filter > returning that seek). > This JIRA is to explore this potential optimization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15410) Utilize the max seek value when all Filters in MUST_PASS_ALL FilterList return SEEK_NEXT_USING_HINT
[ https://issues.apache.org/jira/browse/HBASE-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170909#comment-16170909 ] Hudson commented on HBASE-15410: FAILURE: Integrated in Jenkins build HBase-1.5 #68 (See [https://builds.apache.org/job/HBase-1.5/68/]) HBASE-15410 Utilize the max seek value when all Filters in MUST_PASS_ALL (tedyu: rev ead106324474d2faff11b4e1779e575aec67b59c) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java > Utilize the max seek value when all Filters in MUST_PASS_ALL FilterList > return SEEK_NEXT_USING_HINT > --- > > Key: HBASE-15410 > URL: https://issues.apache.org/jira/browse/HBASE-15410 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Labels: filter, perfomance > Fix For: 2.0.0-alpha-3 > > Attachments: 15410.branch-1.patch, 15410.v1.patch, 15410.v2.patch, > 15410-wip.patch > > > As Preston mentioned in the comment in HBASE-15243: > https://issues.apache.org/jira/browse/HBASE-15243?focusedCommentId=15143557=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15143557 > An optimization for filters returning a SEEK_NEXT_USING_HINT would be to seek > to the highest hint (Any previous/lower row won't be accepted by the filter > returning that seek). > This JIRA is to explore this potential optimization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18624) Added support for clearing BlockCache
[ https://issues.apache.org/jira/browse/HBASE-18624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Jadhav updated HBASE-18624: Attachment: HBASE-18624.master.001.patch > Added support for clearing BlockCache > - > > Key: HBASE-18624 > URL: https://issues.apache.org/jira/browse/HBASE-18624 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0, 1.3.0 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav > Fix For: 1.4.0 > > Attachments: HBASE-18624.branch-1.001.patch, > HBASE-18624.master.001.patch > > > Bulk loading the primary HBase cluster triggers a lot of compactions > resulting in archival/ creation > of multiple HFiles. This process will cause a lot of items to become stale in > replica’s BlockCache. > This patch will help users to clear the block cache for a given table by > either using shell or API. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18713) Fix build through make command
[
https://issues.apache.org/jira/browse/HBASE-18713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170883#comment-16170883
]
Ted Yu commented on HBASE-18713:
Updated to commit 21c08df9079b8cc9dd1c8c2208a7604fc12fca12 .
Following the commands given above, I still got:
{code}
In file included from ./folly/Range.h:33:0,
from ./folly/Conv.h:29,
from ./folly/String.h:36,
from ./folly/ExceptionWrapper.h:22,
from include/hbase/connection/rpc-client.h:23,
from src/hbase/connection/rpc-client.cc:20:
src/hbase/connection/rpc-client.cc: In member function
'folly::Future
hbase::RpcClient::SendRequest(std::shared_ptr,
std::unique_ptr)':
src/hbase/connection/rpc-client.cc:95:21: error: 'class
folly::exception_wrapper' has no member named 'get_exception'
{code}
> Fix build through make command
> --
>
> Key: HBASE-18713
> URL: https://issues.apache.org/jira/browse/HBASE-18713
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>
> make all' resulted in build error using docker VM in HBASE-14850 branch:
> {code}
> connection/rpc-client.cc: In member function
> 'folly::Future
> hbase::RpcClient::SendRequest(std::shared_ptr,
> std::unique_ptr)':
> connection/rpc-client.cc:95:21: error: 'class folly::exception_wrapper' has
> no member named 'get_exception'
> CHECK(e.cause().get_exception() != nullptr);
> ^
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18713) Fix build through make command
[
https://issues.apache.org/jira/browse/HBASE-18713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170869#comment-16170869
]
Enis Soztutar commented on HBASE-18713:
---
[~ted_yu] I think you need to update your wangle and folly dependency versions
due to HBASE-18720. {{make clean}} and {{make}} works for me.
> Fix build through make command
> --
>
> Key: HBASE-18713
> URL: https://issues.apache.org/jira/browse/HBASE-18713
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>
> make all' resulted in build error using docker VM in HBASE-14850 branch:
> {code}
> connection/rpc-client.cc: In member function
> 'folly::Future
> hbase::RpcClient::SendRequest(std::shared_ptr,
> std::unique_ptr)':
> connection/rpc-client.cc:95:21: error: 'class folly::exception_wrapper' has
> no member named 'get_exception'
> CHECK(e.cause().get_exception() != nullptr);
> ^
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18797) Deprecate Filter#filterKeyValue and add Filter#filterCell
[ https://issues.apache.org/jira/browse/HBASE-18797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170852#comment-16170852 ] Tamas Penzes commented on HBASE-18797: -- [~abhishek.chouhan], so I wrote a quite long comment about why you are wrong. During this activity I recognised what you tried to explain to me and so how wrong I was. Now I see your point. Thanks. Please check my newest version. > Deprecate Filter#filterKeyValue and add Filter#filterCell > - > > Key: HBASE-18797 > URL: https://issues.apache.org/jira/browse/HBASE-18797 > Project: HBase > Issue Type: Sub-task > Components: API, Filters >Reporter: Abhishek Singh Chouhan >Assignee: Tamas Penzes >Priority: Critical > Fix For: 2.0.0-alpha-3 > > > Part of filter package cleanup. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-18842) The hbase shell clone_snaphost command returns bad error message
[
https://issues.apache.org/jira/browse/HBASE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates reassigned HBASE-18842:
---
Assignee: Jesse Yates
> The hbase shell clone_snaphost command returns bad error message
>
>
> Key: HBASE-18842
> URL: https://issues.apache.org/jira/browse/HBASE-18842
> Project: HBase
> Issue Type: Bug
>Reporter: Thoralf Gutierrez
>Assignee: Jesse Yates
>Priority: Minor
>
> When you call the hbase shell clone_snapshot command with a target namespace
> that doesn't exist, you get an error message, but the variable used to
> identify the inexistent namespace is wrong:
> {noformat}
> hbase(main):001:0> clone_snapshot 'someSnapshotName',
> 'someNamespaceName:someTableName'
> ERROR: Unknown namespace someSnapshotName!
> Create a new table by cloning the snapshot content.
> There're no copies of data involved.
> And writing on the newly created table will not influence the snapshot data.
> Examples:
> hbase> clone_snapshot 'snapshotName', 'tableName'
> hbase> clone_snapshot 'snapshotName', 'namespace:tableName'
> {noformat}
> It should rather say:
> {noformat}
> ERROR: Unknown namespace someNamespaceName!
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18842) The hbase shell clone_snaphost command returns bad error message
[
https://issues.apache.org/jira/browse/HBASE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170850#comment-16170850
]
Jesse Yates commented on HBASE-18842:
-
Any chance for a patch [~ThorGutierrez]? Looks like we could do a light hack
where commands could optionally handle exceptions to generate custom ruby
exceptions.
> The hbase shell clone_snaphost command returns bad error message
>
>
> Key: HBASE-18842
> URL: https://issues.apache.org/jira/browse/HBASE-18842
> Project: HBase
> Issue Type: Bug
>Reporter: Thoralf Gutierrez
>Priority: Minor
>
> When you call the hbase shell clone_snapshot command with a target namespace
> that doesn't exist, you get an error message, but the variable used to
> identify the inexistent namespace is wrong:
> {noformat}
> hbase(main):001:0> clone_snapshot 'someSnapshotName',
> 'someNamespaceName:someTableName'
> ERROR: Unknown namespace someSnapshotName!
> Create a new table by cloning the snapshot content.
> There're no copies of data involved.
> And writing on the newly created table will not influence the snapshot data.
> Examples:
> hbase> clone_snapshot 'snapshotName', 'tableName'
> hbase> clone_snapshot 'snapshotName', 'namespace:tableName'
> {noformat}
> It should rather say:
> {noformat}
> ERROR: Unknown namespace someNamespaceName!
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-17852) Add Fault tolerance to HBASE-14417 (Support bulk loaded files in incremental backup)
[ https://issues.apache.org/jira/browse/HBASE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170822#comment-16170822 ] Vladimir Rodionov edited comment on HBASE-17852 at 9/18/17 10:39 PM: - Reopened HBASE-14417. Working on the fix. Created HBASE-18843 was (Author: vrodionov): Reopened HBASE-14417. Working on the fix. > Add Fault tolerance to HBASE-14417 (Support bulk loaded files in incremental > backup) > > > Key: HBASE-17852 > URL: https://issues.apache.org/jira/browse/HBASE-17852 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-17852-v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18843) Add DistCp support to incremental backup with bulk loading
Vladimir Rodionov created HBASE-18843: - Summary: Add DistCp support to incremental backup with bulk loading Key: HBASE-18843 URL: https://issues.apache.org/jira/browse/HBASE-18843 Project: HBase Issue Type: Improvement Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-13346) Clean up Filter package for post 1.0 s/KeyValue/Cell/g
[
https://issues.apache.org/jira/browse/HBASE-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Penzes updated HBASE-13346:
-
Attachment: HBASE-13346.master.003.patch
> Clean up Filter package for post 1.0 s/KeyValue/Cell/g
> --
>
> Key: HBASE-13346
> URL: https://issues.apache.org/jira/browse/HBASE-13346
> Project: HBase
> Issue Type: Bug
> Components: API, Filters
>Affects Versions: 2.0.0
>Reporter: Lars George
>Assignee: Tamas Penzes
>Priority: Critical
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-13346.master.001.patch,
> HBASE-13346.master.002.patch, HBASE-13346.master.003.patch
>
>
> Since we have a bit of a messy Filter API with KeyValue vs Cell reference
> mixed up all over the place, I recommend cleaning this up once and for all.
> There should be no {{KeyValue}} (or {{kv}}, {{kvs}} etc.) in any method or
> parameter name.
> This includes deprecating and renaming filters too, for example
> {{FirstKeyOnlyFilter}}, which really should be named {{FirstKeyValueFilter}}
> as it does _not_ just return the key, but the entire cell. It should be
> deprecated and renamed to {{FirstCellFilter}} (or {{FirstColumnFilter}} if
> you prefer).
> In general we should clarify and settle on {{KeyValue}} vs {{Cell}} vs
> {{Column}} in our naming. The latter two are the only ones going forward with
> the public API, and are used synonymous. We should carefully check which is
> better suited (is it really a specific cell, or the newest cell, aka the
> newest column value) and settle on a naming schema.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-14417. Resolution: Fixed > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v21.txt, 14417-tbl-ext.v22.txt, > 14417-tbl-ext.v23.txt, 14417-tbl-ext.v24.txt, 14417-tbl-ext.v9.txt, > 14417.v11.txt, 14417.v13.txt, 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, > 14417.v24.txt, 14417.v25.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Here is review board: > https://reviews.apache.org/r/57790/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170830#comment-16170830 ] Ted Yu commented on HBASE-14417: Vlad: All the patches on this JIRA were from me. This was resolved almost half year ago. To add DistCp support, open new JIRA. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v21.txt, 14417-tbl-ext.v22.txt, > 14417-tbl-ext.v23.txt, 14417-tbl-ext.v24.txt, 14417-tbl-ext.v9.txt, > 14417.v11.txt, 14417.v13.txt, 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, > 14417.v24.txt, 14417.v25.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Here is review board: > https://reviews.apache.org/r/57790/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-14417: -- Assignee: Ted Yu (was: Vladimir Rodionov) > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v21.txt, 14417-tbl-ext.v22.txt, > 14417-tbl-ext.v23.txt, 14417-tbl-ext.v24.txt, 14417-tbl-ext.v9.txt, > 14417.v11.txt, 14417.v13.txt, 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, > 14417.v24.txt, 14417.v25.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Here is review board: > https://reviews.apache.org/r/57790/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17852) Add Fault tolerance to HBASE-14417 (Support bulk loaded files in incremental backup)
[ https://issues.apache.org/jira/browse/HBASE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170822#comment-16170822 ] Vladimir Rodionov commented on HBASE-17852: --- Reopened HBASE-14417. Working on the fix. > Add Fault tolerance to HBASE-14417 (Support bulk loaded files in incremental > backup) > > > Key: HBASE-17852 > URL: https://issues.apache.org/jira/browse/HBASE-17852 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-17852-v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-14417: --- Assignee: Vladimir Rodionov (was: Ted Yu) > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v21.txt, 14417-tbl-ext.v22.txt, > 14417-tbl-ext.v23.txt, 14417-tbl-ext.v24.txt, 14417-tbl-ext.v9.txt, > 14417.v11.txt, 14417.v13.txt, 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, > 14417.v24.txt, 14417.v25.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Here is review board: > https://reviews.apache.org/r/57790/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18842) The hbase shell clone_snaphost command returns bad error message
Thoralf Gutierrez created HBASE-18842:
-
Summary: The hbase shell clone_snaphost command returns bad error
message
Key: HBASE-18842
URL: https://issues.apache.org/jira/browse/HBASE-18842
Project: HBase
Issue Type: Bug
Reporter: Thoralf Gutierrez
Priority: Minor
When you call the hbase shell clone_snapshot command with a target namespace
that doesn't exist, you get an error message, but the variable used to identify
the inexistent namespace is wrong:
{noformat}
hbase(main):001:0> clone_snapshot 'someSnapshotName',
'someNamespaceName:someTableName'
ERROR: Unknown namespace someSnapshotName!
Create a new table by cloning the snapshot content.
There're no copies of data involved.
And writing on the newly created table will not influence the snapshot data.
Examples:
hbase> clone_snapshot 'snapshotName', 'tableName'
hbase> clone_snapshot 'snapshotName', 'namespace:tableName'
{noformat}
It should rather say:
{noformat}
ERROR: Unknown namespace someNamespaceName!
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170820#comment-16170820 ] Vladimir Rodionov commented on HBASE-14417: --- Reopened to add DistCp support. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v14.txt, 14417-tbl-ext.v18.txt, 14417-tbl-ext.v19.txt, > 14417-tbl-ext.v20.txt, 14417-tbl-ext.v21.txt, 14417-tbl-ext.v22.txt, > 14417-tbl-ext.v23.txt, 14417-tbl-ext.v24.txt, 14417-tbl-ext.v9.txt, > 14417.v11.txt, 14417.v13.txt, 14417.v1.txt, 14417.v21.txt, 14417.v23.txt, > 14417.v24.txt, 14417.v25.txt, 14417.v2.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Here is the review board (out of date): > https://reviews.apache.org/r/54258/ > In order not to miss the hfiles which are loaded into region directories in a > situation where postBulkLoadHFile() hook is not called (bulk load being > interrupted), we record hfile names thru preCommitStoreFile() hook. > At time of incremental backup, we check the presence of such hfiles. If they > are present, they become part of the incremental backup image. > Here is review board: > https://reviews.apache.org/r/57790/ > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17852) Add Fault tolerance to HBASE-14417 (Support bulk loaded files in incremental backup)
[ https://issues.apache.org/jira/browse/HBASE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170818#comment-16170818 ] Vladimir Rodionov commented on HBASE-17852: --- Unfortunately, we have issue with the current implementation of bulk loading - no distcp support. All file copies are done from a client. This definitely, does not scale. My bad, somehow I have missed that during the code review. > Add Fault tolerance to HBASE-14417 (Support bulk loaded files in incremental > backup) > > > Key: HBASE-17852 > URL: https://issues.apache.org/jira/browse/HBASE-17852 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-17852-v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-13346) Clean up Filter package for post 1.0 s/KeyValue/Cell/g
[
https://issues.apache.org/jira/browse/HBASE-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-13346:
Fix Version/s: (was: 2.0.0-alpha-3)
2.0.0-alpha-4
> Clean up Filter package for post 1.0 s/KeyValue/Cell/g
> --
>
> Key: HBASE-13346
> URL: https://issues.apache.org/jira/browse/HBASE-13346
> Project: HBase
> Issue Type: Bug
> Components: API, Filters
>Affects Versions: 2.0.0
>Reporter: Lars George
>Assignee: Tamas Penzes
>Priority: Critical
> Fix For: 2.0.0-alpha-4
>
> Attachments: HBASE-13346.master.001.patch,
> HBASE-13346.master.002.patch
>
>
> Since we have a bit of a messy Filter API with KeyValue vs Cell reference
> mixed up all over the place, I recommend cleaning this up once and for all.
> There should be no {{KeyValue}} (or {{kv}}, {{kvs}} etc.) in any method or
> parameter name.
> This includes deprecating and renaming filters too, for example
> {{FirstKeyOnlyFilter}}, which really should be named {{FirstKeyValueFilter}}
> as it does _not_ just return the key, but the entire cell. It should be
> deprecated and renamed to {{FirstCellFilter}} (or {{FirstColumnFilter}} if
> you prefer).
> In general we should clarify and settle on {{KeyValue}} vs {{Cell}} vs
> {{Column}} in our naming. The latter two are the only ones going forward with
> the public API, and are used synonymous. We should carefully check which is
> better suited (is it really a specific cell, or the newest cell, aka the
> newest column value) and settle on a naming schema.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15410) Utilize the max seek value when all Filters in MUST_PASS_ALL FilterList return SEEK_NEXT_USING_HINT
[ https://issues.apache.org/jira/browse/HBASE-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15410: --- Attachment: 15410.branch-1.patch > Utilize the max seek value when all Filters in MUST_PASS_ALL FilterList > return SEEK_NEXT_USING_HINT > --- > > Key: HBASE-15410 > URL: https://issues.apache.org/jira/browse/HBASE-15410 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Labels: filter, perfomance > Fix For: 2.0.0-alpha-3 > > Attachments: 15410.branch-1.patch, 15410.v1.patch, 15410.v2.patch, > 15410-wip.patch > > > As Preston mentioned in the comment in HBASE-15243: > https://issues.apache.org/jira/browse/HBASE-15243?focusedCommentId=15143557=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15143557 > An optimization for filters returning a SEEK_NEXT_USING_HINT would be to seek > to the highest hint (Any previous/lower row won't be accepted by the filter > returning that seek). > This JIRA is to explore this potential optimization. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18823) Apply RegionInfo to MasterObserver/RegionObserver/WALObserver
[
https://issues.apache.org/jira/browse/HBASE-18823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170738#comment-16170738
]
Hadoop QA commented on HBASE-18823:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 9m
43s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
41m 31s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}121m
42s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
24s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
20s{color} | {color:green} hbase-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
7s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}217m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:5d60123 |
| JIRA Issue | HBASE-18823 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12887699/HBASE-18823.v0.patch |
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a0525825a6e3 3.13.0-123-generic #172-Ubuntu SMP
[jira] [Commented] (HBASE-18832) LTT fails with casting exception for HColumnDescriptor
[
https://issues.apache.org/jira/browse/HBASE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170665#comment-16170665
]
Umesh Agashe commented on HBASE-18832:
--
Thanks for follow up JIRA [~stack]!
> LTT fails with casting exception for HColumnDescriptor
> --
>
> Key: HBASE-18832
> URL: https://issues.apache.org/jira/browse/HBASE-18832
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha-4
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Fix For: 2.0.0-alpha-4
>
> Attachments: hbase-18832.master.001.patch
>
>
> Here is the stack trace:
> {code}
> 2017-09-15 12:38:38,158 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> java.lang.ClassCastException:
> org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder$ModifyableColumnFamilyDescriptor
> cannot be cast to org.apache.hadoop.hbase.HColumnDescriptor
> at
> org.apache.hadoop.hbase.util.LoadTestTool.applyColumnFamilyOptions(LoadTestTool.java:265)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.initTestTable(LoadTestTool.java:540)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.loadTable(LoadTestTool.java:567)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:548)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:262)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:793)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18832) LTT fails with casting exception for HColumnDescriptor
[
https://issues.apache.org/jira/browse/HBASE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-18832:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to master and branch-2. Thanks [~uagashe].
I tried it first and came across HBASE-18841. TODO.
> LTT fails with casting exception for HColumnDescriptor
> --
>
> Key: HBASE-18832
> URL: https://issues.apache.org/jira/browse/HBASE-18832
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha-4
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Fix For: 2.0.0-alpha-4
>
> Attachments: hbase-18832.master.001.patch
>
>
> Here is the stack trace:
> {code}
> 2017-09-15 12:38:38,158 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> java.lang.ClassCastException:
> org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder$ModifyableColumnFamilyDescriptor
> cannot be cast to org.apache.hadoop.hbase.HColumnDescriptor
> at
> org.apache.hadoop.hbase.util.LoadTestTool.applyColumnFamilyOptions(LoadTestTool.java:265)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.initTestTable(LoadTestTool.java:540)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.loadTable(LoadTestTool.java:567)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:548)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:154)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:262)
> at
> org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:793)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18841) ./bin/hbase ltt and pe cannot find their classes when in dev/build context
stack created HBASE-18841: - Summary: ./bin/hbase ltt and pe cannot find their classes when in dev/build context Key: HBASE-18841 URL: https://issues.apache.org/jira/browse/HBASE-18841 Project: HBase Issue Type: Bug Reporter: stack Priority: Minor If I run the below out of a built checkout, it fails unable to find the main in named LoadTestTool class: ./bin/hbase ltt Ditto for: ./bin/hbase pe The main classes are in *-test.jars which we do not include in our cached_classpath.txt file that is our trick for making stuff work in dev context. Investigate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18796) Admin#isTableAvailable returns incorrect result before daughter regions are opened
[ https://issues.apache.org/jira/browse/HBASE-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170607#comment-16170607 ] Andrew Purtell commented on HBASE-18796: Ok, thanks. The precommit looks messed up but I'm not sure it is related. Let me try this change out locally. If it looks good I will commit. > Admin#isTableAvailable returns incorrect result before daughter regions are > opened > -- > > Key: HBASE-18796 > URL: https://issues.apache.org/jira/browse/HBASE-18796 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-18796.branch-1.001.patch, > HBASE-18796.branch-1.001.patch, HBASE-18796.branch-1.002.patch, > HBASE-18796.branch-1.003.patch, HBASE-18796.master.001.patch > > > Admin#isTableAvailable checks if it can getServerName for the meta entries it > reads. During the time of split server location are added to the meta entries > in MetaTableAccessor#splitRegion although the description of the method says > "Does not add the location information to the daughter regions since they are > not open yet.". At this point during the split daughter regions are not > actually open, so we can get to a state where parent is offline, daughters > are not yet open but isTableAvailable returns true. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18142) Deletion of a cell deletes the previous versions too
[ https://issues.apache.org/jira/browse/HBASE-18142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170604#comment-16170604 ] Hudson commented on HBASE-18142: FAILURE: Integrated in Jenkins build HBase-2.0 #535 (See [https://builds.apache.org/job/HBase-2.0/535/]) HBASE-18142 Deletion of a cell deletes the previous versions too (chia7712: rev 03eefa8aa5a98303ab9e2f42a25f9dec909b7094) * (edit) hbase-shell/src/main/ruby/shell/commands/deleteall.rb * (edit) hbase-shell/src/main/ruby/hbase/table.rb * (edit) hbase-shell/src/main/ruby/shell/commands/delete.rb * (edit) hbase-shell/src/test/ruby/hbase/table_test.rb > Deletion of a cell deletes the previous versions too > > > Key: HBASE-18142 > URL: https://issues.apache.org/jira/browse/HBASE-18142 > Project: HBase > Issue Type: Bug > Components: API, shell >Affects Versions: 3.0.0, 1.3.1, 1.2.6, 2.0.0-alpha-1 >Reporter: Karthick >Assignee: ChunHao > Labels: beginner > Fix For: 2.0.0, 1.4.0, 1.3.2, 1.5.0, 1.2.7 > > Attachments: HBASE-18142.branch-1.v0.patch, > HBASE-18142.branch-1.v1.patch, HBASE-18142.master.v0.patch, > HBASE-18142.master.v1.patch, HBASE-18142.master.v2.patch, > HBASE-18142.master.v3.patch, HBASE-18142.master.v4.patch, > HBASE-18142.master.v5.patch, HBASE-18142.master.v6.patch, > HBASE-18142.master.v7.patch, HBASE-18142.master.v8.patch > > > When I tried to delete a cell using it's timestamp in the Hbase Shell, the > previous versions of the same cell also got deleted. But when I tried the > same using the Java API, then the previous versions are not deleted and I can > retrive the previous values. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java > see this file to fix the issue. This method (public Delete addColumn(final > byte [] family, final byte [] qualifier, final long timestamp)) only deletes > the current version of the cell. The previous versions are not deleted. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18835) The return type of ExtendedCell#deepClone should be ExtendedCell rather than Cell
[ https://issues.apache.org/jira/browse/HBASE-18835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170602#comment-16170602 ] Hudson commented on HBASE-18835: FAILURE: Integrated in Jenkins build HBase-2.0 #535 (See [https://builds.apache.org/job/HBase-2.0/535/]) HBASE-18835 The return type of ExtendedCell#deepClone should be (chia7712: rev f380676d4958933da7be75da6c042071dcb77160) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/NoTagsKeyValue.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/IndividualBytesFieldCell.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCell.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/NoTagsByteBufferKeyValue.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ByteBufferKeyValue.java > The return type of ExtendedCell#deepClone should be ExtendedCell rather than > Cell > - > > Key: HBASE-18835 > URL: https://issues.apache.org/jira/browse/HBASE-18835 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0-alpha-4 > > Attachments: HBASE-18835.v0.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
