[jira] [Commented] (HBASE-15053) hbase.client.max.perregion.tasks can affect get or scan operation?
[ https://issues.apache.org/jira/browse/HBASE-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074757#comment-15074757 ] 胡托 commented on HBASE-15053: Thank you！ > hbase.client.max.perregion.tasks can affect get or scan operation? > > > Key: HBASE-15053 > URL: https://issues.apache.org/jira/browse/HBASE-15053 > Project: HBase > Issue Type: Test >Reporter: 胡托 > > In Refrence guide，hbase.client.max.perregion.tasks is descripted if there is > already hbase.client.max.perregion.tasks writes in progress for this region, > new puts won’t be sent to this region until some writes finishes. > Whether can affect the read operation?I want to know,thanks! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15007) update Hadoop support matrix to list 2.6.1+ as supported
[ https://issues.apache.org/jira/browse/HBASE-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074747#comment-15074747 ] Hudson commented on HBASE-15007: SUCCESS: Integrated in HBase-Trunk_matrix #598 (See [https://builds.apache.org/job/HBase-Trunk_matrix/598/]) HBASE-15007 update docs to list Hadoop 2.6.1+ as cool. (busbey: rev 5e2c2e1780d57f2e0086b10cd28d5212d6a3d5d0) * src/main/asciidoc/_chapters/configuration.adoc > update Hadoop support matrix to list 2.6.1+ as supported > > > Key: HBASE-15007 > URL: https://issues.apache.org/jira/browse/HBASE-15007 > Project: HBase > Issue Type: Task > Components: documentation, Operability >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0 > > Attachments: HBASE-15007.1.patch > > > The Hadoop community responded very well to our request for more maintenance > releases and have now put out 2.6.1 - 2.6.3. The first of those included the > fix for our catastrophic failure under HDFS encryption. > We should update the book to point out those versions are fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15018) Inconsistent way of handling TimeoutException in the rpc client implementations
[
https://issues.apache.org/jira/browse/HBASE-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074746#comment-15074746
]
Hudson commented on HBASE-15018:
SUCCESS: Integrated in HBase-Trunk_matrix #598 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/598/])
HBASE-15018 Inconsistent way of handling TimeoutException in the rpc (busbey:
rev 413d663f9262bcdfce67cbc902f7e3153a161fad)
* hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/AbstractTestIPC.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClientImpl.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java
> Inconsistent way of handling TimeoutException in the rpc client
> implementations
> ---
>
> Key: HBASE-15018
> URL: https://issues.apache.org/jira/browse/HBASE-15018
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC
>Affects Versions: 2.0.0, 1.1.0, 1.2.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-15018-v1(1).patch, HBASE-15018-v1.patch,
> HBASE-15018-v2.patch, HBASE-15018-v3-branch-1.patch, HBASE-15018-v3.patch,
> HBASE-15018.patch, HBASE-15018.patch
>
>
> If there is any rpc timeout when using RpcClientImpl then we wrap the
> exception in IOE and throw it,
> {noformat}
> 2015-11-16 04:05:24,935 WARN [main-EventThread.replicationSource,peer2]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because of
> a local or network error:
> java.io.IOException: Call to host-XX:16040 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1271)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1239)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:25690)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:77)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:322)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:308)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1213)
> ... 10 more
> {noformat}
> But that isn't case with AsyncRpcClient, we don't wrap and throw
> CallTimeoutException as it is.
> {noformat}
> 2015-12-21 14:27:33,093 WARN
> [RS_OPEN_REGION-host-XX:16201-0.replicationSource.host-XX%2C16201%2C1450687255593,1]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because
> of a local or network error:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: callId=2,
> method=ReplicateWALEntry, rpcTimeout=60, param {TODO: class
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$ReplicateWALEntryRequest}
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcClient.call(AsyncRpcClient.java:257)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:295)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:23707)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:73)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:387)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInt
[jira] [Commented] (HBASE-15053) hbase.client.max.perregion.tasks can affect get or scan operation?
[ https://issues.apache.org/jira/browse/HBASE-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074741#comment-15074741 ] Jingcheng Du commented on HBASE-15053: -- This property is used in AsyncProcess. As I can see it is only used when background flush in mutate and batch coprocessor call. Read operations should not be affected. By the way, you can write mails to hbase user list or dev list instead of creating a JIRA. And please invalidate this JIRA later. > hbase.client.max.perregion.tasks can affect get or scan operation? > > > Key: HBASE-15053 > URL: https://issues.apache.org/jira/browse/HBASE-15053 > Project: HBase > Issue Type: Test >Reporter: 胡托 > > In Refrence guide，hbase.client.max.perregion.tasks is descripted if there is > already hbase.client.max.perregion.tasks writes in progress for this region, > new puts won’t be sent to this region until some writes finishes. > Whether can affect the read operation?I want to know,thanks! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15031) Fix merge of MVCC and SequenceID performance regression in branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074739#comment-15074739 ] Hudson commented on HBASE-15031: FAILURE: Integrated in HBase-1.1-JDK7 #1632 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1632/]) HBASE-15031 Fix merge of MVCC and SequenceID performance regression in (stack: rev 5140ce0afa4014a41ab27ba9f41cb8e799227b1b) * hbase-server/src/test/java/org/apache/hadoop/hbase/IncrementPerformanceTest.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionIncrement.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementsFromClientSide.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementFromClientSideWithCoprocessor.java > Fix merge of MVCC and SequenceID performance regression in branch-1.0 > - > > Key: HBASE-15031 > URL: https://issues.apache.org/jira/browse/HBASE-15031 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 1.0.3 >Reporter: stack >Assignee: stack > Fix For: 1.1.3, 1.0.4 > > Attachments: 14460.v0.branch-1.0.patch, 15031.v2.branch-1.0.patch, > 15031.v3.branch-1.0.patch, 15031.v4.branch-1.0.patch, > 15031.v5.branch-1.0.patch, 15031.v6.branch-1.0.patch, > 15031.v6.branch-1.0.patch, 15031.v6.branch-1.0.patch, > 15031.v7.branch-1.0.patch, 15031.v8.branch-1.0.patch > > > Subtask with fix for branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074735#comment-15074735
]
Duo Zhang commented on HBASE-14790:
---
{quote}
Soon as you have a basic WAL Duo Zhang, I can try get some basic numbers? I'm
doing a bit of perf testing at mo on write pipeline.
{quote}
Great. But I think I need some time to finish the WAL implementation. The
{{WALProvider}} is not friendly to an async implementation, I need to do some
refactoring first.
Thanks.
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15018) Inconsistent way of handling TimeoutException in the rpc client implementations
[
https://issues.apache.org/jira/browse/HBASE-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074734#comment-15074734
]
Sean Busbey commented on HBASE-15018:
-
it might be. my storage when running under ubuntu is kind of slow. that test
passes without the patch on branch-1.1 and master but fails on both branch-1.2
and branch-1.
let's see how the buildbot does and I can file an issue about 1.2 and 1.3.
> Inconsistent way of handling TimeoutException in the rpc client
> implementations
> ---
>
> Key: HBASE-15018
> URL: https://issues.apache.org/jira/browse/HBASE-15018
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC
>Affects Versions: 2.0.0, 1.1.0, 1.2.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-15018-v1(1).patch, HBASE-15018-v1.patch,
> HBASE-15018-v2.patch, HBASE-15018-v3-branch-1.patch, HBASE-15018-v3.patch,
> HBASE-15018.patch, HBASE-15018.patch
>
>
> If there is any rpc timeout when using RpcClientImpl then we wrap the
> exception in IOE and throw it,
> {noformat}
> 2015-11-16 04:05:24,935 WARN [main-EventThread.replicationSource,peer2]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because of
> a local or network error:
> java.io.IOException: Call to host-XX:16040 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1271)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1239)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:25690)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:77)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:322)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:308)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1213)
> ... 10 more
> {noformat}
> But that isn't case with AsyncRpcClient, we don't wrap and throw
> CallTimeoutException as it is.
> {noformat}
> 2015-12-21 14:27:33,093 WARN
> [RS_OPEN_REGION-host-XX:16201-0.replicationSource.host-XX%2C16201%2C1450687255593,1]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because
> of a local or network error:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: callId=2,
> method=ReplicateWALEntry, rpcTimeout=60, param {TODO: class
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$ReplicateWALEntryRequest}
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcClient.call(AsyncRpcClient.java:257)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:295)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:23707)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:73)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:387)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:370)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.conc
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074732#comment-15074732
]
Lars Hofhansl commented on HBASE-14468:
---
This is almost strictly add-on. We can backport to 0.98 I think. [~apurtell].
(We should also test whether this allows HBase to handle usecases that are
typically handled by Kafka.)
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14468-v1.patch, HBASE-14468-v10.patch,
> HBASE-14468-v2.patch, HBASE-14468-v3.patch, HBASE-14468-v4.patch,
> HBASE-14468-v5.patch, HBASE-14468-v6.patch, HBASE-14468-v7.patch,
> HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15031) Fix merge of MVCC and SequenceID performance regression in branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074725#comment-15074725 ] Hudson commented on HBASE-15031: FAILURE: Integrated in HBase-1.1-JDK8 #1719 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1719/]) HBASE-15031 Fix merge of MVCC and SequenceID performance regression in (stack: rev 5140ce0afa4014a41ab27ba9f41cb8e799227b1b) * hbase-server/src/test/java/org/apache/hadoop/hbase/IncrementPerformanceTest.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementFromClientSideWithCoprocessor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementsFromClientSide.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionIncrement.java > Fix merge of MVCC and SequenceID performance regression in branch-1.0 > - > > Key: HBASE-15031 > URL: https://issues.apache.org/jira/browse/HBASE-15031 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 1.0.3 >Reporter: stack >Assignee: stack > Fix For: 1.1.3, 1.0.4 > > Attachments: 14460.v0.branch-1.0.patch, 15031.v2.branch-1.0.patch, > 15031.v3.branch-1.0.patch, 15031.v4.branch-1.0.patch, > 15031.v5.branch-1.0.patch, 15031.v6.branch-1.0.patch, > 15031.v6.branch-1.0.patch, 15031.v6.branch-1.0.patch, > 15031.v7.branch-1.0.patch, 15031.v8.branch-1.0.patch > > > Subtask with fix for branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15018) Inconsistent way of handling TimeoutException in the rpc client implementations
[
https://issues.apache.org/jira/browse/HBASE-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074723#comment-15074723
]
Ashish Singhi commented on HBASE-15018:
---
Related to env ? Why am I not getting this!
Not sure if Hadoop QA is also reporting it.
> Inconsistent way of handling TimeoutException in the rpc client
> implementations
> ---
>
> Key: HBASE-15018
> URL: https://issues.apache.org/jira/browse/HBASE-15018
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC
>Affects Versions: 2.0.0, 1.1.0, 1.2.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-15018-v1(1).patch, HBASE-15018-v1.patch,
> HBASE-15018-v2.patch, HBASE-15018-v3-branch-1.patch, HBASE-15018-v3.patch,
> HBASE-15018.patch, HBASE-15018.patch
>
>
> If there is any rpc timeout when using RpcClientImpl then we wrap the
> exception in IOE and throw it,
> {noformat}
> 2015-11-16 04:05:24,935 WARN [main-EventThread.replicationSource,peer2]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because of
> a local or network error:
> java.io.IOException: Call to host-XX:16040 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1271)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1239)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:25690)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:77)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:322)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:308)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1213)
> ... 10 more
> {noformat}
> But that isn't case with AsyncRpcClient, we don't wrap and throw
> CallTimeoutException as it is.
> {noformat}
> 2015-12-21 14:27:33,093 WARN
> [RS_OPEN_REGION-host-XX:16201-0.replicationSource.host-XX%2C16201%2C1450687255593,1]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because
> of a local or network error:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: callId=2,
> method=ReplicateWALEntry, rpcTimeout=60, param {TODO: class
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$ReplicateWALEntryRequest}
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcClient.call(AsyncRpcClient.java:257)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:295)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:23707)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:73)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:387)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:370)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
[jira] [Commented] (HBASE-15018) Inconsistent way of handling TimeoutException in the rpc client implementations
[
https://issues.apache.org/jira/browse/HBASE-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074715#comment-15074715
]
Sean Busbey commented on HBASE-15018:
-
hurm. also fails on branch-1 with no patch. sigh.
> Inconsistent way of handling TimeoutException in the rpc client
> implementations
> ---
>
> Key: HBASE-15018
> URL: https://issues.apache.org/jira/browse/HBASE-15018
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC
>Affects Versions: 2.0.0, 1.1.0, 1.2.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-15018-v1(1).patch, HBASE-15018-v1.patch,
> HBASE-15018-v2.patch, HBASE-15018-v3-branch-1.patch, HBASE-15018-v3.patch,
> HBASE-15018.patch, HBASE-15018.patch
>
>
> If there is any rpc timeout when using RpcClientImpl then we wrap the
> exception in IOE and throw it,
> {noformat}
> 2015-11-16 04:05:24,935 WARN [main-EventThread.replicationSource,peer2]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because of
> a local or network error:
> java.io.IOException: Call to host-XX:16040 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1271)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1239)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:25690)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:77)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:322)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:308)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1213)
> ... 10 more
> {noformat}
> But that isn't case with AsyncRpcClient, we don't wrap and throw
> CallTimeoutException as it is.
> {noformat}
> 2015-12-21 14:27:33,093 WARN
> [RS_OPEN_REGION-host-XX:16201-0.replicationSource.host-XX%2C16201%2C1450687255593,1]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because
> of a local or network error:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: callId=2,
> method=ReplicateWALEntry, rpcTimeout=60, param {TODO: class
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$ReplicateWALEntryRequest}
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcClient.call(AsyncRpcClient.java:257)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:295)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:23707)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:73)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:387)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:370)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread
[jira] [Commented] (HBASE-15046) Perf test doing all mutation steps under row lock
[ https://issues.apache.org/jira/browse/HBASE-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074710#comment-15074710 ] Lars Hofhansl commented on HBASE-15046: --- Hey [~stack]. I'm very interested in this. Wanna huddle first thing next year? I've also written a simple load tester for a kind of workload we have, which essentially overrides (i.e. writes new versions of) the same keys (row key + CF + CQ) over and over again in batches; it's likely multiple threads/clients will touch the same key in a batch. > Perf test doing all mutation steps under row lock > - > > Key: HBASE-15046 > URL: https://issues.apache.org/jira/browse/HBASE-15046 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack > Attachments: 1.2.svg, 1.2.v2.svg > > > This issue is about perf testing a redo of the write pipeline so that rather > than: > * take rowlock > * start mvcc > * append to WAL > * add to memstore > * sync WAL > * let go of rowlock > * finish up mvcc > instead try... > * take rowlock > * start mvcc > * append to WAL > * sync WAL > * add to memstore > * finish up mvcc > * let go of rowlock > The latter is more straight-forward undoing need of rolling back memstore if > all does not succeed. > It might be slower though. This issue is a look-see/try it. > The redo will also help address the parent issue in a more general way so we > can do without the special-casing done for branch-1.0 and branch-1.1 done in > a sibling subtask. > Other benefits are that the current write pipeline is copy/pasted in a few > places -- in append, increment and checkand* -- and a refactor will allow us > to fix this duplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15018) Inconsistent way of handling TimeoutException in the rpc client implementations
[
https://issues.apache.org/jira/browse/HBASE-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074707#comment-15074707
]
Sean Busbey commented on HBASE-15018:
-
let me see if it passes on branch-1 for me if I remove the patch.
> Inconsistent way of handling TimeoutException in the rpc client
> implementations
> ---
>
> Key: HBASE-15018
> URL: https://issues.apache.org/jira/browse/HBASE-15018
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC
>Affects Versions: 2.0.0, 1.1.0, 1.2.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-15018-v1(1).patch, HBASE-15018-v1.patch,
> HBASE-15018-v2.patch, HBASE-15018-v3-branch-1.patch, HBASE-15018-v3.patch,
> HBASE-15018.patch, HBASE-15018.patch
>
>
> If there is any rpc timeout when using RpcClientImpl then we wrap the
> exception in IOE and throw it,
> {noformat}
> 2015-11-16 04:05:24,935 WARN [main-EventThread.replicationSource,peer2]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because of
> a local or network error:
> java.io.IOException: Call to host-XX:16040 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1271)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1239)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:25690)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:77)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:322)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:308)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1213)
> ... 10 more
> {noformat}
> But that isn't case with AsyncRpcClient, we don't wrap and throw
> CallTimeoutException as it is.
> {noformat}
> 2015-12-21 14:27:33,093 WARN
> [RS_OPEN_REGION-host-XX:16201-0.replicationSource.host-XX%2C16201%2C1450687255593,1]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because
> of a local or network error:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: callId=2,
> method=ReplicateWALEntry, rpcTimeout=60, param {TODO: class
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$ReplicateWALEntryRequest}
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcClient.call(AsyncRpcClient.java:257)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:295)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:23707)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:73)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:387)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:370)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.T
[jira] [Updated] (HBASE-15054) Allow 0.94 to compile with JDK8
[ https://issues.apache.org/jira/browse/HBASE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-15054: -- Attachment: 15054.txt Simple patch. Will break if somebody inherited from pool map, should list as release note. I think the last 0.94 release should compile with JDK8. > Allow 0.94 to compile with JDK8 > --- > > Key: HBASE-15054 > URL: https://issues.apache.org/jira/browse/HBASE-15054 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: 15054.txt > > > Fix the following two problems: > # PoolMap > # InputSampler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15054) Allow 0.94 to compile with JDK8
Lars Hofhansl created HBASE-15054: - Summary: Allow 0.94 to compile with JDK8 Key: HBASE-15054 URL: https://issues.apache.org/jira/browse/HBASE-15054 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Fix the following two problems: # PoolMap # InputSampler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15018) Inconsistent way of handling TimeoutException in the rpc client implementations
[
https://issues.apache.org/jira/browse/HBASE-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-15018:
--
Attachment: HBASE-15018-v3-branch-1.patch
Patch attached to get Hadoop QA run on branch-1.
> Inconsistent way of handling TimeoutException in the rpc client
> implementations
> ---
>
> Key: HBASE-15018
> URL: https://issues.apache.org/jira/browse/HBASE-15018
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC
>Affects Versions: 2.0.0, 1.1.0, 1.2.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-15018-v1(1).patch, HBASE-15018-v1.patch,
> HBASE-15018-v2.patch, HBASE-15018-v3-branch-1.patch, HBASE-15018-v3.patch,
> HBASE-15018.patch, HBASE-15018.patch
>
>
> If there is any rpc timeout when using RpcClientImpl then we wrap the
> exception in IOE and throw it,
> {noformat}
> 2015-11-16 04:05:24,935 WARN [main-EventThread.replicationSource,peer2]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because of
> a local or network error:
> java.io.IOException: Call to host-XX:16040 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1271)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1239)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:25690)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:77)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:322)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:308)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1213)
> ... 10 more
> {noformat}
> But that isn't case with AsyncRpcClient, we don't wrap and throw
> CallTimeoutException as it is.
> {noformat}
> 2015-12-21 14:27:33,093 WARN
> [RS_OPEN_REGION-host-XX:16201-0.replicationSource.host-XX%2C16201%2C1450687255593,1]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because
> of a local or network error:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: callId=2,
> method=ReplicateWALEntry, rpcTimeout=60, param {TODO: class
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$ReplicateWALEntryRequest}
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcClient.call(AsyncRpcClient.java:257)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:295)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:23707)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:73)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:387)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:370)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:74
[jira] [Commented] (HBASE-15018) Inconsistent way of handling TimeoutException in the rpc client implementations
[
https://issues.apache.org/jira/browse/HBASE-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074700#comment-15074700
]
Ashish Singhi commented on HBASE-15018:
---
I ran locally 5 times on branch-1 with patch applied in my Windows environment
and it is successful all the times.
{noformat}
---
T E S T S
---
Running org.apache.hadoop.hbase.security.token.TestGenerateDelegationToken
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.712 sec - in
org.apache.hadoop.hbase.security.token.T
eDelegationToken
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (secondPartTestsExecution) @
hbase-server ---
[INFO] Tests are skipped.
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 57.953s
[INFO] Finished at: Wed Dec 30 12:29:24 GMT+05:30 2015
[INFO] Final Memory: 32M/76M
[INFO]
---
T E S T S
---
Running org.apache.hadoop.hbase.security.token.TestGenerateDelegationToken
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.892 sec - in
org.apache.hadoop.hbase.security.token.T
eDelegationToken
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (secondPartTestsExecution) @
hbase-server ---
[INFO] Tests are skipped.
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 56.470s
[INFO] Finished at: Wed Dec 30 12:30:46 GMT+05:30 2015
[INFO] Final Memory: 31M/76M
[INFO]
D:\OGHBase-b_1\hbase\hbase-server>
---
T E S T S
---
Running org.apache.hadoop.hbase.security.token.TestGenerateDelegationToken
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.573 sec - in
org.apache.hadoop.hbase.security.token.TestGenerat
eDelegationToken
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (secondPartTestsExecution) @
hbase-server ---
[INFO] Tests are skipped.
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 55.369s
[INFO] Finished at: Wed Dec 30 12:32:13 GMT+05:30 2015
[INFO] Final Memory: 31M/76M
[INFO]
D:\OGHBase-b_1\hbase\hbase-server>
{noformat}
> Inconsistent way of handling TimeoutException in the rpc client
> implementations
> ---
>
> Key: HBASE-15018
> URL: https://issues.apache.org/jira/browse/HBASE-15018
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC
>Affects Versions: 2.0.0, 1.1.0, 1.2.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-15018-v1(1).patch, HBASE-15018-v1.patch,
> HBASE-15018-v2.patch, HBASE-15018-v3.patch, HBASE-15018.patch,
> HBASE-15018.patch
>
>
> If there is any rpc timeout when using RpcClientImpl then we wrap the
> exception in IOE and throw it,
> {noformat}
> 2015-11-16 04:05:24,935 WARN [main-EventThread.replicationSource,peer2]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because of
> a local or network error:
> java.io.IOException: Call to host-XX:16040 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1271)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1239)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:25690)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUt
[jira] [Commented] (HBASE-14949) Skip duplicate entries when replay WAL.
[
https://issues.apache.org/jira/browse/HBASE-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074699#comment-15074699
]
Duo Zhang commented on HBASE-14949:
---
{quote}
This only happens when all datanodes crash, right?
{quote}
This depends on how you define crash. A network problem with enough time could
also lead to this. If all datanodes die then the problem will be data lost...
And still I'd say this is irrelevant with HBASE-15046...
> Skip duplicate entries when replay WAL.
> ---
>
> Key: HBASE-14949
> URL: https://issues.apache.org/jira/browse/HBASE-14949
> Project: HBase
> Issue Type: Sub-task
>Reporter: Heng Chen
> Attachments: HBASE-14949.patch, HBASE-14949_v1.patch,
> HBASE-14949_v2.patch
>
>
> As HBASE-14004 design, there will be duplicate entries in different WAL. It
> happens when one hflush failed, we will close old WAL with 'acked hflushed'
> length, then open a new WAL and write the unacked hlushed entries into it.
> So there maybe some overlap between old WAL and new WAL.
> We should skip the duplicate entries when replay. I think it has no harm to
> current logic, maybe we do it first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14949) Skip duplicate entries when replay WAL.
[
https://issues.apache.org/jira/browse/HBASE-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074695#comment-15074695
]
Heng Chen commented on HBASE-14949:
---
{quote}
If we do not change the logic of WAL, sync failed does not mean the WAL entry
is NOT write into the WAL file. This is the root cause of inconsistency.
{quote}
This only happens when all datanodes crash, right?
> Skip duplicate entries when replay WAL.
> ---
>
> Key: HBASE-14949
> URL: https://issues.apache.org/jira/browse/HBASE-14949
> Project: HBase
> Issue Type: Sub-task
>Reporter: Heng Chen
> Attachments: HBASE-14949.patch, HBASE-14949_v1.patch,
> HBASE-14949_v2.patch
>
>
> As HBASE-14004 design, there will be duplicate entries in different WAL. It
> happens when one hflush failed, we will close old WAL with 'acked hflushed'
> length, then open a new WAL and write the unacked hlushed entries into it.
> So there maybe some overlap between old WAL and new WAL.
> We should skip the duplicate entries when replay. I think it has no harm to
> current logic, maybe we do it first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15046) Perf test doing all mutation steps under row lock
[ https://issues.apache.org/jira/browse/HBASE-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15046: -- Attachment: 1.2.svg 1.2.v2.svg Here is what a RS looks like when YCSB write only is being run against it. Two samples. Trying to change ordering but this doMiniBatchMutation has a lot going on. First few attempts have me locked up. Will keep at it. > Perf test doing all mutation steps under row lock > - > > Key: HBASE-15046 > URL: https://issues.apache.org/jira/browse/HBASE-15046 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack > Attachments: 1.2.svg, 1.2.v2.svg > > > This issue is about perf testing a redo of the write pipeline so that rather > than: > * take rowlock > * start mvcc > * append to WAL > * add to memstore > * sync WAL > * let go of rowlock > * finish up mvcc > instead try... > * take rowlock > * start mvcc > * append to WAL > * sync WAL > * add to memstore > * finish up mvcc > * let go of rowlock > The latter is more straight-forward undoing need of rolling back memstore if > all does not succeed. > It might be slower though. This issue is a look-see/try it. > The redo will also help address the parent issue in a more general way so we > can do without the special-casing done for branch-1.0 and branch-1.1 done in > a sibling subtask. > Other benefits are that the current write pipeline is copy/pasted in a few > places -- in append, increment and checkand* -- and a refactor will allow us > to fix this duplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15053) hbase.client.max.perregion.tasks can affect get or scan operation?
胡托 created HBASE-15053: -- Summary: hbase.client.max.perregion.tasks can affect get or scan operation? Key: HBASE-15053 URL: https://issues.apache.org/jira/browse/HBASE-15053 Project: HBase Issue Type: Test Reporter: 胡托 In Refrence guide，hbase.client.max.perregion.tasks is descripted if there is already hbase.client.max.perregion.tasks writes in progress for this region, new puts won’t be sent to this region until some writes finishes. Whether can affect the read operation?I want to know,thanks! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14949) Skip duplicate entries when replay WAL.
[
https://issues.apache.org/jira/browse/HBASE-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074690#comment-15074690
]
Duo Zhang commented on HBASE-14949:
---
{quote}
If we move sync WAL before update memstore, HBASE-14004 will be fixed naturally
right?
{quote}
No.
If we do not change the logic of WAL, sync failed does not mean the WAL entry
is NOT write into the WAL file. This is the root cause of inconsistency.
> Skip duplicate entries when replay WAL.
> ---
>
> Key: HBASE-14949
> URL: https://issues.apache.org/jira/browse/HBASE-14949
> Project: HBase
> Issue Type: Sub-task
>Reporter: Heng Chen
> Attachments: HBASE-14949.patch, HBASE-14949_v1.patch,
> HBASE-14949_v2.patch
>
>
> As HBASE-14004 design, there will be duplicate entries in different WAL. It
> happens when one hflush failed, we will close old WAL with 'acked hflushed'
> length, then open a new WAL and write the unacked hlushed entries into it.
> So there maybe some overlap between old WAL and new WAL.
> We should skip the duplicate entries when replay. I think it has no harm to
> current logic, maybe we do it first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15018) Inconsistent way of handling TimeoutException in the rpc client implementations
[
https://issues.apache.org/jira/browse/HBASE-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074688#comment-15074688
]
Ashish Singhi commented on HBASE-15018:
---
I ran TestGenerateDelegationToken on master before posting v2 and it was
passing. Let me check on branch-1.
> Inconsistent way of handling TimeoutException in the rpc client
> implementations
> ---
>
> Key: HBASE-15018
> URL: https://issues.apache.org/jira/browse/HBASE-15018
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC
>Affects Versions: 2.0.0, 1.1.0, 1.2.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-15018-v1(1).patch, HBASE-15018-v1.patch,
> HBASE-15018-v2.patch, HBASE-15018-v3.patch, HBASE-15018.patch,
> HBASE-15018.patch
>
>
> If there is any rpc timeout when using RpcClientImpl then we wrap the
> exception in IOE and throw it,
> {noformat}
> 2015-11-16 04:05:24,935 WARN [main-EventThread.replicationSource,peer2]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because of
> a local or network error:
> java.io.IOException: Call to host-XX:16040 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1271)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1239)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:25690)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:77)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:322)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:308)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1213)
> ... 10 more
> {noformat}
> But that isn't case with AsyncRpcClient, we don't wrap and throw
> CallTimeoutException as it is.
> {noformat}
> 2015-12-21 14:27:33,093 WARN
> [RS_OPEN_REGION-host-XX:16201-0.replicationSource.host-XX%2C16201%2C1450687255593,1]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because
> of a local or network error:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: callId=2,
> method=ReplicateWALEntry, rpcTimeout=60, param {TODO: class
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$ReplicateWALEntryRequest}
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcClient.call(AsyncRpcClient.java:257)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:295)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:23707)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:73)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:387)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:370)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
[jira] [Commented] (HBASE-15018) Inconsistent way of handling TimeoutException in the rpc client implementations
[
https://issues.apache.org/jira/browse/HBASE-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074686#comment-15074686
]
Sean Busbey commented on HBASE-15018:
-
I pushed this on master, but when I cherry-pick back to branch-1 I consistently
get a failure from TestGenerateDelegationToken:
{code}
Running org.apache.hadoop.hbase.security.token.TestGenerateDelegationToken
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 270.558 sec <<<
FAILURE! - in org.apache.hadoop.hbase.security.token.TestGenerateDelegationToken
org.apache.hadoop.hbase.security.token.TestGenerateDelegationToken Time
elapsed: 270.548 sec <<< ERROR!
java.lang.RuntimeException: Master not initialized after 20ms seconds
at
org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:225)
at
org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:447)
at
org.apache.hadoop.hbase.security.token.TestGenerateDelegationToken.setUp(TestGenerateDelegationToken.java:129)
{code}
> Inconsistent way of handling TimeoutException in the rpc client
> implementations
> ---
>
> Key: HBASE-15018
> URL: https://issues.apache.org/jira/browse/HBASE-15018
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC
>Affects Versions: 2.0.0, 1.1.0, 1.2.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-15018-v1(1).patch, HBASE-15018-v1.patch,
> HBASE-15018-v2.patch, HBASE-15018-v3.patch, HBASE-15018.patch,
> HBASE-15018.patch
>
>
> If there is any rpc timeout when using RpcClientImpl then we wrap the
> exception in IOE and throw it,
> {noformat}
> 2015-11-16 04:05:24,935 WARN [main-EventThread.replicationSource,peer2]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because of
> a local or network error:
> java.io.IOException: Call to host-XX:16040 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1271)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1239)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:25690)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:77)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:322)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:308)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=510,
> waitTime=180001, operationTimeout=18 expired.
> at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1213)
> ... 10 more
> {noformat}
> But that isn't case with AsyncRpcClient, we don't wrap and throw
> CallTimeoutException as it is.
> {noformat}
> 2015-12-21 14:27:33,093 WARN
> [RS_OPEN_REGION-host-XX:16201-0.replicationSource.host-XX%2C16201%2C1450687255593,1]
> regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because
> of a local or network error:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: callId=2,
> method=ReplicateWALEntry, rpcTimeout=60, param {TODO: class
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$ReplicateWALEntryRequest}
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcClient.call(AsyncRpcClient.java:257)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:295)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:23707)
> at
> org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil
[jira] [Commented] (HBASE-14949) Skip duplicate entries when replay WAL.
[ https://issues.apache.org/jira/browse/HBASE-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074683#comment-15074683 ] Heng Chen commented on HBASE-14949: --- To be honest, i want to wait for HBASE-15046 results. If we move sync WAL before update memstore, HBASE-14004 will be fixed naturally right? Everything will be much more simple if performance is OK. > Skip duplicate entries when replay WAL. > --- > > Key: HBASE-14949 > URL: https://issues.apache.org/jira/browse/HBASE-14949 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen > Attachments: HBASE-14949.patch, HBASE-14949_v1.patch, > HBASE-14949_v2.patch > > > As HBASE-14004 design, there will be duplicate entries in different WAL. It > happens when one hflush failed, we will close old WAL with 'acked hflushed' > length, then open a new WAL and write the unacked hlushed entries into it. > So there maybe some overlap between old WAL and new WAL. > We should skip the duplicate entries when replay. I think it has no harm to > current logic, maybe we do it first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074678#comment-15074678
]
stack commented on HBASE-14790:
---
Soon as you have a basic WAL [~Apache9], I can try get some basic numbers? I'm
doing a bit of perf testing at mo on write pipeline.
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15044: --- Attachment: 15044-v7.txt > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.3.0 > > Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt, 15044-v4.txt, > 15044-v5.txt, 15044-v6.txt, 15044-v7.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074641#comment-15074641
]
Duo Zhang commented on HBASE-14790:
---
https://github.com/Apache9/hbase/blob/HBASE-14790/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FanOutOneBlockAsyncDFSOutput.java
Finally I found that there is no way to truncate a block from client side, only
NameNode can do it. If we do not want to change the protocol, the only way
without writing the pending data again is calling recoverLease. I will try to
add a 'truncateToMinRcvdBytes' flag in protocol when implementing it in HDFS,
but now I think this is enough. I will implement a WAL based on this and
collect some perf results.
Thanks.
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[
https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074639#comment-15074639
]
Hadoop QA commented on HBASE-15044:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779892/15044-v6.txt
against master branch at commit 822fead744a308df7ae45da440047207841d7abc.
ATTACHMENT ID: 12779892
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17071//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17071//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17071//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17071//console
This message is automatically generated.
> Region normalization should be allowed when underlying namespace has quota
> --
>
> Key: HBASE-15044
> URL: https://issues.apache.org/jira/browse/HBASE-15044
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt, 15044-v4.txt,
> 15044-v5.txt, 15044-v6.txt
>
>
> Currently when namespace has quota, HMaster#normalizeRegions() skips the
> tables in the namespace.
> However, performing region merge(s) wouldn't violate quota constraint.
> For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to
> check whether quota is about to be exceeded. If not, region split plan can
> still be executed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15050) Block Ref counting does not work in Region Split cases.
[ https://issues.apache.org/jira/browse/HBASE-15050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15050: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Block Ref counting does not work in Region Split cases. > --- > > Key: HBASE-15050 > URL: https://issues.apache.org/jira/browse/HBASE-15050 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15050.patch, HBASE-15050_1.patch, > HBASE-15050_2.patch, HBASE-15050_3.patch > > > The reference counting on the blocks does not work correctly when the > HalfStorefileReader is used for compaction/scans. > The reason is that getFirstKey and getLastKey API create a new scanner but > does not do the needed close() call and because of that we do not decrement > the count on the blocks. The same impact will also be observed on the ref > count that we maintain on the reader. Issue found when I was trying to test > some other feature with lot of evictions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15050) Block Ref counting does not work in Region Split cases.
[ https://issues.apache.org/jira/browse/HBASE-15050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15050: --- Attachment: HBASE-15050_3.patch Patch with test case that clearly highlights the problem. Will commit this shortly. > Block Ref counting does not work in Region Split cases. > --- > > Key: HBASE-15050 > URL: https://issues.apache.org/jira/browse/HBASE-15050 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15050.patch, HBASE-15050_1.patch, > HBASE-15050_2.patch, HBASE-15050_3.patch > > > The reference counting on the blocks does not work correctly when the > HalfStorefileReader is used for compaction/scans. > The reason is that getFirstKey and getLastKey API create a new scanner but > does not do the needed close() call and because of that we do not decrement > the count on the blocks. The same impact will also be observed on the ref > count that we maintain on the reader. Issue found when I was trying to test > some other feature with lot of evictions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15050) Block Ref counting does not work in Region Split cases.
[ https://issues.apache.org/jira/browse/HBASE-15050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15050: --- Status: Open (was: Patch Available) > Block Ref counting does not work in Region Split cases. > --- > > Key: HBASE-15050 > URL: https://issues.apache.org/jira/browse/HBASE-15050 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15050.patch, HBASE-15050_1.patch, > HBASE-15050_2.patch > > > The reference counting on the blocks does not work correctly when the > HalfStorefileReader is used for compaction/scans. > The reason is that getFirstKey and getLastKey API create a new scanner but > does not do the needed close() call and because of that we do not decrement > the count on the blocks. The same impact will also be observed on the ref > count that we maintain on the reader. Issue found when I was trying to test > some other feature with lot of evictions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14888) ClusterSchema: Add Namespace Operations
[
https://issues.apache.org/jira/browse/HBASE-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074624#comment-15074624
]
Hadoop QA commented on HBASE-14888:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779898/14888v11.patch
against master branch at commit 5e2c2e1780d57f2e0086b10cd28d5212d6a3d5d0.
ATTACHMENT ID: 12779898
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17072//console
This message is automatically generated.
> ClusterSchema: Add Namespace Operations
> ---
>
> Key: HBASE-14888
> URL: https://issues.apache.org/jira/browse/HBASE-14888
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments:
> 0001-Add-in-a-ClusterSchema-Interface.-It-will-have-all-Av2.patch,
> 14888.patch, 14888.v8.txt, 14888v11.patch, 14888v3.txt, 14888v4.txt,
> 14888v5.txt, 14888v6.txt, 14888v7.txt, 14888v9.txt
>
>
> Add in a ClusterSchema Interface. It will have all API for all cluster
> manipulation; adding namespaces, tables, amending column families, etc. The
> idea is to gather up our mess and put it all behind a tidy API that all works
> the same way whatever you changing returning a Future to wait on and behind
> the scenes driving Procedures.
> This patch does namespace operations first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14888) ClusterSchema: Add Namespace Operations
[ https://issues.apache.org/jira/browse/HBASE-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14888: -- Attachment: 14888v11.patch Fix missing annotation and address checkstyle complaints. > ClusterSchema: Add Namespace Operations > --- > > Key: HBASE-14888 > URL: https://issues.apache.org/jira/browse/HBASE-14888 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: > 0001-Add-in-a-ClusterSchema-Interface.-It-will-have-all-Av2.patch, > 14888.patch, 14888.v8.txt, 14888v11.patch, 14888v3.txt, 14888v4.txt, > 14888v5.txt, 14888v6.txt, 14888v7.txt, 14888v9.txt > > > Add in a ClusterSchema Interface. It will have all API for all cluster > manipulation; adding namespaces, tables, amending column families, etc. The > idea is to gather up our mess and put it all behind a tidy API that all works > the same way whatever you changing returning a Future to wait on and behind > the scenes driving Procedures. > This patch does namespace operations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074605#comment-15074605 ] Heng Chen commented on HBASE-15044: --- LGTM except the useless variable 'checkQuota'. > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.3.0 > > Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt, 15044-v4.txt, > 15044-v5.txt, 15044-v6.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074601#comment-15074601 ] Vladimir Rodionov commented on HBASE-14030: --- [~stack] The design doc is here: HBASE-7912, version (v6), last one. > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v3.patch, HBASE-14030-v4.patch, > HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, > HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15047) Try spin lock for MVCC completion
[ https://issues.apache.org/jira/browse/HBASE-15047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074590#comment-15074590 ] Duo Zhang commented on HBASE-15047: --- OK, I should add a limitation 'in production'. In micro benchmark, yes, it can be much faster. But in production environment, as [~stack] said above, thread scheduling is not controlled by us so we can hardly tell how long a spin will continue thus the spin may waste lots of CPU time. And see this paper "Ad Hoc Synchronization Considered Harmful" https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Xiong.pdf The key point here is "Ad Hoc Synchronization" is not easy to understand so people will easily break your initial assumption and cause a very bad performance impact. Thanks. > Try spin lock for MVCC completion > - > > Key: HBASE-15047 > URL: https://issues.apache.org/jira/browse/HBASE-15047 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-15047.patch > > > Waits/Notify is very very expensive since it can cost a thread scheduling. > There should only ever be a few threads ( < Num Cores ) running. So it should > be possible to spin and use compare and swap to update the read point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15047) Try spin lock for MVCC completion
[ https://issues.apache.org/jira/browse/HBASE-15047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074582#comment-15074582 ] Elliott Clark commented on HBASE-15047: --- bq.And usually I do not think a simple spin can perform better than standard locking method. http://askldjd.com/2010/05/16/spin-wait-vs-condition-variable/ http://stackoverflow.com/questions/5869825/when-should-one-use-a-spinlock-instead-of-mutex/5870415#5870415 http://www.alexonlinux.com/pthread-mutex-vs-pthread-spinlock Spin locking is much faster than requesting the operating system to de-schedule and re-schedule a thread. > Try spin lock for MVCC completion > - > > Key: HBASE-15047 > URL: https://issues.apache.org/jira/browse/HBASE-15047 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-15047.patch > > > Waits/Notify is very very expensive since it can cost a thread scheduling. > There should only ever be a few threads ( < Num Cores ) running. So it should > be possible to spin and use compare and swap to update the read point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1
[
https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074568#comment-15074568
]
Hadoop QA commented on HBASE-14030:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779864/HBASE-14030-v26.patch
against master branch at commit 822fead744a308df7ae45da440047207841d7abc.
ATTACHMENT ID: 12779864
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 23 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:red}-1 javac{color}. The applied patch generated 43 javac compiler
warnings (more than the master's current 35 warnings).
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17066//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17066//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17066//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17066//console
This message is automatically generated.
> HBase Backup/Restore Phase 1
>
>
> Key: HBASE-14030
> URL: https://issues.apache.org/jira/browse/HBASE-14030
> Project: HBase
> Issue Type: Umbrella
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch,
> HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch,
> HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch,
> HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch,
> HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch,
> HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch,
> HBASE-14030-v26.patch, HBASE-14030-v3.patch, HBASE-14030-v4.patch,
> HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch,
> HBASE-14030-v8.patch
>
>
> This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design
> doc for the phase description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[
https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074552#comment-15074552
]
Hadoop QA commented on HBASE-15044:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779862/15044-v4.txt
against master branch at commit 822fead744a308df7ae45da440047207841d7abc.
ATTACHMENT ID: 12779862
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17065//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17065//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17065//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17065//console
This message is automatically generated.
> Region normalization should be allowed when underlying namespace has quota
> --
>
> Key: HBASE-15044
> URL: https://issues.apache.org/jira/browse/HBASE-15044
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt, 15044-v4.txt,
> 15044-v5.txt, 15044-v6.txt
>
>
> Currently when namespace has quota, HMaster#normalizeRegions() skips the
> tables in the namespace.
> However, performing region merge(s) wouldn't violate quota constraint.
> For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to
> check whether quota is about to be exceeded. If not, region split plan can
> still be executed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15007) update Hadoop support matrix to list 2.6.1+ as supported
[ https://issues.apache.org/jira/browse/HBASE-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15007: Resolution: Fixed Status: Resolved (was: Patch Available) > update Hadoop support matrix to list 2.6.1+ as supported > > > Key: HBASE-15007 > URL: https://issues.apache.org/jira/browse/HBASE-15007 > Project: HBase > Issue Type: Task > Components: documentation, Operability >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0 > > Attachments: HBASE-15007.1.patch > > > The Hadoop community responded very well to our request for more maintenance > releases and have now put out 2.6.1 - 2.6.3. The first of those included the > fix for our catastrophic failure under HDFS encryption. > We should update the book to point out those versions are fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15044: --- Attachment: 15044-v6.txt Patch v6 fixes compilation. > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.3.0 > > Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt, 15044-v4.txt, > 15044-v5.txt, 15044-v6.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15007) update Hadoop support matrix to list 2.6.1+ as supported
[ https://issues.apache.org/jira/browse/HBASE-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074540#comment-15074540 ] Sean Busbey commented on HBASE-15007: - I tried to make it mirror the phrasing on the 2.7 line, using "2.6.1+" to encompass all the later versions. > update Hadoop support matrix to list 2.6.1+ as supported > > > Key: HBASE-15007 > URL: https://issues.apache.org/jira/browse/HBASE-15007 > Project: HBase > Issue Type: Task > Components: documentation, Operability >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0 > > Attachments: HBASE-15007.1.patch > > > The Hadoop community responded very well to our request for more maintenance > releases and have now put out 2.6.1 - 2.6.3. The first of those included the > fix for our catastrophic failure under HDFS encryption. > We should update the book to point out those versions are fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14888) ClusterSchema: Add Namespace Operations
[
https://issues.apache.org/jira/browse/HBASE-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074539#comment-15074539
]
Hadoop QA commented on HBASE-14888:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779880/14888v9.txt
against master branch at commit 822fead744a308df7ae45da440047207841d7abc.
ATTACHMENT ID: 12779880
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestInterfaceAudienceAnnotations
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17068//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17068//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17068//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17068//console
This message is automatically generated.
> ClusterSchema: Add Namespace Operations
> ---
>
> Key: HBASE-14888
> URL: https://issues.apache.org/jira/browse/HBASE-14888
> Project: HBase
> Issue Type: Sub-task
> Components: API
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments:
> 0001-Add-in-a-ClusterSchema-Interface.-It-will-have-all-Av2.patch,
> 14888.patch, 14888.v8.txt, 14888v3.txt, 14888v4.txt, 14888v5.txt,
> 14888v6.txt, 14888v7.txt, 14888v9.txt
>
>
> Add in a ClusterSchema Interface. It will have all API for all cluster
> manipulation; adding namespaces, tables, amending column families, etc. The
> idea is to gather up our mess and put it all behind a tidy API that all works
> the same way whatever you changing returning a Future to wait on and behind
> the scenes driving Procedures.
> This patch does namespace operations first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[
https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074537#comment-15074537
]
Hadoop QA commented on HBASE-15044:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779890/15044-v5.txt
against master branch at commit 822fead744a308df7ae45da440047207841d7abc.
ATTACHMENT ID: 12779890
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail with Hadoop version 2.4.0.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java:[215,3]
org.apache.hadoop.hbase.master.TestCatalogJanitor.MockMasterServices is not
abstract and does not override abstract method getRegionNormalizer() in
org.apache.hadoop.hbase.master.MasterServices
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:testCompile
(default-testCompile) on project hbase-server: Compilation failure
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java:[215,3]
org.apache.hadoop.hbase.master.TestCatalogJanitor.MockMasterServices is not
abstract and does not override abstract method getRegionNormalizer() in
org.apache.hadoop.hbase.master.MasterServices
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hbase-server
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17070//console
This message is automatically generated.
> Region normalization should be allowed when underlying namespace has quota
> --
>
> Key: HBASE-15044
> URL: https://issues.apache.org/jira/browse/HBASE-15044
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt, 15044-v4.txt,
> 15044-v5.txt
>
>
> Currently when namespace has quota, HMaster#normalizeRegions() skips the
> tables in the namespace.
> However, performing region merge(s) wouldn't violate quota constraint.
> For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to
> check whether quota is about to be exceeded. If not, region split plan can
> still be executed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074523#comment-15074523 ] stack commented on HBASE-14030: --- Where do I go read up on the plan? The issue that this one points to for design goes back to an issue with lots of talk and a few versions of a design up on it as well as some pretty fundamental issues w/ the approach. To review I need to review the whole issue this points at or just the last version of the design doc with your additions to it? > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v3.patch, HBASE-14030-v4.patch, > HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, > HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15044: --- Attachment: 15044-v5.txt Patch v5 addresses Heng's comment. Call to checkQuotaToSplitRegion() is avoided. Now quota info is managed by NamespaceQuotaManager > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.3.0 > > Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt, 15044-v4.txt, > 15044-v5.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15007) update Hadoop support matrix to list 2.6.1+ as supported
[ https://issues.apache.org/jira/browse/HBASE-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074516#comment-15074516 ] stack commented on HBASE-15007: --- +1 Pretty important. Want to mention later versions of 2.6? > update Hadoop support matrix to list 2.6.1+ as supported > > > Key: HBASE-15007 > URL: https://issues.apache.org/jira/browse/HBASE-15007 > Project: HBase > Issue Type: Task > Components: documentation, Operability >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0 > > Attachments: HBASE-15007.1.patch > > > The Hadoop community responded very well to our request for more maintenance > releases and have now put out 2.6.1 - 2.6.3. The first of those included the > fix for our catastrophic failure under HDFS encryption. > We should update the book to point out those versions are fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074503#comment-15074503
]
Jonathan Hsieh commented on HBASE-15052:
+1 if bot passes.
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.0.3, 0.98.16.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Attachments: HBASE-15052-v0.patch
>
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14987) Compaction marker whose region name doesn't match current region's needs to be handled
[
https://issues.apache.org/jira/browse/HBASE-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074498#comment-15074498
]
Hadoop QA commented on HBASE-14987:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779853/14987-v5.txt
against master branch at commit 822fead744a308df7ae45da440047207841d7abc.
ATTACHMENT ID: 12779853
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17063//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17063//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17063//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17063//console
This message is automatically generated.
> Compaction marker whose region name doesn't match current region's needs to
> be handled
> --
>
> Key: HBASE-14987
> URL: https://issues.apache.org/jira/browse/HBASE-14987
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Stephen Yuan Jiang
> Attachments: 14987-suggest.txt, 14987-v1.txt, 14987-v2.txt,
> 14987-v2.txt, 14987-v3.txt, 14987-v4.txt, 14987-v5.txt
>
>
> One customer encountered the following error when replaying recovered edits,
> leading to region open failure:
> {code}
> region=table1,d6b-2282-9223370590058224807-U-9856557-
> EJ452727-16313786400171,1449616291799.fa8a526f2578eb3630bb08a4b1648f5d.,
> starting to roll back the global memstore size.
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Compaction marker
> from WAL table_name: "table1"
> encoded_region_name: "d389c70fde9ec07971d0cfd20ef8f575"
> ...
> region_name:
> "table1,d6b-2282-9223370590058224807-U-9856557-EJ452727-16313786400171,1449089609367.d389c70fde9ec07971d0cfd20ef8f575."
> targetted for region d389c70fde9ec07971d0cfd20ef8f575 does not match this
> region: {ENCODED => fa8a526f2578eb3630bb08a4b1648f5d, NAME =>
> 'table1,d6b-2282-
> 9223370590058224807-U-9856557-EJ452727-16313786400171,1449616291799.fa8a526f2578eb3630bb08a4b1648f5d.',
> STARTKEY => 'd6b-2282-9223370590058224807-U-9856557-EJ452727-
> 16313786400171', ENDKEY =>
> 'd76-2553-9223370588576178807-U-7416904-EK875822-1766218060'}
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkTargetRegion(HRegion.java:4592)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALCompactionMarker(HRegion.java:3831)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3747)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3601)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:911)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:789)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:762)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5774)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5744)
> {code}
> This was likely caused by the following action of hbck:
> {code}
> 15/12/08 18:11:34 INFO util.HBaseFsck: [hbasefsck-pool1-t37] Movin
[jira] [Commented] (HBASE-15047) Try spin lock for MVCC completion
[ https://issues.apache.org/jira/browse/HBASE-15047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074495#comment-15074495 ] Duo Zhang commented on HBASE-15047: --- I think this is just a spin, not a spin lock? And usually I do not think a simple spin can perform better than standard locking method. Let's see some perfs results first? Thanks. > Try spin lock for MVCC completion > - > > Key: HBASE-15047 > URL: https://issues.apache.org/jira/browse/HBASE-15047 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-15047.patch > > > Waits/Notify is very very expensive since it can cost a thread scheduling. > There should only ever be a few threads ( < Num Cores ) running. So it should > be possible to spin and use compare and swap to update the read point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-15052:
Affects Version/s: 1.2.0
2.0.0
1.1.2
1.0.3
0.98.16.1
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.0.3, 0.98.16.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Attachments: HBASE-15052-v0.patch
>
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-15052:
Status: Patch Available (was: Open)
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Attachments: HBASE-15052-v0.patch
>
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-15052:
Attachment: HBASE-15052-v0.patch
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Attachments: HBASE-15052-v0.patch
>
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-15052:
Attachment: HBASE-15052-v0.patch
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
Matteo Bertozzi created HBASE-15052:
---
Summary: Use EnvironmentEdgeManager in ReplicationSource
Key: HBASE-15052
URL: https://issues.apache.org/jira/browse/HBASE-15052
Project: HBase
Issue Type: Bug
Components: Replication
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
ReplicationSource is passing System.currentTimeMillis() to
MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
EnvironmentEdgeManager.currentTime().
{code}
// if there was nothing to ship and it's not an error
// set "ageOfLastShippedOp" to to indicate that we're current
metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
long age = EnvironmentEdgeManager.currentTime() - timestamp;
{code}
we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-15052:
Attachment: (was: HBASE-15052-v0.patch)
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14888) ClusterSchema: Add Namespace Operations
[ https://issues.apache.org/jira/browse/HBASE-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14888: -- Attachment: 14888v9.txt Address the hang in tests on startup (I'm pretty sure I fixed this once before... ) Remove mess from HBaseAdmin. > ClusterSchema: Add Namespace Operations > --- > > Key: HBASE-14888 > URL: https://issues.apache.org/jira/browse/HBASE-14888 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: > 0001-Add-in-a-ClusterSchema-Interface.-It-will-have-all-Av2.patch, > 14888.patch, 14888.v8.txt, 14888v3.txt, 14888v4.txt, 14888v5.txt, > 14888v6.txt, 14888v7.txt, 14888v9.txt > > > Add in a ClusterSchema Interface. It will have all API for all cluster > manipulation; adding namespaces, tables, amending column families, etc. The > idea is to gather up our mess and put it all behind a tidy API that all works > the same way whatever you changing returning a Future to wait on and behind > the scenes driving Procedures. > This patch does namespace operations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15035) bulkloading hfiles with tags that require splits do not preserve tags
[ https://issues.apache.org/jira/browse/HBASE-15035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074463#comment-15074463 ] Jonathan Hsieh commented on HBASE-15035: I checked and you are right. I've filed HBASE-15051 to cover your suggestion and other cleanup around HFileContext so that these bugs are harder to write. > bulkloading hfiles with tags that require splits do not preserve tags > - > > Key: HBASE-15035 > URL: https://issues.apache.org/jira/browse/HBASE-15035 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.98.0, 1.0.0, 2.0.0, 1.1.0, 1.2.0, 1.3.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4 > > Attachments: HBASE-15035-v2.patch, HBASE-15035-v3.patch, > HBASE-15035-v4.patch, HBASE-15035.patch > > > When an hfile is created with cell tags present and it is bulk loaded into > hbase the tags will be present when loaded into a single region. If the bulk > load hfile spans multiple regions, bulk load automatically splits the > original hfile into a set of split hfiles corresponding to each of the > regions that the original covers. > Since 0.98, tags are not copied into the newly created split hfiles. (the > default for "includeTags" of the HFileContextBuilder [1] is uninitialized > which defaults to false). This means acls, ttls, mob pointers and other tag > stored values will not be bulk loaded in. > [1] > https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java#L40 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15051) Refactor HFileReaderImpl HFIleContext usage
Jonathan Hsieh created HBASE-15051: -- Summary: Refactor HFileReaderImpl HFIleContext usage Key: HBASE-15051 URL: https://issues.apache.org/jira/browse/HBASE-15051 Project: HBase Issue Type: Improvement Reporter: Jonathan Hsieh In discusssion from HBASE-15035, [~ram_krish] noted a different approach for guranteeing that HFileContext settings are passed in bulk load. This patch will take that idea and extend. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15047) Try spin lock for MVCC completion
[ https://issues.apache.org/jira/browse/HBASE-15047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074440#comment-15074440 ] Elliott Clark commented on HBASE-15047: --- bq.Radical. I'm on a mission to remove locks. Every lock we have is a place for contention and long wait times. The jury is still out on this instance being worth it though. bq.who says the currently scheduled thread is next in line That shouldn't matter as long as someone is next. They should all be waiting. bq.what is to prevent all CPUs being here spinning with no thread free to make forward progess Yeah that's the possible downside. That you have so many threads using cpu that you can't get any io done. If needed we can add a sleep if the spin has been going too long. However I want to try these things before adding more complexity. > Try spin lock for MVCC completion > - > > Key: HBASE-15047 > URL: https://issues.apache.org/jira/browse/HBASE-15047 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-15047.patch > > > Waits/Notify is very very expensive since it can cost a thread scheduling. > There should only ever be a few threads ( < Num Cores ) running. So it should > be possible to spin and use compare and swap to update the read point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15047) Try spin lock for MVCC completion
[ https://issues.apache.org/jira/browse/HBASE-15047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074434#comment-15074434 ] stack commented on HBASE-15047: --- Radical. At first blush, we will be waiting too long in mvcc to spin and who says the currently scheduled thread is next in line for mvcc and what is to prevent all CPUs being here spinning with no thread free to make forward progess? I'll try it though. > Try spin lock for MVCC completion > - > > Key: HBASE-15047 > URL: https://issues.apache.org/jira/browse/HBASE-15047 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-15047.patch > > > Waits/Notify is very very expensive since it can cost a thread scheduling. > There should only ever be a few threads ( < Num Cores ) running. So it should > be possible to spin and use compare and swap to update the read point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14853) Add on protobuf to c++ chain
[
https://issues.apache.org/jira/browse/HBASE-14853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074409#comment-15074409
]
Hadoop QA commented on HBASE-14853:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779865/HBASE-14853-v2.patch
against master branch at commit 822fead744a308df7ae45da440047207841d7abc.
ATTACHMENT ID: 12779865
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation, build,
or dev-support patch that doesn't require tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17064//console
This message is automatically generated.
> Add on protobuf to c++ chain
>
>
> Key: HBASE-14853
> URL: https://issues.apache.org/jira/browse/HBASE-14853
> Project: HBase
> Issue Type: Sub-task
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-14853-v1.patch, HBASE-14853-v2.patch,
> HBASE-14853.patch
>
>
> We have protobufs.
> We need c++ libraries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15047) Try spin lock for MVCC completion
[ https://issues.apache.org/jira/browse/HBASE-15047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15047: -- Attachment: HBASE-15047.patch This will greatly change the way perf/flame graphs look. We will now be in user mode code a lot more. However that should result in much faster response times as there should be no scheduler overhead. > Try spin lock for MVCC completion > - > > Key: HBASE-15047 > URL: https://issues.apache.org/jira/browse/HBASE-15047 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-15047.patch > > > Waits/Notify is very very expensive since it can cost a thread scheduling. > There should only ever be a few threads ( < Num Cores ) running. So it should > be possible to spin and use compare and swap to update the read point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15047) Try spin lock for MVCC completion
[ https://issues.apache.org/jira/browse/HBASE-15047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15047: -- Status: Patch Available (was: Open) > Try spin lock for MVCC completion > - > > Key: HBASE-15047 > URL: https://issues.apache.org/jira/browse/HBASE-15047 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-15047.patch > > > Waits/Notify is very very expensive since it can cost a thread scheduling. > There should only ever be a few threads ( < Num Cores ) running. So it should > be possible to spin and use compare and swap to update the read point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15047) Try spin lock for MVCC completion
[ https://issues.apache.org/jira/browse/HBASE-15047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074402#comment-15074402 ] Elliott Clark commented on HBASE-15047: --- https://reviews.facebook.net/D52407 > Try spin lock for MVCC completion > - > > Key: HBASE-15047 > URL: https://issues.apache.org/jira/browse/HBASE-15047 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > > Waits/Notify is very very expensive since it can cost a thread scheduling. > There should only ever be a few threads ( < Num Cores ) running. So it should > be possible to spin and use compare and swap to update the read point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14888) ClusterSchema: Add Namespace Operations
[ https://issues.apache.org/jira/browse/HBASE-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074398#comment-15074398 ] stack commented on HBASE-14888: --- [~jmhsieh] No. The change in hbaseadmin is detritus. Let me fix. Tests used to pass but running locally I see hang up. Let me check... > ClusterSchema: Add Namespace Operations > --- > > Key: HBASE-14888 > URL: https://issues.apache.org/jira/browse/HBASE-14888 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: > 0001-Add-in-a-ClusterSchema-Interface.-It-will-have-all-Av2.patch, > 14888.patch, 14888.v8.txt, 14888v3.txt, 14888v4.txt, 14888v5.txt, > 14888v6.txt, 14888v7.txt > > > Add in a ClusterSchema Interface. It will have all API for all cluster > manipulation; adding namespaces, tables, amending column families, etc. The > idea is to gather up our mess and put it all behind a tidy API that all works > the same way whatever you changing returning a Future to wait on and behind > the scenes driving Procedures. > This patch does namespace operations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14853) Add on protobuf to c++ chain
[ https://issues.apache.org/jira/browse/HBASE-14853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074390#comment-15074390 ] stack commented on HBASE-14853: --- Why you have to copy the files? You can't have build generate from their current location (I'm sure you tried it and a reasons why it don't work). Would it be silly doing git submodule in this 'if' subdir? > Add on protobuf to c++ chain > > > Key: HBASE-14853 > URL: https://issues.apache.org/jira/browse/HBASE-14853 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-14853-v1.patch, HBASE-14853-v2.patch, > HBASE-14853.patch > > > We have protobufs. > We need c++ libraries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15011) turn off the jdk8 javadoc linter. :(
[
https://issues.apache.org/jira/browse/HBASE-15011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074389#comment-15074389
]
Hudson commented on HBASE-15011:
SUCCESS: Integrated in HBase-1.3 #476 (See
[https://builds.apache.org/job/HBase-1.3/476/])
HBASE-15011 turn off the jdk8 javadoc linter. (busbey: rev
9cc01b4029d8159dd528b0dc413464285d07fbfc)
* pom.xml
> turn off the jdk8 javadoc linter. :(
>
>
> Key: HBASE-15011
> URL: https://issues.apache.org/jira/browse/HBASE-15011
> Project: HBase
> Issue Type: Bug
> Components: build, documentation
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Labels: beginner
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15011.1.patch, HBASE-15011.2.patch,
> HBASE-15011.2.patch, HBASE-15011.3.patch
>
>
> there's a new javadoc warning that causes warnings on all of our new patches
> and (I believe) breaks us on jdk8.
> Thanks to [~saint@gmail.com] for chasing it down on HBASE-14849.
> {code}
> 1 warning
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java:430:
> warning - @param argument "hfilesDir" is not a parameter name.
> [INFO]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[
https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074385#comment-15074385
]
Heng Chen commented on HBASE-15044:
---
{quote}
I think this is fine. The master is going to execute region split plan which
would result in increment of region count for the underlying table.
{quote}
I have two questions:
* could we handle split plan failed if we update quota info here?
* If we update quota info when plan type is split, should we update quota
when split plan is merge?
> Region normalization should be allowed when underlying namespace has quota
> --
>
> Key: HBASE-15044
> URL: https://issues.apache.org/jira/browse/HBASE-15044
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt, 15044-v4.txt
>
>
> Currently when namespace has quota, HMaster#normalizeRegions() skips the
> tables in the namespace.
> However, performing region merge(s) wouldn't violate quota constraint.
> For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to
> check whether quota is about to be exceeded. If not, region split plan can
> still be executed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15023) Reenable TestShell and TestStochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074371#comment-15074371 ] Hudson commented on HBASE-15023: FAILURE: Integrated in HBase-1.2 #481 (See [https://builds.apache.org/job/HBase-1.2/481/]) HBASE-15023 Reenable TestShell and TestStochasticLoadBalancer; (stack: rev f2338be4d84d1cff1eb27e87c1302f9d896d4744) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java > Reenable TestShell and TestStochasticLoadBalancer > - > > Key: HBASE-15023 > URL: https://issues.apache.org/jira/browse/HBASE-15023 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15023.patch, 15023.patch > > > Parent disabled these tests when test runs were flakier than they are now. > Try reenabling them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15011) turn off the jdk8 javadoc linter. :(
[
https://issues.apache.org/jira/browse/HBASE-15011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074372#comment-15074372
]
Hudson commented on HBASE-15011:
FAILURE: Integrated in HBase-1.2 #481 (See
[https://builds.apache.org/job/HBase-1.2/481/])
HBASE-15011 turn off the jdk8 javadoc linter. (busbey: rev
02ad6504cd5c9b65833cd5034cbca233665c398d)
* pom.xml
> turn off the jdk8 javadoc linter. :(
>
>
> Key: HBASE-15011
> URL: https://issues.apache.org/jira/browse/HBASE-15011
> Project: HBase
> Issue Type: Bug
> Components: build, documentation
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Labels: beginner
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15011.1.patch, HBASE-15011.2.patch,
> HBASE-15011.2.patch, HBASE-15011.3.patch
>
>
> there's a new javadoc warning that causes warnings on all of our new patches
> and (I believe) breaks us on jdk8.
> Thanks to [~saint@gmail.com] for chasing it down on HBASE-14849.
> {code}
> 1 warning
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java:430:
> warning - @param argument "hfilesDir" is not a parameter name.
> [INFO]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15031) Fix merge of MVCC and SequenceID performance regression in branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074357#comment-15074357 ] stack commented on HBASE-15031: --- Addendum added test files left out on first commit (spotted by [~mbertozzi]) > Fix merge of MVCC and SequenceID performance regression in branch-1.0 > - > > Key: HBASE-15031 > URL: https://issues.apache.org/jira/browse/HBASE-15031 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 1.0.3 >Reporter: stack >Assignee: stack > Fix For: 1.1.3, 1.0.4 > > Attachments: 14460.v0.branch-1.0.patch, 15031.v2.branch-1.0.patch, > 15031.v3.branch-1.0.patch, 15031.v4.branch-1.0.patch, > 15031.v5.branch-1.0.patch, 15031.v6.branch-1.0.patch, > 15031.v6.branch-1.0.patch, 15031.v6.branch-1.0.patch, > 15031.v7.branch-1.0.patch, 15031.v8.branch-1.0.patch > > > Subtask with fix for branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14853) Add on protobuf to c++ chain
[ https://issues.apache.org/jira/browse/HBASE-14853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14853: -- Attachment: HBASE-14853-v2.patch > Add on protobuf to c++ chain > > > Key: HBASE-14853 > URL: https://issues.apache.org/jira/browse/HBASE-14853 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-14853-v1.patch, HBASE-14853-v2.patch, > HBASE-14853.patch > > > We have protobufs. > We need c++ libraries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14030: -- Attachment: HBASE-14030-v26.patch v26. small checkstyle fix > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v3.patch, HBASE-14030-v4.patch, > HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, > HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14030: -- Status: Patch Available (was: Open) > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v3.patch, HBASE-14030-v4.patch, > HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, > HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14030: -- Status: Open (was: Patch Available) > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v3.patch, HBASE-14030-v4.patch, HBASE-14030-v5.patch, > HBASE-14030-v6.patch, HBASE-14030-v7.patch, HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15044: --- Hadoop Flags: Reviewed Fix Version/s: 1.3.0 2.0.0 > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.3.0 > > Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt, 15044-v4.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15044: --- Attachment: 15044-v4.txt Patch v4 addresses checkstyle warning > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt, 15044-v4.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15007) update Hadoop support matrix to list 2.6.1+ as supported
[
https://issues.apache.org/jira/browse/HBASE-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074328#comment-15074328
]
Hadoop QA commented on HBASE-15007:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779839/HBASE-15007.1.patch
against master branch at commit 822fead744a308df7ae45da440047207841d7abc.
ATTACHMENT ID: 12779839
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17061//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17061//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17061//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17061//console
This message is automatically generated.
> update Hadoop support matrix to list 2.6.1+ as supported
>
>
> Key: HBASE-15007
> URL: https://issues.apache.org/jira/browse/HBASE-15007
> Project: HBase
> Issue Type: Task
> Components: documentation, Operability
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Fix For: 2.0.0
>
> Attachments: HBASE-15007.1.patch
>
>
> The Hadoop community responded very well to our request for more maintenance
> releases and have now put out 2.6.1 - 2.6.3. The first of those included the
> fix for our catastrophic failure under HDFS encryption.
> We should update the book to point out those versions are fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[
https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074324#comment-15074324
]
Hadoop QA commented on HBASE-15044:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779841/15044-v3.txt
against master branch at commit 822fead744a308df7ae45da440047207841d7abc.
ATTACHMENT ID: 12779841
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17062//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17062//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17062//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17062//console
This message is automatically generated.
> Region normalization should be allowed when underlying namespace has quota
> --
>
> Key: HBASE-15044
> URL: https://issues.apache.org/jira/browse/HBASE-15044
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt
>
>
> Currently when namespace has quota, HMaster#normalizeRegions() skips the
> tables in the namespace.
> However, performing region merge(s) wouldn't violate quota constraint.
> For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to
> check whether quota is about to be exceeded. If not, region split plan can
> still be executed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15011) turn off the jdk8 javadoc linter. :(
[
https://issues.apache.org/jira/browse/HBASE-15011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074254#comment-15074254
]
Hudson commented on HBASE-15011:
FAILURE: Integrated in HBase-1.2-IT #372 (See
[https://builds.apache.org/job/HBase-1.2-IT/372/])
HBASE-15011 turn off the jdk8 javadoc linter. (busbey: rev
02ad6504cd5c9b65833cd5034cbca233665c398d)
* pom.xml
> turn off the jdk8 javadoc linter. :(
>
>
> Key: HBASE-15011
> URL: https://issues.apache.org/jira/browse/HBASE-15011
> Project: HBase
> Issue Type: Bug
> Components: build, documentation
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Labels: beginner
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15011.1.patch, HBASE-15011.2.patch,
> HBASE-15011.2.patch, HBASE-15011.3.patch
>
>
> there's a new javadoc warning that causes warnings on all of our new patches
> and (I believe) breaks us on jdk8.
> Thanks to [~saint@gmail.com] for chasing it down on HBASE-14849.
> {code}
> 1 warning
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java:430:
> warning - @param argument "hfilesDir" is not a parameter name.
> [INFO]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15023) Reenable TestShell and TestStochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074253#comment-15074253 ] Hudson commented on HBASE-15023: FAILURE: Integrated in HBase-1.2-IT #372 (See [https://builds.apache.org/job/HBase-1.2-IT/372/]) HBASE-15023 Reenable TestShell and TestStochasticLoadBalancer; (stack: rev f2338be4d84d1cff1eb27e87c1302f9d896d4744) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java > Reenable TestShell and TestStochasticLoadBalancer > - > > Key: HBASE-15023 > URL: https://issues.apache.org/jira/browse/HBASE-15023 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15023.patch, 15023.patch > > > Parent disabled these tests when test runs were flakier than they are now. > Try reenabling them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14987) Compaction marker whose region name doesn't match current region's needs to be handled
[
https://issues.apache.org/jira/browse/HBASE-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14987:
---
Attachment: 14987-v5.txt
Patch v5 selectively skips replaying compaction marker for primary region.
> Compaction marker whose region name doesn't match current region's needs to
> be handled
> --
>
> Key: HBASE-14987
> URL: https://issues.apache.org/jira/browse/HBASE-14987
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Stephen Yuan Jiang
> Attachments: 14987-suggest.txt, 14987-v1.txt, 14987-v2.txt,
> 14987-v2.txt, 14987-v3.txt, 14987-v4.txt, 14987-v5.txt
>
>
> One customer encountered the following error when replaying recovered edits,
> leading to region open failure:
> {code}
> region=table1,d6b-2282-9223370590058224807-U-9856557-
> EJ452727-16313786400171,1449616291799.fa8a526f2578eb3630bb08a4b1648f5d.,
> starting to roll back the global memstore size.
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Compaction marker
> from WAL table_name: "table1"
> encoded_region_name: "d389c70fde9ec07971d0cfd20ef8f575"
> ...
> region_name:
> "table1,d6b-2282-9223370590058224807-U-9856557-EJ452727-16313786400171,1449089609367.d389c70fde9ec07971d0cfd20ef8f575."
> targetted for region d389c70fde9ec07971d0cfd20ef8f575 does not match this
> region: {ENCODED => fa8a526f2578eb3630bb08a4b1648f5d, NAME =>
> 'table1,d6b-2282-
> 9223370590058224807-U-9856557-EJ452727-16313786400171,1449616291799.fa8a526f2578eb3630bb08a4b1648f5d.',
> STARTKEY => 'd6b-2282-9223370590058224807-U-9856557-EJ452727-
> 16313786400171', ENDKEY =>
> 'd76-2553-9223370588576178807-U-7416904-EK875822-1766218060'}
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkTargetRegion(HRegion.java:4592)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayWALCompactionMarker(HRegion.java:3831)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3747)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3601)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:911)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:789)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:762)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5774)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5744)
> {code}
> This was likely caused by the following action of hbck:
> {code}
> 15/12/08 18:11:34 INFO util.HBaseFsck: [hbasefsck-pool1-t37] Moving files
> from
> hdfs://Zealand/hbase/data/default/table1/d389c70fde9ec07971d0cfd20ef8f575/recovered.edits
> into containing region
> hdfs://Zealand/hbase/data/default/table1/fa8a526f2578eb3630bb08a4b1648f5d/recovered.edits
> {code}
> The recovered.edits for d389c70fde9ec07971d0cfd20ef8f575 contained compaction
> marker which couldn't be replayed against fa8a526f2578eb3630bb08a4b1648f5d
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15043) region_status.rb broken with TypeError: no public constructors for Java::OrgApacheHadoopHbaseClient::HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-15043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074226#comment-15074226 ] Hudson commented on HBASE-15043: FAILURE: Integrated in HBase-Trunk_matrix #597 (See [https://builds.apache.org/job/HBase-Trunk_matrix/597/]) HBASE-15043 region_status.rb broken with TypeError: no public (tedyu: rev 9b8895ba295ee66fefda115f30ab727590e487e8) * bin/region_status.rb * bin/replication/copy_tables_desc.rb * bin/shutdown_regionserver.rb * bin/draining_servers.rb > region_status.rb broken with TypeError: no public constructors for > Java::OrgApacheHadoopHbaseClient::HBaseAdmin > --- > > Key: HBASE-15043 > URL: https://issues.apache.org/jira/browse/HBASE-15043 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic > Fix For: 2.0.0 > > Attachments: HBASE-15043-v2.patch, HBASE-15043-v3.patch, > HBASE-15043.patch > > > Here is full error: > TypeError: no public constructors for > Java::OrgApacheHadoopHbaseClient::HBaseAdmin > (root) at bin/region_status.rb:75 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15011) turn off the jdk8 javadoc linter. :(
[
https://issues.apache.org/jira/browse/HBASE-15011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074227#comment-15074227
]
Hudson commented on HBASE-15011:
FAILURE: Integrated in HBase-Trunk_matrix #597 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/597/])
HBASE-15011 turn off the jdk8 javadoc linter. (busbey: rev
822fead744a308df7ae45da440047207841d7abc)
* pom.xml
> turn off the jdk8 javadoc linter. :(
>
>
> Key: HBASE-15011
> URL: https://issues.apache.org/jira/browse/HBASE-15011
> Project: HBase
> Issue Type: Bug
> Components: build, documentation
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Labels: beginner
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15011.1.patch, HBASE-15011.2.patch,
> HBASE-15011.2.patch, HBASE-15011.3.patch
>
>
> there's a new javadoc warning that causes warnings on all of our new patches
> and (I believe) breaks us on jdk8.
> Thanks to [~saint@gmail.com] for chasing it down on HBASE-14849.
> {code}
> 1 warning
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java:430:
> warning - @param argument "hfilesDir" is not a parameter name.
> [INFO]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14867) SimpleRegionNormalizer needs to have better heuristics to trigger merge operation
[ https://issues.apache.org/jira/browse/HBASE-14867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074228#comment-15074228 ] Hudson commented on HBASE-14867: FAILURE: Integrated in HBase-Trunk_matrix #597 (See [https://builds.apache.org/job/HBase-Trunk_matrix/597/]) HBASE-14867 SimpleRegionNormalizer needs to have better heuristics to (tedyu: rev 1e4992c6eccb81166cdda842a68644fa962a3fdc) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java > SimpleRegionNormalizer needs to have better heuristics to trigger merge > operation > - > > Key: HBASE-14867 > URL: https://issues.apache.org/jira/browse/HBASE-14867 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.2.0 >Reporter: Romil Choksi >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14867-v2.txt, 14867-v3.txt, 14867-v4.txt > > > SimpleRegionNormalizer needs to have better heuristics to trigger merge > operation. SimpleRegionNormalizer is not able to trigger a merge action if > the table's smallest region has neighboring regions that are larger than > table's average region size, whereas there are other smaller regions whose > combined size is less than the average region size. > For example, > - Consider a table with six region, say r1 to r6. > - Keep r1 as empty and create some data say, 100K rows of data for each of > the regions r2, r3 and r4. Create smaller amount of data for regions r5 and > r6, say about 27K rows of data. > - Run the normalizer. Verify the number the regions for that table and also > check the master log to see if any merge action was triggered as a result of > normalization. > In such scenario, it would be better to have a merge action triggered for > those two smaller regions r5 and r6 even though either of them is not the > smallest one -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14867) SimpleRegionNormalizer needs to have better heuristics to trigger merge operation
[ https://issues.apache.org/jira/browse/HBASE-14867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074188#comment-15074188 ] Hudson commented on HBASE-14867: SUCCESS: Integrated in HBase-1.3-IT #416 (See [https://builds.apache.org/job/HBase-1.3-IT/416/]) HBASE-14867 SimpleRegionNormalizer needs to have better heuristics to (tedyu: rev c4173777dd776b23c9a09b3eca827690fc6ee840) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java > SimpleRegionNormalizer needs to have better heuristics to trigger merge > operation > - > > Key: HBASE-14867 > URL: https://issues.apache.org/jira/browse/HBASE-14867 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.2.0 >Reporter: Romil Choksi >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14867-v2.txt, 14867-v3.txt, 14867-v4.txt > > > SimpleRegionNormalizer needs to have better heuristics to trigger merge > operation. SimpleRegionNormalizer is not able to trigger a merge action if > the table's smallest region has neighboring regions that are larger than > table's average region size, whereas there are other smaller regions whose > combined size is less than the average region size. > For example, > - Consider a table with six region, say r1 to r6. > - Keep r1 as empty and create some data say, 100K rows of data for each of > the regions r2, r3 and r4. Create smaller amount of data for regions r5 and > r6, say about 27K rows of data. > - Run the normalizer. Verify the number the regions for that table and also > check the master log to see if any merge action was triggered as a result of > normalization. > In such scenario, it would be better to have a merge action triggered for > those two smaller regions r5 and r6 even though either of them is not the > smallest one -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15011) turn off the jdk8 javadoc linter. :(
[
https://issues.apache.org/jira/browse/HBASE-15011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074187#comment-15074187
]
Hudson commented on HBASE-15011:
SUCCESS: Integrated in HBase-1.3-IT #416 (See
[https://builds.apache.org/job/HBase-1.3-IT/416/])
HBASE-15011 turn off the jdk8 javadoc linter. (busbey: rev
9cc01b4029d8159dd528b0dc413464285d07fbfc)
* pom.xml
> turn off the jdk8 javadoc linter. :(
>
>
> Key: HBASE-15011
> URL: https://issues.apache.org/jira/browse/HBASE-15011
> Project: HBase
> Issue Type: Bug
> Components: build, documentation
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Labels: beginner
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15011.1.patch, HBASE-15011.2.patch,
> HBASE-15011.2.patch, HBASE-15011.3.patch
>
>
> there's a new javadoc warning that causes warnings on all of our new patches
> and (I believe) breaks us on jdk8.
> Thanks to [~saint@gmail.com] for chasing it down on HBASE-14849.
> {code}
> 1 warning
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java:430:
> warning - @param argument "hfilesDir" is not a parameter name.
> [INFO]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15023) Reenable TestShell and TestStochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074186#comment-15074186 ] Hudson commented on HBASE-15023: SUCCESS: Integrated in HBase-1.3-IT #416 (See [https://builds.apache.org/job/HBase-1.3-IT/416/]) HBASE-15023 Reenable TestShell and TestStochasticLoadBalancer (stack: rev 1098dfd918f76eb6cc6ccd2a6b423e876d782f0d) * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java > Reenable TestShell and TestStochasticLoadBalancer > - > > Key: HBASE-15023 > URL: https://issues.apache.org/jira/browse/HBASE-15023 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15023.patch, 15023.patch > > > Parent disabled these tests when test runs were flakier than they are now. > Try reenabling them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14804) HBase shell's create table command ignores 'NORMALIZATION_ENABLED' attribute
[
https://issues.apache.org/jira/browse/HBASE-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14804:
---
Fix Version/s: (was: 0.98.17)
Dropped the patch from 0.98 due to normalization feature absent in 0.98 branch
> HBase shell's create table command ignores 'NORMALIZATION_ENABLED' attribute
>
>
> Key: HBASE-14804
> URL: https://issues.apache.org/jira/browse/HBASE-14804
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Romil Choksi
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14804.v0-trunk.patch, HBASE-14804.v1-trunk.patch
>
>
> I am trying to create a new table and set the NORMALIZATION_ENABLED as true,
> but seems like the argument NORMALIZATION_ENABLED is being ignored. And the
> attribute NORMALIZATION_ENABLED is not displayed on doing a desc command on
> that table
> {code}
> hbase(main):020:0> create 'test-table-4', 'cf', {NORMALIZATION_ENABLED =>
> 'true'}
> An argument ignored (unknown or overridden): NORMALIZATION_ENABLED
> 0 row(s) in 4.2670 seconds
> => Hbase::Table - test-table-4
> hbase(main):021:0> desc 'test-table-4'
> Table test-table-4 is ENABLED
>
>
> test-table-4
>
>
> COLUMN FAMILIES DESCRIPTION
>
>
> {NAME => 'cf', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL =>
> 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOC
> KCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
>
>
> 1 row(s) in 0.0430 seconds
> {code}
> However, on doing an alter command on that table we can set the
> NORMALIZATION_ENABLED attribute for that table
> {code}
> hbase(main):022:0> alter 'test-table-4', {NORMALIZATION_ENABLED => 'true'}
> Unknown argument ignored: NORMALIZATION_ENABLED
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 2.3640 seconds
> hbase(main):023:0> desc 'test-table-4'
> Table test-table-4 is ENABLED
>
>
> test-table-4, {TABLE_ATTRIBUTES => {NORMALIZATION_ENABLED => 'true'}
>
>
> COLUMN FAMILIES DESCRIPTION
>
>
> {NAME => 'cf', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL =>
> 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOC
> KCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
>
>
> 1 row(s) in 0.0190 seconds
> {code}
> I think it would be better to have a single step process to enable
> normalization while creating the table itself, rather than a two step process
> to alter the table later on to enable normalization
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15044: --- Component/s: Balancer > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074156#comment-15074156 ] Mikhail Antonov commented on HBASE-15044: - (i think we used to set component == Balancer on normalization jiras, so they're easier to find) > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074154#comment-15074154 ] Mikhail Antonov commented on HBASE-15044: - [~te...@apache.org] +1, lgtm > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14867) SimpleRegionNormalizer needs to have better heuristics to trigger merge operation
[ https://issues.apache.org/jira/browse/HBASE-14867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074149#comment-15074149 ] Hudson commented on HBASE-14867: SUCCESS: Integrated in HBase-1.3 #475 (See [https://builds.apache.org/job/HBase-1.3/475/]) HBASE-14867 SimpleRegionNormalizer needs to have better heuristics to (tedyu: rev c4173777dd776b23c9a09b3eca827690fc6ee840) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java > SimpleRegionNormalizer needs to have better heuristics to trigger merge > operation > - > > Key: HBASE-14867 > URL: https://issues.apache.org/jira/browse/HBASE-14867 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.2.0 >Reporter: Romil Choksi >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14867-v2.txt, 14867-v3.txt, 14867-v4.txt > > > SimpleRegionNormalizer needs to have better heuristics to trigger merge > operation. SimpleRegionNormalizer is not able to trigger a merge action if > the table's smallest region has neighboring regions that are larger than > table's average region size, whereas there are other smaller regions whose > combined size is less than the average region size. > For example, > - Consider a table with six region, say r1 to r6. > - Keep r1 as empty and create some data say, 100K rows of data for each of > the regions r2, r3 and r4. Create smaller amount of data for regions r5 and > r6, say about 27K rows of data. > - Run the normalizer. Verify the number the regions for that table and also > check the master log to see if any merge action was triggered as a result of > normalization. > In such scenario, it would be better to have a merge action triggered for > those two smaller regions r5 and r6 even though either of them is not the > smallest one -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14888) ClusterSchema: Add Namespace Operations
[ https://issues.apache.org/jira/browse/HBASE-14888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074141#comment-15074141 ] Jonathan Hsieh commented on HBASE-14888: nit: Is the hbase admin change intentional? The failed tests could be related - they are all instances of the master taking too long to start. Maybe the change from a async start of the tablenamespacemanager to the a synchronous startup of the /clusterschemasterservice caused master startup to hiccup/hang? > ClusterSchema: Add Namespace Operations > --- > > Key: HBASE-14888 > URL: https://issues.apache.org/jira/browse/HBASE-14888 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: > 0001-Add-in-a-ClusterSchema-Interface.-It-will-have-all-Av2.patch, > 14888.patch, 14888.v8.txt, 14888v3.txt, 14888v4.txt, 14888v5.txt, > 14888v6.txt, 14888v7.txt > > > Add in a ClusterSchema Interface. It will have all API for all cluster > manipulation; adding namespaces, tables, amending column families, etc. The > idea is to gather up our mess and put it all behind a tidy API that all works > the same way whatever you changing returning a Future to wait on and behind > the scenes driving Procedures. > This patch does namespace operations first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15044: --- Status: Patch Available (was: Open) > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15044: --- Attachment: 15044-v3.txt Patch v3 adds test where split plan gets skipped due to namespace quota constraint. Mikhail: Mind taking another look ? Thanks > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15044-v1.txt, 15044-v2.txt, 15044-v3.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14804) HBase shell's create table command ignores 'NORMALIZATION_ENABLED' attribute
[
https://issues.apache.org/jira/browse/HBASE-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074121#comment-15074121
]
Mikhail Antonov commented on HBASE-14804:
-
I thought whole normalization thing was only committed to master and 1.2+?
> HBase shell's create table command ignores 'NORMALIZATION_ENABLED' attribute
>
>
> Key: HBASE-14804
> URL: https://issues.apache.org/jira/browse/HBASE-14804
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Romil Choksi
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: HBASE-14804.v0-trunk.patch, HBASE-14804.v1-trunk.patch
>
>
> I am trying to create a new table and set the NORMALIZATION_ENABLED as true,
> but seems like the argument NORMALIZATION_ENABLED is being ignored. And the
> attribute NORMALIZATION_ENABLED is not displayed on doing a desc command on
> that table
> {code}
> hbase(main):020:0> create 'test-table-4', 'cf', {NORMALIZATION_ENABLED =>
> 'true'}
> An argument ignored (unknown or overridden): NORMALIZATION_ENABLED
> 0 row(s) in 4.2670 seconds
> => Hbase::Table - test-table-4
> hbase(main):021:0> desc 'test-table-4'
> Table test-table-4 is ENABLED
>
>
> test-table-4
>
>
> COLUMN FAMILIES DESCRIPTION
>
>
> {NAME => 'cf', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL =>
> 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOC
> KCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
>
>
> 1 row(s) in 0.0430 seconds
> {code}
> However, on doing an alter command on that table we can set the
> NORMALIZATION_ENABLED attribute for that table
> {code}
> hbase(main):022:0> alter 'test-table-4', {NORMALIZATION_ENABLED => 'true'}
> Unknown argument ignored: NORMALIZATION_ENABLED
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 2.3640 seconds
> hbase(main):023:0> desc 'test-table-4'
> Table test-table-4 is ENABLED
>
>
> test-table-4, {TABLE_ATTRIBUTES => {NORMALIZATION_ENABLED => 'true'}
>
>
> COLUMN FAMILIES DESCRIPTION
>
>
> {NAME => 'cf', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL =>
> 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOC
> KCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
>
>
> 1 row(s) in 0.0190 seconds
> {code}
> I think it would be better to have a single step process to enable
> normalization while creating the table itself, rather than a two step process
> to alter the table later on to enable normalization
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15044) Region normalization should be allowed when underlying namespace has quota
[ https://issues.apache.org/jira/browse/HBASE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074120#comment-15074120 ] Mikhail Antonov commented on HBASE-15044: - +1 > Region normalization should be allowed when underlying namespace has quota > -- > > Key: HBASE-15044 > URL: https://issues.apache.org/jira/browse/HBASE-15044 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15044-v1.txt, 15044-v2.txt > > > Currently when namespace has quota, HMaster#normalizeRegions() skips the > tables in the namespace. > However, performing region merge(s) wouldn't violate quota constraint. > For region split, NamespaceAuditor#checkQuotaToSplitRegion() can be called to > check whether quota is about to be exceeded. If not, region split plan can > still be executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
