[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-16870:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to branch-1.1+ except branch-1.3
[~mantonov] Feel free to reopen this issue if you want this to go into
branch-1.3.
Thanks [~zghaobac] for the patches and thanks [~tedyu] for reviewing.
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 1.2.3, 1.1.7
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16870-ADDENDUM-v1.patch,
> HBASE-16870-ADDENDUM.patch, HBASE-16870-branch-1-v1.patch,
> HBASE-16870-branch-1-v1.patch, HBASE-16870-branch-1.1-v1.patch,
> HBASE-16870-branch-1.1-v1.patch, HBASE-16870-branch-1.1.patch,
> HBASE-16870-branch-1.3-v1.patch, HBASE-16870-branch-1.3-v1.patch,
> HBASE-16870-branch-1.3.patch, HBASE-16870-branch-1.patch,
> HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception
[
https://issues.apache.org/jira/browse/HBASE-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15597274#comment-15597274
]
Hudson commented on HBASE-16815:
FAILURE: Integrated in Jenkins build HBase-1.4 #489 (See
[https://builds.apache.org/job/HBase-1.4/489/])
HBASE-16815 Low scan ratio in RPC queue tuning triggers divide by zero
(matteo.bertozzi: rev e395bf91dbafc7d892966122972e70c5446f3a09)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java
> Low scan ratio in RPC queue tuning triggers divide by zero exception
>
>
> Key: HBASE-16815
> URL: https://issues.apache.org/jira/browse/HBASE-16815
> Project: HBase
> Issue Type: Bug
> Components: regionserver, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Lars George
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16815-branch-1.2.patch, HBASE-16815.patch
>
>
> Trying the following settings:
> {noformat}
>
> hbase.ipc.server.callqueue.handler.factor
> 0.5
>
>
> hbase.ipc.server.callqueue.read.ratio
> 0.5
>
>
> hbase.ipc.server.callqueue.scan.ratio
> 0.1
>
> {noformat}
> With 30 default handlers, this means 15 queues. Further, it means 8 write
> queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to
> {{0}}. The debug log confirms it, as the tertiary check omits the scan
> details when they are zero:
> {noformat}
> 2016-10-12 12:50:27,305 INFO [main] ipc.SimpleRpcScheduler: Using fifo as
> user call queue, count=15
> 2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default
> writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
> {noformat}
> But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}}
> nevertheless and that does this:
> {code}
> for (int i = 0; i < numHandlers; i++) {
> final int index = qindex + (i % qsize);
> String name = "RpcServer." + threadPrefix + ".handler=" +
> handlers.size() + ",queue=" +
> index + ",port=" + port;
> {code}
> The modulo triggers then
> {noformat}
> 2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
> at
> org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
> at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> ... 7 more
> {noformat}
> That causes the server to not even start. I would suggest we either skip the
> {{startHandler()}} call altogether, or make it zero aware.
> Another possible option is to reserve at least _one_ scan handler/queue when
> the scan ratio is greater than zero, but only of there is more than one read
> handler/que
[jira] [Commented] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception
[
https://issues.apache.org/jira/browse/HBASE-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15597264#comment-15597264
]
Hudson commented on HBASE-16815:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #52 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/52/])
HBASE-16815 Low scan ratio in RPC queue tuning triggers divide by zero
(matteo.bertozzi: rev 5526c947082ce37e93f0a6c330e6828f2fadaede)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java
> Low scan ratio in RPC queue tuning triggers divide by zero exception
>
>
> Key: HBASE-16815
> URL: https://issues.apache.org/jira/browse/HBASE-16815
> Project: HBase
> Issue Type: Bug
> Components: regionserver, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Lars George
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16815-branch-1.2.patch, HBASE-16815.patch
>
>
> Trying the following settings:
> {noformat}
>
> hbase.ipc.server.callqueue.handler.factor
> 0.5
>
>
> hbase.ipc.server.callqueue.read.ratio
> 0.5
>
>
> hbase.ipc.server.callqueue.scan.ratio
> 0.1
>
> {noformat}
> With 30 default handlers, this means 15 queues. Further, it means 8 write
> queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to
> {{0}}. The debug log confirms it, as the tertiary check omits the scan
> details when they are zero:
> {noformat}
> 2016-10-12 12:50:27,305 INFO [main] ipc.SimpleRpcScheduler: Using fifo as
> user call queue, count=15
> 2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default
> writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
> {noformat}
> But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}}
> nevertheless and that does this:
> {code}
> for (int i = 0; i < numHandlers; i++) {
> final int index = qindex + (i % qsize);
> String name = "RpcServer." + threadPrefix + ".handler=" +
> handlers.size() + ",queue=" +
> index + ",port=" + port;
> {code}
> The modulo triggers then
> {noformat}
> 2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
> at
> org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
> at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> ... 7 more
> {noformat}
> That causes the server to not even start. I would suggest we either skip the
> {{startHandler()}} call altogether, or make it zero aware.
> Another possible option is to reserve at least _one_ scan handler/queue when
> the scan ratio is greater than zero, but only of there is more than one read
> han
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-16870:
--
Affects Version/s: 1.4.0
2.0.0
1.2.3
1.1.7
Fix Version/s: 1.1.8
1.2.4
1.4.0
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 1.2.3, 1.1.7
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16870-ADDENDUM-v1.patch,
> HBASE-16870-ADDENDUM.patch, HBASE-16870-branch-1-v1.patch,
> HBASE-16870-branch-1-v1.patch, HBASE-16870-branch-1.1-v1.patch,
> HBASE-16870-branch-1.1-v1.patch, HBASE-16870-branch-1.1.patch,
> HBASE-16870-branch-1.3-v1.patch, HBASE-16870-branch-1.3-v1.patch,
> HBASE-16870-branch-1.3.patch, HBASE-16870-branch-1.patch,
> HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15709) Handle large edits for asynchronous WAL
[
https://issues.apache.org/jira/browse/HBASE-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15597245#comment-15597245
]
Duo Zhang commented on HBASE-15709:
---
{quote}
Block size seems fine as limit. Block would run on till opportunity to roll so
could be bigger than 256M, right?
{quote}
I'm not sure if there are some checks of the block size at datanode or
namenode. And if there is no check for now, they maybe added later, we should
not rely on an unstable feature...
Anyway, let me try to add the support for chunking and finish this issue. At
least this will move the limit from 16M to 256M...
Thanks.
> Handle large edits for asynchronous WAL
> ---
>
> Key: HBASE-15709
> URL: https://issues.apache.org/jira/browse/HBASE-15709
> Project: HBase
> Issue Type: Sub-task
> Components: io, wal
>Reporter: Duo Zhang
>Priority: Critical
>
> First, FanOutOneBlockAsyncDFSOutput can not work if the buffered data is
> larger than PacketReceiver.MAX_PACKET_SIZE(16MB).
> Second, since we only allow one block here, we need to make sure we do not
> exceed the block size after writing a large chunk of data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16887) Allow setting different hadoopcheck versions in precommit for different branches
[ https://issues.apache.org/jira/browse/HBASE-16887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15597239#comment-15597239 ] Duo Zhang commented on HBASE-16887: --- Seems no reply... :( > Allow setting different hadoopcheck versions in precommit for different > branches > > > Key: HBASE-16887 > URL: https://issues.apache.org/jira/browse/HBASE-16887 > Project: HBase > Issue Type: Bug > Components: build >Reporter: Duo Zhang > > http://hbase.apache.org/book.html#hadoop > The supportted hadoop versions are different for different HBase versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15684) Fix the broken log file size accounting
[
https://issues.apache.org/jira/browse/HBASE-15684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15597216#comment-15597216
]
Duo Zhang commented on HBASE-15684:
---
Let me pick this up.
> Fix the broken log file size accounting
> ---
>
> Key: HBASE-15684
> URL: https://issues.apache.org/jira/browse/HBASE-15684
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15684.patch
>
>
> {code}
> long oldFileLen = 0L;
> doReplaceWriter(oldPath, newPath, nextWriter);
> {code}
> Should be
> {code}
> long oldFileLen = doReplaceWriter(oldPath, newPath, nextWriter);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception
[
https://issues.apache.org/jira/browse/HBASE-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15597194#comment-15597194
]
Hudson commented on HBASE-16815:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #46 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/46/])
HBASE-16815 Low scan ratio in RPC queue tuning triggers divide by zero
(matteo.bertozzi: rev 5526c947082ce37e93f0a6c330e6828f2fadaede)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java
> Low scan ratio in RPC queue tuning triggers divide by zero exception
>
>
> Key: HBASE-16815
> URL: https://issues.apache.org/jira/browse/HBASE-16815
> Project: HBase
> Issue Type: Bug
> Components: regionserver, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Lars George
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16815-branch-1.2.patch, HBASE-16815.patch
>
>
> Trying the following settings:
> {noformat}
>
> hbase.ipc.server.callqueue.handler.factor
> 0.5
>
>
> hbase.ipc.server.callqueue.read.ratio
> 0.5
>
>
> hbase.ipc.server.callqueue.scan.ratio
> 0.1
>
> {noformat}
> With 30 default handlers, this means 15 queues. Further, it means 8 write
> queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to
> {{0}}. The debug log confirms it, as the tertiary check omits the scan
> details when they are zero:
> {noformat}
> 2016-10-12 12:50:27,305 INFO [main] ipc.SimpleRpcScheduler: Using fifo as
> user call queue, count=15
> 2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default
> writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
> {noformat}
> But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}}
> nevertheless and that does this:
> {code}
> for (int i = 0; i < numHandlers; i++) {
> final int index = qindex + (i % qsize);
> String name = "RpcServer." + threadPrefix + ".handler=" +
> handlers.size() + ",queue=" +
> index + ",port=" + port;
> {code}
> The modulo triggers then
> {noformat}
> 2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
> at
> org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
> at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> ... 7 more
> {noformat}
> That causes the server to not even start. I would suggest we either skip the
> {{startHandler()}} call altogether, or make it zero aware.
> Another possible option is to reserve at least _one_ scan handler/queue when
> the scan ratio is greater than zero, but only of there is more than one read
> han
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[
https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15597123#comment-15597123
]
stack commented on HBASE-16890:
---
I reran WALPerformanceEvaluation and here is what I got:
Current
{code}
11697968
11697969 4204706.715579 task-clock (msec) #7.284 CPUs utilized
11697970 51,803,614 context-switches #0.012 M/sec
11697971 6,685,043 cpu-migrations#0.002 M/sec
11697972 18,132,994 page-faults #0.004 M/sec
11697973 7,791,964,493,250 cycles#1.853 GHz
11697974 stalled-cycles-frontend
11697975 stalled-cycles-backend
11697976 1,927,527,070,105 instructions #0.25 insns per cycle
11697977335,996,633,114 branches # 79.910 M/sec
11697978 8,301,975,795 branch-misses #2.47% of all branches
11697979
11697980 577.245436180 seconds time elapsed
{code}
With async enabled
{code}
1596
1597 2820443.128629 task-clock (msec) #9.413 CPUs utilized
1598 27,279,634 context-switches #0.010 M/sec
1599 4,548,444 cpu-migrations#0.002 M/sec
1600 9,174,991 page-faults #0.003 M/sec
1601 6,222,195,076,963 cycles#2.206 GHz
1602 stalled-cycles-frontend
1603 stalled-cycles-backend
1604 1,448,534,066,827 instructions #0.23 insns per cycle
1605244,568,362,833 branches # 86.713 M/sec
1606 4,213,196,541 branch-misses #1.72% of all branches
1607
1608 299.621230438 seconds time elapsed
{code}
With 100 threads, async finishes in half the time. Let me try with the server
up. Here is what I ran:
{code}
for i in 100; do for j in 1; do export
HBASE_CLASSPATH_PREFIX=`pwd`/hbase/lib/hbase-server-1.4.0-SNAPSHOT-tests.jar ;
./hbase/bin/hbase --config conf_hbase classpath; perf stat ./hbase/bin/hbase
--config /home/stack/conf_hbase
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation -threads $i -iterations
25 -keySize 50 -valueSize 100 &> "/tmp/wal.5.async.${i}.${j}.txt"; done;
done
{code}
> Analyze the performance of AsyncWAL and fix the same
>
>
> Key: HBASE-16890
> URL: https://issues.apache.org/jira/browse/HBASE-16890
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: contention.png, contention_defaultWAL.png
>
>
> Tests reveal that AsyncWAL under load in single node cluster performs slower
> than the Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16929) Move default method of shipped to Shipper interface
[
https://issues.apache.org/jira/browse/HBASE-16929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15597062#comment-15597062
]
Hudson commented on HBASE-16929:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1830 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1830/])
HBASE-16929 Move default method of shipped to Shipper interface (tedyu: rev
ac415f85f3dd0ecf01a655f3b72351a4b5c675bd)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Shipper.java
> Move default method of shipped to Shipper interface
> ---
>
> Key: HBASE-16929
> URL: https://issues.apache.org/jira/browse/HBASE-16929
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16929.v1.txt
>
>
> HBASE-16626 added default method of shipped() to RegionScanner.
> However, when building master branch of Phoenix against 2.0 SNAPSHOT, I got:
> {code}
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/DelegateRegionScanner.java:[27,8]
> org.apache.phoenix.coprocessor.DelegateRegionScanner is not abstract and
> does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java:[344,36]
> is
> not abstract and does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> {code}
> Here is the snippet for DelegateRegionScanner:
> {code}
> public class DelegateRegionScanner implements RegionScanner {
> {code}
> It seems adding default method in RegionScanner is not enough for downstream
> projects.
> After moving the default method to Shipper interface, the above two
> compilation errors are gone in Phoenix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception
[
https://issues.apache.org/jira/browse/HBASE-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15597064#comment-15597064
]
Hudson commented on HBASE-16815:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1830 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1830/])
HBASE-16815 Low scan ratio in RPC queue tuning triggers divide by zero
(matteo.bertozzi: rev 641513757496842c7077ab5c46618a892768d14c)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java
> Low scan ratio in RPC queue tuning triggers divide by zero exception
>
>
> Key: HBASE-16815
> URL: https://issues.apache.org/jira/browse/HBASE-16815
> Project: HBase
> Issue Type: Bug
> Components: regionserver, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Lars George
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16815-branch-1.2.patch, HBASE-16815.patch
>
>
> Trying the following settings:
> {noformat}
>
> hbase.ipc.server.callqueue.handler.factor
> 0.5
>
>
> hbase.ipc.server.callqueue.read.ratio
> 0.5
>
>
> hbase.ipc.server.callqueue.scan.ratio
> 0.1
>
> {noformat}
> With 30 default handlers, this means 15 queues. Further, it means 8 write
> queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to
> {{0}}. The debug log confirms it, as the tertiary check omits the scan
> details when they are zero:
> {noformat}
> 2016-10-12 12:50:27,305 INFO [main] ipc.SimpleRpcScheduler: Using fifo as
> user call queue, count=15
> 2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default
> writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
> {noformat}
> But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}}
> nevertheless and that does this:
> {code}
> for (int i = 0; i < numHandlers; i++) {
> final int index = qindex + (i % qsize);
> String name = "RpcServer." + threadPrefix + ".handler=" +
> handlers.size() + ",queue=" +
> index + ",port=" + port;
> {code}
> The modulo triggers then
> {noformat}
> 2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
> at
> org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
> at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> ... 7 more
> {noformat}
> That causes the server to not even start. I would suggest we either skip the
> {{startHandler()}} call altogether, or make it zero aware.
> Another possible option is to reserve at least _one_ scan handler/queue when
> the scan ratio is greater than zero, but only of there is more than one
[jira] [Commented] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15597063#comment-15597063
]
Hudson commented on HBASE-16829:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1830 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1830/])
HBASE-16829 DemoClient should detect secure mode (tedyu: rev
4983c742e07e4853a1a5490645f62bdc1f2b5c26)
* (edit)
hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift2/DemoClient.java
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16829.v1.txt, 16829.v2.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-14058: Fix Version/s: 1.2.0 > Stabilizing default heap memory tuner > - > > Key: HBASE-14058 > URL: https://issues.apache.org/jira/browse/HBASE-14058 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Abhilash >Assignee: Abhilash > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 0001-Stabilizing-default-heap-memory-tuner.patch, > HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, > before_modifications.png > > > The memory tuner works well in general cases but when we have a work load > that is both read heavy as well as write heavy the tuner does too many > tuning. We should try to control the number of tuner operation and stabilize > it. The main problem was that the tuner thinks it is in steady state even if > it sees just one neutral tuner period thus does too many tuning operations > and too many reverts that too with large step sizes(step size was set to > maximum even after one neutral period). So to stop this I have thought of > these steps: > 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically > ~62% periods will lie outside this range, which means 62% of the data points > are considered either high or low which is too much. Use μ + δ*0.8 and μ - > δ*0.8 instead. On expectations it will decrease number of tuner operations > per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values > will be considered to be high and 31% will be considered to be low (2*0.31 * > 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% > will be high(2*0.22*0.22 ~ 0.10). > 2) Defining proper steady state by looking at past few periods(it is equal to > hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last > tuner operation. We say tuner is in steady state when last few tuner periods > were NEUTRAL. We keep decreasing step size unless it is extremely low. Then > leave system in that state for some time. > 3) Rather then decreasing step size only while reverting, decrease the > magnitude of step size whenever we are trying to revert tuning done in last > few periods(sum the changes of last few periods and compare to current step) > rather than just looking at last period. When its magnitude gets too low then > make tuner steps NEUTRAL(no operation). This will cause step size to > continuously decrease unless we reach steady state. After that tuning process > will restart (tuner step size rests again when we reach steady state). > 4) The tuning done in last few periods will be decaying sum of past tuner > steps with sign. This parameter will be positive for increase in memstore and > negative for increase in block cache. Rather than using arithmetic mean we > use this to give more priority to recent tuner steps. > Please see the attachments. One represents the size of memstore(green) and > size of block cache(blue) adjusted by tuner without these modification and > other with the above modifications. The x-axis is time axis and y-axis is the > fraction of heap memory available to memstore and block cache at that time(it > always sums up to 80%). I configured min/max ranges for both components to > 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and > 0.7). In both cases the tuner tries to distribute memory by giving ~15% to > memstore and ~65% to block cache. But the modified one does it much more > smoothly. > I got these results from YCSB test. The test was doing approximately 5000 > inserts and 500 reads per second (for one region server). The results can be > further fine tuned and number of tuner operation can be reduced with these > changes in configuration. > For more fine tuning: > a) lower max step size (suggested = 4%) > b) lower min step size ( default if also fine ) > To further decrease frequency of tuning operations: > c) increase the number of lookup periods ( in the tests it was just 10, > default is 60 ) > d) increase tuner period ( in the tests it was just 20 secs, default is > 60secs) > I used smaller tuner period/ number of look up periods to get more data > points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596996#comment-15596996
]
Hudson commented on HBASE-16829:
SUCCESS: Integrated in Jenkins build HBase-1.4 #488 (See
[https://builds.apache.org/job/HBase-1.4/488/])
HBASE-16829 DemoClient should detect secure mode (tedyu: rev
9fda0dd09820de895313b9b06fc020e7f758c78f)
* (edit)
hbase-examples/src/main/java/org/apache/hadoop/hbase/thrift2/DemoClient.java
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16829.v1.txt, 16829.v2.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16930) AssignmentManager.java:checkWals() function can recur infinitely
[
https://issues.apache.org/jira/browse/HBASE-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596919#comment-15596919
]
Hadoop QA commented on HBASE-16930:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 4s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 51s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 43s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 157m 32s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| Timed out junit tests |
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface |
| | org.apache.hadoop.hbase.coprocessor.TestOpenTableInCoprocessor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834772/HBASE-16930.v1.patch |
| JIRA Issue | HBASE-16930 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 219efa57cc30 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / e223639 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4148/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4148/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4148/artifact/patchprocess/patch-unit-hbase-server.txt
|
| T
[jira] [Commented] (HBASE-15709) Handle large edits for asynchronous WAL
[ https://issues.apache.org/jira/browse/HBASE-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596906#comment-15596906 ] stack commented on HBASE-15709: --- Block size seems fine as limit. Block would run on till opportunity to roll so could be bigger than 256M, right? DFSClient#DFSOS has its own machinery. There is nought we can do to influence its operation. Changing Q size or packet size doesn't matter. We are careful aggregating syncs and appends on our side but then the DFSClient#DFSOS machinery does its own thing regardless (we need to be in control of the packet sending). Given this, AsyncWAL for this reason and others is the path forward. Let us know how can help. Will look at Rams perf issue this w/e. > Handle large edits for asynchronous WAL > --- > > Key: HBASE-15709 > URL: https://issues.apache.org/jira/browse/HBASE-15709 > Project: HBase > Issue Type: Sub-task > Components: io, wal >Reporter: Duo Zhang >Priority: Critical > > First, FanOutOneBlockAsyncDFSOutput can not work if the buffered data is > larger than PacketReceiver.MAX_PACKET_SIZE(16MB). > Second, since we only allow one block here, we need to make sure we do not > exceed the block size after writing a large chunk of data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16754) Regions failing compaction due to referencing non-existent store file
[
https://issues.apache.org/jira/browse/HBASE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596895#comment-15596895
]
Hudson commented on HBASE-16754:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #45 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/45/])
HBASE-16754 All WALSplitter OutputSinks should process compaction events
(garyh: rev fdb921c89346addb9daba338a1472a4bb76fb2e1)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java
* (add) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java
> Regions failing compaction due to referencing non-existent store file
> -
>
> Key: HBASE-16754
> URL: https://issues.apache.org/jira/browse/HBASE-16754
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.3
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5
>
> Attachments: HBASE-16754.001.patch, HBASE-16754.branch-1.001.patch,
> HBASE-16754.branch-1.2.001.patch
>
>
> Running a mixed read write workload on a recent build off branch-1.3, we are
> seeing compactions occasionally fail with errors like the following (actual
> filenames replaced with placeholders):
> {noformat}
> 16/09/27 16:57:28 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 116
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
>
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
>
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1644)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:373)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$100(CompactSplitThread.java:59)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:498)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:568)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/09/27 17:01:31 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 115
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache
[jira] [Commented] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[
https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596875#comment-15596875
]
Hadoop QA commented on HBASE-16744:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 41s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 17m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
56s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 7s
{color} | {color:red} hbase-protocol-shaded in master has 22 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 58s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 17m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 44 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 22s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 51s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 129m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestLockManager |
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestLockProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834775/HBASE-16744.master.013.patch
|
| JIRA Issue | HBASE-
[jira] [Updated] (HBASE-16929) Move default method of shipped to Shipper interface
[
https://issues.apache.org/jira/browse/HBASE-16929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16929:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the review, Stephen.
> Move default method of shipped to Shipper interface
> ---
>
> Key: HBASE-16929
> URL: https://issues.apache.org/jira/browse/HBASE-16929
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16929.v1.txt
>
>
> HBASE-16626 added default method of shipped() to RegionScanner.
> However, when building master branch of Phoenix against 2.0 SNAPSHOT, I got:
> {code}
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/DelegateRegionScanner.java:[27,8]
> org.apache.phoenix.coprocessor.DelegateRegionScanner is not abstract and
> does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java:[344,36]
> is
> not abstract and does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> {code}
> Here is the snippet for DelegateRegionScanner:
> {code}
> public class DelegateRegionScanner implements RegionScanner {
> {code}
> It seems adding default method in RegionScanner is not enough for downstream
> projects.
> After moving the default method to Shipper interface, the above two
> compilation errors are gone in Phoenix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15709) Handle large edits for asynchronous WAL
[ https://issues.apache.org/jira/browse/HBASE-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596763#comment-15596763 ] Duo Zhang commented on HBASE-15709: --- I can add the support of entries larger than 16M, as you said, split it into several chunks. But here comes another limit, the block size. Our fan out output is 'one block'... So there will always be a size limit for AsyncFSWAL. 16M or 256M. And the two WALs will break our serial replication feature. So if we decide to use AsyncFSWAL then we must reject the large edits... If you guys think 16M is too small and 256M is OK then I could try to add chunking support for AsyncFSWAL. Thanks. > Handle large edits for asynchronous WAL > --- > > Key: HBASE-15709 > URL: https://issues.apache.org/jira/browse/HBASE-15709 > Project: HBase > Issue Type: Sub-task > Components: io, wal >Reporter: Duo Zhang >Priority: Critical > > First, FanOutOneBlockAsyncDFSOutput can not work if the buffered data is > larger than PacketReceiver.MAX_PACKET_SIZE(16MB). > Second, since we only allow one block here, we need to make sure we do not > exceed the block size after writing a large chunk of data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception
[
https://issues.apache.org/jira/browse/HBASE-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-16815:
Fix Version/s: (was: 1.3.1)
> Low scan ratio in RPC queue tuning triggers divide by zero exception
>
>
> Key: HBASE-16815
> URL: https://issues.apache.org/jira/browse/HBASE-16815
> Project: HBase
> Issue Type: Bug
> Components: regionserver, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Lars George
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16815-branch-1.2.patch, HBASE-16815.patch
>
>
> Trying the following settings:
> {noformat}
>
> hbase.ipc.server.callqueue.handler.factor
> 0.5
>
>
> hbase.ipc.server.callqueue.read.ratio
> 0.5
>
>
> hbase.ipc.server.callqueue.scan.ratio
> 0.1
>
> {noformat}
> With 30 default handlers, this means 15 queues. Further, it means 8 write
> queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to
> {{0}}. The debug log confirms it, as the tertiary check omits the scan
> details when they are zero:
> {noformat}
> 2016-10-12 12:50:27,305 INFO [main] ipc.SimpleRpcScheduler: Using fifo as
> user call queue, count=15
> 2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default
> writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
> {noformat}
> But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}}
> nevertheless and that does this:
> {code}
> for (int i = 0; i < numHandlers; i++) {
> final int index = qindex + (i % qsize);
> String name = "RpcServer." + threadPrefix + ".handler=" +
> handlers.size() + ",queue=" +
> index + ",port=" + port;
> {code}
> The modulo triggers then
> {noformat}
> 2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
> at
> org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
> at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> ... 7 more
> {noformat}
> That causes the server to not even start. I would suggest we either skip the
> {{startHandler()}} call altogether, or make it zero aware.
> Another possible option is to reserve at least _one_ scan handler/queue when
> the scan ratio is greater than zero, but only of there is more than one read
> handler/queue to begin with. Otherwise the scan handler/queue should be zero
> and share the one read handler/queue.
> Makes sense?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception
[
https://issues.apache.org/jira/browse/HBASE-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-16815:
Fix Version/s: 1.3.1
> Low scan ratio in RPC queue tuning triggers divide by zero exception
>
>
> Key: HBASE-16815
> URL: https://issues.apache.org/jira/browse/HBASE-16815
> Project: HBase
> Issue Type: Bug
> Components: regionserver, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Lars George
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16815-branch-1.2.patch, HBASE-16815.patch
>
>
> Trying the following settings:
> {noformat}
>
> hbase.ipc.server.callqueue.handler.factor
> 0.5
>
>
> hbase.ipc.server.callqueue.read.ratio
> 0.5
>
>
> hbase.ipc.server.callqueue.scan.ratio
> 0.1
>
> {noformat}
> With 30 default handlers, this means 15 queues. Further, it means 8 write
> queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to
> {{0}}. The debug log confirms it, as the tertiary check omits the scan
> details when they are zero:
> {noformat}
> 2016-10-12 12:50:27,305 INFO [main] ipc.SimpleRpcScheduler: Using fifo as
> user call queue, count=15
> 2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default
> writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
> {noformat}
> But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}}
> nevertheless and that does this:
> {code}
> for (int i = 0; i < numHandlers; i++) {
> final int index = qindex + (i % qsize);
> String name = "RpcServer." + threadPrefix + ".handler=" +
> handlers.size() + ",queue=" +
> index + ",port=" + port;
> {code}
> The modulo triggers then
> {noformat}
> 2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
> at
> org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
> at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> ... 7 more
> {noformat}
> That causes the server to not even start. I would suggest we either skip the
> {{startHandler()}} call altogether, or make it zero aware.
> Another possible option is to reserve at least _one_ scan handler/queue when
> the scan ratio is greater than zero, but only of there is more than one read
> handler/queue to begin with. Otherwise the scan handler/queue should be zero
> and share the one read handler/queue.
> Makes sense?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception
[
https://issues.apache.org/jira/browse/HBASE-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596753#comment-15596753
]
Mikhail Antonov commented on HBASE-16815:
-
yeah I'm working on making a tag / rc now [~mbertozzi] let's hold on that, i
have a git hash I'm settling at..
> Low scan ratio in RPC queue tuning triggers divide by zero exception
>
>
> Key: HBASE-16815
> URL: https://issues.apache.org/jira/browse/HBASE-16815
> Project: HBase
> Issue Type: Bug
> Components: regionserver, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Lars George
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16815-branch-1.2.patch, HBASE-16815.patch
>
>
> Trying the following settings:
> {noformat}
>
> hbase.ipc.server.callqueue.handler.factor
> 0.5
>
>
> hbase.ipc.server.callqueue.read.ratio
> 0.5
>
>
> hbase.ipc.server.callqueue.scan.ratio
> 0.1
>
> {noformat}
> With 30 default handlers, this means 15 queues. Further, it means 8 write
> queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to
> {{0}}. The debug log confirms it, as the tertiary check omits the scan
> details when they are zero:
> {noformat}
> 2016-10-12 12:50:27,305 INFO [main] ipc.SimpleRpcScheduler: Using fifo as
> user call queue, count=15
> 2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default
> writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
> {noformat}
> But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}}
> nevertheless and that does this:
> {code}
> for (int i = 0; i < numHandlers; i++) {
> final int index = qindex + (i % qsize);
> String name = "RpcServer." + threadPrefix + ".handler=" +
> handlers.size() + ",queue=" +
> index + ",port=" + port;
> {code}
> The modulo triggers then
> {noformat}
> 2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
> at
> org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
> at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> ... 7 more
> {noformat}
> That causes the server to not even start. I would suggest we either skip the
> {{startHandler()}} call altogether, or make it zero aware.
> Another possible option is to reserve at least _one_ scan handler/queue when
> the scan ratio is greater than zero, but only of there is more than one read
> handler/queue to begin with. Otherwise the scan handler/queue should be zero
> and share the one read handler/queue.
> Makes sense?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception
[
https://issues.apache.org/jira/browse/HBASE-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596747#comment-15596747
]
Matteo Bertozzi commented on HBASE-16815:
-
committed to 1.1, 1.2, branch-1 and master.
holding off 1.3 until [~mantonov] gives the ok, since he seems to be working on
cutting the 1.3 branch
> Low scan ratio in RPC queue tuning triggers divide by zero exception
>
>
> Key: HBASE-16815
> URL: https://issues.apache.org/jira/browse/HBASE-16815
> Project: HBase
> Issue Type: Bug
> Components: regionserver, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Lars George
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16815-branch-1.2.patch, HBASE-16815.patch
>
>
> Trying the following settings:
> {noformat}
>
> hbase.ipc.server.callqueue.handler.factor
> 0.5
>
>
> hbase.ipc.server.callqueue.read.ratio
> 0.5
>
>
> hbase.ipc.server.callqueue.scan.ratio
> 0.1
>
> {noformat}
> With 30 default handlers, this means 15 queues. Further, it means 8 write
> queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to
> {{0}}. The debug log confirms it, as the tertiary check omits the scan
> details when they are zero:
> {noformat}
> 2016-10-12 12:50:27,305 INFO [main] ipc.SimpleRpcScheduler: Using fifo as
> user call queue, count=15
> 2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default
> writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
> {noformat}
> But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}}
> nevertheless and that does this:
> {code}
> for (int i = 0; i < numHandlers; i++) {
> final int index = qindex + (i % qsize);
> String name = "RpcServer." + threadPrefix + ".handler=" +
> handlers.size() + ",queue=" +
> index + ",port=" + port;
> {code}
> The modulo triggers then
> {noformat}
> 2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
> at
> org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
> at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> ... 7 more
> {noformat}
> That causes the server to not even start. I would suggest we either skip the
> {{startHandler()}} call altogether, or make it zero aware.
> Another possible option is to reserve at least _one_ scan handler/queue when
> the scan ratio is greater than zero, but only of there is more than one read
> handler/queue to begin with. Otherwise the scan handler/queue should be zero
> and share the one read handler/queue.
> Makes sense?
--
This message was sent by Atlassian JI
[jira] [Commented] (HBASE-15347) Update CHANGES.txt for 1.3
[ https://issues.apache.org/jira/browse/HBASE-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596748#comment-15596748 ] Mikhail Antonov commented on HBASE-15347: - (also cc [~stack] just in case) > Update CHANGES.txt for 1.3 > -- > > Key: HBASE-15347 > URL: https://issues.apache.org/jira/browse/HBASE-15347 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > Attachments: HBASE-15347-branch-1.3.v1.patch > > > Going to post the steps in preparing changes file for 1.3 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16815) Low scan ratio in RPC queue tuning triggers divide by zero exception
[
https://issues.apache.org/jira/browse/HBASE-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-16815:
Resolution: Fixed
Fix Version/s: 1.1.8
1.2.4
1.4.0
2.0.0
Status: Resolved (was: Patch Available)
> Low scan ratio in RPC queue tuning triggers divide by zero exception
>
>
> Key: HBASE-16815
> URL: https://issues.apache.org/jira/browse/HBASE-16815
> Project: HBase
> Issue Type: Bug
> Components: regionserver, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Lars George
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16815-branch-1.2.patch, HBASE-16815.patch
>
>
> Trying the following settings:
> {noformat}
>
> hbase.ipc.server.callqueue.handler.factor
> 0.5
>
>
> hbase.ipc.server.callqueue.read.ratio
> 0.5
>
>
> hbase.ipc.server.callqueue.scan.ratio
> 0.1
>
> {noformat}
> With 30 default handlers, this means 15 queues. Further, it means 8 write
> queues and 7 read queues. 10% of that is {{0.7}} which is then floor'ed to
> {{0}}. The debug log confirms it, as the tertiary check omits the scan
> details when they are zero:
> {noformat}
> 2016-10-12 12:50:27,305 INFO [main] ipc.SimpleRpcScheduler: Using fifo as
> user call queue, count=15
> 2016-10-12 12:50:27,311 DEBUG [main] ipc.RWQueueRpcExecutor: FifoRWQ.default
> writeQueues=7 writeHandlers=15 readQueues=8 readHandlers=14
> {noformat}
> But the code in {{RWQueueRpcExecutor}} calls {{RpcExecutor.startHandler()}}
> nevertheless and that does this:
> {code}
> for (int i = 0; i < numHandlers; i++) {
> final int index = qindex + (i % qsize);
> String name = "RpcServer." + threadPrefix + ".handler=" +
> handlers.size() + ",queue=" +
> index + ",port=" + port;
> {code}
> The modulo triggers then
> {noformat}
> 2016-10-12 11:41:22,810 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2524)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.startHandlers(RpcExecutor.java:125)
> at
> org.apache.hadoop.hbase.ipc.RWQueueRpcExecutor.startHandlers(RWQueueRpcExecutor.java:178)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.start(RpcExecutor.java:78)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.start(SimpleRpcScheduler.java:272)
> at org.apache.hadoop.hbase.ipc.RpcServer.start(RpcServer.java:2212)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.start(RSRpcServices.java:1143)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:615)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:396)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
> ... 7 more
> {noformat}
> That causes the server to not even start. I would suggest we either skip the
> {{startHandler()}} call altogether, or make it zero aware.
> Another possible option is to reserve at least _one_ scan handler/queue when
> the scan ratio is greater than zero, but only of there is more than one read
> handler/queue to begin with. Otherwise the scan handler/queue should be zero
> and share the one read handler/queue.
> Makes sense?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
[
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596743#comment-15596743
]
Matteo Bertozzi commented on HBASE-14865:
-
yeah, fixes are in 2.6.1 and 2.7.1 and we updated the doc with the supported
versions here HBASE-16884
> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: 14865-master-v7.patch, HBASE-14865-branch-1.2.patch,
> HBASE-14865-branch-1.patch, HBASE-14865-branch-1.patch,
> HBASE-14865-master-v2.patch, HBASE-14865-master-v3.patch,
> HBASE-14865-master-v4.patch, HBASE-14865-master-v5.patch,
> HBASE-14865-master-v6.patch, HBASE-14865-master-v7.patch,
> HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of
> authentication/integrity/privacy. It is the used to set
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll
> have to take a downtime. Rolling upgrade will create a situation where some
> nodes have old value and some have new, which'll prevent any communication
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order.
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16929) Move default method of shipped to Shipper interface
[
https://issues.apache.org/jira/browse/HBASE-16929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596740#comment-15596740
]
Hadoop QA commented on HBASE-16929:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 43s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 16s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.TestNamespace |
| | org.apache.hadoop.hbase.snapshot.TestMobRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834760/16929.v1.txt |
| JIRA Issue | HBASE-16929 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 374b1ea0cdd2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e223639 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4145/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
h
[jira] [Commented] (HBASE-15347) Update CHANGES.txt for 1.3
[ https://issues.apache.org/jira/browse/HBASE-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596739#comment-15596739 ] Mikhail Antonov commented on HBASE-15347: - [~busbey] FYI I've been looking through the changex.txt for 1.3 and find some data generated by Jira surprising. I"m looking why the above consistency check returned so many mismatches. E.g. HBASE-10844 is added to auto-generated release notes in JIRA, though it's obvious that it was marked as "fixed" for 1.2.0 minor release? > Update CHANGES.txt for 1.3 > -- > > Key: HBASE-15347 > URL: https://issues.apache.org/jira/browse/HBASE-15347 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > Attachments: HBASE-15347-branch-1.3.v1.patch > > > Going to post the steps in preparing changes file for 1.3 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16891) Try copying to the Netty ByteBuf directly from the WALEdit
[ https://issues.apache.org/jira/browse/HBASE-16891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596736#comment-15596736 ] stack commented on HBASE-16891: --- We doing native checksumming in both places? > Try copying to the Netty ByteBuf directly from the WALEdit > -- > > Key: HBASE-16891 > URL: https://issues.apache.org/jira/browse/HBASE-16891 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > -> The FanOutOneBlockAsyncDFSOutput is a much sophisticated dfs client model > that works with Netty ByteBuf. Here we hold on connection to the datanodes > using > Netty Channels. And the idea is to write data direclty to these channels. > AsyncHLog gets an append call. The AysncWAL uses the HBase's > ByteArrayOutputSTream and so the content of the cell is written to this BAOS > and that is again > copied to the netty Bytebuf in the FanOutOneBlockAsyncDFSOutput. > So when the sync call happens this FanoutDFSoutput does the checksum > calcualtion itself and then writes the content of this buffer direclty to the > DN channel. > -> In case of FSHLOg this is different. When an append call comes we direclty > write the content to the FSDataOutputStream (it is copied to this stream). > Then here internally there is a checkSum calculation that happens. when a > sync call happens there is noth ing to do except to notify the NN to flush > the latest > data. > AS we can see from the above that there are two copies in AsyncWAL > -> From the Cell to the BAOS > -> From the BAOS to the Netty byte buf > -> On sync() call, do check sum and finally flush the netty byte buf to the > DN channel > In case of FSHLog > -> From cell to the FSDataoutputstream. data is copied. Check sum happens > here. > -> Sync call just tries to notify the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15709) Handle large edits for asynchronous WAL
[ https://issues.apache.org/jira/browse/HBASE-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15709: -- Priority: Critical (was: Major) How we going to reject big Cells? Would it be crazy running two WALs per RegionServer? One the new async and the other the old version so we could still do big types? (Yes, it would). Is this a limitation of fan-out only? Why so? When a bigger than 16M patch comes in on current WAL, how is it handled? Its cut up into checksum sized chunks and sent down the pipeline. Is it because of the fanout? > Handle large edits for asynchronous WAL > --- > > Key: HBASE-15709 > URL: https://issues.apache.org/jira/browse/HBASE-15709 > Project: HBase > Issue Type: Sub-task > Components: io, wal >Reporter: Duo Zhang >Priority: Critical > > First, FanOutOneBlockAsyncDFSOutput can not work if the buffered data is > larger than PacketReceiver.MAX_PACKET_SIZE(16MB). > Second, since we only allow one block here, we need to make sure we do not > exceed the block size after writing a large chunk of data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16910) Avoid NPE when starting StochasticLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-16910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596725#comment-15596725
]
Hudson commented on HBASE-16910:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1829 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1829/])
HBASE-16910 Avoid NPE when starting StochasticLoadBalancer (Guanghao (tedyu:
rev e2236396713cfc1cef61150a569e972c2faaca04)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit)
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
> Avoid NPE when starting StochasticLoadBalancer
> --
>
> Key: HBASE-16910
> URL: https://issues.apache.org/jira/browse/HBASE-16910
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16910-v1.patch, HBASE-16910.patch
>
>
> When master start, it initialize StochasticLoadBalancer.
> {code}
> this.balancer.setClusterStatus(getClusterStatus());
> this.balancer.setMasterServices(this);
> {code}
> It first setClusterStatus(), then setMasterService(). But in setClusterStatus
> method, it use master service which is not initialized. So it will throw NPE.
> {code}
> int tablesCount = isByTable ? services.getTableDescriptors().getAll().size()
> : 1;
> {code}
> It happens when set isByTable is ture.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596719#comment-15596719 ] stack commented on HBASE-16890: --- Thanks [~ram_krish] for opening this issue. Lets fix this. It must be something basic given how our current DFSClient works. Let me try and look too over w/e. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16829:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the review, Josh.
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16829.v1.txt, 16829.v2.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14329) Report region in transition only ever operates on one region
[
https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596659#comment-15596659
]
Hadoop QA commented on HBASE-14329:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 3s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 23s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 125m 16s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestReplicaWithCluster |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834758/HBASE-14329.patch |
| JIRA Issue | HBASE-14329 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9acdaeb20637 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e223639 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4143/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/4143/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4143/artifact/patchprocess/patc
[jira] [Updated] (HBASE-16786) Procedure V2 - Move ZK-lock's uses to Procedure framework locks (LockProcedure)
[ https://issues.apache.org/jira/browse/HBASE-16786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16786: - Attachment: HBASE-16786.master.006.patch > Procedure V2 - Move ZK-lock's uses to Procedure framework locks > (LockProcedure) > --- > > Key: HBASE-16786 > URL: https://issues.apache.org/jira/browse/HBASE-16786 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16786.master.001.patch, > HBASE-16786.master.002.patch, HBASE-16786.master.003.patch, > HBASE-16786.master.004.patch, HBASE-16786.master.005.patch, > HBASE-16786.master.006.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596642#comment-15596642
]
Hadoop QA commented on HBASE-16829:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 15s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s
{color} | {color:green} hbase-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 7s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834765/16829.v2.txt |
| JIRA Issue | HBASE-16829 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c2fec7424de9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e223639 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4146/testReport/ |
| modules | C: hbase-examples U: hbase-examples |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4146/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> DemoClient should detect secure mode
>
>
>
[jira] [Updated] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[ https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16744: - Attachment: HBASE-16744.master.013.patch > Procedure V2 - Lock procedures to allow clients to acquire locks on > tables/namespaces/regions > - > > Key: HBASE-16744 > URL: https://issues.apache.org/jira/browse/HBASE-16744 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16744.master.001.patch, > HBASE-16744.master.002.patch, HBASE-16744.master.003.patch, > HBASE-16744.master.004.patch, HBASE-16744.master.005.patch, > HBASE-16744.master.006.patch, HBASE-16744.master.007.patch, > HBASE-16744.master.008.patch, HBASE-16744.master.009.patch, > HBASE-16744.master.010.patch, HBASE-16744.master.011.patch, > HBASE-16744.master.012.patch, HBASE-16744.master.013.patch > > > Will help us get rid of ZK locks. > Will be useful for external tools like hbck, future backup manager, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16930) AssignmentManager.java:checkWals() function can recur infinitely
[ https://issues.apache.org/jira/browse/HBASE-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596612#comment-15596612 ] Ted Yu commented on HBASE-16930: lgtm > AssignmentManager.java:checkWals() function can recur infinitely > > > Key: HBASE-16930 > URL: https://issues.apache.org/jira/browse/HBASE-16930 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Attachments: HBASE-16930.v1.patch > > > Recursion in AssignmentManager.checkWals() uses the same input argument. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16930) AssignmentManager.java:checkWals() function can recur infinitely
[ https://issues.apache.org/jira/browse/HBASE-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-16930: - Release Note: Fixed potential infinite recursion in AssignmentManager.checkWals(). Status: Patch Available (was: In Progress) > AssignmentManager.java:checkWals() function can recur infinitely > > > Key: HBASE-16930 > URL: https://issues.apache.org/jira/browse/HBASE-16930 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Attachments: HBASE-16930.v1.patch > > > Recursion in AssignmentManager.checkWals() uses the same input argument. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16930) AssignmentManager.java:checkWals() function can recur infinitely
[
https://issues.apache.org/jira/browse/HBASE-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596610#comment-15596610
]
Umesh Agashe commented on HBASE-16930:
--
See the code below:
{code}
private boolean checkWals(FileSystem fs, Path dir) throws IOException {
if (!fs.exists(dir)) {
LOG.debug(dir + " doesn't exist");
return false;
}
if (!fs.getFileStatus(dir).isDirectory()) {
LOG.warn(dir + " is not a directory");
return false;
}
FileStatus[] files = FSUtils.listStatus(fs, dir);
if (files == null || files.length == 0) {
LOG.debug(dir + " has no files");
return false;
}
for (int i = 0; i < files.length; i++) {
if (files[i].isFile() && files[i].getLen() > 0) {
LOG.debug(dir + " has a non-empty file: " + files[i].getPath());
return true;
} else if (files[i].isDirectory() && checkWals(fs, dir)) {
LOG.debug(dir + " is a directory and has a non-empty file: " +
files[i].getPath());
return true;
}
}
LOG.debug("Found 0 non-empty wal files for :" + dir);
return false;
}
{code}
dir is input arg and without changing it passed to call to checkWals() if child
directory is found.
> AssignmentManager.java:checkWals() function can recur infinitely
>
>
> Key: HBASE-16930
> URL: https://issues.apache.org/jira/browse/HBASE-16930
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Attachments: HBASE-16930.v1.patch
>
>
> Recursion in AssignmentManager.checkWals() uses the same input argument.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16930) AssignmentManager.java:checkWals() function can recur infinitely
[
https://issues.apache.org/jira/browse/HBASE-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe updated HBASE-16930:
-
Description: Recursion in AssignmentManager.checkWals() uses the same input
argument. (was: Recursion in AssignmentManager.checkWals() uses the same input
argument. See the code below:
{code}
private boolean checkWals(FileSystem fs, Path dir) throws IOException {
if (!fs.exists(dir)) {
LOG.debug(dir + " doesn't exist");
return false;
}
if (!fs.getFileStatus(dir).isDirectory()) {
LOG.warn(dir + " is not a directory");
return false;
}
FileStatus[] files = FSUtils.listStatus(fs, dir);
if (files == null || files.length == 0) {
LOG.debug(dir + " has no files");
return false;
}
for (int i = 0; i < files.length; i++) {
if (files[i].isFile() && files[i].getLen() > 0) {
LOG.debug(dir + " has a non-empty file: " + files[i].getPath());
return true;
} else if (files[i].isDirectory() && checkWals(fs, dir)) {
LOG.debug(dir + " is a directory and has a non-empty file: " +
files[i].getPath());
return true;
}
}
LOG.debug("Found 0 non-empty wal files for :" + dir);
return false;
}
{code}
dir is input arg and without changing it passed to call to checkWals() if child
directory is found.)
> AssignmentManager.java:checkWals() function can recur infinitely
>
>
> Key: HBASE-16930
> URL: https://issues.apache.org/jira/browse/HBASE-16930
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Attachments: HBASE-16930.v1.patch
>
>
> Recursion in AssignmentManager.checkWals() uses the same input argument.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16930) AssignmentManager.java:checkWals() function can recur infinitely
[
https://issues.apache.org/jira/browse/HBASE-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe updated HBASE-16930:
-
Attachment: HBASE-16930.v1.patch
Fixes potential infinite recursion in AssignmentManager.checkWals().
> AssignmentManager.java:checkWals() function can recur infinitely
>
>
> Key: HBASE-16930
> URL: https://issues.apache.org/jira/browse/HBASE-16930
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Attachments: HBASE-16930.v1.patch
>
>
> Recursion in AssignmentManager.checkWals() uses the same input argument. See
> the code below:
> {code}
> private boolean checkWals(FileSystem fs, Path dir) throws IOException {
> if (!fs.exists(dir)) {
> LOG.debug(dir + " doesn't exist");
> return false;
> }
> if (!fs.getFileStatus(dir).isDirectory()) {
> LOG.warn(dir + " is not a directory");
> return false;
> }
> FileStatus[] files = FSUtils.listStatus(fs, dir);
> if (files == null || files.length == 0) {
> LOG.debug(dir + " has no files");
> return false;
> }
> for (int i = 0; i < files.length; i++) {
> if (files[i].isFile() && files[i].getLen() > 0) {
> LOG.debug(dir + " has a non-empty file: " + files[i].getPath());
> return true;
> } else if (files[i].isDirectory() && checkWals(fs, dir)) {
> LOG.debug(dir + " is a directory and has a non-empty file: " +
> files[i].getPath());
> return true;
> }
> }
> LOG.debug("Found 0 non-empty wal files for :" + dir);
> return false;
> }
> {code}
> dir is input arg and without changing it passed to call to checkWals() if
> child directory is found.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HBASE-16930) AssignmentManager.java:checkWals() function can recur infinitely
[
https://issues.apache.org/jira/browse/HBASE-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-16930 started by Umesh Agashe.
> AssignmentManager.java:checkWals() function can recur infinitely
>
>
> Key: HBASE-16930
> URL: https://issues.apache.org/jira/browse/HBASE-16930
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>
> Recursion in AssignmentManager.checkWals() uses the same input argument. See
> the code below:
> {code}
> private boolean checkWals(FileSystem fs, Path dir) throws IOException {
> if (!fs.exists(dir)) {
> LOG.debug(dir + " doesn't exist");
> return false;
> }
> if (!fs.getFileStatus(dir).isDirectory()) {
> LOG.warn(dir + " is not a directory");
> return false;
> }
> FileStatus[] files = FSUtils.listStatus(fs, dir);
> if (files == null || files.length == 0) {
> LOG.debug(dir + " has no files");
> return false;
> }
> for (int i = 0; i < files.length; i++) {
> if (files[i].isFile() && files[i].getLen() > 0) {
> LOG.debug(dir + " has a non-empty file: " + files[i].getPath());
> return true;
> } else if (files[i].isDirectory() && checkWals(fs, dir)) {
> LOG.debug(dir + " is a directory and has a non-empty file: " +
> files[i].getPath());
> return true;
> }
> }
> LOG.debug("Found 0 non-empty wal files for :" + dir);
> return false;
> }
> {code}
> dir is input arg and without changing it passed to call to checkWals() if
> child directory is found.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16929) Move default method of shipped to Shipper interface
[
https://issues.apache.org/jira/browse/HBASE-16929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596598#comment-15596598
]
Stephen Yuan Jiang commented on HBASE-16929:
LGTM
> Move default method of shipped to Shipper interface
> ---
>
> Key: HBASE-16929
> URL: https://issues.apache.org/jira/browse/HBASE-16929
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16929.v1.txt
>
>
> HBASE-16626 added default method of shipped() to RegionScanner.
> However, when building master branch of Phoenix against 2.0 SNAPSHOT, I got:
> {code}
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/DelegateRegionScanner.java:[27,8]
> org.apache.phoenix.coprocessor.DelegateRegionScanner is not abstract and
> does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java:[344,36]
> is
> not abstract and does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> {code}
> Here is the snippet for DelegateRegionScanner:
> {code}
> public class DelegateRegionScanner implements RegionScanner {
> {code}
> It seems adding default method in RegionScanner is not enough for downstream
> projects.
> After moving the default method to Shipper interface, the above two
> compilation errors are gone in Phoenix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16929) Move default method of shipped to Shipper interface
[
https://issues.apache.org/jira/browse/HBASE-16929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Yuan Jiang updated HBASE-16929:
---
Affects Version/s: 2.0.0
> Move default method of shipped to Shipper interface
> ---
>
> Key: HBASE-16929
> URL: https://issues.apache.org/jira/browse/HBASE-16929
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16929.v1.txt
>
>
> HBASE-16626 added default method of shipped() to RegionScanner.
> However, when building master branch of Phoenix against 2.0 SNAPSHOT, I got:
> {code}
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/DelegateRegionScanner.java:[27,8]
> org.apache.phoenix.coprocessor.DelegateRegionScanner is not abstract and
> does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java:[344,36]
> is
> not abstract and does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> {code}
> Here is the snippet for DelegateRegionScanner:
> {code}
> public class DelegateRegionScanner implements RegionScanner {
> {code}
> It seems adding default method in RegionScanner is not enough for downstream
> projects.
> After moving the default method to Shipper interface, the above two
> compilation errors are gone in Phoenix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16689) Durability == ASYNC_WAL means no SYNC
[ https://issues.apache.org/jira/browse/HBASE-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596595#comment-15596595 ] stack commented on HBASE-16689: --- If ASYNC_WAL, the most data you could lose in a crash of the RegionServer currently is ~dfs.client-write-packet-size (default=64k). The DFSOutputStream takes writes until it gets a DFSPacket-ful. It then queues the packet for the data DFSOS#Streamer thread to send down the DN pipeline for the DFSOS#Streamer#ResponseProcessor to ACK back. W/o syncs, there is a continuous fill the packet, send, and so on. Syncs mess up the rhythm by cutting the DFSPacket off before it fills not returning until the ResponseProcessor ACKs-back that the cut-off packet was sent. Unfortunately DFSOS is opaque giving us no insight on where it is in the above back and forth. We could try proxying the state by sizing our append and taking action at DFSPacket size hoping we align w/ DFSOS internals but unsatisfactory. When no sync'ing we run much faster and the above packet flow seems to keep up w/ our append rate (Needs exposure). The 'fix' here is, yuck, a (yet-another!) watcher thread. It wakes up every N ms and looks to see if unsynced data in the WAL and if so, sends in a sync. > Durability == ASYNC_WAL means no SYNC > - > > Key: HBASE-16689 > URL: https://issues.apache.org/jira/browse/HBASE-16689 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 1.0.3, 1.1.6, 1.2.3 >Reporter: stack >Assignee: stack >Priority: Critical > > Setting DURABILITY=ASYNC_WAL on a Table suspends all syncs for all table > Table appends. If all tables on a cluster have this setting, data is flushed > from the RS to the DN at some arbitrary time and a bunch may just hang out in > DFSClient buffers on the RS-side indefinitely if writes are sporadic, at > least until there is a WAL roll -- a log roll sends a sync through the write > pipeline to flush out any outstanding appends -- or a region close which does > similar or we crash and drop the data in buffers RS. > This is probably not what a user expects when they set ASYNC_WAL (We don't > doc anywhere that I could find clearly what ASYNC_WAL means). Worse, old-time > users probably associate ASYNC_WAL and DEFERRED_FLUSH, an old > HTableDescriptor config that was deprecated and replaced by ASYNC_WAL. > DEFERRED_FLUSH ran a background thread -- LogSyncer -- that on a configurable > interval, sent a sync down the write pipeline so any outstanding appends > since last last interval start get pushed out to the DN. ASYNC_WAL doesn't > do this (see below for history on how we let go of the LogSyncer feature). > Of note, we always sync meta edits. You can't turn this off. Also, given WALs > are per regionserver, if other regions on the RS are from tables that have > sync set, these writes will push out to the DN any appends done on tables > that have DEFERRED/ASYNC_WAL set. > To fix, we could do a few things: > * Simple and comprehensive would be always queuing a sync, even if ASYNC_WAL > is set but we let go of Handlers as soon as we write the memstore -- we don't > wait on the sync to complete as we do with the default setting of > Durability=SYNC_WAL. > * Be like a 'real' database and add in a sync after N bytes of data have > been appended (configurable) or after M milliseconds have passed, which ever > threshold happens first. The size check would be easy. The sync-ever-M-millis > would mean another thread. > * Doc what ASYNC_WAL means (and other durability options) > Let me take a look and report back. Will file a bit of history on how we got > here in next comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16930) AssignmentManager.java:checkWals() function can recur infinitely
[
https://issues.apache.org/jira/browse/HBASE-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe updated HBASE-16930:
-
Description:
Recursion in AssignmentManager.checkWals() uses the same input argument. See
the code below:
{code}
private boolean checkWals(FileSystem fs, Path dir) throws IOException {
if (!fs.exists(dir)) {
LOG.debug(dir + " doesn't exist");
return false;
}
if (!fs.getFileStatus(dir).isDirectory()) {
LOG.warn(dir + " is not a directory");
return false;
}
FileStatus[] files = FSUtils.listStatus(fs, dir);
if (files == null || files.length == 0) {
LOG.debug(dir + " has no files");
return false;
}
for (int i = 0; i < files.length; i++) {
if (files[i].isFile() && files[i].getLen() > 0) {
LOG.debug(dir + " has a non-empty file: " + files[i].getPath());
return true;
} else if (files[i].isDirectory() && checkWals(fs, dir)) {
LOG.debug(dir + " is a directory and has a non-empty file: " +
files[i].getPath());
return true;
}
}
LOG.debug("Found 0 non-empty wal files for :" + dir);
return false;
}
{code}
dir is input arg and without changing it passed to call to checkWals() if child
directory is found.
was:
Recursion in AssignmentManager.checkWals() uses the same input argument. See
the code below:
{code}
private boolean checkWals(FileSystem fs, Path dir) throws IOException {
if (!fs.exists(dir)) {
LOG.debug(dir + " doesn't exist");
return false;
}
if (!fs.getFileStatus(dir).isDirectory()) {
LOG.warn(dir + " is not a directory");
return false;
}
FileStatus[] files = FSUtils.listStatus(fs, dir);
if (files == null || files.length == 0) {
LOG.debug(dir + " has no files");
return false;
}
for (int i = 0; i < files.length; i++) {
if (files[i].isFile() && files[i].getLen() > 0) {
LOG.debug(dir + " has a non-empty file: " + files[i].getPath());
return true;
} else if (files[i].isDirectory() && checkWals(fs, dir)) {
LOG.debug(dir + " is a directory and has a non-empty file: " +
files[i].getPath());
return true;
}
}
LOG.debug("Found 0 non-empty wal files for :" + dir);
return false;
}
{code}
> AssignmentManager.java:checkWals() function can recur infinitely
>
>
> Key: HBASE-16930
> URL: https://issues.apache.org/jira/browse/HBASE-16930
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>
> Recursion in AssignmentManager.checkWals() uses the same input argument. See
> the code below:
> {code}
> private boolean checkWals(FileSystem fs, Path dir) throws IOException {
> if (!fs.exists(dir)) {
> LOG.debug(dir + " doesn't exist");
> return false;
> }
> if (!fs.getFileStatus(dir).isDirectory()) {
> LOG.warn(dir + " is not a directory");
> return false;
> }
> FileStatus[] files = FSUtils.listStatus(fs, dir);
> if (files == null || files.length == 0) {
> LOG.debug(dir + " has no files");
> return false;
> }
> for (int i = 0; i < files.length; i++) {
> if (files[i].isFile() && files[i].getLen() > 0) {
> LOG.debug(dir + " has a non-empty file: " + files[i].getPath());
> return true;
> } else if (files[i].isDirectory() && checkWals(fs, dir)) {
> LOG.debug(dir + " is a directory and has a non-empty file: " +
> files[i].getPath());
> return true;
> }
> }
> LOG.debug("Found 0 non-empty wal files for :" + dir);
> return false;
> }
> {code}
> dir is input arg and without changing it passed to call to checkWals() if
> child directory is found.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16930) AssignmentManager.java:checkWals() function can recur infinitely
Umesh Agashe created HBASE-16930:
Summary: AssignmentManager.java:checkWals() function can recur
infinitely
Key: HBASE-16930
URL: https://issues.apache.org/jira/browse/HBASE-16930
Project: HBase
Issue Type: Bug
Components: Region Assignment
Reporter: Umesh Agashe
Assignee: Umesh Agashe
Recursion in AssignmentManager.checkWals() uses the same input argument. See
the code below:
{code}
private boolean checkWals(FileSystem fs, Path dir) throws IOException {
if (!fs.exists(dir)) {
LOG.debug(dir + " doesn't exist");
return false;
}
if (!fs.getFileStatus(dir).isDirectory()) {
LOG.warn(dir + " is not a directory");
return false;
}
FileStatus[] files = FSUtils.listStatus(fs, dir);
if (files == null || files.length == 0) {
LOG.debug(dir + " has no files");
return false;
}
for (int i = 0; i < files.length; i++) {
if (files[i].isFile() && files[i].getLen() > 0) {
LOG.debug(dir + " has a non-empty file: " + files[i].getPath());
return true;
} else if (files[i].isDirectory() && checkWals(fs, dir)) {
LOG.debug(dir + " is a directory and has a non-empty file: " +
files[i].getPath());
return true;
}
}
LOG.debug("Found 0 non-empty wal files for :" + dir);
return false;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16829:
---
Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
2.0.0
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16829.v1.txt, 16829.v2.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596531#comment-15596531
]
Josh Elser commented on HBASE-16829:
v2 looks better, Ted!
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16829.v1.txt, 16829.v2.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16077) Replication status doesnt show failed RS metrics in CLI
[
https://issues.apache.org/jira/browse/HBASE-16077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596532#comment-15596532
]
Demai Ni commented on HBASE-16077:
--
[~bibinchundatt], would you please elaborate a bit detail what do you expect
from the output of the CLI command? thanks
> Replication status doesnt show failed RS metrics in CLI
>
>
> Key: HBASE-16077
> URL: https://issues.apache.org/jira/browse/HBASE-16077
> Project: HBase
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>
> Steps to reproduce
>
> # Create 2 clusters and configure replication
> # Create TABLE 1 and enable table replication
> # Shutdown Cluster 2 for short period.
> # Load data to TABLE 1
> # Shutdown Region Server whr Region of TABLE 1 is available
> # Check metrics using CLI
> {noformat}
> hbase(main):003:0* status 'replication'
> 2016-06-14 00:58:04,664 INFO [main] ipc.AbstractRpcClient: RPC Server
> Kerberos principal name for service=MasterService is
> hbase/hadoop.hadoop@hadoop.com
> version 1.0.2
> 3 live servers
> host-10-19-92-200:
>SOURCE: PeerID=11, SizeOfLogQueue=0, ShippedBatches=30,
> ShippedOps=1351, ShippedBytes=1513127672, LogReadInBytes=662648911,
> LogEditsRead=1546, LogEditsFiltered=1409, SizeOfLogToReplicate=0,
> TimeWillBeTakenForLogToReplicate=0, ShippedHFiles=0, SizeOfHFileRefsQueue=0,
> AgeOfLastShippedOp=0, TimeStampsOfLastShippedOp=Tue Jun 14 00:58:01 IST 2016,
> Replication Lag=0
>SINK : AppliedBatches=2, AppliedOps=5, AppliedHFiles=3,
> AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Mon Jun 13 02:18:06 IST 2016
> host-10-19-92-187:
>SOURCE: PeerID=11, SizeOfLogQueue=0, ShippedBatches=0, ShippedOps=0,
> ShippedBytes=0, LogReadInBytes=65719, LogEditsRead=112, LogEditsFiltered=112,
> SizeOfLogToReplicate=0, TimeWillBeTakenForLogToReplicate=0, ShippedHFiles=0,
> SizeOfHFileRefsQueue=0, AgeOfLastShippedOp=0, TimeStampsOfLastShippedOp=Tue
> Jun 14 00:58:01 IST 2016, Replication Lag=0
>SINK : AppliedBatches=0, AppliedOps=0, AppliedHFiles=0,
> AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Mon Jun 13 09:07:20 IST 2016
> host-10-19-92-188:
>SOURCE: PeerID=11, SizeOfLogQueue=0, ShippedBatches=39,
> ShippedOps=1730, ShippedBytes=1937609744, LogReadInBytes=848439638,
> LogEditsRead=1671, LogEditsFiltered=1497, SizeOfLogToReplicate=0,
> TimeWillBeTakenForLogToReplicate=0, ShippedHFiles=0, SizeOfHFileRefsQueue=0,
> AgeOfLastShippedOp=0, TimeStampsOfLastShippedOp=Tue Jun 14 00:58:03 IST 2016,
> Replication Lag=0
>SINK : AppliedBatches=1, AppliedOps=1, AppliedHFiles=0,
> AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Mon Jun 13 01:53:53 IST 2016
> {noformat}
> *JMX output*
> {noformat}
> {
> "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication",
> "modelerType" : "RegionServer,sub=Replication",
> "tag.Context" : "regionserver",
> "tag.Hostname" : "host-10-19-92-200",
> "source.11.sizeOfLogToReplicate" : 537,
> "source.11-host-10-19-92-187,21302,1465787242095.sizeOfLogToReplicate" :
> 282766680,
> "source.shippedHFiles" : 0,
> "source.ageOfLastShippedOp" : 0,
> "source.11.shippedHFiles" : 0,
> "source.11-host-10-19-92-187,21302,1465787242095.ageOfLastShippedOp" : 0,
> "source.shippedKBs" : 1477663,
> "source.sizeOfHFileRefsQueue" : 0,
> "source.logReadInBytes" : 691148656,
> "source.11-host-10-19-92-187,21302,1465787242095.logEditsRead" : 39,
> "source.11-host-10-19-92-187,21302,1465787242095.shippedOps" : 0,
> "source.11.logEditsFiltered" : 1244,
> "source.sizeOfLogQueue" : 4,
> "source.timeWillBeTakenForLogToReplicate" : 1,
> "sink.ageOfLastAppliedOp" : 0,
>
> "source.11-host-10-19-92-187,21302,1465787242095.timeWillBeTakenForLogToReplicate"
> : 0,
> "source.logEditsRead" : 1420,
> "source.11.sizeOfLogQueue" : 0,
> "source.11-host-10-19-92-187,21302,1465787242095.logEditsFiltered" : 32,
> "source.11-host-10-19-92-187,21302,1465787242095.shippedHFiles" : 0,
> "source.shippedOps" : 1351,
> "source.11.shippedKBs" : 1477663,
> "source.11.logReadInBytes" : 662562515,
> "sink.appliedHFiles" : 3,
> "source.11.sizeOfHFileRefsQueue" : 0,
> "source.logEditsFiltered" : 1276,
> "source.shippedBytes" : 1513127672,
> "source.11-host-10-19-92-187,21302,1465787242095.shippedBatches" : 0,
> "source.11.shippedBytes" : 1513127672,
> "sink.appliedOps" : 5,
> "source.11-host-10-19-92-187,21302,1465787242095.sizeOfLogQueue" : 4,
> "source.11.shippedBatches" : 30,
> "source.11-host-10-19-92-187,21302,1465787242095.sizeOfHFileRefsQueue" :
> 0,
> "source.11.timeWillBeTakenForLogToReplicate" : 1,
> "source.11-host-10-19-92-187,21302,1465787242095.logReadInBytes" :
[jira] [Updated] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16829:
---
Attachment: 16829.v2.txt
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16829.v1.txt, 16829.v2.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16827: -- Description: # Make server_name type ServerName (from HBase.proto) # Make all fields optional # Remove duplicate fields from Backup.proto was:Make server_name type ServerName (from HBase.proto) > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch, > HBASE-16827-v3.patch, HBASE-16827-v4.patch, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > # Make server_name type ServerName (from HBase.proto) > # Make all fields optional > # Remove duplicate fields from Backup.proto -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596519#comment-15596519
]
Josh Elser commented on HBASE-16829:
Actually, they were just geting a normal GSSException when the client used
"hbase" but the thrift2 server was using a principal with a different
"primary", but thanks for the mention anyways :)
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16829.v1.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596517#comment-15596517
]
Josh Elser commented on HBASE-16829:
{code}
+ int slashIdx = principal.indexOf("/");
+ int atIdx = principal.indexOf("@");
+ int idx = slashIdx != -1 ? slashIdx : atIdx;
+ user = principal.substring(0, idx);
{code}
There is a class somewhere that can do this parsing for you..
org.apache.hadoop.security.authentication.util.KerberosName. I guess it is
marked as LimitedPrivate, so maybe better to just leave it as you have it. Oh,
but the realm is optional, so you should not require that you find an {{@}}
symbol.
Otherwise, LGTM.
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16829.v1.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-12074) TestLogRollingNoCluster#testContendedLogRolling() failed
[
https://issues.apache.org/jira/browse/HBASE-12074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar resolved HBASE-12074.
---
Resolution: Duplicate
Fixed via HBASE-16824. This issue was originally filed against 0.98 which did
not get HBASE-16824. Please reopen if the issue is still seen there.
> TestLogRollingNoCluster#testContendedLogRolling() failed
>
>
> Key: HBASE-12074
> URL: https://issues.apache.org/jira/browse/HBASE-12074
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Stephen Yuan Jiang
>
> TestLogRollingNoCluster#testContendedLogRolling() failed on a 0.98 run. I am
> trying to understand the context.
> The failure is this:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster.testContendedLogRolling(TestLogRollingNoCluster.java:80)
> {code}
> Caused because one of the Appenders calling FSHLog.sync() threw IOE because
> of concurrent close:
> {code}
> 4-09-23 16:36:39,530 FATAL [pool-1-thread-1-WAL.AsyncSyncer0]
> wal.FSHLog$AsyncSyncer(1246): Error while AsyncSyncer sync, request close of
> hlog
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:168)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1241)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:165)
> ... 2 more
> 2014-09-23 16:36:39,531 INFO [32] wal.TestLogRollingNoCluster$Appender(137):
> Caught exception from Appender:32
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:168)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1241)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:165)
> ... 2 more
> 2014-09-23 16:36:39,532 INFO [19] wal.TestLogRollingNoCluster$Appender(137):
> Caught exception from Appender:19
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:168)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1241)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:165)
> ... 2 more
> {code}
> The code is:
> {code}
> public void sync() throws IOException {
> try {
> this.output.flush();
> this.output.sync();
> } catch (NullPointerException npe) {
> // Concurrent close...
> throw new IOException(npe);
> }
> }
> {code}
> I think the test case written exactly to catch this case:
> {code}
>* Spin up a bunch of threads and have them all append to a WAL. Roll the
>* WAL frequently to try and trigger NPE.
> {code}
> This is why I am reporting since I don't have much context. It may not be a
> test issue, but an actual bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596488#comment-15596488
]
Enis Soztutar commented on HBASE-16698:
---
On master, when I do a jstack with some load, almost all the handlers are
waiting for sync(), and since the memstore insert does not happen until sync()
completes, we do not have to wait for the latch. Actually, the latch can be
removed as well (for Durability.SYNC_WAL). For async, we still have to have the
latch I think.
The other reason I was asking about this is that I have a hacked up patch which
divides the batchMutate() into 3 phases. The first phase is the parts to
compute the WALEdit to write, second part is the actual append + sync() and the
last part is all other steps after sync() (memstore insert, etc). This
architecture makes it possible to do the flush pipelining idea talked about
here: http://pandis.net/resources/vldb10aether.pdf (section 4). The theory is
that we can free up the handler to process more requests while we are waiting
for the sync to happen. After sync some other handler or thread will complete
the work.
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch, hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596469#comment-15596469 ] Ted Yu commented on HBASE-16827: Please update the description with actual changes. Meanwhile I am running backup tests locally. > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch, > HBASE-16827-v3.patch, HBASE-16827-v4.patch, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > Make server_name type ServerName (from HBase.proto) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[
https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596429#comment-15596429
]
Hadoop QA commented on HBASE-16827:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HBASE-16827 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834759/HBASE-16827-v4.patch |
| JIRA Issue | HBASE-16827 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4144/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Backup.proto refactoring
>
>
> Key: HBASE-16827
> URL: https://issues.apache.org/jira/browse/HBASE-16827
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch,
> HBASE-16827-v3.patch, HBASE-16827-v4.patch,
> org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt
>
>
> Make server_name type ServerName (from HBase.proto)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16929) Move default method of shipped to Shipper interface
[
https://issues.apache.org/jira/browse/HBASE-16929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16929:
---
Attachment: 16929.v1.txt
> Move default method of shipped to Shipper interface
> ---
>
> Key: HBASE-16929
> URL: https://issues.apache.org/jira/browse/HBASE-16929
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16929.v1.txt
>
>
> HBASE-16626 added default method of shipped() to RegionScanner.
> However, when building master branch of Phoenix against 2.0 SNAPSHOT, I got:
> {code}
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/DelegateRegionScanner.java:[27,8]
> org.apache.phoenix.coprocessor.DelegateRegionScanner is not abstract and
> does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java:[344,36]
> is
> not abstract and does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> {code}
> Here is the snippet for DelegateRegionScanner:
> {code}
> public class DelegateRegionScanner implements RegionScanner {
> {code}
> It seems adding default method in RegionScanner is not enough for downstream
> projects.
> After moving the default method to Shipper interface, the above two
> compilation errors are gone in Phoenix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16929) Move default method of shipped to Shipper interface
[
https://issues.apache.org/jira/browse/HBASE-16929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16929:
---
Fix Version/s: 2.0.0
Status: Patch Available (was: Open)
> Move default method of shipped to Shipper interface
> ---
>
> Key: HBASE-16929
> URL: https://issues.apache.org/jira/browse/HBASE-16929
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16929.v1.txt
>
>
> HBASE-16626 added default method of shipped() to RegionScanner.
> However, when building master branch of Phoenix against 2.0 SNAPSHOT, I got:
> {code}
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/DelegateRegionScanner.java:[27,8]
> org.apache.phoenix.coprocessor.DelegateRegionScanner is not abstract and
> does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> [ERROR]
> /a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java:[344,36]
> is
> not abstract and does not override abstract method shipped() in
> org.apache.hadoop.hbase.regionserver.Shipper
> {code}
> Here is the snippet for DelegateRegionScanner:
> {code}
> public class DelegateRegionScanner implements RegionScanner {
> {code}
> It seems adding default method in RegionScanner is not enough for downstream
> projects.
> After moving the default method to Shipper interface, the above two
> compilation errors are gone in Phoenix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16929) Move default method of shipped to Shipper interface
Ted Yu created HBASE-16929:
--
Summary: Move default method of shipped to Shipper interface
Key: HBASE-16929
URL: https://issues.apache.org/jira/browse/HBASE-16929
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
HBASE-16626 added default method of shipped() to RegionScanner.
However, when building master branch of Phoenix against 2.0 SNAPSHOT, I got:
{code}
[ERROR]
/a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/DelegateRegionScanner.java:[27,8]
org.apache.phoenix.coprocessor.DelegateRegionScanner is not abstract and does
not override abstract method shipped() in
org.apache.hadoop.hbase.regionserver.Shipper
[ERROR]
/a/phoenix/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/BaseScannerRegionObserver.java:[344,36]
is not
abstract and does not override abstract method shipped() in
org.apache.hadoop.hbase.regionserver.Shipper
{code}
Here is the snippet for DelegateRegionScanner:
{code}
public class DelegateRegionScanner implements RegionScanner {
{code}
It seems adding default method in RegionScanner is not enough for downstream
projects.
After moving the default method to Shipper interface, the above two compilation
errors are gone in Phoenix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16827: -- Attachment: HBASE-16827-v4.patch v4. cc: [~tedyu] > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch, > HBASE-16827-v3.patch, HBASE-16827-v4.patch, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > Make server_name type ServerName (from HBase.proto) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16877) Remove directory layout/ filesystem references form Master
[
https://issues.apache.org/jira/browse/HBASE-16877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe updated HBASE-16877:
-
Description:
Remove directory layout/ filesystem references form Master and add required
APIs to MasterStorage/ RegionStorage
{code}
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterWalManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlacementMaintainer.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureEnv.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java
{code}
was:
Remove directory layout/ filesystem references form Master and add required
APIs to MasterStorage/ RegionStorage
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterWalManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlacementMaintainer.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureEnv.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java
> Remove directory layout/ filesystem references form Master
> --
>
> Key: HBASE-16877
> URL: https://issues.apache.org/jira/browse/HBASE-16877
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>
> Remove directory layout/ filesystem references form Master and add required
> APIs to MasterStorage/ RegionStorage
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterWalManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlacementMaintainer.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureEnv.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eddie Elizondo updated HBASE-14329: --- Comment: was deleted (was: Check and throw an exception for every element in the RegionInfoList) > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 1.3.1 > > Attachments: HBASE-14329.patch > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eddie Elizondo updated HBASE-14329: --- Status: Patch Available (was: In Progress) Check and throw an exception for every element in the RegionInfoList > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 1.3.1 > > Attachments: HBASE-14329.patch > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eddie Elizondo updated HBASE-14329: --- Status: Patch Available (was: Open) Check and throw an exception for every element in the RegionInfoList > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 1.3.1 > > Attachments: HBASE-14329.patch > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eddie Elizondo updated HBASE-14329: --- Status: In Progress (was: Patch Available) > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 1.3.1 > > Attachments: HBASE-14329.patch > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596390#comment-15596390
]
Hadoop QA commented on HBASE-16829:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 1s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
6s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 49s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834745/16829.v1.txt |
| JIRA Issue | HBASE-16829 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 42a6dbc105c3 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e223639 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4142/testReport/ |
| modules | C: hbase-examples U: hbase-examples |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4142/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> DemoClient should detect secure mode
>
>
>
[jira] [Updated] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eddie Elizondo updated HBASE-14329: --- Attachment: HBASE-14329.patch > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 1.3.1 > > Attachments: HBASE-14329.patch > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eddie Elizondo updated HBASE-14329: --- Status: Patch Available (was: Open) > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 1.3.1 > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eddie Elizondo updated HBASE-14329: --- Status: Open (was: Patch Available) > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 1.3.1 > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work stopped] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-14329 stopped by Eddie Elizondo. -- > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 1.3.1 > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-14329 started by Eddie Elizondo. -- > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 1.3.1 > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596337#comment-15596337
]
Ted Yu commented on HBASE-16829:
Thanks to Josh who found the case where mismatched user name for
hbase.thrift.kerberos.principal led to the mysterious SocketTimeoutException.
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16829.v1.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16829:
---
Status: Patch Available (was: Open)
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16829.v1.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-16829:
--
Assignee: Ted Yu
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16829.v1.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16829) DemoClient should detect secure mode
[
https://issues.apache.org/jira/browse/HBASE-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16829:
---
Attachment: 16829.v1.txt
> DemoClient should detect secure mode
>
>
> Key: HBASE-16829
> URL: https://issues.apache.org/jira/browse/HBASE-16829
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Attachments: 16829.v1.txt
>
>
> Sometimes user runs DemoClient against secure cluster using the following
> command:
> {code}
> java -cp `hbase classpath` org.apache.hadoop.hbase.thrift2.DemoClient host
> 9090
> {code}
> This wouldn't work.
> DemoClient should detect the presence of secure cluster - e.g. through the
> following config:
> {code}
>
> hbase.thrift.authentication.type
> kerberos
>
> {code}
> User can still specify 'false' as the last parameter if the target cluster is
> insecure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16867) Procedure V2 - Check ACLs for remote HBaseLock
[ https://issues.apache.org/jira/browse/HBASE-16867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16867: - Attachment: HBASE-16867.master.004.patch > Procedure V2 - Check ACLs for remote HBaseLock > -- > > Key: HBASE-16867 > URL: https://issues.apache.org/jira/browse/HBASE-16867 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16867.master.001.patch, > HBASE-16867.master.002.patch, HBASE-16867.master.003.patch, > HBASE-16867.master.004.patch > > > HBaseLock was added in HBASE-16744 to allow clients to take locks on > namespace/table/regions. Check acls for the 2 rpcs that were added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16754) Regions failing compaction due to referencing non-existent store file
[
https://issues.apache.org/jira/browse/HBASE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596142#comment-15596142
]
Hudson commented on HBASE-16754:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #50 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/50/])
HBASE-16754 All WALSplitter OutputSinks should process compaction events
(garyh: rev fdb921c89346addb9daba338a1472a4bb76fb2e1)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
* (add) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java
> Regions failing compaction due to referencing non-existent store file
> -
>
> Key: HBASE-16754
> URL: https://issues.apache.org/jira/browse/HBASE-16754
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.3
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5
>
> Attachments: HBASE-16754.001.patch, HBASE-16754.branch-1.001.patch,
> HBASE-16754.branch-1.2.001.patch
>
>
> Running a mixed read write workload on a recent build off branch-1.3, we are
> seeing compactions occasionally fail with errors like the following (actual
> filenames replaced with placeholders):
> {noformat}
> 16/09/27 16:57:28 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 116
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
>
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
>
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1644)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:373)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$100(CompactSplitThread.java:59)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:498)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:568)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/09/27 17:01:31 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 115
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache
[jira] [Commented] (HBASE-15347) Update CHANGES.txt for 1.3
[ https://issues.apache.org/jira/browse/HBASE-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596132#comment-15596132 ] Mikhail Antonov commented on HBASE-15347: - This step `comm -1 -3 CHANGES_1.3.0_git.txt CHANGES_1.3.0_jira.txt` (issues in jira but not in git) report 452 discrepancies in Jira :( Looking through. > Update CHANGES.txt for 1.3 > -- > > Key: HBASE-15347 > URL: https://issues.apache.org/jira/browse/HBASE-15347 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > Attachments: HBASE-15347-branch-1.3.v1.patch > > > Going to post the steps in preparing changes file for 1.3 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-14329) Report region in transition only ever operates on one region
[ https://issues.apache.org/jira/browse/HBASE-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling reassigned HBASE-14329: - Assignee: Eddie Elizondo > Report region in transition only ever operates on one region > > > Key: HBASE-14329 > URL: https://issues.apache.org/jira/browse/HBASE-14329 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Eddie Elizondo > Fix For: 1.3.1 > > > Report region in transition takes a list of regions but it only ever operates > on one region however more than one region can be reported. > Seems like this could cause some serious weirdness on failure cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16754) Regions failing compaction due to referencing non-existent store file
[
https://issues.apache.org/jira/browse/HBASE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-16754:
--
Resolution: Fixed
Fix Version/s: 1.2.5
Status: Resolved (was: Patch Available)
Committed to branch-1.2+.
> Regions failing compaction due to referencing non-existent store file
> -
>
> Key: HBASE-16754
> URL: https://issues.apache.org/jira/browse/HBASE-16754
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.3
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5
>
> Attachments: HBASE-16754.001.patch, HBASE-16754.branch-1.001.patch,
> HBASE-16754.branch-1.2.001.patch
>
>
> Running a mixed read write workload on a recent build off branch-1.3, we are
> seeing compactions occasionally fail with errors like the following (actual
> filenames replaced with placeholders):
> {noformat}
> 16/09/27 16:57:28 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 116
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
>
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
>
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1644)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:373)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$100(CompactSplitThread.java:59)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:498)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:568)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/09/27 17:01:31 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 115
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
>
[jira] [Commented] (HBASE-15347) Update CHANGES.txt for 1.3
[ https://issues.apache.org/jira/browse/HBASE-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596054#comment-15596054 ] Mikhail Antonov commented on HBASE-15347: - Restarting the work on release notes for 1.3. https://paste.apache.org/WuCo - initial list of things-in-git-but-not-in jira that I've cleaned up. - Some jiras committed then reverted are false-positives - some jiras with missed or incorrect fixVersion - bunch of jiras labeled against 1.3.1, whereas should be as 1.3.0 > Update CHANGES.txt for 1.3 > -- > > Key: HBASE-15347 > URL: https://issues.apache.org/jira/browse/HBASE-15347 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > Attachments: HBASE-15347-branch-1.3.v1.patch > > > Going to post the steps in preparing changes file for 1.3 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[ https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16824: Fix Version/s: 1.1.8 1.2.5 1.4.0 1.3.0 2.0.0 > Writer.flush() can be called on already closed streams in WAL roll > -- > > Key: HBASE-16824 > URL: https://issues.apache.org/jira/browse/HBASE-16824 > Project: HBase > Issue Type: Bug >Reporter: Atri Sharma >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8 > > Attachments: hbase-16824_v1.patch, hbase-16824_v2.patch, > hbase-16824_v2.patch > > > In https://issues.apache.org/jira/browse/HBASE-12074, we hit an error if an > async thread calls flush on a WAL record already closed as the WAL is being > rotated. This JIRA investigates if setting the new WAL record path as the > first operation during WAL rotation will fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16824) Writer.flush() can be called on already closed streams in WAL roll
[ https://issues.apache.org/jira/browse/HBASE-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596048#comment-15596048 ] Mikhail Antonov commented on HBASE-16824: - set fixVersions based on the reported integ-history above > Writer.flush() can be called on already closed streams in WAL roll > -- > > Key: HBASE-16824 > URL: https://issues.apache.org/jira/browse/HBASE-16824 > Project: HBase > Issue Type: Bug >Reporter: Atri Sharma >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8 > > Attachments: hbase-16824_v1.patch, hbase-16824_v2.patch, > hbase-16824_v2.patch > > > In https://issues.apache.org/jira/browse/HBASE-12074, we hit an error if an > async thread calls flush on a WAL record already closed as the WAL is being > rotated. This JIRA investigates if setting the new WAL record path as the > first operation during WAL rotation will fix the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12074) TestLogRollingNoCluster#testContendedLogRolling() failed
[
https://issues.apache.org/jira/browse/HBASE-12074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596033#comment-15596033
]
Mikhail Antonov commented on HBASE-12074:
-
Should this one be closed as a duplicate, as indicated by the linked jira?
> TestLogRollingNoCluster#testContendedLogRolling() failed
>
>
> Key: HBASE-12074
> URL: https://issues.apache.org/jira/browse/HBASE-12074
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Stephen Yuan Jiang
>
> TestLogRollingNoCluster#testContendedLogRolling() failed on a 0.98 run. I am
> trying to understand the context.
> The failure is this:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster.testContendedLogRolling(TestLogRollingNoCluster.java:80)
> {code}
> Caused because one of the Appenders calling FSHLog.sync() threw IOE because
> of concurrent close:
> {code}
> 4-09-23 16:36:39,530 FATAL [pool-1-thread-1-WAL.AsyncSyncer0]
> wal.FSHLog$AsyncSyncer(1246): Error while AsyncSyncer sync, request close of
> hlog
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:168)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1241)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:165)
> ... 2 more
> 2014-09-23 16:36:39,531 INFO [32] wal.TestLogRollingNoCluster$Appender(137):
> Caught exception from Appender:32
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:168)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1241)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:165)
> ... 2 more
> 2014-09-23 16:36:39,532 INFO [19] wal.TestLogRollingNoCluster$Appender(137):
> Caught exception from Appender:19
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:168)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1241)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:165)
> ... 2 more
> {code}
> The code is:
> {code}
> public void sync() throws IOException {
> try {
> this.output.flush();
> this.output.sync();
> } catch (NullPointerException npe) {
> // Concurrent close...
> throw new IOException(npe);
> }
> }
> {code}
> I think the test case written exactly to catch this case:
> {code}
>* Spin up a bunch of threads and have them all append to a WAL. Roll the
>* WAL frequently to try and trigger NPE.
> {code}
> This is why I am reporting since I don't have much context. It may not be a
> test issue, but an actual bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-16723:
Fix Version/s: (was: 1.3.1)
1.3.0
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 0.98.24
>
> Attachments: HBASE-16723-0.98.patch, HBASE-16723-V2.patch,
> HBASE-16723-V3.patch, HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover
[
https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-16807:
Fix Version/s: (was: 1.3.1)
1.3.0
> RegionServer will fail to report new active Hmaster until
> HMaster/RegionServer failover
> ---
>
> Key: HBASE-16807
> URL: https://issues.apache.org/jira/browse/HBASE-16807
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.8
>
> Attachments: HBASE-16807-0.98.patch, HBASE-16807-branch-1.1.patch,
> HBASE-16807-branch-1.2.patch, HBASE-16807-branch-1.3.patch,
> HBASE-16807-branch-1.patch, HBASE-16807.patch
>
>
> It's little weird, but it happened in the product environment that few
> RegionServer missed master znode create notification on master failover. In
> that case ZooKeeperNodeTracker will not refresh the cached data and
> MasterAddressTracker will always return old active HM detail to Region server
> on ServiceException.
> Though We create region server stub on failure but without refreshing the
> MasterAddressTracker data.
> In HRegionServer.createRegionServerStatusStub()
> {code}
> boolean refresh = false; // for the first time, use cached data
> RegionServerStatusService.BlockingInterface intf = null;
> boolean interrupted = false;
> try {
> while (keepLooping()) {
> sn = this.masterAddressTracker.getMasterAddress(refresh);
> if (sn == null) {
> if (!keepLooping()) {
> // give up with no connection.
> LOG.debug("No master found and cluster is stopped; bailing out");
> return null;
> }
> if (System.currentTimeMillis() > (previousLogTime + 1000)) {
> LOG.debug("No master found; retry");
> previousLogTime = System.currentTimeMillis();
> }
> refresh = true; // let's try pull it from ZK directly
> if (sleep(200)) {
> interrupted = true;
> }
> continue;
> }
> {code}
> Here we refresh node only when 'sn' is NULL otherwise it will use same cached
> data.
> So in above case RegionServer will never report active HMaster successfully
> until HMaster failover or RegionServer restart.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14439) New/Improved Filesystem Abstractions
[ https://issues.apache.org/jira/browse/HBASE-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596007#comment-15596007 ] Sean Busbey commented on HBASE-14439: - note: TEST sub-tasks probably need to happen after the non-TEST sub-tasks. > New/Improved Filesystem Abstractions > > > Key: HBASE-14439 > URL: https://issues.apache.org/jira/browse/HBASE-14439 > Project: HBase > Issue Type: Improvement >Reporter: Ben Lau >Assignee: Sean Busbey > Attachments: abstraction.patch > > > Ticket for work in progress on new FileSystem abstractions. Previously, we > (Yahoo) submitted a ticket that would add support for humongous (1 million > region+) tables via a hierarchical layout (HBASE-13991). However open source > is moving in a similar but not identical direction in the future and so the > patch will not be merged into open source. > We will be working on a different patch now with folks from open source. It > will create/add to 2 layers-- a path abstraction layer and a use-oriented > abstraction layer. The path abstraction layer is epitomized by classes like > FsUtils (and in the patch new classes like AFsLayout). The use oriented > abstraction layer is epitomized by existing classes like > MasterFileSystem/HRegionFileSystem (and possibly new classes later) that > build on the path abstraction layer and focus on 'doing things' (eg creating > regions) and less on the gritty details like the paths. > This work on abstracting and isolating the paths from the use cases will help > Yahoo not diverge too much from open source with its internal 'Humongous' > table hierarchical layout, while also helping open source move further > towards the eventual goal of redoing the FS layout in a similar (but > different) hierarchical layout later that focuses on data directory > uniformity (unlike the humongous patch) and storing hierarchy in the meta > table instead which enables new optimizations (see HBASE-14090.) > Attached to this ticket is some work we've done at Yahoo so far that will be > put into an open source HBase branch for further collaboration. The patch is > not meant to be complete yet and is a work in progress. (Please wait on > patch comments/reviews.) It also includes some Yahoo-specific 'humongous' > layout code that will be removed before submission in open source. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16664: Fix Version/s: (was: 1.3.1) 1.3.0 > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1-v2.patch, > HBASE-16664-branch-1.1-v1.patch, HBASE-16664-branch-1.2-v1.patch, > HBASE-16664-branch-1.3-v1.patch, HBASE-16664-branch-1.3-v2.patch, > HBASE-16664-branch-1.3.v3.patch, HBASE-16664-branch-1.v3.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, HBASE-16664-v6.patch, > HBASE-16664-v7.patch, testhcm.patch > > > Rpc/operation timeout logic in AsyncProcess is broken. And Table's > set*Timeout does not take effect in its AP or BufferedMutator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596001#comment-15596001 ] Mikhail Antonov commented on HBASE-16664: - [~Apache9] [~chenheng] acked, thanks. labeled as in for 1.3.0 > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: 1.patch, HBASE-16664-branch-1-v1.patch, > HBASE-16664-branch-1-v1.patch, HBASE-16664-branch-1-v2.patch, > HBASE-16664-branch-1.1-v1.patch, HBASE-16664-branch-1.2-v1.patch, > HBASE-16664-branch-1.3-v1.patch, HBASE-16664-branch-1.3-v2.patch, > HBASE-16664-branch-1.3.v3.patch, HBASE-16664-branch-1.v3.patch, > HBASE-16664-v1.patch, HBASE-16664-v2.patch, HBASE-16664-v3.patch, > HBASE-16664-v4.patch, HBASE-16664-v5.patch, HBASE-16664-v6.patch, > HBASE-16664-v7.patch, testhcm.patch > > > Rpc/operation timeout logic in AsyncProcess is broken. And Table's > set*Timeout does not take effect in its AP or BufferedMutator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16928) TEST: cleanup misc tests to ensure no direct filesystem use
Sean Busbey created HBASE-16928: --- Summary: TEST: cleanup misc tests to ensure no direct filesystem use Key: HBASE-16928 URL: https://issues.apache.org/jira/browse/HBASE-16928 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey clean up references to filesystem and legacy implementation in a few tests that don't fit well in our other issues: hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin1.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestClusterId.java hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestRootPath.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16927) TEST: update tests for misc regionserver tests
Sean Busbey created HBASE-16927: --- Summary: TEST: update tests for misc regionserver tests Key: HBASE-16927 URL: https://issues.apache.org/jira/browse/HBASE-16927 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey ensure remaining tests for regionserver use masterstorage and regionstorage rather than legacy implementation or direct filesystem access. hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction
[
https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595991#comment-15595991
]
stack commented on HBASE-13408:
---
[~ram_krish], [~anoopsamjohn], [~eshcar], [~ebortnik] and myself met last Weds
morning, the 19th of October to chat about where we are all at on inmemory
compaction. Here are some rough notes:
{code}
1. Reiteration that inmemory compaction needs to be on all the time with no
associated perf degradation and with minimal config required to get benefit.
2. Need more testing and with more variety (Zipfian so inmemory compaction gets
a chance to shine). We'll all pitch in here.
3. When to merge (Eshcar "It is just a question of when.."). Back and forth.
Concern that current default of merge on every flush is a default that will
make inmemory compaction look bad because it for sure generates loads of
garbage. Suggestion to go to other extreme where we merge only at flush-to-disk
letting the pipeline build up in memory.
4. We'll pick up CellChunkMap at the next meeting but meantime wukk revive a
rumored existing umbrella issue. CCM is where we'll get biggest bang for the
buck so excited to get this done. Need to solve the cell-too-big-issue still.
Maybe split its dev some between Y! and Intel/St.Ack... TBD.
5. Ram and Anoop fixed some GC issues in "HBASE-16608 Introducing the ability
to merge ImmutableSegments without copy-compaction or SQM usage" and will put
up a new version of patch w/ fixes. Generally agreed patch is close to commit.
The Y! crew are on holiday until next Tuesday.
Did the note to the dev list go up on state of inmemory compaction (after
Tuesday -- smile).
...
The back-and-forth about when/how to merge will probably continue into
HBase-16417; we have to keep looking for the sweet spot.
{code}
> HBase In-Memory Memstore Compaction
> ---
>
> Key: HBASE-13408
> URL: https://issues.apache.org/jira/browse/HBASE-13408
> Project: HBase
> Issue Type: New Feature
>Reporter: Eshcar Hillel
> Attachments: HBASE-13408-trunk-v01.patch,
> HBASE-13408-trunk-v02.patch, HBASE-13408-trunk-v03.patch,
> HBASE-13408-trunk-v04.patch, HBASE-13408-trunk-v05.patch,
> HBASE-13408-trunk-v06.patch, HBASE-13408-trunk-v07.patch,
> HBASE-13408-trunk-v08.patch, HBASE-13408-trunk-v09.patch,
> HBASE-13408-trunk-v10.patch,
> HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf,
> HBaseIn-MemoryMemstoreCompactionDesignDocument-ver03.pdf,
> HBaseIn-MemoryMemstoreCompactionDesignDocument-ver04.pdf,
> HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf,
> InMemoryMemstoreCompactionEvaluationResults.pdf,
> InMemoryMemstoreCompactionMasterEvaluationResults.pdf,
> InMemoryMemstoreCompactionScansEvaluationResults.pdf,
> StoreSegmentandStoreSegmentScannerClassHierarchies.pdf
>
>
> A store unit holds a column family in a region, where the memstore is its
> in-memory component. The memstore absorbs all updates to the store; from time
> to time these updates are flushed to a file on disk, where they are
> compacted. Unlike disk components, the memstore is not compacted until it is
> written to the filesystem and optionally to block-cache. This may result in
> underutilization of the memory due to duplicate entries per row, for example,
> when hot data is continuously updated.
> Generally, the faster the data is accumulated in memory, more flushes are
> triggered, the data sinks to disk more frequently, slowing down retrieval of
> data, even if very recent.
> In high-churn workloads, compacting the memstore can help maintain the data
> in memory, and thereby speed up data retrieval.
> We suggest a new compacted memstore with the following principles:
> 1.The data is kept in memory for as long as possible
> 2.Memstore data is either compacted or in process of being compacted
> 3.Allow a panic mode, which may interrupt an in-progress compaction and
> force a flush of part of the memstore.
> We suggest applying this optimization only to in-memory column families.
> A design document is attached.
> This feature was previously discussed in HBASE-5311.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16926) TEST: update unit tests for io package
Umesh Agashe created HBASE-16926:
Summary: TEST: update unit tests for io package
Key: HBASE-16926
URL: https://issues.apache.org/jira/browse/HBASE-16926
Project: HBase
Issue Type: Sub-task
Components: Filesystem Integration
Reporter: Umesh Agashe
List of files:
{code}
hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestFileLink.java
hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHFileLink.java
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16914) TEST: Remove directory layout/ filesystem references form hbck unit tests
[
https://issues.apache.org/jira/browse/HBASE-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-16914:
Description:
Remove directory layout/ filesystem references form hbck unit tests. List of
files:
{code}
hbase-server/src/test/java/org/apache/hadoop/hbase/util/BaseTestHBaseFsck.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckMOB.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckOneRS.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckTwoRS.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java
{code}
was:
Remove directory layout/ filesystem references form hbck unit tests. List of
files:
{code}
hbase-server/src/test/java/org/apache/hadoop/hbase/util/BaseTestHBaseFsck.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckMOB.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckOneRS.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckTwoRS.java
{code}
> TEST: Remove directory layout/ filesystem references form hbck unit tests
> -
>
> Key: HBASE-16914
> URL: https://issues.apache.org/jira/browse/HBASE-16914
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Umesh Agashe
>
> Remove directory layout/ filesystem references form hbck unit tests. List of
> files:
> {code}
> hbase-server/src/test/java/org/apache/hadoop/hbase/util/BaseTestHBaseFsck.java
> hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckMOB.java
> hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckOneRS.java
> hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckTwoRS.java
> hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16925) TEST: update compaction tests
[ https://issues.apache.org/jira/browse/HBASE-16925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16925: Description: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSplitThread.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionPolicy.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java > TEST: update compaction tests > - > > Key: HBASE-16925 > URL: https://issues.apache.org/jira/browse/HBASE-16925 > Project: HBase > Issue Type: Sub-task >Reporter: Sean Busbey > > hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSplitThread.java > hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionPolicy.java > hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16924) TEST: update merge tests
[ https://issues.apache.org/jira/browse/HBASE-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16924: Description: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransaction.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java > TEST: update merge tests > > > Key: HBASE-16924 > URL: https://issues.apache.org/jira/browse/HBASE-16924 > Project: HBase > Issue Type: Sub-task >Reporter: Sean Busbey > > hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransaction.java > hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java > hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
