[jira] [Commented] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425991#comment-15425991
]
Hudson commented on HBASE-16429:
FAILURE: Integrated in Jenkins build HBase-1.3-IT #793 (See
[https://builds.apache.org/job/HBase-1.3-IT/793/])
HBASE-16429 FSHLog: deadlock if rollWriter called when ring buffer (liyu: rev
e3ed8ecdb3ce7ee727b525f88bc58650559ebc7e)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> FSHLog: deadlock if rollWriter called when ring buffer filled with appends
> --
>
> Key: HBASE-16429
> URL: https://issues.apache.org/jira/browse/HBASE-16429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2, 1.2.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7
>
> Attachments: HBASE-16429.patch
>
>
> Recently we experienced an online problem that all handlers are stuck.
> Checking the jstack we could see all handler threads waiting for
> RingBuffer.next, while the single ring buffer consumer dead waiting for
> {{safePointReleasedLatch}} to count down:
> {noformat}
> Normal handler thread:
> "B.defaultRpcServer.handler=126,queue=9,port=16020" daemon prio=10
> tid=0x7efd4b44f800 nid=0x15f29 runnable [0x7efd3db7b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1222)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:736)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:698)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2095)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:774)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:756)
> RingBufferEventHandler thread waiting for safePointReleasedLatch:
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.append-pool2-t1"
> prio=10 tid=0x7efd320d nid=0x1777b waiting on condition
> [0x7efd2d2fa000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f01b48d9178> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.safePointAttained(FSHLog.java:1866)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:2066)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1909)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Wo
[jira] [Commented] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425990#comment-15425990
]
Yu Li commented on HBASE-16429:
---
It seems the post-commit job failed since [build
#344|https://builds.apache.org/job/HBase-1.4/344/#showFailuresLink] and main
error is caused by {{HBaseConfiguration.checkDefaultsVersion}} invocation, such
as NPE or unknown version, like below:
{noformat}
java.lang.RuntimeException: hbase-default.xml file seems to be for an older
version of HBase (1.4.0-SNAPSHOT), this version is Unknown
at
org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:71)
at
org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81)
at
org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96)
at
org.apache.hadoop.hbase.io.crypto.aes.TestAES.testAESAlgorithm(TestAES.java:54)
{noformat}
will check whether there's any JIRA already tracking this, or open a new one to
resolve it.
> FSHLog: deadlock if rollWriter called when ring buffer filled with appends
> --
>
> Key: HBASE-16429
> URL: https://issues.apache.org/jira/browse/HBASE-16429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2, 1.2.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7
>
> Attachments: HBASE-16429.patch
>
>
> Recently we experienced an online problem that all handlers are stuck.
> Checking the jstack we could see all handler threads waiting for
> RingBuffer.next, while the single ring buffer consumer dead waiting for
> {{safePointReleasedLatch}} to count down:
> {noformat}
> Normal handler thread:
> "B.defaultRpcServer.handler=126,queue=9,port=16020" daemon prio=10
> tid=0x7efd4b44f800 nid=0x15f29 runnable [0x7efd3db7b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1222)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:736)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:698)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2095)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:774)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:756)
> RingBufferEventHandler thread waiting for safePointReleasedLatch:
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.append-pool2-t1"
> prio=10 tid=0x7efd320d nid=0x1777b waiting on condition
> [0x7efd2d2fa000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f01b48d9178> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.safePointAttained(FSHLog.java:1866)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:2066)
[jira] [Commented] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425979#comment-15425979
]
Hudson commented on HBASE-16429:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #4 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/4/])
HBASE-16429 FSHLog: deadlock if rollWriter called when ring buffer (liyu: rev
e09b52fdb182135100d759d76ae7cee15007dd11)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> FSHLog: deadlock if rollWriter called when ring buffer filled with appends
> --
>
> Key: HBASE-16429
> URL: https://issues.apache.org/jira/browse/HBASE-16429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2, 1.2.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7
>
> Attachments: HBASE-16429.patch
>
>
> Recently we experienced an online problem that all handlers are stuck.
> Checking the jstack we could see all handler threads waiting for
> RingBuffer.next, while the single ring buffer consumer dead waiting for
> {{safePointReleasedLatch}} to count down:
> {noformat}
> Normal handler thread:
> "B.defaultRpcServer.handler=126,queue=9,port=16020" daemon prio=10
> tid=0x7efd4b44f800 nid=0x15f29 runnable [0x7efd3db7b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1222)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:736)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:698)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2095)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:774)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:756)
> RingBufferEventHandler thread waiting for safePointReleasedLatch:
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.append-pool2-t1"
> prio=10 tid=0x7efd320d nid=0x1777b waiting on condition
> [0x7efd2d2fa000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f01b48d9178> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.safePointAttained(FSHLog.java:1866)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:2066)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1909)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Wo
[jira] [Commented] (HBASE-15635) Mean age of Blocks in cache (seconds) on webUI should be greater than zero
[ https://issues.apache.org/jira/browse/HBASE-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425980#comment-15425980 ] Hudson commented on HBASE-15635: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #4 (See [https://builds.apache.org/job/HBase-1.2-JDK7/4/]) HBASE-15635 Mean age of Blocks in cache (seconds) on webUI should be (chenheng: rev a84ff653ad2af8102d2acbf6102634b477cb7c8b) * (edit) hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java > Mean age of Blocks in cache (seconds) on webUI should be greater than zero > -- > > Key: HBASE-15635 > URL: https://issues.apache.org/jira/browse/HBASE-15635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.17 >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.1.6, 1.3.1, 1.2.3, 0.98.22 > > Attachments: 0.98.png, 7BFFAF68-0807-400C-853F-706B498449E1.png, > HBASE-15635.0.98.patch, HBASE-15635.patch, HBASE-15635_branch-1.2.patch, > HBASE-15635_v1.patch, HBASE-15635_v2.patch, HBASE-15635_v2.patch, master.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16444) CellUtil#getSumOfCellKeyElementLengths() should consider KEY_INFRASTRUCTURE_SIZE
[
https://issues.apache.org/jira/browse/HBASE-16444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16444:
---
Description:
Currently CellUtil#getSumOfCellKeyElementLengths() considers
{code}
return cell.getRowLength() + cell.getFamilyLength() +
cell.getQualifierLength() +
KeyValue.TIMESTAMP_TYPE_SIZE;
{code}
It can consider the 2 byte ROWLEN and 1 byte FAMILY_LEN also because with the
current way of things we are sure how our key is structured.
But pls note that
{code}
// This will be a low estimate. Will do for now.
return getSumOfCellKeyElementLengths(cell);
{code}
It says clearly it is going to be a low estimate. But in the write path there
should be no harm in adding the complete KEY_INFRA_SIZE.
was:
Currently HFileWriterImpl#getSumOfCellKeyElementLengths() considers
{code}
return cell.getRowLength() + cell.getFamilyLength() +
cell.getQualifierLength() +
KeyValue.TIMESTAMP_TYPE_SIZE;
{code}
It can consider the 2 byte ROWLEN and 1 byte FAMILY_LEN also because with the
current way of things we are sure how our key is structured.
But pls note that
{code}
// This will be a low estimate. Will do for now.
return getSumOfCellKeyElementLengths(cell);
{code}
It says clearly it is going to be a low estimate. But in the write path there
should be no harm in adding the complete KEY_INFRA_SIZE.
> CellUtil#getSumOfCellKeyElementLengths() should consider
> KEY_INFRASTRUCTURE_SIZE
>
>
> Key: HBASE-16444
> URL: https://issues.apache.org/jira/browse/HBASE-16444
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> Currently CellUtil#getSumOfCellKeyElementLengths() considers
> {code}
> return cell.getRowLength() + cell.getFamilyLength() +
> cell.getQualifierLength() +
> KeyValue.TIMESTAMP_TYPE_SIZE;
> {code}
> It can consider the 2 byte ROWLEN and 1 byte FAMILY_LEN also because with the
> current way of things we are sure how our key is structured.
> But pls note that
> {code}
> // This will be a low estimate. Will do for now.
> return getSumOfCellKeyElementLengths(cell);
> {code}
> It says clearly it is going to be a low estimate. But in the write path there
> should be no harm in adding the complete KEY_INFRA_SIZE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16444) HFileWriterImpl#getSumOfCellKeyElementLengths() should consider KEY_INFRASTRUCTURE_SIZE
ramkrishna.s.vasudevan created HBASE-16444:
--
Summary: HFileWriterImpl#getSumOfCellKeyElementLengths() should
consider KEY_INFRASTRUCTURE_SIZE
Key: HBASE-16444
URL: https://issues.apache.org/jira/browse/HBASE-16444
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Currently HFileWriterImpl#getSumOfCellKeyElementLengths() considers
{code}
return cell.getRowLength() + cell.getFamilyLength() +
cell.getQualifierLength() +
KeyValue.TIMESTAMP_TYPE_SIZE;
{code}
It can consider the 2 byte ROWLEN and 1 byte FAMILY_LEN also because with the
current way of things we are sure how our key is structured.
But pls note that
{code}
// This will be a low estimate. Will do for now.
return getSumOfCellKeyElementLengths(cell);
{code}
It says clearly it is going to be a low estimate. But in the write path there
should be no harm in adding the complete KEY_INFRA_SIZE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16444) CellUtil#getSumOfCellKeyElementLengths() should consider KEY_INFRASTRUCTURE_SIZE
[
https://issues.apache.org/jira/browse/HBASE-16444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16444:
---
Summary: CellUtil#getSumOfCellKeyElementLengths() should consider
KEY_INFRASTRUCTURE_SIZE (was: HFileWriterImpl#getSumOfCellKeyElementLengths()
should consider KEY_INFRASTRUCTURE_SIZE)
> CellUtil#getSumOfCellKeyElementLengths() should consider
> KEY_INFRASTRUCTURE_SIZE
>
>
> Key: HBASE-16444
> URL: https://issues.apache.org/jira/browse/HBASE-16444
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> Currently HFileWriterImpl#getSumOfCellKeyElementLengths() considers
> {code}
> return cell.getRowLength() + cell.getFamilyLength() +
> cell.getQualifierLength() +
> KeyValue.TIMESTAMP_TYPE_SIZE;
> {code}
> It can consider the 2 byte ROWLEN and 1 byte FAMILY_LEN also because with the
> current way of things we are sure how our key is structured.
> But pls note that
> {code}
> // This will be a low estimate. Will do for now.
> return getSumOfCellKeyElementLengths(cell);
> {code}
> It says clearly it is going to be a low estimate. But in the write path there
> should be no harm in adding the complete KEY_INFRA_SIZE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425972#comment-15425972
]
Yu Li commented on HBASE-16429:
---
The newly added UT case in TestHRegion simulated the scenario by decreasing
ring buffer size, JFYI.
> FSHLog: deadlock if rollWriter called when ring buffer filled with appends
> --
>
> Key: HBASE-16429
> URL: https://issues.apache.org/jira/browse/HBASE-16429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2, 1.2.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7
>
> Attachments: HBASE-16429.patch
>
>
> Recently we experienced an online problem that all handlers are stuck.
> Checking the jstack we could see all handler threads waiting for
> RingBuffer.next, while the single ring buffer consumer dead waiting for
> {{safePointReleasedLatch}} to count down:
> {noformat}
> Normal handler thread:
> "B.defaultRpcServer.handler=126,queue=9,port=16020" daemon prio=10
> tid=0x7efd4b44f800 nid=0x15f29 runnable [0x7efd3db7b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1222)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:736)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:698)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2095)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:774)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:756)
> RingBufferEventHandler thread waiting for safePointReleasedLatch:
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.append-pool2-t1"
> prio=10 tid=0x7efd320d nid=0x1777b waiting on condition
> [0x7efd2d2fa000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f01b48d9178> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.safePointAttained(FSHLog.java:1866)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:2066)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1909)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:756)
> {noformat}
> {{FSHLog#replaceWriter}} will call {{SafePointZigZagLatch#releaseSafePoint}}
> to count down {{safePointReleasedLatch}}, but replaceWriter got stuck when
> trying to publish a sync onto ring buffer:
> {noforma
[jira] [Commented] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425969#comment-15425969
]
Yu Li commented on HBASE-16429:
---
The issue will happen as long as rollWriter happens when ring buffer filled up,
and since there're multiple producer but only one consumer of the ring buffer
inside FSHLog, it will be filled up when either heavy write load or slow
consuming.
Considering the fact that most operation inside
{{RingBufferEventHandler#onEvent}} (append, sync, etc.) are asynchronous and
non-blocking, the main case causing ring buffer filled up should be heavy write
load, and slow hdfs response should have little impact here.
> FSHLog: deadlock if rollWriter called when ring buffer filled with appends
> --
>
> Key: HBASE-16429
> URL: https://issues.apache.org/jira/browse/HBASE-16429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2, 1.2.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7
>
> Attachments: HBASE-16429.patch
>
>
> Recently we experienced an online problem that all handlers are stuck.
> Checking the jstack we could see all handler threads waiting for
> RingBuffer.next, while the single ring buffer consumer dead waiting for
> {{safePointReleasedLatch}} to count down:
> {noformat}
> Normal handler thread:
> "B.defaultRpcServer.handler=126,queue=9,port=16020" daemon prio=10
> tid=0x7efd4b44f800 nid=0x15f29 runnable [0x7efd3db7b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1222)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:736)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:698)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2095)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:774)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:756)
> RingBufferEventHandler thread waiting for safePointReleasedLatch:
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.append-pool2-t1"
> prio=10 tid=0x7efd320d nid=0x1777b waiting on condition
> [0x7efd2d2fa000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f01b48d9178> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.safePointAttained(FSHLog.java:1866)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:2066)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1909)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.
[jira] [Updated] (HBASE-16405) Change read path Bloom check to work with Cells with out any copy
[ https://issues.apache.org/jira/browse/HBASE-16405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16405: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Change read path Bloom check to work with Cells with out any copy > - > > Key: HBASE-16405 > URL: https://issues.apache.org/jira/browse/HBASE-16405 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16405.patch, HBASE-16405_V2.patch, > HBASE-16405_V3.patch, HBASE-16405_V4.patch, HBASE-16405_V4.patch > > > Currently the read path though it works with Cells, still does go with the > assumption that we need to copy the Cells to KeyValue's key format and then > use the byte[] version of hash() function to calculate the Bloom hash. In > case of incoming cell's being KeyValue type we try to use the > KeyValue.getKey(), KeyValue.getKeyOffset() APIs. With this sub-task we will > get rid of all those and things will work with Cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16405) Change read path Bloom check to work with Cells with out any copy
[ https://issues.apache.org/jira/browse/HBASE-16405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16405: --- Status: Open (was: Patch Available) > Change read path Bloom check to work with Cells with out any copy > - > > Key: HBASE-16405 > URL: https://issues.apache.org/jira/browse/HBASE-16405 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16405.patch, HBASE-16405_V2.patch, > HBASE-16405_V3.patch, HBASE-16405_V4.patch, HBASE-16405_V4.patch > > > Currently the read path though it works with Cells, still does go with the > assumption that we need to copy the Cells to KeyValue's key format and then > use the byte[] version of hash() function to calculate the Bloom hash. In > case of incoming cell's being KeyValue type we try to use the > KeyValue.getKey(), KeyValue.getKeyOffset() APIs. With this sub-task we will > get rid of all those and things will work with Cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16405) Change read path Bloom check to work with Cells with out any copy
[ https://issues.apache.org/jira/browse/HBASE-16405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16405: --- Attachment: HBASE-16405_V4.patch > Change read path Bloom check to work with Cells with out any copy > - > > Key: HBASE-16405 > URL: https://issues.apache.org/jira/browse/HBASE-16405 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16405.patch, HBASE-16405_V2.patch, > HBASE-16405_V3.patch, HBASE-16405_V4.patch, HBASE-16405_V4.patch > > > Currently the read path though it works with Cells, still does go with the > assumption that we need to copy the Cells to KeyValue's key format and then > use the byte[] version of hash() function to calculate the Bloom hash. In > case of incoming cell's being KeyValue type we try to use the > KeyValue.getKey(), KeyValue.getKeyOffset() APIs. With this sub-task we will > get rid of all those and things will work with Cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425959#comment-15425959 ] Phil Yang commented on HBASE-16388: --- Will we use non-blocking Stub in blocking Table interface? This patch only handles BlockingStub. And I think AsyncTable won't block users' threads. So we need not have any limit of concurrent requests for a server in Stub if it is only used in AsyncTable. > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16405) Change read path Bloom check to work with Cells with out any copy
[
https://issues.apache.org/jira/browse/HBASE-16405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425941#comment-15425941
]
Hadoop QA commented on HBASE-16405:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 4s {color}
| {color:red} Docker failed to build yetus/hbase:date2016-08-18. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824287/HBASE-16405_V4.patch |
| JIRA Issue | HBASE-16405 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3138/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Change read path Bloom check to work with Cells with out any copy
> -
>
> Key: HBASE-16405
> URL: https://issues.apache.org/jira/browse/HBASE-16405
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
>Reporter: ramkrishna.s.vasudevan
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16405.patch, HBASE-16405_V2.patch,
> HBASE-16405_V3.patch, HBASE-16405_V4.patch
>
>
> Currently the read path though it works with Cells, still does go with the
> assumption that we need to copy the Cells to KeyValue's key format and then
> use the byte[] version of hash() function to calculate the Bloom hash. In
> case of incoming cell's being KeyValue type we try to use the
> KeyValue.getKey(), KeyValue.getKeyOffset() APIs. With this sub-task we will
> get rid of all those and things will work with Cells.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16405) Change read path Bloom check to work with Cells with out any copy
[ https://issues.apache.org/jira/browse/HBASE-16405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16405: --- Attachment: HBASE-16405_V4.patch > Change read path Bloom check to work with Cells with out any copy > - > > Key: HBASE-16405 > URL: https://issues.apache.org/jira/browse/HBASE-16405 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16405.patch, HBASE-16405_V2.patch, > HBASE-16405_V3.patch, HBASE-16405_V4.patch > > > Currently the read path though it works with Cells, still does go with the > assumption that we need to copy the Cells to KeyValue's key format and then > use the byte[] version of hash() function to calculate the Bloom hash. In > case of incoming cell's being KeyValue type we try to use the > KeyValue.getKey(), KeyValue.getKeyOffset() APIs. With this sub-task we will > get rid of all those and things will work with Cells. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16432) Revisit the asynchronous ipc implementation
[ https://issues.apache.org/jira/browse/HBASE-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425916#comment-15425916 ] Anoop Sam John commented on HBASE-16432: That is fine then > Revisit the asynchronous ipc implementation > --- > > Key: HBASE-16432 > URL: https://issues.apache.org/jira/browse/HBASE-16432 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang > > Seems the current approach of implementing AsyncTable is in trouble as we > expose some netty classes in our public interfaces. > I agree that we should not do this. The AsyncTable should be implemented > using the asynchronous protobuf stub, and we could use CompletableFuture or > Deferred instead of netty's Future. I think the problem of netty's future is > that it is tighten with netty's executor. This makes it impossible to use > without netty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425911#comment-15425911 ] Duo Zhang commented on HBASE-16388: --- How do you deal with the async call? Although it is not used right now and I plan to reimplement it... > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16388-v1.patch, HBASE-16388-v2.patch, > HBASE-16388-v2.patch > > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16432) Revisit the asynchronous ipc implementation
[
https://issues.apache.org/jira/browse/HBASE-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425904#comment-15425904
]
Duo Zhang edited comment on HBASE-16432 at 8/18/16 5:12 AM:
{quote}
Means we will be exposing PB async stuff instead of netty?
{quote}
Not necessary. The AsyncTable interface will use CompletableFuture or something
like Deferred. The PB stuffs are only used in the implementation.
was (Author: apache9):
{quota}
Means we will be exposing PB async stuff instead of netty?
{quota}
Not necessary. The AsyncTable interface will use CompletableFuture or something
like Deferred. The PB stuffs are only used in the implementation.
> Revisit the asynchronous ipc implementation
> ---
>
> Key: HBASE-16432
> URL: https://issues.apache.org/jira/browse/HBASE-16432
> Project: HBase
> Issue Type: Umbrella
>Reporter: Duo Zhang
>
> Seems the current approach of implementing AsyncTable is in trouble as we
> expose some netty classes in our public interfaces.
> I agree that we should not do this. The AsyncTable should be implemented
> using the asynchronous protobuf stub, and we could use CompletableFuture or
> Deferred instead of netty's Future. I think the problem of netty's future is
> that it is tighten with netty's executor. This makes it impossible to use
> without netty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16432) Revisit the asynchronous ipc implementation
[
https://issues.apache.org/jira/browse/HBASE-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425904#comment-15425904
]
Duo Zhang commented on HBASE-16432:
---
{quota}
Means we will be exposing PB async stuff instead of netty?
{quota}
Not necessary. The AsyncTable interface will use CompletableFuture or something
like Deferred. The PB stuffs are only used in the implementation.
> Revisit the asynchronous ipc implementation
> ---
>
> Key: HBASE-16432
> URL: https://issues.apache.org/jira/browse/HBASE-16432
> Project: HBase
> Issue Type: Umbrella
>Reporter: Duo Zhang
>
> Seems the current approach of implementing AsyncTable is in trouble as we
> expose some netty classes in our public interfaces.
> I agree that we should not do this. The AsyncTable should be implemented
> using the asynchronous protobuf stub, and we could use CompletableFuture or
> Deferred instead of netty's Future. I think the problem of netty's future is
> that it is tighten with netty's executor. This makes it impossible to use
> without netty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16432) Revisit the asynchronous ipc implementation
[ https://issues.apache.org/jira/browse/HBASE-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425896#comment-15425896 ] Anoop Sam John commented on HBASE-16432: bq.Seems the current approach of implementing AsyncTable is in trouble as we expose some netty classes in our public interfaces. bq. Cut off the relationship between AsyncTable and Netty. We could implement the AsyncTable with protobuf async stub. Means we will be exposing PB async stuff instead of netty? That also should be avoided. We should not be exposing any other third party class/interface directly > Revisit the asynchronous ipc implementation > --- > > Key: HBASE-16432 > URL: https://issues.apache.org/jira/browse/HBASE-16432 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang > > Seems the current approach of implementing AsyncTable is in trouble as we > expose some netty classes in our public interfaces. > I agree that we should not do this. The AsyncTable should be implemented > using the asynchronous protobuf stub, and we could use CompletableFuture or > Deferred instead of netty's Future. I think the problem of netty's future is > that it is tighten with netty's executor. This makes it impossible to use > without netty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-16440:
---
Attachment: HBASE-16440_V2.patch
Addressed comments from Stack.
Added a test. With out the patch it some times fails as we will keep more than
the maxCount #chunks in the pool!
Thanks for the review stack.
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16440.patch, HBASE-16440_V2.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-16440:
---
Status: Patch Available (was: Open)
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16440.patch, HBASE-16440_V2.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-16440:
---
Fix Version/s: 2.0.0
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16440.patch, HBASE-16440_V2.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425891#comment-15425891
]
Hudson commented on HBASE-16429:
FAILURE: Integrated in Jenkins build HBase-1.3 #819 (See
[https://builds.apache.org/job/HBase-1.3/819/])
HBASE-16429 FSHLog: deadlock if rollWriter called when ring buffer (liyu: rev
e3ed8ecdb3ce7ee727b525f88bc58650559ebc7e)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
> FSHLog: deadlock if rollWriter called when ring buffer filled with appends
> --
>
> Key: HBASE-16429
> URL: https://issues.apache.org/jira/browse/HBASE-16429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2, 1.2.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7
>
> Attachments: HBASE-16429.patch
>
>
> Recently we experienced an online problem that all handlers are stuck.
> Checking the jstack we could see all handler threads waiting for
> RingBuffer.next, while the single ring buffer consumer dead waiting for
> {{safePointReleasedLatch}} to count down:
> {noformat}
> Normal handler thread:
> "B.defaultRpcServer.handler=126,queue=9,port=16020" daemon prio=10
> tid=0x7efd4b44f800 nid=0x15f29 runnable [0x7efd3db7b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1222)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:736)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:698)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2095)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:774)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:756)
> RingBufferEventHandler thread waiting for safePointReleasedLatch:
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.append-pool2-t1"
> prio=10 tid=0x7efd320d nid=0x1777b waiting on condition
> [0x7efd2d2fa000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f01b48d9178> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.safePointAttained(FSHLog.java:1866)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:2066)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1909)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.r
[jira] [Commented] (HBASE-16299) Update REST API scanner with ability to do reverse scan
[ https://issues.apache.org/jira/browse/HBASE-16299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425888#comment-15425888 ] Minwoo Kang commented on HBASE-16299: - I appreciate your comment. I closed the RB requests. > Update REST API scanner with ability to do reverse scan > --- > > Key: HBASE-16299 > URL: https://issues.apache.org/jira/browse/HBASE-16299 > Project: HBase > Issue Type: Improvement > Components: REST >Affects Versions: 1.1.2 > Environment: Not environment specific (tested on HDP 2.4.2) >Reporter: Bjorn Olsen >Assignee: Minwoo Kang >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16299.branch-1.3.001.patch, > HBASE-16299.master.001.patch > > > HBASE-4811 "Support reverse scan" was implemented from version 0.98.0, and is > available in the Java API. > However this functionality is not yet exposed via REST. > Example of expected API call: > http://server:port/table/*?startrow=1&endrow=10&reversed=true"; > (Returns rows ordered by key in reverse, eg from 9*** to 1*** ) > Based on my (limited) understanding this should be simple to add. > See org.apach.hadoop.hbase.rest.TableResource.getScanResource > This function creates a Scan object with parameters passed in from the REST > API (I assume). > Adding this functionality may be as simple as adding a "reversed" parameter > to the REST API which passes down to where the Scan object is created in > TableResource.getScanResource. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425882#comment-15425882
]
Heng Chen commented on HBASE-16429:
---
[~carp84] which scenario, the issue will happen? Heavy write load and slow
hdfs response?
> FSHLog: deadlock if rollWriter called when ring buffer filled with appends
> --
>
> Key: HBASE-16429
> URL: https://issues.apache.org/jira/browse/HBASE-16429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2, 1.2.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7
>
> Attachments: HBASE-16429.patch
>
>
> Recently we experienced an online problem that all handlers are stuck.
> Checking the jstack we could see all handler threads waiting for
> RingBuffer.next, while the single ring buffer consumer dead waiting for
> {{safePointReleasedLatch}} to count down:
> {noformat}
> Normal handler thread:
> "B.defaultRpcServer.handler=126,queue=9,port=16020" daemon prio=10
> tid=0x7efd4b44f800 nid=0x15f29 runnable [0x7efd3db7b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1222)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:736)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:698)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2095)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:774)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:756)
> RingBufferEventHandler thread waiting for safePointReleasedLatch:
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.append-pool2-t1"
> prio=10 tid=0x7efd320d nid=0x1777b waiting on condition
> [0x7efd2d2fa000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f01b48d9178> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.safePointAttained(FSHLog.java:1866)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:2066)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1909)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:756)
> {noformat}
> {{FSHLog#replaceWriter}} will call {{SafePointZigZagLatch#releaseSafePoint}}
> to count down {{safePointReleasedLatch}}, but replaceWriter got stuck when
> trying to publish a sync onto ring buffer:
> {noformat
[jira] [Commented] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425877#comment-15425877
]
Hudson commented on HBASE-16429:
FAILURE: Integrated in Jenkins build HBase-1.4 #355 (See
[https://builds.apache.org/job/HBase-1.4/355/])
HBASE-16429 FSHLog: deadlock if rollWriter called when ring buffer (liyu: rev
e05b002ed53c20cb885e773c7189efe90977858a)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
> FSHLog: deadlock if rollWriter called when ring buffer filled with appends
> --
>
> Key: HBASE-16429
> URL: https://issues.apache.org/jira/browse/HBASE-16429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2, 1.2.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7
>
> Attachments: HBASE-16429.patch
>
>
> Recently we experienced an online problem that all handlers are stuck.
> Checking the jstack we could see all handler threads waiting for
> RingBuffer.next, while the single ring buffer consumer dead waiting for
> {{safePointReleasedLatch}} to count down:
> {noformat}
> Normal handler thread:
> "B.defaultRpcServer.handler=126,queue=9,port=16020" daemon prio=10
> tid=0x7efd4b44f800 nid=0x15f29 runnable [0x7efd3db7b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1222)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:736)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:698)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2095)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:774)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:756)
> RingBufferEventHandler thread waiting for safePointReleasedLatch:
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.append-pool2-t1"
> prio=10 tid=0x7efd320d nid=0x1777b waiting on condition
> [0x7efd2d2fa000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f01b48d9178> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.safePointAttained(FSHLog.java:1866)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:2066)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1909)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.r
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[ https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425865#comment-15425865 ] Ted Yu commented on HBASE-14450: I was looking at WALInputFormat#WALRecordReader#nextKeyValue() Strangely the log statements in the above method were not observed in test output. > HBase Backup/Restore Phase 3: Multiwal support > -- > > Key: HBASE-14450 > URL: https://issues.apache.org/jira/browse/HBASE-14450 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 14450-roll.v2.txt, 14450.roll-wait.txt, 14450.v2.txt, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt, > testIncrementalBackup-8-15.txt > > > We need to support multiwal configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16255) Backup/Restore IT
[ https://issues.apache.org/jira/browse/HBASE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16255: -- Attachment: HBASE-16255-v5.patch v5. > Backup/Restore IT > - > > Key: HBASE-16255 > URL: https://issues.apache.org/jira/browse/HBASE-16255 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: HBASE-16255-v1.patch, HBASE-16255-v2.patch, > HBASE-16255-v3.patch, HBASE-16255-v4.patch, HBASE-16255-v5.patch > > > Integration test for backup restore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425860#comment-15425860
]
Anoop Sam John commented on HBASE-16440:
I just moved the Chunk class outside and renamed it.. OK I will keep the name
Chunk. Just addition is the isPooled() method.
bq.In getChunk, you have three returns throughout the method. You don't want to
just have one on the end?
Ok I can change.
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-16440.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-16429:
--
Resolution: Fixed
Fix Version/s: 1.1.7
1.2.3
1.4.0
1.3.0
2.0.0
Status: Resolved (was: Patch Available)
Pushed into all 1.1+ branches. Thanks [~stack] for review.
> FSHLog: deadlock if rollWriter called when ring buffer filled with appends
> --
>
> Key: HBASE-16429
> URL: https://issues.apache.org/jira/browse/HBASE-16429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2, 1.2.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7
>
> Attachments: HBASE-16429.patch
>
>
> Recently we experienced an online problem that all handlers are stuck.
> Checking the jstack we could see all handler threads waiting for
> RingBuffer.next, while the single ring buffer consumer dead waiting for
> {{safePointReleasedLatch}} to count down:
> {noformat}
> Normal handler thread:
> "B.defaultRpcServer.handler=126,queue=9,port=16020" daemon prio=10
> tid=0x7efd4b44f800 nid=0x15f29 runnable [0x7efd3db7b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1222)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:736)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:698)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2095)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:774)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:756)
> RingBufferEventHandler thread waiting for safePointReleasedLatch:
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.append-pool2-t1"
> prio=10 tid=0x7efd320d nid=0x1777b waiting on condition
> [0x7efd2d2fa000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f01b48d9178> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.safePointAttained(FSHLog.java:1866)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:2066)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1909)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:756)
> {noformat}
> {{FSHLog#replaceWriter}} will call {{SafePointZigZagLatch#releaseSafePoint}}
> to count down {{safePointRel
[jira] [Commented] (HBASE-16433) Remove AsyncRpcChannel related stuffs
[ https://issues.apache.org/jira/browse/HBASE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425842#comment-15425842 ] Duo Zhang commented on HBASE-16433: --- So what's your options [~jurmous] and [~stack]? I have also posted a comment on HBASE-16432 to describe my plan. The first thing we need to reach a consensus is that the AsyncTable should be implement use the protobuf async stub. Thanks. > Remove AsyncRpcChannel related stuffs > - > > Key: HBASE-16433 > URL: https://issues.apache.org/jira/browse/HBASE-16433 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16433.patch > > > AsyncRpcChannel can not be used by protobuf stub. We should implement the > async logic along with the RpcChannel interface of protobuf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15635) Mean age of Blocks in cache (seconds) on webUI should be greater than zero
[ https://issues.apache.org/jira/browse/HBASE-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425841#comment-15425841 ] Heng Chen commented on HBASE-15635: --- commit to branch-1.0, branch-1.1, branch-1.2 > Mean age of Blocks in cache (seconds) on webUI should be greater than zero > -- > > Key: HBASE-15635 > URL: https://issues.apache.org/jira/browse/HBASE-15635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.17 >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.1.6, 1.3.1, 1.2.3, 0.98.22 > > Attachments: 0.98.png, 7BFFAF68-0807-400C-853F-706B498449E1.png, > HBASE-15635.0.98.patch, HBASE-15635.patch, HBASE-15635_branch-1.2.patch, > HBASE-15635_v1.patch, HBASE-15635_v2.patch, HBASE-15635_v2.patch, master.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15635) Mean age of Blocks in cache (seconds) on webUI should be greater than zero
[ https://issues.apache.org/jira/browse/HBASE-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15635: -- Fix Version/s: 1.2.3 1.1.6 1.0.4 > Mean age of Blocks in cache (seconds) on webUI should be greater than zero > -- > > Key: HBASE-15635 > URL: https://issues.apache.org/jira/browse/HBASE-15635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.17 >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.1.6, 1.3.1, 1.2.3, 0.98.22 > > Attachments: 0.98.png, 7BFFAF68-0807-400C-853F-706B498449E1.png, > HBASE-15635.0.98.patch, HBASE-15635.patch, HBASE-15635_branch-1.2.patch, > HBASE-15635_v1.patch, HBASE-15635_v2.patch, HBASE-15635_v2.patch, master.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15635) Mean age of Blocks in cache (seconds) on webUI should be greater than zero
[ https://issues.apache.org/jira/browse/HBASE-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-15635: -- Attachment: HBASE-15635_branch-1.2.patch upload patch for branch-1.2 > Mean age of Blocks in cache (seconds) on webUI should be greater than zero > -- > > Key: HBASE-15635 > URL: https://issues.apache.org/jira/browse/HBASE-15635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.17 >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.4.0, 1.3.1, 0.98.22 > > Attachments: 0.98.png, 7BFFAF68-0807-400C-853F-706B498449E1.png, > HBASE-15635.0.98.patch, HBASE-15635.patch, HBASE-15635_branch-1.2.patch, > HBASE-15635_v1.patch, HBASE-15635_v2.patch, HBASE-15635_v2.patch, master.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16318) fail build if license isn't in whitelist
[ https://issues.apache.org/jira/browse/HBASE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425828#comment-15425828 ] Hudson commented on HBASE-16318: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #3 (See [https://builds.apache.org/job/HBase-1.2-JDK7/3/]) HBASE-16318 consistently use the correct name for 'Apache License, (busbey: rev 3f30d420c773ab173252d2bfadf2df095d715878) * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm * (edit) pom.xml * (edit) hbase-resource-bundle/src/main/resources/supplemental-models.xml HBASE-16318 fail build while rendering velocity template if dependency (busbey: rev ca1b49dd04db6d94b19dc097c52df763baa60f39) * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm > fail build if license isn't in whitelist > > > Key: HBASE-16318 > URL: https://issues.apache.org/jira/browse/HBASE-16318 > Project: HBase > Issue Type: Sub-task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 0.98.22, 1.1.7 > > Attachments: HBASE-16318.0.patch, HBASE-16318.1.patch, > HBASE-16318.2.patch, HBASE-16318.3.patch > > > we use supplemental-models.xml to make sure we have consistent names and > descriptions for licenses. we also know what licenses we expect to see in our > build. If we see a different one > # fail the velocity template process > # if possible, include some information about why this happened -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16318) fail build if license isn't in whitelist
[ https://issues.apache.org/jira/browse/HBASE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425824#comment-15425824 ] Hudson commented on HBASE-16318: FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #3 (See [https://builds.apache.org/job/HBase-1.2-JDK8/3/]) HBASE-16318 consistently use the correct name for 'Apache License, (busbey: rev 3f30d420c773ab173252d2bfadf2df095d715878) * (edit) pom.xml * (edit) hbase-resource-bundle/src/main/resources/supplemental-models.xml * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm HBASE-16318 fail build while rendering velocity template if dependency (busbey: rev ca1b49dd04db6d94b19dc097c52df763baa60f39) * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm > fail build if license isn't in whitelist > > > Key: HBASE-16318 > URL: https://issues.apache.org/jira/browse/HBASE-16318 > Project: HBase > Issue Type: Sub-task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 0.98.22, 1.1.7 > > Attachments: HBASE-16318.0.patch, HBASE-16318.1.patch, > HBASE-16318.2.patch, HBASE-16318.3.patch > > > we use supplemental-models.xml to make sure we have consistent names and > descriptions for licenses. we also know what licenses we expect to see in our > build. If we see a different one > # fail the velocity template process > # if possible, include some information about why this happened -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16378) Procedure v2 - Make ProcedureException extend HBaseException
[ https://issues.apache.org/jira/browse/HBASE-16378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425815#comment-15425815 ] Hudson commented on HBASE-16378: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1434 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1434/]) HBASE-16378 Procedure v2 - Make ProcedureException extend HBaseException (matteo.bertozzi: rev e637a61e26aeb442ea1c1593fdbd0b15b48f5642) * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureException.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureException.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/ForeignExceptionUtil.java > Procedure v2 - Make ProcedureException extend HBaseException > > > Key: HBASE-16378 > URL: https://issues.apache.org/jira/browse/HBASE-16378 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0, 1.1.5, 1.2.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16378-v0.patch, HBASE-16378-v1.patch > > > Make ProcedureException extend HBaseException, so we can avoid stuff like > HBASE-15591. and avoid try/catch ProcedureException and direct rethrows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16432) Revisit the asynchronous ipc implementation
[ https://issues.apache.org/jira/browse/HBASE-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425790#comment-15425790 ] Duo Zhang commented on HBASE-16432: --- My plan is to do this with 3 steps 1. Cut off the relationship between AsyncTable and Netty. We could implement the AsyncTable with protobuf async stub. After this the development of AsyncTable could continue since the async stub is already supported by the current AsyncRpcClient. 2. Add RpcChannel support for RpcClientImpl. The current implementation is wrong... We schedule a blocking task into the global event loop, this may block others as it is 'global'...At this time, the AsyncTable can run on both RpcClientImpl and AsyncRpcClient. 3. Do some refactoring on AbstractRpcClient and reimplement the AsyncRpcClient. Thanks. > Revisit the asynchronous ipc implementation > --- > > Key: HBASE-16432 > URL: https://issues.apache.org/jira/browse/HBASE-16432 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang > > Seems the current approach of implementing AsyncTable is in trouble as we > expose some netty classes in our public interfaces. > I agree that we should not do this. The AsyncTable should be implemented > using the asynchronous protobuf stub, and we could use CompletableFuture or > Deferred instead of netty's Future. I think the problem of netty's future is > that it is tighten with netty's executor. This makes it impossible to use > without netty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16426) Remove company affiliations from committer list
[ https://issues.apache.org/jira/browse/HBASE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-16426: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Pushed to master. Will be live when the site is next pushed. > Remove company affiliations from committer list > --- > > Key: HBASE-16426 > URL: https://issues.apache.org/jira/browse/HBASE-16426 > Project: HBase > Issue Type: Task > Components: documentation, website >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-16426.patch > > > An email thread on the dev mailing list came to the consensus that company > affiliations in the committer list are difficult to keep up to date and not > worth it. This JIRA is to track removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16318) fail build if license isn't in whitelist
[ https://issues.apache.org/jira/browse/HBASE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425682#comment-15425682 ] Hudson commented on HBASE-16318: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1766 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1766/]) HBASE-16318 consistently use the correct name for 'Apache License, (busbey: rev 5eb0a540f885c99a87a55a2c38765a1a89a16bdc) * (edit) hbase-resource-bundle/src/main/resources/supplemental-models.xml * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm * (edit) pom.xml HBASE-16318 fail build while rendering velocity template if dependency (busbey: rev d82cf96f2253cbd5bf7684f5d6ffb5642a44e739) * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm > fail build if license isn't in whitelist > > > Key: HBASE-16318 > URL: https://issues.apache.org/jira/browse/HBASE-16318 > Project: HBase > Issue Type: Sub-task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 0.98.22, 1.1.7 > > Attachments: HBASE-16318.0.patch, HBASE-16318.1.patch, > HBASE-16318.2.patch, HBASE-16318.3.patch > > > we use supplemental-models.xml to make sure we have consistent names and > descriptions for licenses. we also know what licenses we expect to see in our > build. If we see a different one > # fail the velocity template process > # if possible, include some information about why this happened -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16318) fail build if license isn't in whitelist
[ https://issues.apache.org/jira/browse/HBASE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425663#comment-15425663 ] Hudson commented on HBASE-16318: FAILURE: Integrated in Jenkins build HBase-1.3 #818 (See [https://builds.apache.org/job/HBase-1.3/818/]) HBASE-16318 consistently use the correct name for 'Apache License, (busbey: rev 75950de09ea9b0f1dc893f0724c73fb2a618e4ed) * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm * (edit) hbase-resource-bundle/src/main/resources/supplemental-models.xml * (edit) pom.xml HBASE-16318 fail build while rendering velocity template if dependency (busbey: rev 4e78f7a99e4d3fd264fb64446b52bf082b4ee6e6) * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm > fail build if license isn't in whitelist > > > Key: HBASE-16318 > URL: https://issues.apache.org/jira/browse/HBASE-16318 > Project: HBase > Issue Type: Sub-task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 0.98.22, 1.1.7 > > Attachments: HBASE-16318.0.patch, HBASE-16318.1.patch, > HBASE-16318.2.patch, HBASE-16318.3.patch > > > we use supplemental-models.xml to make sure we have consistent names and > descriptions for licenses. we also know what licenses we expect to see in our > build. If we see a different one > # fail the velocity template process > # if possible, include some information about why this happened -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16094) Procedure v2 - Improve cleaning up of proc wals
[
https://issues.apache.org/jira/browse/HBASE-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425650#comment-15425650
]
Hadoop QA commented on HBASE-16094:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 32s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 48s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 57s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-17 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824256/HBASE-16094.master.006.patch
|
| JIRA Issue | HBASE-16094 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 776bfccc6012 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e637a61 |
| Default Java | 1.7.0_101 |
| Multi-JDK versions
[jira] [Commented] (HBASE-16433) Remove AsyncRpcChannel related stuffs
[ https://issues.apache.org/jira/browse/HBASE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425638#comment-15425638 ] Duo Zhang commented on HBASE-16433: --- I agree that the protobuf stub is not a good abstraction of event-driven code, but I do not think it is a good idea to partially bypass it. The string name mapping in the protobuf code is generated by protoc, so it is much safer than our hand-written mapping code. None or all, my suggestion. We should either implement our code on the suck but work stub, or change to use something new such as grpc as our RPC layer. Thanks. > Remove AsyncRpcChannel related stuffs > - > > Key: HBASE-16433 > URL: https://issues.apache.org/jira/browse/HBASE-16433 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16433.patch > > > AsyncRpcChannel can not be used by protobuf stub. We should implement the > async logic along with the RpcChannel interface of protobuf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16318) fail build if license isn't in whitelist
[ https://issues.apache.org/jira/browse/HBASE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425637#comment-15425637 ] Hudson commented on HBASE-16318: FAILURE: Integrated in Jenkins build HBase-1.4 #354 (See [https://builds.apache.org/job/HBase-1.4/354/]) HBASE-16318 consistently use the correct name for 'Apache License, (busbey: rev df4953c22e8578922d5f4fa77b7a78c30a9b0a1a) * (edit) hbase-resource-bundle/src/main/resources/supplemental-models.xml * (edit) pom.xml * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm HBASE-16318 fail build while rendering velocity template if dependency (busbey: rev 0175e432c81bb97070e0d584689e41f368ace82a) * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm > fail build if license isn't in whitelist > > > Key: HBASE-16318 > URL: https://issues.apache.org/jira/browse/HBASE-16318 > Project: HBase > Issue Type: Sub-task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 0.98.22, 1.1.7 > > Attachments: HBASE-16318.0.patch, HBASE-16318.1.patch, > HBASE-16318.2.patch, HBASE-16318.3.patch > > > we use supplemental-models.xml to make sure we have consistent names and > descriptions for licenses. we also know what licenses we expect to see in our > build. If we see a different one > # fail the velocity template process > # if possible, include some information about why this happened -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425636#comment-15425636
]
Hadoop QA commented on HBASE-16429:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 100m 26s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 12s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-17 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824065/HBASE-16429.patch |
| JIRA Issue | HBASE-16429 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2941c1a670b5 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e637a61 |
| Default Java | 1.7.0_101 |
| Multi-JDK versions | /usr/lib/jvm/java
[jira] [Commented] (HBASE-16433) Remove AsyncRpcChannel related stuffs
[ https://issues.apache.org/jira/browse/HBASE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425628#comment-15425628 ] Duo Zhang commented on HBASE-16433: --- On a mobile phone so reply all things here, include the questions on RB. The parameter type in protobuf stub is the actual type of request and in the general call method it is Message, so this means we can not find the error at compile time. And for the RpcController, I do not think the 'possible null' is a problem if the upper layer need it to receive something. For exapmle, I think we will always have a pcrc when doing scan because we need it to get the CellScanner, otherwise the logic is broken. For AsyncTable implementation, you always need a pcrc to check if we are in error. > Remove AsyncRpcChannel related stuffs > - > > Key: HBASE-16433 > URL: https://issues.apache.org/jira/browse/HBASE-16433 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16433.patch > > > AsyncRpcChannel can not be used by protobuf stub. We should implement the > async logic along with the RpcChannel interface of protobuf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16431) Add missing method in class HTableWrapper
[ https://issues.apache.org/jira/browse/HBASE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425615#comment-15425615 ] stack commented on HBASE-16431: --- What [~dimaspivak] said [~F7753] Where do you see the problem. I just checked compile of branch-1.2 and seems good. Thanks. > Add missing method in class HTableWrapper > - > > Key: HBASE-16431 > URL: https://issues.apache.org/jira/browse/HBASE-16431 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.2 >Reporter: Biao Ma > > Class `HTableWrapper` must either declared abstract or implement abstract > method 'setRpcTimeout(int)' in 'Table' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16094) Procedure v2 - Improve cleaning up of proc wals
[ https://issues.apache.org/jira/browse/HBASE-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16094: - Attachment: HBASE-16094.master.006.patch > Procedure v2 - Improve cleaning up of proc wals > --- > > Key: HBASE-16094 > URL: https://issues.apache.org/jira/browse/HBASE-16094 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16094.master.001.patch, > HBASE-16094.master.002.patch, HBASE-16094.master.003.patch, > HBASE-16094.master.004.patch, HBASE-16094.master.005.patch, > HBASE-16094.master.006.patch > > > Avoid accumulating too many wals. > We remove logs in 3 cases: > - No procedures are running. We can remove all the logs. > - All procedures are updated/written in the last log. We can remove all the > logs, aside the active one. > - Remove log if does not contains “active” procIds > https://github.com/apache/hbase/blob/master/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L865 > The last point, Remove log if does not contains “active” procIds, needs to be > improved. Basically, even if a log contains state of an active proc, we can > delete it if a later log contains a newer state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16426) Remove company affiliations from committer list
[ https://issues.apache.org/jira/browse/HBASE-16426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425599#comment-15425599 ] Andrew Purtell commented on HBASE-16426: +1 > Remove company affiliations from committer list > --- > > Key: HBASE-16426 > URL: https://issues.apache.org/jira/browse/HBASE-16426 > Project: HBase > Issue Type: Task > Components: documentation, website >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-16426.patch > > > An email thread on the dev mailing list came to the consensus that company > affiliations in the committer list are difficult to keep up to date and not > worth it. This JIRA is to track removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16094) Procedure v2 - Improve cleaning up of proc wals
[
https://issues.apache.org/jira/browse/HBASE-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425589#comment-15425589
]
Hadoop QA commented on HBASE-16094:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 47s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 48s {color}
| {color:red} hbase-procedure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 24s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.procedure2.store.wal.TestWALProcedureStore |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-17 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824250/HBASE-16094.master.005.patch
|
| JIRA Issue | HBASE-16094 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 459cc471b520 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-person
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425586#comment-15425586 ] stack commented on HBASE-16438: --- Could it be a tag? > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16443) Add readRegionServerLastLogRollResult() variant which accepts server name parameter
Ted Yu created HBASE-16443:
--
Summary: Add readRegionServerLastLogRollResult() variant which
accepts server name parameter
Key: HBASE-16443
URL: https://issues.apache.org/jira/browse/HBASE-16443
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Priority: Minor
Currently in LogRollBackupSubprocedure#RSRollLogTask#call, we have
{code}
HashMap serverTimestampMap =
table.readRegionServerLastLogRollResult(backupRoot);
String host = rss.getServerName().getHostname();
int port = rss.getServerName().getPort();
String server = host + ":" + port;
Long sts = serverTimestampMap.get(host);
{code}
Last log roll result is retrieved for all the servers. Yet only the timestamp
for relevant server is used.
This issue adds a variant of readRegionServerLastLogRollResult() which takes
server name as second parameter.
The variant would be used in LogRollBackupSubprocedure#RSRollLogTask#call().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16318) fail build if license isn't in whitelist
[ https://issues.apache.org/jira/browse/HBASE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425562#comment-15425562 ] Hudson commented on HBASE-16318: FAILURE: Integrated in Jenkins build HBase-1.3-IT #792 (See [https://builds.apache.org/job/HBase-1.3-IT/792/]) HBASE-16318 consistently use the correct name for 'Apache License, (busbey: rev 75950de09ea9b0f1dc893f0724c73fb2a618e4ed) * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm * (edit) hbase-resource-bundle/src/main/resources/supplemental-models.xml * (edit) pom.xml HBASE-16318 fail build while rendering velocity template if dependency (busbey: rev 4e78f7a99e4d3fd264fb64446b52bf082b4ee6e6) * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm > fail build if license isn't in whitelist > > > Key: HBASE-16318 > URL: https://issues.apache.org/jira/browse/HBASE-16318 > Project: HBase > Issue Type: Sub-task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 0.98.22, 1.1.7 > > Attachments: HBASE-16318.0.patch, HBASE-16318.1.patch, > HBASE-16318.2.patch, HBASE-16318.3.patch > > > we use supplemental-models.xml to make sure we have consistent names and > descriptions for licenses. we also know what licenses we expect to see in our > build. If we see a different one > # fail the velocity template process > # if possible, include some information about why this happened -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16094) Procedure v2 - Improve cleaning up of proc wals
[ https://issues.apache.org/jira/browse/HBASE-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16094: - Attachment: HBASE-16094.master.005.patch > Procedure v2 - Improve cleaning up of proc wals > --- > > Key: HBASE-16094 > URL: https://issues.apache.org/jira/browse/HBASE-16094 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16094.master.001.patch, > HBASE-16094.master.002.patch, HBASE-16094.master.003.patch, > HBASE-16094.master.004.patch, HBASE-16094.master.005.patch > > > Avoid accumulating too many wals. > We remove logs in 3 cases: > - No procedures are running. We can remove all the logs. > - All procedures are updated/written in the last log. We can remove all the > logs, aside the active one. > - Remove log if does not contains “active” procIds > https://github.com/apache/hbase/blob/master/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L865 > The last point, Remove log if does not contains “active” procIds, needs to be > improved. Basically, even if a log contains state of an active proc, we can > delete it if a later log contains a newer state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16422) Tighten our guarantees on compatibility across patch versions
[ https://issues.apache.org/jira/browse/HBASE-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425518#comment-15425518 ] Hudson commented on HBASE-16422: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1433 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1433/]) HBASE-16422 Tighten our guarantees on compatibility across patch (stack: rev 2261c8c31a46fc79d9b930db90752ab60945b74a) * (edit) src/main/asciidoc/_chapters/upgrading.adoc > Tighten our guarantees on compatibility across patch versions > - > > Key: HBASE-16422 > URL: https://issues.apache.org/jira/browse/HBASE-16422 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: stack >Assignee: stack > Attachments: HBASE-16422.master.001.patch, > HBASE-16422.master.002.patch > > > Out of the dev discussion [1], comes this change that tightens up our > guarantees adding new API; that we'll do it in a non-breaking way across > patch versions. > 1. http://mail-archives.apache.org/mod_mbox/hbase-dev/201608.mbox/browser -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16318) fail build if license isn't in whitelist
[ https://issues.apache.org/jira/browse/HBASE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425517#comment-15425517 ] Hudson commented on HBASE-16318: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1433 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1433/]) HBASE-16318 consistently use the correct name for 'Apache License, (busbey: rev cc12be39e568b873c7699e38c18d299db037453b) * (edit) pom.xml * (edit) hbase-resource-bundle/src/main/resources/supplemental-models.xml * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm HBASE-16318 fail build while rendering velocity template if dependency (busbey: rev e51fcdd7788271d33cf02240155429358d71627b) * (edit) hbase-resource-bundle/src/main/resources/META-INF/LICENSE.vm > fail build if license isn't in whitelist > > > Key: HBASE-16318 > URL: https://issues.apache.org/jira/browse/HBASE-16318 > Project: HBase > Issue Type: Sub-task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 0.98.22, 1.1.7 > > Attachments: HBASE-16318.0.patch, HBASE-16318.1.patch, > HBASE-16318.2.patch, HBASE-16318.3.patch > > > we use supplemental-models.xml to make sure we have consistent names and > descriptions for licenses. we also know what licenses we expect to see in our > build. If we see a different one > # fail the velocity template process > # if possible, include some information about why this happened -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16433) Remove AsyncRpcChannel related stuffs
[ https://issues.apache.org/jira/browse/HBASE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425354#comment-15425354 ] stack edited comment on HBASE-16433 at 8/17/16 10:33 PM: - PayloadCarryingRpcController is indeed a pain (Weird sidenote is that I just discovered, it is not always present). [~Apache9] sleeping at mo. [~jurmous] So you think do his suggestion of a callback on Call on the implementation we get back from the createProtobufRpcChannel call? was (Author: stack): PayloadCarryingRpcController is indeed a pain and I just discovered, not always present. [~Apache9] sleeping at mo. [~jurmous] So you think do his suggestion of a callback on Call on the implementation we get back from the createProtobufRpcChannel call? > Remove AsyncRpcChannel related stuffs > - > > Key: HBASE-16433 > URL: https://issues.apache.org/jira/browse/HBASE-16433 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16433.patch > > > AsyncRpcChannel can not be used by protobuf stub. We should implement the > async logic along with the RpcChannel interface of protobuf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425514#comment-15425514
]
Hudson commented on HBASE-12721:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1433 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1433/])
HBASE-12721 Create Docker container cluster infrastructure to enable (busbey:
rev ccf5d27d7aa238c8398d2818928a71f39bd749a0)
* (add) dev-support/apache_hbase_topology/profile.cfg
* (add) dev-support/apache_hbase_topology/Dockerfile
* (add) dev-support/apache_hbase_topology/configurations.cfg
* (add) dev-support/apache_hbase_topology/ssh/id_rsa.pub
* (add) dev-support/apache_hbase_topology/__init__.py
* (add) dev-support/apache_hbase_topology/actions.py
* (add) dev-support/apache_hbase_topology/ssh/id_rsa
* (add) dev-support/apache_hbase_topology/README.md
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Fix For: 2.0.0
>
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch,
> HBASE-12721_v7.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15635) Mean age of Blocks in cache (seconds) on webUI should be greater than zero
[ https://issues.apache.org/jira/browse/HBASE-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425515#comment-15425515 ] Hudson commented on HBASE-15635: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1433 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1433/]) HBASE-15635 Mean age of Blocks in cache (seconds) on webUI should be (stack: rev fb4ef5357115c708b0857bdce0a0c2f4d38fdd10) * (edit) hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheUtil.java > Mean age of Blocks in cache (seconds) on webUI should be greater than zero > -- > > Key: HBASE-15635 > URL: https://issues.apache.org/jira/browse/HBASE-15635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.17 >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.4.0, 1.3.1, 0.98.22 > > Attachments: 0.98.png, 7BFFAF68-0807-400C-853F-706B498449E1.png, > HBASE-15635.0.98.patch, HBASE-15635.patch, HBASE-15635_v1.patch, > HBASE-15635_v2.patch, HBASE-15635_v2.patch, master.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16384) Update report-flakies.py script to allow specifying a list of build ids to be excluded
[ https://issues.apache.org/jira/browse/HBASE-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425513#comment-15425513 ] Hudson commented on HBASE-16384: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1433 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1433/]) HBASE-16384 Update report-flakies.py script to allow specifying a list (appy: rev 7999bb9bd379550d0b537c088005b73ff63f1857) * (edit) dev-support/report-flakies.py * (edit) dev-support/findHangingTests.py > Update report-flakies.py script to allow specifying a list of build ids to be > excluded > -- > > Key: HBASE-16384 > URL: https://issues.apache.org/jira/browse/HBASE-16384 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16384.master.001.patch, > HBASE-16384.master.002.patch, HBASE-16384.master.003.patch, > HBASE-16384.master.004.patch, HBASE-16384.master.005.patch, > HBASE-16384.master.006.patch, HBASE-16384.master.007.patch, > HBASE-16384.master.008.patch, HBASE-16384.master.009.patch > > > Sometimes, builds fail mysteriously and leave lots of tests hanging. This > makes [flaky > list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html] > go crazy. > This patch adds that feature to specify build ids to exclude in > report-flakies.py. > If we find that a build screwed up, we can exclude it using "exclude=" option > in --urls param and rerun the job to fix the flaky list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16434) Improve flaky dashboard
[
https://issues.apache.org/jira/browse/HBASE-16434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425509#comment-15425509
]
Hadoop QA commented on HBASE-16434:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} pylint {color} | {color:red} 0m 5s {color}
| {color:red} The patch generated 1 new + 21 unchanged - 20 fixed = 22 total
(was 41) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 11s
{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 14s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-17 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824243/HBASE-16434.master.001.patch
|
| JIRA Issue | HBASE-16434 |
| Optional Tests | asflicense pylint |
| uname | Linux 45c81e6c656f 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / e637a61 |
| pylint | v1.6.4 |
| pylint |
https://builds.apache.org/job/PreCommit-HBASE-Build/3134/artifact/patchprocess/diff-patch-pylint.txt
|
| hbaseprotoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/3134/artifact/patchprocess/patch-hbaseprotoc-root.txt
|
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3134/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Improve flaky dashboard
> ---
>
> Key: HBASE-16434
> URL: https://issues.apache.org/jira/browse/HBASE-16434
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
>Priority: Minor
> Attachments: HBASE-16434.master.001.patch
>
>
> - Sort list of tests in descending order of flakyness
> - add date and flaky tests count
> - local hrefs to job results
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: 16179.v0.txt > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Attachments: 16179.v0.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: (was: 16179.v0.txt) > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425498#comment-15425498 ] Ted Yu commented on HBASE-16179: With patch v0, here are the compilation errors: http://pastebin.com/BA1czzdT > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16318) fail build if license isn't in whitelist
[ https://issues.apache.org/jira/browse/HBASE-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16318: Resolution: Fixed Status: Resolved (was: Patch Available) pushed to all active branches. I'm not sure where folks are on their RC making, so I left the fix versions at the ones last modified by the RMs. > fail build if license isn't in whitelist > > > Key: HBASE-16318 > URL: https://issues.apache.org/jira/browse/HBASE-16318 > Project: HBase > Issue Type: Sub-task > Components: build, dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 0.98.22, 1.1.7 > > Attachments: HBASE-16318.0.patch, HBASE-16318.1.patch, > HBASE-16318.2.patch, HBASE-16318.3.patch > > > we use supplemental-models.xml to make sure we have consistent names and > descriptions for licenses. we also know what licenses we expect to see in our > build. If we see a different one > # fail the velocity template process > # if possible, include some information about why this happened -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16434) Improve flaky dashboard
[ https://issues.apache.org/jira/browse/HBASE-16434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16434: - Status: Patch Available (was: Open) > Improve flaky dashboard > --- > > Key: HBASE-16434 > URL: https://issues.apache.org/jira/browse/HBASE-16434 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16434.master.001.patch > > > - Sort list of tests in descending order of flakyness > - add date and flaky tests count > - local hrefs to job results -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16434) Improve flaky dashboard
[ https://issues.apache.org/jira/browse/HBASE-16434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16434: - Attachment: HBASE-16434.master.001.patch > Improve flaky dashboard > --- > > Key: HBASE-16434 > URL: https://issues.apache.org/jira/browse/HBASE-16434 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16434.master.001.patch > > > - Sort list of tests in descending order of flakyness > - add date and flaky tests count > - local hrefs to job results -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16378) Procedure v2 - Make ProcedureException extend HBaseException
[ https://issues.apache.org/jira/browse/HBASE-16378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16378: Resolution: Fixed Fix Version/s: (was: 1.2.4) (was: 1.1.7) (was: 1.3.0) Status: Resolved (was: Patch Available) > Procedure v2 - Make ProcedureException extend HBaseException > > > Key: HBASE-16378 > URL: https://issues.apache.org/jira/browse/HBASE-16378 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0, 1.1.5, 1.2.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16378-v0.patch, HBASE-16378-v1.patch > > > Make ProcedureException extend HBaseException, so we can avoid stuff like > HBASE-15591. and avoid try/catch ProcedureException and direct rethrows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425439#comment-15425439
]
stack commented on HBASE-16429:
---
Clever. Thank you for figuring this out and fixing. Nice one. +1.
> FSHLog: deadlock if rollWriter called when ring buffer filled with appends
> --
>
> Key: HBASE-16429
> URL: https://issues.apache.org/jira/browse/HBASE-16429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2, 1.2.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-16429.patch
>
>
> Recently we experienced an online problem that all handlers are stuck.
> Checking the jstack we could see all handler threads waiting for
> RingBuffer.next, while the single ring buffer consumer dead waiting for
> {{safePointReleasedLatch}} to count down:
> {noformat}
> Normal handler thread:
> "B.defaultRpcServer.handler=126,queue=9,port=16020" daemon prio=10
> tid=0x7efd4b44f800 nid=0x15f29 runnable [0x7efd3db7b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1222)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:736)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:698)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2095)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:774)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:756)
> RingBufferEventHandler thread waiting for safePointReleasedLatch:
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.append-pool2-t1"
> prio=10 tid=0x7efd320d nid=0x1777b waiting on condition
> [0x7efd2d2fa000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f01b48d9178> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.safePointAttained(FSHLog.java:1866)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:2066)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1909)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:756)
> {noformat}
> {{FSHLog#replaceWriter}} will call {{SafePointZigZagLatch#releaseSafePoint}}
> to count down {{safePointReleasedLatch}}, but replaceWriter got stuck when
> trying to publish a sync onto ring buffer:
> {noformat}
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.logRoller"
> daemon prio=10
[jira] [Updated] (HBASE-16429) FSHLog: deadlock if rollWriter called when ring buffer filled with appends
[
https://issues.apache.org/jira/browse/HBASE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16429:
--
Status: Patch Available (was: Open)
> FSHLog: deadlock if rollWriter called when ring buffer filled with appends
> --
>
> Key: HBASE-16429
> URL: https://issues.apache.org/jira/browse/HBASE-16429
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.2, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-16429.patch
>
>
> Recently we experienced an online problem that all handlers are stuck.
> Checking the jstack we could see all handler threads waiting for
> RingBuffer.next, while the single ring buffer consumer dead waiting for
> {{safePointReleasedLatch}} to count down:
> {noformat}
> Normal handler thread:
> "B.defaultRpcServer.handler=126,queue=9,port=16020" daemon prio=10
> tid=0x7efd4b44f800 nid=0x15f29 runnable [0x7efd3db7b000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:349)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog.append(FSHLog.java:1222)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3188)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:736)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:698)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2095)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:774)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:102)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:756)
> RingBufferEventHandler thread waiting for safePointReleasedLatch:
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.append-pool2-t1"
> prio=10 tid=0x7efd320d nid=0x1777b waiting on condition
> [0x7efd2d2fa000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f01b48d9178> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.safePointAttained(FSHLog.java:1866)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.attainSafePoint(FSHLog.java:2066)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1909)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:756)
> {noformat}
> {{FSHLog#replaceWriter}} will call {{SafePointZigZagLatch#releaseSafePoint}}
> to count down {{safePointReleasedLatch}}, but replaceWriter got stuck when
> trying to publish a sync onto ring buffer:
> {noformat}
> "regionserver/hadoop0369.et2.tbsite.net/11.251.152.226:16020.logRoller"
> daemon prio=10 tid=0x7efd320c8800 nid=0x16123 runnable
> [0x7efd311f6000]
>java.lang
[jira] [Commented] (HBASE-16224) Reduce the number of RPCs for the large PUTs
[
https://issues.apache.org/jira/browse/HBASE-16224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425435#comment-15425435
]
Hadoop QA commented on HBASE-16224:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m 42s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 141m 22s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-17 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824220/HBASE-16224-v9.patch |
| JIRA Issue | HBASE-16224 |
| Optional Tests | asflicense javac javadoc uni
[jira] [Commented] (HBASE-15936) Sort out how to push to apache/ repositories on Docker Hub
[ https://issues.apache.org/jira/browse/HBASE-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425431#comment-15425431 ] Dima Spivak commented on HBASE-15936: - Please do, [~jfarrell]! We already have [~busbey] and [~apurtell] from the HBase PMC as volunteers with permissions for the Apache HBase Bintray Docker registry, but if there can be something setup that is specifically for pre-release stuff that can be pushed on demand, that would be ideal. In that case, any recommendation on how to handle doing so from Jenkins? Can we get a "Jenkins user" account set up or is there some way to securely store user credentials for this purpose? > Sort out how to push to apache/ repositories on Docker Hub > -- > > Key: HBASE-15936 > URL: https://issues.apache.org/jira/browse/HBASE-15936 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: Dima Spivak >Assignee: Dima Spivak > Labels: docker > > It looks like there's already a [Docker Hub username for Apache > stuff|https://hub.docker.com/r/apache/]. We should figure out how to use it > to push the Docker images we use/build for testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16440) MemstoreChunkPool might cross its maxCount of chunks to pool
[
https://issues.apache.org/jira/browse/HBASE-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425422#comment-15425422
]
stack commented on HBASE-16440:
---
MemstoreLABChunk has a lot of Chunk in it. You don't want to subclass? Seems
same except for isPooled. You don't want to add support for this to the base
Chunk class? Chunk is a nice name (smile). The method to get one of these
things is getChunk.
In getChunk, you have three returns throughout the method. You don't want to
just have one on the end?
Otherwise patch looks good. Any chance of a test that can repro the scenario
you describe above?
Nice [~anoop.hbase]
> MemstoreChunkPool might cross its maxCount of chunks to pool
>
>
> Key: HBASE-16440
> URL: https://issues.apache.org/jira/browse/HBASE-16440
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Attachments: HBASE-16440.patch
>
>
> {code}
> void putbackChunks(BlockingQueue chunks) {
> int maxNumToPutback = this.maxCount - reclaimedChunks.size();
> if (maxNumToPutback <= 0) {
> return;
> }
> chunks.drainTo(reclaimedChunks, maxNumToPutback);
> // clear reference of any non-reclaimable chunks
> if (chunks.size() > 0) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Left " + chunks.size() + " unreclaimable chunks, removing
> them from queue");
> }
> chunks.clear();
> }
> }
> {code}
> There is no synchroization. 2 threads might be calling this API as part of a
> MSLAB close. (Once the memstore is flushed). It pass all the chunks used by
> it. (Those might not have been come from pool also). We try to put back
> chunks such that it is not crossing maxCount. Suppose maxCount is 10 and
> currently no chunks in 'reclaimedChunks'. Say both threads at line one. Both
> see 'maxNumToPutback ' as 10 and that will make 20 chunks being pooled.
> Similar issue is in putbackChunk(Chunk chunk) API also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15936) Sort out how to push to apache/ repositories on Docker Hub
[ https://issues.apache.org/jira/browse/HBASE-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425416#comment-15425416 ] Jake Farrell commented on HBASE-15936: -- the ASF bintray would probably be the best place for that, I can setup HBase for that if you would like > Sort out how to push to apache/ repositories on Docker Hub > -- > > Key: HBASE-15936 > URL: https://issues.apache.org/jira/browse/HBASE-15936 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: Dima Spivak >Assignee: Dima Spivak > Labels: docker > > It looks like there's already a [Docker Hub username for Apache > stuff|https://hub.docker.com/r/apache/]. We should figure out how to use it > to push the Docker images we use/build for testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16433) Remove AsyncRpcChannel related stuffs
[ https://issues.apache.org/jira/browse/HBASE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425354#comment-15425354 ] stack commented on HBASE-16433: --- PayloadCarryingRpcController is indeed a pain and I just discovered, not always present. [~Apache9] sleeping at mo. [~jurmous] So you think do his suggestion of a callback on Call on the implementation we get back from the createProtobufRpcChannel call? > Remove AsyncRpcChannel related stuffs > - > > Key: HBASE-16433 > URL: https://issues.apache.org/jira/browse/HBASE-16433 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16433.patch > > > AsyncRpcChannel can not be used by protobuf stub. We should implement the > async logic along with the RpcChannel interface of protobuf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16384) Update report-flakies.py script to allow specifying a list of build ids to be excluded
[ https://issues.apache.org/jira/browse/HBASE-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16384: - Fix Version/s: 2.0.0 > Update report-flakies.py script to allow specifying a list of build ids to be > excluded > -- > > Key: HBASE-16384 > URL: https://issues.apache.org/jira/browse/HBASE-16384 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16384.master.001.patch, > HBASE-16384.master.002.patch, HBASE-16384.master.003.patch, > HBASE-16384.master.004.patch, HBASE-16384.master.005.patch, > HBASE-16384.master.006.patch, HBASE-16384.master.007.patch, > HBASE-16384.master.008.patch, HBASE-16384.master.009.patch > > > Sometimes, builds fail mysteriously and leave lots of tests hanging. This > makes [flaky > list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html] > go crazy. > This patch adds that feature to specify build ids to exclude in > report-flakies.py. > If we find that a build screwed up, we can exclude it using "exclude=" option > in --urls param and rerun the job to fix the flaky list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16384) Update report-flakies.py script to allow specifying a list of build ids to be excluded
[ https://issues.apache.org/jira/browse/HBASE-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16384: - Resolution: Fixed Status: Resolved (was: Patch Available) > Update report-flakies.py script to allow specifying a list of build ids to be > excluded > -- > > Key: HBASE-16384 > URL: https://issues.apache.org/jira/browse/HBASE-16384 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16384.master.001.patch, > HBASE-16384.master.002.patch, HBASE-16384.master.003.patch, > HBASE-16384.master.004.patch, HBASE-16384.master.005.patch, > HBASE-16384.master.006.patch, HBASE-16384.master.007.patch, > HBASE-16384.master.008.patch, HBASE-16384.master.009.patch > > > Sometimes, builds fail mysteriously and leave lots of tests hanging. This > makes [flaky > list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html] > go crazy. > This patch adds that feature to specify build ids to exclude in > report-flakies.py. > If we find that a build screwed up, we can exclude it using "exclude=" option > in --urls param and rerun the job to fix the flaky list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15963) Create Jenkins automation to build apache-hbase clusterdock images
[
https://issues.apache.org/jira/browse/HBASE-15963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak updated HBASE-15963:
Parent Issue: HBASE-16442 (was: HBASE-12721)
> Create Jenkins automation to build apache-hbase clusterdock images
> --
>
> Key: HBASE-15963
> URL: https://issues.apache.org/jira/browse/HBASE-15963
> Project: HBase
> Issue Type: Sub-task
> Components: build
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> One of the main goals of {{clusterdock}} is to enable testing of distributed
> Apache HBase clusters built from source. There's no better way to facilitate
> this than to regularly (or continuously) build images of the branches we care
> about for use by other people and automation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15964) Create Jenkins automation to run ITBLL on upstream infrastructure
[
https://issues.apache.org/jira/browse/HBASE-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak updated HBASE-15964:
Parent Issue: HBASE-16442 (was: HBASE-12721)
> Create Jenkins automation to run ITBLL on upstream infrastructure
> -
>
> Key: HBASE-15964
> URL: https://issues.apache.org/jira/browse/HBASE-15964
> Project: HBase
> Issue Type: Sub-task
> Components: integration tests
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Being able to run {{IntegrationTestBigLinkedList}} on demand on, for example,
> a five-node Apache HBase cluster, would be cool. Also having different chaos
> monkeys running would be even cooler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16442) Create Jenkins automation for clusterdock
Dima Spivak created HBASE-16442: --- Summary: Create Jenkins automation for clusterdock Key: HBASE-16442 URL: https://issues.apache.org/jira/browse/HBASE-16442 Project: HBase Issue Type: New Feature Reporter: Dima Spivak Assignee: Dima Spivak -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16131) Add optional docker image build and publish steps to release documentation and build_rc.sh
[ https://issues.apache.org/jira/browse/HBASE-16131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak updated HBASE-16131: Parent Issue: HBASE-16441 (was: HBASE-12721) > Add optional docker image build and publish steps to release documentation > and build_rc.sh > -- > > Key: HBASE-16131 > URL: https://issues.apache.org/jira/browse/HBASE-16131 > Project: HBase > Issue Type: Sub-task > Components: build, community, documentation, test >Reporter: Andrew Purtell >Assignee: Dima Spivak > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15962) Create Jenkins automation to build clusterdock framework regularly
[
https://issues.apache.org/jira/browse/HBASE-15962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak resolved HBASE-15962.
-
Resolution: Invalid
Since the framework no longer lives under HBase, there's no need for this to be
a concern of HBase's.
> Create Jenkins automation to build clusterdock framework regularly
> --
>
> Key: HBASE-15962
> URL: https://issues.apache.org/jira/browse/HBASE-15962
> Project: HBase
> Issue Type: Sub-task
> Components: build
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> To avoid the need for a virtualenv or anything else annoying, {{clusterdock}}
> itself runs out of a Docker container. We'll need to come up with a way to
> build this framework regularly (ideally continuously) as changes go in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15936) Sort out how to push to apache/ repositories on Docker Hub
[ https://issues.apache.org/jira/browse/HBASE-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak resolved HBASE-15936. - Resolution: Fixed Docker Hub images are for automated builds only. Now trying to find out where to push the images we're creating pre-release on a thread to infra@... > Sort out how to push to apache/ repositories on Docker Hub > -- > > Key: HBASE-15936 > URL: https://issues.apache.org/jira/browse/HBASE-15936 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: Dima Spivak >Assignee: Dima Spivak > Labels: docker > > It looks like there's already a [Docker Hub username for Apache > stuff|https://hub.docker.com/r/apache/]. We should figure out how to use it > to push the Docker images we use/build for testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15961) Create documentation on how to use of clusterdock
[ https://issues.apache.org/jira/browse/HBASE-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak updated HBASE-15961: Parent Issue: HBASE-16441 (was: HBASE-12721) > Create documentation on how to use of clusterdock > - > > Key: HBASE-15961 > URL: https://issues.apache.org/jira/browse/HBASE-15961 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Dima Spivak >Assignee: Dima Spivak > > Ref guide should include information on starting Docker container-based > distributed HBase clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16441) Document use of Apache HBase clusterdock topology
Dima Spivak created HBASE-16441: --- Summary: Document use of Apache HBase clusterdock topology Key: HBASE-16441 URL: https://issues.apache.org/jira/browse/HBASE-16441 Project: HBase Issue Type: New Feature Components: documentation Reporter: Dima Spivak Assignee: Dima Spivak -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425317#comment-15425317
]
Dima Spivak commented on HBASE-12721:
-
Thanks, [~busbey]. I'll move those tasks to live under a new umbrella JIRA
having to do with integrating the topology into testing. Documentation is
blocked at the moment by the uncertainty of where we should be pushing Docker
images. Once that's resolved, I'll hammer out updates to the ref guide and the
RC scripts before taking a stab at Jenkins jobs to run {{hbase-it}} upstream
somewhere.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Fix For: 2.0.0
>
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch,
> HBASE-12721_v7.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak updated HBASE-12721:
Release Note:
Downstream users wishing to test HBase in a "distributed" fashion (multiple
"nodes" running as separate containers on the same host) can now do so in an
automated fashion while leveraging Docker for process isolation via the
clusterdock project.
For details see the README.md in the dev-support/apache_hbase_topology folder.
was:
Downstream users wishing to test HBase in a pseudo-distributed fashion can now
do so in an automated fashion while leveraging Docker for process isolation via
the clusterdock project.
For details see the README.md in the dev-support/apache_hbase_topology folder.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Fix For: 2.0.0
>
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch,
> HBASE-12721_v7.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13094) Consider Filters that are evaluated before deletes and see delete markers
[ https://issues.apache.org/jira/browse/HBASE-13094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425313#comment-15425313 ] Lars Hofhansl commented on HBASE-13094: --- Yep. Or we come up with an API that enforces this structurally. For example we could adapt #4 to add an API to Get/Scan that instead only controls whether the Filter.filterKeyValue (set as before with setFilter) is evaluated either pre or post deletes. The effect would be the same as the patch I have, but with a clean no-magic API. (Although that almost leads me back to the approach of indicating the "pre-postness" in the Filter itself). Or one could think about option #3 above. > Consider Filters that are evaluated before deletes and see delete markers > - > > Key: HBASE-13094 > URL: https://issues.apache.org/jira/browse/HBASE-13094 > Project: HBase > Issue Type: Brainstorming > Components: regionserver, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 13094-0.98.txt > > > That would be good for full control filtering of all cells, such as needed > for some transaction implementations. > [~ghelmling] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16384) Update report-flakies.py script to allow specifying a list of build ids to be excluded
[
https://issues.apache.org/jira/browse/HBASE-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425310#comment-15425310
]
Dima Spivak commented on HBASE-16384:
-
Well FWIW, the only issue I cared about was the import order warning: you
should put {{logging}} and {{os}} imports before Jinja2 to be compliant with
PEP 8. I guess a follow-on JIRA is fine.
> Update report-flakies.py script to allow specifying a list of build ids to be
> excluded
> --
>
> Key: HBASE-16384
> URL: https://issues.apache.org/jira/browse/HBASE-16384
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16384.master.001.patch,
> HBASE-16384.master.002.patch, HBASE-16384.master.003.patch,
> HBASE-16384.master.004.patch, HBASE-16384.master.005.patch,
> HBASE-16384.master.006.patch, HBASE-16384.master.007.patch,
> HBASE-16384.master.008.patch, HBASE-16384.master.009.patch
>
>
> Sometimes, builds fail mysteriously and leave lots of tests hanging. This
> makes [flaky
> list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html]
> go crazy.
> This patch adds that feature to specify build ids to exclude in
> report-flakies.py.
> If we find that a build screwed up, we can exclude it using "exclude=" option
> in --urls param and rerun the job to fix the flaky list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425307#comment-15425307
]
Sean Busbey commented on HBASE-12721:
-
Do the subtasks need to get promoted to full tasks, or closed out?
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Fix For: 2.0.0
>
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch,
> HBASE-12721_v7.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16433) Remove AsyncRpcChannel related stuffs
[ https://issues.apache.org/jira/browse/HBASE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425297#comment-15425297 ] Jurriaan Mous edited comment on HBASE-16433 at 8/17/16 8:27 PM: I implemented two ways to do the Async Calls: createProtobufRpcChannel which handles everything the protobuf way with RpcChannel and createRpcChannel which exposes a bit of the Netty internals. I wanted to have this second route so the Table impl doesn't need to do everything with wrapping in cumbersome RpcController and RpcCallback which are not developer friendly for async communication. You can see some hacking around the RpcChannel barrier with the PayloadCarryingRpcController to just pass on variables to the Rpc layers. I wanted something simpler for the Async client with a more direct connection to what is happening in the Rpc layer. I am agreeing that leaking Netty is not the way forward and I want to replace that with java native CompletableFuture which is Java 8 only. So we can pass that on from the lower RPC layers without any hacking around the Protobuf RPC layers. But it is being hold up by HBASE-15624 and that was hold back by the just fixed HBASE-15882 So my vote would be to leave the more direct way of communicating with the RPC layers with AsyncRpcChannel in but to base it on CompletableFuture as is in patch v1 in HBASE-15978. Then it would also be easier to land the AsyncTableImpl so it is more visible why the AsyncRpcChannel is there. was (Author: jurmous): I implemented two ways to do the Async Calls: createProtobufRpcChannel which handles everything the protobuf way with RpcChannel and createRpcChannel which exposes a bit of the Netty internals. I wanted to have this second route so the Table impl doesn't need to do everything with wrapping in cumbersome RpcController and RpcCallback which are not developer friendly for a complete async interfaces. You see now hacking around that barrier with PayloadCarryingRpcController to just pass on variables to the Rpc layers. I wanted something simpler for the Async client with a more direct connection to what is happening in the Rpc layer. I am agreeing that leaking Netty is not the way forward and I want to replace that with java native CompletableFuture which is Java 8 only. So we can pass that on from the lower RPC layers without any hacking around the Protobuf RPC layers. But it is being hold up by HBASE-15624 and that was hold back by the just fixed HBASE-15882 So my vote would be to leave the more direct way of communicating with the RPC layers with AsyncRpcChannel in but to base it on CompletableFuture as is in patch v1 in HBASE-15978. Then it would also be easier to land the AsyncTableImpl so it is more visible why the AsyncRpcChannel is there. > Remove AsyncRpcChannel related stuffs > - > > Key: HBASE-16433 > URL: https://issues.apache.org/jira/browse/HBASE-16433 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16433.patch > > > AsyncRpcChannel can not be used by protobuf stub. We should implement the > async logic along with the RpcChannel interface of protobuf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-12721:
Resolution: Fixed
Fix Version/s: 2.0.0
Release Note:
Downstream users wishing to test HBase in a pseudo-distributed fashion can now
do so in an automated fashion while leveraging Docker for process isolation via
the clusterdock project.
For details see the README.md in the dev-support/apache_hbase_topology folder.
Status: Resolved (was: Patch Available)
pushed to master! That sufficient for now? Maybe branch-specific backports in
follow-on?
Took a go at a release note, [~dimaspivak] or whomever else feel free to update.
thanks a ton for this [~dimaspivak]. is this baked enough for us to expand our
"getting started" section for pseudo distributed?
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Fix For: 2.0.0
>
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch,
> HBASE-12721_v7.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16384) Update report-flakies.py script to allow specifying a list of build ids to be excluded
[ https://issues.apache.org/jira/browse/HBASE-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425303#comment-15425303 ] Sean Busbey commented on HBASE-16384: - +1. I agree that a clean-up follow-on is the way to go for the lint issues. > Update report-flakies.py script to allow specifying a list of build ids to be > excluded > -- > > Key: HBASE-16384 > URL: https://issues.apache.org/jira/browse/HBASE-16384 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16384.master.001.patch, > HBASE-16384.master.002.patch, HBASE-16384.master.003.patch, > HBASE-16384.master.004.patch, HBASE-16384.master.005.patch, > HBASE-16384.master.006.patch, HBASE-16384.master.007.patch, > HBASE-16384.master.008.patch, HBASE-16384.master.009.patch > > > Sometimes, builds fail mysteriously and leave lots of tests hanging. This > makes [flaky > list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html] > go crazy. > This patch adds that feature to specify build ids to exclude in > report-flakies.py. > If we find that a build screwed up, we can exclude it using "exclude=" option > in --urls param and rerun the job to fix the flaky list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16433) Remove AsyncRpcChannel related stuffs
[ https://issues.apache.org/jira/browse/HBASE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425297#comment-15425297 ] Jurriaan Mous commented on HBASE-16433: --- I implemented two ways to do the Async Calls: createProtobufRpcChannel which handles everything the protobuf way with RpcChannel and createRpcChannel which exposes a bit of the Netty internals. I wanted to have this second route so the Table impl doesn't need to do everything with wrapping in cumbersome RpcController and RpcCallback which are not developer friendly for a complete async interfaces. You see now hacking around that barrier with PayloadCarryingRpcController to just pass on variables to the Rpc layers. I wanted something simpler for the Async client with a more direct connection to what is happening in the Rpc layer. I am agreeing that leaking Netty is not the way forward and I want to replace that with java native CompletableFuture which is Java 8 only. So we can pass that on from the lower RPC layers without any hacking around the Protobuf RPC layers. But it is being hold up by HBASE-15624 and that was hold back by the just fixed HBASE-15882 So my vote would be to leave the more direct way of communicating with the RPC layers with AsyncRpcChannel in but to base it on CompletableFuture as is in patch v1 in HBASE-15978. Then it would also be easier to land the AsyncTableImpl so it is more visible why the AsyncRpcChannel is there. > Remove AsyncRpcChannel related stuffs > - > > Key: HBASE-16433 > URL: https://issues.apache.org/jira/browse/HBASE-16433 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16433.patch > > > AsyncRpcChannel can not be used by protobuf stub. We should implement the > async logic along with the RpcChannel interface of protobuf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16384) Update report-flakies.py script to allow specifying a list of build ids to be excluded
[ https://issues.apache.org/jira/browse/HBASE-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425276#comment-15425276 ] Appy commented on HBASE-16384: -- Net pylint from patch are -ve, i.e. it's fixing a lot of old ones. The net count is bigger because it's showing old pylints from the newly added file in patch. These are not being introduced in this patch, so i'd rather not keep delaying this patch for fixing old lints. Old ones can be fixed overtime in future patches. > Update report-flakies.py script to allow specifying a list of build ids to be > excluded > -- > > Key: HBASE-16384 > URL: https://issues.apache.org/jira/browse/HBASE-16384 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16384.master.001.patch, > HBASE-16384.master.002.patch, HBASE-16384.master.003.patch, > HBASE-16384.master.004.patch, HBASE-16384.master.005.patch, > HBASE-16384.master.006.patch, HBASE-16384.master.007.patch, > HBASE-16384.master.008.patch, HBASE-16384.master.009.patch > > > Sometimes, builds fail mysteriously and leave lots of tests hanging. This > makes [flaky > list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html] > go crazy. > This patch adds that feature to specify build ids to exclude in > report-flakies.py. > If we find that a build screwed up, we can exclude it using "exclude=" option > in --urls param and rerun the job to fix the flaky list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16419) check REPLICATION_SCOPE's value more stringently
[
https://issues.apache.org/jira/browse/HBASE-16419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16419:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the fix, Guangxu.
> check REPLICATION_SCOPE's value more stringently
>
>
> Key: HBASE-16419
> URL: https://issues.apache.org/jira/browse/HBASE-16419
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0, 1.2.2
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16419-branch-1.2-v1.patch,
> HBASE-16419-branch-1.2-v2.patch, HBASE-16419-v1.patch
>
>
> When create table or modify table, the master will check if the value of
> REPLICATION_SCOPE is less than 0, however the value of REPLICATION_SCOPE must
> be 0 or 1. Otherwise will lead to regionserver shutdown, so I think should
> be check the values of REPLICATION_SCOPE more stringent.
> Beginning I don't fully understand the usage of REPLICATION_SCOPE, then set
> REPLICATION_SCOPE to 2 by mistake.when I insert data to table,the
> regionservers abort one by one,finanly
> the cluster abort，the exceptions as follow:
> {quote}
> 2016-08-16 12:34:45,245 WARN [regionserver/host:60023.append-pool1-t1]
> wal.FSHLog: Append sequenceId=94, requesting roll of WAL
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.protobuf.generated.WALProtos$FamilyScope$Builder.setScopeType(WALProtos.java:3939)
> at org.apache.hadoop.hbase.wal.WALKey.getBuilder(WALKey.java:618)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:118)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1886)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2016-08-16 12:34:45,293 INFO [MemStoreFlusher.0] regionserver.HStore: Added
> hdfs://hbase-test-27/hbase1.2.2/data/default/usertable/2aa98c17845c9c6d5c8760b87b3ba09a/i/35825c94e72945c0bf7df3f0adefa1b6,
> entries=1161600, sequenceid=59, filesize=167.6 M
> 2016-08-16 12:34:45,296 FATAL [MemStoreFlusher.0] regionserver.HRegionServer:
> ABORTING region server hbase-10-166-141-99,60023,1471262434177: Replay of WAL
> required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region:
> usertable,,1471262560009.2aa98c17845c9c6d5c8760b87b3ba09a.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2427)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2105)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2067)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1958)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1884)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException:
> Append sequenceId=94, requesting roll of WAL
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1898)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ... 1 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.protobuf.generated.WALProtos$FamilyScope$Builder.setScopeType(WALProtos.java:3939)
> at
[jira] [Commented] (HBASE-16419) check REPLICATION_SCOPE's value more stringently
[
https://issues.apache.org/jira/browse/HBASE-16419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425256#comment-15425256
]
Hudson commented on HBASE-16419:
FAILURE: Integrated in Jenkins build HBase-1.4 #353 (See
[https://builds.apache.org/job/HBase-1.4/353/])
HBASE-16419 check REPLICATION_SCOPE's value more stringently (Guangxu (tedyu:
rev a944a198791859331f9541ee454afe810db69c45)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> check REPLICATION_SCOPE's value more stringently
>
>
> Key: HBASE-16419
> URL: https://issues.apache.org/jira/browse/HBASE-16419
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0, 1.2.2
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16419-branch-1.2-v1.patch,
> HBASE-16419-branch-1.2-v2.patch, HBASE-16419-v1.patch
>
>
> When create table or modify table, the master will check if the value of
> REPLICATION_SCOPE is less than 0, however the value of REPLICATION_SCOPE must
> be 0 or 1. Otherwise will lead to regionserver shutdown, so I think should
> be check the values of REPLICATION_SCOPE more stringent.
> Beginning I don't fully understand the usage of REPLICATION_SCOPE, then set
> REPLICATION_SCOPE to 2 by mistake.when I insert data to table,the
> regionservers abort one by one,finanly
> the cluster abort，the exceptions as follow:
> {quote}
> 2016-08-16 12:34:45,245 WARN [regionserver/host:60023.append-pool1-t1]
> wal.FSHLog: Append sequenceId=94, requesting roll of WAL
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.protobuf.generated.WALProtos$FamilyScope$Builder.setScopeType(WALProtos.java:3939)
> at org.apache.hadoop.hbase.wal.WALKey.getBuilder(WALKey.java:618)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:118)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1886)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2016-08-16 12:34:45,293 INFO [MemStoreFlusher.0] regionserver.HStore: Added
> hdfs://hbase-test-27/hbase1.2.2/data/default/usertable/2aa98c17845c9c6d5c8760b87b3ba09a/i/35825c94e72945c0bf7df3f0adefa1b6,
> entries=1161600, sequenceid=59, filesize=167.6 M
> 2016-08-16 12:34:45,296 FATAL [MemStoreFlusher.0] regionserver.HRegionServer:
> ABORTING region server hbase-10-166-141-99,60023,1471262434177: Replay of WAL
> required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region:
> usertable,,1471262560009.2aa98c17845c9c6d5c8760b87b3ba09a.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2427)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2105)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2067)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1958)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1884)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException:
> Append sequenceId=94, requesting roll of WAL
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1898)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.c
[jira] [Commented] (HBASE-15635) Mean age of Blocks in cache (seconds) on webUI should be greater than zero
[ https://issues.apache.org/jira/browse/HBASE-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425257#comment-15425257 ] Hudson commented on HBASE-15635: FAILURE: Integrated in Jenkins build HBase-1.4 #353 (See [https://builds.apache.org/job/HBase-1.4/353/]) HBASE-15635 Mean age of Blocks in cache (seconds) on webUI should be (stack: rev 3d7bcd020e03ca89dccab18dd87cde453beeeafa) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java * (edit) hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheUtil.java > Mean age of Blocks in cache (seconds) on webUI should be greater than zero > -- > > Key: HBASE-15635 > URL: https://issues.apache.org/jira/browse/HBASE-15635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.17 >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.4.0, 1.3.1, 0.98.22 > > Attachments: 0.98.png, 7BFFAF68-0807-400C-853F-706B498449E1.png, > HBASE-15635.0.98.patch, HBASE-15635.patch, HBASE-15635_v1.patch, > HBASE-15635_v2.patch, HBASE-15635_v2.patch, master.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16384) Update report-flakies.py script to allow specifying a list of build ids to be excluded
[ https://issues.apache.org/jira/browse/HBASE-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425245#comment-15425245 ] Dima Spivak commented on HBASE-16384: - You got a few more Pylint warnings with your newest patch, [~appy]. Once those are fixed, if it works as expected on your machine, +1 from me. > Update report-flakies.py script to allow specifying a list of build ids to be > excluded > -- > > Key: HBASE-16384 > URL: https://issues.apache.org/jira/browse/HBASE-16384 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16384.master.001.patch, > HBASE-16384.master.002.patch, HBASE-16384.master.003.patch, > HBASE-16384.master.004.patch, HBASE-16384.master.005.patch, > HBASE-16384.master.006.patch, HBASE-16384.master.007.patch, > HBASE-16384.master.008.patch, HBASE-16384.master.009.patch > > > Sometimes, builds fail mysteriously and leave lots of tests hanging. This > makes [flaky > list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html] > go crazy. > This patch adds that feature to specify build ids to exclude in > report-flakies.py. > If we find that a build screwed up, we can exclude it using "exclude=" option > in --urls param and rerun the job to fix the flaky list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
