[jira] [Commented] (HBASE-11279) Block cache could be disabled by mistake
[
https://issues.apache.org/jira/browse/HBASE-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014902#comment-14014902
]
Jimmy Xiang commented on HBASE-11279:
-
Log shows 2014-05-31 14:49:16,075 INFO [pool-1-thread-1]
hfile.CacheConfig(275): CacheConfig:disabled
> Block cache could be disabled by mistake
>
>
> Key: HBASE-11279
> URL: https://issues.apache.org/jira/browse/HBASE-11279
> Project: HBase
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>
> There is a weird test failure:
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hbase.io.hfile.TestPrefetch.readStoreFile(TestPrefetch.java:96)
> at
> org.apache.hadoop.hbase.io.hfile.TestPrefetch.testPrefetch(TestPrefetch.java:66)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Looked into it, and found it could be because CacheConfig tries to compare a
> float with a long. Probably we should do this instead:
> {noformat}
> -if (cachePercentage == 0L) {
> +if (cachePercentage <= 0.0001f) {
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11279) Block cache could be disabled by mistake
Jimmy Xiang created HBASE-11279:
---
Summary: Block cache could be disabled by mistake
Key: HBASE-11279
URL: https://issues.apache.org/jira/browse/HBASE-11279
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
There is a weird test failure:
{noformat}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.io.hfile.TestPrefetch.readStoreFile(TestPrefetch.java:96)
at
org.apache.hadoop.hbase.io.hfile.TestPrefetch.testPrefetch(TestPrefetch.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}
Looked into it, and found it could be because CacheConfig tries to compare a
float with a long. Probably we should do this instead:
{noformat}
-if (cachePercentage == 0L) {
+if (cachePercentage <= 0.0001f) {
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11277) RPCServer threads can wedge under high load
[
https://issues.apache.org/jira/browse/HBASE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014899#comment-14014899
]
Hudson commented on HBASE-11277:
SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #306 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/306/])
HBASE-11277 RpcServer threads can wedge under high load (apurtell: rev
7257dbc58fe27ec3b1b573707bb0a8f6fde3d462)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
> RPCServer threads can wedge under high load
> ---
>
> Key: HBASE-11277
> URL: https://issues.apache.org/jira/browse/HBASE-11277
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.2, 0.98.3
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.99.0, 0.96.3, 0.98.3
>
> Attachments: HBASE-11277.patch
>
>
> This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load,
> RPCServer threads can wedge, fail to make progess, and consume 100% CPU time
> on a core indefinitely.
> Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE
> threads are mostly in EPollArrayWrapper.epollWait or
> FileDispatcherImpl.read0. The number of threads found in
> FileDispatcherImpl.read0 correspond to the number of runaway threads expected
> based on looking at 'top' output. These look like:
> {noformat}
> Thread 64758: (state = IN_NATIVE)
> - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int)
> @bci=0 (Compiled frame; information may be imprecise)
> - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int)
> @bci=4, line=39 (Compiled frame)
> - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor,
> java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223
> (Compil
> ed frame)
> - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long,
> sun.nio.ch.NativeDispatcher) @bci=48, line=197 (Compiled frame)
> - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379
> (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel,
> java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess()
> @bci=509, line=1488 (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey)
> @bci=23, line=790 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97,
> line=581 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1,
> line=556 (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1145 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11277) RPCServer threads can wedge under high load
[
https://issues.apache.org/jira/browse/HBASE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014883#comment-14014883
]
Hudson commented on HBASE-11277:
SUCCESS: Integrated in HBase-TRUNK #5163 (See
[https://builds.apache.org/job/HBase-TRUNK/5163/])
HBASE-11277 RpcServer threads can wedge under high load (apurtell: rev
7f2f7c23a7bd274925f6438d316d8e2e8d8709d6)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
> RPCServer threads can wedge under high load
> ---
>
> Key: HBASE-11277
> URL: https://issues.apache.org/jira/browse/HBASE-11277
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.2, 0.98.3
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.99.0, 0.96.3, 0.98.3
>
> Attachments: HBASE-11277.patch
>
>
> This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load,
> RPCServer threads can wedge, fail to make progess, and consume 100% CPU time
> on a core indefinitely.
> Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE
> threads are mostly in EPollArrayWrapper.epollWait or
> FileDispatcherImpl.read0. The number of threads found in
> FileDispatcherImpl.read0 correspond to the number of runaway threads expected
> based on looking at 'top' output. These look like:
> {noformat}
> Thread 64758: (state = IN_NATIVE)
> - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int)
> @bci=0 (Compiled frame; information may be imprecise)
> - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int)
> @bci=4, line=39 (Compiled frame)
> - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor,
> java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223
> (Compil
> ed frame)
> - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long,
> sun.nio.ch.NativeDispatcher) @bci=48, line=197 (Compiled frame)
> - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379
> (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel,
> java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess()
> @bci=509, line=1488 (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey)
> @bci=23, line=790 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97,
> line=581 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1,
> line=556 (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1145 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11275) [AccessController] postCreateTable hook fails when another CP creates table on their startup
[
https://issues.apache.org/jira/browse/HBASE-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014884#comment-14014884
]
Hudson commented on HBASE-11275:
SUCCESS: Integrated in HBase-TRUNK #5163 (See
[https://builds.apache.org/job/HBase-TRUNK/5163/])
HBASE-11275 [AccessController] postCreateTable hook fails when another CP
creates table on their startup (Anoop Sam John) (apurtell: rev
af63fba391afa26545656f6335e75eb3577d5477)
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
> [AccessController] postCreateTable hook fails when another CP creates table
> on their startup
>
>
> Key: HBASE-11275
> URL: https://issues.apache.org/jira/browse/HBASE-11275
> Project: HBase
> Issue Type: Bug
> Components: security
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0, 0.98.3
>
> Attachments: HBASE-11275.patch
>
>
> I am using AC and VC together.( In the required order with AC first)
> Still I am getting the below Exception
> {code}
> 2014-05-29 14:10:55,574 ERROR [ActiveMasterManager] client.AsyncProcess(313):
> Failed to get region location
> org.apache.hadoop.hbase.TableNotFoundException: hbase:acl
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1056)
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:995)
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:952)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:380)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:309)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:263)
>at
> org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1022)
>at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>at
> org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>at
> org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>at
> org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:1390)
>at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:619)
>at org.apache.hadoop.hbase.master.HMaster.access$3(HMaster.java:462)
>at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1258)
>at java.lang.Thread.run(Thread.java:745)
> 2014-05-29 14:10:55,587 ERROR [ActiveMasterManager] master.HMaster(621):
> Coprocessor postStartMaster() hook failed
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: hbase:acl: 1 time,
>at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:180)
>at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1(AsyncProcess.java:179)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1024)
>at
> org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1025)
>at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>at
> org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>at
> org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>at
> org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterC
[jira] [Commented] (HBASE-11275) [AccessController] postCreateTable hook fails when another CP creates table on their startup
[
https://issues.apache.org/jira/browse/HBASE-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014881#comment-14014881
]
Hudson commented on HBASE-11275:
SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #305 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/305/])
HBASE-11275 [AccessController] postCreateTable hook fails when another CP
creates table on their startup (Anoop Sam John) (apurtell: rev
f5338d7e0a6901467696fe32f0d39796662f471c)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> [AccessController] postCreateTable hook fails when another CP creates table
> on their startup
>
>
> Key: HBASE-11275
> URL: https://issues.apache.org/jira/browse/HBASE-11275
> Project: HBase
> Issue Type: Bug
> Components: security
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0, 0.98.3
>
> Attachments: HBASE-11275.patch
>
>
> I am using AC and VC together.( In the required order with AC first)
> Still I am getting the below Exception
> {code}
> 2014-05-29 14:10:55,574 ERROR [ActiveMasterManager] client.AsyncProcess(313):
> Failed to get region location
> org.apache.hadoop.hbase.TableNotFoundException: hbase:acl
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1056)
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:995)
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:952)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:380)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:309)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:263)
>at
> org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1022)
>at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>at
> org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>at
> org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>at
> org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:1390)
>at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:619)
>at org.apache.hadoop.hbase.master.HMaster.access$3(HMaster.java:462)
>at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1258)
>at java.lang.Thread.run(Thread.java:745)
> 2014-05-29 14:10:55,587 ERROR [ActiveMasterManager] master.HMaster(621):
> Coprocessor postStartMaster() hook failed
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: hbase:acl: 1 time,
>at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:180)
>at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1(AsyncProcess.java:179)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1024)
>at
> org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1025)
>at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>at
> org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>at
> org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>at
> org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost
[jira] [Commented] (HBASE-11277) RPCServer threads can wedge under high load
[
https://issues.apache.org/jira/browse/HBASE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014875#comment-14014875
]
Hudson commented on HBASE-11277:
SUCCESS: Integrated in hbase-0.96-hadoop2 #279 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/279/])
HBASE-11277 RpcServer threads can wedge under high load (apurtell: rev
0c374ceea9374c4743a427c7029335a975ad3f14)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
> RPCServer threads can wedge under high load
> ---
>
> Key: HBASE-11277
> URL: https://issues.apache.org/jira/browse/HBASE-11277
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.2, 0.98.3
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.99.0, 0.96.3, 0.98.3
>
> Attachments: HBASE-11277.patch
>
>
> This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load,
> RPCServer threads can wedge, fail to make progess, and consume 100% CPU time
> on a core indefinitely.
> Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE
> threads are mostly in EPollArrayWrapper.epollWait or
> FileDispatcherImpl.read0. The number of threads found in
> FileDispatcherImpl.read0 correspond to the number of runaway threads expected
> based on looking at 'top' output. These look like:
> {noformat}
> Thread 64758: (state = IN_NATIVE)
> - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int)
> @bci=0 (Compiled frame; information may be imprecise)
> - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int)
> @bci=4, line=39 (Compiled frame)
> - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor,
> java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223
> (Compil
> ed frame)
> - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long,
> sun.nio.ch.NativeDispatcher) @bci=48, line=197 (Compiled frame)
> - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379
> (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel,
> java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess()
> @bci=509, line=1488 (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey)
> @bci=23, line=790 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97,
> line=581 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1,
> line=556 (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1145 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11275) [AccessController] postCreateTable hook fails when another CP creates table on their startup
[
https://issues.apache.org/jira/browse/HBASE-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014872#comment-14014872
]
Hudson commented on HBASE-11275:
SUCCESS: Integrated in HBase-0.98 #324 (See
[https://builds.apache.org/job/HBase-0.98/324/])
HBASE-11275 [AccessController] postCreateTable hook fails when another CP
creates table on their startup (Anoop Sam John) (apurtell: rev
f5338d7e0a6901467696fe32f0d39796662f471c)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> [AccessController] postCreateTable hook fails when another CP creates table
> on their startup
>
>
> Key: HBASE-11275
> URL: https://issues.apache.org/jira/browse/HBASE-11275
> Project: HBase
> Issue Type: Bug
> Components: security
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0, 0.98.3
>
> Attachments: HBASE-11275.patch
>
>
> I am using AC and VC together.( In the required order with AC first)
> Still I am getting the below Exception
> {code}
> 2014-05-29 14:10:55,574 ERROR [ActiveMasterManager] client.AsyncProcess(313):
> Failed to get region location
> org.apache.hadoop.hbase.TableNotFoundException: hbase:acl
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1056)
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:995)
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:952)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:380)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:309)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:263)
>at
> org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1022)
>at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>at
> org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>at
> org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>at
> org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:1390)
>at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:619)
>at org.apache.hadoop.hbase.master.HMaster.access$3(HMaster.java:462)
>at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1258)
>at java.lang.Thread.run(Thread.java:745)
> 2014-05-29 14:10:55,587 ERROR [ActiveMasterManager] master.HMaster(621):
> Coprocessor postStartMaster() hook failed
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: hbase:acl: 1 time,
>at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:180)
>at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1(AsyncProcess.java:179)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1024)
>at
> org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1025)
>at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>at
> org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>at
> org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>at
> org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCopro
[jira] [Commented] (HBASE-11277) RPCServer threads can wedge under high load
[
https://issues.apache.org/jira/browse/HBASE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014871#comment-14014871
]
Hudson commented on HBASE-11277:
SUCCESS: Integrated in HBase-0.98 #324 (See
[https://builds.apache.org/job/HBase-0.98/324/])
HBASE-11277 RpcServer threads can wedge under high load (apurtell: rev
7257dbc58fe27ec3b1b573707bb0a8f6fde3d462)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
> RPCServer threads can wedge under high load
> ---
>
> Key: HBASE-11277
> URL: https://issues.apache.org/jira/browse/HBASE-11277
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.2, 0.98.3
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.99.0, 0.96.3, 0.98.3
>
> Attachments: HBASE-11277.patch
>
>
> This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load,
> RPCServer threads can wedge, fail to make progess, and consume 100% CPU time
> on a core indefinitely.
> Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE
> threads are mostly in EPollArrayWrapper.epollWait or
> FileDispatcherImpl.read0. The number of threads found in
> FileDispatcherImpl.read0 correspond to the number of runaway threads expected
> based on looking at 'top' output. These look like:
> {noformat}
> Thread 64758: (state = IN_NATIVE)
> - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int)
> @bci=0 (Compiled frame; information may be imprecise)
> - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int)
> @bci=4, line=39 (Compiled frame)
> - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor,
> java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223
> (Compil
> ed frame)
> - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long,
> sun.nio.ch.NativeDispatcher) @bci=48, line=197 (Compiled frame)
> - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379
> (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel,
> java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess()
> @bci=509, line=1488 (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey)
> @bci=23, line=790 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97,
> line=581 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1,
> line=556 (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1145 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11277) RPCServer threads can wedge under high load
[
https://issues.apache.org/jira/browse/HBASE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014853#comment-14014853
]
Hudson commented on HBASE-11277:
FAILURE: Integrated in hbase-0.96 #404 (See
[https://builds.apache.org/job/hbase-0.96/404/])
HBASE-11277 RpcServer threads can wedge under high load (apurtell: rev
0c374ceea9374c4743a427c7029335a975ad3f14)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
> RPCServer threads can wedge under high load
> ---
>
> Key: HBASE-11277
> URL: https://issues.apache.org/jira/browse/HBASE-11277
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.2, 0.98.3
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.99.0, 0.96.3, 0.98.3
>
> Attachments: HBASE-11277.patch
>
>
> This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load,
> RPCServer threads can wedge, fail to make progess, and consume 100% CPU time
> on a core indefinitely.
> Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE
> threads are mostly in EPollArrayWrapper.epollWait or
> FileDispatcherImpl.read0. The number of threads found in
> FileDispatcherImpl.read0 correspond to the number of runaway threads expected
> based on looking at 'top' output. These look like:
> {noformat}
> Thread 64758: (state = IN_NATIVE)
> - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int)
> @bci=0 (Compiled frame; information may be imprecise)
> - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int)
> @bci=4, line=39 (Compiled frame)
> - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor,
> java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223
> (Compil
> ed frame)
> - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long,
> sun.nio.ch.NativeDispatcher) @bci=48, line=197 (Compiled frame)
> - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379
> (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel,
> java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess()
> @bci=509, line=1488 (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey)
> @bci=23, line=790 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97,
> line=581 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1,
> line=556 (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1145 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10692) The Multi TableMap job don't support the security HBase cluster
[ https://issues.apache.org/jira/browse/HBASE-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10692: --- Fix Version/s: (was: 0.98.4) 0.98.3 > The Multi TableMap job don't support the security HBase cluster > --- > > Key: HBASE-10692 > URL: https://issues.apache.org/jira/browse/HBASE-10692 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 0.99.0, 0.98.3, 0.94.21 > > Attachments: HBASE-10692-0.94-v1.diff, HBASE-10692-trunk-v1.diff, > HBASE-10692-trunk-v2.diff > > > HBASE-3996 adds the support of multiple tables and scanners as input to the > mapper in map/reduce jobs. But it don't support the security HBase cluster. > [~erank] [~bbaugher] > Ps: HBASE-3996 only support multiple tables from the same HBase cluster. > Should we support multiple tables from the different clusters? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11220) Add listeners to ServerManager and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-11220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11220: --- Fix Version/s: (was: 0.98.4) 0.98.3 > Add listeners to ServerManager and AssignmentManager > > > Key: HBASE-11220 > URL: https://issues.apache.org/jira/browse/HBASE-11220 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 0.99.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.99.0, 0.98.3 > > Attachments: HBASE-11220-v0.patch, HBASE-11220-v1.patch > > > Add support for listeners to ServerManager and AssignmentManager. > This will allows to get notified about servers added/removed or regions > added/removed/moved. > I'm planning to use this in the MasterProcedureManager. Since we are starting > using the Procedures for distributed operations, we must add support for RS > joining or Regions moving. At the moment the operation on the "moving" set of > RSs is "lost". -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11255) Negative request num in region load
[
https://issues.apache.org/jira/browse/HBASE-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-11255:
---
Fix Version/s: (was: 0.98.4)
0.98.3
> Negative request num in region load
> ---
>
> Key: HBASE-11255
> URL: https://issues.apache.org/jira/browse/HBASE-11255
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.99.0
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 0.99.0, 0.96.3, 0.98.3
>
> Attachments: HBASE-11255-trunk-v1.diff
>
>
> We found that the request number of region is negative in long-running hbase
> cluster.
> The is because of improper cast in HRegionServer#createRegionLoad
> {code}
> ...
> .setReadRequestsCount((int)r.readRequestsCount.get())
> .setWriteRequestsCount((int) r.writeRequestsCount.get())
> {code}
> The patch is simple and just removes the cast.
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11259) Compression.java different compressions load system classpath differently causing errors
[ https://issues.apache.org/jira/browse/HBASE-11259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11259: --- Fix Version/s: (was: 0.98.4) 0.98.3 > Compression.java different compressions load system classpath differently > causing errors > > > Key: HBASE-11259 > URL: https://issues.apache.org/jira/browse/HBASE-11259 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.99.0 >Reporter: Enoch Hsu >Assignee: Enoch Hsu >Priority: Minor > Fix For: 0.99.0, 0.98.3 > > Attachments: HBASE-11259.patch > > > I stumbled upon this issue when testing Snappy compression on hbase tables > with a webserver. > On the webserver the system classpath of the server JVM did not include the > path to hadoop-common.jar and since it is hardcoded to only retrieve the > system classpath it ran into a NoClassDefFoundException using hte following > call ClassLoader.getSystemClassLoader() > However LZ4 compression works using this private method > getClassLoaderForCodec() which attempts to load the threads classpath first > and the system classpath last. > I propose to change all the ClassLoader.getSystemClassLoader() calls to > getClassLoaderForCodec() to allow for consistent behavior in loading > compression classes and to also make it easier for users to make sure they do > not run into RuntimeExceptions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11272) Backport HBASE-11218 (Data loss in HBase standalone mode) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11272: --- Fix Version/s: (was: 0.98.4) 0.98.3 > Backport HBASE-11218 (Data loss in HBase standalone mode) to 0.98 > - > > Key: HBASE-11272 > URL: https://issues.apache.org/jira/browse/HBASE-11272 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.98.3 > > Attachments: HBASE-11272-0.98.patch > > > Consider backporting the fix on HBASE-11218 to 0.98. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11277) RPCServer threads can wedge under high load
[
https://issues.apache.org/jira/browse/HBASE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-11277:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to 0.96+. Thanks for the reviews!
> RPCServer threads can wedge under high load
> ---
>
> Key: HBASE-11277
> URL: https://issues.apache.org/jira/browse/HBASE-11277
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.2, 0.98.3
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.99.0, 0.96.3, 0.98.3
>
> Attachments: HBASE-11277.patch
>
>
> This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load,
> RPCServer threads can wedge, fail to make progess, and consume 100% CPU time
> on a core indefinitely.
> Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE
> threads are mostly in EPollArrayWrapper.epollWait or
> FileDispatcherImpl.read0. The number of threads found in
> FileDispatcherImpl.read0 correspond to the number of runaway threads expected
> based on looking at 'top' output. These look like:
> {noformat}
> Thread 64758: (state = IN_NATIVE)
> - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int)
> @bci=0 (Compiled frame; information may be imprecise)
> - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int)
> @bci=4, line=39 (Compiled frame)
> - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor,
> java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223
> (Compil
> ed frame)
> - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long,
> sun.nio.ch.NativeDispatcher) @bci=48, line=197 (Compiled frame)
> - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379
> (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel,
> java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess()
> @bci=509, line=1488 (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey)
> @bci=23, line=790 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97,
> line=581 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1,
> line=556 (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1145 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11275) [AccessController] postCreateTable hook fails when another CP creates table on their startup
[
https://issues.apache.org/jira/browse/HBASE-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-11275:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to master and 0.98
> [AccessController] postCreateTable hook fails when another CP creates table
> on their startup
>
>
> Key: HBASE-11275
> URL: https://issues.apache.org/jira/browse/HBASE-11275
> Project: HBase
> Issue Type: Bug
> Components: security
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0, 0.98.3
>
> Attachments: HBASE-11275.patch
>
>
> I am using AC and VC together.( In the required order with AC first)
> Still I am getting the below Exception
> {code}
> 2014-05-29 14:10:55,574 ERROR [ActiveMasterManager] client.AsyncProcess(313):
> Failed to get region location
> org.apache.hadoop.hbase.TableNotFoundException: hbase:acl
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1056)
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:995)
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:952)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:380)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:309)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:263)
>at
> org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1022)
>at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>at
> org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>at
> org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>at
> org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:1390)
>at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:619)
>at org.apache.hadoop.hbase.master.HMaster.access$3(HMaster.java:462)
>at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1258)
>at java.lang.Thread.run(Thread.java:745)
> 2014-05-29 14:10:55,587 ERROR [ActiveMasterManager] master.HMaster(621):
> Coprocessor postStartMaster() hook failed
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: hbase:acl: 1 time,
>at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:180)
>at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1(AsyncProcess.java:179)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1024)
>at
> org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1025)
>at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>at
> org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>at
> org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>at
> org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:1390)
>at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:619)
>at org.apache.hadoop.hbase.master.HMaster.access$3(HMaster.java:462)
>at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1258)
>at java.lang.Thread.run(Thread.java:745)
> {code}
> During the master start, the postStartMaster on the AC is called 1st and it
> creates acl table. Th
[jira] [Commented] (HBASE-11277) RPCServer threads can wedge under high load
[
https://issues.apache.org/jira/browse/HBASE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014727#comment-14014727
]
Andrew Purtell commented on HBASE-11277:
I am going to commit this in a few hours to roll a new RC, unless objection.
> RPCServer threads can wedge under high load
> ---
>
> Key: HBASE-11277
> URL: https://issues.apache.org/jira/browse/HBASE-11277
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.2, 0.98.3
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.99.0, 0.96.3, 0.98.3
>
> Attachments: HBASE-11277.patch
>
>
> This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load,
> RPCServer threads can wedge, fail to make progess, and consume 100% CPU time
> on a core indefinitely.
> Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE
> threads are mostly in EPollArrayWrapper.epollWait or
> FileDispatcherImpl.read0. The number of threads found in
> FileDispatcherImpl.read0 correspond to the number of runaway threads expected
> based on looking at 'top' output. These look like:
> {noformat}
> Thread 64758: (state = IN_NATIVE)
> - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int)
> @bci=0 (Compiled frame; information may be imprecise)
> - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int)
> @bci=4, line=39 (Compiled frame)
> - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor,
> java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223
> (Compil
> ed frame)
> - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long,
> sun.nio.ch.NativeDispatcher) @bci=48, line=197 (Compiled frame)
> - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379
> (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel,
> java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess()
> @bci=509, line=1488 (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey)
> @bci=23, line=790 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97,
> line=581 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1,
> line=556 (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1145 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11275) [AccessController] postCreateTable hook fails when another CP creates table on their startup
[
https://issues.apache.org/jira/browse/HBASE-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014726#comment-14014726
]
Andrew Purtell commented on HBASE-11275:
I am going to commit this in a few hours to roll a new RC, unless objection. We
can update the test later.
> [AccessController] postCreateTable hook fails when another CP creates table
> on their startup
>
>
> Key: HBASE-11275
> URL: https://issues.apache.org/jira/browse/HBASE-11275
> Project: HBase
> Issue Type: Bug
> Components: security
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0, 0.98.3
>
> Attachments: HBASE-11275.patch
>
>
> I am using AC and VC together.( In the required order with AC first)
> Still I am getting the below Exception
> {code}
> 2014-05-29 14:10:55,574 ERROR [ActiveMasterManager] client.AsyncProcess(313):
> Failed to get region location
> org.apache.hadoop.hbase.TableNotFoundException: hbase:acl
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1056)
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:995)
>at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:952)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:380)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:309)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:263)
>at
> org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1022)
>at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>at
> org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>at
> org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>at
> org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:1390)
>at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:619)
>at org.apache.hadoop.hbase.master.HMaster.access$3(HMaster.java:462)
>at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1258)
>at java.lang.Thread.run(Thread.java:745)
> 2014-05-29 14:10:55,587 ERROR [ActiveMasterManager] master.HMaster(621):
> Coprocessor postStartMaster() hook failed
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: hbase:acl: 1 time,
>at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:180)
>at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1(AsyncProcess.java:179)
>at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1024)
>at
> org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1025)
>at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1357)
>at org.apache.hadoop.hbase.client.HTable.put(HTable.java:957)
>at
> org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
>at
> org.apache.hadoop.hbase.security.access.AccessController.postCreateTable(AccessController.java:867)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:258)
>at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1123)
>at
> org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:231)
>at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:1390)
>at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:619)
>at org.apache.hadoop.hbase.master.HMaster.access$3(HMaster.java:462)
>at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1258)
>at java.lang.Thread.run(Thread.java:745)
> {code}
> During the master start, the postStartMaster on the AC is called 1st
[jira] [Commented] (HBASE-11277) RPCServer threads can wedge under high load
[
https://issues.apache.org/jira/browse/HBASE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014719#comment-14014719
]
Andrew Purtell commented on HBASE-11277:
Agreed with your points [~lhofhansl]. The other side is not disconnected and
there is space in the buffer. My understanding of the code after this change is
now in the data phase as long as we have read > 0 we will loop back to do
another immediate read call, but otherwise we will exit handling for the
connection and move on. We won't come back to the connection again until it is
selectable for reading again. In this sense I think the fix is correct. Without
the fix when a channel is selectable we go to read from it, and spin on reads
until the entire message is transmitted no matter how many times we spin in a
hot loop. With the fix when a channel is selectable we go to read from it but
then set it aside when read returns 0 for useful work.
> RPCServer threads can wedge under high load
> ---
>
> Key: HBASE-11277
> URL: https://issues.apache.org/jira/browse/HBASE-11277
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.2, 0.98.3
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.99.0, 0.96.3, 0.98.3
>
> Attachments: HBASE-11277.patch
>
>
> This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load,
> RPCServer threads can wedge, fail to make progess, and consume 100% CPU time
> on a core indefinitely.
> Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE
> threads are mostly in EPollArrayWrapper.epollWait or
> FileDispatcherImpl.read0. The number of threads found in
> FileDispatcherImpl.read0 correspond to the number of runaway threads expected
> based on looking at 'top' output. These look like:
> {noformat}
> Thread 64758: (state = IN_NATIVE)
> - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int)
> @bci=0 (Compiled frame; information may be imprecise)
> - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int)
> @bci=4, line=39 (Compiled frame)
> - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor,
> java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223
> (Compil
> ed frame)
> - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long,
> sun.nio.ch.NativeDispatcher) @bci=48, line=197 (Compiled frame)
> - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379
> (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel,
> java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess()
> @bci=509, line=1488 (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey)
> @bci=23, line=790 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97,
> line=581 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1,
> line=556 (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1145 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11278) Port HBASE-11201 'Enable global procedure members to return values to procedure master' to 0.98
Ted Yu created HBASE-11278: -- Summary: Port HBASE-11201 'Enable global procedure members to return values to procedure master' to 0.98 Key: HBASE-11278 URL: https://issues.apache.org/jira/browse/HBASE-11278 Project: HBase Issue Type: Task Reporter: Ted Yu HBASE-11201 adds the capability for procedure members to return data back to the master. This issue is to backport the feature to 0.98 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11246) Backport HBASE-11234 'FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result'
[ https://issues.apache.org/jira/browse/HBASE-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11246: --- Fix Version/s: (was: 0.98.4) > Backport HBASE-11234 'FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong > result' > --- > > Key: HBASE-11246 > URL: https://issues.apache.org/jira/browse/HBASE-11246 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Priority: Critical > > HBASE-11234 fixed bug in FastDiffDeltaEncoder. > This issue is to backport the fix to earlier releases. > For 0.96 / 0.98, the fix should also cover prefix tree encoding. This would > allow TestReversibleScanners#testReversibleStoreFileScanner to iterate > through all the data block encodings. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
[
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-11234:
---
Fix Version/s: (was: 0.98.4)
0.98.3
> FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
>
>
> Key: HBASE-11234
> URL: https://issues.apache.org/jira/browse/HBASE-11234
> Project: HBase
> Issue Type: Bug
>Reporter: chunhui shen
>Assignee: chunhui shen
>Priority: Critical
> Fix For: 0.99.0, 0.96.3, 0.98.3, 0.94.21
>
> Attachments: 11234-94.patch, 11234-96.patch,
> 11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch
>
>
> As Ted found,
> With this change:
> {noformat}
> Index:
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
> ===
> ---
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
>(revision 1596579)
> +++
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
>(working copy)
> @@ -51,6 +51,7 @@
> import org.apache.hadoop.hbase.filter.FilterList.Operator;
> import org.apache.hadoop.hbase.filter.PageFilter;
> import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
> +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
> import org.apache.hadoop.hbase.io.hfile.CacheConfig;
> import org.apache.hadoop.hbase.io.hfile.HFileContext;
> import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
> @@ -90,6 +91,7 @@
> CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
> HFileContextBuilder hcBuilder = new HFileContextBuilder();
> hcBuilder.withBlockSize(2 * 1024);
> +hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
> HFileContext hFileContext = hcBuilder.build();
> StoreFile.Writer writer = new StoreFile.WriterBuilder(
> TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
> {noformat}
> I got:
> java.lang.AssertionError:
> expected:
> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
> at
> org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
> After debugging, it seems the method of
> FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause
> hfilescanner#seekBefore returns wrong result.
> The solution is simple, see the patch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
