[jira] [Updated] (HBASE-9321) Contention getting the current user in RpcClient$Connection.writeRequest
[
https://issues.apache.org/jira/browse/HBASE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9321:
---
Attachment: trunk-9321_v4.patch
Attached v4, fixed TestScannerResource
Contention getting the current user in RpcClient$Connection.writeRequest
Key: HBASE-9321
URL: https://issues.apache.org/jira/browse/HBASE-9321
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.2
Reporter: Jean-Daniel Cryans
Assignee: Jimmy Xiang
Priority: Critical
Fix For: 0.98.0, 0.96.0
Attachments: trunk-9321.patch, trunk-9321_v2.patch,
trunk-9321_v3.patch, trunk-9321_v4.patch
I've been running tests on clusters with lots of regions, about 400, and
I'm seeing weird contention in the client.
This one I see a lot, hundreds and sometimes thousands of threads are blocked
like this:
{noformat}
htable-pool4-t74 daemon prio=10 tid=0x7f2254114000 nid=0x2a99 waiting
for monitor entry [0x7f21f9e94000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- waiting to lock 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
While the holder is doing this:
{noformat}
htable-pool17-t55 daemon prio=10 tid=0x7f2244408000 nid=0x2a98 runnable
[0x7f21f9f95000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.getStackAccessControlContext(Native
Method)
at java.security.AccessController.getContext(AccessController.java:487)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- locked 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9321) Contention getting the current user in RpcClient$Connection.writeRequest
[
https://issues.apache.org/jira/browse/HBASE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9321:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I checked in patch v4. Thanks.
Contention getting the current user in RpcClient$Connection.writeRequest
Key: HBASE-9321
URL: https://issues.apache.org/jira/browse/HBASE-9321
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.2
Reporter: Jean-Daniel Cryans
Assignee: Jimmy Xiang
Priority: Critical
Fix For: 0.98.0, 0.96.0
Attachments: trunk-9321.patch, trunk-9321_v2.patch,
trunk-9321_v3.patch, trunk-9321_v4.patch
I've been running tests on clusters with lots of regions, about 400, and
I'm seeing weird contention in the client.
This one I see a lot, hundreds and sometimes thousands of threads are blocked
like this:
{noformat}
htable-pool4-t74 daemon prio=10 tid=0x7f2254114000 nid=0x2a99 waiting
for monitor entry [0x7f21f9e94000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- waiting to lock 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
While the holder is doing this:
{noformat}
htable-pool17-t55 daemon prio=10 tid=0x7f2244408000 nid=0x2a98 runnable
[0x7f21f9f95000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.getStackAccessControlContext(Native
Method)
at java.security.AccessController.getContext(AccessController.java:487)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- locked 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9321) Contention getting the current user in RpcClient$Connection.writeRequest
[
https://issues.apache.org/jira/browse/HBASE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9321:
---
Status: Patch Available (was: Open)
Contention getting the current user in RpcClient$Connection.writeRequest
Key: HBASE-9321
URL: https://issues.apache.org/jira/browse/HBASE-9321
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.2
Reporter: Jean-Daniel Cryans
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.0
Attachments: trunk-9321.patch, trunk-9321_v2.patch
I've been running tests on clusters with lots of regions, about 400, and
I'm seeing weird contention in the client.
This one I see a lot, hundreds and sometimes thousands of threads are blocked
like this:
{noformat}
htable-pool4-t74 daemon prio=10 tid=0x7f2254114000 nid=0x2a99 waiting
for monitor entry [0x7f21f9e94000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- waiting to lock 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
While the holder is doing this:
{noformat}
htable-pool17-t55 daemon prio=10 tid=0x7f2244408000 nid=0x2a98 runnable
[0x7f21f9f95000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.getStackAccessControlContext(Native
Method)
at java.security.AccessController.getContext(AccessController.java:487)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- locked 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9321) Contention getting the current user in RpcClient$Connection.writeRequest
[
https://issues.apache.org/jira/browse/HBASE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9321:
---
Attachment: trunk-9321_v2.patch
Attached patch v2, which removed connection sharing among users, created a
connection per user, connections are cached, closed/removed if idle for some
time (default 10 minutes) to limit the # of connection leftover.
Tested with a non-secure cluster.
Contention getting the current user in RpcClient$Connection.writeRequest
Key: HBASE-9321
URL: https://issues.apache.org/jira/browse/HBASE-9321
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.2
Reporter: Jean-Daniel Cryans
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.0
Attachments: trunk-9321.patch, trunk-9321_v2.patch
I've been running tests on clusters with lots of regions, about 400, and
I'm seeing weird contention in the client.
This one I see a lot, hundreds and sometimes thousands of threads are blocked
like this:
{noformat}
htable-pool4-t74 daemon prio=10 tid=0x7f2254114000 nid=0x2a99 waiting
for monitor entry [0x7f21f9e94000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- waiting to lock 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
While the holder is doing this:
{noformat}
htable-pool17-t55 daemon prio=10 tid=0x7f2244408000 nid=0x2a98 runnable
[0x7f21f9f95000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.getStackAccessControlContext(Native
Method)
at java.security.AccessController.getContext(AccessController.java:487)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- locked 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9321) Contention getting the current user in RpcClient$Connection.writeRequest
[
https://issues.apache.org/jira/browse/HBASE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-9321:
-
Priority: Critical (was: Major)
Contention getting the current user in RpcClient$Connection.writeRequest
Key: HBASE-9321
URL: https://issues.apache.org/jira/browse/HBASE-9321
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.2
Reporter: Jean-Daniel Cryans
Assignee: Jimmy Xiang
Priority: Critical
Fix For: 0.98.0, 0.96.0
Attachments: trunk-9321.patch, trunk-9321_v2.patch
I've been running tests on clusters with lots of regions, about 400, and
I'm seeing weird contention in the client.
This one I see a lot, hundreds and sometimes thousands of threads are blocked
like this:
{noformat}
htable-pool4-t74 daemon prio=10 tid=0x7f2254114000 nid=0x2a99 waiting
for monitor entry [0x7f21f9e94000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- waiting to lock 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
While the holder is doing this:
{noformat}
htable-pool17-t55 daemon prio=10 tid=0x7f2244408000 nid=0x2a98 runnable
[0x7f21f9f95000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.getStackAccessControlContext(Native
Method)
at java.security.AccessController.getContext(AccessController.java:487)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- locked 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9321) Contention getting the current user in RpcClient$Connection.writeRequest
[
https://issues.apache.org/jira/browse/HBASE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9321:
---
Attachment: trunk-9321_v3.patch
Attached v3 that fixed the unit test.
Contention getting the current user in RpcClient$Connection.writeRequest
Key: HBASE-9321
URL: https://issues.apache.org/jira/browse/HBASE-9321
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.2
Reporter: Jean-Daniel Cryans
Assignee: Jimmy Xiang
Priority: Critical
Fix For: 0.98.0, 0.96.0
Attachments: trunk-9321.patch, trunk-9321_v2.patch,
trunk-9321_v3.patch
I've been running tests on clusters with lots of regions, about 400, and
I'm seeing weird contention in the client.
This one I see a lot, hundreds and sometimes thousands of threads are blocked
like this:
{noformat}
htable-pool4-t74 daemon prio=10 tid=0x7f2254114000 nid=0x2a99 waiting
for monitor entry [0x7f21f9e94000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- waiting to lock 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
While the holder is doing this:
{noformat}
htable-pool17-t55 daemon prio=10 tid=0x7f2244408000 nid=0x2a98 runnable
[0x7f21f9f95000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.getStackAccessControlContext(Native
Method)
at java.security.AccessController.getContext(AccessController.java:487)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- locked 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9321) Contention getting the current user in RpcClient$Connection.writeRequest
[
https://issues.apache.org/jira/browse/HBASE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9321:
---
Assignee: Jimmy Xiang
Status: Open (was: Patch Available)
Contention getting the current user in RpcClient$Connection.writeRequest
Key: HBASE-9321
URL: https://issues.apache.org/jira/browse/HBASE-9321
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.2
Reporter: Jean-Daniel Cryans
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.0
Attachments: trunk-9321.patch
I've been running tests on clusters with lots of regions, about 400, and
I'm seeing weird contention in the client.
This one I see a lot, hundreds and sometimes thousands of threads are blocked
like this:
{noformat}
htable-pool4-t74 daemon prio=10 tid=0x7f2254114000 nid=0x2a99 waiting
for monitor entry [0x7f21f9e94000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- waiting to lock 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
While the holder is doing this:
{noformat}
htable-pool17-t55 daemon prio=10 tid=0x7f2244408000 nid=0x2a98 runnable
[0x7f21f9f95000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.getStackAccessControlContext(Native
Method)
at java.security.AccessController.getContext(AccessController.java:487)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- locked 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9321) Contention getting the current user in RpcClient$Connection.writeRequest
[
https://issues.apache.org/jira/browse/HBASE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9321:
---
Status: Patch Available (was: Open)
Added a configuration so that we do that only if proxy user support is enabled,
which is required to support impersonation.
Contention getting the current user in RpcClient$Connection.writeRequest
Key: HBASE-9321
URL: https://issues.apache.org/jira/browse/HBASE-9321
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.2
Reporter: Jean-Daniel Cryans
Fix For: 0.98.0, 0.96.0
Attachments: trunk-9321.patch
I've been running tests on clusters with lots of regions, about 400, and
I'm seeing weird contention in the client.
This one I see a lot, hundreds and sometimes thousands of threads are blocked
like this:
{noformat}
htable-pool4-t74 daemon prio=10 tid=0x7f2254114000 nid=0x2a99 waiting
for monitor entry [0x7f21f9e94000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- waiting to lock 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
While the holder is doing this:
{noformat}
htable-pool17-t55 daemon prio=10 tid=0x7f2244408000 nid=0x2a98 runnable
[0x7f21f9f95000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.getStackAccessControlContext(Native
Method)
at java.security.AccessController.getContext(AccessController.java:487)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- locked 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9321) Contention getting the current user in RpcClient$Connection.writeRequest
[
https://issues.apache.org/jira/browse/HBASE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9321:
---
Attachment: trunk-9321.patch
Contention getting the current user in RpcClient$Connection.writeRequest
Key: HBASE-9321
URL: https://issues.apache.org/jira/browse/HBASE-9321
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.2
Reporter: Jean-Daniel Cryans
Fix For: 0.98.0, 0.96.0
Attachments: trunk-9321.patch
I've been running tests on clusters with lots of regions, about 400, and
I'm seeing weird contention in the client.
This one I see a lot, hundreds and sometimes thousands of threads are blocked
like this:
{noformat}
htable-pool4-t74 daemon prio=10 tid=0x7f2254114000 nid=0x2a99 waiting
for monitor entry [0x7f21f9e94000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- waiting to lock 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
While the holder is doing this:
{noformat}
htable-pool17-t55 daemon prio=10 tid=0x7f2244408000 nid=0x2a98 runnable
[0x7f21f9f95000]
java.lang.Thread.State: RUNNABLE
at java.security.AccessController.getStackAccessControlContext(Native
Method)
at java.security.AccessController.getContext(AccessController.java:487)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:466)
- locked 0xfb5ad000 (a java.lang.Class for
org.apache.hadoop.security.UserGroupInformation)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1013)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1407)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1634)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1691)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:27339)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at
org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
