[
https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851023#comment-15851023
]
William Yang edited comment on PHOENIX-3360 at 2/3/17 4:08 AM:
---------------------------------------------------------------
I don't think that current patch has solved the problem because it is still
setting the RPC class in RegionServer level instead of region level.
We've encountered this problem too, our resolution is simple. The only thing we
need to do is to set specified priority ONLY for index updates that originated
from RS. We can do this in IndexWriterUtils#getDefaultDelegateHTableFactory()
and CoprocessorHTableFactory. Method getDefaultDelegateHTableFactory() creates
an htable factory to get all HTableInterface objects to do index updates in
current region server. If we can make these HTableInterface objects have this
RPC config, then all index update requests will have the right priority and
will not interfere with other RS-RS requests.
Right now, IndexWriterUtils#getDefaultDelegateHTableFactory() is called at each
data table regions init, then each index region has its own
CoprocessorHTableFactory object. We need to do :
* create a global CoprocessorHTableFactory object in
getDefaultDelegateHTableFactory(), thus all data table regions will share the
same CoprocessorHTableFactory.
* in CoprocessorHTableFactory's ctor, create an HBase connection configured
with our RPC controller class.
* in CoprocessorHTableFactory#getTable(), get the table from the connection
created in ctor instead of calling CoprocessorEnvironment#getTable()
The only target here is to make sure all HTableInterface objects for updating
index tables are created from the specified connection. I've done this and it
works fine.
was (Author: yhxx511):
I don't think that current patch has solved the problem because it is still
setting the RPC class in RegionServer level instead of region level.
We've encountered this problem too, our resolution is simple. The only thing we
need to do is to set specified priority ONLY for index updates that originated
from RS. We can do this in IndexWriterUtils#getDefaultDelegateHTableFactory()
and CoprocessorHTableFactory. Method getDefaultDelegateHTableFactory() creates
an htable factory to get all HTableInterface objects to do index updates in
current region server. If we can make these HTableInterface objects have this
RPC config, then all index update requests will have the right priority and
will not interfere with other RS-RS requests.
Right now, IndexWriterUtils#getDefaultDelegateHTableFactory() is called at each
data table regions init, then each index region has its own
CoprocessorHTableFactory object. We need to do :
* create a single HBase connection for current RS and this connection is
configured with this RPC class
* all data table regions get HTableInterface from this connection instead of
calling CoprocessorEnvironment#getTable() before doing a batch mutations to
their index regions.
You can create this connection in getDefaultDelegateHTableFactory() or just let
all regions share the same CoprocessorHTableFactory object. No matter which way
you choose, the only target here is to make sure all HTableInterface objects
for updating index tables are created from the specified connection. In this
case, CoprocessorHTableFactory is no longer a suitable name as we don't get
table from coprocessor env, but from an explicit connection object.
I've done this and it works fine.
> Secondary index configuration is wrong
> --------------------------------------
>
> Key: PHOENIX-3360
> URL: https://issues.apache.org/jira/browse/PHOENIX-3360
> Project: Phoenix
> Issue Type: Bug
> Reporter: Enis Soztutar
> Assignee: Rajeshbabu Chintaguntla
> Priority: Critical
> Fix For: 4.10.0
>
> Attachments: PHOENIX-3360.patch
>
>
> IndexRpcScheduler allocates some handler threads and uses a higher priority
> for RPCs. The corresponding IndexRpcController is not used by default as it
> is, but used through ServerRpcControllerFactory that we configure from Ambari
> by default which sets the priority of the outgoing RPCs to either metadata
> priority, or the index priority.
> However, after reading code of IndexRpcController / ServerRpcController it
> seems that the IndexRPCController DOES NOT look at whether the outgoing RPC
> is for an Index table or not. It just sets ALL rpc priorities to be the index
> priority. The intention seems to be the case that ONLY on servers, we
> configure ServerRpcControllerFactory, and with clients we NEVER configure
> ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We
> configure ServerRpcControllerFactory from Ambari, which in affect makes it so
> that ALL rpcs from Phoenix are only handled by the index handlers by default.
> It means all deadlock cases are still there.
> The documentation in https://phoenix.apache.org/secondary_indexing.html is
> also wrong in this sense. It does not talk about server side / client side.
> Plus this way of configuring different values is not how HBase configuration
> is deployed. We cannot have the configuration show the
> ServerRpcControllerFactory even only for server nodes, because the clients
> running on those nodes will also see the wrong values.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
