[
https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488564#comment-15488564
]
James Taylor commented on PHOENIX-3072:
---------------------------------------
Thanks for getting back to this, [~enis] and for the explanations.
+1 with one minor typo in comment (remote -> remove):
{code}
+ /**
+ * The priority property for an hbase table. This is already in HTD, but
older versions of
+ * HBase do not have this, so we re-defined it here. Once Phoenix is
HBase-1.3+, we can remote.
+ */
+ public static final String PRIORITY = "PRIORITY";
{code}
bq. We should however clean up the code base. It has a lot of whitespace and
indentation issues already.
Agreed. I try to clean this up opportunistically too. One way to do this is to
make the whitespace cleanups, but then produce a patch that ignores whitespace
changes (to make it easier to review).
bq. I've thought about this, but it seems dangerous to alter the existing
tables when Phoenix is upgraded.
How about if we commit the patch as-is to 4.8 branches (since we don't want
upgrade code for patch releases) and then add the upgrade code for 4.9? We do
do these kinds of things at upgrade time - for example for 4.8, we dropped all
local indexes and re-create them asynchronously. You can do an {{ALTER TABLE
<table name> SET PRIORITY=<priority>}} for all non local indexes. This will do
an online change to the HBase metadata (unless the HBase config for this has
been overridden) so it should be ok.
> Deadlock on region opening with secondary index recovery
> --------------------------------------------------------
>
> Key: PHOENIX-3072
> URL: https://issues.apache.org/jira/browse/PHOENIX-3072
> Project: Phoenix
> Issue Type: Bug
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 4.9.0, 4.8.1
>
> Attachments: phoenix-3072_v1.patch, phoenix-3072_v2.patch
>
>
> There is a distributed deadlock happening in clusters with some moderate
> number of regions for the data tables and secondary index tables and cluster
> and it is cluster restart or some large failure. We have seen this in a
> couple of production cases already.
> Opening of regions in hbase is performed by a thread pool with 3 threads by
> default. Every regionserver can open 3 regions at a time. However, opening
> data table regions has to write to multiple index regions during WAL
> recovery. All other region open requests are queued up in a single queue.
> This causes a deadlock, since the secondary index regions are also opened by
> the same thread pools that we do the work. So if there is greater number of
> data table regions then available number of region opening threads from
> regionservers, the secondary index region open requests just wait to be
> processed in the queue. Since these index regions are not open, the region
> opening of data table regions just block the region opening threads for a
> long time.
> One proposed fix is to use a different thread pool for opening regions of the
> secondary index tables so that we will not deadlock. See HBASE-16095 for the
> HBase-level fix. In Phoenix, we just have to set the priority for secondary
> index tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
