[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491791#comment-15491791 ] Hudson commented on PHOENIX-3072: - SUCCESS: Integrated in Jenkins build Phoenix-4.8-HBase-1.2 #23 (See [https://builds.apache.org/job/Phoenix-4.8-HBase-1.2/23/]) PHOENIX-3072 Deadlock on region opening with secondary index recovery (elserj: rev 0b0db61038597661f6c62bd56674755cfea9aaf2) * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/IndexIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: PHOENIX-3072_v3.patch, PHOENIX-3072_v4.patch, > phoenix-3072_v1.patch, phoenix-3072_v2.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491719#comment-15491719 ] Hudson commented on PHOENIX-3072: - FAILURE: Integrated in Jenkins build Phoenix-master #1401 (See [https://builds.apache.org/job/Phoenix-master/1401/]) PHOENIX-3072 Deadlock on region opening with secondary index recovery (elserj: rev 2c2b552cd7c611c63677b0be075065504d579469) * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/QueryConstants.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/index/IndexIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: PHOENIX-3072_v3.patch, PHOENIX-3072_v4.patch, > phoenix-3072_v1.patch, phoenix-3072_v2.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491395#comment-15491395 ] Enis Soztutar commented on PHOENIX-3072: Thanks James, Josh. > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: PHOENIX-3072_v3.patch, PHOENIX-3072_v4.patch, > phoenix-3072_v1.patch, phoenix-3072_v2.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491175#comment-15491175 ] Josh Elser commented on PHOENIX-3072: - bq. I have a moment, let me get this in. This took a bit more time than expected. v4 didn't come back to master cleanly for me. Worked through it though and cherry-pick'ing+testing now > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: PHOENIX-3072_v3.patch, PHOENIX-3072_v4.patch, > phoenix-3072_v1.patch, phoenix-3072_v2.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490623#comment-15490623 ] Josh Elser commented on PHOENIX-3072: - bq. Would you have time to commit this, Enis Soztutar to 4.8 and 4.x branches? I have a moment, let me get this in. > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: PHOENIX-3072_v3.patch, PHOENIX-3072_v4.patch, > phoenix-3072_v1.patch, phoenix-3072_v2.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[
https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15489620#comment-15489620
]
Hadoop QA commented on PHOENIX-3072:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12828388/PHOENIX-3072_v4.patch
against master branch at commit c0f72b536684bdbb6f3e597d5d93e767732bd007.
ATTACHMENT ID: 12828388
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
34 warning messages.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+{ false, false, false }, { false, false, true }, { false,
true, false }, { false, true, true },
+{ true, false, false }, { true, false, true }, { true, true,
false }, { true, true, true }
+String ddl ="CREATE TABLE " + fullTableName +
BaseTest.TEST_TABLE_SCHEMA + tableDDLOptions;
+ddl = "CREATE " + (localIndex ? "LOCAL" : "") + " INDEX " +
indexName + " ON " + fullTableName
+String ddl ="CREATE TABLE " + fullTableName +
BaseTest.TEST_TABLE_SCHEMA + tableDDLOptions;
+ddl = "CREATE " + (localIndex ? "LOCAL" : "") + " INDEX " +
indexName + " ON " + fullTableName
+ddl = "CREATE " + (localIndex ? "LOCAL" : "") + " INDEX " +
indexName + " ON " + fullTableName
+String ddl ="CREATE TABLE " + fullTableName +
BaseTest.TEST_TABLE_SCHEMA + tableDDLOptions;
+ddl = "CREATE " + (localIndex ? "LOCAL" : "") + " INDEX " +
indexName + " ON " + fullTableName + " (int_col2)";
+rs = conn.createStatement().executeQuery("SELECT int_col2,
COUNT(*) FROM " + fullTableName + " GROUP BY int_col2");
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/573//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/573//artifact/patchprocess/patchReleaseAuditWarnings.txt
Javadoc warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/573//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/573//console
This message is automatically generated.
> Deadlock on region opening with secondary index recovery
>
>
> Key: PHOENIX-3072
> URL: https://issues.apache.org/jira/browse/PHOENIX-3072
> Project: Phoenix
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 4.9.0, 4.8.1
>
> Attachments: PHOENIX-3072_v3.patch, PHOENIX-3072_v4.patch,
> phoenix-3072_v1.patch, phoenix-3072_v2.patch
>
>
> There is a distributed deadlock happening in clusters with some moderate
> number of regions for the data tables and secondary index tables and cluster
> and it is cluster restart or some large failure. We have seen this in a
> couple of production cases already.
> Opening of regions in hbase is performed by a thread pool with 3 threads by
> default. Every regionserver can open 3 regions at a time. However, opening
> data table regions has to write to multiple index regions during WAL
> recovery. All other region open requests are queued up in a single queue.
> This causes a deadlock, since the secondary index regions are also opened by
> the same thread pools that we do the work. So if there is greater number of
> data table regions then available number of region opening threads from
> regionservers, the secondary index region open requests just wait to be
> processed in the queue. Since these index regions are not open, the region
> opening of data table regions just block the region opening threads for a
> long time.
> One proposed fix is to use a different thread pool for opening regions of the
> secondary index tables so that we will not deadlock. See HBASE-16095 for the
> HBase-level fix. In Phoenix, we just have to set the priority for secondary
> index tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[
https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15489204#comment-15489204
]
Hadoop QA commented on PHOENIX-3072:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12828373/PHOENIX-3072_v3.patch
against master branch at commit c0f72b536684bdbb6f3e597d5d93e767732bd007.
ATTACHMENT ID: 12828373
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
34 warning messages.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/572//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/572//artifact/patchprocess/patchReleaseAuditWarnings.txt
Javadoc warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/572//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/572//console
This message is automatically generated.
> Deadlock on region opening with secondary index recovery
>
>
> Key: PHOENIX-3072
> URL: https://issues.apache.org/jira/browse/PHOENIX-3072
> Project: Phoenix
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 4.9.0, 4.8.1
>
> Attachments: PHOENIX-3072_v3.patch, phoenix-3072_v1.patch,
> phoenix-3072_v2.patch
>
>
> There is a distributed deadlock happening in clusters with some moderate
> number of regions for the data tables and secondary index tables and cluster
> and it is cluster restart or some large failure. We have seen this in a
> couple of production cases already.
> Opening of regions in hbase is performed by a thread pool with 3 threads by
> default. Every regionserver can open 3 regions at a time. However, opening
> data table regions has to write to multiple index regions during WAL
> recovery. All other region open requests are queued up in a single queue.
> This causes a deadlock, since the secondary index regions are also opened by
> the same thread pools that we do the work. So if there is greater number of
> data table regions then available number of region opening threads from
> regionservers, the secondary index region open requests just wait to be
> processed in the queue. Since these index regions are not open, the region
> opening of data table regions just block the region opening threads for a
> long time.
> One proposed fix is to use a different thread pool for opening regions of the
> secondary index tables so that we will not deadlock. See HBASE-16095 for the
> HBase-level fix. In Phoenix, we just have to set the priority for secondary
> index tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488707#comment-15488707 ] James Taylor commented on PHOENIX-3072: --- There's one more case where we don't need this PRIORITY flag and that's for immutable tables. These indexes don't use the IndexedWALEditCodec so they won't have this issue. Let me figure out how you can determine that at that point in the code. > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch, phoenix-3072_v2.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488591#comment-15488591 ] Enis Soztutar commented on PHOENIX-3072: Created PHOENIX-3274 as a follow up. Agreed that we can commit this, and do the upgrade handling there. > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch, phoenix-3072_v2.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[
https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488564#comment-15488564
]
James Taylor commented on PHOENIX-3072:
---
Thanks for getting back to this, [~enis] and for the explanations.
+1 with one minor typo in comment (remote -> remove):
{code}
+/**
+ * The priority property for an hbase table. This is already in HTD, but
older versions of
+ * HBase do not have this, so we re-defined it here. Once Phoenix is
HBase-1.3+, we can remote.
+ */
+public static final String PRIORITY = "PRIORITY";
{code}
bq. We should however clean up the code base. It has a lot of whitespace and
indentation issues already.
Agreed. I try to clean this up opportunistically too. One way to do this is to
make the whitespace cleanups, but then produce a patch that ignores whitespace
changes (to make it easier to review).
bq. I've thought about this, but it seems dangerous to alter the existing
tables when Phoenix is upgraded.
How about if we commit the patch as-is to 4.8 branches (since we don't want
upgrade code for patch releases) and then add the upgrade code for 4.9? We do
do these kinds of things at upgrade time - for example for 4.8, we dropped all
local indexes and re-create them asynchronously. You can do an {{ALTER TABLE
SET PRIORITY=}} for all non local indexes. This will do
an online change to the HBase metadata (unless the HBase config for this has
been overridden) so it should be ok.
> Deadlock on region opening with secondary index recovery
>
>
> Key: PHOENIX-3072
> URL: https://issues.apache.org/jira/browse/PHOENIX-3072
> Project: Phoenix
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 4.9.0, 4.8.1
>
> Attachments: phoenix-3072_v1.patch, phoenix-3072_v2.patch
>
>
> There is a distributed deadlock happening in clusters with some moderate
> number of regions for the data tables and secondary index tables and cluster
> and it is cluster restart or some large failure. We have seen this in a
> couple of production cases already.
> Opening of regions in hbase is performed by a thread pool with 3 threads by
> default. Every regionserver can open 3 regions at a time. However, opening
> data table regions has to write to multiple index regions during WAL
> recovery. All other region open requests are queued up in a single queue.
> This causes a deadlock, since the secondary index regions are also opened by
> the same thread pools that we do the work. So if there is greater number of
> data table regions then available number of region opening threads from
> regionservers, the secondary index region open requests just wait to be
> processed in the queue. Since these index regions are not open, the region
> opening of data table regions just block the region opening threads for a
> long time.
> One proposed fix is to use a different thread pool for opening regions of the
> secondary index tables so that we will not deadlock. See HBASE-16095 for the
> HBase-level fix. In Phoenix, we just have to set the priority for secondary
> index tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488407#comment-15488407 ] Enis Soztutar commented on PHOENIX-3072: bq. It's difficult to tell what's changed with all the whitespace diffs. Can you generate a patch without that? Sure. We should however clean up the code base. It has a lot of whitespace and indentation issues already. bq. It looks like you're setting a new "PRIORITY" attribute on table descriptor for indexes? How/where is this used? (never mind on this - I see it's part of an HBase JIRA). Yep, it is introduced and used via HBASE-16095. bq. How will you handle local indexes since the table descriptor is the same data and index table Local indexes will not have this problem since they are in the same table. There is no inter-dependency between index regions and data table regions. bq. Minor nit: is I suppose you're not using the HBase static constant for "PRIORITY" because this doesn't appear until HBase 1.3? Maybe we should define one in QueryConstants with a comment? Done in v2. bq. Didn't priority get exposed as an attribute on operations now? If so, would that be an alternate implementation mechanism which is a bit more flexible? This is not related to RPCs at all. The deadlock happens at region opening. BTW the patch for per-operation priorities is not committed yet I think on the HBase side. bq. What about existing tables and indexes - I didn't see any upgrade code that sets this for those. If setting priority on operation is an option, that'd get around this. I've thought about this, but it seems dangerous to alter the existing tables when Phoenix is upgraded. That is why there is no upgrade handling. Altering an existing table might have implications on availability, etc. Do we do this kind of alter for other features on Phoenix upgrade? If so we can hook into that. > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch, phoenix-3072_v2.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15480286#comment-15480286 ] James Taylor commented on PHOENIX-3072: --- Ping, [~enis]. We'd like to cut an RC for 4.8.1 next week. Will you have a few spare cycles to complete this one? Maybe [~an...@apache.org] can help? > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475067#comment-15475067 ] James Taylor commented on PHOENIX-3072: --- If we can get in shape for 4.8.1, that'd be great, [~enis]. I agree, it seems important. Some questions/comments: - It's difficult to tell what's changed with all the whitespace diffs. Can you generate a patch without that? - It looks like you're setting a new "PRIORITY" attribute on table descriptor for indexes? How/where is this used? - How will you handle local indexes since the table descriptor is the same data and index table? Should we add it as a column descriptor attribute instead, or would we not know which column families are involved when we're using this info? - Minor nit: is I suppose you're not using the HBase static constant for "PRIORITY" because this doesn't appear until HBase 1.3? Maybe we should define one in QueryConstants with a comment? - Didn't priority get exposed as an attribute on operations now? If so, would that be an alternate implementation mechanism which is a bit more flexible? - What about existing tables and indexes - I didn't see any upgrade code that sets this for those. If setting priority on operation is an option, that'd get around this. > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475049#comment-15475049 ] Andrew Purtell commented on PHOENIX-3072: - Then +1 for not pushing out > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475018#comment-15475018 ] Enis Soztutar commented on PHOENIX-3072: We should commit this for 4.8.1. We have seen this in multiple production clusters already. Although there is a known workaround, it is a hassle and very user un-friendly. > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475015#comment-15475015 ] Enis Soztutar commented on PHOENIX-3072: bq. On the RS, we already make index table updates higher priority than data table updates This happens on the region open, and does not involve the RPC scheduling. In a cluster restart, all of the index and data table regions will be opened by the regionservers. There is only 3 threads that does the opening of regions by default, and for the data tables, the opening of the region blocks on doing the index updates. However, if the index regions are not opened yet, then they will not succeed even if the regionserver RPC scheduling works. The index regions will be waiting on the same "region opening queue" to be opened by the same regionserver. bq. Also, would you mind generating a patch that ignores whitespace changes as it's difficult to find the change you've made. Sorry, the existing code is full with extra whitespace, and my Eclipse settings is to truncate these as a save action. This is to make sure that my patches do not introduce any more extra whitespaces. I can put the patch in RB/github if you want. > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15474982#comment-15474982 ] Lars Hofhansl commented on PHOENIX-3072: [~enis], [~jamestaylor], where are we with this? Push to 4.8.2? > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418049#comment-15418049 ] James Taylor commented on PHOENIX-3072: --- I'm confused by this, [~enis]. On the RS, we already make index table updates higher priority than data table updates. Why doesn't this solve the issue? Also, would you mind generating a patch that ignores whitespace changes as it's difficult to find the change you've made. > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418050#comment-15418050 ] James Taylor commented on PHOENIX-3072: --- I'm confused by this, [~enis]. On the RS, we already make index table updates higher priority than data table updates. Why doesn't this solve the issue? Also, would you mind generating a patch that ignores whitespace changes as it's difficult to find the change you've made. > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387571#comment-15387571 ] Ankit Singhal commented on PHOENIX-3072: [~enis], how about updating the existing SYSTEM and index table descriptor? > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3072) Deadlock on region opening with secondary index recovery
[ https://issues.apache.org/jira/browse/PHOENIX-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380474#comment-15380474 ] Josh Elser commented on PHOENIX-3072: - +1 from me. The approach makes sense and should be low-impact. Cross-referenced your changes here with the changes in HBASE-16095. Nits: lots of whitespace changes which would be good to undo (had to hunt and peck to find the actual changes) and the changes to generateTableDescriptor() and isLocalIndexTable() have incorrect indentation. > Deadlock on region opening with secondary index recovery > > > Key: PHOENIX-3072 > URL: https://issues.apache.org/jira/browse/PHOENIX-3072 > Project: Phoenix > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 4.9.0, 4.8.1 > > Attachments: phoenix-3072_v1.patch > > > There is a distributed deadlock happening in clusters with some moderate > number of regions for the data tables and secondary index tables and cluster > and it is cluster restart or some large failure. We have seen this in a > couple of production cases already. > Opening of regions in hbase is performed by a thread pool with 3 threads by > default. Every regionserver can open 3 regions at a time. However, opening > data table regions has to write to multiple index regions during WAL > recovery. All other region open requests are queued up in a single queue. > This causes a deadlock, since the secondary index regions are also opened by > the same thread pools that we do the work. So if there is greater number of > data table regions then available number of region opening threads from > regionservers, the secondary index region open requests just wait to be > processed in the queue. Since these index regions are not open, the region > opening of data table regions just block the region opening threads for a > long time. > One proposed fix is to use a different thread pool for opening regions of the > secondary index tables so that we will not deadlock. See HBASE-16095 for the > HBase-level fix. In Phoenix, we just have to set the priority for secondary > index tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
