[
https://issues.apache.org/jira/browse/PHOENIX-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16335181#comment-16335181
]
Josh Elser commented on PHOENIX-4537:
-------------------------------------
bq. The upgrade should not happen from a server-side connection. We have code
in place for this already. See QueryUtil.getConnectionOnServer():
[~jamestaylor], again, I think these are two separate things.
System table _migration_ to the SYSTEM schema happens irregardless of whether
it comes from client or server. I have explicitly looked at this code in
ConnectionQueryServicesImpl and confirmed that this is how it works ("intent"
of how it is meant to work aside). We are in agreement that schema migration of
the System tables does not happen on Connections initiated from the server.
bq. If we're going to go this route, can't we prevent this migration in a
similar manner?
This is what Sergey was hinting at, that the real smell is that we should not
be moving System tables into the SYSTEM schema from a server-side connection.
If the consensus is that we don't want to fix this "now" and work towards a fix
that addresses the server-side SYSTEM schema movement, that's fine. However,
this is blocking me at $dayjob and I need a fix for it now.
bq. What about the idea I outlined above of doing PHOENIX-4530 and removing the
clearTsOnDisabledIndexes altogether?
To be honest, I understand the words you've written there, but am at a loss as
to how to go about implementing it :). If we prevent movement of system tables
to the SYSTEM schema by server-side Connections, that lowers the "need" to
implement your suggestion in PHOENIX-4530, right?
> RegionServer initiating compaction can trigger schema migration and deadlock
> the system
> ---------------------------------------------------------------------------------------
>
> Key: PHOENIX-4537
> URL: https://issues.apache.org/jira/browse/PHOENIX-4537
> Project: Phoenix
> Issue Type: Bug
> Reporter: Romil Choksi
> Assignee: Josh Elser
> Priority: Critical
> Fix For: 5.0.0, 4.14.0
>
> Attachments: PHOENIX-4537.001.patch
>
>
> [~sergey.soldatov] has been doing some great digging around a test failure
> we've been seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}}
> and {{phoenix.schema.mapSystemTablesToNamespace=true}} in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run
> before a client first connects
> When the RegionServer initiates the compaction, it will end up running
> {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}} which opens a
> Phoenix connection. While the RegionServer won't upgrade system tables, it
> *will* try to migrate them into the schema mapped variants (e.g.
> SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the
> SYSTEM.CATALOG table. However, the SYSTEM.CATALOG table can't be disabled
> until the region is CLOSED, and the region cannot be CLOSED until the
> compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table
> migrations, but Sergey and [~elserj] both think that this will end badly
> (RegionServers falling over because they expect the tables to be migrated and
> they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
