[
https://issues.apache.org/jira/browse/PHOENIX-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17695285#comment-17695285
]
Viraj Jasani edited comment on PHOENIX-6883 at 3/2/23 12:34 AM:
----------------------------------------------------------------
-One of the important considerations here should be for tenant views
CQSI#getTables.-
-We can replace CQSI#getTable from "making RPC call to the server serving
SYSTEM.CATALOG region" to "making RPC call to the server serving data table
regions (let's say first region of the table)". We should be able to do this
for Upserts or Selects queries to base table, but when we need to query Upsert
or Select on tenant view, CQSI#getTable would not know which server to make RPC
call to because it doesn't have info reg base table for the given tenant view
in it's first attempt.-
-For this case, it looks like going to SYSTEM.CATALOG is the only way?-
The improvement to CQSI#getTable is to be considered later.
was (Author: vjasani):
One of the important considerations here should be for tenant views
CQSI#getTables.
We can replace CQSI#getTable from "making RPC call to the server serving
SYSTEM.CATALOG region" to "making RPC call to the server serving data table
regions (let's say first region of the table)". We should be able to do this
for Upserts or Selects queries to base table, but when we need to query Upsert
or Select on tenant view, CQSI#getTable would not know which server to make RPC
call to because it doesn't have info reg base table for the given tenant view
in it's first attempt.
For this case, it looks like going to SYSTEM.CATALOG is the only way?
> Phoenix metadata caching redesign
> ---------------------------------
>
> Key: PHOENIX-6883
> URL: https://issues.apache.org/jira/browse/PHOENIX-6883
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Kadir Ozdemir
> Assignee: Rushabh Shah
> Priority: Major
>
> PHOENIX-6761 improves the client side metadata caching by eliminating the
> separate cache for each connection. This improvement results in memory and
> compute savings since it eliminates copying CQSI level cache every time a
> Phoenix connection is created, and also replaces the inefficient the CQSI
> level cache implementation with Guava Cache from Google.
> Despite this improvement, the overall metadata caching architecture begs for
> redesign. This is because every operation in Phoenix need to make multiple
> RPCs to metadata servers for the SYSTEM.CATALOG table (please see
> PHOENIX-6860) to ensure the latest metadata changes are visible to clients.
> These constant RPCs makes the region servers serving SYSTEM.CATALOG hot spot
> and thus leads to poor performance and availability issues.
> The UPDATE_CACHE_FREQUENCY configuration parameter specifies how frequently
> the client cache is updated. However, setting this parameter to a non-zero
> value results in stale caching. Stale caching can cause data integrity
> issues. For example, if an index table creation is not visible to the client,
> Phoenix would skip updating the index table in the write path. That's why is
> this parameter is typically set to zero. However, this defeats the purpose of
> client side metadata caching.
> The redesign of the metadata caching architecture is to directly address this
> issue by making sure that the client metadata caching is always used (that
> is, UPDATE_CACHE_FREQUENCY is set to NEVER) but still ensures the data
> integrity. This is achieved by three main changes.
> The first change is to introduce server side metadata caching in all region
> servers. Currently, the server side metadata caching is used on the region
> servers serving SYSTEM.CATALOG. This metadata caching should be strongly
> consistent such that the metadata updates should include invalidating the
> corresponding entries on the server side caches. This would ensure the server
> cache would not become stale.
> The second change is that the Phoenix client passes the LAST_DDL_TIMESTAMP
> table attribute along with scan and mutation operations to the server regions
> (more accurately to the Phoenix coprocessors). Then the Phoenix coprocessors
> would check the timestamp on a given operation against with the timestamp in
> its server side cache to validate that the client did not use stale metadata
> when it prepared the operation. If the client did use stale metadata then the
> coprocessor would return an exception (this exception can be called
> StaleClientMetadataCacheException) to the client.
> The third change is that upon receiving StaleClientMetadataCacheException the
> Phoenix client makes an RPC call to the metadata server to update the client
> cache, reconstruct the operation with the updated cached, and retry the
> operation.
> This redesign would require updating client and server metadata caches only
> when metadata is stale instead of updating the client metadata cache for each
> (scan or mutation) operation. This would eliminate hot spotting on the
> metadata servers and thus poor performance and availability issues caused by
> this hot spotting.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
