[
https://issues.apache.org/jira/browse/IMPALA-7961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758860#comment-16758860
]
bharath v commented on IMPALA-7961:
-----------------------------------
After digging into this a bit more, the issue here seems to be a race in "IF
NOT EXISTS" + sync_ddl=true combination. For ex: "create table if not exists
foo(..)" and the *table foo already exists* in the catalog server.
In such a case we do the following:
{noformat}
Table existingTbl = catalog_.getTableNoThrow(tableName.getDb(),
tableName.getTbl());
if (params.if_not_exists && existingTbl != null) {
addSummary(response, "Table already exists.");
LOG.trace(String.format("Skipping table creation because %s already
exists and " +
"IF NOT EXISTS was specified.", tableName));
existingTbl.getLock().lock();
try {
addTableToCatalogUpdate(existingTbl, response.getResult());
return false;
} finally {
existingTbl.getLock().unlock();
}
}
{noformat}
We add the *{{existingTbl}}* to the DDL response and wait on a topic update
covering its version.
{noformat}
if (req.isSync_ddl()) {
resp.getResult().setVersion(catalog_.waitForSyncDdlVersion(resp.getResult()));
}
public long waitForSyncDdlVersion(TCatalogUpdateResult result) throws
CatalogException {
.................
long topicVersionForUpdates =
getCoveringTopicUpdateVersion(result.getUpdated_catalog_objects());
..................
{noformat}
The catch here is that since this "existingTbl" could've been unchanged for a
period (no version bumps), it's topic entry could be past
TOPIC_UPDATE_LOG_GC_FREQUENCY and the entry could be potentially GC'ed, in
which case, unless there is a version bump on this table, it wouldn't be added
again to the {{topicUpdateLog_}}. This means that the
{{waitForSyncDdlVersion()}} would loop until it exhausts retries as nothing
would add the table to the log unless modified. I could reproduce this with
aggressive topicUpdate_ GCs using the attached patch.
"create table if not exists" is a specific example and it is quiet possible
that there are other manifestations of this issue with the same theme of trying
to add objects to DDL objects to responses without adding them to topicUpdateLog
> Concurrent catalog heavy workloads can cause queries with SYNC_DDL to fail
> fast
> -------------------------------------------------------------------------------
>
> Key: IMPALA-7961
> URL: https://issues.apache.org/jira/browse/IMPALA-7961
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.12.0, Impala 3.1.0
> Reporter: bharath v
> Assignee: bharath v
> Priority: Critical
>
> When catalog server is under heavy load with concurrent updates to objects,
> queries with SYNC_DDL can fail with the following message.
> *User facing error message:*
> {noformat}
> ERROR: CatalogException: Couldn't retrieve the catalog topic version for the
> SYNC_DDL operation after 3 attempts.The operation has been successfully
> executed but its effects may have not been broadcast to all the coordinators.
> {noformat}
> *Exception from the catalog server log:*
> {noformat}
> I1031 00:00:49.168761 1127039 CatalogServiceCatalog.java:1903] Operation
> using SYNC_DDL is waiting for catalog topic version: 236535. Time to identify
> topic version (msec): 1088
> I1031 00:00:49.168824 1125528 CatalogServiceCatalog.java:1903] Operation
> using SYNC_DDL is waiting for catalog topic version: 236535. Time to identify
> topic version (msec): 12625
> I1031 00:00:49.168851 1131986 jni-util.cc:230]
> org.apache.impala.catalog.CatalogException: Couldn't retrieve the catalog
> topic version for the SYNC_DDL operation after 3 attempts.The operation has
> been successfully executed but its effects may have not been broadcast to all
> the coordinators.
> at
> org.apache.impala.catalog.CatalogServiceCatalog.waitForSyncDdlVersion(CatalogServiceCatalog.java:1891)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:336)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
> ::::
> {noformat}
> *What this means*
> The Catalog operation is actually successful (the change has been committed
> to HMS and Catalog server cache) but the Catalog server noticed that it is
> taking longer than expected time for it to broadcast the changes (for
> whatever reason) and instead of hanging in there, it fails fast. The
> coordinators are expected to eventually sync up in the background.
> *Problem*
> - This violates the contract of the SYNC_DDL query option since the query
> returns early.
> - This is a behavioral regression from pre IMPALA-5058 state where the
> queries would wait forever for SYNC_DDL based changes to propagate.
> *Notes*
> - Usual suspect here is heavily concurrent catalog operations with long
> running DDLs.
> - Introduced by IMPALA-5058
> - My understanding is that this also applies to the Catalog V2 (or
> LocalCatalog mode) since we still rely on the CatalogServer for DDL
> orchestration and hence it takes this codepath.
> Please refer to the jira comment for technical explanation as to why this is
> happening (to be updated).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
