Quanlong Huang created IMPALA-9211: -------------------------------------- Summary: CreateTable with sync_ddl may fail with concurrent INVALIDATE METADATA Key: IMPALA-9211 URL: https://issues.apache.org/jira/browse/IMPALA-9211 Project: IMPALA Issue Type: Bug Components: Catalog Reporter: Quanlong Huang Assignee: Quanlong Huang
The execution flow for CreateTable is {code:java} hold metastoreDdlLock_ | create table in HMS | if (success) add Table to catalog cache by calling CatalogServiceCatalog.addTable() |--free metastoreDdlLock_{code} The execution flow of CatalogServiceCatalog.addTable() is {code:java} get db from dbCache init an IncompleteTable for the table hold writeLock of versionLock_ | incr catalog version and assign it to the table | add table to the db |-- free writeLock of versionLock_ {code} The execution flow for INVALIDATE METADATA is (major part in reset()): {code:java} hold writeLock of versionLock_ | create a new dbCache | for each db init a new item in new dbCache | get table names in db from HMS | for each table in the db | init a new table object and put it into the dbCache | replace original dbCache with the new one |-- free writeLock of versionLock_{code} *Buggy scenario:* {code:java} Thread-1(CreateTable): - create table in HMS - get db from dbCache - waiting for writeLock of versionLock_ Thread-2(Invalidate Metadata): - holds the writeLock of versionLock_ - replacing dbCache with a new one. - find the new table in HMS, create an IncompleteTable for it using catalogVersion=900 - add it into the new db object Thread-1(CreateTable): - get the writeLock of versionLock_ - Incr catalog version to 1000 and assign it to the table. - Add the table to the stale db object. - Then wait in waitForSyncDdlVersion() for the update to be sent. Thread-3(catalog-update-gathering): - get current catalog verion (1000) as toVersion. - Collecting updates in version range (800, 1000]. - Sent the created table with catalogVersion=900. Thread-1(CreateTable): - Find an older version (900) is sent. - Keep waiting a verion > 1000 for this table to be sent. Let's say no other modifications on the table happen. It remains catalogVersion=900. Thread-3(catalog-update-gathering): - Collecting updates in version range (1000, 1234]. - Do nothing for the table since its version not in range. {code} The created Table is added into a stale Db object. Reset() op already creates a new Db object. Thread-1 will finally run out of waiting attemps and throw an exception as described inĀ IMPALA-9135. -- This message was sent by Atlassian Jira (v8.3.4#803005)