Quanlong Huang created IMPALA-9211:
--------------------------------------
Summary: CreateTable with sync_ddl may fail with concurrent
INVALIDATE METADATA
Key: IMPALA-9211
URL: https://issues.apache.org/jira/browse/IMPALA-9211
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
Assignee: Quanlong Huang
The execution flow for CreateTable is
{code:java}
hold metastoreDdlLock_
| create table in HMS
| if (success) add Table to catalog cache by calling
CatalogServiceCatalog.addTable()
|--free metastoreDdlLock_{code}
The execution flow of CatalogServiceCatalog.addTable() is
{code:java}
get db from dbCache
init an IncompleteTable for the table
hold writeLock of versionLock_
| incr catalog version and assign it to the table
| add table to the db
|-- free writeLock of versionLock_
{code}
The execution flow for INVALIDATE METADATA is (major part in reset()):
{code:java}
hold writeLock of versionLock_
| create a new dbCache
| for each db init a new item in new dbCache
| get table names in db from HMS
| for each table in the db
| init a new table object and put it into the dbCache
| replace original dbCache with the new one
|-- free writeLock of versionLock_{code}
*Buggy scenario:*
{code:java}
Thread-1(CreateTable):
- create table in HMS
- get db from dbCache
- waiting for writeLock of versionLock_
Thread-2(Invalidate Metadata):
- holds the writeLock of versionLock_
- replacing dbCache with a new one.
- find the new table in HMS, create an IncompleteTable for it using
catalogVersion=900
- add it into the new db object
Thread-1(CreateTable):
- get the writeLock of versionLock_
- Incr catalog version to 1000 and assign it to the table.
- Add the table to the stale db object.
- Then wait in waitForSyncDdlVersion() for the update to be sent.
Thread-3(catalog-update-gathering):
- get current catalog verion (1000) as toVersion.
- Collecting updates in version range (800, 1000].
- Sent the created table with catalogVersion=900.
Thread-1(CreateTable):
- Find an older version (900) is sent.
- Keep waiting a verion > 1000 for this table to be sent. Let's say no other
modifications on the table happen. It remains catalogVersion=900.
Thread-3(catalog-update-gathering):
- Collecting updates in version range (1000, 1234].
- Do nothing for the table since its version not in range.
{code}
The created Table is added into a stale Db object. Reset() op already creates a
new Db object. Thread-1 will finally run out of waiting attemps and throw an
exception as described in IMPALA-9135.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
