[
https://issues.apache.org/jira/browse/IMPALA-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868173#comment-16868173
]
Robbie Zhang commented on IMPALA-7093:
--------------------------------------
Hi [~bharathv] , thanks for your comment.
My fix only changes the behavior within one catalog update. The risks are:
1) The ImpaladCatalog keeps the removed table/function objects
2) The ImpaladCatalog doesn't replace stale table/function objects
For 1), I think it's alright unless for some reason the catalogd doesn't
include the removed table/function objects into deleteLog_. But for 2), I do
find a scenario in which it could happen. For example, when the catalogd is
restarted while impala daemons are running, the catalog object versions are
reset and might be lower than the version of objects in impala daemons. It will
definitely break my fix. So I just came up with an idea to improve my fix. It's
not so smart but it should work. I think my fix can be improved as:
a)
[ImpaladCatalog.addCatalogObject|https://github.com/apache/impala/blob/30c3cd95a42cacbfa2dbb0b29a4757745af942c3/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java#L291]
removes existing table object first then adds it back, just as what we do for
function objects;
b) ImpaladCatalog.addCatalogObject adds the name of all updated
table/function objects to a list or map, and
[ImpaladCatalog.updateCatalog|https://github.com/apache/impala/blob/30c3cd95a42cacbfa2dbb0b29a4757745af942c3/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java#L226]
removes all table/function object which are not in the list or map.
With the above improvement, I believe the fix has no side effect as long as
the performance is acceptable. How do you reckon?
> Tables briefly appear to not exist after INVALIDATE METADATA or catalog
> restart
> -------------------------------------------------------------------------------
>
> Key: IMPALA-7093
> URL: https://issues.apache.org/jira/browse/IMPALA-7093
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.12.0, Impala 2.13.0
> Reporter: Todd Lipcon
> Priority: Major
>
> I'm doing some stress testing of Impala 2.13 (recent snapshot build) and hit
> the following sequence:
> {code}
> {"query": "SHOW TABLES in consistency_test", "type": "call", "id": 3}
> {"type": "response", "id": 3, "results": [["t1"]]}
> {"query": "INVALIDATE METADATA", "type": "call", "id": 7}
> {"type": "response", "id": 7}
> {"query": "DESCRIBE consistency_test.t1", "type": "call", "id": 9}
> {"type": "response", "id": 9, "error": "AnalysisException: Could not resolve
> path: 'consistency_test.t1'\n"}
> {code}
> i.e. 'SHOW TABLES' shows that a table exists, but then shortly after an
> INVALIDATE METADATA, an attempt to describe a table indicates that the table
> does not exist. This is a single-threaded test case against a single impalad.
> I also saw a similar behavior that issuing queries to an impalad shortly
> after a catalogd restart could transiently show tables not existing that in
> fact exist.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
