Hello Bharath Vissapragada, Philip Zeyliger, Dimitris Tsirogiannis, Vuk Ercegovac,
I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8958 to look at the new patch set (#4). Change subject: IMPALA-5152: Introduce metadata loading phase ...................................................................... IMPALA-5152: Introduce metadata loading phase Reworks the collection and loading of missing metadata when compiling a statement. Introduces a new metadata-loading phase between parsing and analysis. Summary of the new compilation flow: 1. Parse statement. 2. Collect all table references from the parsed statement and generate a list of tables that need to be loaded for analysis to succeed. 3. Request missing metadata and wait for it to arrive. As views become loaded we expand the set of required tables based on the view definitions. This step populates a statement-local table cache that contains all loaded tables relevant to the statement. 4. Create a new Analyzer with the table cache and analyze the statement. During analysis only the table cache is consulted for table metadata, the ImpaladCatalog is not used for that purpose anymore. 5. Authorize the statement. 6. Plan generation as usual. The intent of the existing code was to collect all tables missing metadata during analysis, load the metadata, and then re-analyze the statement (and repeat those steps until all metadata is loaded). Unfortunately, the relevant code was hard-to-follow, subtle and not well tested, and therefore it was broken in several ways over the course of time. For example, the introduction of path analysis for nested types subtly broke the intended behavior, and there are other similar examples. The serial table loading observed in the JIRA was caused by the following code in the resolution of table references: for (all path interpretations) { try { // Try to resolve the path; might call getTable() which // throws for nonexistent tables. } catch (AnalysisException e) { if (analyzer.hasMissingTbls()) throw e; } } The following example illustrates the problem: SELECT * FROM a.b, x.y When resolving the path "a.b" we consider that "a" could be a database or a table. Similarly, "b" could be a table or a nested collection. If the path resolution for "a.b" adds a missing table entry, then the path resolution for "x.y" could exit prematurely, without trying the other path interpretations that would lead to adding the expected missing table. So effectively, the tables end up being loaded one-by-one. Testing: - A core/hdfs run succeeded - No new tests were added because the existing functional tests provide good coverage of various metadata loading scenarios. - The issue reported in IMPALA-5152 is basically impossible now. Adding FE unit tests for that bug specifically would require ugly changes to the new code to enable such testing. Change-Id: I68d32d5acd4a6f6bc6cedb05e6cc5cf604d24a55 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/AuthorizationStmt.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateFunctionStmtBase.java M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DescribeDbStmt.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DropStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LimitElement.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SetStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateFunctionStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowFunctionsStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowGrantRoleStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowTablesStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java A fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/common/AnalysisException.java M fe/src/main/java/org/apache/impala/common/ImpalaException.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/PlannerContext.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/main/java/org/apache/impala/util/EventSequence.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeModifyStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeUpsertStmtTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ExprNdvTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java A fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java 69 files changed, 1,480 insertions(+), 957 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/8958/4 -- To view, visit http://gerrit.cloudera.org:8080/8958 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I68d32d5acd4a6f6bc6cedb05e6cc5cf604d24a55 Gerrit-Change-Number: 8958 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com> Gerrit-Reviewer: Philip Zeyliger <phi...@cloudera.com> Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com>