[ https://issues.apache.org/jira/browse/IMPALA-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482917#comment-16482917 ]
Pranay Singh edited comment on IMPALA-6994 at 5/21/18 7:04 PM: --------------------------------------------------------------- Checked the code in updatePartitionsFromHms() looks like there is little room of optimization here, skipping CREATE/RECREATE/DROP of a partition will cause inconsistency between HMS and catalog will result in failure of DML statements like INSERT, when used with SYNC_DDL option which requires underlying data and metadata changes to be propagated to all Impala nodes. -Pranay was (Author: pranay_singh): Checked the code in updatePartitionsFromHms() looks like there is little room of optimization here, skipping CREATE/RECREATE/DROP of a partition will cause inconsistency between HMS and catalog this will result in failure of DML statements like INSERT when used with SYNC_DDL option which requires underlying data and metadata changes to be propagated to all Impala nodes. -Pranay > Avoid reloading a table's HMS data for file-only operations > ----------------------------------------------------------- > > Key: IMPALA-6994 > URL: https://issues.apache.org/jira/browse/IMPALA-6994 > Project: IMPALA > Issue Type: Improvement > Components: Catalog > Affects Versions: Impala 2.12.0 > Reporter: Balazs Jeszenszky > Assignee: Pranay Singh > Priority: Major > > Reloading file metadata for HDFS tables (e.g. as a final step in an 'insert') > is done via > https://github.com/apache/impala/blob/branch-2.12.0/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L628 > , which calls > https://github.com/apache/impala/blob/branch-2.12.0/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1243 > HdfsTable.load has no option to only load file metadata. HMS metadata will > also be reloaded every time, which is an unnecessary overhead (and potential > point of failure) when adding files to existing locations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org