[ https://issues.apache.org/jira/browse/IMPALA-12277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on IMPALA-12277 started by Sai Hemanth Gantasala. ------------------------------------------------------ > metadata reload of INSERT failed by NullPointerException: Invalid partition > name > -------------------------------------------------------------------------------- > > Key: IMPALA-12277 > URL: https://issues.apache.org/jira/browse/IMPALA-12277 > Project: IMPALA > Issue Type: Bug > Components: Catalog > Reporter: Quanlong Huang > Assignee: Sai Hemanth Gantasala > Priority: Major > > INSERT into a partition that exists in catalogd but doesn't exist in HMS will > fail in metadata reloading on the partition. The cause is that updateCatalog > doesn't create the partition in HMS (since catalogd is not aware of the > non-existence of the partition in HMS): > [https://github.com/apache/impala/blob/d0fe4c604f72d41019832513ebf65cfe8f469953/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L6697-L6699] > When reloading the partition, catalogd first removes it since it doesn't > exist in HMS: > [https://github.com/apache/impala/blob/d0fe4c604f72d41019832513ebf65cfe8f469953/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1530-L1531] > It then try to reload it, which hits NullPointerException at: > [https://github.com/apache/impala/blob/d0fe4c604f72d41019832513ebf65cfe8f469953/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1566] > To reproduce the issue, launch Impala with event processing disabled so > catalogd can be unsynced with HMS. Create a partitioned table in Impala with > one partition: > {noformat} > bin/start-impala-cluster.py --catalogd_args=--hms_event_polling_interval_s=0 > impala> create table my_part2 (id int) partitioned by (p int) stored as > textfile; > impala> insert into my_part2 partition(p=0) values (0); {noformat} > Drop the partition in Hive: > {code:sql} > hive> alter table my_part2 drop partition (p=0);{code} > Then insert the partition again in Impala > {code:sql} > impala> insert into my_part2 partition(p=0) values (1); > ERROR: TableLoadingException: Failed to load metadata for table: > default.my_part2 > CAUSED BY: NullPointerException: Invalid partition name: p=0 > {code} > The exception: > {noformat} > E0710 19:34:43.569339 4413 JniUtil.java:183] > bb4452d18eafe116:eaf16c4000000000] Error in Update catalog for > default.my_part2. Time spent: 1s186ms > I0710 19:34:43.569918 4413 jni-util.cc:288] > bb4452d18eafe116:eaf16c4000000000] > org.apache.impala.catalog.TableLoadingException: Failed to load metadata for > table: default.my_part2 > at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1308) > at > org.apache.impala.service.CatalogOpExecutor.loadTableMetadata(CatalogOpExecutor.java:1521) > at > org.apache.impala.service.CatalogOpExecutor.updateCatalog(CatalogOpExecutor.java:6863) > at > org.apache.impala.service.JniCatalog.lambda$updateCatalog$16(JniCatalog.java:471) > at > org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90) > at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58) > at > org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89) > at > org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100) > at > org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:230) > at > org.apache.impala.service.JniCatalog.updateCatalog(JniCatalog.java:470) > Caused by: java.lang.NullPointerException: Invalid partition name: p=0 > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:907) > at > org.apache.impala.catalog.HdfsTable.getPartitionsForNames(HdfsTable.java:1766) > at > org.apache.impala.catalog.HdfsTable$PartitionDeltaUpdater.apply(HdfsTable.java:1566) > at > org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(HdfsTable.java:1447) > at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1282) > ... 9 more{noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org