[ https://issues.apache.org/jira/browse/IMPALA-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sai Hemanth Gantasala resolved IMPALA-12680. -------------------------------------------- Fix Version/s: Impala 4.5.0 Resolution: Fixed > NullPointerException in addHmsPartitions() during MetastoreEventsProcessor > switch state from PAUSED to ACTIVE > ------------------------------------------------------------------------------------------------------------- > > Key: IMPALA-12680 > URL: https://issues.apache.org/jira/browse/IMPALA-12680 > Project: IMPALA > Issue Type: Bug > Components: Catalog > Reporter: Quanlong Huang > Assignee: Sai Hemanth Gantasala > Priority: Critical > Fix For: Impala 4.5.0 > > Attachments: alterTableAddPartitionProfile.txt, catalogd.INFO.gz, > impalad.INFO.gz > > > Event processing is paused during a global INVALIDATE METADATA. > catalog_.isEventProcessingActive() returns false in this state. > > If an AlterTableAddPartition statement is running during that time, we could > pass in a null value for the 'partitionToEventId' map in here: > {code:java} > Map<String, Long> partitionToEventId = > catalog_.isEventProcessingActive() ? > Maps.newHashMap() : null; > List<Partition> addedHmsPartitions = > addHmsPartitionsInTransaction(msClient, > tbl, allHmsPartitionsToAdd, partitionToEventId, ifNotExists); {code} > > https://github.com/apache/impala/blob/fcda98ad99c13324e3ab09f2e92d331d0304bb8e/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L4404-L4407 > If the global INVALIDATE METADATA finishes and event processing is back to > the ACTIVE state before the AlterTableAddPartition statement runs into > addHmsPartitions(), we will have an non-empty 'partitionToEventSubMap' here: > {code:java} > List<NotificationEvent> events = > getNextMetastoreEventsIfEnabled(eventId, > event -> AddPartitionEvent.ADD_PARTITION_EVENT_TYPE > .equals(event.getEventType()) > && msTbl.getDbName().equalsIgnoreCase(event.getDbName()) > && > msTbl.getTableName().equalsIgnoreCase(event.getTableName())); > Map<Partition, Long> partitionToEventSubMap = Maps.newHashMap(); > getPartitionsFromEvent(events, partitionToEventSubMap); > // set the eventId to last one which we received so the we fetch the > next > // set of events correctly > if (!events.isEmpty()) { > eventId = events.get(events.size() - 1).getEventId(); > } > if (partitionToEventSubMap.isEmpty()) { > // if partitions couldn't be fetched from events, use the one > returned by > // add_partitions call above. > addedHmsPartitions.addAll(addedPartitions); > } else { > Preconditions.checkNotNull(partitionToEventId); // <-- This will > fail{code} > https://github.com/apache/impala/blob/fcda98ad99c13324e3ab09f2e92d331d0304bb8e/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L5052-L5069 > Then the AlterTableAddPartition statement fails with NullPointerException: > {code:java} > I0104 02:46:32.075830 1010 jni-util.cc:302] > 4a4eae34f60ba947:b6b2bcfc00000000] java.lang.NullPointerException > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:889) > at > org.apache.impala.service.CatalogOpExecutor.addHmsPartitions(CatalogOpExecutor.java:5051) > at > org.apache.impala.service.CatalogOpExecutor.addHmsPartitionsInTransaction(CatalogOpExecutor.java:5082) > at > org.apache.impala.service.CatalogOpExecutor.alterTableAddPartitions(CatalogOpExecutor.java:4388) > at > org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:1136) > at > org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:450) > at > org.apache.impala.service.JniCatalog.lambda$execDdl$3(JniCatalog.java:304) > at > org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90) > at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58) > at > org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89) > at > org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100) > at > org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:233) > at > org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:247) > at > org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:303){code} -- This message was sent by Atlassian Jira (v8.20.10#820010)