Quanlong Huang created IMPALA-10372: ---------------------------------------
Summary: Data loading fails with foreign key constraint violation in postgresql Key: IMPALA-10372 URL: https://issues.apache.org/jira/browse/IMPALA-10372 Project: IMPALA Issue Type: Bug Reporter: Quanlong Huang Saw a failure in data loading in a private build: {code} ERROR: DROP TABLE IF EXISTS functional_seq_record_bzip.insert_only_transactional_table Traceback (most recent call last): File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/bin/load-data.py", line 208, in exec_impala_query_from_file result = impala_client.execute(query) File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py", line 187, in execute handle = self.__execute_query(query_string.strip(), user=user) File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py", line 363, in __execute_query handle = self.execute_query_async(query_string, user=user) File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py", line 357, in execute_query_async handle = self.__do_rpc(lambda: self.imp_service.query(query,)) File "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py", line 520, in __do_rpc raise ImpalaBeeswaxException(self.__build_error_message(b), b) ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: ImpalaRuntimeException: Error making 'dropTable' RPC to Hive Metastore: CAUSED BY: MetaException: Exception thrown flushing changes to datastore {code} The root exception is {code} ERROR: update or delete on table "TBLS" violates foreign key constraint "MV_TABLES_USED_FK2" on table "MV_TABLES_USED" {code} Found it in hive-metastore.log: {code} 2020-12-01T04:37:32,946 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27: source:127.0.0.1 get_table : tbl=hive.functional_seq_record_bzip.insert_only_transactional_table 2020-12-01T04:37:32,946 INFO [pool-6-thread-18] HiveMetaStore.audit: ugi=jenkins ip=127.0.0.1 cmd=source:127.0.0.1 get_table : tbl=hive.functional_seq_record_bzip.insert_only_transactional_table 2020-12-01T04:37:32,955 INFO [pool-6-thread-18] metastore.MetastoreDefaultTransformer: Starting translation for processor Impala4.0.0-SNAPSHOT@localhost on list 1 2020-12-01T04:37:32,955 INFO [pool-6-thread-18] metastore.MetastoreDefaultTransformer: Table insert_only_transactional_table,#bucket=0,isBucketed:false,tableType=MANAGED_TABLE,tableCapabilities=HIVEMANAGEDINSERTREAD,HIVEMANAGEDINSERTWRITE 2020-12-01T04:37:32,955 INFO [pool-6-thread-18] metastore.MetastoreDefaultTransformer: Abilities for match: Table type=MANAGED_TABLE,accesstype is RW 2020-12-01T04:37:32,955 INFO [pool-6-thread-18] metastore.MetastoreDefaultTransformer: Transformer return list of 1 2020-12-01T04:37:32,961 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27: source:127.0.0.1 drop_table : tbl=hive.functional_seq_record_bzip.insert_only_transactional_table 2020-12-01T04:37:32,961 INFO [pool-6-thread-18] HiveMetaStore.audit: ugi=jenkins ip=127.0.0.1 cmd=source:127.0.0.1 drop_table : tbl=hive.functional_seq_record_bzip.insert_only_transactional_table 2020-12-01T04:37:33,143 ERROR [pool-6-thread-18] metastore.RetryingHMSHandler: Retrying HMSHandler after 2000 ms (attempt 1 of 10) with error: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore ...... 2020-12-01T04:37:53,690 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27: source:127.0.0.1 drop_table : tbl=hive.functional_seq_record_bzip.insert_only_transactional_table 2020-12-01T04:37:53,690 INFO [pool-6-thread-18] HiveMetaStore.audit: ugi=jenkins ip=127.0.0.1 cmd=source:127.0.0.1 drop_table : tbl=hive.functional_seq_record_bzip.insert_only_transactional_table 2020-12-01T04:37:53,709 ERROR [pool-6-thread-18] metastore.RetryingHMSHandler: HMSHandler Fatal error: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543) at org.datanucleus.api.jdo.JDOPersistenceManager.flush(JDOPersistenceManager.java:2058) at org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:10827) at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at com.sun.proxy.$Proxy27.addNotificationEvent(Unknown Source) at org.apache.hive.hcatalog.listener.DbNotificationListener.process(DbNotificationListener.java:1123) at org.apache.hive.hcatalog.listener.DbNotificationListener.onDropTable(DbNotificationListener.java:234) at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$19.notify(MetaStoreListenerNotifier.java:103) at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:285) at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:347) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2991) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3231) at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at com.sun.proxy.$Proxy28.drop_table_with_environment_context(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:16875) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:16859) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) NestedThrowablesStackTrace: java.sql.BatchUpdateException: Batch entry 0 DELETE FROM "TBLS" WHERE "TBL_ID"=713 was aborted: ERROR: update or delete on table "TBLS" violates foreign key constraint "MV_TABLES_USED_FK2" on table "MV_TABLES_USED" Detail: Key (TBL_ID)=(713) is still referenced from table "MV_TABLES_USED". Call getNextException to see other errors in the batch. at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:159) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2268) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:510) at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:851) at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:874) at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1563) at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:125) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:366) at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:676) at org.datanucleus.store.rdbms.SQLController.processStatementsForConnection(SQLController.java:644) at org.datanucleus.store.rdbms.SQLController$1.transactionFlushed(SQLController.java:731) at org.datanucleus.store.connection.AbstractManagedConnection.transactionFlushed(AbstractManagedConnection.java:89) at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionFlushed(ConnectionManagerImpl.java:450) at org.datanucleus.TransactionImpl.flush(TransactionImpl.java:210) at org.datanucleus.ExecutionContextImpl.flushInternal(ExecutionContextImpl.java:4060) at org.datanucleus.ExecutionContextThreadedImpl.flushInternal(ExecutionContextThreadedImpl.java:450) at org.datanucleus.ExecutionContextImpl.flush(ExecutionContextImpl.java:4001) at org.datanucleus.ExecutionContextThreadedImpl.flush(ExecutionContextThreadedImpl.java:436) at org.datanucleus.api.jdo.JDOPersistenceManager.flush(JDOPersistenceManager.java:2040) at org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:10827) at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at com.sun.proxy.$Proxy27.addNotificationEvent(Unknown Source) at org.apache.hive.hcatalog.listener.DbNotificationListener.process(DbNotificationListener.java:1123) at org.apache.hive.hcatalog.listener.DbNotificationListener.onDropTable(DbNotificationListener.java:234) at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$19.notify(MetaStoreListenerNotifier.java:103) at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:285) at org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:347) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2991) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:3231) at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at com.sun.proxy.$Proxy28.drop_table_with_environment_context(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:16875) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:16859) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "TBLS" violates foreign key constraint "MV_TABLES_USED_FK2" on table "MV_TABLES_USED" Detail: Key (TBL_ID)=(713) is still referenced from table "MV_TABLES_USED". at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267) ... 50 more {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)