[ https://issues.apache.org/jira/browse/IMPALA-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe McDonnell resolved IMPALA-10057. ------------------------------------ Fix Version/s: Impala 4.2.0 Resolution: Fixed > TransactionKeepalive NoClassDefFoundError floods logs during JDBC_TEST/FE_TEST > ------------------------------------------------------------------------------ > > Key: IMPALA-10057 > URL: https://issues.apache.org/jira/browse/IMPALA-10057 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure > Affects Versions: Impala 4.0.0 > Reporter: Joe McDonnell > Assignee: Joe McDonnell > Priority: Major > Labels: flaky > Fix For: Impala 4.2.0 > > > For the both the normal tests and the docker-based tests, the Impala logs > generated during the FE_TEST/JDBC_TEST can be huge: > > {noformat} > $ du -c -h fe_test/ee_tests > 4.0K fe_test/ee_tests/minidumps/statestored > 4.0K fe_test/ee_tests/minidumps/impalad > 4.0K fe_test/ee_tests/minidumps/catalogd > 16K fe_test/ee_tests/minidumps > 352K fe_test/ee_tests/profiles > 81G fe_test/ee_tests > 81G total{noformat} > Creating a tarball of these logs takes 10 minutes. The Impalad/catalogd logs > are filled with this error over and over: > {noformat} > E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected > exception thrown > Java exception follows: > java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: > org/apache/impala/common/TransactionKeepalive$HeartbeatContext > at > org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.NoClassDefFoundError: > org/apache/impala/common/TransactionKeepalive$HeartbeatContext > ... 2 more > Caused by: java.lang.ClassNotFoundException: > org.apache.impala.common.TransactionKeepalive$HeartbeatContext > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > ... 2 more{noformat} > Two interesting points: > # The frontend/jdbc tests are passing, so all of these errors in the impalad > logs are not impacting tests. > # These errors aren't concurrently with any of the other tests (ee tests, > custom cluster tests, etc). > This is happening on normal core runs (including the GVO job that does > FE_TEST/JDBC_TEST) on both Ubuntu and Centos 7. It is also happening on > docker-based tests. A theory is that FE_TEST/JDBC_TEST have an Impala cluster > running and then invoke maven to run the tests. Maven could manipulate jars > while Impala is running. Maybe there is a race-condition or conflict when > manipulating those jars that could cause the NoClassDefFoundError. It makes > no sense for Impala not to be able to find > TransactionKeepalive$HeartbeatContext. > When it happens, it is in a tight loop, printing the message more than once > per millisecond. It fills the ERROR, WARNING, and INFO logs with that > message, sometimes for multiple Impalads and/or catalogd. -- This message was sent by Atlassian Jira (v8.20.7#820007)