[
https://issues.apache.org/jira/browse/IMPALA-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-10057.
------------------------------------
Fix Version/s: Impala 4.2.0
Resolution: Fixed
> TransactionKeepalive NoClassDefFoundError floods logs during JDBC_TEST/FE_TEST
> ------------------------------------------------------------------------------
>
> Key: IMPALA-10057
> URL: https://issues.apache.org/jira/browse/IMPALA-10057
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Labels: flaky
> Fix For: Impala 4.2.0
>
>
> For the both the normal tests and the docker-based tests, the Impala logs
> generated during the FE_TEST/JDBC_TEST can be huge:
>
> {noformat}
> $ du -c -h fe_test/ee_tests
> 4.0K fe_test/ee_tests/minidumps/statestored
> 4.0K fe_test/ee_tests/minidumps/impalad
> 4.0K fe_test/ee_tests/minidumps/catalogd
> 16K fe_test/ee_tests/minidumps
> 352K fe_test/ee_tests/profiles
> 81G fe_test/ee_tests
> 81G total{noformat}
> Creating a tarball of these logs takes 10 minutes. The Impalad/catalogd logs
> are filled with this error over and over:
> {noformat}
> E0903 02:25:39.453887 12060 TransactionKeepalive.java:137] Unexpected
> exception thrown
> Java exception follows:
> java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError:
> org/apache/impala/common/TransactionKeepalive$HeartbeatContext
> at
> org.apache.impala.common.TransactionKeepalive$DaemonThread.run(TransactionKeepalive.java:114)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/impala/common/TransactionKeepalive$HeartbeatContext
> ... 2 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.impala.common.TransactionKeepalive$HeartbeatContext
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 2 more{noformat}
> Two interesting points:
> # The frontend/jdbc tests are passing, so all of these errors in the impalad
> logs are not impacting tests.
> # These errors aren't concurrently with any of the other tests (ee tests,
> custom cluster tests, etc).
> This is happening on normal core runs (including the GVO job that does
> FE_TEST/JDBC_TEST) on both Ubuntu and Centos 7. It is also happening on
> docker-based tests. A theory is that FE_TEST/JDBC_TEST have an Impala cluster
> running and then invoke maven to run the tests. Maven could manipulate jars
> while Impala is running. Maybe there is a race-condition or conflict when
> manipulating those jars that could cause the NoClassDefFoundError. It makes
> no sense for Impala not to be able to find
> TransactionKeepalive$HeartbeatContext.
> When it happens, it is in a tight loop, printing the message more than once
> per millisecond. It fills the ERROR, WARNING, and INFO logs with that
> message, sometimes for multiple Impalads and/or catalogd.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
