Bunakov Ilya created BIGTOP-4438:
------------------------------------
Summary: Problem compatibility of Hadoop components (HDFS,
Mapreduce, Hive, Spark)
Key: BIGTOP-4438
URL: https://issues.apache.org/jira/browse/BIGTOP-4438
Project: Bigtop
Issue Type: Bug
Components: hadoop, hive
Affects Versions: 3.4.0
Environment: bigtop 3.4.0
ubuntu-24.04
hadoop 3.3.6
hive 4.0.1
Reporter: Bunakov Ilya
In the new version of BigTop 3.4.0 there are problems with compatibility of
Hadoop components (HDFS, Mapreduce, Hive).
When running Mapreduce applications there are problems with dependencies,
example:
~ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-3.3.6.jar
teragen 100000000 /tmp/teragenout
java.lang.RuntimeException: java.lang.VerifyError: class
org.apache.log4j.bridge.LogEventAdapter overrides final method getTimeStamp.()J
at
org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:149)
at
org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:181)
at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:266)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:347)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:344)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:344)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:471)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:448)
at org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:434)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:478)
Caused by: java.lang.VerifyError: class org.apache.log4j.bridge.LogEventAdapter
overrides final method getTimeStamp.()J
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.log4j.builders.BuilderManager.<clinit>(BuilderManager.java:58)
at
org.apache.log4j.config.Log4j1Configuration.<init>(Log4j1Configuration.java:46)
at
org.apache.log4j.config.PropertiesConfiguration.<init>(PropertiesConfiguration.java:90)
at
org.apache.log4j.config.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:58)
at
org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:499)
at
org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:404)
at
org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:318)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:690)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
at
org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
at
org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:46)
at
org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:41)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:186)
at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:57)
at org.apache.hadoop.fs.local.RawLocalFs.<init>(RawLocalFs.java:43)
at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:37)
at org.apache.hadoop.fs.local.LocalFs.<init>(LocalFs.java:51)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:143)
... 12 more
Similar stack trace when using the tez on yarn.
Hey did anyone manage to get Hive 4.0.1 working with Spark 3.5.3? SparkSQL can
query {{show databases}} and successfully displays all available databases, but
invoking {{select * from xyz}} fails with {{{}HiveException: unable to fetch
table xyz. Invalid method name 'get_table'{}}}. Adding the jars from hive to
spark and specifying {{spark.sql.hive.metastore.version 4.0.1}} throws an error
about unsupported version and all queries fail. Is there a workaround?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
