Hi all, We made some changes to hive which require changes to the hive jars that Spark is bundled with. Since Spark 3.3.1 comes bundled with Hive 2.3.9 jars, we built our changes in Hive 2.3.9 and put the necessary jars under $SPARK_HOME/jars (replacing the original jars that were there), everything works fine.
However since I wanted to make use of spark.jars.packages to download jars at runtime, I thought what would also work is if I deleted the original hive jars from $SPARK_HOME/jars and download the same jars at runtime. Apparently spark.jars.packages should add these jars to the classpath. Instead I get a NoClassDefFoundError downloading the same Jars: ``` Caused by: java.lang.reflect.InvocationTargetException: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/metadata/HiveException at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:227) ... 87 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/metadata/HiveException at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:75) ... 92 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.metadata.HiveException at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source) at java.base/java.lang.ClassLoader.loadClass(Unknown Source) ``` The class HiveException should already be available in the jars that have been supplied by spark.jars.packages... Any idea what could be wrong? Thanks, Yeachan
