[
https://issues.apache.org/jira/browse/SPARK-26839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781903#comment-16781903
]
Sean Owen commented on SPARK-26839:
-----------------------------------
Just sharing where I have gotten to with this as well:
I think we need to update IsolatedClientLoader to use getPlatformClassLoader()
on Java 9+ as the parent. Right now it uses
getSystemClassloader().getParent().getParent(), which is the boot classloader,
which is actually always represented by null. (There's an assert in the code
that checks if loading Hive classes fails, and it always fails due to an NPE!
That part is easy.
The problem here is actually datanucleus, which needs to be 5.1+ for Java 9, it
seems. I was uneasy about updating from 3.2 because Hive uses it, but it seems
to get past the immediate error here. I have a PR that does both of these
things.
However it now fails for not finding BONECP, which is the pool I think Hive
uses. It's on the classpath though. I suspect it's something to do with how we
try to isolate Hive-related classes. I put BONECP on the classpath explicitly
and still no dice. That's where I got so far.
> on JDK11, IsolatedClientLoader must be able to load java.sql classes
> --------------------------------------------------------------------
>
> Key: SPARK-26839
> URL: https://issues.apache.org/jira/browse/SPARK-26839
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Imran Rashid
> Priority: Major
>
> This might be very specific to my fork & a kind of weird system setup I'm
> working on, I haven't completely confirmed yet, but I wanted to report it
> anyway in case anybody else sees this.
> When I try to do anything which touches the metastore on java11, I
> immediately get errors from IsolatedClientLoader that it can't load anything
> in java.sql. eg.
> {noformat}
> scala> spark.sql("show tables").show()
> java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError:
> java/sql/SQLTransientException when creating Hive client using classpath:
> file:/home/systest/jdk-11.0.2/, ...
> ...
> Caused by: java.lang.ClassNotFoundException: java.sql.SQLTransientException
> at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:230)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:219)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> {noformat}
> After a bit of debugging, I also discovered that the {{rootClassLoader}} is
> {{null}} in {{IsolatedClientLoader}}. I think this would work if either
> {{rootClassLoader}} could load those classes, or if {{isShared()}} was
> changed to allow any class starting with "java." (I'm not sure why it only
> allows "java.lang" and "java.net" currently.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
