AngersZhuuuu opened a new pull request #25775: [WIP][SPARK-29015][SQL][test-hadoop3.2][test-java11] fix bug for jdk11, can't use 'add jar' jar's class as create table serde URL: https://github.com/apache/spark/pull/25775 ### What changes were proposed in this pull request? As show in issue [SPARK-29015](https://issues.apache.org/jira/browse/SPARK-29015) In hadoop 3.2 & jdk11, when we use 'add jar' to add a jar, then use it's class as serde class. we can create table successfully but when select data throw ClassNotFound exception. When we call HiveClientImpl#createTable(), it's called withHiveState(), it will get serde class's constructor with HiveClientImpl's clientLoader's classLoader. When we use 'ADD JAR' sql, jar will be added to this ClassLoader. In jdk11, when we call ``` select * from addJar ``` In sql analyze process, it will call HiveClientImpl#withHiveState() In jdk11, this thread's ThreadLocal SessionState will be attached with HiveClientImpl's state. When check serde class in HiveTableScanExec, this SessionState will be used in TableDesc.getDeserializerClass, but doesn't in control of HiveClientImpl#withHiveState Means it won't replace SessionState's conf's ClassLoader as clientLoader's class loader. So it can't found class with SessionState's origin ClassLoader. So we should detach SessionState after we have use it already. Then TaskDesc.getDeserializerClass will use classLoader of currentThread. Since we have se `sharedState.jarClassLoader` to current thread when we call execute() in SparkExecuteStatement . then here will use sharedState.jarClassLoader. In this Jar ClassLoader, it will contain jars : -- jar HIVEAUXCONF [SPARK-28954](https://github.com/apache/spark/pull/25653) [SPARK-28840](https://github.com/apache/spark/pull/25542) SQL 'ADD JAR' ### Why are the changes needed? support for jdk11 ### Does this PR introduce any user-facing change? NO ### How was this patch tested? UT
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
