The more I look at this the more confusing it gets, it looks like some places Hive is using the class loader from the SessionState.getConf() and sometimes from the thread context. I'd expect the thread context class loader and the session state one to be the same as the thread context is set by sessionstate. However I'm looking at the start method on SessionState and it doesn't set the classloader at all so registerJars may be getting the wrong class loader.
Thanks Shawn -----Original Message----- From: Shawn Weeks <swe...@weeksconsulting.us> Sent: Saturday, March 9, 2019 10:44 AM To: dev@hive.apache.org Subject: RE: Custom UDF Loses Depenencies I filed HIVE-21409 for this issue. It looks like the registerJars method in SessionState is still using the thread context class loader instead of grabbing the one from SessionState. I'm not sure how it happens but it looks like if you have a bunch of parallel sessions from something like Oozie right after Hive starts the class path can get really polluted quickly. I need to go look at the mechanism behind the "delete jars" command as running that on a brand new session seems to wipe all the extra stuff out of the class loader. Thanks -----Original Message----- From: Gopal Vijayaraghavan <gop...@apache.org> Sent: Wednesday, March 6, 2019 3:01 PM To: dev@hive.apache.org Subject: Re: Custom UDF Loses Depenencies > When we register a jar on the Hive console. Hive creates a fresh URL > classloader which includes the path of the current jar to be registered and > all the jar paths of the parent classloader. The parent classlaoder is the > current ThreadContextClassLoader. Once the URLClassloader is created Hive > sets ... That looks like the root-cause of a different class of issues. Since the fresh classloader is picking up the URLs, the URLs keep growing. Worse, the Hadoop classes loaded from shims are coming off the thread loader & being reloaded a million times. Cheers, Gopal
