What version of Hadoop are y'all using? Which parts of Hadoop are you using?
Can you try relying on the hadoop-client stuff from Hadoop 3? it won't be any better about use of the thread-context classloader (unfortunately) but it does do a fair job of cutting down on the number of third party dependencies present out of the box. On Fri, Feb 1, 2019 at 3:33 PM John Lilley <john.lil...@redpointglobal.com> wrote: > > I realize this is something of a broad question, but I hope that someone has > already had to deal with it and can share some experience. Our application > has multiple classloaders. This is to avoid dependency conflicts between the > various APIs that our software accesses, including but not limited to Hadoop. > Without using a multi-classloader strategy (or Java 9 modules, which we > cannot yet support) we encounter a lot of dependency conflicts between the > Hadoop jars and everything else. So, suffice it to say that we have an > URLClassLoader for all of the Hadoop jars, and other classloaders for other > set of libraries. > > > > The problem is this: The system classloader (the one that the JVM starts > with) does not have the Hadoop jars. This means that the thread-context > classloader, which is the system classloader by default, does not have the > Hadoop jars. Unfortunately, Hadoop APIs of all sorts seem to use the > thread-context classloader implicitly, which leads to all manner of > ClassNotFound errors, ServiceLoaders returning empty lists, factories not > finding implementations, etc. This is driving us slowly mad, because new > “not found” issues crop up occasionally in the field as our customers > discover new security options and other configuration variants, which leads > down code paths that fail to find classes or implementations of interfaces. > > > > You may suggest the obvious: Always set the thread-context classloader to be > the Hadoop classloader. Unfortunately this doesn’t seem to be a solution. > First, threads get spawned which have the default system classloader as the > thread-context-classloader, and I don’t have control over all of those > threads. Second, it is error-prone and probably too expensive to save and > restore the thread-context classloader at every possible Hadoop entry point. > > > > The only answer I’ve found is to learn by trial-and-error which classes are > loaded by the thread-context classloader (or by ServiceLoader), and force > them to pre-load where I’ve set the thread-context classloader. Is there a > better way? If not, is there a list of such classes? > > > > Thanks, > > John Lilley > > -- busbey --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
