Is that an assumption we can make? I think we'd run into an issue in this situation:
*In primary jar:* def makeDynamicObject(clazz: String) = Class.forName(clazz).newInstance() *In app code:* sc.addJar("dynamicjar.jar") ... rdd.map(x => makeDynamicObject("some.class.from.DynamicJar")) It might be fair to say that the user should make sure to use the context classloader when instantiating dynamic classes, but I think it's weird that this code would work on Spark standalone but not on YARN. -Sandy On Wed, May 21, 2014 at 2:10 PM, Xiangrui Meng <men...@gmail.com> wrote: > I think adding jars dynamically should work as long as the primary jar > and the secondary jars do not depend on dynamically added jars, which > should be the correct logic. -Xiangrui > > On Wed, May 21, 2014 at 1:40 PM, DB Tsai <dbt...@stanford.edu> wrote: > > This will be another separate story. > > > > Since in the yarn deployment, as Sandy said, the app.jar will be always > in > > the systemclassloader which means any object instantiated in app.jar will > > have parent loader of systemclassloader instead of custom one. As a > result, > > the custom class loader in yarn will never work without specifically > using > > reflection. > > > > Solution will be not using system classloader in the classloader > hierarchy, > > and add all the resources in system one into custom one. This is the > > approach of tomcat takes. > > > > Or we can directly overwirte the system class loader by calling the > > protected method `addURL` which will not work and throw exception if the > > code is wrapped in security manager. > > > > > > Sincerely, > > > > DB Tsai > > ------------------------------------------------------- > > My Blog: https://www.dbtsai.com > > LinkedIn: https://www.linkedin.com/in/dbtsai > > > > > > On Wed, May 21, 2014 at 1:13 PM, Sandy Ryza <sandy.r...@cloudera.com> > wrote: > > > >> This will solve the issue for jars added upon application submission, > but, > >> on top of this, we need to make sure that anything dynamically added > >> through sc.addJar works as well. > >> > >> To do so, we need to make sure that any jars retrieved via the driver's > >> HTTP server are loaded by the same classloader that loads the jars > given on > >> app submission. To achieve this, we need to either use the same > >> classloader for both system jars and user jars, or make sure that the > user > >> jars given on app submission are under the same classloader used for > >> dynamically added jars. > >> > >> On Tue, May 20, 2014 at 5:59 PM, Xiangrui Meng <men...@gmail.com> > wrote: > >> > >> > Talked with Sandy and DB offline. I think the best solution is sending > >> > the secondary jars to the distributed cache of all containers rather > >> > than just the master, and set the classpath to include spark jar, > >> > primary app jar, and secondary jars before executor starts. In this > >> > way, user only needs to specify secondary jars via --jars instead of > >> > calling sc.addJar inside the code. It also solves the scalability > >> > problem of serving all the jars via http. > >> > > >> > If this solution sounds good, I can try to make a patch. > >> > > >> > Best, > >> > Xiangrui > >> > > >> > On Mon, May 19, 2014 at 10:04 PM, DB Tsai <dbt...@stanford.edu> > wrote: > >> > > In 1.0, there is a new option for users to choose which classloader > has > >> > > higher priority via spark.files.userClassPathFirst, I decided to > submit > >> > the > >> > > PR for 0.9 first. We use this patch in our lab and we can use those > >> jars > >> > > added by sc.addJar without reflection. > >> > > > >> > > https://github.com/apache/spark/pull/834 > >> > > > >> > > Can anyone comment if it's a good approach? > >> > > > >> > > Thanks. > >> > > > >> > > > >> > > Sincerely, > >> > > > >> > > DB Tsai > >> > > ------------------------------------------------------- > >> > > My Blog: https://www.dbtsai.com > >> > > LinkedIn: https://www.linkedin.com/in/dbtsai > >> > > > >> > > > >> > > On Mon, May 19, 2014 at 7:42 PM, DB Tsai <dbt...@stanford.edu> > wrote: > >> > > > >> > >> Good summary! We fixed it in branch 0.9 since our production is > still > >> in > >> > >> 0.9. I'm porting it to 1.0 now, and hopefully will submit PR for > 1.0 > >> > >> tonight. > >> > >> > >> > >> > >> > >> Sincerely, > >> > >> > >> > >> DB Tsai > >> > >> ------------------------------------------------------- > >> > >> My Blog: https://www.dbtsai.com > >> > >> LinkedIn: https://www.linkedin.com/in/dbtsai > >> > >> > >> > >> > >> > >> On Mon, May 19, 2014 at 7:38 PM, Sandy Ryza < > sandy.r...@cloudera.com > >> > >wrote: > >> > >> > >> > >>> It just hit me why this problem is showing up on YARN and not on > >> > >>> standalone. > >> > >>> > >> > >>> The relevant difference between YARN and standalone is that, on > YARN, > >> > the > >> > >>> app jar is loaded by the system classloader instead of Spark's > custom > >> > URL > >> > >>> classloader. > >> > >>> > >> > >>> On YARN, the system classloader knows about [the classes in the > spark > >> > >>> jars, > >> > >>> the classes in the primary app jar]. The custom classloader > knows > >> > about > >> > >>> [the classes in secondary app jars] and has the system > classloader as > >> > its > >> > >>> parent. > >> > >>> > >> > >>> A few relevant facts (mostly redundant with what Sean pointed > out): > >> > >>> * Every class has a classloader that loaded it. > >> > >>> * When an object of class B is instantiated inside of class A, the > >> > >>> classloader used for loading B is the classloader that was used > for > >> > >>> loading > >> > >>> A. > >> > >>> * When a classloader fails to load a class, it lets its parent > >> > classloader > >> > >>> try. If its parent succeeds, its parent becomes the "classloader > >> that > >> > >>> loaded it". > >> > >>> > >> > >>> So suppose class B is in a secondary app jar and class A is in the > >> > primary > >> > >>> app jar: > >> > >>> 1. The custom classloader will try to load class A. > >> > >>> 2. It will fail, because it only knows about the secondary jars. > >> > >>> 3. It will delegate to its parent, the system classloader. > >> > >>> 4. The system classloader will succeed, because it knows about the > >> > primary > >> > >>> app jar. > >> > >>> 5. A's classloader will be the system classloader. > >> > >>> 6. A tries to instantiate an instance of class B. > >> > >>> 7. B will be loaded with A's classloader, which is the system > >> > classloader. > >> > >>> 8. Loading B will fail, because A's classloader, which is the > system > >> > >>> classloader, doesn't know about the secondary app jars. > >> > >>> > >> > >>> In Spark standalone, A and B are both loaded by the custom > >> > classloader, so > >> > >>> this issue doesn't come up. > >> > >>> > >> > >>> -Sandy > >> > >>> > >> > >>> On Mon, May 19, 2014 at 7:07 PM, Patrick Wendell < > pwend...@gmail.com > >> > > >> > >>> wrote: > >> > >>> > >> > >>> > Having a user add define a custom class inside of an added jar > and > >> > >>> > instantiate it directly inside of an executor is definitely > >> supported > >> > >>> > in Spark and has been for a really long time (several years). > This > >> is > >> > >>> > something we do all the time in Spark. > >> > >>> > > >> > >>> > DB - I'd hold off on a re-architecting of this until we identify > >> > >>> > exactly what is causing the bug you are running into. > >> > >>> > > >> > >>> > In a nutshell, when the bytecode "new Foo()" is run on the > >> executor, > >> > >>> > it will ask the driver for the class over HTTP using a custom > >> > >>> > classloader. Something in that pipeline is breaking here, > possibly > >> > >>> > related to the YARN deployment stuff. > >> > >>> > > >> > >>> > > >> > >>> > On Mon, May 19, 2014 at 12:29 AM, Sean Owen <so...@cloudera.com > > > >> > wrote: > >> > >>> > > I don't think a customer classloader is necessary. > >> > >>> > > > >> > >>> > > Well, it occurs to me that this is no new problem. Hadoop, > >> Tomcat, > >> > etc > >> > >>> > > all run custom user code that creates new user objects without > >> > >>> > > reflection. I should go see how that's done. Maybe it's > totally > >> > valid > >> > >>> > > to set the thread's context classloader for just this purpose, > >> and > >> > I > >> > >>> > > am not thinking clearly. > >> > >>> > > > >> > >>> > > On Mon, May 19, 2014 at 8:26 AM, Andrew Ash < > >> and...@andrewash.com> > >> > >>> > wrote: > >> > >>> > >> Sounds like the problem is that classloaders always look in > >> their > >> > >>> > parents > >> > >>> > >> before themselves, and Spark users want executors to pick up > >> > classes > >> > >>> > from > >> > >>> > >> their custom code before the ones in Spark plus its > >> dependencies. > >> > >>> > >> > >> > >>> > >> Would a custom classloader that delegates to the parent after > >> > first > >> > >>> > >> checking itself fix this up? > >> > >>> > >> > >> > >>> > >> > >> > >>> > >> On Mon, May 19, 2014 at 12:17 AM, DB Tsai < > dbt...@stanford.edu> > >> > >>> wrote: > >> > >>> > >> > >> > >>> > >>> Hi Sean, > >> > >>> > >>> > >> > >>> > >>> It's true that the issue here is classloader, and due to the > >> > >>> > classloader > >> > >>> > >>> delegation model, users have to use reflection in the > executors > >> > to > >> > >>> > pick up > >> > >>> > >>> the classloader in order to use those classes added by > >> sc.addJars > >> > >>> APIs. > >> > >>> > >>> However, it's very inconvenience for users, and not > documented > >> in > >> > >>> > spark. > >> > >>> > >>> > >> > >>> > >>> I'm working on a patch to solve it by calling the protected > >> > method > >> > >>> > addURL > >> > >>> > >>> in URLClassLoader to update the current default > classloader, so > >> > no > >> > >>> > >>> customClassLoader anymore. I wonder if this is an good way > to > >> go. > >> > >>> > >>> > >> > >>> > >>> private def addURL(url: URL, loader: URLClassLoader){ > >> > >>> > >>> try { > >> > >>> > >>> val method: Method = > >> > >>> > >>> classOf[URLClassLoader].getDeclaredMethod("addURL", > >> classOf[URL]) > >> > >>> > >>> method.setAccessible(true) > >> > >>> > >>> method.invoke(loader, url) > >> > >>> > >>> } > >> > >>> > >>> catch { > >> > >>> > >>> case t: Throwable => { > >> > >>> > >>> throw new IOException("Error, could not add URL to > >> system > >> > >>> > >>> classloader") > >> > >>> > >>> } > >> > >>> > >>> } > >> > >>> > >>> } > >> > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> Sincerely, > >> > >>> > >>> > >> > >>> > >>> DB Tsai > >> > >>> > >>> ------------------------------------------------------- > >> > >>> > >>> My Blog: https://www.dbtsai.com > >> > >>> > >>> LinkedIn: https://www.linkedin.com/in/dbtsai > >> > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> On Sun, May 18, 2014 at 11:57 PM, Sean Owen < > >> so...@cloudera.com> > >> > >>> > wrote: > >> > >>> > >>> > >> > >>> > >>> > I might be stating the obvious for everyone, but the issue > >> > here is > >> > >>> > not > >> > >>> > >>> > reflection or the source of the JAR, but the ClassLoader. > The > >> > >>> basic > >> > >>> > >>> > rules are this. > >> > >>> > >>> > > >> > >>> > >>> > "new Foo" will use the ClassLoader that defines Foo. This > is > >> > >>> usually > >> > >>> > >>> > the ClassLoader that loaded whatever it is that first > >> > referenced > >> > >>> Foo > >> > >>> > >>> > and caused it to be loaded -- usually the ClassLoader > holding > >> > your > >> > >>> > >>> > other app classes. > >> > >>> > >>> > > >> > >>> > >>> > ClassLoaders can have a parent-child relationship. > >> ClassLoaders > >> > >>> > always > >> > >>> > >>> > look in their parent before themselves. > >> > >>> > >>> > > >> > >>> > >>> > (Careful then -- in contexts like Hadoop or Tomcat where > your > >> > app > >> > >>> is > >> > >>> > >>> > loaded in a child ClassLoader, and you reference a class > that > >> > >>> Hadoop > >> > >>> > >>> > or Tomcat also has (like a lib class) you will get the > >> > container's > >> > >>> > >>> > version!) > >> > >>> > >>> > > >> > >>> > >>> > When you load an external JAR it has a separate > ClassLoader > >> > which > >> > >>> > does > >> > >>> > >>> > not necessarily bear any relation to the one containing > your > >> > app > >> > >>> > >>> > classes, so yeah it is not generally going to make "new > Foo" > >> > work. > >> > >>> > >>> > > >> > >>> > >>> > Reflection lets you pick the ClassLoader, yes. > >> > >>> > >>> > > >> > >>> > >>> > I would not call setContextClassLoader. > >> > >>> > >>> > > >> > >>> > >>> > On Mon, May 19, 2014 at 12:00 AM, Sandy Ryza < > >> > >>> > sandy.r...@cloudera.com> > >> > >>> > >>> > wrote: > >> > >>> > >>> > > I spoke with DB offline about this a little while ago > and > >> he > >> > >>> > confirmed > >> > >>> > >>> > that > >> > >>> > >>> > > he was able to access the jar from the driver. > >> > >>> > >>> > > > >> > >>> > >>> > > The issue appears to be a general Java issue: you can't > >> > directly > >> > >>> > >>> > > instantiate a class from a dynamically loaded jar. > >> > >>> > >>> > > > >> > >>> > >>> > > I reproduced it locally outside of Spark with: > >> > >>> > >>> > > --- > >> > >>> > >>> > > URLClassLoader urlClassLoader = new > URLClassLoader(new > >> > >>> URL[] { > >> > >>> > new > >> > >>> > >>> > > File("myotherjar.jar").toURI().toURL() }, null); > >> > >>> > >>> > > > >> > >>> Thread.currentThread().setContextClassLoader(urlClassLoader); > >> > >>> > >>> > > MyClassFromMyOtherJar obj = new > >> MyClassFromMyOtherJar(); > >> > >>> > >>> > > --- > >> > >>> > >>> > > > >> > >>> > >>> > > I was able to load the class with reflection. > >> > >>> > >>> > > >> > >>> > >>> > >> > >>> > > >> > >>> > >> > >> > >> > >> > >> > > >> >