Ok thank you, that brings some light into the matter. I've checked it further and saw that they are creating a new ClassLoader which is based on the URLClassLoader with an empty array of URLs. This still seems to work in the normal state and has only a problem in Twill.
For now, I've "fixed" it by manually loading the necessary classes from a classloader I'd get from any class, which is defined in the same jar (SchemaChangeException.class.getClassLoader().loadClass(className)). This can now find the correct classes, since I get exactly the classloader you've mentioned. So now, instead of using this dirty hack, I can just remember the thread context classloader of my runnable thread and set it as the parent. Thank you very much, I think that will do the trick. On Tue, Jun 10, 2014 at 1:00 AM, Terence Yim <[email protected]> wrote: > Hi Peter, > > I am not familiar with how janino works, but it seems to me that it > may not be using context ClassLoader to load classes or as least the > thread that is compiling the generated class does not have the context > ClassLoader set properly. > > The way that Twill works is pretty straightforward. It creates a > "launcher.jar", which has no dependency on any library and start the > JVM in a YARN container like this: > > java -cp launcher.jar .... > > Hence the system classloader has no user/library classes, but only the > Launcher class. > > Then in the Launcher.main() method, it creates a URLClassLoader, using > all the jars + .class files inside the "container.jar" file, to load > the user TwillRunnable. It also sets it as the context ClassLoader of > the thread that calls the "run()" method. So, if you want to load > class manually (through ClassLoader or Class.forName) in a different > thread than the "run()" thread, you'll have to use set the context > ClassLoader of that thread or explicitly construct the ClassLoader > with the correct parent ClassLoader. > > Terence > > On Mon, Jun 9, 2014 at 12:32 PM, Peter Grman <[email protected]> > wrote: > > Hi, I'm currently in the process of porting an Apache project for > Big-Data > > analysis (Apache Drill) to YARN. I'm using Twill for that manner and it > > almost works. > > > > The last problem, I can't seem to figure out, is an exception from > janino. > > Apache Drill uses the compiler to create classes on the fly and that > works > > fine when I run Drill directly. But when run in Apache Twill I get the > > exception from Janino that a class couldn't be loaded. I've added this > > specific class as a dependency and also could find it in the container > > jar-file which was created by Twill. > > > > And the class really is there, because the class which can't be found, is > > the same as the exception which is thrown. - Yes that same class to which > > the original CompileException is transformed in the end, is the same as > the > > class which is imported as first and can't be found. I suspect that there > > is a problem with how twill loads the jars and that's the reason why > janino > > can't find the class - since it works running it directly from command > line > > - but I don't know where to look anymore. > > > > Any ideas? > > Thanks for your help > > > > The stacktrace from the exception is here: > > https://gist.github.com/pgrm/d29d33f356601b1a1a3f > > The class definition which can't be compiled alone is here: > > https://gist.github.com/pgrm/a589edb64317f5cb1207 > > And my Twill-Application definition can be found here: > > > https://github.com/pgrm/incubator-drill/blob/YARN-support/yarn-integration/src/main/java/org/apache/drill/yarn/integration/RunDrill.java > > > > And here also one more the the exception trace: > > > > 2014-06-09T18:30:40,093Z ERROR o.a.d.e.p.i.p.ProjectRecordBatch [zk1] > > [37daf04b-7d82-4d2f-987c-59851f2aeafe:frag:0:0] > > AbstractSingleRecordBatch:next(AbstractSingleRecordBatch.java:60) - > > Failure during query > > org.apache.drill.exec.exception.SchemaChangeException: Failure while > > attempting to load generated class > > > > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:243) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) > > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) > > > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > at > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > > > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) > > at > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) > > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) > > > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > Caused by: org.apache.drill.exec.exception.ClassTransformationException: > > Failure generating transformation classes for value: > > package org.apache.drill.exec.test.generated; > > > > > > import org.apache.drill.exec.exception.SchemaChangeException; > > import org.apache.drill.exec.expr.holders.BitHolder; > > import org.apache.drill.exec.expr.holders.VarCharHolder; > > import org.apache.drill.exec.ops.FragmentContext; > > import org.apache.drill.exec.record.RecordBatch; > > import org.apache.drill.exec.vector.RepeatedVarCharVector; > > import org.apache.drill.exec.vector.VarCharVector; > > import > org.apache.drill.exec.vector.complex.impl.RepeatedVarCharReaderImpl; > > > > public class ProjectorGen0 { > > > > > > RepeatedVarCharVector vv0; > > RepeatedVarCharReaderImpl reader4; > > VarCharVector vv5; > > > > > > public boolean doEval(int inIndex, int outIndex) > > throws SchemaChangeException > > > > { > > { > > VarCharHolder out3 = new VarCharHolder(); > > > > complex: > > vv0 .getAccessor().getReader().setPosition((inIndex)); > > > > reader4 .read(0, out3); > > BitHolder out8 = new BitHolder(); > > > > out8 .value = 1; > > if (!vv5 .getMutator().setSafe((outIndex), out3)) { > > > > out8 .value = 0; > > } > > if (out8 .value == 0) { > > return false; > > } > > > > } > > { > > return true; > > > > } > > } > > > > > > public void doSetup(FragmentContext context, RecordBatch incoming, > > RecordBatch outgoing) > > throws SchemaChangeException > > > > { > > { > > int[] fieldIds1 = new int[ 1 ] ; > > > > fieldIds1 [ 0 ] = 0; > > Object tmp2 = > > (incoming).getValueAccessorById(RepeatedVarCharVector.class, > > fieldIds1).getValueVector(); > > if (tmp2 == null) { > > throw new SchemaChangeException("Failure while loading > > vector vv0 with id: > > org.apache.drill.exec.record.TypedFieldId@1cf4a5a0."); > > } > > vv0 = ((RepeatedVarCharVector) tmp2); > > > > reader4 = ((RepeatedVarCharReaderImpl) vv0 > > .getAccessor().getReader()); > > int[] fieldIds6 = new int[ 1 ] ; > > > > fieldIds6 [ 0 ] = 0; > > Object tmp7 = > > (outgoing).getValueAccessorById(VarCharVector.class, > > fieldIds6).getValueVector(); > > > > if (tmp7 == null) { > > throw new SchemaChangeException("Failure while loading > > vector vv5 with id: > > org.apache.drill.exec.record.TypedFieldId@1ce776c0."); > > } > > vv5 = ((VarCharVector) tmp7); > > > > } > > } > > > > > > } > > > > at > org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:302) > > at > org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185) > > > > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) > > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) > > > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > at > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > > > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) > > at > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) > > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) > > > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > Caused by: org.codehaus.commons.compiler.CompileException: Line 4, > > Column 8: Imported class > > "org.apache.drill.exec.exception.SchemaChangeException" could not be > > loaded > > at > org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:9014) > > at > org.codehaus.janino.UnitCompiler.import2(UnitCompiler.java:192) > > at > org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:104) > > at > org.codehaus.janino.UnitCompiler$1.visitSingleTypeImportDeclaration(UnitCompiler.java:166) > > at > org.codehaus.janino.Java$CompilationUnit$SingleTypeImportDeclaration.accept(Java.java:171) > > > > at org.codehaus.janino.UnitCompiler.<init>(UnitCompiler.java:164) > > at > org.apache.drill.exec.compile.JaninoClassCompiler.getClassByteCode(JaninoClassCompiler.java:53) > > at > org.apache.drill.exec.compile.QueryClassLoader.getClassByteCode(QueryClassLoader.java:69) > > > > at > org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:256) > > at > org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185) > > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240) > > > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) > > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > > > at > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) > > > > at > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) > > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > > > 2014-06-09T18:30:40,109Z ERROR o.a.d.e.p.i.ScreenCreator$ScreenRoot > > [zk1] [37daf04b-7d82-4d2f-987c-59851f2aeafe:frag:0:0] > > ErrorHelper:logAndConvertError(ErrorHelper.java:60) - Error > > 5625373a-729d-4f3f-a262-6f46f2f879d9: Screen received stop request > > sent. > > org.codehaus.commons.compiler.CompileException: Line 4, Column 8: > > Imported class "org.apache.drill.exec.exception.SchemaChangeException" > > could not be loaded > > at > org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:9014) > > at > org.codehaus.janino.UnitCompiler.import2(UnitCompiler.java:192) > > at > org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:104) > > at > org.codehaus.janino.UnitCompiler$1.visitSingleTypeImportDeclaration(UnitCompiler.java:166) > > at > org.codehaus.janino.Java$CompilationUnit$SingleTypeImportDeclaration.accept(Java.java:171) > > > > at org.codehaus.janino.UnitCompiler.<init>(UnitCompiler.java:164) > > at > org.apache.drill.exec.compile.JaninoClassCompiler.getClassByteCode(JaninoClassCompiler.java:53) > > at > org.apache.drill.exec.compile.QueryClassLoader.getClassByteCode(QueryClassLoader.java:69) > > > > at > org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:256) > > at > org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185) > > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240) > > > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) > > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > > > at > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) > > > > at > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) > > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) >
