Hi Peter, I am not familiar with how janino works, but it seems to me that it may not be using context ClassLoader to load classes or as least the thread that is compiling the generated class does not have the context ClassLoader set properly.
The way that Twill works is pretty straightforward. It creates a "launcher.jar", which has no dependency on any library and start the JVM in a YARN container like this: java -cp launcher.jar .... Hence the system classloader has no user/library classes, but only the Launcher class. Then in the Launcher.main() method, it creates a URLClassLoader, using all the jars + .class files inside the "container.jar" file, to load the user TwillRunnable. It also sets it as the context ClassLoader of the thread that calls the "run()" method. So, if you want to load class manually (through ClassLoader or Class.forName) in a different thread than the "run()" thread, you'll have to use set the context ClassLoader of that thread or explicitly construct the ClassLoader with the correct parent ClassLoader. Terence On Mon, Jun 9, 2014 at 12:32 PM, Peter Grman <[email protected]> wrote: > Hi, I'm currently in the process of porting an Apache project for Big-Data > analysis (Apache Drill) to YARN. I'm using Twill for that manner and it > almost works. > > The last problem, I can't seem to figure out, is an exception from janino. > Apache Drill uses the compiler to create classes on the fly and that works > fine when I run Drill directly. But when run in Apache Twill I get the > exception from Janino that a class couldn't be loaded. I've added this > specific class as a dependency and also could find it in the container > jar-file which was created by Twill. > > And the class really is there, because the class which can't be found, is > the same as the exception which is thrown. - Yes that same class to which > the original CompileException is transformed in the end, is the same as the > class which is imported as first and can't be found. I suspect that there > is a problem with how twill loads the jars and that's the reason why janino > can't find the class - since it works running it directly from command line > - but I don't know where to look anymore. > > Any ideas? > Thanks for your help > > The stacktrace from the exception is here: > https://gist.github.com/pgrm/d29d33f356601b1a1a3f > The class definition which can't be compiled alone is here: > https://gist.github.com/pgrm/a589edb64317f5cb1207 > And my Twill-Application definition can be found here: > https://github.com/pgrm/incubator-drill/blob/YARN-support/yarn-integration/src/main/java/org/apache/drill/yarn/integration/RunDrill.java > > And here also one more the the exception trace: > > 2014-06-09T18:30:40,093Z ERROR o.a.d.e.p.i.p.ProjectRecordBatch [zk1] > [37daf04b-7d82-4d2f-987c-59851f2aeafe:frag:0:0] > AbstractSingleRecordBatch:next(AbstractSingleRecordBatch.java:60) - > Failure during query > org.apache.drill.exec.exception.SchemaChangeException: Failure while > attempting to load generated class > > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:243) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > at > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) > at > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: org.apache.drill.exec.exception.ClassTransformationException: > Failure generating transformation classes for value: > package org.apache.drill.exec.test.generated; > > > import org.apache.drill.exec.exception.SchemaChangeException; > import org.apache.drill.exec.expr.holders.BitHolder; > import org.apache.drill.exec.expr.holders.VarCharHolder; > import org.apache.drill.exec.ops.FragmentContext; > import org.apache.drill.exec.record.RecordBatch; > import org.apache.drill.exec.vector.RepeatedVarCharVector; > import org.apache.drill.exec.vector.VarCharVector; > import org.apache.drill.exec.vector.complex.impl.RepeatedVarCharReaderImpl; > > public class ProjectorGen0 { > > > RepeatedVarCharVector vv0; > RepeatedVarCharReaderImpl reader4; > VarCharVector vv5; > > > public boolean doEval(int inIndex, int outIndex) > throws SchemaChangeException > > { > { > VarCharHolder out3 = new VarCharHolder(); > > complex: > vv0 .getAccessor().getReader().setPosition((inIndex)); > > reader4 .read(0, out3); > BitHolder out8 = new BitHolder(); > > out8 .value = 1; > if (!vv5 .getMutator().setSafe((outIndex), out3)) { > > out8 .value = 0; > } > if (out8 .value == 0) { > return false; > } > > } > { > return true; > > } > } > > > public void doSetup(FragmentContext context, RecordBatch incoming, > RecordBatch outgoing) > throws SchemaChangeException > > { > { > int[] fieldIds1 = new int[ 1 ] ; > > fieldIds1 [ 0 ] = 0; > Object tmp2 = > (incoming).getValueAccessorById(RepeatedVarCharVector.class, > fieldIds1).getValueVector(); > if (tmp2 == null) { > throw new SchemaChangeException("Failure while loading > vector vv0 with id: > org.apache.drill.exec.record.TypedFieldId@1cf4a5a0."); > } > vv0 = ((RepeatedVarCharVector) tmp2); > > reader4 = ((RepeatedVarCharReaderImpl) vv0 > .getAccessor().getReader()); > int[] fieldIds6 = new int[ 1 ] ; > > fieldIds6 [ 0 ] = 0; > Object tmp7 = > (outgoing).getValueAccessorById(VarCharVector.class, > fieldIds6).getValueVector(); > > if (tmp7 == null) { > throw new SchemaChangeException("Failure while loading > vector vv5 with id: > org.apache.drill.exec.record.TypedFieldId@1ce776c0."); > } > vv5 = ((VarCharVector) tmp7); > > } > } > > > } > > at > org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:302) > at > org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185) > > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > at > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) > at > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: org.codehaus.commons.compiler.CompileException: Line 4, > Column 8: Imported class > "org.apache.drill.exec.exception.SchemaChangeException" could not be > loaded > at > org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:9014) > at org.codehaus.janino.UnitCompiler.import2(UnitCompiler.java:192) > at org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:104) > at > org.codehaus.janino.UnitCompiler$1.visitSingleTypeImportDeclaration(UnitCompiler.java:166) > at > org.codehaus.janino.Java$CompilationUnit$SingleTypeImportDeclaration.accept(Java.java:171) > > at org.codehaus.janino.UnitCompiler.<init>(UnitCompiler.java:164) > at > org.apache.drill.exec.compile.JaninoClassCompiler.getClassByteCode(JaninoClassCompiler.java:53) > at > org.apache.drill.exec.compile.QueryClassLoader.getClassByteCode(QueryClassLoader.java:69) > > at > org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:256) > at > org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185) > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > at > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) > > at > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > 2014-06-09T18:30:40,109Z ERROR o.a.d.e.p.i.ScreenCreator$ScreenRoot > [zk1] [37daf04b-7d82-4d2f-987c-59851f2aeafe:frag:0:0] > ErrorHelper:logAndConvertError(ErrorHelper.java:60) - Error > 5625373a-729d-4f3f-a262-6f46f2f879d9: Screen received stop request > sent. > org.codehaus.commons.compiler.CompileException: Line 4, Column 8: > Imported class "org.apache.drill.exec.exception.SchemaChangeException" > could not be loaded > at > org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:9014) > at org.codehaus.janino.UnitCompiler.import2(UnitCompiler.java:192) > at org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:104) > at > org.codehaus.janino.UnitCompiler$1.visitSingleTypeImportDeclaration(UnitCompiler.java:166) > at > org.codehaus.janino.Java$CompilationUnit$SingleTypeImportDeclaration.accept(Java.java:171) > > at org.codehaus.janino.UnitCompiler.<init>(UnitCompiler.java:164) > at > org.apache.drill.exec.compile.JaninoClassCompiler.getClassByteCode(JaninoClassCompiler.java:53) > at > org.apache.drill.exec.compile.QueryClassLoader.getClassByteCode(QueryClassLoader.java:69) > > at > org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:256) > at > org.apache.drill.exec.ops.FragmentContext.getImplementationClass(FragmentContext.java:185) > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:240) > > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:57) > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > > at > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:45) > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:94) > > at > org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:80) > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744)
