Seems "org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler" is not a valid class name passed to classloader.
Java Doc: Any class name provided as a String <https://docs.oracle.com/javase/7/docs/api/java/lang/String.html> parameter to methods in ClassLoader must be a binary name as defined by The Java™ Language Specification. Examples of valid class names include: "java.lang.String" "javax.swing.JSpinner$DefaultEditor" "java.security.KeyStore$Builder$FileBuilder$1" "java.net.URLClassLoader$3$1" Code extract: org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader public final IClass loadIClass(String fieldDescriptor) throws ClassNotFoundException { .... result = this.findIClass(fieldDescriptor); .... } On Tue, Aug 30, 2022 at 3:55 PM Zheng Ni <nizheng1...@gmail.com> wrote: > Hi Brain & Singh, > > Thanks for the reply. Actually, beam version from 2.35.0 to 2.41.0 uses > the same calcite version, which is 1.28.0. So i do not think there is any > difference between them. but I will give it a try. > > Basides, i found beam try to load > class > "org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler" > by the context classloader. In my app, i have a customized CL to do this. I > could solve the NPE issue with some code change inside my CL cause this > class is a nest class. After renaming to > "org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata$ExplainVisibility$Handler" > my CL could find the resource. > However, there will be another issue pop up. > > 1 > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader..loadIClass() > start to be exec > 2. > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.ClassLoaderIClassLoader.findIClass() > will call my CL to find the above class. > 3. > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.defineIClass() > will save the returned class to a map using value from func > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getDescriptor() > as the key. In my case, the key is > "org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata$ExplainVisibility$Handler" > 4. > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.loadIClass() > then will check the map with a diff key, which value is > "org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler" > 5. Got exception: > findIClass("Lorg/apache/beam/vendor/calcite/v1_28_0/org/apache/calcite/rel/metadata/BuiltInMetadata/ExplainVisibility/Handler;")" > did not call "defineIClass()"!? > > > Stack trace: > java.lang.Exception: java.lang.reflect.InvocationTargetException : > java.lang.reflect.InvocationTargetException : > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.InternalCompilerException: > "findIClass("Lorg/apache/beam/vendor/calcite/v1_28_0/org/apache/calcite/rel/metadata/BuiltInMetadata/ExplainVisibility/Handler;")" > did not call "defineIClass()"!? > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java: > 332) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java: > 8673) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java: > 6689) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java: > 6661) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$14400(UnitCompiler.java: > 231) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java: > 6556) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java: > 6551) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$ReferenceType.accept(Java.java: > 4271) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java: > 6551) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$1300(UnitCompiler.java: > 231) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$36.getInterfaces2(UnitCompiler.java: > 10134) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getInterfaces(IClass.java: > 498) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java: > 264) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java: > 238) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java: > 472) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java: > 412) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java: > 231) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java: > 391) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java: > 386) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java: > 1692) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java: > 386) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java: > 359) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java: > 231) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java: > 333) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java: > 330) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$CompilationUnit.accept(Java.java: > 367) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java: > 330) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java: > 245) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java: > 473) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java: > 223) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java: > 209) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java: > 82) > at > org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java: > 77) > at > org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java: > 361) > at > org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.generateCompileAndInstantiate(JaninoRelMetadataProvider.java: > 182) > at > org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java: > 75) > at > org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java: > 165) > at > org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java: > 3529) > at > org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java: > 2278) > at > org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java: > 2155) > at > org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java: > 2045) > at > org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.get(LocalCache.java: > 3951) > at > org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java: > 3974) > at > org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java: > 4958) > at > org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java: > 381) > at > org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java: > 108) > at > org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQuery.isVisibleInExplain(RelMetadataQuery.java: > 857) > at > org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.explain_(RelWriterImpl.java: > 67) > at > org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.done(RelWriterImpl.java: > 151) > at > org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java: > 252) > at > org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java: > 2409) > at > org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java: > 2392) > at > org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java: > 189) > at > org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java: > 112) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java: > 171) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java: > 110) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java: 548) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java: 482) > at org.apache.beam.sdk.values.PCollection.apply(PCollection.java: 363) > > On Tue, Aug 30, 2022 at 11:40 AM P Singh <bigdatadevelop...@gmail.com> > wrote: > >> Hi, >> >> Use apache-beam==2.37.0 >> >> On Wed, 31 Aug 2022 at 00:02, Brian Hulette via user < >> user@beam.apache.org> wrote: >> >>> Hi Zheng, >>> Could you share a minimal example that reproduces the issue? >>> Also, have you tried using Beam >2.35.0? I'm curious if this happens in >>> the 2.41.0 release as well. >>> >>> Brian >>> >>> On Tue, Aug 30, 2022 at 11:10 AM Zheng Ni <nizheng1...@gmail.com> wrote: >>> >>>> Hi There, >>>> >>>> I am using beam 2.35.0 to build a simple sql based pipeline but got >>>> below error. I am trying to debug into the code and could not found this >>>> class: >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler. >>>> Not sure if any lib missing in my project. >>>> >>>> Thanks, >>>> Zheng >>>> >>>> >>>> POM: >>>> <dependency> >>>> <groupId>org.apache.beam</groupId> >>>> <artifactId>beam-sdks-java-extensions-sql</artifactId> >>>> <version>2.35.0</version> >>>> </dependency> >>>> >>>> >>>> Stack trace: >>>> Caused by: >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.InternalCompilerException: >>>> Compiling "GeneratedMetadata_ExplainVisibilityHandler" in Line 1, Column >>>> 14: null >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:369) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:231) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:333) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:330) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$CompilationUnit.accept(Java.java:367) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:330) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:245) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:473) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:223) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:209) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:82) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:77) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java:361) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.generateCompileAndInstantiate(JaninoRelMetadataProvider.java:182) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:75) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.get(LocalCache.java:3951) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:381) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:108) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQuery.isVisibleInExplain(RelMetadataQuery.java:857) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.explain_(RelWriterImpl.java:67) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.done(RelWriterImpl.java:151) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:252) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:2409) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:2392) >>>> at >>>> org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:189) >>>> at >>>> org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:112) >>>> at >>>> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:171) >>>> at >>>> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:110) >>>> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548) >>>> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:482) >>>> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:363) >>>> ... 117 common frames omitted >>>> Caused by: java.lang.NullPointerException: null >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.ReflectionIClass.getDescriptor2(ReflectionIClass.java:125) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getDescriptor(IClass.java:529) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.defineIClass(IClassLoader.java:396) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:93) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:319) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:8673) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6689) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6661) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$14400(UnitCompiler.java:231) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java:6556) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java:6551) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$ReferenceType.accept(Java.java:4271) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6551) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$1300(UnitCompiler.java:231) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$36.getInterfaces2(UnitCompiler.java:10134) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getInterfaces(IClass.java:498) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java:264) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java:238) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:472) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:412) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:231) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:391) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:386) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1692) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:386) >>>> at >>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:359) >>>> >>>
