Hi Brain & Singh,
Thanks for the reply. Actually, beam version from 2.35.0 to 2.41.0 uses the
same calcite version, which is 1.28.0. So i do not think there is any
difference between them. but I will give it a try.
Basides, i found beam try to load
class
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler"
by the context classloader. In my app, i have a customized CL to do this. I
could solve the NPE issue with some code change inside my CL cause this
class is a nest class. After renaming to
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata$ExplainVisibility$Handler"
my CL could find the resource.
However, there will be another issue pop up.
1
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader..loadIClass()
start to be exec
2.
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.ClassLoaderIClassLoader.findIClass()
will call my CL to find the above class.
3.
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.defineIClass()
will save the returned class to a map using value from func
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getDescriptor()
as the key. In my case, the key is
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata$ExplainVisibility$Handler"
4.
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.loadIClass()
then will check the map with a diff key, which value is
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler"
5. Got exception:
findIClass("Lorg/apache/beam/vendor/calcite/v1_28_0/org/apache/calcite/rel/metadata/BuiltInMetadata/ExplainVisibility/Handler;")"
did not call "defineIClass()"!?
Stack trace:
java.lang.Exception: java.lang.reflect.InvocationTargetException :
java.lang.reflect.InvocationTargetException :
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.InternalCompilerException:
"findIClass("Lorg/apache/beam/vendor/calcite/v1_28_0/org/apache/calcite/rel/metadata/BuiltInMetadata/ExplainVisibility/Handler;")"
did not call "defineIClass()"!?
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:
332)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:
8673)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:
6689)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:
6661)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$14400(UnitCompiler.java:
231)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java:
6556)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java:
6551)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$ReferenceType.accept(Java.java:
4271)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:
6551)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$1300(UnitCompiler.java:
231)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$36.getInterfaces2(UnitCompiler.java:
10134)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getInterfaces(IClass.java:
498)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java:
264)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java:
238)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:
472)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:
412)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:
231)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:
391)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:
386)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:
1692)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:
386)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:
359)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:
231)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:
333)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:
330)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$CompilationUnit.accept(Java.java:
367)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:
330)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:
245)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:
473)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:
223)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:
209)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:
82)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:
77)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java:
361)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.generateCompileAndInstantiate(JaninoRelMetadataProvider.java:
182)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:
75)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:
165)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:
3529)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:
2278)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:
2155)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:
2045)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.get(LocalCache.java:
3951)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:
3974)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:
4958)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:
381)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:
108)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQuery.isVisibleInExplain(RelMetadataQuery.java:
857)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.explain_(RelWriterImpl.java:
67)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.done(RelWriterImpl.java:
151)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:
252)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:
2409)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:
2392)
at
org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:
189)
at
org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:
112)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:
171)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:
110)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java: 548)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java: 482)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java: 363)
On Tue, Aug 30, 2022 at 11:40 AM P Singh <[email protected]>
wrote:
> Hi,
>
> Use apache-beam==2.37.0
>
> On Wed, 31 Aug 2022 at 00:02, Brian Hulette via user <[email protected]>
> wrote:
>
>> Hi Zheng,
>> Could you share a minimal example that reproduces the issue?
>> Also, have you tried using Beam >2.35.0? I'm curious if this happens in
>> the 2.41.0 release as well.
>>
>> Brian
>>
>> On Tue, Aug 30, 2022 at 11:10 AM Zheng Ni <[email protected]> wrote:
>>
>>> Hi There,
>>>
>>> I am using beam 2.35.0 to build a simple sql based pipeline but got
>>> below error. I am trying to debug into the code and could not found this
>>> class:
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler.
>>> Not sure if any lib missing in my project.
>>>
>>> Thanks,
>>> Zheng
>>>
>>>
>>> POM:
>>> <dependency>
>>> <groupId>org.apache.beam</groupId>
>>> <artifactId>beam-sdks-java-extensions-sql</artifactId>
>>> <version>2.35.0</version>
>>> </dependency>
>>>
>>>
>>> Stack trace:
>>> Caused by:
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.InternalCompilerException:
>>> Compiling "GeneratedMetadata_ExplainVisibilityHandler" in Line 1, Column
>>> 14: null
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:369)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:231)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:333)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:330)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$CompilationUnit.accept(Java.java:367)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:330)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:245)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:473)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:223)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:209)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:82)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:77)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java:361)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.generateCompileAndInstantiate(JaninoRelMetadataProvider.java:182)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:75)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:381)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:108)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQuery.isVisibleInExplain(RelMetadataQuery.java:857)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.explain_(RelWriterImpl.java:67)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.done(RelWriterImpl.java:151)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:252)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:2409)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:2392)
>>> at
>>> org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:189)
>>> at
>>> org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:112)
>>> at
>>> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:171)
>>> at
>>> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:110)
>>> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
>>> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:482)
>>> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:363)
>>> ... 117 common frames omitted
>>> Caused by: java.lang.NullPointerException: null
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.ReflectionIClass.getDescriptor2(ReflectionIClass.java:125)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getDescriptor(IClass.java:529)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.defineIClass(IClassLoader.java:396)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:93)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:319)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:8673)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6689)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6661)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$14400(UnitCompiler.java:231)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java:6556)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java:6551)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$ReferenceType.accept(Java.java:4271)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6551)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$1300(UnitCompiler.java:231)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$36.getInterfaces2(UnitCompiler.java:10134)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getInterfaces(IClass.java:498)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java:264)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java:238)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:472)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:412)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:231)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:391)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:386)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1692)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:386)
>>> at
>>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:359)
>>>
>>
