Re: [question] Good Course to learn beam

[P Singh]

Hi,

Yeah I agree this book covers pretty much what you need as beginner, though
I would suggest building stuff along reading.

On Wed, 31 Aug 2022 at 03:08, Adeel  wrote:

> Hi,
>
> The packt ebook is a lot better, guide yourself practically as you
> read the book. Plus, it is from beam committer.
>
>
> https://www.packtpub.com/product/building-big-data-pipelines-with-apache-beam/9781800564930
>
> Thanks,
> Adeel
>
> On Tue, 30 Aug 2022 at 22:06, Evan Galpin  wrote:
> >
> > +dev for additional visibility/input
> >
> > On Mon, Aug 29, 2022 at 11:10 AM Leandro Nahabedian via user <
> user@beam.apache.org> wrote:
> >>
> >> Hi community!
> >>
> >> I'm looking for a good course to learn apache beam and I saw this one
> which I believe is good, since it has very good feedback from students.
> What do you think?
> >>
> >> Thanks in advance for your help
> >>
> >> Cheers,
> >> Leandro
> >>
> >> --
> >>
> >> __
> >> Leandro Nahabedian
> >> Data Engineer
> >> O: 908.883.4369
> >>
>

Re: [Question] Beam SQL failed with NPE

[Zheng Ni]

Seems
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler"
is not a valid class name passed to classloader.

Java Doc:

Any class name provided as a String
 parameter
to methods in ClassLoader must be a binary name as defined by The Java™
Language Specification.

Examples of valid class names include:

   "java.lang.String"
   "javax.swing.JSpinner$DefaultEditor"
   "java.security.KeyStore$Builder$FileBuilder$1"
   "java.net.URLClassLoader$3$1"


Code extract:
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader
public final IClass loadIClass(String fieldDescriptor) throws
ClassNotFoundException {
 
 result = this.findIClass(fieldDescriptor);
 
}



On Tue, Aug 30, 2022 at 3:55 PM Zheng Ni  wrote:

> Hi Brain & Singh,
>
> Thanks for the reply. Actually, beam version from 2.35.0 to 2.41.0 uses
> the same calcite version, which is 1.28.0. So i do not think there is any
> difference between them. but I will give it a try.
>
> Basides, i found beam try to load
> class 
> "org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler"
> by the context classloader. In my app, i have a customized CL to do this. I
> could solve the NPE issue with some code change inside my CL cause this
> class is a nest class. After renaming to
> "org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata$ExplainVisibility$Handler"
> my CL could find the resource.
> However, there will be another issue pop up.
>
> 1
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader..loadIClass()
>  start to be exec
> 2.
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.ClassLoaderIClassLoader.findIClass()
>  will call my CL to find the above class.
> 3. 
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.defineIClass()
> will save the returned class to a map using value from func
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getDescriptor()
>  as the key. In my case, the key is
> "org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata$ExplainVisibility$Handler"
> 4.
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.loadIClass()
> then will check the map with a diff key, which value is
> "org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler"
> 5. Got exception:
> findIClass("Lorg/apache/beam/vendor/calcite/v1_28_0/org/apache/calcite/rel/metadata/BuiltInMetadata/ExplainVisibility/Handler;")"
> did not call "defineIClass()"!?
>
>
> Stack trace:
> java.lang.Exception: java.lang.reflect.InvocationTargetException :
> java.lang.reflect.InvocationTargetException :
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.InternalCompilerException:
> "findIClass("Lorg/apache/beam/vendor/calcite/v1_28_0/org/apache/calcite/rel/metadata/BuiltInMetadata/ExplainVisibility/Handler;")"
> did not call "defineIClass()"!?
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:
> 332)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:
> 8673)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:
> 6689)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:
> 6661)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$14400(UnitCompiler.java:
> 231)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java:
> 6556)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java:
> 6551)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$ReferenceType.accept(Java.java:
> 4271)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:
> 6551)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$1300(UnitCompiler.java:
> 231)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$36.getInterfaces2(UnitCompiler.java:
> 10134)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getInterfaces(IClass.java:
> 498)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java:
> 264)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java:
> 238)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:
> 472)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompil

Re: [Question] Beam SQL failed with NPE

[Zheng Ni]

Hi Brain & Singh,

Thanks for the reply. Actually, beam version from 2.35.0 to 2.41.0 uses the
same calcite version, which is 1.28.0. So i do not think there is any
difference between them. but I will give it a try.

Basides, i found beam try to load
class 
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler"
by the context classloader. In my app, i have a customized CL to do this. I
could solve the NPE issue with some code change inside my CL cause this
class is a nest class. After renaming to
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata$ExplainVisibility$Handler"
my CL could find the resource.
However, there will be another issue pop up.

1
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader..loadIClass()
 start to be exec
2.
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.ClassLoaderIClassLoader.findIClass()
 will call my CL to find the above class.
3. 
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.defineIClass()
will save the returned class to a map using value from func
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getDescriptor()
 as the key. In my case, the key is
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata$ExplainVisibility$Handler"
4.
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.loadIClass()
then will check the map with a diff key, which value is
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler"
5. Got exception:
findIClass("Lorg/apache/beam/vendor/calcite/v1_28_0/org/apache/calcite/rel/metadata/BuiltInMetadata/ExplainVisibility/Handler;")"
did not call "defineIClass()"!?


Stack trace:
java.lang.Exception: java.lang.reflect.InvocationTargetException :
java.lang.reflect.InvocationTargetException :
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.InternalCompilerException:
"findIClass("Lorg/apache/beam/vendor/calcite/v1_28_0/org/apache/calcite/rel/metadata/BuiltInMetadata/ExplainVisibility/Handler;")"
did not call "defineIClass()"!?
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:
332)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:
8673)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:
6689)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:
6661)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$14400(UnitCompiler.java:
231)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java:
6556)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$22.visitReferenceType(UnitCompiler.java:
6551)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$ReferenceType.accept(Java.java:
4271)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:
6551)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$1300(UnitCompiler.java:
231)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$36.getInterfaces2(UnitCompiler.java:
10134)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getInterfaces(IClass.java:
498)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java:
264)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.IClass.getIMethods(IClass.java:
238)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:
472)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:
412)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:
231)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:
391)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:
386)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:
1692)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:
386)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:
359)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:
231)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:
333)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompile

Re: [question] Good Course to learn beam

[Adeel]

Hi,

The packt ebook is a lot better, guide yourself practically as you
read the book. Plus, it is from beam committer.

https://www.packtpub.com/product/building-big-data-pipelines-with-apache-beam/9781800564930

Thanks,
Adeel

On Tue, 30 Aug 2022 at 22:06, Evan Galpin  wrote:
>
> +dev for additional visibility/input
>
> On Mon, Aug 29, 2022 at 11:10 AM Leandro Nahabedian via user 
>  wrote:
>>
>> Hi community!
>>
>> I'm looking for a good course to learn apache beam and I saw this one which 
>> I believe is good, since it has very good feedback from students. What do 
>> you think?
>>
>> Thanks in advance for your help
>>
>> Cheers,
>> Leandro
>>
>> --
>>
>> __
>> Leandro Nahabedian
>> Data Engineer
>> O: 908.883.4369
>>

Re: [question] Good Course to learn beam

[Evan Galpin]

+dev  for additional visibility/input

On Mon, Aug 29, 2022 at 11:10 AM Leandro Nahabedian via user <
user@beam.apache.org> wrote:

> Hi community!
>
> I'm looking for a good course to learn apache beam and I saw this one
> 
> which I believe is good, since it has very good feedback from students.
> What do you think?
>
> Thanks in advance for your help
>
> Cheers,
> Leandro
>
> --
> [image: dialpad] 
> __
> *Leandro Nahabedian *
> Data Engineer
> O: 908.883.4369
>
>

Re: [Question] Beam SQL failed with NPE

[P Singh]

Hi,

Use apache-beam==2.37.0

On Wed, 31 Aug 2022 at 00:02, Brian Hulette via user 
wrote:

> Hi Zheng,
> Could you share a minimal example that reproduces the issue?
> Also, have you tried using Beam >2.35.0? I'm curious if this happens in
> the 2.41.0 release as well.
>
> Brian
>
> On Tue, Aug 30, 2022 at 11:10 AM Zheng Ni  wrote:
>
>> Hi There,
>>
>> I am using beam 2.35.0 to build a simple sql based pipeline but got below
>> error. I am trying to debug into the code and could not found this class:
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler.
>> Not sure if any lib missing in my project.
>>
>> Thanks,
>> Zheng
>>
>>
>> POM:
>> 
>> org.apache.beam
>> beam-sdks-java-extensions-sql
>> 2.35.0
>> 
>>
>>
>> Stack trace:
>> Caused by:
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.InternalCompilerException:
>> Compiling "GeneratedMetadata_ExplainVisibilityHandler" in Line 1, Column
>> 14: null
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:369)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:231)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:333)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:330)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$CompilationUnit.accept(Java.java:367)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:330)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:245)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:473)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:223)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:209)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:82)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:77)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java:361)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.generateCompileAndInstantiate(JaninoRelMetadataProvider.java:182)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:75)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:381)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:108)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQuery.isVisibleInExplain(RelMetadataQuery.java:857)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.explain_(RelWriterImpl.java:67)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.done(RelWriterImpl.java:151)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:252)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:2409)
>> at
>> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:2392)
>> at
>> org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(

Re: [Question] Beam SQL failed with NPE

[Brian Hulette via user]

Hi Zheng,
Could you share a minimal example that reproduces the issue?
Also, have you tried using Beam >2.35.0? I'm curious if this happens in the
2.41.0 release as well.

Brian

On Tue, Aug 30, 2022 at 11:10 AM Zheng Ni  wrote:

> Hi There,
>
> I am using beam 2.35.0 to build a simple sql based pipeline but got below
> error. I am trying to debug into the code and could not found this class:
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler.
> Not sure if any lib missing in my project.
>
> Thanks,
> Zheng
>
>
> POM:
> 
> org.apache.beam
> beam-sdks-java-extensions-sql
> 2.35.0
> 
>
>
> Stack trace:
> Caused by:
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.InternalCompilerException:
> Compiling "GeneratedMetadata_ExplainVisibilityHandler" in Line 1, Column
> 14: null
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:369)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:231)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:333)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:330)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$CompilationUnit.accept(Java.java:367)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:330)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:245)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:473)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:223)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:209)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:82)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:77)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java:361)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.generateCompileAndInstantiate(JaninoRelMetadataProvider.java:182)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:75)
> at
> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
> at
> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
> at
> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
> at
> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
> at
> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
> at
> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
> at
> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
> at
> org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:381)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:108)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQuery.isVisibleInExplain(RelMetadataQuery.java:857)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.explain_(RelWriterImpl.java:67)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.done(RelWriterImpl.java:151)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:252)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:2409)
> at
> org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:2392)
> at
> org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:189)
> at
> org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:112)
> at
> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.

[Question] Beam SQL failed with NPE

[Zheng Ni]

Hi There,

I am using beam 2.35.0 to build a simple sql based pipeline but got below
error. I am trying to debug into the code and could not found this class:
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.BuiltInMetadata.ExplainVisibility.Handler.
Not sure if any lib missing in my project.

Thanks,
Zheng


POM:

org.apache.beam
beam-sdks-java-extensions-sql
2.35.0



Stack trace:
Caused by:
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.InternalCompilerException:
Compiling "GeneratedMetadata_ExplainVisibilityHandler" in Line 1, Column
14: null
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:369)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:231)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:333)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler$1.visitCompilationUnit(UnitCompiler.java:330)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.Java$CompilationUnit.accept(Java.java:367)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:330)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:245)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:473)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:223)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:209)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:82)
at
org.apache.beam.vendor.calcite.v1_28_0.org.codehaus.commons.compiler.Cookable.cook(Cookable.java:77)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.compile(JaninoRelMetadataProvider.java:361)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.generateCompileAndInstantiate(JaninoRelMetadataProvider.java:182)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:75)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at
org.apache.beam.vendor.calcite.v1_28_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:381)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:108)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.metadata.RelMetadataQuery.isVisibleInExplain(RelMetadataQuery.java:857)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.explain_(RelWriterImpl.java:67)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.externalize.RelWriterImpl.done(RelWriterImpl.java:151)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:252)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:2409)
at
org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptUtil.toString(RelOptUtil.java:2392)
at
org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:189)
at
org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:112)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:171)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:110)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:482)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:363)
 ... 117 common frames omitted
Caused by: java.lang.NullPointerException: null
at
org.apache.beam.vendor.calcite.v1_

Re: Checkpointing on Google Cloud Dataflow Runner

[Reuven Lax via user]

Snapshots are expected to happen nearly instantaneously. While processing
is paused while the snapshot is in progress, the pause should usually be
very brief. It's true that Dataflow does not support automated snapshots -
you would have to create them yourself using a cron.

Checkpoints on Flink aren't simply automated snapshot mechanism.
Checkpoints are how Flink implements consistent, exactly-once processing.
Dataflow on the other hand continuously checkpoints records, so doesn't
need global checkpoints for exactly-once processing.

Reuven

On Tue, Aug 30, 2022 at 5:10 AM Will Baker  wrote:

> I looked into snapshots and they do seem useful for providing a means
> to save state and resume, however they aren't as seamless as I was
> hoping for with the automatic checkpointing that is supported by other
> runners. It looked like snapshots would be user initiated and would
> pause the pipeline while the snapshot was being created. I could
> imagine how this would be set up on an automated schedule, but would
> still prefer something more light-weight like checkpoints.
>
> On Mon, Aug 29, 2022 at 8:11 PM Reuven Lax  wrote:
> >
> > Google Cloud Dataflow does support snapshots. Is this what you were
> looking for?
> >
> > On Mon, Aug 29, 2022 at 4:04 PM Kenneth Knowles  wrote:
> >>
> >> Hi Will, David,
> >>
> >> I think you'll find the best source of answer for this sort of question
> on the user@beam list. I've put that in the To: line with a BCC: to the
> dev@beam list so everyone knows they can find the thread there. If I have
> misunderstood, and your question has to do with building Beam itself, feel
> free to move it back.
> >>
> >> Kenn
> >>
> >> On Mon, Aug 29, 2022 at 2:24 PM Will Baker  wrote:
> >>>
> >>> Hello!
> >>>
> >>> I am wondering about using checkpoints with Beam running on Google
> >>> Cloud Dataflow.
> >>>
> >>> The docs indicate that checkpoints are not supported by Google Cloud
> >>> Dataflow:
> https://beam.apache.org/documentation/runners/capability-matrix/additional-common-features-not-yet-part-of-the-beam-model/
> >>>
> >>> Is there a recommended approach to handling checkpointing on Google
> >>> Cloud Dataflow when using streaming sources like Kinesis and Kafka, so
> >>> that a pipeline could be resumed from where it left off if it needs to
> >>> be stopped or crashes for some reason?
> >>>
> >>> Thanks!
> >>> Will Baker
>

Re: Checkpointing on Google Cloud Dataflow Runner

[Will Baker]

I looked into snapshots and they do seem useful for providing a means
to save state and resume, however they aren't as seamless as I was
hoping for with the automatic checkpointing that is supported by other
runners. It looked like snapshots would be user initiated and would
pause the pipeline while the snapshot was being created. I could
imagine how this would be set up on an automated schedule, but would
still prefer something more light-weight like checkpoints.

On Mon, Aug 29, 2022 at 8:11 PM Reuven Lax  wrote:
>
> Google Cloud Dataflow does support snapshots. Is this what you were looking 
> for?
>
> On Mon, Aug 29, 2022 at 4:04 PM Kenneth Knowles  wrote:
>>
>> Hi Will, David,
>>
>> I think you'll find the best source of answer for this sort of question on 
>> the user@beam list. I've put that in the To: line with a BCC: to the 
>> dev@beam list so everyone knows they can find the thread there. If I have 
>> misunderstood, and your question has to do with building Beam itself, feel 
>> free to move it back.
>>
>> Kenn
>>
>> On Mon, Aug 29, 2022 at 2:24 PM Will Baker  wrote:
>>>
>>> Hello!
>>>
>>> I am wondering about using checkpoints with Beam running on Google
>>> Cloud Dataflow.
>>>
>>> The docs indicate that checkpoints are not supported by Google Cloud
>>> Dataflow:  
>>> https://beam.apache.org/documentation/runners/capability-matrix/additional-common-features-not-yet-part-of-the-beam-model/
>>>
>>> Is there a recommended approach to handling checkpointing on Google
>>> Cloud Dataflow when using streaming sources like Kinesis and Kafka, so
>>> that a pipeline could be resumed from where it left off if it needs to
>>> be stopped or crashes for some reason?
>>>
>>> Thanks!
>>> Will Baker