Re: Extract Multiple Features in a Flexible and Extensible Way

I realized that you can just use normal Python control flow while constructing your pipeline. No additional Beam functionality necessary. On Thu, Feb 20, 2020 at 2:14 PM Xander Song wrote: > I have written a feature extraction pipeline in which I extract two > features using ParDos and combine t

Re: Implementing custom session with max event/element count

Hi Kenn, Let me know if I'm missing something in below email. I was going through stateful processing blog posts but that wouldn't solve the issue since Window cannot be broken on operations done after WindowFn. Wondering what I could leverage to enforce max events in a Window that has a gap durat

Re: Beam 2.19.0 / Flink 1.9.1 - Session cluster error when submitting job "Multiple environments cannot be created in detached mode"

Hi Tobi, This seems like a bug with Beam 2.19. I filed https://issues.apache.org/jira/browse/BEAM-9345 to track the issue. > What puzzles me is that the session cluster should be allowed to have multiple environments in detached mode - or am I wrong? It looks like that check is removed in Flink

Extract Multiple Features in a Flexible and Extensible Way

I have written a feature extraction pipeline in which I extract two features using ParDos and combine the results with a CoGroupByKey. with beam.Pipeline() as p: input = p | 'read input' >> beam.io.ReadFromText(input_path) first_feature = input | 'extract first feature' >> beam.ParDo(Ext

Beam 2.19.0 / Flink 1.9.1 - Session cluster error when submitting job "Multiple environments cannot be created in detached mode"

Hello, I am trying to upgrade from a Flink session cluster 1.8 to 1.9 and from Beam 2.16.0 to 2.19.0. Everything went quite smoothly, the local runner and the local Flink runner work flawlessly. However when I: 1. Generate a Beam jar for the FlinkRunner via maven (mvn package -PFlinkRunner) 2