Re: How windowing is implemented on Flink runner

2024-06-12 Thread Ruben Vargas
I imagined it but wasn't sure! Thanks for the clarification! On Wed, Jun 12, 2024 at 1:42 PM Robert Bradshaw via user wrote: > > Beam implements Windowing itself (via state and timers) rather than > deferring to Flink's implementation. > > On Wed, Jun 12, 2024 at 11:55 AM Ruben Vargas wrote: >

Re: How windowing is implemented on Flink runner

2024-06-12 Thread Robert Bradshaw via user
Beam implements Windowing itself (via state and timers) rather than deferring to Flink's implementation. On Wed, Jun 12, 2024 at 11:55 AM Ruben Vargas wrote: > > Hello guys > > May be a silly question, > > But in the Flink runner, the window implementation uses the Flink > windowing? Does that

How windowing is implemented on Flink runner

2024-06-12 Thread Ruben Vargas
Hello guys May be a silly question, But in the Flink runner, the window implementation uses the Flink windowing? Does that mean the runner will have performance issues like Flink itself? see this: https://issues.apache.org/jira/browse/FLINK-7001 I'm asking because I see the issue, it mentions

Re: Paralalelism of a side input

2024-06-12 Thread Robert Bradshaw via user
On Wed, Jun 12, 2024 at 7:56 AM Ruben Vargas wrote: > > The approach looks good. but one question > > My understanding is that this will schedule for example 8 operators across > the workers, but only one of them will be processing, the others remain idle? > Are those consuming resources in

Re: Paralalelism of a side input

2024-06-12 Thread Ruben Vargas
The approach looks good. but one question My understanding is that this will schedule for example 8 operators across the workers, but only one of them will be processing, the others remain idle? Are those consuming resources in some way? I'm assuming may be is not significant. Thanks. El El

Re: Apache Bean on GCP / Forcing to use py 3.11

2024-06-12 Thread XQ Hu via user
Any reason to use this? RUN pip install avro-python3 pyarrow==0.15.1 apache-beam[gcp]==2.30.0 pandas-datareader==0.9.0 It is typically recommended to use the latest Beam and build the docker image using the requirements released for each Beam, for example,