Re: Session window ad sideinput
If you can make use of Java SDK, you can use GenerateSequence. With the python SDK, you should be able to use GenerateSequence via xlang. Do you have a Jira for the bugs with the PeriodicImpulse? Cheers Reza On Mon, 4 Jan 2021 at 20:12, Manninger, Matyas wrote: > Dear Reza, > > Thanks for the suggestion, that is the solution I was going for but > unfortunately PeriodicImpulse has some bugs. I also posted a question about > that in this mail list but no success there so far so I am looking for > alternatives. > > On Tue, 22 Dec 2020 at 12:36, Reza Ardeshir Rokni > wrote: > >> Hi, >> >> Why the need for session windows? Could you make use of a Global Window >> for the side input, as per the following pattern: >> >> https://beam.apache.org/documentation/patterns/side-inputs/ >> >> Cheers >> Reza >> >> >> >> On Tue, 22 Dec 2020 at 01:17, Manninger, Matyas < >> matyas.mannin...@veolia.com> wrote: >> >>> Dear Beam users, >>> >>> I am writing a streaming pipeline that has static tables as side inputs. >>> These tables change from time to time and I want the side inputs to be >>> updated at some intervals. I am planning on triggering the update by >>> sending a message through pubsub. When a new message arrives, the side >>> input should be updated. I would like to do this with session windows but >>> on the beam documentation page session windows are depicted as lasting from >>> the first input in the window to the last input in the window and the gap >>> seems to not belong to any window. So if I would use this as a side >>> input how would my main stream be matched to windows? If I send a signal >>> every day and the gap is set to 1 hour, for example, would the window close >>> after 1 hour and for the next 23 hours all the elements in the main >>> stream would be matched to no side input? >>> >>> Thanks for any help or tips on how to solve this or what is the expected >>> behaviour. >>> >>> BR, >>> Matyas Manninger >>> >>
Re: Session window ad sideinput
Dear Reza, Thanks for the suggestion, that is the solution I was going for but unfortunately PeriodicImpulse has some bugs. I also posted a question about that in this mail list but no success there so far so I am looking for alternatives. On Tue, 22 Dec 2020 at 12:36, Reza Ardeshir Rokni wrote: > Hi, > > Why the need for session windows? Could you make use of a Global Window > for the side input, as per the following pattern: > > https://beam.apache.org/documentation/patterns/side-inputs/ > > Cheers > Reza > > > > On Tue, 22 Dec 2020 at 01:17, Manninger, Matyas < > matyas.mannin...@veolia.com> wrote: > >> Dear Beam users, >> >> I am writing a streaming pipeline that has static tables as side inputs. >> These tables change from time to time and I want the side inputs to be >> updated at some intervals. I am planning on triggering the update by >> sending a message through pubsub. When a new message arrives, the side >> input should be updated. I would like to do this with session windows but >> on the beam documentation page session windows are depicted as lasting from >> the first input in the window to the last input in the window and the gap >> seems to not belong to any window. So if I would use this as a side >> input how would my main stream be matched to windows? If I send a signal >> every day and the gap is set to 1 hour, for example, would the window close >> after 1 hour and for the next 23 hours all the elements in the main >> stream would be matched to no side input? >> >> Thanks for any help or tips on how to solve this or what is the expected >> behaviour. >> >> BR, >> Matyas Manninger >> >
Re: Session window ad sideinput
Hi, Why the need for session windows? Could you make use of a Global Window for the side input, as per the following pattern: https://beam.apache.org/documentation/patterns/side-inputs/ Cheers Reza On Tue, 22 Dec 2020 at 01:17, Manninger, Matyas wrote: > Dear Beam users, > > I am writing a streaming pipeline that has static tables as side inputs. > These tables change from time to time and I want the side inputs to be > updated at some intervals. I am planning on triggering the update by > sending a message through pubsub. When a new message arrives, the side > input should be updated. I would like to do this with session windows but > on the beam documentation page session windows are depicted as lasting from > the first input in the window to the last input in the window and the gap > seems to not belong to any window. So if I would use this as a side > input how would my main stream be matched to windows? If I send a signal > every day and the gap is set to 1 hour, for example, would the window close > after 1 hour and for the next 23 hours all the elements in the main > stream would be matched to no side input? > > Thanks for any help or tips on how to solve this or what is the expected > behaviour. > > BR, > Matyas Manninger >
Session window ad sideinput
Dear Beam users, I am writing a streaming pipeline that has static tables as side inputs. These tables change from time to time and I want the side inputs to be updated at some intervals. I am planning on triggering the update by sending a message through pubsub. When a new message arrives, the side input should be updated. I would like to do this with session windows but on the beam documentation page session windows are depicted as lasting from the first input in the window to the last input in the window and the gap seems to not belong to any window. So if I would use this as a side input how would my main stream be matched to windows? If I send a signal every day and the gap is set to 1 hour, for example, would the window close after 1 hour and for the next 23 hours all the elements in the main stream would be matched to no side input? Thanks for any help or tips on how to solve this or what is the expected behaviour. BR, Matyas Manninger
