Have you explored processing time timers? https://beam.apache.org/releases/javadoc/2.43.0/org/apache/beam/sdk/state/TimeDomain.html#PROCESSING_TIME
On Wed, 23 Nov 2022 at 13:46, Sigalit Eliazov <e.siga...@gmail.com> wrote: > Hi all, > > the flow in our pipeline is: > > 1. read event X from kafka. open fixed window of 30 sec. > 2. read event subscription from kafka. open GlobalWindow and store a > state of all subscriptions. > 3. match X and Y using key and if there is a match send an event to > another kafka topic. (we use the state as side input) > > if a user unsubscribes (meaning we read from a different source an > event of unsubscribe) we would like to delete the relevant entry from > the state. > Can this be achieved only using state and not using some external cache/db? > > I am aware there is an option to add timers on state but the > expiration logic is not time based. > Any suggestions? > > Thanks in advance > Sigalit >
