Hi all, I've spent nearly 2 weeks trying to figure a solution to my requirement as below. If anyone can advise, that would be great.
1. There're going to be 2000 transactions per second as StreamRaw, I'm going to tumbleWindow them as StreamA, let's say every 15 seconds. Then I'm going to countWindow StreamA as StreamB, let's say every 20 events. 2. For every event in StreamRaw as E, I need to find exact one event in StreamB which is earlier than E and closest to E. Then some comparison will be proceeded. For example, if timestamp in E is 9:46:38, there should be an event in StreamB with timestamp 9:46:30 because I use 15 seconds interval. I tried CEP using StreamRaw, however, I didn't figure out how to involve StreamB and get the exact one event in condition method. I tried tableAPI and SQL, it throws time attribute error during the second window method. *window(Tumble).group().select().window(Slide).group().select()* Seems there's no way to tell Flink the time attribute after the first window.group(). I then tried to convert it into table first then leftoutJoin them. But Flink tells me it's not supported. Is Flink able to do this? If not, I'll go for other alternatives. Thanks again if someone can help.
