Hello! I have built a Flink pipeline that involve
1) Reading events from a Kinesis Stream 2) Create two DataStream (using .filter) with events of type 'A' going in stream1 and event of type 'B' going in stream2 3) Transform stream1 into a Table and use Table API to do a simple window tumble and group by to counts the events 4) Interval join stream1 with stream2 in order to filter out some event in stream1 that are not in stream2 5) Transform the result of the interval join into a table and use Table API to do a simple Tumble Window Group by to count the events 6) Join 3) and 5) and transform back to a stream that sinks to an output kinesis stream I have read the documentation that shows some examples of Unit Testing but I'm scratching my end to know how I'm going to be able to IT test my pipeline to make sure all the computation are correct given an exact input dataset? Is there a proper way of writing IT to test my pipepleine? Or will have I have to bring up a Flink cluster (with docker for example, fire events with a python scripts and then check the results by reading the output stream? Thank you!
