Hello, I am having a Flink pipeline processing data in several overlapping(sliding) windows such that they span [t_i, t_i + T], where t_i is the window starting time and T is the window size. The overlap is such that t_(I+1) - t_i = T/6 (i.e on every window size there is 6 overlapping windows).
When deploying in my CI/CD process to Kubernetes, there are sometimes serialisation problems due to change of Flink DAG, checkpoint states and etc as a symptom of, for instance a change of definition of these classes (adding/removing a field). This leads to the fact that the process cannot start from the save point that I am saving during a deploy. How could this be managed in an efficient way? I understand that the way I am using windowing is not optimal here so lets not focus on those solutions. Currently, my only approach is: * Shutdown the streaming process in a controlled manner (replying the running version with new configs, terminating the stream when events are arriving after a certain timepjoint) * After termination, move the time-point ( offset, I am using Kafka) backwards in time, in this case T + eps to allow rebuffering of the windows. * Start the servicer reading from the new timepjoint, but not emitting any output events until it has passed a defined time-point (in this case the time-point of termination). Do you have any suggestions on how to improve this process? Best regards and thanks in advance for any input, William Flink Version: 1.6.2 -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
