Hi, I'm quite new to the world of stream and batch processing. I've been reading about pipelined regions in Flink and am quite confused by what it means. My specific problem involves a streaming job that looks like the following:
1. There is a Kafka source that takes in an input data that sets off a series of operations 2. As part of the first operation, I have an operator that produces multiple values, each of which has to be fed into several different operators in parallel 3. The operators each produce a result which I keyBy and merge together using the union operator 4. The merged result is then written to a Kafka sink The problem is that when one of the parallel operators throws an exception, all the tasks in the entire pipeline gets restarted including the source which then replays the input data and the process starts off once again. My question is if it's possible to make the tasks of only the branch that failed restart rather than the whole job. I do realize that it is possible to split up the job such that the first operator produces its output to a sink and having that as the source to the subsequent operations can mitigate the problem. I was just wondering if it's possible in the scenario that I have described above. In general, how can I "create" a pipelined region? Thanks, Sunny -- Secure Link Services Group Zürich: The Circle 37, 8058 Zürich-Airport, Switzerland Munich: Tal 44, 80331 München, Germany Dubai: Building 3, 3rd Floor, Dubai Design District, Dubai, United Arab Emirates Dhaka: Midas Center, Road 16, Dhanmondi, Dhaka 1209, Bangladesh Thimphu: Bhutan Innovation Tech Center, Babesa, P.O. Box 633, Thimphu, Bhutan Visit us: www.selise.ch <http://www.selise.ch> -- *Important Note: This e-mail and any attachment are confidential and may contain trade secrets and may well also be legally privileged or otherwise protected from disclosure. If you have received it in error, you are on notice of its status. Please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. If you are not the intended recipient please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation.*
