Re: Issue with Checkpoint restore( Beam pipeline)

2017-11-30 Thread Jins George
Thanks Aljoscha. I have not tried with 1.3. I will try and check the behavior. Regarding setting UIDs to operators from Beam, do you know if thats something planned for a near future release ? Thanks, Jins George On 11/30/2017 01:48 AM, Aljoscha Krettek wrote: Hi, I think you might be run

Re: Issue with Checkpoint restore( Beam pipeline)

2017-11-30 Thread Aljoscha Krettek
Hi, I think you might be running into a problem that is hard to solve with Flink 1.2 and Beam. As you noticed, it's a problem that Beam doesn't assign UIDs to operators, which is a problem. Flink 1.3 and even more Flink 1.4 are a bit more lenient in accepting changes to the graph, so you might

Issue with Checkpoint restore( Beam pipeline)

2017-11-29 Thread Jins George
Hi, I am running a Beam Pipeline on Flink 1.2 and facing an issue in restoring a job from checkpoint. If I modify my beam pipeline to add a new operator and try to restore from the externalized checkpoint, I get the error /java.lang.IllegalStateException: Invalid Invalid number of operator