Re: [DISCUSS] REST API behaviour when user main method throws error

2023-11-23 Thread Danny Cranmer
Hey David, Thanks for the feedback. > For streaming, this is probably something we could address by book-keeping jobs submitted by the jar and canceling them on exception. I considered two options. 1/ cancel on error as you proposed. This has the downside that it could start processing data temp

Re: [DISCUSS] REST API behaviour when user main method throws error

2023-11-20 Thread David Morávek
Hi Danny, > My current proposal is that the REST API should not leave the Flink cluster in an inconsistent state. Regarding consistency, Flink only cares about individual jobs, but I can see your point. For streaming, this is probably something we could address by book-keeping jobs submitted by

[DISCUSS] REST API behaviour when user main method throws error

2023-11-14 Thread Danny Cranmer
Hey all, We run Flink clusters in session mode; we upload the user jar and then invoke "/jars/:jarid/run" [1] REST API endpoint. We have noticed a discrepancy in the run endpoint and were hoping to get some feedback from the community before proposing a FLIP or Jira to fix it. Some problem contex