Hi Allison Thanks for driving this FLIP. It's a valuable feature for batch jobs. This helps keep "Drop Per-Job Mode [1]" going.
+1 for this proposal. However, it seems that the change in this FLIP is not detailed enough. I have a few questions. 1. The config 'execution.attached' is not only used in per-job mode, but also in session mode to shutdown the cluster. IMHO, it's better to keep this option name. 2. This FLIP only mentions YARN mode. I believe this feature should work in both YARN and Kubernetes mode. 3. Within the attach mode, we support two features: execution.shutdown-on-attached-exit and client.heartbeat.timeout. These should also be taken into account. 4. The Application Mode will shut down once the job has been completed. So, if we use the flink client to poll job status via REST API for attach mode, there is a chance that the client will not be able to retrieve the job finish status. Perhaps FLINK-24113[3] will help with this. [1]https://issues.apache.org/jira/browse/FLINK-26000 [2] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#session-mode [2]https://issues.apache.org/jira/browse/FLINK-24113 Best, Weihua On Thu, Aug 10, 2023 at 10:47 AM liu ron <ron9....@gmail.com> wrote: > Hi, Allison > > Thanks for driving this proposal, it looks cool for batch jobs under > application mode. But after reading your FLIP document and [1], I have a > question. Why do you want to rename the execution.attached configuration to > client.attached.after.submission and at the same time deprecate > execution.attached? Based on your design, I understand the role of these > two options are the same. Introducing a new option would increase the cost > of understanding and use for the user, so why not follow the idea discussed > in FLINK-25495 and make Application mode support attached.execution. > > [1] https://issues.apache.org/jira/browse/FLINK-25495 > > Best, > Ron > > Venkatakrishnan Sowrirajan <vsowr...@asu.edu> 于2023年8月9日周三 02:07写道： > > > This is definitely a useful feature especially for the flink batch > > execution workloads using flow orchestrators like Airflow, Azkaban, Oozie > > etc. Thanks for reviving this issue and starting a FLIP. > > > > Regards > > Venkata krishnan > > > > > > On Mon, Aug 7, 2023 at 4:09 PM Allison Chang > <alch...@linkedin.com.invalid > > > > > wrote: > > > > > Hi all, > > > > > > I am opening this thread to discuss this proposal to support attached > > > execution on Flink Application Completion for Batch Jobs. The link to > the > > > FLIP proposal is here: > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-323*3A*Support*Attached*Execution*on*Flink*Application*Completion*for*Batch*Jobs__;JSsrKysrKysrKys!!IKRxdwAv5BmarQ!friFO6bJub5FKSLhPIzA6kv-7uffv-zXlv9ZLMKqj_xMcmZl62HhsgvwDXSCS5hfSeyHZgoAVSFg3fk7ChaAFNKi$ > > > > > > This FLIP proposes adding back attached execution for Application Mode. > > In > > > the past attached execution was supported for the per-job mode, which > > will > > > be deprecated and we want to include this feature back into Application > > > mode. > > > > > > Please reply to this email thread and share your thoughts/opinions. > > > > > > Thank you! > > > > > > Allison Chang > > > > > >
