Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Hi, Thank you everyone for the valuable comments, if there are no new messages by then, I will start a vote on Monday. Thanks, Ferenc On Monday, 3 June 2024 at 17:27, Jeyhun Karimov wrote: > > > Hi Ferenc, > > Thanks for the proposal. +1 for it! This FLIP will improve the user > experience. > > Regards, > Jeyhun > > > > > > On Mon, Jun 3, 2024 at 1:50 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hi Hang, > > > > Thank you for your inputs, both points make sense, updated the > > FLIP according to them. > > > > Best, > > Ferenc > > > > On Friday, 31 May 2024 at 04:31, Hang Ruan ruanhang1...@gmail.com wrote: > > > > > Hi, Ferenc. > > > > > > +1 for this proposal. This FLIP will help to make the CLI clearer for > > > users. > > > > > > I think we should better add an example in the FLIP about how to use the > > > application mode with the new CLI. > > > Besides that, we need to add some new tests for this change instead of > > > only > > > using the existed tests. > > > > > > Best, > > > Hang > > > > > > Mate Czagany czmat...@gmail.com 于2024年5月29日周三 19:57写道： > > > > > > > Hi Ferenc, > > > > > > > > Thanks for the FLIP, +1 from me for the proposal. I think these changes > > > > would be a great solution to all the confusion that comes from these > > > > two > > > > action parameters. > > > > > > > > Best regards, > > > > Mate > > > > > > > > Ferenc Csaky ferenc.cs...@pm.me.invalid ezt írta (időpont: 2024. máj. > > > > 28., K, 16:13): > > > > > > > > > Thank you Xintong for your input. > > > > > > > > > > I prepared a FLIP for this change [1], looking forward for any > > > > > other opinions. > > > > > > > > > > Thanks, > > > > > Ferenc > > > > > > > > > > [1] > > > > https://docs.google.com/document/d/1EX74rFp9bMKdfoGkz1ASOM6Ibw32rRxIadX72zs2zoY/edit?usp=sharing > > > > > > > On Friday, 17 May 2024 at 07:04, Xintong Song tonysong...@gmail.com > > > > > wrote: > > > > > > > > > > > AFAIK, the main purpose of having `run-application` was to make > > > > > > sure > > > > > > the user is aware that application mode is used, which executes the > > > > > > main > > > > > > method of the user program in JM rather than in client. This was > > > > > > important > > > > > > at the time application mode was first introduced, but maybe not > > > > > > that > > > > > > important anymore, given that per-job mode is deprecated and likely > > > > > > removed > > > > > > in 2.0. Therefore, +1 for the proposal. > > > > > > > > > > > > Best, > > > > > > > > > > > > Xintong > > > > > > > > > > > > On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky > > > > > > ferenc.cs...@pm.me.invalid > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hello devs, > > > > > > > > > > > > > > I saw quite some examples when customers were confused about > > > > > > > run, and > > > > > > > run- > > > > > > > application in the Flink CLI and I was wondering about the > > > > > > > necessity > > > > > > > of > > > > > > > deploying > > > > > > > Application Mode (AM) jobs with a different command, than > > > > > > > Session and > > > > > > > Per-Job mode jobs. > > > > > > > > > > > > > > I can see a point that YarnDeploymentTarget [1] and > > > > > > > KubernetesDeploymentTarget > > > > > > > [2] are part of their own maven modules and not known in > > > > > > > flink-clients, > > > > > > > so the > > > > > > > deployment mode validations are happening during cluster > > > > > > > deployment > > > > > > > in > > > > > > > their specific > > > > > > > ClusterDescriptor implementation [3]. Although these are > > > > > > > implementation > > > > > > > details that > > > > > > > IMO should not define user-facing APIs. > > > > > > > > > > > > > > The command line setup is the same for both run and > > > > > > > run-application, > > > > > > > so > > > > > > > I think there > > > > > > > is a quite simple way to achieve a unified flink run experience, > > > > > > > but > > > > > > > I > > > > > > > might missed > > > > > > > something so I would appreciate any inputs on this topic. > > > > > > > > > > > > > > Based on my assumptions I think it would be possible to > > > > > > > deprecate the > > > > > > > run- > > > > > > > application in Flink 1.20 and remove it completely in Flink 2.0. > > > > > > > I > > > > > > > already put together a > > > > > > > PoC [4], and I was able to deploy AM jobs like this: > > > > > > > > > > > > > > flink run --target kubernetes-application ... > > > > > > > > > > > > > > If others also agree with this, I would be happy to open a FLIP. > > > > > > > WDYT? > > > > > > > > > > > > > > Thanks, > > > > > > > Ferenc > > > > > > > > > > > > > > [1] > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java > > > > > > > > > [2] > > > > https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java
Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Hi Ferenc, Thanks for the proposal. +1 for it! This FLIP will improve the user experience. Regards, Jeyhun On Mon, Jun 3, 2024 at 1:50 PM Ferenc Csaky wrote: > Hi Hang, > > Thank you for your inputs, both points make sense, updated the > FLIP according to them. > > Best, > Ferenc > > > > > On Friday, 31 May 2024 at 04:31, Hang Ruan wrote: > > > > > > > Hi, Ferenc. > > > > +1 for this proposal. This FLIP will help to make the CLI clearer for > users. > > > > I think we should better add an example in the FLIP about how to use the > > application mode with the new CLI. > > Besides that, we need to add some new tests for this change instead of > only > > using the existed tests. > > > > Best, > > Hang > > > > Mate Czagany czmat...@gmail.com 于2024年5月29日周三 19:57写道： > > > > > Hi Ferenc, > > > > > > Thanks for the FLIP, +1 from me for the proposal. I think these changes > > > would be a great solution to all the confusion that comes from these > two > > > action parameters. > > > > > > Best regards, > > > Mate > > > > > > Ferenc Csaky ferenc.cs...@pm.me.invalid ezt írta (időpont: 2024. máj. > > > 28., K, 16:13): > > > > > > > Thank you Xintong for your input. > > > > > > > > I prepared a FLIP for this change [1], looking forward for any > > > > other opinions. > > > > > > > > Thanks, > > > > Ferenc > > > > > > > > [1] > > > > > > > https://docs.google.com/document/d/1EX74rFp9bMKdfoGkz1ASOM6Ibw32rRxIadX72zs2zoY/edit?usp=sharing > > > > > > > On Friday, 17 May 2024 at 07:04, Xintong Song tonysong...@gmail.com > > > > wrote: > > > > > > > > > AFAIK, the main purpose of having `run-application` was to make > sure > > > > > the user is aware that application mode is used, which executes the > > > > > main > > > > > method of the user program in JM rather than in client. This was > > > > > important > > > > > at the time application mode was first introduced, but maybe not > that > > > > > important anymore, given that per-job mode is deprecated and likely > > > > > removed > > > > > in 2.0. Therefore, +1 for the proposal. > > > > > > > > > > Best, > > > > > > > > > > Xintong > > > > > > > > > > On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky > > > > > ferenc.cs...@pm.me.invalid > > > > > > > > > > wrote: > > > > > > > > > > > Hello devs, > > > > > > > > > > > > I saw quite some examples when customers were confused about > run, and > > > > > > run- > > > > > > application in the Flink CLI and I was wondering about the > necessity > > > > > > of > > > > > > deploying > > > > > > Application Mode (AM) jobs with a different command, than > Session and > > > > > > Per-Job mode jobs. > > > > > > > > > > > > I can see a point that YarnDeploymentTarget [1] and > > > > > > KubernetesDeploymentTarget > > > > > > [2] are part of their own maven modules and not known in > > > > > > flink-clients, > > > > > > so the > > > > > > deployment mode validations are happening during cluster > deployment > > > > > > in > > > > > > their specific > > > > > > ClusterDescriptor implementation [3]. Although these are > > > > > > implementation > > > > > > details that > > > > > > IMO should not define user-facing APIs. > > > > > > > > > > > > The command line setup is the same for both run and > run-application, > > > > > > so > > > > > > I think there > > > > > > is a quite simple way to achieve a unified flink run experience, > but > > > > > > I > > > > > > might missed > > > > > > something so I would appreciate any inputs on this topic. > > > > > > > > > > > > Based on my assumptions I think it would be possible to > deprecate the > > > > > > run- > > > > > > application in Flink 1.20 and remove it completely in Flink 2.0. > I > > > > > > already put together a > > > > > > PoC [4], and I was able to deploy AM jobs like this: > > > > > > > > > > > > flink run --target kubernetes-application ... > > > > > > > > > > > > If others also agree with this, I would be happy to open a FLIP. > > > > > > WDYT? > > > > > > > > > > > > Thanks, > > > > > > Ferenc > > > > > > > > > > > > [1] > > > > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java > > > > > > > > > [2] > > > > > > > https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java > > > > > > > > > [3] > > > > > > > https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206 > > > > > > > > > [4] > > > > > > > https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0 >
Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Hi Hang, Thank you for your inputs, both points make sense, updated the FLIP according to them. Best, Ferenc On Friday, 31 May 2024 at 04:31, Hang Ruan wrote: > > > Hi, Ferenc. > > +1 for this proposal. This FLIP will help to make the CLI clearer for users. > > I think we should better add an example in the FLIP about how to use the > application mode with the new CLI. > Besides that, we need to add some new tests for this change instead of only > using the existed tests. > > Best, > Hang > > Mate Czagany czmat...@gmail.com 于2024年5月29日周三 19:57写道： > > > Hi Ferenc, > > > > Thanks for the FLIP, +1 from me for the proposal. I think these changes > > would be a great solution to all the confusion that comes from these two > > action parameters. > > > > Best regards, > > Mate > > > > Ferenc Csaky ferenc.cs...@pm.me.invalid ezt írta (időpont: 2024. máj. > > 28., K, 16:13): > > > > > Thank you Xintong for your input. > > > > > > I prepared a FLIP for this change [1], looking forward for any > > > other opinions. > > > > > > Thanks, > > > Ferenc > > > > > > [1] > > > > https://docs.google.com/document/d/1EX74rFp9bMKdfoGkz1ASOM6Ibw32rRxIadX72zs2zoY/edit?usp=sharing > > > > > On Friday, 17 May 2024 at 07:04, Xintong Song tonysong...@gmail.com > > > wrote: > > > > > > > AFAIK, the main purpose of having `run-application` was to make sure > > > > the user is aware that application mode is used, which executes the > > > > main > > > > method of the user program in JM rather than in client. This was > > > > important > > > > at the time application mode was first introduced, but maybe not that > > > > important anymore, given that per-job mode is deprecated and likely > > > > removed > > > > in 2.0. Therefore, +1 for the proposal. > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky > > > > ferenc.cs...@pm.me.invalid > > > > > > > > wrote: > > > > > > > > > Hello devs, > > > > > > > > > > I saw quite some examples when customers were confused about run, and > > > > > run- > > > > > application in the Flink CLI and I was wondering about the necessity > > > > > of > > > > > deploying > > > > > Application Mode (AM) jobs with a different command, than Session and > > > > > Per-Job mode jobs. > > > > > > > > > > I can see a point that YarnDeploymentTarget [1] and > > > > > KubernetesDeploymentTarget > > > > > [2] are part of their own maven modules and not known in > > > > > flink-clients, > > > > > so the > > > > > deployment mode validations are happening during cluster deployment > > > > > in > > > > > their specific > > > > > ClusterDescriptor implementation [3]. Although these are > > > > > implementation > > > > > details that > > > > > IMO should not define user-facing APIs. > > > > > > > > > > The command line setup is the same for both run and run-application, > > > > > so > > > > > I think there > > > > > is a quite simple way to achieve a unified flink run experience, but > > > > > I > > > > > might missed > > > > > something so I would appreciate any inputs on this topic. > > > > > > > > > > Based on my assumptions I think it would be possible to deprecate the > > > > > run- > > > > > application in Flink 1.20 and remove it completely in Flink 2.0. I > > > > > already put together a > > > > > PoC [4], and I was able to deploy AM jobs like this: > > > > > > > > > > flink run --target kubernetes-application ... > > > > > > > > > > If others also agree with this, I would be happy to open a FLIP. > > > > > WDYT? > > > > > > > > > > Thanks, > > > > > Ferenc > > > > > > > > > > [1] > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java > > > > > > > [2] > > > > https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java > > > > > > > [3] > > > > https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206 > > > > > > > [4] > > > > https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0
Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Hi, Ferenc. +1 for this proposal. This FLIP will help to make the CLI clearer for users. I think we should better add an example in the FLIP about how to use the application mode with the new CLI. Besides that, we need to add some new tests for this change instead of only using the existed tests. Best, Hang Mate Czagany 于2024年5月29日周三 19:57写道： > Hi Ferenc, > > Thanks for the FLIP, +1 from me for the proposal. I think these changes > would be a great solution to all the confusion that comes from these two > action parameters. > > Best regards, > Mate > > Ferenc Csaky ezt írta (időpont: 2024. máj. > 28., K, 16:13): > > > Thank you Xintong for your input. > > > > I prepared a FLIP for this change [1], looking forward for any > > other opinions. > > > > Thanks, > > Ferenc > > > > [1] > > > https://docs.google.com/document/d/1EX74rFp9bMKdfoGkz1ASOM6Ibw32rRxIadX72zs2zoY/edit?usp=sharing > > > > > > > > On Friday, 17 May 2024 at 07:04, Xintong Song > > wrote: > > > > > > > > > > > AFAIK, the main purpose of having `run-application` was to make sure > > > the user is aware that application mode is used, which executes the > main > > > method of the user program in JM rather than in client. This was > > important > > > at the time application mode was first introduced, but maybe not that > > > important anymore, given that per-job mode is deprecated and likely > > removed > > > in 2.0. Therefore, +1 for the proposal. > > > > > > Best, > > > > > > Xintong > > > > > > > > > > > > On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky > ferenc.cs...@pm.me.invalid > > > > > > wrote: > > > > > > > Hello devs, > > > > > > > > I saw quite some examples when customers were confused about run, and > > run- > > > > application in the Flink CLI and I was wondering about the necessity > of > > > > deploying > > > > Application Mode (AM) jobs with a different command, than Session and > > > > Per-Job mode jobs. > > > > > > > > I can see a point that YarnDeploymentTarget [1] and > > > > KubernetesDeploymentTarget > > > > [2] are part of their own maven modules and not known in > flink-clients, > > > > so the > > > > deployment mode validations are happening during cluster deployment > in > > > > their specific > > > > ClusterDescriptor implementation [3]. Although these are > implementation > > > > details that > > > > IMO should not define user-facing APIs. > > > > > > > > The command line setup is the same for both run and run-application, > so > > > > I think there > > > > is a quite simple way to achieve a unified flink run experience, but > I > > > > might missed > > > > something so I would appreciate any inputs on this topic. > > > > > > > > Based on my assumptions I think it would be possible to deprecate the > > run- > > > > application in Flink 1.20 and remove it completely in Flink 2.0. I > > > > already put together a > > > > PoC [4], and I was able to deploy AM jobs like this: > > > > > > > > flink run --target kubernetes-application ... > > > > > > > > If others also agree with this, I would be happy to open a FLIP. > WDYT? > > > > > > > > Thanks, > > > > Ferenc > > > > > > > > [1] > > > > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java > > > > [2] > > > > > > > https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java > > > > [3] > > > > > > > https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206 > > > > [4] > > > > > > > https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0 > > >
Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Hi Ferenc, Thanks for the FLIP, +1 from me for the proposal. I think these changes would be a great solution to all the confusion that comes from these two action parameters. Best regards, Mate Ferenc Csaky ezt írta (időpont: 2024. máj. 28., K, 16:13): > Thank you Xintong for your input. > > I prepared a FLIP for this change [1], looking forward for any > other opinions. > > Thanks, > Ferenc > > [1] > https://docs.google.com/document/d/1EX74rFp9bMKdfoGkz1ASOM6Ibw32rRxIadX72zs2zoY/edit?usp=sharing > > > > On Friday, 17 May 2024 at 07:04, Xintong Song > wrote: > > > > > > > AFAIK, the main purpose of having `run-application` was to make sure > > the user is aware that application mode is used, which executes the main > > method of the user program in JM rather than in client. This was > important > > at the time application mode was first introduced, but maybe not that > > important anymore, given that per-job mode is deprecated and likely > removed > > in 2.0. Therefore, +1 for the proposal. > > > > Best, > > > > Xintong > > > > > > > > On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > > > wrote: > > > > > Hello devs, > > > > > > I saw quite some examples when customers were confused about run, and > run- > > > application in the Flink CLI and I was wondering about the necessity of > > > deploying > > > Application Mode (AM) jobs with a different command, than Session and > > > Per-Job mode jobs. > > > > > > I can see a point that YarnDeploymentTarget [1] and > > > KubernetesDeploymentTarget > > > [2] are part of their own maven modules and not known in flink-clients, > > > so the > > > deployment mode validations are happening during cluster deployment in > > > their specific > > > ClusterDescriptor implementation [3]. Although these are implementation > > > details that > > > IMO should not define user-facing APIs. > > > > > > The command line setup is the same for both run and run-application, so > > > I think there > > > is a quite simple way to achieve a unified flink run experience, but I > > > might missed > > > something so I would appreciate any inputs on this topic. > > > > > > Based on my assumptions I think it would be possible to deprecate the > run- > > > application in Flink 1.20 and remove it completely in Flink 2.0. I > > > already put together a > > > PoC [4], and I was able to deploy AM jobs like this: > > > > > > flink run --target kubernetes-application ... > > > > > > If others also agree with this, I would be happy to open a FLIP. WDYT? > > > > > > Thanks, > > > Ferenc > > > > > > [1] > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java > > > [2] > > > > https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java > > > [3] > > > > https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206 > > > [4] > > > > https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0 >
Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Thank you Xintong for your input. I prepared a FLIP for this change [1], looking forward for any other opinions. Thanks, Ferenc [1] https://docs.google.com/document/d/1EX74rFp9bMKdfoGkz1ASOM6Ibw32rRxIadX72zs2zoY/edit?usp=sharing On Friday, 17 May 2024 at 07:04, Xintong Song wrote: > > > AFAIK, the main purpose of having `run-application` was to make sure > the user is aware that application mode is used, which executes the main > method of the user program in JM rather than in client. This was important > at the time application mode was first introduced, but maybe not that > important anymore, given that per-job mode is deprecated and likely removed > in 2.0. Therefore, +1 for the proposal. > > Best, > > Xintong > > > > On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hello devs, > > > > I saw quite some examples when customers were confused about run, and run- > > application in the Flink CLI and I was wondering about the necessity of > > deploying > > Application Mode (AM) jobs with a different command, than Session and > > Per-Job mode jobs. > > > > I can see a point that YarnDeploymentTarget [1] and > > KubernetesDeploymentTarget > > [2] are part of their own maven modules and not known in flink-clients, > > so the > > deployment mode validations are happening during cluster deployment in > > their specific > > ClusterDescriptor implementation [3]. Although these are implementation > > details that > > IMO should not define user-facing APIs. > > > > The command line setup is the same for both run and run-application, so > > I think there > > is a quite simple way to achieve a unified flink run experience, but I > > might missed > > something so I would appreciate any inputs on this topic. > > > > Based on my assumptions I think it would be possible to deprecate the run- > > application in Flink 1.20 and remove it completely in Flink 2.0. I > > already put together a > > PoC [4], and I was able to deploy AM jobs like this: > > > > flink run --target kubernetes-application ... > > > > If others also agree with this, I would be happy to open a FLIP. WDYT? > > > > Thanks, > > Ferenc > > > > [1] > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java > > [2] > > https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java > > [3] > > https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206 > > [4] > > https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0
Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
AFAIK, the main purpose of having `run-application` was to make sure the user is aware that application mode is used, which executes the main method of the user program in JM rather than in client. This was important at the time application mode was first introduced, but maybe not that important anymore, given that per-job mode is deprecated and likely removed in 2.0. Therefore, +1 for the proposal. Best, Xintong On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky wrote: > Hello devs, > > I saw quite some examples when customers were confused about run, and run- > application in the Flink CLI and I was wondering about the necessity of > deploying > Application Mode (AM) jobs with a different command, than Session and > Per-Job mode jobs. > > I can see a point that YarnDeploymentTarget [1] and > KubernetesDeploymentTarget > [2] are part of their own maven modules and not known in flink-clients, > so the > deployment mode validations are happening during cluster deployment in > their specific > ClusterDescriptor implementation [3]. Although these are implementation > details that > IMO should not define user-facing APIs. > > The command line setup is the same for both run and run-application, so > I think there > is a quite simple way to achieve a unified flink run experience, but I > might missed > something so I would appreciate any inputs on this topic. > > Based on my assumptions I think it would be possible to deprecate the run- > application in Flink 1.20 and remove it completely in Flink 2.0. I > already put together a > PoC [4], and I was able to deploy AM jobs like this: > > flink run --target kubernetes-application ... > > If others also agree with this, I would be happy to open a FLIP. WDYT? > > Thanks, > Ferenc > > [1] > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java > [2] > https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java > [3] > https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206 > [4] > https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0
[DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Hello devs, I saw quite some examples when customers were confused about run, and run- application in the Flink CLI and I was wondering about the necessity of deploying Application Mode (AM) jobs with a different command, than Session and Per-Job mode jobs. I can see a point that YarnDeploymentTarget [1] and KubernetesDeploymentTarget [2] are part of their own maven modules and not known in flink-clients, so the deployment mode validations are happening during cluster deployment in their specific ClusterDescriptor implementation [3]. Although these are implementation details that IMO should not define user-facing APIs. The command line setup is the same for both run and run-application, so I think there is a quite simple way to achieve a unified flink run experience, but I might missed something so I would appreciate any inputs on this topic. Based on my assumptions I think it would be possible to deprecate the run- application in Flink 1.20 and remove it completely in Flink 2.0. I already put together a PoC [4], and I was able to deploy AM jobs like this: flink run --target kubernetes-application ... If others also agree with this, I would be happy to open a FLIP. WDYT? Thanks, Ferenc [1] https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java [2] https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java [3] https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206 [4] https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0
