Re: Flink job manager conditional start of flink jobs
Hi, nage, I agree to the Shammon's suggestion after reading the context. Maybe you need a 'Job Management Service' to manage all jobs among different namespaces. I think the job management is not suitable for implementation in the Flink engine. Best, Hang Shammon FY 于2023年4月13日周四 11:34写道: > Hi > > The job in ns2 has the permission to stop the job in ns1? How about > managing the relationship in your `Job Submission Service` if it exists. > The service can check and stop the job in ns1 before it submitting the job > to ns2, what do you think? > > Best, > Shammon FY > > > On Thu, Apr 13, 2023 at 10:50 AM naga sudhakar > wrote: > >> Hi, >> Thanks for your reply. >> It is slightly different, would be happy to have any suggestion for the >> scenario you mentioned. >> My scenario: I have 2 namespaces say ns1,ns2. I have to make sure only >> one of ns1 or ns2 should run my flink jobs. Say initially ns1 is running >> flink jobs, later planned to move them to ns2. Now when I start in ns2, I >> can make an api call to ns1 jobmanager about running jobs and if no jobs >> then only I should start in ns2. I can introduce this logic inside the >> flink job java main method where my total streaming logic present. So if I >> identify then, how can I stop the initiated job? >> >> Thanks, >> Nagasudhakar. >> >> On Thu, 13 Apr, 2023, 7:08 am Shammon FY, wrote: >> >>> Hi naga >>> >>> Could you provide a specific description of your scene? It sounds like >>> your requirement requires a uniqueness check to ensure that there are no >>> multiple identical jobs running simultaneously, right? >>> >>> Best, >>> Shammon FY >>> >>> On Wed, Apr 12, 2023 at 4:08 PM naga sudhakar >>> wrote: >>> Thanks for your email. I am looking more in terms of running these flinkk jobs in multi names pace environment and make sure only one namespace flink jobs are running. So on the Job manager when i try to start a flink job, it has to check if it's allowed to run in this namespace or not and accordingly flink job shud turn into running state otherwise it shud cancel by itself On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, wrote: > Hi, > > Is the job you want to start running or already finished? > If the job is running, this is simply a failover or a JM failover > case. > While if the job has finished, there's no such feature that can > restart the job > automatically, AFAIK. The job has to be submitted again. > > On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar > wrote: > >> Hi Team, >> Greetings!! >> Just wanted to know when job manager or task manager is being >> restarted, is there a way to run the existing flink jobs based on a >> condition? Same query when I am starting flink job fresh also. >> >> Please let me know if any more information is required from my side. >> >> Thanks & Regards >> Nagasudhakar Sajja. >> >
Re: Flink job manager conditional start of flink jobs
Hi The job in ns2 has the permission to stop the job in ns1? How about managing the relationship in your `Job Submission Service` if it exists. The service can check and stop the job in ns1 before it submitting the job to ns2, what do you think? Best, Shammon FY On Thu, Apr 13, 2023 at 10:50 AM naga sudhakar wrote: > Hi, > Thanks for your reply. > It is slightly different, would be happy to have any suggestion for the > scenario you mentioned. > My scenario: I have 2 namespaces say ns1,ns2. I have to make sure only one > of ns1 or ns2 should run my flink jobs. Say initially ns1 is running flink > jobs, later planned to move them to ns2. Now when I start in ns2, I can > make an api call to ns1 jobmanager about running jobs and if no jobs then > only I should start in ns2. I can introduce this logic inside the flink job > java main method where my total streaming logic present. So if I identify > then, how can I stop the initiated job? > > Thanks, > Nagasudhakar. > > On Thu, 13 Apr, 2023, 7:08 am Shammon FY, wrote: > >> Hi naga >> >> Could you provide a specific description of your scene? It sounds like >> your requirement requires a uniqueness check to ensure that there are no >> multiple identical jobs running simultaneously, right? >> >> Best, >> Shammon FY >> >> On Wed, Apr 12, 2023 at 4:08 PM naga sudhakar >> wrote: >> >>> Thanks for your email. >>> I am looking more in terms of running these flinkk jobs in multi names >>> pace environment and make sure only one namespace flink jobs are running. >>> So on the Job manager when i try to start a flink job, it has to check >>> if it's allowed to run in this namespace or not and accordingly flink job >>> shud turn into running state otherwise it shud cancel by itself >>> >>> >>> >>> On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, wrote: >>> Hi, Is the job you want to start running or already finished? If the job is running, this is simply a failover or a JM failover case. While if the job has finished, there's no such feature that can restart the job automatically, AFAIK. The job has to be submitted again. On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar wrote: > Hi Team, > Greetings!! > Just wanted to know when job manager or task manager is being > restarted, is there a way to run the existing flink jobs based on a > condition? Same query when I am starting flink job fresh also. > > Please let me know if any more information is required from my side. > > Thanks & Regards > Nagasudhakar Sajja. >
Re: Flink job manager conditional start of flink jobs
Hi, Thanks for your reply. It is slightly different, would be happy to have any suggestion for the scenario you mentioned. My scenario: I have 2 namespaces say ns1,ns2. I have to make sure only one of ns1 or ns2 should run my flink jobs. Say initially ns1 is running flink jobs, later planned to move them to ns2. Now when I start in ns2, I can make an api call to ns1 jobmanager about running jobs and if no jobs then only I should start in ns2. I can introduce this logic inside the flink job java main method where my total streaming logic present. So if I identify then, how can I stop the initiated job? Thanks, Nagasudhakar. On Thu, 13 Apr, 2023, 7:08 am Shammon FY, wrote: > Hi naga > > Could you provide a specific description of your scene? It sounds like > your requirement requires a uniqueness check to ensure that there are no > multiple identical jobs running simultaneously, right? > > Best, > Shammon FY > > On Wed, Apr 12, 2023 at 4:08 PM naga sudhakar > wrote: > >> Thanks for your email. >> I am looking more in terms of running these flinkk jobs in multi names >> pace environment and make sure only one namespace flink jobs are running. >> So on the Job manager when i try to start a flink job, it has to check if >> it's allowed to run in this namespace or not and accordingly flink job >> shud turn into running state otherwise it shud cancel by itself >> >> >> >> On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, wrote: >> >>> Hi, >>> >>> Is the job you want to start running or already finished? >>> If the job is running, this is simply a failover or a JM failover case. >>> While if the job has finished, there's no such feature that can restart >>> the job >>> automatically, AFAIK. The job has to be submitted again. >>> >>> On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar >>> wrote: >>> Hi Team, Greetings!! Just wanted to know when job manager or task manager is being restarted, is there a way to run the existing flink jobs based on a condition? Same query when I am starting flink job fresh also. Please let me know if any more information is required from my side. Thanks & Regards Nagasudhakar Sajja. >>>
Re: Flink job manager conditional start of flink jobs
Hi naga Could you provide a specific description of your scene? It sounds like your requirement requires a uniqueness check to ensure that there are no multiple identical jobs running simultaneously, right? Best, Shammon FY On Wed, Apr 12, 2023 at 4:08 PM naga sudhakar wrote: > Thanks for your email. > I am looking more in terms of running these flinkk jobs in multi names > pace environment and make sure only one namespace flink jobs are running. > So on the Job manager when i try to start a flink job, it has to check if > it's allowed to run in this namespace or not and accordingly flink job > shud turn into running state otherwise it shud cancel by itself > > > > On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, wrote: > >> Hi, >> >> Is the job you want to start running or already finished? >> If the job is running, this is simply a failover or a JM failover case. >> While if the job has finished, there's no such feature that can restart >> the job >> automatically, AFAIK. The job has to be submitted again. >> >> On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar >> wrote: >> >>> Hi Team, >>> Greetings!! >>> Just wanted to know when job manager or task manager is being restarted, >>> is there a way to run the existing flink jobs based on a condition? Same >>> query when I am starting flink job fresh also. >>> >>> Please let me know if any more information is required from my side. >>> >>> Thanks & Regards >>> Nagasudhakar Sajja. >>> >>
Re: Flink job manager conditional start of flink jobs
Thanks for your email. I am looking more in terms of running these flinkk jobs in multi names pace environment and make sure only one namespace flink jobs are running. So on the Job manager when i try to start a flink job, it has to check if it's allowed to run in this namespace or not and accordingly flink job shud turn into running state otherwise it shud cancel by itself On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, wrote: > Hi, > > Is the job you want to start running or already finished? > If the job is running, this is simply a failover or a JM failover case. > While if the job has finished, there's no such feature that can restart > the job > automatically, AFAIK. The job has to be submitted again. > > On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar > wrote: > >> Hi Team, >> Greetings!! >> Just wanted to know when job manager or task manager is being restarted, >> is there a way to run the existing flink jobs based on a condition? Same >> query when I am starting flink job fresh also. >> >> Please let me know if any more information is required from my side. >> >> Thanks & Regards >> Nagasudhakar Sajja. >> >
Re: Flink job manager conditional start of flink jobs
Hi, Is the job you want to start running or already finished? If the job is running, this is simply a failover or a JM failover case. While if the job has finished, there's no such feature that can restart the job automatically, AFAIK. The job has to be submitted again. On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar wrote: > Hi Team, > Greetings!! > Just wanted to know when job manager or task manager is being restarted, > is there a way to run the existing flink jobs based on a condition? Same > query when I am starting flink job fresh also. > > Please let me know if any more information is required from my side. > > Thanks & Regards > Nagasudhakar Sajja. >
Flink job manager conditional start of flink jobs
Hi Team, Greetings!! Just wanted to know when job manager or task manager is being restarted, is there a way to run the existing flink jobs based on a condition? Same query when I am starting flink job fresh also. Please let me know if any more information is required from my side. Thanks & Regards Nagasudhakar Sajja.