Query about flink job manager dashboard
> Hi Team, > Greetings!!! > I am a software developer using apache flink and deploying flink jobs > using the same. I have two queries about flink job manager dashboard. Can > you please help with below? > > 1) is it possible to add login mechanism for the flink job manager dash > board and have a role based mechanism for viewing running jobs, cancelling > jobs, adding the jobs? > 2) is it possible to disable to dash bord display but use api to do the > same operations using API? > > > Thanks, > Nagasudhakar. >
Query about flink job manager dashboard
> > Hi Team, > Greetings!!! > I am a software developer using apache flink and deploying flink jobs > using the same. I have two queries about flink job manager dashboard. Can > you please help with below? > > 1) is it possible to add login mechanism for the flink job manager dash > board and have a role based mechanism for viewing running jobs, cancelling > jobs, adding the jobs? > 2) is it possible to disable to dash bord display but use api to do the > same operations using API? > > > Thanks, > Nagasudhakar. >
Re: Query about flink job manager dashboard
Thank you for your response. On Thu, 24 Nov, 2022, 2:07 PM Martijn Visser, wrote: > Hi, > > 1) No, that's currently not possible. > 2) You could consider disabling to disallow uploading new JARs and/or > cancelling jobs from the UI. See > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#advanced-options-for-flink-web-ui > > Best regards, > > Martijn > > On Thu, Nov 24, 2022 at 3:44 AM naga sudhakar > wrote: > >> Hi Team, >>> Greetings!!! >>> I am a software developer using apache flink and deploying flink jobs >>> using the same. I have two queries about flink job manager dashboard. Can >>> you please help with below? >>> >>> 1) is it possible to add login mechanism for the flink job manager dash >>> board and have a role based mechanism for viewing running jobs, cancelling >>> jobs, adding the jobs? >>> 2) is it possible to disable to dash bord display but use api to do the >>> same operations using API? >>> >>> >>> Thanks, >>> Nagasudhakar. >>> >>
Re: Query about flink job manager dashboard
Hi,
We are able to disable this cancela nd upload otpion in ui.
But this is having issues with endpoints for below.
Get call for /jars to list all uploaded jars and post call
/jars/{jarid}/run are giving 404 after disabling the two flags.
Is the process of uploading jars and running a jar with specific id changes
after this change?
Please suggest.
Thanks & Regards,
Nagasudhakar
On Thu, 24 Nov, 2022, 2:07 PM Martijn Visser,
wrote:
> Hi,
>
> 1) No, that's currently not possible.
> 2) You could consider disabling to disallow uploading new JARs and/or
> cancelling jobs from the UI. See
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#advanced-options-for-flink-web-ui
>
> Best regards,
>
> Martijn
>
> On Thu, Nov 24, 2022 at 3:44 AM naga sudhakar
> wrote:
>
>> Hi Team,
>>> Greetings!!!
>>> I am a software developer using apache flink and deploying flink jobs
>>> using the same. I have two queries about flink job manager dashboard. Can
>>> you please help with below?
>>>
>>> 1) is it possible to add login mechanism for the flink job manager dash
>>> board and have a role based mechanism for viewing running jobs, cancelling
>>> jobs, adding the jobs?
>>> 2) is it possible to disable to dash bord display but use api to do the
>>> same operations using API?
>>>
>>>
>>> Thanks,
>>> Nagasudhakar.
>>>
>>
Re: Query about flink job manager dashboard
After disabling the cancel, submit flags facing issues with below api
calls.
1) /jars giving 404
2) /jars/upload
3) /jars/{jarid}/run
Is there any config changes needed to have these apis work?
On Mon, 28 Nov, 2022, 7:00 PM naga sudhakar, wrote:
> Hi,
> We are able to disable this cancela nd upload otpion in ui.
> But this is having issues with endpoints for below.
> Get call for /jars to list all uploaded jars and post call
> /jars/{jarid}/run are giving 404 after disabling the two flags.
> Is the process of uploading jars and running a jar with specific id
> changes after this change?
>
> Please suggest.
>
> Thanks & Regards,
> Nagasudhakar
>
> On Thu, 24 Nov, 2022, 2:07 PM Martijn Visser,
> wrote:
>
>> Hi,
>>
>> 1) No, that's currently not possible.
>> 2) You could consider disabling to disallow uploading new JARs and/or
>> cancelling jobs from the UI. See
>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#advanced-options-for-flink-web-ui
>>
>> Best regards,
>>
>> Martijn
>>
>> On Thu, Nov 24, 2022 at 3:44 AM naga sudhakar
>> wrote:
>>
>>> Hi Team,
>>>> Greetings!!!
>>>> I am a software developer using apache flink and deploying flink jobs
>>>> using the same. I have two queries about flink job manager dashboard. Can
>>>> you please help with below?
>>>>
>>>> 1) is it possible to add login mechanism for the flink job manager dash
>>>> board and have a role based mechanism for viewing running jobs, cancelling
>>>> jobs, adding the jobs?
>>>> 2) is it possible to disable to dash bord display but use api to do the
>>>> same operations using API?
>>>>
>>>>
>>>> Thanks,
>>>> Nagasudhakar.
>>>>
>>>
Re: Query about flink job manager dashboard
Any suggestions for these apis to work after applying these configuration?
Basically I suspect webupload directory cobfif is not taken also when these
were set to false.
web.submit.enable: Enables uploading and starting jobs through the Flink UI
(true by default). Please note that even when this is disabled, session
clusters still accept jobs through REST requests (HTTP calls). This flag
only guards the feature to upload jobs in the UI.
web.cancel.enable: Enables canceling jobs through the Flink UI (true by
default). Please note that even when this is disabled, session clusters
still cancel jobs through REST requests (HTTP calls). This flag only guards
the feature to cancel jobs in the UI.
web.upload.dir: The directory where to store uploaded jobs. Only used when
web.submit.enable is true
On Wed, 30 Nov, 2022, 10:46 AM naga sudhakar,
wrote:
> After disabling the cancel, submit flags facing issues with below api
> calls.
>
> 1) /jars giving 404
> 2) /jars/upload
> 3) /jars/{jarid}/run
>
> Is there any config changes needed to have these apis work?
>
>
> On Mon, 28 Nov, 2022, 7:00 PM naga sudhakar,
> wrote:
>
>> Hi,
>> We are able to disable this cancela nd upload otpion in ui.
>> But this is having issues with endpoints for below.
>> Get call for /jars to list all uploaded jars and post call
>> /jars/{jarid}/run are giving 404 after disabling the two flags.
>> Is the process of uploading jars and running a jar with specific id
>> changes after this change?
>>
>> Please suggest.
>>
>> Thanks & Regards,
>> Nagasudhakar
>>
>> On Thu, 24 Nov, 2022, 2:07 PM Martijn Visser,
>> wrote:
>>
>>> Hi,
>>>
>>> 1) No, that's currently not possible.
>>> 2) You could consider disabling to disallow uploading new JARs and/or
>>> cancelling jobs from the UI. See
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#advanced-options-for-flink-web-ui
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> On Thu, Nov 24, 2022 at 3:44 AM naga sudhakar
>>> wrote:
>>>
>>>> Hi Team,
>>>>> Greetings!!!
>>>>> I am a software developer using apache flink and deploying flink jobs
>>>>> using the same. I have two queries about flink job manager dashboard. Can
>>>>> you please help with below?
>>>>>
>>>>> 1) is it possible to add login mechanism for the flink job manager
>>>>> dash board and have a role based mechanism for viewing running jobs,
>>>>> cancelling jobs, adding the jobs?
>>>>> 2) is it possible to disable to dash bord display but use api to do
>>>>> the same operations using API?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Nagasudhakar.
>>>>>
>>>>
Re: Query about flink job manager dashboard
Is it possible to disable to dashboard port running, but still api running
on different port? Let us know if we can configure this?
On Mon, 5 Dec, 2022, 6:08 AM naga sudhakar, wrote:
> Any suggestions for these apis to work after applying these configuration?
> Basically I suspect webupload directory cobfif is not taken also when these
> were set to false.
>
> web.submit.enable: Enables uploading and starting jobs through the Flink
> UI (true by default). Please note that even when this is disabled, session
> clusters still accept jobs through REST requests (HTTP calls). This flag
> only guards the feature to upload jobs in the UI.
> web.cancel.enable: Enables canceling jobs through the Flink UI (true by
> default). Please note that even when this is disabled, session clusters
> still cancel jobs through REST requests (HTTP calls). This flag only guards
> the feature to cancel jobs in the UI.
> web.upload.dir: The directory where to store uploaded jobs. Only used when
> web.submit.enable is true
>
> On Wed, 30 Nov, 2022, 10:46 AM naga sudhakar,
> wrote:
>
>> After disabling the cancel, submit flags facing issues with below api
>> calls.
>>
>> 1) /jars giving 404
>> 2) /jars/upload
>> 3) /jars/{jarid}/run
>>
>> Is there any config changes needed to have these apis work?
>>
>>
>> On Mon, 28 Nov, 2022, 7:00 PM naga sudhakar,
>> wrote:
>>
>>> Hi,
>>> We are able to disable this cancela nd upload otpion in ui.
>>> But this is having issues with endpoints for below.
>>> Get call for /jars to list all uploaded jars and post call
>>> /jars/{jarid}/run are giving 404 after disabling the two flags.
>>> Is the process of uploading jars and running a jar with specific id
>>> changes after this change?
>>>
>>> Please suggest.
>>>
>>> Thanks & Regards,
>>> Nagasudhakar
>>>
>>> On Thu, 24 Nov, 2022, 2:07 PM Martijn Visser,
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> 1) No, that's currently not possible.
>>>> 2) You could consider disabling to disallow uploading new JARs and/or
>>>> cancelling jobs from the UI. See
>>>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#advanced-options-for-flink-web-ui
>>>>
>>>> Best regards,
>>>>
>>>> Martijn
>>>>
>>>> On Thu, Nov 24, 2022 at 3:44 AM naga sudhakar
>>>> wrote:
>>>>
>>>>> Hi Team,
>>>>>> Greetings!!!
>>>>>> I am a software developer using apache flink and deploying flink jobs
>>>>>> using the same. I have two queries about flink job manager dashboard. Can
>>>>>> you please help with below?
>>>>>>
>>>>>> 1) is it possible to add login mechanism for the flink job manager
>>>>>> dash board and have a role based mechanism for viewing running jobs,
>>>>>> cancelling jobs, adding the jobs?
>>>>>> 2) is it possible to disable to dash bord display but use api to do
>>>>>> the same operations using API?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Nagasudhakar.
>>>>>>
>>>>>
Flink job manager conditional start of flink jobs
Hi Team, Greetings!! Just wanted to know when job manager or task manager is being restarted, is there a way to run the existing flink jobs based on a condition? Same query when I am starting flink job fresh also. Please let me know if any more information is required from my side. Thanks & Regards Nagasudhakar Sajja.
Re: Flink job manager conditional start of flink jobs
Thanks for your email. I am looking more in terms of running these flinkk jobs in multi names pace environment and make sure only one namespace flink jobs are running. So on the Job manager when i try to start a flink job, it has to check if it's allowed to run in this namespace or not and accordingly flink job shud turn into running state otherwise it shud cancel by itself On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, wrote: > Hi, > > Is the job you want to start running or already finished? > If the job is running, this is simply a failover or a JM failover case. > While if the job has finished, there's no such feature that can restart > the job > automatically, AFAIK. The job has to be submitted again. > > On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar > wrote: > >> Hi Team, >> Greetings!! >> Just wanted to know when job manager or task manager is being restarted, >> is there a way to run the existing flink jobs based on a condition? Same >> query when I am starting flink job fresh also. >> >> Please let me know if any more information is required from my side. >> >> Thanks & Regards >> Nagasudhakar Sajja. >> >
Re: Flink job manager conditional start of flink jobs
Hi, Thanks for your reply. It is slightly different, would be happy to have any suggestion for the scenario you mentioned. My scenario: I have 2 namespaces say ns1,ns2. I have to make sure only one of ns1 or ns2 should run my flink jobs. Say initially ns1 is running flink jobs, later planned to move them to ns2. Now when I start in ns2, I can make an api call to ns1 jobmanager about running jobs and if no jobs then only I should start in ns2. I can introduce this logic inside the flink job java main method where my total streaming logic present. So if I identify then, how can I stop the initiated job? Thanks, Nagasudhakar. On Thu, 13 Apr, 2023, 7:08 am Shammon FY, wrote: > Hi naga > > Could you provide a specific description of your scene? It sounds like > your requirement requires a uniqueness check to ensure that there are no > multiple identical jobs running simultaneously, right? > > Best, > Shammon FY > > On Wed, Apr 12, 2023 at 4:08 PM naga sudhakar > wrote: > >> Thanks for your email. >> I am looking more in terms of running these flinkk jobs in multi names >> pace environment and make sure only one namespace flink jobs are running. >> So on the Job manager when i try to start a flink job, it has to check if >> it's allowed to run in this namespace or not and accordingly flink job >> shud turn into running state otherwise it shud cancel by itself >> >> >> >> On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, wrote: >> >>> Hi, >>> >>> Is the job you want to start running or already finished? >>> If the job is running, this is simply a failover or a JM failover case. >>> While if the job has finished, there's no such feature that can restart >>> the job >>> automatically, AFAIK. The job has to be submitted again. >>> >>> On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar >>> wrote: >>> >>>> Hi Team, >>>> Greetings!! >>>> Just wanted to know when job manager or task manager is being >>>> restarted, is there a way to run the existing flink jobs based on a >>>> condition? Same query when I am starting flink job fresh also. >>>> >>>> Please let me know if any more information is required from my side. >>>> >>>> Thanks & Regards >>>> Nagasudhakar Sajja. >>>> >>>
Moving from flinkkafkaconsumer to kafkasource issues
Hi Team, Greetings of the day.. we are on flink 1.16.1 version and using flinkkafkaconsumer today. When I replaced it with kafkasource,it's failing with not able to connect with kafka jaas configuration. Error says Kafka client entry not found in /tmp/jass config file. We are passing the flonk runtime arg for the security.auth.login conf details. Same was working g with flink Kafka consumer without any issues. Did anyone face this issue? Any suggestion? One of the stack over flow suggestion was to have the content of jaas config into java code,but not use full Thanks, Nagasudhakar
