Re: Retrieving Flink job ID/YARN Id programmatically
Hi Lei, if you use public JobExecutionResult StreamExecutionEnvironment#execute() You can retrieve the job id through the result. result.getJobID() Best, Arvid On Mon, Nov 25, 2019 at 3:50 AM Ana wrote: > Hi Lei, > > To add, you may use Hadoop Resource Manager REST APIs > https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html. > I'm also running Flink application on YARN and use this API for that > purpose. If you find other way or a much better solution, please let me > know! > > Regards, > Ana > > On Fri, Nov 22, 2019 at 10:58 AM vino yang wrote: > >> Hi Lei, >> >> It would be better to use Flink's RESTful API to fetch the information of >> the running jobs[1]. >> >> [1]: >> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-1 >> >> Best, >> Vino >> >> Lei Nie 于2019年11月22日周五 上午4:14写道： >> >>> I looked at the code, and >>> StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is >>> generating a random ID unrelated to the actual ID used. >>> >>> Is there any way to fetch the real ID at runtime? >>> Use case: fetch most recent checkpoint from stable storage for >>> automated restarts. Most recent checkpoint has form >>> ".../checkpoints/flink_app_id/chk-123" >>> >>> On Thu, Nov 21, 2019 at 11:28 AM Lei Nie wrote: >>> > >>> > This does not get the correct id: >>> > StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID = >>> > eea5abc21dd8743a4090f4a3a660f9e8 >>> > Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a >>> > >>> > >>> > >>> > On Thu, Nov 7, 2019 at 6:56 PM vino yang >>> wrote: >>> > > >>> > > Hi Lei Nie, >>> > > >>> > > You can use >>> `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the >>> job id. >>> > > >>> > > Best, >>> > > Vino >>> > > >>> > > Lei Nie 于2019年11月8日周五 上午8:38写道： >>> > >> >>> > >> Hello, >>> > >> I am currently executing streaming jobs via >>> StreamExecutionEnvironment. Is it possible to retrieve the Flink job >>> ID/YARN ID within the context of a job? I'd like to be able to >>> automatically register the job such that monitoring jobs can run (REST api >>> requires for example job id). >>> > >> >>> > >> Thanks >>> >>
Re: Retrieving Flink job ID/YARN Id programmatically
Hi Lei, To add, you may use Hadoop Resource Manager REST APIs https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html. I'm also running Flink application on YARN and use this API for that purpose. If you find other way or a much better solution, please let me know! Regards, Ana On Fri, Nov 22, 2019 at 10:58 AM vino yang wrote: > Hi Lei, > > It would be better to use Flink's RESTful API to fetch the information of > the running jobs[1]. > > [1]: > https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-1 > > Best, > Vino > > Lei Nie 于2019年11月22日周五 上午4:14写道： > >> I looked at the code, and >> StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is >> generating a random ID unrelated to the actual ID used. >> >> Is there any way to fetch the real ID at runtime? >> Use case: fetch most recent checkpoint from stable storage for >> automated restarts. Most recent checkpoint has form >> ".../checkpoints/flink_app_id/chk-123" >> >> On Thu, Nov 21, 2019 at 11:28 AM Lei Nie wrote: >> > >> > This does not get the correct id: >> > StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID = >> > eea5abc21dd8743a4090f4a3a660f9e8 >> > Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a >> > >> > >> > >> > On Thu, Nov 7, 2019 at 6:56 PM vino yang wrote: >> > > >> > > Hi Lei Nie, >> > > >> > > You can use >> `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the >> job id. >> > > >> > > Best, >> > > Vino >> > > >> > > Lei Nie 于2019年11月8日周五 上午8:38写道： >> > >> >> > >> Hello, >> > >> I am currently executing streaming jobs via >> StreamExecutionEnvironment. Is it possible to retrieve the Flink job >> ID/YARN ID within the context of a job? I'd like to be able to >> automatically register the job such that monitoring jobs can run (REST api >> requires for example job id). >> > >> >> > >> Thanks >> >
Re: Retrieving Flink job ID/YARN Id programmatically
Hi Lei, It would be better to use Flink's RESTful API to fetch the information of the running jobs[1]. [1]: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-1 Best, Vino Lei Nie 于2019年11月22日周五 上午4:14写道： > I looked at the code, and > StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is > generating a random ID unrelated to the actual ID used. > > Is there any way to fetch the real ID at runtime? > Use case: fetch most recent checkpoint from stable storage for > automated restarts. Most recent checkpoint has form > ".../checkpoints/flink_app_id/chk-123" > > On Thu, Nov 21, 2019 at 11:28 AM Lei Nie wrote: > > > > This does not get the correct id: > > StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID = > > eea5abc21dd8743a4090f4a3a660f9e8 > > Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a > > > > > > > > On Thu, Nov 7, 2019 at 6:56 PM vino yang wrote: > > > > > > Hi Lei Nie, > > > > > > You can use > `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the > job id. > > > > > > Best, > > > Vino > > > > > > Lei Nie 于2019年11月8日周五 上午8:38写道： > > >> > > >> Hello, > > >> I am currently executing streaming jobs via > StreamExecutionEnvironment. Is it possible to retrieve the Flink job > ID/YARN ID within the context of a job? I'd like to be able to > automatically register the job such that monitoring jobs can run (REST api > requires for example job id). > > >> > > >> Thanks >
Re: Retrieving Flink job ID/YARN Id programmatically
I looked at the code, and StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is generating a random ID unrelated to the actual ID used. Is there any way to fetch the real ID at runtime? Use case: fetch most recent checkpoint from stable storage for automated restarts. Most recent checkpoint has form ".../checkpoints/flink_app_id/chk-123" On Thu, Nov 21, 2019 at 11:28 AM Lei Nie wrote: > > This does not get the correct id: > StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID = > eea5abc21dd8743a4090f4a3a660f9e8 > Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a > > > > On Thu, Nov 7, 2019 at 6:56 PM vino yang wrote: > > > > Hi Lei Nie, > > > > You can use > > `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the > > job id. > > > > Best, > > Vino > > > > Lei Nie 于2019年11月8日周五 上午8:38写道： > >> > >> Hello, > >> I am currently executing streaming jobs via StreamExecutionEnvironment. Is > >> it possible to retrieve the Flink job ID/YARN ID within the context of a > >> job? I'd like to be able to automatically register the job such that > >> monitoring jobs can run (REST api requires for example job id). > >> > >> Thanks
Re: Retrieving Flink job ID/YARN Id programmatically
This does not get the correct id: StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID = eea5abc21dd8743a4090f4a3a660f9e8 Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a On Thu, Nov 7, 2019 at 6:56 PM vino yang wrote: > > Hi Lei Nie, > > You can use `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` > to get the job id. > > Best, > Vino > > Lei Nie 于2019年11月8日周五 上午8:38写道： >> >> Hello, >> I am currently executing streaming jobs via StreamExecutionEnvironment. Is >> it possible to retrieve the Flink job ID/YARN ID within the context of a >> job? I'd like to be able to automatically register the job such that >> monitoring jobs can run (REST api requires for example job id). >> >> Thanks
Re: Retrieving Flink job ID/YARN Id programmatically
Hi Lei Nie, You can use `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the job id. Best, Vino Lei Nie 于2019年11月8日周五 上午8:38写道： > Hello, > I am currently executing streaming jobs via StreamExecutionEnvironment. Is > it possible to retrieve the Flink job ID/YARN ID within the context of a > job? I'd like to be able to automatically register the job such that > monitoring jobs can run (REST api requires for example job id). > > Thanks >
Retrieving Flink job ID/YARN Id programmatically
Hello, I am currently executing streaming jobs via StreamExecutionEnvironment. Is it possible to retrieve the Flink job ID/YARN ID within the context of a job? I'd like to be able to automatically register the job such that monitoring jobs can run (REST api requires for example job id). Thanks
