Re: how to specify yarnqueue when starting a new job programmatically?
Hi Vitaliy, in the case of a session cluster you cannot influence the queue programmatically since Flink uses the value configured via yarn.application.queue which is read from the flink-conf.yaml. However, there is a way to influence the yarn queue programmatically if you use the per job mode. What you need to do is to create a StreamExecutionEnvironment manually by passing a Configuration instance. In this configuration instance you can set the respective configuration key to the desired yarn queue value. Please be aware that with this approach, you won't load Flink's global configuration specified in flink-conf.yaml. Cheers, Till On Thu, Mar 12, 2020 at 12:02 PM Xintong Song wrote: > Do you mean in the job java code, where you call “env.execute()”? > > I don't think so. The yarn queue is a Flink cluster level configuration, > and I don't think you can change the cluster level configurations in your > job code. You job might even be created after the cluster is started, e.g., > in a Flink YARN Session.[1] > > Thank you~ > > Xintong Song > > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html#flink-yarn-session > > On Thu, Mar 12, 2020 at 6:20 PM Vitaliy Semochkin > wrote: > >> Thank you Xintong Song, >> >> is there any way to queue programmatically, i.e. via java code? >> >> Regards, >> Vitaliy >> >> On Thu, Mar 12, 2020 at 5:56 AM Xintong Song >> wrote: >> >>> Hi Vitaliy, >>> >>> You can specify a yarn queue by either setting the configuration option >>> 'yarn.application.queue' [1], or using the command line option '-qu' (or >>> '--queue') [2]. >>> >>> Thank you~ >>> >>> Xintong Song >>> >>> >>> [1] >>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#yarn-application-queue >>> [2] >>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html#start-a-session >>> >>> On Thu, Mar 12, 2020 at 3:56 AM Vitaliy Semochkin >>> wrote: >>> Hi, How can I specify a yarn queue when I start a new job programmatically? Regards, Vitaliy >>>
Re: how to specify yarnqueue when starting a new job programmatically?
Do you mean in the job java code, where you call “env.execute()”? I don't think so. The yarn queue is a Flink cluster level configuration, and I don't think you can change the cluster level configurations in your job code. You job might even be created after the cluster is started, e.g., in a Flink YARN Session.[1] Thank you~ Xintong Song [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html#flink-yarn-session On Thu, Mar 12, 2020 at 6:20 PM Vitaliy Semochkin wrote: > Thank you Xintong Song, > > is there any way to queue programmatically, i.e. via java code? > > Regards, > Vitaliy > > On Thu, Mar 12, 2020 at 5:56 AM Xintong Song > wrote: > >> Hi Vitaliy, >> >> You can specify a yarn queue by either setting the configuration option >> 'yarn.application.queue' [1], or using the command line option '-qu' (or >> '--queue') [2]. >> >> Thank you~ >> >> Xintong Song >> >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#yarn-application-queue >> [2] >> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html#start-a-session >> >> On Thu, Mar 12, 2020 at 3:56 AM Vitaliy Semochkin >> wrote: >> >>> Hi, >>> >>> How can I specify a yarn queue when I start a new job programmatically? >>> >>> Regards, >>> Vitaliy >>> >>
Re: how to specify yarnqueue when starting a new job programmatically?
Thank you Xintong Song, is there any way to queue programmatically, i.e. via java code? Regards, Vitaliy On Thu, Mar 12, 2020 at 5:56 AM Xintong Song wrote: > Hi Vitaliy, > > You can specify a yarn queue by either setting the configuration option > 'yarn.application.queue' [1], or using the command line option '-qu' (or > '--queue') [2]. > > Thank you~ > > Xintong Song > > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#yarn-application-queue > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html#start-a-session > > On Thu, Mar 12, 2020 at 3:56 AM Vitaliy Semochkin > wrote: > >> Hi, >> >> How can I specify a yarn queue when I start a new job programmatically? >> >> Regards, >> Vitaliy >> >
Re: how to specify yarnqueue when starting a new job programmatically?
Hi Vitaliy, You can specify a yarn queue by either setting the configuration option 'yarn.application.queue' [1], or using the command line option '-qu' (or '--queue') [2]. Thank you~ Xintong Song [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#yarn-application-queue [2] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html#start-a-session On Thu, Mar 12, 2020 at 3:56 AM Vitaliy Semochkin wrote: > Hi, > > How can I specify a yarn queue when I start a new job programmatically? > > Regards, > Vitaliy >
how to specify yarnqueue when starting a new job programmatically?
Hi, How can I specify a yarn queue when I start a new job programmatically? Regards, Vitaliy
