Re: Container JMX port setting / discovery for Flink on YARN
Thanks Robert, it works like a charm. 2016-11-25 12:55 GMT+03:00 Robert Metzger : > Hi Yury, > > Flink is using its own JMX server instance (not the JVM's one). Therefore, > you can configure the server yourself. > Check out this documentation page: https://ci.apache.org/ > projects/flink/flink-docs-release-1.2/monitoring/metrics.html#reporter > > metrics.reporter.my_jmx_reporter.class: > org.apache.flink.metrics.jmx.JMXReporter > metrics.reporter.my_jmx_reporter.port: 9020-9040 > > > Flink will print the port it is using in the end into the log of each > TaskManager. > > > On Fri, Nov 25, 2016 at 10:53 AM, Yury Ruchin > wrote: > >> Thanks Stefan! I think this would help if I had just one container per >> node. But that's not my case - there are multiple TaskManagers running on >> the same node, so setting the same value will likely result in port >> conflict. >> >> 2016-11-25 12:28 GMT+03:00 Stefan Richter : >> >>> Hi, >>> >>> can you try adding the following to your flink.yaml? >>> >>> env.java.opts: -Dcom.sun.management.jmxremote >>> -Dcom.sun.management.jmxremote.port= >>> -Dcom.sun.management.jmxremote.authenticate=false >>> -Dcom.sun.management.jmxremote.ssl=false >>> >>> Best, >>> Stefan >>> >>> >>> > Am 24.11.2016 um 16:47 schrieb Yury Ruchin : >>> > >>> > Hi, >>> > >>> > I want to enable JMX for my Flink streaming app running in YARN >>> session. How can I specify which ports containers will listen to? If I >>> cannot control it (e. g. they will be chosen randomly) - how can I detect >>> which ports were picked by containers: inspecting logs, looking at the Web >>> UI etc.? >>> > >>> > Example: in Apache Storm it is easy to derive JMX port numbers from >>> worker port (slot) numbers and pass it as a JVM argument to the worker >>> start command, so that every slot is always associated with a stable and >>> well-known JMX port. >>> > >>> > Any clue is appreciated. Thanks! >>> >>> >> >
Re: Container JMX port setting / discovery for Flink on YARN
Hi Yury, Flink is using its own JMX server instance (not the JVM's one). Therefore, you can configure the server yourself. Check out this documentation page: https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html#reporter metrics.reporter.my_jmx_reporter.class: org.apache.flink.metrics.jmx.JMXReporter metrics.reporter.my_jmx_reporter.port: 9020-9040 Flink will print the port it is using in the end into the log of each TaskManager. On Fri, Nov 25, 2016 at 10:53 AM, Yury Ruchin wrote: > Thanks Stefan! I think this would help if I had just one container per > node. But that's not my case - there are multiple TaskManagers running on > the same node, so setting the same value will likely result in port > conflict. > > 2016-11-25 12:28 GMT+03:00 Stefan Richter : > >> Hi, >> >> can you try adding the following to your flink.yaml? >> >> env.java.opts: -Dcom.sun.management.jmxremote >> -Dcom.sun.management.jmxremote.port= >> -Dcom.sun.management.jmxremote.authenticate=false >> -Dcom.sun.management.jmxremote.ssl=false >> >> Best, >> Stefan >> >> >> > Am 24.11.2016 um 16:47 schrieb Yury Ruchin : >> > >> > Hi, >> > >> > I want to enable JMX for my Flink streaming app running in YARN >> session. How can I specify which ports containers will listen to? If I >> cannot control it (e. g. they will be chosen randomly) - how can I detect >> which ports were picked by containers: inspecting logs, looking at the Web >> UI etc.? >> > >> > Example: in Apache Storm it is easy to derive JMX port numbers from >> worker port (slot) numbers and pass it as a JVM argument to the worker >> start command, so that every slot is always associated with a stable and >> well-known JMX port. >> > >> > Any clue is appreciated. Thanks! >> >> >
Re: Container JMX port setting / discovery for Flink on YARN
Thanks Stefan! I think this would help if I had just one container per node. But that's not my case - there are multiple TaskManagers running on the same node, so setting the same value will likely result in port conflict. 2016-11-25 12:28 GMT+03:00 Stefan Richter : > Hi, > > can you try adding the following to your flink.yaml? > > env.java.opts: -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.port= > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > > Best, > Stefan > > > > Am 24.11.2016 um 16:47 schrieb Yury Ruchin : > > > > Hi, > > > > I want to enable JMX for my Flink streaming app running in YARN session. > How can I specify which ports containers will listen to? If I cannot > control it (e. g. they will be chosen randomly) - how can I detect which > ports were picked by containers: inspecting logs, looking at the Web UI > etc.? > > > > Example: in Apache Storm it is easy to derive JMX port numbers from > worker port (slot) numbers and pass it as a JVM argument to the worker > start command, so that every slot is always associated with a stable and > well-known JMX port. > > > > Any clue is appreciated. Thanks! > >
Re: Container JMX port setting / discovery for Flink on YARN
Hi, can you try adding the following to your flink.yaml? env.java.opts: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port= -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false Best, Stefan > Am 24.11.2016 um 16:47 schrieb Yury Ruchin : > > Hi, > > I want to enable JMX for my Flink streaming app running in YARN session. How > can I specify which ports containers will listen to? If I cannot control it > (e. g. they will be chosen randomly) - how can I detect which ports were > picked by containers: inspecting logs, looking at the Web UI etc.? > > Example: in Apache Storm it is easy to derive JMX port numbers from worker > port (slot) numbers and pass it as a JVM argument to the worker start > command, so that every slot is always associated with a stable and well-known > JMX port. > > Any clue is appreciated. Thanks!