Identify metrics belonging to the "same" task manager in kubernetes

2021-08-25 Thread gaurav kulkarni
Hi,  We have multiple flink clusters running in kubernetes. We plan to enable prometheus on these clusters. Looks like flink metrics emitted are of the format: "flink_taskmanager_Status_JVM_GarbageCollector_G1_Young_Generation_Time{host="10_244_2_6",tm_id="10_244_2_6:6122_2e3d7a",} 65.0" 1.  Si

Re: Identify metrics belonging to the "same" task manager in kubernetes

2021-08-26 Thread Chesnay Schepler
1) Can you clarify what you mean with "same"? Are you referring to something like an index, i.e., "This is TaskManager #2", without being tied to a specific process? 2) It is not possible for the PrometheusReporter. On 26/08/2021 04:22, gaurav kulkarni wrote: Hi, We have multiple flink clust

Re: Identify metrics belonging to the "same" task manager in kubernetes

2021-08-26 Thread gaurav kulkarni
Thanks for the response!  1) That's correct. I was wondering if there is any logical ID for task manager that stays the same even if its moved to a different pod and is a part of metrics emitted. The scenario is if a TM#1 was running on pod#1 and then is moved to pod#2 and if we can correlate t

Re: Identify metrics belonging to the "same" task manager in kubernetes

2021-08-26 Thread Chesnay Schepler
1) As is there is no way to accomplish this. 2) Yes (datadog, graphite) but if you are happy with Prometheus I instead recommend to fork the reporter and adjust it accordingly. Would only custom prefixes/suffixes work for your use-case, or would a custom label also work? 3) Sure, there's ple

Re: Identify metrics belonging to the "same" task manager in kubernetes

2021-08-26 Thread gaurav kulkarni
Thanks for the response! For #2, custom labels should work too for our case.  Thanks,Gaurav On Thursday, August 26, 2021, 08:28:27 AM PDT, Chesnay Schepler wrote: 1) As is there is no way to accomplish this. 2) Yes (datadog, graphite) but if you are happy with Prometheus I instea

Re: Identify metrics belonging to the "same" task manager in kubernetes

2021-08-26 Thread gaurav kulkarni
Hi,  I have another question: What mechanisms are usually used to correlate prometheus flink metrics for kubernetes?  Thanks,Gaurav  On Thursday, August 26, 2021, 10:06:30 AM PDT, gaurav kulkarni wrote: Thanks for the response! For #2, custom labels should work too for our case.  Than