Re: Flink-Metrics Prometheus - Native Histograms / Native Counters
Sounds good - I haven't looked too much into the comparison of the old and new client. Opened a ticket: https://issues.apache.org/jira/browse/FLINK-32508 I may have some time to support this ticket in 2 weeks but anyone can feel free to start working on it. Thanks, Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Fri, Jun 30, 2023 at 10:34 AM Martijn Visser wrote: > Hi Ryan, > > If I look at the changelog for the simpleclient 0.10 [1], they've switched > their data model. So if you upgrade to the later version, the data model > for existing Flink Prometheus users would be broken IIRC. That's why I > think option 1 is more clean: it provides the option to the user to choose > which package they want to use. Either the new one, with a new data model, > or the current one, with the existing data model. > > Best regards, > > Martijn > > [1] https://github.com/prometheus/client_java/releases/tag/parent-0.10.0 > > On Fri, Jun 30, 2023 at 4:23 PM Ryan van Huuksloot > wrote: > > > I'd have to check but the original plan is to upgrade the client but keep > > the flink-metrics-prometheus implementation the same. This should keep > the > > metrics collection consistent even with the client upgrade - this would > > need to be verified. > > > > But if that is the worry then we could create a new package to keep > things > > distinct. > > > > Thanks, > > > > Ryan van Huuksloot > > Sr. Production Engineer | Streaming Platform > > [image: Shopify] > > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email > > > > > > > > On Fri, Jun 30, 2023 at 10:02 AM Martijn Visser < > martijnvis...@apache.org> > > wrote: > > > > > Hi Patrick, > > > > > > Yeah, but you would need the latest version of the client, which would > > > break the implementation for the current, outdated one, wouldn't it? > > > > > > Best regards, > > > > > > Martijn > > > > > > On Fri, Jun 30, 2023 at 3:35 PM Ryan van Huuksloot > > > wrote: > > > > > > > Hi Martijn, > > > > > > > > Option 2 and 3 would use a single client. It would just register the > > > > metrics differently. > > > > > > > > Does that make sense? Does that change your perspective? > > > > > > > > Thanks, > > > > > > > > Ryan van Huuksloot > > > > Sr. Production Engineer | Streaming Platform > > > > [image: Shopify] > > > > < > > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email > > > > > > > > > > > > > > > > On Fri, Jun 30, 2023 at 7:49 AM Martijn Visser < > > martijnvis...@apache.org > > > > > > > > wrote: > > > > > > > > > Hi Ryan, > > > > > > > > > > I think option 2 and option 3 won't work, because there can be only > > one > > > > > version of the client. I don't think we should make a clean break > on > > > > > metrics in a minor version, but only in major. All in all, I think > > > > option 1 > > > > > would be the best. We could deprecate the existing one and remove > it > > > > > with Flink 2.0 imho. > > > > > > > > > > Best regards, > > > > > > > > > > Martijn > > > > > > > > > > On Thu, Jun 29, 2023 at 5:56 PM Ryan van Huuksloot > > > > > wrote: > > > > > > > > > > > Hi Martijn, > > > > > > > > > > > > Our team shared the same concern. We've considered a few options: > > > > > > > > > > > > > > > > > > *1. Add a new package such as `flink-metrics-prometheus-native` > and > > > > > > eventually deprecate the original.* > > > > > > *Pros:* > > > > > > - Supports backward compatibility. > > > > > > *Cons:* > > > > > > - Two packages to maintain in the interim. > > > > > > - Not consistent with other metrics packages. > > > > > > > > > > > > *2. Maintain the same logic in flink-metrics-prometheus and write > > new > > > > > > natively typed metrics to a different metric name in Prometheus, >
[jira] [Created] (FLINK-32508) Flink-Metrics Prometheus - Native Histograms / Native Counters
Ryan van Huuksloot created FLINK-32508: -- Summary: Flink-Metrics Prometheus - Native Histograms / Native Counters Key: FLINK-32508 URL: https://issues.apache.org/jira/browse/FLINK-32508 Project: Flink Issue Type: Technical Debt Components: Runtime / Metrics Reporter: Ryan van Huuksloot Fix For: 1.18.0, 1.19.0 There are new metric types in Prometheus that would allow for the exporter to write Counters and Histograms as Native metrics in prometheus (vs writing as Gauges). This requires an update to the Prometheus Client which has changed it's spec. To accommodate the new metric types while retaining the old option for prometheus metrics, the recommendation is to *Add a new package such as `flink-metrics-prometheus-native` and eventually deprecate the original.* Discussed more on the mailing list: https://lists.apache.org/thread/kbo3973whb8nj5xvkpvhxrmgtmnbkhlv -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Flink-Metrics Prometheus - Native Histograms / Native Counters
Hi Ryan, If I look at the changelog for the simpleclient 0.10 [1], they've switched their data model. So if you upgrade to the later version, the data model for existing Flink Prometheus users would be broken IIRC. That's why I think option 1 is more clean: it provides the option to the user to choose which package they want to use. Either the new one, with a new data model, or the current one, with the existing data model. Best regards, Martijn [1] https://github.com/prometheus/client_java/releases/tag/parent-0.10.0 On Fri, Jun 30, 2023 at 4:23 PM Ryan van Huuksloot wrote: > I'd have to check but the original plan is to upgrade the client but keep > the flink-metrics-prometheus implementation the same. This should keep the > metrics collection consistent even with the client upgrade - this would > need to be verified. > > But if that is the worry then we could create a new package to keep things > distinct. > > Thanks, > > Ryan van Huuksloot > Sr. Production Engineer | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Fri, Jun 30, 2023 at 10:02 AM Martijn Visser > wrote: > > > Hi Patrick, > > > > Yeah, but you would need the latest version of the client, which would > > break the implementation for the current, outdated one, wouldn't it? > > > > Best regards, > > > > Martijn > > > > On Fri, Jun 30, 2023 at 3:35 PM Ryan van Huuksloot > > wrote: > > > > > Hi Martijn, > > > > > > Option 2 and 3 would use a single client. It would just register the > > > metrics differently. > > > > > > Does that make sense? Does that change your perspective? > > > > > > Thanks, > > > > > > Ryan van Huuksloot > > > Sr. Production Engineer | Streaming Platform > > > [image: Shopify] > > > < > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email > > > > > > > > > > > > On Fri, Jun 30, 2023 at 7:49 AM Martijn Visser < > martijnvis...@apache.org > > > > > > wrote: > > > > > > > Hi Ryan, > > > > > > > > I think option 2 and option 3 won't work, because there can be only > one > > > > version of the client. I don't think we should make a clean break on > > > > metrics in a minor version, but only in major. All in all, I think > > > option 1 > > > > would be the best. We could deprecate the existing one and remove it > > > > with Flink 2.0 imho. > > > > > > > > Best regards, > > > > > > > > Martijn > > > > > > > > On Thu, Jun 29, 2023 at 5:56 PM Ryan van Huuksloot > > > > wrote: > > > > > > > > > Hi Martijn, > > > > > > > > > > Our team shared the same concern. We've considered a few options: > > > > > > > > > > > > > > > *1. Add a new package such as `flink-metrics-prometheus-native` and > > > > > eventually deprecate the original.* > > > > > *Pros:* > > > > > - Supports backward compatibility. > > > > > *Cons:* > > > > > - Two packages to maintain in the interim. > > > > > - Not consistent with other metrics packages. > > > > > > > > > > *2. Maintain the same logic in flink-metrics-prometheus and write > new > > > > > natively typed metrics to a different metric name in Prometheus, in > > > > > addition to the original metric.* > > > > > > > > > > *Pros:* > > > > > - Supports backward compatibility. > > > > > *Cons:* > > > > > - Nearly doubles the metrics being captured by default. > > > > > - The naming convention will permanently differ when the original > > names > > > > are > > > > > deprecated. > > > > > - The original names will likely be deprecated at some point. > > > > > > > > > > *3. Maintain the same logic in flink-metrics-prometheus. However, > if > > > you > > > > > use a flink-conf option, natively typed metrics would be written to > > the > > > > > same names instead of the original metric types.* > > > > > > > > > > *Pros:* > > > > > - Supports backwards compatibility > > > > > - No double metrics > > > > > *Cons:* > > > > &g
Re: Flink-Metrics Prometheus - Native Histograms / Native Counters
I'd have to check but the original plan is to upgrade the client but keep the flink-metrics-prometheus implementation the same. This should keep the metrics collection consistent even with the client upgrade - this would need to be verified. But if that is the worry then we could create a new package to keep things distinct. Thanks, Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Fri, Jun 30, 2023 at 10:02 AM Martijn Visser wrote: > Hi Patrick, > > Yeah, but you would need the latest version of the client, which would > break the implementation for the current, outdated one, wouldn't it? > > Best regards, > > Martijn > > On Fri, Jun 30, 2023 at 3:35 PM Ryan van Huuksloot > wrote: > > > Hi Martijn, > > > > Option 2 and 3 would use a single client. It would just register the > > metrics differently. > > > > Does that make sense? Does that change your perspective? > > > > Thanks, > > > > Ryan van Huuksloot > > Sr. Production Engineer | Streaming Platform > > [image: Shopify] > > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email > > > > > > > > On Fri, Jun 30, 2023 at 7:49 AM Martijn Visser > > > wrote: > > > > > Hi Ryan, > > > > > > I think option 2 and option 3 won't work, because there can be only one > > > version of the client. I don't think we should make a clean break on > > > metrics in a minor version, but only in major. All in all, I think > > option 1 > > > would be the best. We could deprecate the existing one and remove it > > > with Flink 2.0 imho. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Thu, Jun 29, 2023 at 5:56 PM Ryan van Huuksloot > > > wrote: > > > > > > > Hi Martijn, > > > > > > > > Our team shared the same concern. We've considered a few options: > > > > > > > > > > > > *1. Add a new package such as `flink-metrics-prometheus-native` and > > > > eventually deprecate the original.* > > > > *Pros:* > > > > - Supports backward compatibility. > > > > *Cons:* > > > > - Two packages to maintain in the interim. > > > > - Not consistent with other metrics packages. > > > > > > > > *2. Maintain the same logic in flink-metrics-prometheus and write new > > > > natively typed metrics to a different metric name in Prometheus, in > > > > addition to the original metric.* > > > > > > > > *Pros:* > > > > - Supports backward compatibility. > > > > *Cons:* > > > > - Nearly doubles the metrics being captured by default. > > > > - The naming convention will permanently differ when the original > names > > > are > > > > deprecated. > > > > - The original names will likely be deprecated at some point. > > > > > > > > *3. Maintain the same logic in flink-metrics-prometheus. However, if > > you > > > > use a flink-conf option, natively typed metrics would be written to > the > > > > same names instead of the original metric types.* > > > > > > > > *Pros:* > > > > - Supports backwards compatibility > > > > - No double metrics > > > > *Cons:* > > > > - Increases the maintenance burden. > > > > - Would require future migrations > > > > > > > > *4. Make a clean break and swap the types in > flink-metrics-prometheus, > > > > releasing it in 1.18 or 1.19 with a note.* > > > > > > > > *Pros:* > > > > - Avoids duplicate metrics and packages. > > > > - No future maintenance burden. > > > > *Cons:* > > > > -Introduces a breaking change. > > > > - Metrics may silently fail in dashboards if the graphs do not > support > > > the > > > > new data type (I would need to conduct more testing to determine how > > > often > > > > this occurs). > > > > > > > > I lean towards option 4, and we would communicate the change > internally > > > as > > > > part of a minor version upgrade. I'm open to other ideas and would > > > welcome > > > > further discussion on what the OSS community prefers. > > > > > > > > Thanks, > > > > > >
Re: Flink-Metrics Prometheus - Native Histograms / Native Counters
Hi Patrick, Yeah, but you would need the latest version of the client, which would break the implementation for the current, outdated one, wouldn't it? Best regards, Martijn On Fri, Jun 30, 2023 at 3:35 PM Ryan van Huuksloot wrote: > Hi Martijn, > > Option 2 and 3 would use a single client. It would just register the > metrics differently. > > Does that make sense? Does that change your perspective? > > Thanks, > > Ryan van Huuksloot > Sr. Production Engineer | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Fri, Jun 30, 2023 at 7:49 AM Martijn Visser > wrote: > > > Hi Ryan, > > > > I think option 2 and option 3 won't work, because there can be only one > > version of the client. I don't think we should make a clean break on > > metrics in a minor version, but only in major. All in all, I think > option 1 > > would be the best. We could deprecate the existing one and remove it > > with Flink 2.0 imho. > > > > Best regards, > > > > Martijn > > > > On Thu, Jun 29, 2023 at 5:56 PM Ryan van Huuksloot > > wrote: > > > > > Hi Martijn, > > > > > > Our team shared the same concern. We've considered a few options: > > > > > > > > > *1. Add a new package such as `flink-metrics-prometheus-native` and > > > eventually deprecate the original.* > > > *Pros:* > > > - Supports backward compatibility. > > > *Cons:* > > > - Two packages to maintain in the interim. > > > - Not consistent with other metrics packages. > > > > > > *2. Maintain the same logic in flink-metrics-prometheus and write new > > > natively typed metrics to a different metric name in Prometheus, in > > > addition to the original metric.* > > > > > > *Pros:* > > > - Supports backward compatibility. > > > *Cons:* > > > - Nearly doubles the metrics being captured by default. > > > - The naming convention will permanently differ when the original names > > are > > > deprecated. > > > - The original names will likely be deprecated at some point. > > > > > > *3. Maintain the same logic in flink-metrics-prometheus. However, if > you > > > use a flink-conf option, natively typed metrics would be written to the > > > same names instead of the original metric types.* > > > > > > *Pros:* > > > - Supports backwards compatibility > > > - No double metrics > > > *Cons:* > > > - Increases the maintenance burden. > > > - Would require future migrations > > > > > > *4. Make a clean break and swap the types in flink-metrics-prometheus, > > > releasing it in 1.18 or 1.19 with a note.* > > > > > > *Pros:* > > > - Avoids duplicate metrics and packages. > > > - No future maintenance burden. > > > *Cons:* > > > -Introduces a breaking change. > > > - Metrics may silently fail in dashboards if the graphs do not support > > the > > > new data type (I would need to conduct more testing to determine how > > often > > > this occurs). > > > > > > I lean towards option 4, and we would communicate the change internally > > as > > > part of a minor version upgrade. I'm open to other ideas and would > > welcome > > > further discussion on what the OSS community prefers. > > > > > > Thanks, > > > > > > Ryan van Huuksloot > > > Sr. Production Engineer | Streaming Platform > > > [image: Shopify] > > > < > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email > > > > > > > > > > > > On Thu, Jun 29, 2023 at 4:23 AM Martijn Visser < > martijnvis...@apache.org > > > > > > wrote: > > > > > > > Hi Ryan, > > > > > > > > I think there definitely is an interest in the > > > > flink-metrics-prometheus, but I do see some challenges as well. Given > > > > that the Prometheus simpleclient doesn't yet have a major version, > > > > there are breaking changes happening in that. If we would update > this, > > > > it can/probably breaks the metrics for users, which is an undesirable > > > > situation. Any thoughts on how we could avoid that situation? > > > > > > > > Best regards, > > > > > > > > Martijn > > > > > > > > On
Re: Flink-Metrics Prometheus - Native Histograms / Native Counters
Hi Martijn, Option 2 and 3 would use a single client. It would just register the metrics differently. Does that make sense? Does that change your perspective? Thanks, Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Fri, Jun 30, 2023 at 7:49 AM Martijn Visser wrote: > Hi Ryan, > > I think option 2 and option 3 won't work, because there can be only one > version of the client. I don't think we should make a clean break on > metrics in a minor version, but only in major. All in all, I think option 1 > would be the best. We could deprecate the existing one and remove it > with Flink 2.0 imho. > > Best regards, > > Martijn > > On Thu, Jun 29, 2023 at 5:56 PM Ryan van Huuksloot > wrote: > > > Hi Martijn, > > > > Our team shared the same concern. We've considered a few options: > > > > > > *1. Add a new package such as `flink-metrics-prometheus-native` and > > eventually deprecate the original.* > > *Pros:* > > - Supports backward compatibility. > > *Cons:* > > - Two packages to maintain in the interim. > > - Not consistent with other metrics packages. > > > > *2. Maintain the same logic in flink-metrics-prometheus and write new > > natively typed metrics to a different metric name in Prometheus, in > > addition to the original metric.* > > > > *Pros:* > > - Supports backward compatibility. > > *Cons:* > > - Nearly doubles the metrics being captured by default. > > - The naming convention will permanently differ when the original names > are > > deprecated. > > - The original names will likely be deprecated at some point. > > > > *3. Maintain the same logic in flink-metrics-prometheus. However, if you > > use a flink-conf option, natively typed metrics would be written to the > > same names instead of the original metric types.* > > > > *Pros:* > > - Supports backwards compatibility > > - No double metrics > > *Cons:* > > - Increases the maintenance burden. > > - Would require future migrations > > > > *4. Make a clean break and swap the types in flink-metrics-prometheus, > > releasing it in 1.18 or 1.19 with a note.* > > > > *Pros:* > > - Avoids duplicate metrics and packages. > > - No future maintenance burden. > > *Cons:* > > -Introduces a breaking change. > > - Metrics may silently fail in dashboards if the graphs do not support > the > > new data type (I would need to conduct more testing to determine how > often > > this occurs). > > > > I lean towards option 4, and we would communicate the change internally > as > > part of a minor version upgrade. I'm open to other ideas and would > welcome > > further discussion on what the OSS community prefers. > > > > Thanks, > > > > Ryan van Huuksloot > > Sr. Production Engineer | Streaming Platform > > [image: Shopify] > > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email > > > > > > > > On Thu, Jun 29, 2023 at 4:23 AM Martijn Visser > > > wrote: > > > > > Hi Ryan, > > > > > > I think there definitely is an interest in the > > > flink-metrics-prometheus, but I do see some challenges as well. Given > > > that the Prometheus simpleclient doesn't yet have a major version, > > > there are breaking changes happening in that. If we would update this, > > > it can/probably breaks the metrics for users, which is an undesirable > > > situation. Any thoughts on how we could avoid that situation? > > > > > > Best regards, > > > > > > Martijn > > > > > > On Tue, Jun 20, 2023 at 3:53 PM Ryan van Huuksloot > > > wrote: > > > > > > > > Following up, any interest in flink-metrics-prometheus? It is quite a > > > stale > > > > package. I would be interested in contributing - time permitting. > > > > > > > > Ryan van Huuksloot > > > > Sr. Production Engineer | Streaming Platform > > > > [image: Shopify] > > > > < > > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email > > > > > > > > > > > > > > > > On Thu, Jun 15, 2023 at 2:16 PM Ryan van Huuksloot < > > > > ryan.vanhuuksl...@shopify.com> wrote: > > > > > > > > > Hello, > > > > > > > > > > Internally we use the flink-metrics-prometheus jar and we noticed > > that > > > the > > > > > code is a little out of date. Primarily, there are new metric types > > in > > > > > Prometheus that would allow for the exporter to write Counters and > > > > > Histograms as Native metrics in prometheus (vs writing as Gauges). > > > > > > > > > > I noticed that there was a closed PR for the simpleclient: > > > > > https://github.com/apache/flink/pull/21047 - which has what is > > > required > > > > > for the native metrics but may cause other maintenance tickets. > > > > > > > > > > Is there any appetite from the community to update this exporter? > > > > > > > > > > Thanks, > > > > > > > > > > Ryan van Huuksloot > > > > > Sr. Production Engineer | Streaming Platform > > > > > [image: Shopify] > > > > > < > > > > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > > > > > > > > >
Re: Flink-Metrics Prometheus - Native Histograms / Native Counters
Hi Ryan, I think option 2 and option 3 won't work, because there can be only one version of the client. I don't think we should make a clean break on metrics in a minor version, but only in major. All in all, I think option 1 would be the best. We could deprecate the existing one and remove it with Flink 2.0 imho. Best regards, Martijn On Thu, Jun 29, 2023 at 5:56 PM Ryan van Huuksloot wrote: > Hi Martijn, > > Our team shared the same concern. We've considered a few options: > > > *1. Add a new package such as `flink-metrics-prometheus-native` and > eventually deprecate the original.* > *Pros:* > - Supports backward compatibility. > *Cons:* > - Two packages to maintain in the interim. > - Not consistent with other metrics packages. > > *2. Maintain the same logic in flink-metrics-prometheus and write new > natively typed metrics to a different metric name in Prometheus, in > addition to the original metric.* > > *Pros:* > - Supports backward compatibility. > *Cons:* > - Nearly doubles the metrics being captured by default. > - The naming convention will permanently differ when the original names are > deprecated. > - The original names will likely be deprecated at some point. > > *3. Maintain the same logic in flink-metrics-prometheus. However, if you > use a flink-conf option, natively typed metrics would be written to the > same names instead of the original metric types.* > > *Pros:* > - Supports backwards compatibility > - No double metrics > *Cons:* > - Increases the maintenance burden. > - Would require future migrations > > *4. Make a clean break and swap the types in flink-metrics-prometheus, > releasing it in 1.18 or 1.19 with a note.* > > *Pros:* > - Avoids duplicate metrics and packages. > - No future maintenance burden. > *Cons:* > -Introduces a breaking change. > - Metrics may silently fail in dashboards if the graphs do not support the > new data type (I would need to conduct more testing to determine how often > this occurs). > > I lean towards option 4, and we would communicate the change internally as > part of a minor version upgrade. I'm open to other ideas and would welcome > further discussion on what the OSS community prefers. > > Thanks, > > Ryan van Huuksloot > Sr. Production Engineer | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Thu, Jun 29, 2023 at 4:23 AM Martijn Visser > wrote: > > > Hi Ryan, > > > > I think there definitely is an interest in the > > flink-metrics-prometheus, but I do see some challenges as well. Given > > that the Prometheus simpleclient doesn't yet have a major version, > > there are breaking changes happening in that. If we would update this, > > it can/probably breaks the metrics for users, which is an undesirable > > situation. Any thoughts on how we could avoid that situation? > > > > Best regards, > > > > Martijn > > > > On Tue, Jun 20, 2023 at 3:53 PM Ryan van Huuksloot > > wrote: > > > > > > Following up, any interest in flink-metrics-prometheus? It is quite a > > stale > > > package. I would be interested in contributing - time permitting. > > > > > > Ryan van Huuksloot > > > Sr. Production Engineer | Streaming Platform > > > [image: Shopify] > > > < > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email > > > > > > > > > > > > On Thu, Jun 15, 2023 at 2:16 PM Ryan van Huuksloot < > > > ryan.vanhuuksl...@shopify.com> wrote: > > > > > > > Hello, > > > > > > > > Internally we use the flink-metrics-prometheus jar and we noticed > that > > the > > > > code is a little out of date. Primarily, there are new metric types > in > > > > Prometheus that would allow for the exporter to write Counters and > > > > Histograms as Native metrics in prometheus (vs writing as Gauges). > > > > > > > > I noticed that there was a closed PR for the simpleclient: > > > > https://github.com/apache/flink/pull/21047 - which has what is > > required > > > > for the native metrics but may cause other maintenance tickets. > > > > > > > > Is there any appetite from the community to update this exporter? > > > > > > > > Thanks, > > > > > > > > Ryan van Huuksloot > > > > Sr. Production Engineer | Streaming Platform > > > > [image: Shopify] > > > > < > > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > > > > >
Re: Flink-Metrics Prometheus - Native Histograms / Native Counters
Hi Martijn, Our team shared the same concern. We've considered a few options: *1. Add a new package such as `flink-metrics-prometheus-native` and eventually deprecate the original.* *Pros:* - Supports backward compatibility. *Cons:* - Two packages to maintain in the interim. - Not consistent with other metrics packages. *2. Maintain the same logic in flink-metrics-prometheus and write new natively typed metrics to a different metric name in Prometheus, in addition to the original metric.* *Pros:* - Supports backward compatibility. *Cons:* - Nearly doubles the metrics being captured by default. - The naming convention will permanently differ when the original names are deprecated. - The original names will likely be deprecated at some point. *3. Maintain the same logic in flink-metrics-prometheus. However, if you use a flink-conf option, natively typed metrics would be written to the same names instead of the original metric types.* *Pros:* - Supports backwards compatibility - No double metrics *Cons:* - Increases the maintenance burden. - Would require future migrations *4. Make a clean break and swap the types in flink-metrics-prometheus, releasing it in 1.18 or 1.19 with a note.* *Pros:* - Avoids duplicate metrics and packages. - No future maintenance burden. *Cons:* -Introduces a breaking change. - Metrics may silently fail in dashboards if the graphs do not support the new data type (I would need to conduct more testing to determine how often this occurs). I lean towards option 4, and we would communicate the change internally as part of a minor version upgrade. I'm open to other ideas and would welcome further discussion on what the OSS community prefers. Thanks, Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Thu, Jun 29, 2023 at 4:23 AM Martijn Visser wrote: > Hi Ryan, > > I think there definitely is an interest in the > flink-metrics-prometheus, but I do see some challenges as well. Given > that the Prometheus simpleclient doesn't yet have a major version, > there are breaking changes happening in that. If we would update this, > it can/probably breaks the metrics for users, which is an undesirable > situation. Any thoughts on how we could avoid that situation? > > Best regards, > > Martijn > > On Tue, Jun 20, 2023 at 3:53 PM Ryan van Huuksloot > wrote: > > > > Following up, any interest in flink-metrics-prometheus? It is quite a > stale > > package. I would be interested in contributing - time permitting. > > > > Ryan van Huuksloot > > Sr. Production Engineer | Streaming Platform > > [image: Shopify] > > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email > > > > > > > > On Thu, Jun 15, 2023 at 2:16 PM Ryan van Huuksloot < > > ryan.vanhuuksl...@shopify.com> wrote: > > > > > Hello, > > > > > > Internally we use the flink-metrics-prometheus jar and we noticed that > the > > > code is a little out of date. Primarily, there are new metric types in > > > Prometheus that would allow for the exporter to write Counters and > > > Histograms as Native metrics in prometheus (vs writing as Gauges). > > > > > > I noticed that there was a closed PR for the simpleclient: > > > https://github.com/apache/flink/pull/21047 - which has what is > required > > > for the native metrics but may cause other maintenance tickets. > > > > > > Is there any appetite from the community to update this exporter? > > > > > > Thanks, > > > > > > Ryan van Huuksloot > > > Sr. Production Engineer | Streaming Platform > > > [image: Shopify] > > > < > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > >
Re: Flink-Metrics Prometheus - Native Histograms / Native Counters
Hi Ryan, I think there definitely is an interest in the flink-metrics-prometheus, but I do see some challenges as well. Given that the Prometheus simpleclient doesn't yet have a major version, there are breaking changes happening in that. If we would update this, it can/probably breaks the metrics for users, which is an undesirable situation. Any thoughts on how we could avoid that situation? Best regards, Martijn On Tue, Jun 20, 2023 at 3:53 PM Ryan van Huuksloot wrote: > > Following up, any interest in flink-metrics-prometheus? It is quite a stale > package. I would be interested in contributing - time permitting. > > Ryan van Huuksloot > Sr. Production Engineer | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Thu, Jun 15, 2023 at 2:16 PM Ryan van Huuksloot < > ryan.vanhuuksl...@shopify.com> wrote: > > > Hello, > > > > Internally we use the flink-metrics-prometheus jar and we noticed that the > > code is a little out of date. Primarily, there are new metric types in > > Prometheus that would allow for the exporter to write Counters and > > Histograms as Native metrics in prometheus (vs writing as Gauges). > > > > I noticed that there was a closed PR for the simpleclient: > > https://github.com/apache/flink/pull/21047 - which has what is required > > for the native metrics but may cause other maintenance tickets. > > > > Is there any appetite from the community to update this exporter? > > > > Thanks, > > > > Ryan van Huuksloot > > Sr. Production Engineer | Streaming Platform > > [image: Shopify] > > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > >
Re: Flink-Metrics Prometheus - Native Histograms / Native Counters
Following up, any interest in flink-metrics-prometheus? It is quite a stale package. I would be interested in contributing - time permitting. Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Thu, Jun 15, 2023 at 2:16 PM Ryan van Huuksloot < ryan.vanhuuksl...@shopify.com> wrote: > Hello, > > Internally we use the flink-metrics-prometheus jar and we noticed that the > code is a little out of date. Primarily, there are new metric types in > Prometheus that would allow for the exporter to write Counters and > Histograms as Native metrics in prometheus (vs writing as Gauges). > > I noticed that there was a closed PR for the simpleclient: > https://github.com/apache/flink/pull/21047 - which has what is required > for the native metrics but may cause other maintenance tickets. > > Is there any appetite from the community to update this exporter? > > Thanks, > > Ryan van Huuksloot > Sr. Production Engineer | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> >
Flink-Metrics Prometheus - Native Histograms / Native Counters
Hello, Internally we use the flink-metrics-prometheus jar and we noticed that the code is a little out of date. Primarily, there are new metric types in Prometheus that would allow for the exporter to write Counters and Histograms as Native metrics in prometheus (vs writing as Gauges). I noticed that there was a closed PR for the simpleclient: https://github.com/apache/flink/pull/21047 - which has what is required for the native metrics but may cause other maintenance tickets. Is there any appetite from the community to update this exporter? Thanks, Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email>
[jira] [Created] (FLINK-29970) Prometheus cannot collect flink metrics
钟洋洋 created FLINK-29970:
---
Summary: Prometheus cannot collect flink metrics
Key: FLINK-29970
URL: https://issues.apache.org/jira/browse/FLINK-29970
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes
Affects Versions: 1.14.6, 1.15.2
Reporter: 钟洋洋
When I use the native k8s method to deploy my flink application cluster, if I
do not manually deploy a Service to expose the 9249 port of my jobmanager and
taskmanager, the prometheus I deployed in k8s cannot collect the metrics in my
flink. Should flink generate these services itself when deploying.
My deploy command is（Some content is omitted）
{code:sh}
flink run-application --target-application -Dmetrics.reporters=prom
-Dmetric.reporter.prom.class=org.apache.flink.metrics.prometheus.PrometheusReport
-Dkubernetes.jobmanager.annotaions=prometheus.io/scrape:true,prometheus.io/port:9249
-DDkubernetes.jobmanager.annotaions=prometheus.io/scrape:true,prometheus.io/port:9249{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
FW: 1.13.2 - Flink Metrics not updated during job execution
From: "Oliveira, Joao / Kuehne + Nagel / Opo MI-DH" Date: Wednesday, 3 August 2022 at 09:54 To: "u...@flink.apache.org" Cc: "Melo, Jose / Kuehne + Nagel / Opo MI-DH" Subject: 1.13.2 - Flink Metrics not updated during job execution Hi Flink team, My name is João Oliveira, i’m living in Portugal and I’m working at Kuehne+Nagel. We are creating a process to handle millions of records with apache flink to support logistics data pipelines. We are moving from kinesis sources/sink to kafka sources/sink. However, in the flink dashboard, the job metrics are not being updated in the near-real-time. Do you want can be wrong with the job/version? Evidence in the attachments Best, João Oliveira Kuehne+Nagel
[jira] [Created] (FLINK-27914) Integrate JOSDK metrics with Flink Metrics reporter
Gyula Fora created FLINK-27914: -- Summary: Integrate JOSDK metrics with Flink Metrics reporter Key: FLINK-27914 URL: https://issues.apache.org/jira/browse/FLINK-27914 Project: Flink Issue Type: New Feature Components: Kubernetes Operator Reporter: Gyula Fora Fix For: kubernetes-operator-1.1.0 The Java Operator SDK comes with an internal metric interface that could be implemented to forward metrics/measurements to the Flink metric registries. We should investigate and implement this if possible. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27661) [Metric]Flink-Metrics pushgateway support authentication
jiangchunyang created FLINK-27661: - Summary: [Metric]Flink-Metrics pushgateway support authentication Key: FLINK-27661 URL: https://issues.apache.org/jira/browse/FLINK-27661 Project: Flink Issue Type: Improvement Components: Runtime / Metrics Environment: Flink：1.13.0 Reporter: jiangchunyang Fix For: 1.13.0 We found that the native PushGateway does not support authentication. As a result, the metrics data in on YARN mode cannot be reported to pushGateway with authentication. Although we have some other solutions, such as landing files and others, we think pushGateway is the best solution. So I decided to do some implementation on my own, and will submit pr to the community later -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27163) Fix typo issue in Flink Metrics documentation
hao wang created FLINK-27163: Summary: Fix typo issue in Flink Metrics documentation Key: FLINK-27163 URL: https://issues.apache.org/jira/browse/FLINK-27163 Project: Flink Issue Type: Bug Components: Documentation Affects Versions: 1.15.0, 1.16.0 Reporter: hao wang Attachments: 20220411145958283.png The Cluster module in the metrics documentation has five items，but only four are specified. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26230) [JUnit5 Migration] Module: flink-metrics-core
Chesnay Schepler created FLINK-26230: Summary: [JUnit5 Migration] Module: flink-metrics-core Key: FLINK-26230 URL: https://issues.apache.org/jira/browse/FLINK-26230 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics, Tests Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26229) [JUnit5 Migration] Module: flink-metrics-datadog
Chesnay Schepler created FLINK-26229: Summary: [JUnit5 Migration] Module: flink-metrics-datadog Key: FLINK-26229 URL: https://issues.apache.org/jira/browse/FLINK-26229 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics, Tests Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26228) [JUnit5 Migration] Module: flink-metrics-dropwizard
Chesnay Schepler created FLINK-26228: Summary: [JUnit5 Migration] Module: flink-metrics-dropwizard Key: FLINK-26228 URL: https://issues.apache.org/jira/browse/FLINK-26228 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics, Tests Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26227) [JUnit5 Migration] Module: flink-metrics-graphite
Chesnay Schepler created FLINK-26227: Summary: [JUnit5 Migration] Module: flink-metrics-graphite Key: FLINK-26227 URL: https://issues.apache.org/jira/browse/FLINK-26227 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics, Tests Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26226) [JUnit5 Migration] Module: flink-metrics-influxdb
Chesnay Schepler created FLINK-26226: Summary: [JUnit5 Migration] Module: flink-metrics-influxdb Key: FLINK-26226 URL: https://issues.apache.org/jira/browse/FLINK-26226 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics, Tests Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26225) [JUnit5 Migration] Module: flink-metrics-jmx
Chesnay Schepler created FLINK-26225: Summary: [JUnit5 Migration] Module: flink-metrics-jmx Key: FLINK-26225 URL: https://issues.apache.org/jira/browse/FLINK-26225 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics, Tests Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26221) [JUnit5 Migration] Module: flink-metrics-prometheus
Chesnay Schepler created FLINK-26221: Summary: [JUnit5 Migration] Module: flink-metrics-prometheus Key: FLINK-26221 URL: https://issues.apache.org/jira/browse/FLINK-26221 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics, Tests Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26219) [JUnit5 Migration] Module: flink-metrics-slf4j
Chesnay Schepler created FLINK-26219: Summary: [JUnit5 Migration] Module: flink-metrics-slf4j Key: FLINK-26219 URL: https://issues.apache.org/jira/browse/FLINK-26219 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics, Tests Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26215) [JUnit5 Migration] Module: flink-metrics-statsd
Chesnay Schepler created FLINK-26215: Summary: [JUnit5 Migration] Module: flink-metrics-statsd Key: FLINK-26215 URL: https://issues.apache.org/jira/browse/FLINK-26215 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24896) Failed to execute goal com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp (default) on project flink-metrics-core
Yun Gao created FLINK-24896:
---
Summary: Failed to execute goal
com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp (default) on project
flink-metrics-core
Key: FLINK-24896
URL: https://issues.apache.org/jira/browse/FLINK-24896
Project: Flink
Issue Type: Bug
Components: Build System / Azure Pipelines
Affects Versions: 1.15.0
Reporter: Yun Gao
{code:java}
[ERROR] Failed to execute goal
com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp (default) on project
flink-metrics-core: Execution default of goal
com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp failed: Marshalling
of XML document failed: Implementation of JAXB-API has not been found on module
path or classpath. com.sun.xml.internal.bind.v2.ContextFactory -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :flink-metrics-core
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-24514) Incorrect Flink Metrics on Job-Internal-Restart:
Alok Singh created FLINK-24514: -- Summary: Incorrect Flink Metrics on Job-Internal-Restart: Key: FLINK-24514 URL: https://issues.apache.org/jira/browse/FLINK-24514 Project: Flink Issue Type: Bug Components: Runtime / Metrics Affects Versions: 1.12.1 Reporter: Alok Singh Attachments: Screenshot 2021-10-12 at 4.46.49 PM.png, Screenshot 2021-10-12 at 4.47.17 PM.png, Screenshot 2021-10-12 at 4.47.29 PM.png, Screenshot 2021-10-12 at 4.47.41 PM.png We have been seeing metrics showing multi-folded values after Flink Job restarts (due to some internal exceptions for example something like while deployment, the job didn't get the Task Managers in time and then it restarted on its own.) Metrics implementation: # We have done metrics implementation using Meter. # We are using Accumulators.scala to define our metrics name as Value and use this as key and MeterView as value to define it under a Map in CustomMetrics.scala. # For MeterView object creation, we use object of AtomicLongCounter.scala class which extends Counter interface and override its methods. (Attached code files for the same to understand better) # We register the metrics inside FilterReportsForSummaryAnalysis.scala. Some points to remember: # Not all job internal restarts cause incorrect metrics. # When there are internal job-restarts which caused incorrect metrics, then if we manually restart the job (Killing it and restarting using or not using savepoints), the metrics show correct value after this manual restart.(Given that on manual restarts, no other potential exception happened again which could cause an internal restarts) # We are using Flink Delay Restart Strategy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21773) Remove dependency from flink-metrics-statsd
Chesnay Schepler created FLINK-21773: Summary: Remove dependency from flink-metrics-statsd Key: FLINK-21773 URL: https://issues.apache.org/jira/browse/FLINK-21773 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21772) Remove dependency from flink-metrics-slf4j
Chesnay Schepler created FLINK-21772: Summary: Remove dependency from flink-metrics-slf4j Key: FLINK-21772 URL: https://issues.apache.org/jira/browse/FLINK-21772 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21761) Remove dependency from flink-metrics-influxdb
Chesnay Schepler created FLINK-21761: Summary: Remove dependency from flink-metrics-influxdb Key: FLINK-21761 URL: https://issues.apache.org/jira/browse/FLINK-21761 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21762) Remove dependency from flink-metrics-prometheus
Chesnay Schepler created FLINK-21762: Summary: Remove dependency from flink-metrics-prometheus Key: FLINK-21762 URL: https://issues.apache.org/jira/browse/FLINK-21762 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21760) Remove dependency from flink-metrics-jmx
Chesnay Schepler created FLINK-21760: Summary: Remove dependency from flink-metrics-jmx Key: FLINK-21760 URL: https://issues.apache.org/jira/browse/FLINK-21760 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18573) plugins/metrics-influx/flink-metrics-influxdb-1.11.0.jar META-INF has no dir named "services" , but "service"
zhangyunyun created FLINK-18573: --- Summary: plugins/metrics-influx/flink-metrics-influxdb-1.11.0.jar META-INF has no dir named "services" , but "service" Key: FLINK-18573 URL: https://issues.apache.org/jira/browse/FLINK-18573 Project: Flink Issue Type: Bug Components: Runtime / Metrics Affects Versions: 1.11.0 Reporter: zhangyunyun It causes the error: 2020-07-13 09:08:46.146 [main] WARN org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be found for reporter influxdb. Available factories: [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, org.apache.flink.metrics.graphite.GraphiteReporterFactory, org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, org.apache.flink.metrics.statsd.StatsDReporterFactory, org.apache.flink.metrics.prometheus.PrometheusReporterFactory, org.apache.flink.metrics.jmx.JMXReporterFactory]. 2020-07-13 09:08:46.149 [main] INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting metrics in intervals of 60 SECONDS for reporter slf4j of type org.apache.flink.metrics.slf4j.Slf4jReporter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17556) FATAL: Thread 'flink-metrics-akka.remote.default-remote-dispatcher-3' produced an uncaught exception. Stopping the process... java.lang.OutOfMemoryError: Direct buffer m
Tammy zhang created FLINK-17556: --- Summary: FATAL: Thread 'flink-metrics-akka.remote.default-remote-dispatcher-3' produced an uncaught exception. Stopping the process... java.lang.OutOfMemoryError: Direct buffer memory Key: FLINK-17556 URL: https://issues.apache.org/jira/browse/FLINK-17556 Project: Flink Issue Type: Bug Reporter: Tammy zhang My job consumes the data in kafka and then processes the data. After the job lasts for a while, the following error appears: ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler - FATAL: Thread 'flink-metrics-akka.remote.default-remote-dispatcher-3' produced an uncaught exception. Stopping the process... java.lang.OutOfMemoryError: Direct buffer memory i have set the "max.poll.records" propertity is "250", and it does not work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16635) Incompatible okio dependency in flink-metrics-influxdb module
Till Rohrmann created FLINK-16635:
-
Summary: Incompatible okio dependency in flink-metrics-influxdb
module
Key: FLINK-16635
URL: https://issues.apache.org/jira/browse/FLINK-16635
Project: Flink
Issue Type: Bug
Components: Runtime / Metrics
Affects Versions: 1.10.0
Reporter: Till Rohrmann
Fix For: 1.10.1, 1.11.0
With FLINK-12147 we bumped {{influxdb-java}} from version {{2.14}} to {{2.16}}.
At the same time we fix the okio dependency to version {{1.14.0}}. Since
{{influxdb-java}} transitive dependency {{converter-moshi:jar:2.6.1}} requires
{{moshi:jar:1.8.0}} which requires {{okio:jar:1.16.0}}, the influxdb metric
reporter fails as described
[here|https://stackoverflow.com/q/60667654/4815083]. We should fix this
incompatibility by removing the dependency management entry for okio.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15832) Artifact flink-metrics-core-tests will be uploaded twice
static-max created FLINK-15832:
--
Summary: Artifact flink-metrics-core-tests will be uploaded twice
Key: FLINK-15832
URL: https://issues.apache.org/jira/browse/FLINK-15832
Project: Flink
Issue Type: Bug
Components: Build System
Affects Versions: 1.9.1
Reporter: static-max
I built Flink 1.9.1 myself and merged the changes from
[https://github.com/apache/flink/pull/10936].
When I uploaded the artifacts to our repository (using {{mvn deploy }}
{{-DaltDeploymentRepository}}) the build fails as {{flink-metrics-core-tests}}
will be uploaded twice and we have redeployments disabled.
I'm not sure if other artifacts are affected as well, as I enabled redeployment
as a quick workaround.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-15356) Add applicationId to existing flink metrics running on yarn
Forward Xu created FLINK-15356: -- Summary: Add applicationId to existing flink metrics running on yarn Key: FLINK-15356 URL: https://issues.apache.org/jira/browse/FLINK-15356 Project: Flink Issue Type: Improvement Components: Runtime / Metrics Reporter: Forward Xu When sending metrics to Prometheus, these systems have only the Flink job ID, and the Flink job ID is UUID, which cannot be associated with the application job on the yarn. Therefore, we need to increase the applicationId when running on yarn. This helps us to accurately find the corresponding job on yarn when the metric is abnormal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14831) When edit flink-metrics-influxdb, need add metrics.md by hand
ouyangwulin created FLINK-14831:
---
Summary: When edit flink-metrics-influxdb, need add metrics.md by
hand
Key: FLINK-14831
URL: https://issues.apache.org/jira/browse/FLINK-14831
Project: Flink
Issue Type: Improvement
Reporter: ouyangwulin
When edit flink-metrics-influxdb, need add metrics.md by hand. AND
{code:java}
mvn package -Dgenerate-config-docs -pl flink-docs -am -nsu -DskipTests{code}
, Is not work
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-14332) add flink-metrics-signalfx to flink-metrics
Luke Sun created FLINK-14332: Summary: add flink-metrics-signalfx to flink-metrics Key: FLINK-14332 URL: https://issues.apache.org/jira/browse/FLINK-14332 Project: Flink Issue Type: New Feature Components: Runtime / Metrics Reporter: Luke Sun To support the [SignalFX|[https://www.signalfx.com/]] API and library dependencies: * com.signalfx.public:signalfx-java * com.signalfx.public:signalfx-codahale and add shaded package because io.dropwizard.metrics:metrics-core dependency is not equal to flink-metrics-dropwizard. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Flink metrics with parallel operator
Hello, I have been working on a flink project and need some help with the metric system. I have a logic inside a process function which side outputs a particular message on certain failure parameters. This process function has a parallelism > 1. How do I keep track of the failed messages through a counter metrics which is scoped to the operator across all parallel instances? Regards, Sibendu
[jira] [Created] (FLINK-12423) Add Timer metric type for flink-metrics module
Armstrong Nova created FLINK-12423:
--
Summary: Add Timer metric type for flink-metrics module
Key: FLINK-12423
URL: https://issues.apache.org/jira/browse/FLINK-12423
Project: Flink
Issue Type: Improvement
Components: Runtime / Metrics
Reporter: Armstrong Nova
Assignee: Armstrong Nova
Hi guys,
Currently, Flink only support 4 registering metrics, {{Counters}},
{{Gauges}}, {{Histograms}} and {{Meters}}. If we want to measure the time cost
metric, for example P75th, P99th, P999th， right now can only use Histograms
metric. But Histograms metric not support TimeUnit type (second, millisecond,
microsecond, nanosecond). So it's not convenient to collaborate with outside
metric system.
The Codahale/DropWizard support Time meter, so we can wrap it in Flink.
Thanks for your time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11174) flink Metrics Prometheus labels support chinese
Fan weiwen created FLINK-11174: -- Summary: flink Metrics Prometheus labels support chinese Key: FLINK-11174 URL: https://issues.apache.org/jira/browse/FLINK-11174 Project: Flink Issue Type: Improvement Components: Metrics Affects Versions: 1.7.0, 1.6.2 Reporter: Fan weiwen use flink metrics and Prometheus my job name is chinese but org.apache.flink.metrics.prometheus.AbstractPrometheusReporter replaceInvalidChars only support [a-zA-Z0-9:_] so my job name is replaceAll i think labels key is [a-zA-Z0-9:_] ok but labels value can support chinese? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11014) Relocate flink-metrics-graphite's dropwizard dependencies
Till Rohrmann created FLINK-11014: - Summary: Relocate flink-metrics-graphite's dropwizard dependencies Key: FLINK-11014 URL: https://issues.apache.org/jira/browse/FLINK-11014 Project: Flink Issue Type: Improvement Components: Build System, Metrics Affects Versions: 1.7.0 Reporter: Till Rohrmann Fix For: 1.8.0 Currently the `flink-metrics-graphite` module shades its dropwizard dependencies. In order to not interfere with user code, I think it would be good to also relocate them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10998) flink-metrics-ganglia has LGPL dependency
Chesnay Schepler created FLINK-10998:
Summary: flink-metrics-ganglia has LGPL dependency
Key: FLINK-10998
URL: https://issues.apache.org/jira/browse/FLINK-10998
Project: Flink
Issue Type: Bug
Components: Metrics
Affects Versions: 1.6.2, 1.5.5, 1.7.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Fix For: 1.5.6, 1.6.3, 1.7.0
{{flink-metrics-ganglia}} depends on {{info.ganglia.gmetric4j:gmetric4j}} which
depends on {{org.acplt:oncrpc}}.
{{org.acplt:oncrpc}} is licensed under the LGPL, which is a
[category-x|https://www.apache.org/legal/resolved.html#what-can-we-not-include-in-an-asf-project-category-x]
license.
For the time being we should drop this module from all future releases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10423) Forward RocksDB memory metrics to Flink metrics reporter
Seth Wiesman created FLINK-10423: Summary: Forward RocksDB memory metrics to Flink metrics reporter Key: FLINK-10423 URL: https://issues.apache.org/jira/browse/FLINK-10423 Project: Flink Issue Type: New Feature Components: Metrics, State Backends, Checkpointing Reporter: Seth Wiesman Assignee: Seth Wiesman RocksDB contains a number of metrics at the column family level about current memory usage, open memtables, etc that would be useful to users wishing greater insight what rocksdb is doing. This work is inspired heavily by the comments on this rocksdb issue thread (https://github.com/facebook/rocksdb/issues/3216#issuecomment-348779233) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10221) Flink metrics documentation io section table layout error
vinoyang created FLINK-10221: Summary: Flink metrics documentation io section table layout error Key: FLINK-10221 URL: https://issues.apache.org/jira/browse/FLINK-10221 Project: Flink Issue Type: Bug Affects Versions: 1.6.0 Reporter: vinoyang Assignee: vinoyang see here : https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html#io -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10035) ConcurrentModificationException with flink-metrics-slf4j
Nico Kruber created FLINK-10035:
---
Summary: ConcurrentModificationException with flink-metrics-slf4j
Key: FLINK-10035
URL: https://issues.apache.org/jira/browse/FLINK-10035
Project: Flink
Issue Type: Bug
Components: Metrics
Affects Versions: 1.5.2
Reporter: Nico Kruber
{code}
2018-08-02 15:45:08,052 WARN
org.apache.flink.runtime.metrics.MetricRegistryImpl - Error while
reporting metrics
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
at
org.apache.flink.metrics.slf4j.Slf4jReporter.report(Slf4jReporter.java:95)
at
org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:427)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
https://api.travis-ci.org/v3/job/411307171/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-8553) switch flink-metrics-datadog to async mode
Bowen Li created FLINK-8553: --- Summary: switch flink-metrics-datadog to async mode Key: FLINK-8553 URL: https://issues.apache.org/jira/browse/FLINK-8553 Project: Flink Issue Type: Improvement Components: Metrics Affects Versions: 1.4.0 Reporter: Bowen Li Assignee: Bowen Li Fix For: 1.5.0 Even though currently flink-metrics-datadog is designed as `fire-and-forget`, it's still using sync calls which may block or slow down core. Need to switch it to async mode. cc [~Zentol] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-7907) Flink Metrics documentation missing Scala examples
Colin Williams created FLINK-7907: - Summary: Flink Metrics documentation missing Scala examples Key: FLINK-7907 URL: https://issues.apache.org/jira/browse/FLINK-7907 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Colin Williams Priority: Minor The Flink metrics documentation is missing Scala examples for many of the metrics types. To be consistent there should be Scala examples for all the types. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Question about Flink Metrics
Hi Chesnay, That sounds great to me. I think I will be interested in it. Best Regards, Tony Wei 2017-09-26 21:57 GMT+08:00 Chesnay Schepler : > Hello, > > i see the value in supporting this, and it's also quite easy to do so > actually. > > I've filed FLINK-7692, containing instructions on how to implement this. > > @Tony Are you interested in implementing this? > > > > On 26.09.2017 14:10, Tony Wei wrote: > >> Hi Hai Zhou, >> >> It's a good idea to implement my own reporter, but I think it is not the >> best solution. >> After all, reporter needs to be set well when starting the cluster. It is >> not efficient to update cluster whenever you have a new metric for a new >> streaming job. >> >> Anyway, it is still a workaround for now. Thank you! >> >> Best Regards, >> Tony Wei >> >> >> 2017-09-26 19:13 GMT+08:00 Hai Zhou : >> >> Hi Tony, >>> >>> you can consider implementing a reporter, use a trick to convert the >>> flink's metrics to the structure that suits your needs. >>> >>> This is just my personal practice, hoping to help you. >>> >>> Cheers, >>> Hai Zhou >>> >>> >>> 在 2017年9月26日，17:49，Tony Wei 写道： >>> >>> Hi, >>> >>> Recently, I am using PrometheusReporter to monitor every metrics from >>> Flink. >>> >>> I found that the metric name in Prometheus will map to the identifier >>> from >>> User Scope and System Scope [1], and the labels will map to Variables >>> [2]. >>> >>> To monitor the same metrics from Prometheus, I would like to use labels >>> to differentiate them. >>> Under the job/task/operator scope, it words fine to me. However, its not >>> convenient to me to monitor partitions' states from Kakfa consumer, >>> because >>> I couldn't place partition id like a tag on each metric. All partition >>> states like current commit offset will be a unique metric in Prometheus. >>> It's hard to use visualization tool such as Grafana to monitor them. >>> >>> My question is: Is it possible to add tags on Metric, instead of using >>> `.addGroup()`? >>> If not, will it be a new feature on Flink Metrics in the future? Since I >>> am not sure about how other reporters work, I am afraid that it is not a >>> good design to just fulfill the requirement on particular reporter. >>> >>> Please guide and thanks for your help. >>> >>> Best Regards, >>> Tony Wei >>> >>> [1]: https://ci.apache.org/projects/flink/flink-docs- >>> release-1.3/monitoring/metrics.html#scope >>> [2]: https://ci.apache.org/projects/flink/flink-docs- >>> release-1.3/monitoring/metrics.html#list-of-all-variables >>> >>> >>> >>> >
Re: Question about Flink Metrics
Hello, i see the value in supporting this, and it's also quite easy to do so actually. I've filed FLINK-7692, containing instructions on how to implement this. @Tony Are you interested in implementing this? On 26.09.2017 14:10, Tony Wei wrote: Hi Hai Zhou, It's a good idea to implement my own reporter, but I think it is not the best solution. After all, reporter needs to be set well when starting the cluster. It is not efficient to update cluster whenever you have a new metric for a new streaming job. Anyway, it is still a workaround for now. Thank you! Best Regards, Tony Wei 2017-09-26 19:13 GMT+08:00 Hai Zhou : Hi Tony, you can consider implementing a reporter, use a trick to convert the flink's metrics to the structure that suits your needs. This is just my personal practice, hoping to help you. Cheers, Hai Zhou 在 2017年9月26日，17:49，Tony Wei 写道： Hi, Recently, I am using PrometheusReporter to monitor every metrics from Flink. I found that the metric name in Prometheus will map to the identifier from User Scope and System Scope [1], and the labels will map to Variables [2]. To monitor the same metrics from Prometheus, I would like to use labels to differentiate them. Under the job/task/operator scope, it words fine to me. However, its not convenient to me to monitor partitions' states from Kakfa consumer, because I couldn't place partition id like a tag on each metric. All partition states like current commit offset will be a unique metric in Prometheus. It's hard to use visualization tool such as Grafana to monitor them. My question is: Is it possible to add tags on Metric, instead of using `.addGroup()`? If not, will it be a new feature on Flink Metrics in the future? Since I am not sure about how other reporters work, I am afraid that it is not a good design to just fulfill the requirement on particular reporter. Please guide and thanks for your help. Best Regards, Tony Wei [1]: https://ci.apache.org/projects/flink/flink-docs- release-1.3/monitoring/metrics.html#scope [2]: https://ci.apache.org/projects/flink/flink-docs- release-1.3/monitoring/metrics.html#list-of-all-variables
Re: Question about Flink Metrics
Hi Hai Zhou, It's a good idea to implement my own reporter, but I think it is not the best solution. After all, reporter needs to be set well when starting the cluster. It is not efficient to update cluster whenever you have a new metric for a new streaming job. Anyway, it is still a workaround for now. Thank you! Best Regards, Tony Wei 2017-09-26 19:13 GMT+08:00 Hai Zhou : > Hi Tony, > > you can consider implementing a reporter, use a trick to convert the > flink's metrics to the structure that suits your needs. > > This is just my personal practice, hoping to help you. > > Cheers, > Hai Zhou > > > 在 2017年9月26日，17:49，Tony Wei 写道： > > Hi, > > Recently, I am using PrometheusReporter to monitor every metrics from > Flink. > > I found that the metric name in Prometheus will map to the identifier from > User Scope and System Scope [1], and the labels will map to Variables [2]. > > To monitor the same metrics from Prometheus, I would like to use labels > to differentiate them. > Under the job/task/operator scope, it words fine to me. However, its not > convenient to me to monitor partitions' states from Kakfa consumer, because > I couldn't place partition id like a tag on each metric. All partition > states like current commit offset will be a unique metric in Prometheus. > It's hard to use visualization tool such as Grafana to monitor them. > > My question is: Is it possible to add tags on Metric, instead of using > `.addGroup()`? > If not, will it be a new feature on Flink Metrics in the future? Since I > am not sure about how other reporters work, I am afraid that it is not a > good design to just fulfill the requirement on particular reporter. > > Please guide and thanks for your help. > > Best Regards, > Tony Wei > > [1]: https://ci.apache.org/projects/flink/flink-docs- > release-1.3/monitoring/metrics.html#scope > [2]: https://ci.apache.org/projects/flink/flink-docs- > release-1.3/monitoring/metrics.html#list-of-all-variables > > >
Re: Question about Flink Metrics
Hi Tony, you can consider implementing a reporter, use a trick to convert the flink's metrics to the structure that suits your needs. This is just my personal practice, hoping to help you. Cheers, Hai Zhou > 在 2017年9月26日，17:49，Tony Wei 写道： > > Hi, > > Recently, I am using PrometheusReporter to monitor every metrics from Flink. > > I found that the metric name in Prometheus will map to the identifier from > User Scope and System Scope [1], and the labels will map to Variables [2]. > > To monitor the same metrics from Prometheus, I would like to use labels to > differentiate them. > Under the job/task/operator scope, it words fine to me. However, its not > convenient to me to monitor partitions' states from Kakfa consumer, because I > couldn't place partition id like a tag on each metric. All partition states > like current commit offset will be a unique metric in Prometheus. It's hard > to use visualization tool such as Grafana to monitor them. > > My question is: Is it possible to add tags on Metric, instead of using > `.addGroup()`? > If not, will it be a new feature on Flink Metrics in the future? Since I am > not sure about how other reporters work, I am afraid that it is not a good > design to just fulfill the requirement on particular reporter. > > Please guide and thanks for your help. > > Best Regards, > Tony Wei > > [1]: > https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#scope > > <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#scope> > [2]: > https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#list-of-all-variables > > <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#list-of-all-variables>
Question about Flink Metrics
Hi, Recently, I am using PrometheusReporter to monitor every metrics from Flink. I found that the metric name in Prometheus will map to the identifier from User Scope and System Scope [1], and the labels will map to Variables [2]. To monitor the same metrics from Prometheus, I would like to use labels to differentiate them. Under the job/task/operator scope, it words fine to me. However, its not convenient to me to monitor partitions' states from Kakfa consumer, because I couldn't place partition id like a tag on each metric. All partition states like current commit offset will be a unique metric in Prometheus. It's hard to use visualization tool such as Grafana to monitor them. My question is: Is it possible to add tags on Metric, instead of using `.addGroup()`? If not, will it be a new feature on Flink Metrics in the future? Since I am not sure about how other reporters work, I am afraid that it is not a good design to just fulfill the requirement on particular reporter. Please guide and thanks for your help. Best Regards, Tony Wei [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#scope [2]: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#list-of-all-variables
[jira] [Created] (FLINK-6431) Activate strict checkstyle for flink-metrics
Chesnay Schepler created FLINK-6431: --- Summary: Activate strict checkstyle for flink-metrics Key: FLINK-6431 URL: https://issues.apache.org/jira/browse/FLINK-6431 Project: Flink Issue Type: Improvement Components: Metrics Reporter: Chesnay Schepler Assignee: Chesnay Schepler -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Flink Metrics
https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html Or this: https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/metrics.html if you prefer Flink 1.1 On Mon, 17 Oct 2016 at 19:16 amir bahmanyari wrote: > Hi colleagues, > Is there a link that described Flink Matrices & provides example on how to > utilize it pls? > I really appreciate it... > Cheers > > -- > *From:* Till Rohrmann > *To:* u...@flink.apache.org > *Cc:* dev@flink.apache.org > *Sent:* Monday, October 17, 2016 12:52 AM > *Subject:* Re: Flink Metrics > > Hi Govind, > > I think the DropwizardMeterWrapper implementation is just a reference > implementation where it was decided to report the minute rate. You can > define your own meter class which allows to configure the rate interval > accordingly. > > Concerning Timers, I think nobody requested this metric so far. If you > want, then you can open a JIRA issue and contribute it. The community would > really appreciate that. > > Cheers, > Till > > > On Mon, Oct 17, 2016 at 5:26 AM, Govindarajan Srinivasaraghavan < > govindragh...@gmail.com> wrote: > > > Hi, > > > > I am currently using flink 1.2 snapshot and instrumenting my pipeline > with > > flink metrics. One small suggestion I have is currently the Meter > interface > > only supports getRate() which is always the one minute rate. > > > > It would great if all the rates (1 min, 5 min & 15 min) are exposed to > get > > a better picture in terms of performance. > > > > Also is there any reason why timers are not part of flink metrics core? > > > > Regards, > > Govind > > > > >
Re: Flink Metrics
Hi colleagues,Is there a link that described Flink Matrices & provides example on how to utilize it pls?I really appreciate it...Cheers From: Till Rohrmann To: u...@flink.apache.org Cc: dev@flink.apache.org Sent: Monday, October 17, 2016 12:52 AM Subject: Re: Flink Metrics Hi Govind, I think the DropwizardMeterWrapper implementation is just a reference implementation where it was decided to report the minute rate. You can define your own meter class which allows to configure the rate interval accordingly. Concerning Timers, I think nobody requested this metric so far. If you want, then you can open a JIRA issue and contribute it. The community would really appreciate that. Cheers, Till On Mon, Oct 17, 2016 at 5:26 AM, Govindarajan Srinivasaraghavan < govindragh...@gmail.com> wrote: > Hi, > > I am currently using flink 1.2 snapshot and instrumenting my pipeline with > flink metrics. One small suggestion I have is currently the Meter interface > only supports getRate() which is always the one minute rate. > > It would great if all the rates (1 min, 5 min & 15 min) are exposed to get > a better picture in terms of performance. > > Also is there any reason why timers are not part of flink metrics core? > > Regards, > Govind >
Re: Flink Metrics
Hi Govind, I think the DropwizardMeterWrapper implementation is just a reference implementation where it was decided to report the minute rate. You can define your own meter class which allows to configure the rate interval accordingly. Concerning Timers, I think nobody requested this metric so far. If you want, then you can open a JIRA issue and contribute it. The community would really appreciate that. Cheers, Till On Mon, Oct 17, 2016 at 5:26 AM, Govindarajan Srinivasaraghavan < govindragh...@gmail.com> wrote: > Hi, > > I am currently using flink 1.2 snapshot and instrumenting my pipeline with > flink metrics. One small suggestion I have is currently the Meter interface > only supports getRate() which is always the one minute rate. > > It would great if all the rates (1 min, 5 min & 15 min) are exposed to get > a better picture in terms of performance. > > Also is there any reason why timers are not part of flink metrics core? > > Regards, > Govind >
Flink Metrics
Hi, I am currently using flink 1.2 snapshot and instrumenting my pipeline with flink metrics. One small suggestion I have is currently the Meter interface only supports getRate() which is always the one minute rate. It would great if all the rates (1 min, 5 min & 15 min) are exposed to get a better picture in terms of performance. Also is there any reason why timers are not part of flink metrics core? Regards, Govind
[jira] [Created] (FLINK-4186) Expose Kafka metrics through Flink metrics
Robert Metzger created FLINK-4186: - Summary: Expose Kafka metrics through Flink metrics Key: FLINK-4186 URL: https://issues.apache.org/jira/browse/FLINK-4186 Project: Flink Issue Type: Improvement Components: Kafka Connector Affects Versions: 1.1.0 Reporter: Robert Metzger Assignee: Robert Metzger Currently, we expose the Kafka metrics through Flink's accumulators. We can now use the metrics system in Flink to report Kafka metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4173) Replace maven-assembly-plugin by maven-shade-plugin in flink-metrics
Till Rohrmann created FLINK-4173:
Summary: Replace maven-assembly-plugin by maven-shade-plugin in
flink-metrics
Key: FLINK-4173
URL: https://issues.apache.org/jira/browse/FLINK-4173
Project: Flink
Issue Type: Bug
Components: Metrics
Affects Versions: 1.1.0
Reporter: Till Rohrmann
Assignee: Chesnay Schepler
Priority: Minor
Fix For: 1.1.0
The modules {{flink-metrics-dropwizard}}, {{flink-metrics-ganglia}} and
{{flink-metrics-graphite}} use the {{maven-assembly-plugin}} to build a fat
jar. The resulting fat jar has the suffix {{jar-with-dependencies}}. In order
to make the naming consistent with the rest of the system we should create a
fat-jar without this suffix.
Additionally we could replace the {{maven-assembly-plugin}} with the
{{maven-shade-plugin}} to make it consistent with the rest of the system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
