Hi, My issue has been resolved through discussion with AWS support.
It turns out that Kinesis Data Analytics reports to CloudWatch in a way I did not expect. The way to view the accurate values for Flink counters is with Average in CloudWatch metrics. Below is the response from AWS support, for anyone who might benefit from this in the future: After discussing this issue with our Kinesis Data Analytics team, I was informed that currently metrics are reported every 15 seconds, hence every minute there would be 4 records, which might be the explanation to why you are seeing this number of records. However, please understand that this frequency may change in the future so please do not rely on this frequency as the source of truth. We would recommend to use base alarms on the AVG, P99 or other similar metric statistics rather than looking and examining individual values or verifying the numbers of values are as expected. In addition, depending on how these metrics are calculated, displaying them with different statistic would show different graphs and scenarios. Please kindly refer to the "CloudWatch statistics definitions" [1] documentation for more details on how metric data are aggregated over specified periods of time with different statistic choices for metrics. Below are some examples of the most commonly used statistic options: -"SampleCount" is the number of data points during the period. -"Sum" is the sum of the values of the all data points collected during the period. -"Average" is the value of Sum/SampleCount during the specified period. -"Percentile (p)" indicates the relative standing of a value in a dataset. For example, p99 is the 99th percentile and means that 99 percent of the data within the period is lower than this value and 1 percent of the data is higher than this value. -”Minimum“ is the lowest value observed during the specified period. -“Maximum" is the highest value observed during the specified period. In answer to your question regarding how custom metrics are pushed, in the documentation you have provided [2], it says that "Custom metrics in Kinesis Data Analytics use the Apache Flink metric system", Kinesis Data Analytics only report these metrics to CloudWatch as you define these custom metrics. [1] https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Statistics-definitions.html [2] https://docs.aws.amazon.com/kinesisanalytics/latest/java/monitoring-metrics-custom.html Regards, Shane
