Hi Community, I want to propose addition of metrics like gauges, meters, counters and historgram for the following components. 1) Addition of metrics for Container Stats. 2) Addition of metrics for Operator Stats. 3) Addition of metrics for Stram Application Master stats. 4) Addition of metrics for JVM related stats for all containers.
To implement them would be using metrics dropwizard api's. ( http://metrics.dropwizard.io/) Use cases: 1) Can be directly pushed to external visualisation system like Graphite. 2) Can be viewed in visualVM tools through JMX. 3) Can be outputted to console. 4) It is also possible to push the metrics to custom sink. We will also need to write sinks and reporter, if required for custom sinks. Design/Implementation approach: Way #1: 1) Create new annotations like @MetricTypeGauge, @MetricTypeMeter, @MetricTypeCounter, @MetricTypeHistogram. They can be both fields and methods. 2) Add them to respective methods or fields like StreamingContainer, StreamingAppMasterService for extraction of relevant metrics. 3) While Node creation ( InputNode/GeneticNode/OiONode), we create and initialise the metrics registry depending on components. 4) While collectMetrics() part of operator runner thread ( InputNode.run /GenericNode.run), we actually invoke the annotations methods and collect different types of metrics. 5) We can have a sink which pushes the metrics to reporter like Console, JMX etc. Way #2: Use existing AutoMetrics annotations, convert some metrics to different types like gauge, counter etc..But this cannot be done generically as we don't know the types. Still more investigation is going on this approach. I would prefer first way. Note: There are some complications, if two operators are deployed on same jvm conatiner. But I think it can be resolved by creating two different metrics registry with unique id from JVM. Let me know your thoughts on this. Thanks, Deepak
