+1 for approach 1. As Thomas mentioned, I think there metrics layer should be abstracted out from its implementation. This way one can plugin different metrics systems in apex.
Also keep in mind that there is a lot of code which uses AutoMetrics annotations. We should help a smooth transition from that. As next release will be a major version release, this is a good opportunity for getting rid of old AutoMetrics and counters functionality. Regards, Chinmay. On Thu, 10 May 2018, 4:47 am Vlad Rozov, <vro...@apache.org> wrote: > +1 for the #1 proposal. > > Thank you, > > Vlad > > On 5/9/18 07:14, Thomas Weise wrote: > > +1 for the initiative > > > > Some thoughts: > > > > - it is probably good to retain a level of abstraction, to avoid direct > > dependency on dropwizard > > - support programmatic metric creation (not just annotations) > > - remove deprecated counter and auto-metric code and migrate operators to > > use new API > > - which metric reporting systems will be supported out of the box > > > > You can also take a look at how this was structured in Flink: > > > https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html > > > > Thanks, > > Thomas > > > > > > On Tue, May 8, 2018 at 8:56 AM, Deepak Narkhede <mailtodeep...@gmail.com > > > > wrote: > > > >> Hi Community, > >> > >> I want to propose addition of metrics like gauges, meters, counters and > >> historgram for the following components. > >> 1) Addition of metrics for Container Stats. > >> 2) Addition of metrics for Operator Stats. > >> 3) Addition of metrics for Stram Application Master stats. > >> 4) Addition of metrics for JVM related stats for all containers. > >> > >> To implement them would be using metrics dropwizard api's. ( > >> http://metrics.dropwizard.io/) > >> Use cases: > >> 1) Can be directly pushed to external visualisation system like > Graphite. > >> 2) Can be viewed in visualVM tools through JMX. > >> 3) Can be outputted to console. > >> 4) It is also possible to push the metrics to custom sink. > >> > >> We will also need to write sinks and reporter, if required for custom > >> sinks. > >> > >> Design/Implementation approach: > >> Way #1: > >> 1) Create new annotations like @MetricTypeGauge, @MetricTypeMeter, > >> @MetricTypeCounter, @MetricTypeHistogram. They can be both fields and > >> methods. > >> 2) Add them to respective methods or fields like StreamingContainer, > >> StreamingAppMasterService for extraction of relevant metrics. > >> 3) While Node creation ( InputNode/GeneticNode/OiONode), we create and > >> initialise the metrics registry depending on components. > >> 4) While collectMetrics() part of operator runner thread ( InputNode.run > >> /GenericNode.run), we actually invoke the annotations methods and > collect > >> different types of metrics. > >> 5) We can have a sink which pushes the metrics to reporter like Console, > >> JMX etc. > >> > >> Way #2: > >> Use existing AutoMetrics annotations, convert some metrics to different > >> types like gauge, counter etc..But this cannot be done generically as we > >> don't know the types. Still more investigation is going on this > approach. > >> > >> I would prefer first way. > >> > >> Note: There are some complications, if two operators are deployed on > same > >> jvm conatiner. But I think it can be resolved by creating two different > >> metrics registry with unique id from JVM. > >> > >> Let me know your thoughts on this. > >> > >> Thanks, > >> Deepak > >> > >
