Hi Ismael Good point to discuss the metric here.
Imho, the IOs should use the metric API to provide specific IO metrics (mile number of split, throughput, reading/writing rate, number of splits, etc). In Camel, each processor (aka IO) provides such metric indicators. The purpose is to provide the metric specific to the IO process (not to the back end). On the other hand, equivalent metrics can be found at the pipeline level and at IO level (it's a question of scope). I would propose to extend the documentation and maybe the API about the IO. Regards JB On Feb 14, 2017, 09:37, at 09:37, "Ismaël Mejía" <[email protected]> wrote: >Hello, > >The new metrics API allows us to integrate some basic metrics into the >Beam >IOs. I have been following some discussions about this on JIRAs/PRs, >and I >think it is important to discuss the subject here so we can have more >awareness and obtain ideas from the community. > >First I want to thank Ben for his work on the metrics API, and Aviem >for >his ongoing work on metrics for IOs, e.g. KafkaIO) that made me aware >of >this subject. > >There are some basic ideas to discuss e.g. > >- What are the responsibilities of Beam IOs in terms of Metrics >(considering the fact that the actual IOs, server + client, usually >provide >their own)? > >- What metrics are relevant to the pipeline (or some particular IOs)? >Kafka >backlog for one could point that a pipeline is behind ingestion rate. > >- Should metrics be calculated on IOs by default or no? > >- If metrics are defined by default does it make sense to allow users >to >disable them? > >Well these are just some questions around the subject so we can create >a >common set of practices to include metrics in the IOs and eventually >improve the transform guide with this. What do you think about this? Do >you >have other questions/ideas? > >Thanks, >Ismaël
