Hello, The new metrics API allows us to integrate some basic metrics into the Beam IOs. I have been following some discussions about this on JIRAs/PRs, and I think it is important to discuss the subject here so we can have more awareness and obtain ideas from the community.
First I want to thank Ben for his work on the metrics API, and Aviem for his ongoing work on metrics for IOs, e.g. KafkaIO) that made me aware of this subject. There are some basic ideas to discuss e.g. - What are the responsibilities of Beam IOs in terms of Metrics (considering the fact that the actual IOs, server + client, usually provide their own)? - What metrics are relevant to the pipeline (or some particular IOs)? Kafka backlog for one could point that a pipeline is behind ingestion rate. - Should metrics be calculated on IOs by default or no? - If metrics are defined by default does it make sense to allow users to disable them? Well these are just some questions around the subject so we can create a common set of practices to include metrics in the IOs and eventually improve the transform guide with this. What do you think about this? Do you have other questions/ideas? Thanks, Ismaël
