Also seems like each IO would benefit from an entry in a Transform Catalog with description of any IO-specific metrics it emits. (technically these may not be what you mean by "framework-emitted metrics")
Kenn On Mon, Mar 2, 2020 at 9:40 AM Alex Amato <ajam...@google.com> wrote: > MonitoringInfoSpecs is effectively a list of metrics > <https://github.com/apache/beam/blob/c0b60195b17b7cdc46d6ad6548cd41a967e71cde/model/pipeline/src/main/proto/metrics.proto#L60>, > but its purpose is to simply define how SDKs should populate MonitoringInfo > protos for a RunnerHarness to interpret. > > These metrics are provided by the Java and Python SDKs, and Go will soon > provide all of them as well. > > But there is no requirement for a particular runner to support any of > these metrics. The DataflowRunner will support these and they metrics are > accessible via Dataflow APIs. I am not sure the state of other runners. > > > > > On Mon, Mar 2, 2020 at 2:47 AM Etienne Chauchot <echauc...@apache.org> > wrote: > >> Hi, >> >> There is a doc about metrics here: >> https://beam.apache.org/documentation/programming-guide/#metrics >> >> You can also export the metrics to sinks (REST http endpoint and >> Graphite), see MetricsOptions class for configuration. >> >> Still, there is no doc for export on website, I'll add some >> >> Best >> >> Etienne >> On 28/02/2020 01:07, Pablo Estrada wrote: >> >> Hi Daniel! >> I think +Alex Amato <ajam...@google.com> had tried to have an inventory >> of metrics at some point. >> Other than that, I don't think we have a document outlining them. >> >> Can you talk about what you plan to do with them? Do you plan to export >> them somehow? Do you plan to add your own? >> Best >> -P. >> >> On Thu, Feb 27, 2020 at 11:33 AM Daniel Chen <danx...@gmail.com> wrote: >> >>> Hi all, >>> >>> I some questions about the reference to the framework metrics emitted by >>> Beam. I would like to leverage these metrics to allow better monitoring of >>> by Beam jobs but cannot find any references to the description or a >>> complete set of emitted metrics. >>> >>> Do we have this information documented anywhere? >>> >>> Thanks, >>> Daniel >>> >>
