+user Begin forwarded message:
> From: Rion Williams <[email protected]> > Date: January 12, 2021 at 4:09:34 PM CST > To: [email protected] > Subject: Accessing Custom Beam Metrics in Dataproc > Reply-To: [email protected] > > Hi all, > > I'm currently in the process of adding some metrics to an existing pipeline > that runs on Google Dataproc via Spark and I'm trying to determine how to > access these metrics and eventually expose them to Stackdriver (to be used > downstream in Grafana dashboards). > > The metrics themselves are fairly simple (a series of counters) and are > defined as such (and accessed in DoFns throughout the pipeline): > > ``` > /** Metrics gathered during Event-related transforms */ > private object Metrics { > // This is used to keep track of any dynamically added data sources > and their counts > val totalMessages: Counter = > BeamMetrics.counter(Events::class.qualifiedName, "messages_total") > } > ``` > > After initially running the pipeline in Dataproc, I wasn't able to see > anything that specifically indicated that the metrics were being exposed at > all. I haven't added any specific configuration to handle this within the > pipeline itself, however I did notice an interface that I may need to > consider implementing called MetricOptions: > > ``` > interface MyPipelineOptions : ... MetricsOptions { ... } > ``` > > So my questions primarily center around: > > - Will Metrics be emitted automatically? Or do I need to explicitly implement > the MetricsOptions interface for the pipeline? > - Does anyone have any experience with handling this (i.e. Pipeline > Metrics > > Stackdriver)? I'd imagine since this is all self-contained within GCP > (Dataproc + Stackdriver), that it wouldn't be too rough to hand that baton > off. > > Any advice / articles / examples would be greatly appreciated! > > Thanks, > > Rion
