Is your job a Dataflow Template job? The error is caused by https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTemplateJob.java#L55 .
So basically DataflowTemplateJob does not support metrics. On Wed, Jun 19, 2024 at 3:57 AM Lahiru Ginnaliya Gamathige < glah...@gmail.com> wrote: > Hi Users, > > In Google Cloud monitoring there is a limit of 100 metrics and when we are > using KafkaIO, the library publishes a bunch of metrics per topic. With our > use we will easily run out of 100 metric limit. > > We want to stop KafkaIO from publishing metrics and I do not see this is > configurable. So I am trying to write a metric filtering logic (we are > using beam version 2.55.1). > I wrote a Sink but when I try to find a way to register the sink I cannot > see a way to do the following in this beam version, > > *MetricsEnvironment.setMetricsSink(new > CustomMetricsSink(options.getProject()));* > > Then I tried to register it like this, > > PipelineResult results = run(options); > results.waitUntilFinish(); > > > > * MetricQueryResults metricQueryResults = > results.metrics().queryMetrics(MetricsFilter.builder().build()); > CustomMetricSink reporter = new CustomMetricSink(options.getProject()); > reporter.writeMetrics(metricQueryResults);* > > With the above code pipeline is failing to start with the > error(java.lang.UnsupportedOperationException: > The result of template creation should not be used.) > > > Do you suggest another solution for this problem (it sounds like a quite > common problem when using kafkaio). Or do you have any suggestion about my > attempts ? > > Regards > Lahiru > >
