Hi Phil,
Your setup looks good to me and you are not using detached mode. MetricsPusher
in streaming mode on flink works so we
need to figure out what is wrong.
- please check that you have port 3000 listening (just in case :) ) using
telnet or any network program- also you need
to have beam-runners-extensions-java-metrics jar in the classpath- please check
the logs for a MetricsPushException
exception. It would not stop the pipeline so you might have missed it but in
case of error, the exception will be
wrapped into an MetricsPushException and be printed to the console.
CCing dev list
BestEtienne
Le lundi 10 décembre 2018 à 08:45 -0600, Phil Franklin a écrit :
> Hi, Etienne! The actual command line is “mvn clean install”, because I set
> the arguments in the code:
>
> public void testFlinkRunner() throws Exception {
> String[] args = new String[]{"--runner=FlinkRunner", "--streaming=true",
> "--inputTopic="+topic};
>
> run(args);
> }
>
> I’m using Beam 2.8.0, and Flink 1.5.4.
>
> Because I wasn’t sure whether I was in detached mode, I called the toString()
> method on the Flink runner result to see
> if I would get FlinkDetachedRunnerResult. This is what was returned:
>
>
>
FlinkRunnerResult{accumulators={__metricscontainers=org.apache.beam.runners.core.metrics.MetricsContainerStepMap@b799384b
> }, runtime=33126}
> I set the options for MetricsHttpSink:
>
> options.setMetricsHttpSinkUrl("http://localhost:3000";);
> options.setMetricsSink(MetricsHttpSink.class);
> This works when I test SparkRunner, so I believe I have it set up correctly
> for MetricsPusher to capture the metrics
> from Flink as well.
> -Phil
> On 2018/12/07 14:46:06, Etienne Chauchot <e...@apache.org> wrote:
> > Hi Phil,>
> > MetricsPusher is tested on all the runners in both batch and streaming
> > mode. I just ran this test in Flink in
> streaming>
> > mode and it works.>
> > What is the command line you are using and which version of Beam?>
> >
> > Please also remember that, as discussed, metrics (other flink features )
> > do not work if flink is used in detached
> mode.>
> >
> > Etienne>
> >
> >
> > Le mardi 04 décembre 2018 à 12:49 -0600, Phil Franklin a écrit :>
> > > I’m having difficulty accessing Beam metrics when using FlinkRunner in
> > > streaming mode. I don’t get any metrics
> from MetricsPusher, though the same setup delivered metrics from SparkRunner.
> Probably for the same reason that
> MetricsPusher doesn’t work, I also don’t get any output when I call an
> instance of MetricsHttpSink directly. The
> problem seems to be that Flink never returns from pipeline.run(), an issue
> that others have referred to as FlinkRunner
> hanging. >
> > > >
> > > Is there a solution for getting metrics in this case that I’m missing?>
> > > >
> > > Thanks!>
> > > -Phil>
> >
>
