Phil,Also, I forgot, as I see no "master=local" or flink equivalent in your
setup make sure you run flink in local mode
(in memory runner local to your machine) because you are trying to push metrics
to http://localhost:3000. Make sure your
app does not get deployed to a remote flink cluster that would not have network
access to your local machine.
Etienne
Le mardi 11 décembre 2018 à 15:07 +0100, Etienne Chauchot a écrit :
> Hi Phil,
> Your setup looks good to me and you are not using detached mode.
> MetricsPusher in streaming mode on flink works so we
> need to figure out what is wrong.
> - please check that you have port 3000 listening (just in case :) ) using
> telnet or any network program- also you need
> to have beam-runners-extensions-java-metrics jar in the classpath- please
> check the logs for a MetricsPushException
> exception. It would not stop the pipeline so you might have missed it but in
> case of error, the exception will be
> wrapped into an MetricsPushException and be printed to the console.
> CCing dev list
> BestEtienne
> Le lundi 10 décembre 2018 à 08:45 -0600, Phil Franklin a écrit :
> > Hi, Etienne! The actual command line is “mvn clean install”, because I set
> > the arguments in the code:
> >
> > public void testFlinkRunner() throws Exception {
> > String[] args = new String[]{"--runner=FlinkRunner",
> > "--streaming=true", "--inputTopic="+topic};
> >
> > run(args);
> > }
> >
> > I’m using Beam 2.8.0, and Flink 1.5.4.
> >
> > Because I wasn’t sure whether I was in detached mode, I called the
> > toString() method on the Flink runner result to
> > see if I would get FlinkDetachedRunnerResult. This is what was returned:
> >
> >
> >
FlinkRunnerResult{accumulators={__metricscontainers=org.apache.beam.runners.core.metrics.MetricsContainerStepMap@b799384b
> > }, runtime=33126}
> > I set the options for MetricsHttpSink:
> >
> > options.setMetricsHttpSinkUrl("http://localhost:3000";);
> > options.setMetricsSink(MetricsHttpSink.class);
> > This works when I test SparkRunner, so I believe I have it set up correctly
> > for MetricsPusher to capture the metrics
> > from Flink as well.
> > -Phil
> > On 2018/12/07 14:46:06, Etienne Chauchot <e...@apache.org> wrote:
> > > Hi Phil,>
> > > MetricsPusher is tested on all the runners in both batch and streaming
> > > mode. I just ran this test in Flink in
> > streaming>
> > > mode and it works.>
> > > What is the command line you are using and which version of Beam?>
> > >
> > > Please also remember that, as discussed, metrics (other flink features )
> > > do not work if flink is used in detached
> > mode.>
> > >
> > > Etienne>
> > >
> > >
> > > Le mardi 04 décembre 2018 à 12:49 -0600, Phil Franklin a écrit :>
> > > > I’m having difficulty accessing Beam metrics when using FlinkRunner in
> > > > streaming mode. I don’t get any metrics
> > from MetricsPusher, though the same setup delivered metrics from
> > SparkRunner. Probably for the same reason that
> > MetricsPusher doesn’t work, I also don’t get any output when I call an
> > instance of MetricsHttpSink directly. The
> > problem seems to be that Flink never returns from pipeline.run(), an issue
> > that others have referred to as
> > FlinkRunner hanging. >
> > > > >
> > > > Is there a solution for getting metrics in this case that I’m missing?>
> > > > >
> > > > Thanks!>
> > > > -Phil>
> > >
> >
