Re: Help: how to get latency value comfortable in Flink1.5?
So there's 2 issues here: 1) your reporter configuration is wrong, configuration values for a specific reporter are prefixed with "metrics.reporter", not "metrics.reporters" (note the "s"). See below for a correct config. metrics.reporters: varuy metrics.reporter.varuy.host: tracing115 metrics.reporter.varuy.port: 2003 metrics.reporter.varuy.class: org.apache.flink.metrics.graphite.GraphiteReporter metrics.reporter.varuy.interval: 1 SECONDS metrics.reporter.varuy.protocol: TCP 2) The reporter only exports data to an external graphite server, so yes you have to start that one separately. On 27.09.2018 08:13, varuy322 wrote: Hi Till, I have copy the flink-metrics-graphite-1.5.2.jar to lib/, and restart flink.It seems make no difference. When I checked the jobmanager log, the metrics configured in flink-conf.yaml, could be load since it printed as below: "2018-09-27 09:50:23,953 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters, varuy 2018-09-27 09:50:23,953 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters.varuy.host, tracing115 2018-09-27 09:50:23,953 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters.varuy.port, 2003 2018-09-27 09:50:23,953 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters.varuy.class, org.apache.flink.metrics.graphite.GraphiteReporter 2018-09-27 09:50:23,954 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters.varuy.interval, 1 SECONDS 2018-09-27 09:50:23,954 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters.varuy.protocol, TCP" However, I found the GraphiteReporter do not run correctly for the info list: "2018-09-27 09:50:24,938 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported." Should I install Graphite Server independently? Best Regards & Thanks Rui,Wang - stay hungry, stay foolish. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Help: how to get latency value comfortable in Flink1.5?
Hi Till, I have copy the flink-metrics-graphite-1.5.2.jar to lib/, and restart flink.It seems make no difference. When I checked the jobmanager log, the metrics configured in flink-conf.yaml, could be load since it printed as below: "2018-09-27 09:50:23,953 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters, varuy 2018-09-27 09:50:23,953 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters.varuy.host, tracing115 2018-09-27 09:50:23,953 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters.varuy.port, 2003 2018-09-27 09:50:23,953 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters.varuy.class, org.apache.flink.metrics.graphite.GraphiteReporter 2018-09-27 09:50:23,954 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters.varuy.interval, 1 SECONDS 2018-09-27 09:50:23,954 INFO org.apache.flink.configuration.GlobalConfiguration- Loading configuration property: metrics.reporters.varuy.protocol, TCP" However, I found the GraphiteReporter do not run correctly for the info list: "2018-09-27 09:50:24,938 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported." Should I install Graphite Server independently? Best Regards & Thanks Rui,Wang - stay hungry, stay foolish. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Help: how to get latency value comfortable in Flink1.5?
Hi Rui,
1) Flink should always fetch records from Kafka if there some independent
of the parallelism of the consumer. The only problem which could appear is
that if you set the parallelism higher than the number of partitions, some
of the source operators won't get a partition assigned. Due to that, they
won't read data from Kafka.
2) numRecordsOutPerSecond should give you a good value for the throughput.
3) Could you check the logs whether the GraphiteReporter could be properly
started? Maybe the reporter jar has not been put in the lib folder.
Cheers,
Till
On Tue, Sep 25, 2018 at 11:30 AM Wang, Rui2 wrote:
> Hi Till,
>
>
>
> I have integrated Kafka 2.0 with Flink1.5.2, and it works well. Now, I
> need to know the performance of Flink, such as throughput vs latency.
>
> Here I just set my stream pipeline as: Kafka(source) -> flatMap -> map ->
> print(sink), with Kafka partitions: 32. I confront some issues:
>
>
>
> 1). If set each parallelism to 1, the job runs ok, but flatMap couldn’t
> receive records from Kafka.
>
> Well, I just set the parallelism of flatMap above 1, the pipeline work
> fine. Does Flink must be parallel when fetch records from Kafka?
>
>
>
> 2).Can I get the throughput by "numRecordsOutPerSecond"?
>
> Also, I think it may be right to override the open function which extends
> RichFlatMapFunction.
>
>
>
> 3).Regarding to the latency of operator(e.g. flatMap), after setting
> env.getConfig().setLatencyTrackingInterval(1000), I can get the latency
> only by the URL as below:
>
> input :
> http://server:8081/jobs/jobid/metrics?get=latency.source_id.X1.source_subtask_index.22.operator_id.Y1.operator_subtask_index.22.latency_stddev
>
> result:
> [{"id":"latency.source_id.X1.source_subtask_index.22.operator_id.Y1.operator_subtask_index.22.latency_stddev","value":"0.3854355088676483"}]
>
>
>
> But, it's a poor user experience, so I want a third-party to gather the
> value and display it, such as Graphite. I have configured the metrics in
> flink-conf.yaml as list below,
>
>
>
> metrics.reporters: varuy
>
> metrics.reporters.varuy.host: tracing115
>
> metrics.reporters.varuy.port: 2003
>
> metrics.reporters.varuy.class:
> org.apache.flink.metrics.graphite.GraphiteReporter
>
> metrics.reporters.varuy.interval: 1 SECONDS
>
> metrics.reporters.varuy.protocol: TCP
>
>
>
> when use cmd "lsof -i:2003" nothing return, I think the graphite seems not
> work.
>
> Could you give some advice to continue my work when you’re free, I’m
> looking forward to your reply!
>
>
>
> Best Regards & Thanks
>
> Rui, Wang
>
