Glad you got it working, and thanks for letting us know! David
On Thu, Sep 10, 2020 at 2:40 PM Pankaj Chand <pankajchanda...@gmail.com> wrote: > Actually, I was wrong. It turns out I was setting the values the wrong way > in conf/flink-conf.yaml. > I set "metrics.latency.interval 100" instead of "metrics.latency.interval: > 100". Sorry about that. > > T > > On Thu, Sep 10, 2020 at 7:05 AM Pankaj Chand <pankajchanda...@gmail.com> > wrote: > >> Thank you, David! >> >> After setting ExecutionConfig for latency tracking in the >> SocketWindowWordCount Java source code and rebuilding and using that >> application jar file, I am now getting the latency tracking metrics using >> the REST endpoint. The below documentation [1] seems to imply that >> merely setting the parameters in flink-conf.yaml would generate the latency >> tracking metrics, which is not happening in my case. >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html#latency-tracking >> >> On Wed, Sep 9, 2020 at 10:08 AM David Anderson <da...@alpinegizmo.com> >> wrote: >> >>> Pankaj, >>> >>> I just checked, and the latency metrics for SocketWindowWordCount show >>> up just fine for me with Flink 1.11.1. For me, the latency metrics existed >>> even before I provided any data on the socket for the job to process. This >>> makes sense, as the latency tracking markers will propagate through the job >>> graph and provide measurements despite the lack of any real data for the >>> job to process. >>> >>> These metrics have horrible names, but for me, >>> >>> curl -s >>> http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics >>> >>> reveals a bunch of latency metrics, and, for example, >>> >>> curl -s >>> http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics?get=latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999 >>> >>> returns >>> >>> >>> [{"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999","value":"105.0"}] >>> >>> Best, >>> David >>> >>> >>> On Wed, Sep 9, 2020 at 3:16 PM Pankaj Chand <pankajchanda...@gmail.com> >>> wrote: >>> >>>> Hi David, >>>> >>>> Thanks for replying! Sorry, I forgot to mention I am using >>>> Flink Version: 1.11.1, Commit ID: 7eb514a. >>>> Is it possible that the default SocketWindowWordCount job is too simple >>>> to generate Latency metrics? Or that the latency metrics disappear from the >>>> output JSON when the data ingestion is zero? >>>> >>>> Thanks, >>>> >>>> Pankaj >>>> >>>> >>>> On Wed, Sep 9, 2020 at 6:27 AM David Anderson <da...@alpinegizmo.com> >>>> wrote: >>>> >>>>> Pankaj, >>>>> >>>>> The Flink web UI doesn't do any visualizations of histogram metrics, >>>>> so the only way to access the latency metrics is either through the REST >>>>> api or a metrics reporter. >>>>> >>>>> The REST endpoint you tried is the correct place to find these metrics >>>>> in all recent versions of Flink, but somewhere back before Flink 1.5 or >>>>> 1.6 >>>>> (if I recall correctly) these metrics were task metrics. So if you are >>>>> using an older version of Flink you'll need to dig deeper. I believe >>>>> you'll >>>>> find them in >>>>> >>>>> /jobs/<jobid>/vertices/<vertexid>/subtasks/metrics >>>>> >>>>> Regards, >>>>> David >>>>> >>>>> >>>>> >>>>> On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand < >>>>> pankajchanda...@gmail.com> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> How do I visualize (or extract) the results for Latency Tracking for >>>>>> a Flink local cluster? I set "metrics.latency.interval 100" in the >>>>>> conf/flink-conf.yaml file, and started the cluster and >>>>>> SocketWindowWordCount job. However, I could not find the latency >>>>>> distributions anywhere in the web UI, nor are there any latency metrics >>>>>> in >>>>>> the Metrics dropdown box for either task. >>>>>> >>>>>> I also set "metrics.latency.granularity "operator"" in >>>>>> conf/flink-conf.yaml, but that did not help. >>>>>> >>>>>> When I tried to query the REST endpoint, I got the following output >>>>>> which did not seem to contain anything related to latency: >>>>>> >>>>>> $ curl -s >>>>>> http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics >>>>>> >>>>>> >>>>>> [{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}] >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Pankaj >>>>>> >>>>>
