Re: How to get Latency Tracking results?
Glad you got it working, and thanks for letting us know!
David
On Thu, Sep 10, 2020 at 2:40 PM Pankaj Chand
wrote:
> Actually, I was wrong. It turns out I was setting the values the wrong way
> in conf/flink-conf.yaml.
> I set "metrics.latency.interval 100" instead of "metrics.latency.interval:
> 100". Sorry about that.
>
> T
>
> On Thu, Sep 10, 2020 at 7:05 AM Pankaj Chand
> wrote:
>
>> Thank you, David!
>>
>> After setting ExecutionConfig for latency tracking in the
>> SocketWindowWordCount Java source code and rebuilding and using that
>> application jar file, I am now getting the latency tracking metrics using
>> the REST endpoint. The below documentation [1] seems to imply that
>> merely setting the parameters in flink-conf.yaml would generate the latency
>> tracking metrics, which is not happening in my case.
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html#latency-tracking
>>
>> On Wed, Sep 9, 2020 at 10:08 AM David Anderson
>> wrote:
>>
>>> Pankaj,
>>>
>>> I just checked, and the latency metrics for SocketWindowWordCount show
>>> up just fine for me with Flink 1.11.1. For me, the latency metrics existed
>>> even before I provided any data on the socket for the job to process. This
>>> makes sense, as the latency tracking markers will propagate through the job
>>> graph and provide measurements despite the lack of any real data for the
>>> job to process.
>>>
>>> These metrics have horrible names, but for me,
>>>
>>> curl -s
>>> http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics
>>>
>>> reveals a bunch of latency metrics, and, for example,
>>>
>>> curl -s
>>> http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics?get=latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999
>>>
>>> returns
>>>
>>>
>>> [{"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999","value":"105.0"}]
>>>
>>> Best,
>>> David
>>>
>>>
>>> On Wed, Sep 9, 2020 at 3:16 PM Pankaj Chand
>>> wrote:
>>>
Hi David,
Thanks for replying! Sorry, I forgot to mention I am using
Flink Version: 1.11.1, Commit ID: 7eb514a.
Is it possible that the default SocketWindowWordCount job is too simple
to generate Latency metrics? Or that the latency metrics disappear from the
output JSON when the data ingestion is zero?
Thanks,
Pankaj
On Wed, Sep 9, 2020 at 6:27 AM David Anderson
wrote:
> Pankaj,
>
> The Flink web UI doesn't do any visualizations of histogram metrics,
> so the only way to access the latency metrics is either through the REST
> api or a metrics reporter.
>
> The REST endpoint you tried is the correct place to find these metrics
> in all recent versions of Flink, but somewhere back before Flink 1.5 or
> 1.6
> (if I recall correctly) these metrics were task metrics. So if you are
> using an older version of Flink you'll need to dig deeper. I believe
> you'll
> find them in
>
> /jobs//vertices//subtasks/metrics
>
> Regards,
> David
>
>
>
> On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand <
> pankajchanda...@gmail.com> wrote:
>
>> Hello,
>>
>> How do I visualize (or extract) the results for Latency Tracking for
>> a Flink local cluster? I set "metrics.latency.interval 100" in the
>> conf/flink-conf.yaml file, and started the cluster and
>> SocketWindowWordCount job. However, I could not find the latency
>> distributions anywhere in the web UI, nor are there any latency metrics
>> in
>> the Metrics dropdown box for either task.
>>
>> I also set "metrics.latency.granularity "operator"" in
>> conf/flink-conf.yaml, but that did not help.
>>
>> When I tried to query the REST endpoint, I got the following output
>> which did not seem to contain anything related to latency:
>>
>> $ curl -s
>> http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics
>>
>>
>> [{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]
>>
>> Thanks,
>>
>> Pankaj
>>
>
Re: How to get Latency Tracking results?
Actually, I was wrong. It turns out I was setting the values the wrong way
in conf/flink-conf.yaml.
I set "metrics.latency.interval 100" instead of "metrics.latency.interval:
100". Sorry about that.
T
On Thu, Sep 10, 2020 at 7:05 AM Pankaj Chand
wrote:
> Thank you, David!
>
> After setting ExecutionConfig for latency tracking in the
> SocketWindowWordCount Java source code and rebuilding and using that
> application jar file, I am now getting the latency tracking metrics using
> the REST endpoint. The below documentation [1] seems to imply that
> merely setting the parameters in flink-conf.yaml would generate the latency
> tracking metrics, which is not happening in my case.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html#latency-tracking
>
> On Wed, Sep 9, 2020 at 10:08 AM David Anderson
> wrote:
>
>> Pankaj,
>>
>> I just checked, and the latency metrics for SocketWindowWordCount show up
>> just fine for me with Flink 1.11.1. For me, the latency metrics existed
>> even before I provided any data on the socket for the job to process. This
>> makes sense, as the latency tracking markers will propagate through the job
>> graph and provide measurements despite the lack of any real data for the
>> job to process.
>>
>> These metrics have horrible names, but for me,
>>
>> curl -s
>> http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics
>>
>> reveals a bunch of latency metrics, and, for example,
>>
>> curl -s
>> http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics?get=latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999
>>
>> returns
>>
>>
>> [{"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999","value":"105.0"}]
>>
>> Best,
>> David
>>
>>
>> On Wed, Sep 9, 2020 at 3:16 PM Pankaj Chand
>> wrote:
>>
>>> Hi David,
>>>
>>> Thanks for replying! Sorry, I forgot to mention I am using
>>> Flink Version: 1.11.1, Commit ID: 7eb514a.
>>> Is it possible that the default SocketWindowWordCount job is too simple
>>> to generate Latency metrics? Or that the latency metrics disappear from the
>>> output JSON when the data ingestion is zero?
>>>
>>> Thanks,
>>>
>>> Pankaj
>>>
>>>
>>> On Wed, Sep 9, 2020 at 6:27 AM David Anderson
>>> wrote:
>>>
Pankaj,
The Flink web UI doesn't do any visualizations of histogram metrics, so
the only way to access the latency metrics is either through the REST api
or a metrics reporter.
The REST endpoint you tried is the correct place to find these metrics
in all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6
(if I recall correctly) these metrics were task metrics. So if you are
using an older version of Flink you'll need to dig deeper. I believe you'll
find them in
/jobs//vertices//subtasks/metrics
Regards,
David
On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand
wrote:
> Hello,
>
> How do I visualize (or extract) the results for Latency Tracking for a
> Flink local cluster? I set "metrics.latency.interval 100" in the
> conf/flink-conf.yaml file, and started the cluster and
> SocketWindowWordCount job. However, I could not find the latency
> distributions anywhere in the web UI, nor are there any latency metrics in
> the Metrics dropdown box for either task.
>
> I also set "metrics.latency.granularity "operator"" in
> conf/flink-conf.yaml, but that did not help.
>
> When I tried to query the REST endpoint, I got the following output
> which did not seem to contain anything related to latency:
>
> $ curl -s
> http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics
>
>
> [{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]
>
> Thanks,
>
> Pankaj
>
Re: How to get Latency Tracking results?
Thank you, David!
After setting ExecutionConfig for latency tracking in the
SocketWindowWordCount Java source code and rebuilding and using that
application jar file, I am now getting the latency tracking metrics using
the REST endpoint. The below documentation [1] seems to imply that
merely setting the parameters in flink-conf.yaml would generate the latency
tracking metrics, which is not happening in my case.
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html#latency-tracking
On Wed, Sep 9, 2020 at 10:08 AM David Anderson
wrote:
> Pankaj,
>
> I just checked, and the latency metrics for SocketWindowWordCount show up
> just fine for me with Flink 1.11.1. For me, the latency metrics existed
> even before I provided any data on the socket for the job to process. This
> makes sense, as the latency tracking markers will propagate through the job
> graph and provide measurements despite the lack of any real data for the
> job to process.
>
> These metrics have horrible names, but for me,
>
> curl -s
> http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics
>
> reveals a bunch of latency metrics, and, for example,
>
> curl -s
> http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics?get=latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999
>
> returns
>
>
> [{"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999","value":"105.0"}]
>
> Best,
> David
>
>
> On Wed, Sep 9, 2020 at 3:16 PM Pankaj Chand
> wrote:
>
>> Hi David,
>>
>> Thanks for replying! Sorry, I forgot to mention I am using Flink Version:
>> 1.11.1, Commit ID: 7eb514a.
>> Is it possible that the default SocketWindowWordCount job is too simple
>> to generate Latency metrics? Or that the latency metrics disappear from the
>> output JSON when the data ingestion is zero?
>>
>> Thanks,
>>
>> Pankaj
>>
>>
>> On Wed, Sep 9, 2020 at 6:27 AM David Anderson
>> wrote:
>>
>>> Pankaj,
>>>
>>> The Flink web UI doesn't do any visualizations of histogram metrics, so
>>> the only way to access the latency metrics is either through the REST api
>>> or a metrics reporter.
>>>
>>> The REST endpoint you tried is the correct place to find these metrics
>>> in all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6
>>> (if I recall correctly) these metrics were task metrics. So if you are
>>> using an older version of Flink you'll need to dig deeper. I believe you'll
>>> find them in
>>>
>>> /jobs//vertices//subtasks/metrics
>>>
>>> Regards,
>>> David
>>>
>>>
>>>
>>> On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand
>>> wrote:
>>>
Hello,
How do I visualize (or extract) the results for Latency Tracking for a
Flink local cluster? I set "metrics.latency.interval 100" in the
conf/flink-conf.yaml file, and started the cluster and
SocketWindowWordCount job. However, I could not find the latency
distributions anywhere in the web UI, nor are there any latency metrics in
the Metrics dropdown box for either task.
I also set "metrics.latency.granularity "operator"" in
conf/flink-conf.yaml, but that did not help.
When I tried to query the REST endpoint, I got the following output
which did not seem to contain anything related to latency:
$ curl -s
http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics
[{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]
Thanks,
Pankaj
>>>
Re: How to get Latency Tracking results?
Pankaj,
I just checked, and the latency metrics for SocketWindowWordCount show up
just fine for me with Flink 1.11.1. For me, the latency metrics existed
even before I provided any data on the socket for the job to process. This
makes sense, as the latency tracking markers will propagate through the job
graph and provide measurements despite the lack of any real data for the
job to process.
These metrics have horrible names, but for me,
curl -s http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics
reveals a bunch of latency metrics, and, for example,
curl -s
http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics?get=latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999
returns
[{"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999","value":"105.0"}]
Best,
David
On Wed, Sep 9, 2020 at 3:16 PM Pankaj Chand
wrote:
> Hi David,
>
> Thanks for replying! Sorry, I forgot to mention I am using Flink Version:
> 1.11.1, Commit ID: 7eb514a.
> Is it possible that the default SocketWindowWordCount job is too simple to
> generate Latency metrics? Or that the latency metrics disappear from the
> output JSON when the data ingestion is zero?
>
> Thanks,
>
> Pankaj
>
>
> On Wed, Sep 9, 2020 at 6:27 AM David Anderson
> wrote:
>
>> Pankaj,
>>
>> The Flink web UI doesn't do any visualizations of histogram metrics, so
>> the only way to access the latency metrics is either through the REST api
>> or a metrics reporter.
>>
>> The REST endpoint you tried is the correct place to find these metrics in
>> all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6
>> (if I recall correctly) these metrics were task metrics. So if you are
>> using an older version of Flink you'll need to dig deeper. I believe you'll
>> find them in
>>
>> /jobs//vertices//subtasks/metrics
>>
>> Regards,
>> David
>>
>>
>>
>> On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand
>> wrote:
>>
>>> Hello,
>>>
>>> How do I visualize (or extract) the results for Latency Tracking for a
>>> Flink local cluster? I set "metrics.latency.interval 100" in the
>>> conf/flink-conf.yaml file, and started the cluster and
>>> SocketWindowWordCount job. However, I could not find the latency
>>> distributions anywhere in the web UI, nor are there any latency metrics in
>>> the Metrics dropdown box for either task.
>>>
>>> I also set "metrics.latency.granularity "operator"" in
>>> conf/flink-conf.yaml, but that did not help.
>>>
>>> When I tried to query the REST endpoint, I got the following output
>>> which did not seem to contain anything related to latency:
>>>
>>> $ curl -s
>>> http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics
>>>
>>>
>>> [{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]
>>>
>>> Thanks,
>>>
>>> Pankaj
>>>
>>
Re: How to get Latency Tracking results?
Hi David,
Thanks for replying! Sorry, I forgot to mention I am using Flink Version:
1.11.1, Commit ID: 7eb514a.
Is it possible that the default SocketWindowWordCount job is too simple to
generate Latency metrics? Or that the latency metrics disappear from the
output JSON when the data ingestion is zero?
Thanks,
Pankaj
On Wed, Sep 9, 2020 at 6:27 AM David Anderson wrote:
> Pankaj,
>
> The Flink web UI doesn't do any visualizations of histogram metrics, so
> the only way to access the latency metrics is either through the REST api
> or a metrics reporter.
>
> The REST endpoint you tried is the correct place to find these metrics in
> all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6
> (if I recall correctly) these metrics were task metrics. So if you are
> using an older version of Flink you'll need to dig deeper. I believe you'll
> find them in
>
> /jobs//vertices//subtasks/metrics
>
> Regards,
> David
>
>
>
> On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand
> wrote:
>
>> Hello,
>>
>> How do I visualize (or extract) the results for Latency Tracking for a
>> Flink local cluster? I set "metrics.latency.interval 100" in the
>> conf/flink-conf.yaml file, and started the cluster and
>> SocketWindowWordCount job. However, I could not find the latency
>> distributions anywhere in the web UI, nor are there any latency metrics in
>> the Metrics dropdown box for either task.
>>
>> I also set "metrics.latency.granularity "operator"" in
>> conf/flink-conf.yaml, but that did not help.
>>
>> When I tried to query the REST endpoint, I got the following output which
>> did not seem to contain anything related to latency:
>>
>> $ curl -s
>> http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics
>>
>>
>> [{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]
>>
>> Thanks,
>>
>> Pankaj
>>
>
Re: How to get Latency Tracking results?
Pankaj,
The Flink web UI doesn't do any visualizations of histogram metrics, so the
only way to access the latency metrics is either through the REST api or a
metrics reporter.
The REST endpoint you tried is the correct place to find these metrics in
all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6
(if I recall correctly) these metrics were task metrics. So if you are
using an older version of Flink you'll need to dig deeper. I believe you'll
find them in
/jobs//vertices//subtasks/metrics
Regards,
David
On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand
wrote:
> Hello,
>
> How do I visualize (or extract) the results for Latency Tracking for a
> Flink local cluster? I set "metrics.latency.interval 100" in the
> conf/flink-conf.yaml file, and started the cluster and
> SocketWindowWordCount job. However, I could not find the latency
> distributions anywhere in the web UI, nor are there any latency metrics in
> the Metrics dropdown box for either task.
>
> I also set "metrics.latency.granularity "operator"" in
> conf/flink-conf.yaml, but that did not help.
>
> When I tried to query the REST endpoint, I got the following output which
> did not seem to contain anything related to latency:
>
> $ curl -s
> http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics
>
>
> [{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]
>
> Thanks,
>
> Pankaj
>
How to get Latency Tracking results?
Hello,
How do I visualize (or extract) the results for Latency Tracking for a
Flink local cluster? I set "metrics.latency.interval 100" in the
conf/flink-conf.yaml file, and started the cluster and
SocketWindowWordCount job. However, I could not find the latency
distributions anywhere in the web UI, nor are there any latency metrics in
the Metrics dropdown box for either task.
I also set "metrics.latency.granularity "operator"" in
conf/flink-conf.yaml, but that did not help.
When I tried to query the REST endpoint, I got the following output which
did not seem to contain anything related to latency:
$ curl -s
http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics
[{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]
Thanks,
Pankaj
