Re: Hello, a question about Dashborad in Flink

[Fabian Hueske]

The REST interface does also provide metrics about the number of records
and the size of the input and output of all tasks.
See:
- /jobs//vertices/
- /jobs//vertices//subtasks//attempts/
in
https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html#details-of-a-running-or-completed-job

However, not all of this data is going over the network because some tasks
can be locally connected.

Best, Fabian

2016-01-29 8:50 GMT+01:00 Philip Lee :

> Thanks,
>
> Is there any way to measure shuffle data (read and write) on Flink or
> Dashboard?
>
> I did not find the network usage metric in it.
>
> Best,
> Phil
>
> On Mon, Jan 25, 2016 at 5:06 PM, Fabian Hueske  wrote:
>
>> You can start a job and then periodically request and store information
>> about the running job and vertices from using corresponding REST calls [1].
>> The data will be in JSON format.
>> After the job finished, you can stop requesting data.
>>
>> Next you parse the JSON, extract the information you need and give it to
>> some plotting library.
>> As I said, it is not possible to pass this data back into Flink's
>> dashboard, but you have to process and plot it yourself.
>>
>> Best, Fabian
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs
>>
>>
>>
>> 2016-01-25 16:15 GMT+01:00 Philip Lee :
>>
>>> Hello,
>>>
>>> According to
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
>>> I cannot retrieve the job history from Dashboard after turnning off JM.
>>>
>>> But as Fabian mentioned here,
>>> "However, you can query all stats that are displayed by the dashboard
>>> via a REST API [1] while the JM is running and save them yourself. This way
>>> you can analyze the data also after the JM was stopped" could you explain
>>> about this sentence in detail.
>>>
>>> I want to evaluate timeline view of each function after a job is done.
>>>
>>> Thanks,
>>> Phil
>>>
>>
>>
>

Re: Hello, a question about Dashborad in Flink

[Philip Lee]

Great,

you menat the difference between narrow shuffle and global shuffle?

I use Flink version 0.9,
but it did not not work to access REST interface when I use "ssh tunnel" to
remote server.

it is from version of probelm?

Best,
Phil



On Fri, Jan 29, 2016 at 9:46 AM, Fabian Hueske  wrote:

> The REST interface does also provide metrics about the number of records
> and the size of the input and output of all tasks.
> See:
> - /jobs//vertices/
> -
> /jobs//vertices//subtasks//attempts/
> in
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html#details-of-a-running-or-completed-job
>
> However, not all of this data is going over the network because some tasks
> can be locally connected.
>
> Best, Fabian
>
> 2016-01-29 8:50 GMT+01:00 Philip Lee :
>
>> Thanks,
>>
>> Is there any way to measure shuffle data (read and write) on Flink or
>> Dashboard?
>>
>> I did not find the network usage metric in it.
>>
>> Best,
>> Phil
>>
>> On Mon, Jan 25, 2016 at 5:06 PM, Fabian Hueske  wrote:
>>
>>> You can start a job and then periodically request and store information
>>> about the running job and vertices from using corresponding REST calls [1].
>>> The data will be in JSON format.
>>> After the job finished, you can stop requesting data.
>>>
>>> Next you parse the JSON, extract the information you need and give it to
>>> some plotting library.
>>> As I said, it is not possible to pass this data back into Flink's
>>> dashboard, but you have to process and plot it yourself.
>>>
>>> Best, Fabian
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs
>>>
>>>
>>>
>>> 2016-01-25 16:15 GMT+01:00 Philip Lee :
>>>
 Hello,

 According to
 http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
 I cannot retrieve the job history from Dashboard after turnning off JM.

 But as Fabian mentioned here,
 "However, you can query all stats that are displayed by the dashboard
 via a REST API [1] while the JM is running and save them yourself. This way
 you can analyze the data also after the JM was stopped" could you explain
 about this sentence in detail.

 I want to evaluate timeline view of each function after a job is done.

 Thanks,
 Phil

>>>
>>>
>>
>

Re: Hello, a question about Dashborad in Flink

[Stephan Ewen]

Hi!

The REST monitoring interface and extended web dashboard were added in
version 0.10

Greetings,
Stephan


On Fri, Jan 29, 2016 at 9:55 AM, Philip Lee  wrote:

> Great,
>
> you menat the difference between narrow shuffle and global shuffle?
>
> I use Flink version 0.9,
> but it did not not work to access REST interface when I use "ssh tunnel"
> to remote server.
>
> it is from version of probelm?
>
> Best,
> Phil
>
>
>
> On Fri, Jan 29, 2016 at 9:46 AM, Fabian Hueske  wrote:
>
>> The REST interface does also provide metrics about the number of records
>> and the size of the input and output of all tasks.
>> See:
>> - /jobs//vertices/
>> -
>> /jobs//vertices//subtasks//attempts/
>> in
>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html#details-of-a-running-or-completed-job
>>
>> However, not all of this data is going over the network because some
>> tasks can be locally connected.
>>
>> Best, Fabian
>>
>> 2016-01-29 8:50 GMT+01:00 Philip Lee :
>>
>>> Thanks,
>>>
>>> Is there any way to measure shuffle data (read and write) on Flink or
>>> Dashboard?
>>>
>>> I did not find the network usage metric in it.
>>>
>>> Best,
>>> Phil
>>>
>>> On Mon, Jan 25, 2016 at 5:06 PM, Fabian Hueske 
>>> wrote:
>>>
 You can start a job and then periodically request and store information
 about the running job and vertices from using corresponding REST calls [1].
 The data will be in JSON format.
 After the job finished, you can stop requesting data.

 Next you parse the JSON, extract the information you need and give it
 to some plotting library.
 As I said, it is not possible to pass this data back into Flink's
 dashboard, but you have to process and plot it yourself.

 Best, Fabian

 [1]
 https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs



 2016-01-25 16:15 GMT+01:00 Philip Lee :

> Hello,
>
> According to
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
> I cannot retrieve the job history from Dashboard after turnning off JM.
>
> But as Fabian mentioned here,
> "However, you can query all stats that are displayed by the dashboard
> via a REST API [1] while the JM is running and save them yourself. This 
> way
> you can analyze the data also after the JM was stopped" could you explain
> about this sentence in detail.
>
> I want to evaluate timeline view of each function after a job is done.
>
> Thanks,
> Phil
>


>>>
>>
>

Re: Hello, a question about Dashborad in Flink

[Philip Lee]

Thanks,

Is there any way to measure shuffle data (read and write) on Flink or
Dashboard?

I did not find the network usage metric in it.

Best,
Phil

On Mon, Jan 25, 2016 at 5:06 PM, Fabian Hueske  wrote:

> You can start a job and then periodically request and store information
> about the running job and vertices from using corresponding REST calls [1].
> The data will be in JSON format.
> After the job finished, you can stop requesting data.
>
> Next you parse the JSON, extract the information you need and give it to
> some plotting library.
> As I said, it is not possible to pass this data back into Flink's
> dashboard, but you have to process and plot it yourself.
>
> Best, Fabian
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs
>
>
>
> 2016-01-25 16:15 GMT+01:00 Philip Lee :
>
>> Hello,
>>
>> According to
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
>> I cannot retrieve the job history from Dashboard after turnning off JM.
>>
>> But as Fabian mentioned here,
>> "However, you can query all stats that are displayed by the dashboard via
>> a REST API [1] while the JM is running and save them yourself. This way you
>> can analyze the data also after the JM was stopped" could you explain about
>> this sentence in detail.
>>
>> I want to evaluate timeline view of each function after a job is done.
>>
>> Thanks,
>> Phil
>>
>
>

Re: Hello, a question about Dashborad in Flink

[Fabian Hueske]

You can start a job and then periodically request and store information
about the running job and vertices from using corresponding REST calls [1].
The data will be in JSON format.
After the job finished, you can stop requesting data.

Next you parse the JSON, extract the information you need and give it to
some plotting library.
As I said, it is not possible to pass this data back into Flink's
dashboard, but you have to process and plot it yourself.

Best, Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html#overview-of-jobs



2016-01-25 16:15 GMT+01:00 Philip Lee :

> Hello,
>
> According to
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Apache-Flink-Web-Dashboard-Completed-Job-history-td4067.html,
> I cannot retrieve the job history from Dashboard after turnning off JM.
>
> But as Fabian mentioned here,
> "However, you can query all stats that are displayed by the dashboard via
> a REST API [1] while the JM is running and save them yourself. This way you
> can analyze the data also after the JM was stopped" could you explain about
> this sentence in detail.
>
> I want to evaluate timeline view of each function after a job is done.
>
> Thanks,
> Phil
>