Re: Monitoring job w/LocalStreamEnvironment

[Piotr Nowojski]

Hi,

Regarding metrics please check the "Writing an Integration test for 
flink-metrics” recent mailing list question. You can either use JMXReporter or 
write some custom reporter for this purpose.

Piotrek

> On 13 Oct 2017, at 20:57, Ken Krugler  wrote:
> 
> Hi Piotr,
> 
> Thanks for responding, see below.
> 
>> On Oct 12, 2017, at 7:51 AM, Piotr Nowojski > > wrote:
>> 
>> Hi,
>> 
>> Have you read the following doc?
>> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html
>>  
>> 
>> 
>> There are some hints regarding testing your application. Especially take a 
>> look at the example with using static field to communicate with the running 
>> job.
> 
> Yes, I’d read those.
> 
> I already have a bunch of Flink metrics, I was hoping to leverage those to 
> know when my test can safely terminate my iteration;
> 
> I guess I could create a metrics wrapper that also logs to a static class 
> during tests.
> 
> Regards,
> 
> — Ken
> 
> 
>>> On 12 Oct 2017, at 16:33, Ken Krugler >> > wrote:
>>> 
>>> Hi all,
>>> 
>>> With an iteration-based workflow, it’s helpful to be able to monitor the 
>>> job counters and explicitly terminate when the test has completed.
>>> 
>>> I didn’t see support for async job creation, though.
>>> 
>>> So I extended LocalStreamEnvironment to add an executeAsync(), which 
>>> returns the LocalFlinkMiniCluster.submitJobDetached() result.
>>> 
>>> But it appears like I need to have a ClusterClient in order to actually 
>>> monitor this job.
>>> 
>>> And ClusterClient is bound in with a lot of CLI code, so I’m hesitant to 
>>> try to extract what I need.
>>> 
>>> Is there an easier/recommended approach to the above?
>>> 
>>> Thanks!
>>> 
>>> — Ken
>>> 
>>> 
>>> http://about.me/kkrugler 
>>> +1 530-210-6378
>>> 
>> 
> 
> 
> http://about.me/kkrugler 
> +1 530-210-6378

Re: Monitoring job w/LocalStreamEnvironment

[Piotr Nowojski]

Hi,

Have you read the following doc?
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html
 


There are some hints regarding testing your application. Especially take a look 
at the example with using static field to communicate with the running job.

Piotrek

> On 12 Oct 2017, at 16:33, Ken Krugler  wrote:
> 
> Hi all,
> 
> With an iteration-based workflow, it’s helpful to be able to monitor the job 
> counters and explicitly terminate when the test has completed.
> 
> I didn’t see support for async job creation, though.
> 
> So I extended LocalStreamEnvironment to add an executeAsync(), which returns 
> the LocalFlinkMiniCluster.submitJobDetached() result.
> 
> But it appears like I need to have a ClusterClient in order to actually 
> monitor this job.
> 
> And ClusterClient is bound in with a lot of CLI code, so I’m hesitant to try 
> to extract what I need.
> 
> Is there an easier/recommended approach to the above?
> 
> Thanks!
> 
> — Ken
> 
> 
> http://about.me/kkrugler
> +1 530-210-6378
> 

Monitoring job w/LocalStreamEnvironment

[Ken Krugler]

Hi all,

With an iteration-based workflow, it’s helpful to be able to monitor the job 
counters and explicitly terminate when the test has completed.

I didn’t see support for async job creation, though.

So I extended LocalStreamEnvironment to add an executeAsync(), which returns 
the LocalFlinkMiniCluster.submitJobDetached() result.

But it appears like I need to have a ClusterClient in order to actually monitor 
this job.

And ClusterClient is bound in with a lot of CLI code, so I’m hesitant to try to 
extract what I need.

Is there an easier/recommended approach to the above?

Thanks!

— Ken


http://about.me/kkrugler
+1 530-210-6378