Re: Multiple restarts of Local Cluster

[Stephan Ewen]

Have a look at the class IOManager and IOManagerAsync, it is a good example
of how we use these hooks for cleanup.

The constructor usually installs them, and the shutdown logic removes them.

On Thu, Sep 3, 2015 at 9:19 PM, Stephan Ewen  wrote:

> Stopping the JVM process clean up all resources, except temp files.
>
> Everything that creates temp files uses a shutdown hook to remove these:
> IOManager, BlobManager, LibraryCache, ...
>
> On Wed, Sep 2, 2015 at 7:40 PM, Sachin Goel 
> wrote:
>
>> I'm not sure what you mean by "Crucial cleanup is in shutdown hooks".
>> Could you elaborate?
>>
>> -- Sachin Goel
>> Computer Science, IIT Delhi
>> m. +91-9871457685
>>
>> On Wed, Sep 2, 2015 at 10:25 PM, Stephan Ewen  wrote:
>>
>>> You can always shut down a cluster manually (via shutdown()) and if the
>>> JVM simply exists, all is well as well. Crucial cleanup is in shutdown
>>> hooks.
>>>
>>> On Wed, Sep 2, 2015 at 6:22 PM, Till Rohrmann 
>>> wrote:
>>>
 If I'm not mistaken, then the cluster should be properly terminated
 when it gets garbage collected. Thus, also when the main method exits.

 On Wed, Sep 2, 2015 at 6:14 PM, Sachin Goel 
 wrote:

> If I'm right, all Tests use either the MultipleProgramTestBase or
> JavaProgramTestBase​. Those shut down the cluster explicitly anyway.
> I will make sure if this is the case.
>
> Regards
> Sachin
>
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>
> On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann 
> wrote:
>
>> Maybe we can create a single PlanExecutor for the LocalEnvironment
>> which is used when calling execute. This of course entails that we
>> don’t call stop on the LocalCluster. For cases where the program
>> exits after calling execute, this should be fine because all resources 
>> will
>> then be released anyway. It might matter for the test execution where 
>> maven
>> reuses the JVMs and where the LocalFlinkMiniCluster won’t be garbage
>> collected right away. You could try it out and see what happens.
>>
>> Cheers,
>> Till
>> ​
>>
>> On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann 
>> wrote:
>>
>>> Oh sorry, then I got the wrong context. I somehow thought it was
>>> about test cases because I read `MultipleProgramTestBase` etc. Sorry my 
>>> bad.
>>>
>>> On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel <
>>> sachingoel0...@gmail.com> wrote:
>>>
 I was under the impression that the @AfterClass annotation can only
 be used in test classes.
 Even so, the idea is that a user program running in the IDE should
 not be starting up the cluster several times [my primary concern is the
 addition of the persist operator], and we certainly cannot ask the 
 user to
 terminate the cluster after execution, while in local mode.

 -- Sachin Goel
 Computer Science, IIT Delhi
 m. +91-9871457685

 On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann  wrote:

> Why is it not possible to shut down the local cluster? Can’t you
> shut it down in the @AfterClass method?
> ​
>
> On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <
> sachingoel0...@gmail.com> wrote:
>
>> Yes. That will work too. However, then it isn't possible to shut
>> down the local cluster. [Is it necessary to do so or does it shut 
>> down
>> automatically when the program exists? I'm not entirely sure.]
>>
>> -- Sachin Goel
>> Computer Science, IIT Delhi
>> m. +91-9871457685
>>
>> On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen 
>> wrote:
>>
>>> Have a look at some other tests, like the checkpointing tests.
>>> They start one cluster manually and keep it running. They connect 
>>> against
>>> it using the remote environment ("localhost",
>>> miniCluster.getJobManagerRpcPort()).
>>>
>>> That works nicely...
>>>
>>> On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <
>>> sachingoel0...@gmail.com> wrote:
>>>
 Hi all
 While using LocalEnvironment, in case the program triggers
 execution several times, the {{LocalFlinkMiniCluster}} is started 
 as many
 times. This can consume a lot of time in setting up and tearing 
 down the
 cluster. Further, this hinders with a new functionality I'm 
 working on
 based on persisted results.
 One potential solution could be to follow the methodology 

Re: Multiple restarts of Local Cluster

[Stephan Ewen]

Stopping the JVM process clean up all resources, except temp files.

Everything that creates temp files uses a shutdown hook to remove these:
IOManager, BlobManager, LibraryCache, ...

On Wed, Sep 2, 2015 at 7:40 PM, Sachin Goel 
wrote:

> I'm not sure what you mean by "Crucial cleanup is in shutdown hooks".
> Could you elaborate?
>
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>
> On Wed, Sep 2, 2015 at 10:25 PM, Stephan Ewen  wrote:
>
>> You can always shut down a cluster manually (via shutdown()) and if the
>> JVM simply exists, all is well as well. Crucial cleanup is in shutdown
>> hooks.
>>
>> On Wed, Sep 2, 2015 at 6:22 PM, Till Rohrmann 
>> wrote:
>>
>>> If I'm not mistaken, then the cluster should be properly terminated when
>>> it gets garbage collected. Thus, also when the main method exits.
>>>
>>> On Wed, Sep 2, 2015 at 6:14 PM, Sachin Goel 
>>> wrote:
>>>
 If I'm right, all Tests use either the MultipleProgramTestBase or
 JavaProgramTestBase​. Those shut down the cluster explicitly anyway.
 I will make sure if this is the case.

 Regards
 Sachin

 -- Sachin Goel
 Computer Science, IIT Delhi
 m. +91-9871457685

 On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann 
 wrote:

> Maybe we can create a single PlanExecutor for the LocalEnvironment
> which is used when calling execute. This of course entails that we
> don’t call stop on the LocalCluster. For cases where the program
> exits after calling execute, this should be fine because all resources 
> will
> then be released anyway. It might matter for the test execution where 
> maven
> reuses the JVMs and where the LocalFlinkMiniCluster won’t be garbage
> collected right away. You could try it out and see what happens.
>
> Cheers,
> Till
> ​
>
> On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann 
> wrote:
>
>> Oh sorry, then I got the wrong context. I somehow thought it was
>> about test cases because I read `MultipleProgramTestBase` etc. Sorry my 
>> bad.
>>
>> On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel > > wrote:
>>
>>> I was under the impression that the @AfterClass annotation can only
>>> be used in test classes.
>>> Even so, the idea is that a user program running in the IDE should
>>> not be starting up the cluster several times [my primary concern is the
>>> addition of the persist operator], and we certainly cannot ask the user 
>>> to
>>> terminate the cluster after execution, while in local mode.
>>>
>>> -- Sachin Goel
>>> Computer Science, IIT Delhi
>>> m. +91-9871457685
>>>
>>> On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann 
>>> wrote:
>>>
 Why is it not possible to shut down the local cluster? Can’t you
 shut it down in the @AfterClass method?
 ​

 On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <
 sachingoel0...@gmail.com> wrote:

> Yes. That will work too. However, then it isn't possible to shut
> down the local cluster. [Is it necessary to do so or does it shut down
> automatically when the program exists? I'm not entirely sure.]
>
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>
> On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen 
> wrote:
>
>> Have a look at some other tests, like the checkpointing tests.
>> They start one cluster manually and keep it running. They connect 
>> against
>> it using the remote environment ("localhost",
>> miniCluster.getJobManagerRpcPort()).
>>
>> That works nicely...
>>
>> On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <
>> sachingoel0...@gmail.com> wrote:
>>
>>> Hi all
>>> While using LocalEnvironment, in case the program triggers
>>> execution several times, the {{LocalFlinkMiniCluster}} is started 
>>> as many
>>> times. This can consume a lot of time in setting up and tearing 
>>> down the
>>> cluster. Further, this hinders with a new functionality I'm working 
>>> on
>>> based on persisted results.
>>> One potential solution could be to follow the methodology in
>>> `MultipleProgramsTestBase`. The user code then would have to reside 
>>> in a
>>> fixed function name, instead of the main method. Or is that too 
>>> cumbersome?
>>>
>>> Regards
>>> Sachin
>>> -- Sachin Goel
>>> Computer Science, IIT Delhi
>>> m. +91-9871457685
>>>
>>
>>
>

Re: Multiple restarts of Local Cluster

[Sachin Goel]

If I'm right, all Tests use either the MultipleProgramTestBase or
JavaProgramTestBase​. Those shut down the cluster explicitly anyway.
I will make sure if this is the case.

Regards
Sachin

-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685

On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann  wrote:

> Maybe we can create a single PlanExecutor for the LocalEnvironment which
> is used when calling execute. This of course entails that we don’t call
> stop on the LocalCluster. For cases where the program exits after calling
> execute, this should be fine because all resources will then be released
> anyway. It might matter for the test execution where maven reuses the JVMs
> and where the LocalFlinkMiniCluster won’t be garbage collected right
> away. You could try it out and see what happens.
>
> Cheers,
> Till
> ​
>
> On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann 
> wrote:
>
>> Oh sorry, then I got the wrong context. I somehow thought it was about
>> test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.
>>
>> On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel 
>> wrote:
>>
>>> I was under the impression that the @AfterClass annotation can only be
>>> used in test classes.
>>> Even so, the idea is that a user program running in the IDE should not
>>> be starting up the cluster several times [my primary concern is the
>>> addition of the persist operator], and we certainly cannot ask the user to
>>> terminate the cluster after execution, while in local mode.
>>>
>>> -- Sachin Goel
>>> Computer Science, IIT Delhi
>>> m. +91-9871457685
>>>
>>> On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann 
>>> wrote:
>>>
 Why is it not possible to shut down the local cluster? Can’t you shut
 it down in the @AfterClass method?
 ​

 On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel 
 wrote:

> Yes. That will work too. However, then it isn't possible to shut down
> the local cluster. [Is it necessary to do so or does it shut down
> automatically when the program exists? I'm not entirely sure.]
>
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>
> On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen  wrote:
>
>> Have a look at some other tests, like the checkpointing tests. They
>> start one cluster manually and keep it running. They connect against it
>> using the remote environment ("localhost",
>> miniCluster.getJobManagerRpcPort()).
>>
>> That works nicely...
>>
>> On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel > > wrote:
>>
>>> Hi all
>>> While using LocalEnvironment, in case the program triggers execution
>>> several times, the {{LocalFlinkMiniCluster}} is started as many times. 
>>> This
>>> can consume a lot of time in setting up and tearing down the cluster.
>>> Further, this hinders with a new functionality I'm working on based on
>>> persisted results.
>>> One potential solution could be to follow the methodology in
>>> `MultipleProgramsTestBase`. The user code then would have to reside in a
>>> fixed function name, instead of the main method. Or is that too 
>>> cumbersome?
>>>
>>> Regards
>>> Sachin
>>> -- Sachin Goel
>>> Computer Science, IIT Delhi
>>> m. +91-9871457685
>>>
>>
>>
>

>>>
>>
>

Re: Multiple restarts of Local Cluster

[Till Rohrmann]

Oh sorry, then I got the wrong context. I somehow thought it was about test
cases because I read `MultipleProgramTestBase` etc. Sorry my bad.

On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel 
wrote:

> I was under the impression that the @AfterClass annotation can only be
> used in test classes.
> Even so, the idea is that a user program running in the IDE should not be
> starting up the cluster several times [my primary concern is the addition
> of the persist operator], and we certainly cannot ask the user to terminate
> the cluster after execution, while in local mode.
>
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>
> On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann 
> wrote:
>
>> Why is it not possible to shut down the local cluster? Can’t you shut it
>> down in the @AfterClass method?
>> ​
>>
>> On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel 
>> wrote:
>>
>>> Yes. That will work too. However, then it isn't possible to shut down
>>> the local cluster. [Is it necessary to do so or does it shut down
>>> automatically when the program exists? I'm not entirely sure.]
>>>
>>> -- Sachin Goel
>>> Computer Science, IIT Delhi
>>> m. +91-9871457685
>>>
>>> On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen  wrote:
>>>
 Have a look at some other tests, like the checkpointing tests. They
 start one cluster manually and keep it running. They connect against it
 using the remote environment ("localhost",
 miniCluster.getJobManagerRpcPort()).

 That works nicely...

 On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel 
 wrote:

> Hi all
> While using LocalEnvironment, in case the program triggers execution
> several times, the {{LocalFlinkMiniCluster}} is started as many times. 
> This
> can consume a lot of time in setting up and tearing down the cluster.
> Further, this hinders with a new functionality I'm working on based on
> persisted results.
> One potential solution could be to follow the methodology in
> `MultipleProgramsTestBase`. The user code then would have to reside in a
> fixed function name, instead of the main method. Or is that too 
> cumbersome?
>
> Regards
> Sachin
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>


>>>
>>
>

Re: Multiple restarts of Local Cluster

[Till Rohrmann]

Why is it not possible to shut down the local cluster? Can’t you shut it
down in the @AfterClass method?
​

On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel 
wrote:

> Yes. That will work too. However, then it isn't possible to shut down the
> local cluster. [Is it necessary to do so or does it shut down automatically
> when the program exists? I'm not entirely sure.]
>
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>
> On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen  wrote:
>
>> Have a look at some other tests, like the checkpointing tests. They start
>> one cluster manually and keep it running. They connect against it using the
>> remote environment ("localhost", miniCluster.getJobManagerRpcPort()).
>>
>> That works nicely...
>>
>> On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel 
>> wrote:
>>
>>> Hi all
>>> While using LocalEnvironment, in case the program triggers execution
>>> several times, the {{LocalFlinkMiniCluster}} is started as many times. This
>>> can consume a lot of time in setting up and tearing down the cluster.
>>> Further, this hinders with a new functionality I'm working on based on
>>> persisted results.
>>> One potential solution could be to follow the methodology in
>>> `MultipleProgramsTestBase`. The user code then would have to reside in a
>>> fixed function name, instead of the main method. Or is that too cumbersome?
>>>
>>> Regards
>>> Sachin
>>> -- Sachin Goel
>>> Computer Science, IIT Delhi
>>> m. +91-9871457685
>>>
>>
>>
>

Re: Multiple restarts of Local Cluster

[Sachin Goel]

I was under the impression that the @AfterClass annotation can only be used
in test classes.
Even so, the idea is that a user program running in the IDE should not be
starting up the cluster several times [my primary concern is the addition
of the persist operator], and we certainly cannot ask the user to terminate
the cluster after execution, while in local mode.

-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685

On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann  wrote:

> Why is it not possible to shut down the local cluster? Can’t you shut it
> down in the @AfterClass method?
> ​
>
> On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel 
> wrote:
>
>> Yes. That will work too. However, then it isn't possible to shut down the
>> local cluster. [Is it necessary to do so or does it shut down automatically
>> when the program exists? I'm not entirely sure.]
>>
>> -- Sachin Goel
>> Computer Science, IIT Delhi
>> m. +91-9871457685
>>
>> On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen  wrote:
>>
>>> Have a look at some other tests, like the checkpointing tests. They
>>> start one cluster manually and keep it running. They connect against it
>>> using the remote environment ("localhost",
>>> miniCluster.getJobManagerRpcPort()).
>>>
>>> That works nicely...
>>>
>>> On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel 
>>> wrote:
>>>
 Hi all
 While using LocalEnvironment, in case the program triggers execution
 several times, the {{LocalFlinkMiniCluster}} is started as many times. This
 can consume a lot of time in setting up and tearing down the cluster.
 Further, this hinders with a new functionality I'm working on based on
 persisted results.
 One potential solution could be to follow the methodology in
 `MultipleProgramsTestBase`. The user code then would have to reside in a
 fixed function name, instead of the main method. Or is that too cumbersome?

 Regards
 Sachin
 -- Sachin Goel
 Computer Science, IIT Delhi
 m. +91-9871457685

>>>
>>>
>>
>

Re: Multiple restarts of Local Cluster

[Stephan Ewen]

You can always shut down a cluster manually (via shutdown()) and if the JVM
simply exists, all is well as well. Crucial cleanup is in shutdown hooks.

On Wed, Sep 2, 2015 at 6:22 PM, Till Rohrmann 
wrote:

> If I'm not mistaken, then the cluster should be properly terminated when
> it gets garbage collected. Thus, also when the main method exits.
>
> On Wed, Sep 2, 2015 at 6:14 PM, Sachin Goel 
> wrote:
>
>> If I'm right, all Tests use either the MultipleProgramTestBase or
>> JavaProgramTestBase​. Those shut down the cluster explicitly anyway.
>> I will make sure if this is the case.
>>
>> Regards
>> Sachin
>>
>> -- Sachin Goel
>> Computer Science, IIT Delhi
>> m. +91-9871457685
>>
>> On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann 
>> wrote:
>>
>>> Maybe we can create a single PlanExecutor for the LocalEnvironment
>>> which is used when calling execute. This of course entails that we
>>> don’t call stop on the LocalCluster. For cases where the program exits
>>> after calling execute, this should be fine because all resources will then
>>> be released anyway. It might matter for the test execution where maven
>>> reuses the JVMs and where the LocalFlinkMiniCluster won’t be garbage
>>> collected right away. You could try it out and see what happens.
>>>
>>> Cheers,
>>> Till
>>> ​
>>>
>>> On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann 
>>> wrote:
>>>
 Oh sorry, then I got the wrong context. I somehow thought it was about
 test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.

 On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel 
 wrote:

> I was under the impression that the @AfterClass annotation can only be
> used in test classes.
> Even so, the idea is that a user program running in the IDE should not
> be starting up the cluster several times [my primary concern is the
> addition of the persist operator], and we certainly cannot ask the user to
> terminate the cluster after execution, while in local mode.
>
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>
> On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann 
> wrote:
>
>> Why is it not possible to shut down the local cluster? Can’t you shut
>> it down in the @AfterClass method?
>> ​
>>
>> On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel > > wrote:
>>
>>> Yes. That will work too. However, then it isn't possible to shut
>>> down the local cluster. [Is it necessary to do so or does it shut down
>>> automatically when the program exists? I'm not entirely sure.]
>>>
>>> -- Sachin Goel
>>> Computer Science, IIT Delhi
>>> m. +91-9871457685
>>>
>>> On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen 
>>> wrote:
>>>
 Have a look at some other tests, like the checkpointing tests. They
 start one cluster manually and keep it running. They connect against it
 using the remote environment ("localhost",
 miniCluster.getJobManagerRpcPort()).

 That works nicely...

 On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <
 sachingoel0...@gmail.com> wrote:

> Hi all
> While using LocalEnvironment, in case the program triggers
> execution several times, the {{LocalFlinkMiniCluster}} is started as 
> many
> times. This can consume a lot of time in setting up and tearing down 
> the
> cluster. Further, this hinders with a new functionality I'm working on
> based on persisted results.
> One potential solution could be to follow the methodology in
> `MultipleProgramsTestBase`. The user code then would have to reside 
> in a
> fixed function name, instead of the main method. Or is that too 
> cumbersome?
>
> Regards
> Sachin
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>


>>>
>>
>

>>>
>>
>

Re: Multiple restarts of Local Cluster

[Till Rohrmann]

Maybe we can create a single PlanExecutor for the LocalEnvironment which is
used when calling execute. This of course entails that we don’t call stop
on the LocalCluster. For cases where the program exits after calling
execute, this should be fine because all resources will then be released
anyway. It might matter for the test execution where maven reuses the JVMs
and where the LocalFlinkMiniCluster won’t be garbage collected right away.
You could try it out and see what happens.

Cheers,
Till
​

On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann  wrote:

> Oh sorry, then I got the wrong context. I somehow thought it was about
> test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.
>
> On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel 
> wrote:
>
>> I was under the impression that the @AfterClass annotation can only be
>> used in test classes.
>> Even so, the idea is that a user program running in the IDE should not be
>> starting up the cluster several times [my primary concern is the addition
>> of the persist operator], and we certainly cannot ask the user to terminate
>> the cluster after execution, while in local mode.
>>
>> -- Sachin Goel
>> Computer Science, IIT Delhi
>> m. +91-9871457685
>>
>> On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann 
>> wrote:
>>
>>> Why is it not possible to shut down the local cluster? Can’t you shut it
>>> down in the @AfterClass method?
>>> ​
>>>
>>> On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel 
>>> wrote:
>>>
 Yes. That will work too. However, then it isn't possible to shut down
 the local cluster. [Is it necessary to do so or does it shut down
 automatically when the program exists? I'm not entirely sure.]

 -- Sachin Goel
 Computer Science, IIT Delhi
 m. +91-9871457685

 On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen  wrote:

> Have a look at some other tests, like the checkpointing tests. They
> start one cluster manually and keep it running. They connect against it
> using the remote environment ("localhost",
> miniCluster.getJobManagerRpcPort()).
>
> That works nicely...
>
> On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel 
> wrote:
>
>> Hi all
>> While using LocalEnvironment, in case the program triggers execution
>> several times, the {{LocalFlinkMiniCluster}} is started as many times. 
>> This
>> can consume a lot of time in setting up and tearing down the cluster.
>> Further, this hinders with a new functionality I'm working on based on
>> persisted results.
>> One potential solution could be to follow the methodology in
>> `MultipleProgramsTestBase`. The user code then would have to reside in a
>> fixed function name, instead of the main method. Or is that too 
>> cumbersome?
>>
>> Regards
>> Sachin
>> -- Sachin Goel
>> Computer Science, IIT Delhi
>> m. +91-9871457685
>>
>
>

>>>
>>
>

Re: Multiple restarts of Local Cluster

[Sachin Goel]

Okay. No problem.
Any suggestions for the correct context though? :')

I don't think something like a {{FlinkProgram}} class is a good idea [User
would need to override a {{program}} method and we will make sure the
cluster is setup only once and torn down properly only after the user code
finishes completely].

However, if this isn't so, the shutting down of cluster becomes impossible.
Can I assume however, that the actor system will be shut down automatically
when the main method exits? After all the JVM will terminate. If so, I can
make some changes in LocalExecutor to start up the cluster only once.

Regards
Sachin

-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685

On Wed, Sep 2, 2015 at 9:33 PM, Till Rohrmann  wrote:

> Oh sorry, then I got the wrong context. I somehow thought it was about
> test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.
>
> On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel 
> wrote:
>
>> I was under the impression that the @AfterClass annotation can only be
>> used in test classes.
>> Even so, the idea is that a user program running in the IDE should not be
>> starting up the cluster several times [my primary concern is the addition
>> of the persist operator], and we certainly cannot ask the user to terminate
>> the cluster after execution, while in local mode.
>>
>> -- Sachin Goel
>> Computer Science, IIT Delhi
>> m. +91-9871457685
>>
>> On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann 
>> wrote:
>>
>>> Why is it not possible to shut down the local cluster? Can’t you shut it
>>> down in the @AfterClass method?
>>> ​
>>>
>>> On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel 
>>> wrote:
>>>
 Yes. That will work too. However, then it isn't possible to shut down
 the local cluster. [Is it necessary to do so or does it shut down
 automatically when the program exists? I'm not entirely sure.]

 -- Sachin Goel
 Computer Science, IIT Delhi
 m. +91-9871457685

 On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen  wrote:

> Have a look at some other tests, like the checkpointing tests. They
> start one cluster manually and keep it running. They connect against it
> using the remote environment ("localhost",
> miniCluster.getJobManagerRpcPort()).
>
> That works nicely...
>
> On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel 
> wrote:
>
>> Hi all
>> While using LocalEnvironment, in case the program triggers execution
>> several times, the {{LocalFlinkMiniCluster}} is started as many times. 
>> This
>> can consume a lot of time in setting up and tearing down the cluster.
>> Further, this hinders with a new functionality I'm working on based on
>> persisted results.
>> One potential solution could be to follow the methodology in
>> `MultipleProgramsTestBase`. The user code then would have to reside in a
>> fixed function name, instead of the main method. Or is that too 
>> cumbersome?
>>
>> Regards
>> Sachin
>> -- Sachin Goel
>> Computer Science, IIT Delhi
>> m. +91-9871457685
>>
>
>

>>>
>>
>

Re: Multiple restarts of Local Cluster

[Sachin Goel]

I'm not sure what you mean by "Crucial cleanup is in shutdown hooks". Could
you elaborate?

-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685

On Wed, Sep 2, 2015 at 10:25 PM, Stephan Ewen  wrote:

> You can always shut down a cluster manually (via shutdown()) and if the
> JVM simply exists, all is well as well. Crucial cleanup is in shutdown
> hooks.
>
> On Wed, Sep 2, 2015 at 6:22 PM, Till Rohrmann 
> wrote:
>
>> If I'm not mistaken, then the cluster should be properly terminated when
>> it gets garbage collected. Thus, also when the main method exits.
>>
>> On Wed, Sep 2, 2015 at 6:14 PM, Sachin Goel 
>> wrote:
>>
>>> If I'm right, all Tests use either the MultipleProgramTestBase or
>>> JavaProgramTestBase​. Those shut down the cluster explicitly anyway.
>>> I will make sure if this is the case.
>>>
>>> Regards
>>> Sachin
>>>
>>> -- Sachin Goel
>>> Computer Science, IIT Delhi
>>> m. +91-9871457685
>>>
>>> On Wed, Sep 2, 2015 at 9:40 PM, Till Rohrmann 
>>> wrote:
>>>
 Maybe we can create a single PlanExecutor for the LocalEnvironment
 which is used when calling execute. This of course entails that we
 don’t call stop on the LocalCluster. For cases where the program exits
 after calling execute, this should be fine because all resources will then
 be released anyway. It might matter for the test execution where maven
 reuses the JVMs and where the LocalFlinkMiniCluster won’t be garbage
 collected right away. You could try it out and see what happens.

 Cheers,
 Till
 ​

 On Wed, Sep 2, 2015 at 6:03 PM, Till Rohrmann 
 wrote:

> Oh sorry, then I got the wrong context. I somehow thought it was about
> test cases because I read `MultipleProgramTestBase` etc. Sorry my bad.
>
> On Wed, Sep 2, 2015 at 6:00 PM, Sachin Goel 
> wrote:
>
>> I was under the impression that the @AfterClass annotation can only
>> be used in test classes.
>> Even so, the idea is that a user program running in the IDE should
>> not be starting up the cluster several times [my primary concern is the
>> addition of the persist operator], and we certainly cannot ask the user 
>> to
>> terminate the cluster after execution, while in local mode.
>>
>> -- Sachin Goel
>> Computer Science, IIT Delhi
>> m. +91-9871457685
>>
>> On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann 
>> wrote:
>>
>>> Why is it not possible to shut down the local cluster? Can’t you
>>> shut it down in the @AfterClass method?
>>> ​
>>>
>>> On Wed, Sep 2, 2015 at 4:56 PM, Sachin Goel <
>>> sachingoel0...@gmail.com> wrote:
>>>
 Yes. That will work too. However, then it isn't possible to shut
 down the local cluster. [Is it necessary to do so or does it shut down
 automatically when the program exists? I'm not entirely sure.]

 -- Sachin Goel
 Computer Science, IIT Delhi
 m. +91-9871457685

 On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen 
 wrote:

> Have a look at some other tests, like the checkpointing tests.
> They start one cluster manually and keep it running. They connect 
> against
> it using the remote environment ("localhost",
> miniCluster.getJobManagerRpcPort()).
>
> That works nicely...
>
> On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel <
> sachingoel0...@gmail.com> wrote:
>
>> Hi all
>> While using LocalEnvironment, in case the program triggers
>> execution several times, the {{LocalFlinkMiniCluster}} is started as 
>> many
>> times. This can consume a lot of time in setting up and tearing down 
>> the
>> cluster. Further, this hinders with a new functionality I'm working 
>> on
>> based on persisted results.
>> One potential solution could be to follow the methodology in
>> `MultipleProgramsTestBase`. The user code then would have to reside 
>> in a
>> fixed function name, instead of the main method. Or is that too 
>> cumbersome?
>>
>> Regards
>> Sachin
>> -- Sachin Goel
>> Computer Science, IIT Delhi
>> m. +91-9871457685
>>
>
>

>>>
>>
>

>>>
>>
>

Re: Multiple restarts of Local Cluster

[Sachin Goel]

Yes. That will work too. However, then it isn't possible to shut down the
local cluster. [Is it necessary to do so or does it shut down automatically
when the program exists? I'm not entirely sure.]

-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685

On Wed, Sep 2, 2015 at 7:59 PM, Stephan Ewen  wrote:

> Have a look at some other tests, like the checkpointing tests. They start
> one cluster manually and keep it running. They connect against it using the
> remote environment ("localhost", miniCluster.getJobManagerRpcPort()).
>
> That works nicely...
>
> On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel 
> wrote:
>
>> Hi all
>> While using LocalEnvironment, in case the program triggers execution
>> several times, the {{LocalFlinkMiniCluster}} is started as many times. This
>> can consume a lot of time in setting up and tearing down the cluster.
>> Further, this hinders with a new functionality I'm working on based on
>> persisted results.
>> One potential solution could be to follow the methodology in
>> `MultipleProgramsTestBase`. The user code then would have to reside in a
>> fixed function name, instead of the main method. Or is that too cumbersome?
>>
>> Regards
>> Sachin
>> -- Sachin Goel
>> Computer Science, IIT Delhi
>> m. +91-9871457685
>>
>
>

Re: Multiple restarts of Local Cluster

[Stephan Ewen]

Have a look at some other tests, like the checkpointing tests. They start
one cluster manually and keep it running. They connect against it using the
remote environment ("localhost", miniCluster.getJobManagerRpcPort()).

That works nicely...

On Wed, Sep 2, 2015 at 4:23 PM, Sachin Goel 
wrote:

> Hi all
> While using LocalEnvironment, in case the program triggers execution
> several times, the {{LocalFlinkMiniCluster}} is started as many times. This
> can consume a lot of time in setting up and tearing down the cluster.
> Further, this hinders with a new functionality I'm working on based on
> persisted results.
> One potential solution could be to follow the methodology in
> `MultipleProgramsTestBase`. The user code then would have to reside in a
> fixed function name, instead of the main method. Or is that too cumbersome?
>
> Regards
> Sachin
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>

Multiple restarts of Local Cluster

[Sachin Goel]

Hi all
While using LocalEnvironment, in case the program triggers execution
several times, the {{LocalFlinkMiniCluster}} is started as many times. This
can consume a lot of time in setting up and tearing down the cluster.
Further, this hinders with a new functionality I'm working on based on
persisted results.
One potential solution could be to follow the methodology in
`MultipleProgramsTestBase`. The user code then would have to reside in a
fixed function name, instead of the main method. Or is that too cumbersome?

Regards
Sachin
-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685