GitHub user mxm opened a pull request: https://github.com/apache/flink/pull/858
[FLINK-2097] Implement job session management This is a joint effort by @StephanEwen and me to introduce a session management in Flink. Session are used to keep a copy of the ExecutionGraph in the job manager for the session lifetime. It is important that the ExecutionGraph is not kept around longer because it consumes a lot of memory. Its intermediate results can also be freed. To integrate sessions properly into Flink, some refactoring was necessary. In particular these are: - JobId is created through the ExecutionEnvironment and passed through - Sessions can be termined by the ExecutionEnvironment or directly through the executor - Session are cancelled implicitly through "reapers" or shutdown hooks in the ExecutionEnvironment, otherwise they time out - LocalExecutor and RemoteExecutor manage sessions - The Client only deals with the communication with the job manager and is agnostic of session management With the session management, we will be able to properly support backtracking of produced intermediate results. This makes calls to count()/collect()/print() efficient and enables to write incremental/interactive jobs. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mxm/flink session-dev Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/858.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #858 ---- commit 9852d392bfe69b056596acfba001ab0a574f0ac0 Author: Maximilian Michels <m...@apache.org> Date: 2015-05-13T15:06:47Z [FLINK-2097] [core] Implement job session management. Sessions make sure that the JobManager does not immediately discard a JobGraph after execution, but keeps it around for further operations to be attached to the graph. That is the basis if interactive sessions. This pull request implements a rudimentary session management. Together with the backtracking #640, this will enable users to submit jobs to the cluster and access intermediate results. Session handling ensures that the results are cleared eventually. ExecutionGraphs are kept as long as - no timeout occurred or - the session has not been explicitly ended commit 65464ad19d39a29d41d071b2a4524b414e297147 Author: Stephan Ewen <se...@apache.org> Date: 2015-05-29T12:35:33Z [FLINK-2097] [core] Improve session management. - The Client manages only connections to the JobManager, it is not job specific - Executors provide a more explicit life cycle and methods to start new sessions - Sessions are handled by the environments - The environments use reapers (local) and shutdown hooks (remote) to ensure session termination when the environment runs out of scope commit 6d89edd4a63fa3971c0246f46c7b8c98f3fc6c30 Author: Maximilian Michels <m...@apache.org> Date: 2015-06-18T14:38:09Z [FLINK-2097] [core] Finalize session management ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---