[
https://issues.apache.org/jira/browse/FLINK-41005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Lane updated FLINK-41005:
------------------------------
Description:
If MiniCluster start() fails part way, the catch block calls close() to
cleanup, but {{running}} is only set at the end of a successful start, and
closeInternal doesn't do anything unless {{running}} is true. Anything started
before the failure will be left running.
{*}Impact:{*}{*}{*}
prod: processes that exit after a failed start (e.g. the MiniCluster
application entrypoint from FLINK-40860) leave behind files in java.io.tmpdir
tests: non-daemon threads for RPC and metrics Pekko actor systems, daemon
threads for blob server listener, IOManager, mini-cluster-io executor and
timers - all of which stay in the surefire fork, plus the REST port stays bound
*Proposed fix:*
In the catch block of {{{}start(){}}}, run the shutdown sequence regardless of
{{running}} (e.g. extract the body of {{closeInternal}} into a method both
use), making each step tolerate services that were never created (null fields,
empty TaskManager list), and wait for it before rethrowing.
*Testing:*
fail {{start()}} at a few points (before and after TaskManagers / dispatcher
components are started) and assert that no services are left running
was:If MiniCluster start() fails part way, the catch block calls close() to
cleanup, but {{running}} is only set at the end of a successful start, and
closeInternal doesn't do anything unless {{running}} is true. Anything started
before the failure will be left running.
> MiniCluster should clean up partially started services when start() fails
> -------------------------------------------------------------------------
>
> Key: FLINK-41005
> URL: https://issues.apache.org/jira/browse/FLINK-41005
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Reporter: Dale Lane
> Priority: Minor
>
> If MiniCluster start() fails part way, the catch block calls close() to
> cleanup, but {{running}} is only set at the end of a successful start, and
> closeInternal doesn't do anything unless {{running}} is true. Anything
> started before the failure will be left running.
> {*}Impact:{*}{*}{*}
> prod: processes that exit after a failed start (e.g. the MiniCluster
> application entrypoint from FLINK-40860) leave behind files in java.io.tmpdir
> tests: non-daemon threads for RPC and metrics Pekko actor systems, daemon
> threads for blob server listener, IOManager, mini-cluster-io executor and
> timers - all of which stay in the surefire fork, plus the REST port stays
> bound
> *Proposed fix:*
> In the catch block of {{{}start(){}}}, run the shutdown sequence regardless
> of {{running}} (e.g. extract the body of {{closeInternal}} into a method both
> use), making each step tolerate services that were never created (null
> fields, empty TaskManager list), and wait for it before rethrowing.
> *Testing:*
> fail {{start()}} at a few points (before and after TaskManagers / dispatcher
> components are started) and assert that no services are left running
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
