rmetzger commented on code in PR #29379: URL: https://github.com/apache/flink/pull/29379#discussion_r4261969131
########## docs/content/docs/deployment/resource-providers/standalone/overview.md: ########## @@ -123,6 +123,58 @@ $ ./bin/taskmanager.sh stop $ ./bin/standalone-job.sh stop ``` +### Application Mode in a MiniCluster + +An application can also be run with the JobManager and a TaskManager in a single process, using the `bin/minicluster.sh` script. +This is useful for low-throughput jobs where running a separate TaskManager is not worth the additional resources. +The process runs the same components as a distributed application cluster, so savepoints, checkpoints, state backends, metrics and the web interface all work in the same way. + +The script takes the same arguments as `bin/standalone-job.sh`, except that it always runs in the foreground (so there is no `start` or `stop`) and arguments for the application should be put after `--`: + +```bash +$ ./bin/minicluster.sh --jars local://$(pwd)/examples/streaming/TopSpeedWindowing.jar +``` + +The web interface is now available at [localhost:8081](http://localhost:8081). No TaskManager needs to be started. + +The full set of arguments is: + +```bash +$ ./bin/minicluster.sh \ + [--jars <jar>[,<jar>...]] \ + [--job-classname <class name>] \ + [--job-id <job id>] \ + [--fromSavepoint <path> [--allowNonRestoredState]] \ + [-D <key>=<value> ...] \ + [-- <application arguments>] +``` + +As with `standalone-job.sh`, if `--jars` is not given, the application is loaded from the `usrlib` folder or the classpath. + +The MiniCluster has one TaskManager with `taskmanager.numberOfTaskSlots` slots, so the parallelism of the application cannot be greater than this. +JobManager and TaskManager process memory options (such as `jobmanager.memory.process.size` and `taskmanager.memory.process.size`) do not apply, and the JVM heap size is set by the JVM defaults unless set using `env.java.opts.jobmanager` or `env.java.opts.all`. + +#### Restarting the process + +If [High Availability]({{< ref "docs/deployment/ha/overview" >}}) is configured, the process can be restarted after a failure and will resume the application from the latest checkpoint. +The application is given a fixed job ID derived from the `high-availability.cluster-id` (or from `--job-id`, if given), so a restarted process recovers the existing job from the high availability metadata rather than starting a new one. +Stopping the process with `SIGTERM` keeps the high availability metadata, so that it can be recovered in the same way. + +Task failures are handled by the application's restart strategy within the process, in the same way as in a distributed cluster. +A fatal error in the JobManager or the TaskManager cannot be recovered within the process, because there is no other process to take over. +Examples are an `OutOfMemoryError`, or a task that does not stop within [`task.cancellation.timeout`]({{< ref "docs/deployment/config" >}}#task-cancellation-timeout). Review Comment: > or a task that does not stop within [`task.cancellation.timeout`] I haven't verified but I thought there's a config flag that controls whether the flink process kills itself in that case or not? or is it always killing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
