FrankYang0529 commented on code in PR #75538:
URL: https://github.com/apache/airflow/pull/75538#discussion_r4572742346
##########
java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Server.kt:
##########
@@ -210,7 +233,14 @@ class CoordinatorServer(
aSocket(selector).tcp().connect(logs).use { socket ->
logger.debug("Connected logs", mapOf("addr" to logs))
LogSender.configure(socket.openWriteChannel(autoFlush = true))
- deferral.await()
+ val hook = Thread({ sendQueuedLogs(socket::close,
socket.socketContext, logFlushTimeout) }, "Airflow log flush")
+ Runtime.getRuntime().addShutdownHook(hook)
+ try {
+ deferral.await()
+ } finally {
+ // The use block closes the socket even when serving fails, so
flush while it is still open.
+ LogSender.detachAndFlush()
Review Comment:
Thanks for the suggestion. The finally now calls `sendQueuedLogs` and then
removes the shutdown hook.
##########
airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst:
##########
@@ -748,6 +748,117 @@ the defaults of the Dag's params, and the conf of a
scheduled run is empty. For
accepts and the cases where it throws, see ``DagRun.getConf`` in the `Java SDK
API reference
<https://airflow.apache.org/docs/java-sdk/stable/>`__.
+.. _java-sdk/cancellation:
+
+Stopping when Airflow asks
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Airflow asks a running task to stop when, for example, a user marks its task
instance as failed. The
+supervisor, the Airflow process that started the task's JVM, then sends the
JVM ``SIGTERM``. If the JVM
+is still running after a grace period, the supervisor sends ``SIGKILL``. When
a user marks the task
+instance failed or success, or clears it, the grace period is
+:ref:`core.killed_task_cleanup_time <config:core__killed_task_cleanup_time>`
seconds.
+
+By default the JVM handles the signal as it does without the SDK: shutdown
hooks run and the JVM exits.
+A process the task started gets the same signals, because the supervisor
signals the JVM's whole process
+group. So a task does not need to stop a process that exits on ``SIGTERM`` and
stays in that group.
+
+To stop anything else, such as a job in an external system, implement
``StoppableTask``. This interface
+adds ``onKill``, the Java counterpart of an operator's ``on_kill``:
+
+.. code-block:: java
+
+ public class ReportTask implements StoppableTask {
+ private final ReportService service = new ReportService();
+ private volatile String jobId;
+
+ @Override
+ public void execute(Context context, Client client) throws Exception {
+ jobId = service.submit();
+ if (context.isCancelled()) {
+ service.cancel(jobId);
+ throw new IllegalStateException("Airflow asked the task to stop");
+ }
+ service.waitFor(jobId); // Throws if the job fails or is cancelled.
+ }
+
+ @Override
+ public void onKill(Context context) throws Exception {
+ if (jobId != null) {
+ service.cancel(jobId);
+ }
+ }
+ }
+
+With annotations, mark the method that stops the task with ``@Airflow.OnKill``
and name the task. The
+method can take a ``Context``, and the annotation processor generates a
``StoppableTask`` that calls it on
+the same object as the task method:
+
+.. code-block:: java
+
+ private final ReportService service = new ReportService();
+ private volatile String jobId;
+
+ @Airflow.Task(id = "build_report", retries = 2)
+ public void buildReport(Context context) throws Exception {
+ jobId = service.submit();
+ if (context.isCancelled()) {
+ service.cancel(jobId);
+ throw new IllegalStateException("Airflow asked the task to stop");
+ }
+ service.waitFor(jobId); // Throws if the job fails or is cancelled.
+ }
+
+ @Airflow.OnKill(task = "build_report")
+ public void stopReport() throws Exception {
+ if (jobId != null) {
+ service.cancel(jobId);
+ }
+ }
+
+Once the SDK has started such a task, it catches the first ``SIGTERM`` or
``SIGINT`` and the JVM keeps
+running. ``Context.isCancelled()`` then returns true and the SDK calls
``onKill``:
+
+* ``onKill`` runs once, on its own daemon thread, while the task keeps
running. So ``onKill`` can stop a task
Review Comment:
Removed the `onKill` bullet list, the paragraph about which signals go to
the JVM, and the paragraph about `isCancelled`/`awaitCancellation` from
java.rst, which now links to `StoppableTask` in the API reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]