FrankYang0529 commented on code in PR #75538:
URL: https://github.com/apache/airflow/pull/75538#discussion_r4572740855


##########
java-sdk/processor/src/main/kotlin/org/apache/airflow/sdk/AirflowProcessor.kt:
##########
@@ -854,28 +927,67 @@ class AirflowProcessor : AbstractProcessor() {
       }
     }
 
+    val owner = ClassName.get(decl.owner)
+    val hook = decl.onKill
+    if (hook != null) {
+      // The instance is made here rather than in a field initializer. If the 
constructor throws, the task
+      // fails and its retries apply, as they do for a task without a hook.
+      executeSpec
+        .addStatement($$"this.instance = new $T()", owner)
+        // If the stop request came before this point, the hook had no 
instance to call, so the hook could
+        // not stop the task method.
+        .beginControlFlow("if (context.isCancelled())")
+        .addStatement($$"throw new $T($S)", IllegalStateException::class.java, 
"Airflow asked the task to stop before it started")
+        .endControlFlow()

Review Comment:
   The SDK calls `execute` right after `hold()`, so a check before binding 
would only catch a request that arrives after `hold()` returns and before 
binding starts. The existing `isCancelled()` check still notices a request that 
arrives during binding once binding finishes, so I left the generated code 
unchanged.



##########
java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Context.kt:
##########
@@ -189,10 +192,63 @@ data class Context(
   /** Registration of the executing task; resolves wired data inputs. */
   internal var taskDef: TaskDef? = null
 
+  // Only from() sets this field. A context made in any other way, including 
with copy(), keeps it null.
+  private var cancellation: Cancellation? = null
+
+  /**
+   * Whether Airflow has asked this task to stop.
+   *
+   * Airflow asks a running task to stop when, for example, a user marks its 
task
+   * instance as failed. The supervisor, which is the Airflow process that 
started
+   * the JVM, then sends the JVM SIGTERM. If the JVM is still running after a
+   * grace period, the supervisor sends SIGKILL.
+   *
+   * Only a [StoppableTask] can call this method, because the SDK catches the 
stop
+   * signal only for that type of task. For any other task, the JVM handles the
+   * signal as it would without the SDK: by default the JVM runs its shutdown 
hooks
+   * and exits. Such a task could never see a stop request, so this method 
throws
+   * instead of returning false.
+   *
+   * The context that the SDK passes to the task keeps seeing the stop request
+   * after the task returns. A context made in any other way never sees a stop
+   * request, and this method returns false for it. That includes a context 
that
+   * a unit test makes to call a task directly, and a context made with 
`copy()`.
+   *
+   * @throws IllegalStateException if the running task is not a 
[StoppableTask].
+   */
+  fun isCancelled(): Boolean = cancellation?.check() ?: false

Review Comment:
   `TaskCannotSeeStopRequestException` extends `IllegalStateException`, and the 
task runner reports it as a failure without a retry.
   
   I'd keep `copy()` as documented: a fallback needs a JVM-wide current 
`Cancellation` that every hand-built `Context` reads, so a hand-built context 
in a plain task would throw instead of returning false.



##########
java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/execution/Cancellation.kt:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.airflow.sdk.execution
+
+import org.apache.airflow.sdk.Context
+import org.apache.airflow.sdk.StoppableTask
+import sun.misc.Signal
+import sun.misc.SignalHandler
+import java.time.Duration
+import java.util.concurrent.CompletableFuture
+import java.util.concurrent.CountDownLatch
+import java.util.concurrent.TimeUnit
+import java.util.concurrent.atomic.AtomicBoolean
+import java.util.concurrent.atomic.AtomicReference
+
+/**
+ * Records whether Airflow has asked the running task to stop. For a 
[StoppableTask], also catches the
+ * signal that carries that request.
+ */
+internal class Cancellation : AutoCloseable {
+  private val claimed = AtomicBoolean()
+  private val requested = CountDownLatch(1)
+  private val signalHandled = CompletableFuture<Unit>()
+
+  @Volatile
+  private var held = false
+
+  @Volatile
+  private var trap: AutoCloseable? = null
+
+  @Volatile
+  private var hook: (() -> Unit)? = null
+
+  fun check(): Boolean {
+    throwIfTaskCannotSeeRequest()
+    return requested.count == 0L
+  }
+
+  fun await(timeout: Duration): Boolean {
+    throwIfTaskCannotSeeRequest()
+    return requested.await(TimeUnit.NANOSECONDS.convert(timeout), 
TimeUnit.NANOSECONDS)
+  }
+
+  private fun throwIfTaskCannotSeeRequest() {
+    check(held) {
+      "Only a StoppableTask can see a stop request. Implement StoppableTask, 
or give an annotated task an @Airflow.OnKill method"
+    }
+  }
+
+  /** Catches the first stop signal for [task] until [close], or throws 
[CannotCatchStopSignalsException]. */
+  fun hold(
+    task: StoppableTask,
+    context: Context,
+  ) {
+    val classLoader = Thread.currentThread().contextClassLoader
+    hook = {
+      // On Java 11 and 17, a signal handler thread has no context class 
loader, and libraries can need one.
+      Thread.currentThread().contextClassLoader = classLoader
+      task.onKill(context)
+    }
+    // Set first: a signal can run onKill, which may check for a stop request, 
before trapStopSignals returns.
+    held = true
+    trap = trapStopSignals(::takeSignal)
+      ?: throw CannotCatchStopSignalsException(
+        "The SDK cannot catch SIGTERM or SIGINT in this JVM, so the task 
cannot see a stop request. The JVM might run with -Xrs, " +
+          "or without the jdk.unsupported module",
+      )
+  }
+
+  /**
+   * Stops later signals from calling onKill and waits for a call that already 
started.
+   * The wait has no timeout because the supervisor's SIGKILL ends a call that 
hangs.
+   */
+  fun finish() {
+    hook = null
+    // join() keeps waiting if the thread is interrupted, and a task can leave 
the thread's interrupt flag set.
+    if (claimed.get()) signalHandled.join()
+  }
+
+  override fun close() {
+    trap?.close()
+  }
+
+  private fun takeSignal(signal: String): Boolean {
+    // A second signal means the sender does not want to wait.
+    if (!claimed.compareAndSet(false, true)) {
+      logger.info("Letting the JVM handle the signal", mapOf("signal" to 
signal))
+      return false
+    }
+    return try {
+      logger.warning("Asking the task to stop", mapOf("signal" to signal))

Review Comment:
   Updated it. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to