dongjoon-hyun opened a new pull request, #55639: URL: https://github.com/apache/spark/pull/55639
### What changes were proposed in this pull request? This PR makes `ExecutorResizeDriverPlugin` reuse the `KubernetesClient` owned by `KubernetesClusterSchedulerBackend` instead of creating its own client via `SparkKubernetesClientFactory.createKubernetesClient(...)`. This is a follow-up to [SPARK-56684](https://issues.apache.org/jira/browse/SPARK-56684), which exposed `KubernetesClusterSchedulerBackend.kubernetesClient` as `private[k8s] val` for exactly this purpose. Key changes in `ExecutorResizePlugin.scala`: - Drop the `private var kubernetesClient: KubernetesClient` field and the `SparkKubernetesClientFactory.createKubernetesClient(...)` call inside `init()`. - In the periodic task, look up the backend lazily via `sparkContext.schedulerBackend match { case backend: KubernetesClusterSchedulerBackend => ... }` (same pattern as `ExecutorRollPlugin`) and pass `backend.kubernetesClient` into `checkAndIncreaseMemory(...)`. Lazy resolution is required because `_plugins` is initialized before `_schedulerBackend` in `SparkContext`. - Remove `kubernetesClient.close()` from `shutdown()`; the client lifecycle is now owned by the backend's `start()` / `stop()`. Test changes in `ExecutorResizePluginSuite.scala`: - Stub `sparkContext.schedulerBackend` to return a mocked `KubernetesClusterSchedulerBackend` whose `kubernetesClient` returns the existing mock client. - Drop the reflection that injected the `kubernetesClient` field (the field no longer exists). - Pass the mock client as the new `kubernetesClient` argument to `checkAndIncreaseMemory`. ### Why are the changes needed? Before this PR, the driver process held two `KubernetesClient` instances pointing at the same API server with the same credentials: one owned by `KubernetesClusterSchedulerBackend` and one created by `ExecutorResizeDriverPlugin`. SPARK-56684 deliberately exposed the backend's client so plugins could share it; this PR realizes that benefit for `ExecutorResizePlugin`. As a side effect, the suite no longer needs reflection to inject the client mock, so the test setup matches normal usage more closely. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs with the updated unit tests. - `build/sbt -Pkubernetes 'kubernetes/testOnly *ExecutorResizePluginSuite'` — 11/11 passed. - `build/sbt -Pkubernetes 'kubernetes/testOnly *KubernetesClusterSchedulerBackendSuite *ExecutorRollPluginSuite'` — 39/39 passed (regression check). - `build/sbt -Pkubernetes 'kubernetes/scalastyle' 'kubernetes/Test/scalastyle'` — clean. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
