dongjoon-hyun opened a new pull request, #764: URL: https://github.com/apache/spark-kubernetes-operator/pull/764
### What changes were proposed in this pull request? This PR aims to remove the operator-side `NetworkPolicy` creation from `SparkAppResourceSpec` in favor of Apache Spark 4.2.0's built-in `NetworkPolicyFeatureStep`. - Remove `buildNetworkPolicy` and its invocation from `SparkAppResourceSpec`. - Remove the corresponding `testNetworkPolicy` test case and adjust the pre-resource count assertion. - Keep the `networkpolicies` RBAC rule in the Helm chart because the operator still needs it to create the `NetworkPolicy` generated by Spark's `NetworkPolicyFeatureStep`. ### Why are the changes needed? The operator has created a `NetworkPolicy` per `SparkApplication` since SPARK-55085. After SPARK-58194 upgraded the Spark dependency to 4.2.0, Spark's `KubernetesDriverBuilder` also generates a functionally identical `NetworkPolicy` via `NetworkPolicyFeatureStep` (SPARK-55653). As a result, every `SparkApplication` ends up with two duplicated policies. ``` $ kubectl get networkpolicy NAME POD-SELECTOR AGE pi-0-network-policy spark-app-selector=pi-0,spark-role=executor 41s pi-0-policy spark-app-selector=pi-0,spark-role=executor 40s ``` Both policies have the same pod selector and ingress rule (allowing traffic only from the pods of the same Spark application). Delegating to Spark's `NetworkPolicyFeatureStep` removes the duplication and avoids maintaining the same logic in two places. ### Does this PR introduce _any_ user-facing change? Yes, only one `NetworkPolicy` (`<appId>-policy`, created by Spark) is deployed per `SparkApplication` instead of two. The security behavior is unchanged because the two policies were functionally identical. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
