dongjoon-hyun opened a new pull request, #841: URL: https://github.com/apache/spark-kubernetes-operator/pull/841
### What changes were proposed in this pull request? This PR aims to fill in the missing CPU and memory requests of the `master` and `worker` `PodSet`s when `KueueWorkloadFactory` builds a Kueue `Workload` for a `SparkCluster`. A missing request is filled in as follows. Requests that are already in the pod template are kept, and a missing request defaults to the limit, like Kubernetes. | PodSet | CPU | Memory | |---|---|---| | `master` | `1` | `SPARK_DAEMON_MEMORY` (default `1g`) + overhead | | `worker` | `SPARK_WORKER_CORES` (default `1`) | `SPARK_DAEMON_MEMORY` (default `1g`) + `SPARK_WORKER_MEMORY` (if set) + overhead | - No new configuration is introduced. The values come from the same environment variables that Spark standalone reads (`SparkClassCommandBuilder` and `WorkerArguments`). Only literal `value`s of the container `env` are used. `valueFrom` is ignored. - Like Spark, a memory string without a unit is in bytes. - The overhead is `max(0.10 * memory, 384MiB)`, the default rule of the `SparkApplication` pod sets. - Spark standalone has no setting for the number of master cores, so the minimum value `1` is used unless the pod template specifies it. ### Why are the changes needed? Unlike the driver and executor pods of `SparkApplication`, Spark does not set any requests on the master and worker pods. So far, `KueueWorkloadFactory` copied the `StatefulSet` pod templates into the `PodSet`s as they were. As a result, a `SparkCluster` without `resources.requests` in its templates produced a `Workload` with zero usage, and Kueue admitted it without accounting it against the `ClusterQueue` quota. ### Does this PR introduce _any_ user-facing change? No. The Kueue integration is not released yet. ### How was this patch tested? Pass the CIs with the newly added and updated test cases in `KueueWorkloadFactoryTest`. - `testBuildWorkloadForSparkCluster` (updated) - `testBuildWorkloadForSparkClusterFillsMissingRequests` - `testBuildWorkloadForSparkClusterUsesLimitsAsMissingRequests` - `testCalculateDaemonMemoryMiB` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
