kunwp1 opened a new pull request, #8986: URL: https://github.com/apache/texera/pull/8986
### What changes were proposed in this PR? Problem. When a computing unit fails to start because the cluster is full, an admin cannot see from the app how much of the cluster is left (#6476), and the cluster capacity panel (#6482) has nothing to show. The backend only lists pods in the computing-unit pool namespace and never reads node capacity. Fix. Adds `GET /api/computing-unit/admin/cluster-capacity` (ADMIN only), which returns `total`, `allocated` and `available` for CPU (millicores), memory (bytes) and GPU (devices). `total` is the allocatable of every Ready, uncordoned node, and `allocated` is the requests of the pods running on them in every namespace, with init containers and runtime-class overhead counted the way kube-scheduler counts them. That is the figure `kubectl describe node` reports as "Allocated resources", so `available` bounds what a new unit can claim. Live usage from metrics-server is not used because it reads below what is committed and has no GPU metric. The pod list is narrowed on the API server to pods that have not finished, so a cluster that keeps many finished pods does not ship them all. On a deployment without Kubernetes computing units it answers 404, and a Kubernetes API failure comes back with the cluster's own message. Nodes are cluster-scoped, so the chart adds a ClusterRole that can only `list` `nodes` and `pods`, bound to the computing unit manager's service account. This is the one new permission: the manager can now list pods in every namespace, where it could previously see only its own pool. ``` Before: admin sees a unit stuck Pending -> kubectl describe nodes -> adds up the requests by hand After: admin sees a unit stuck Pending -> GET /computing-unit/admin/cluster-capacity -> available CPU, memory, GPU ``` Not included. The panel that shows it (#6482) and telling the owner on terminate (#6483). `available` is an upper bound, not a go or no-go: it sums across nodes, so capacity spread thinly over many nodes can still refuse a unit that needs one node's worth, and it ignores the pool's ResourceQuota, node placement (taints, node selectors, runtime-class scheduling), pod-level resources and native sidecars. The ClusterRole is unconditional, with no chart value to turn it off. ### Any related issues, documentation, discussions? Closes #6481. Part of #6476. Feeds the panel in #6482 and follows #8888, which added the page. ### How was this PR tested? Tests were added for the aggregation, the client wrapper, the guard and the endpoint. They cover node and pod filtering (cordoned, not Ready, no Ready condition, no status, unbound pods, finished pods), the effective-request rule (container sum, init containers, overhead, containers with no resources), decimal and fractional quantities, a custom GPU resource key, `available` floored at zero, the wrapper reading nodes and the unfinished pods of every namespace, the 404 on a non-Kubernetes deployment, a Kubernetes failure keeping its message, and the JSON field names the panel will read. The tests were written first and watched fail on a stub, apart from the JSON field-name test, which pins the response shape. Breaking the aggregation on purpose (finished pods counted, init containers summed instead of maxed, no floor at zero, a not-Ready node counted, a cordoned node counted, overhead dropped) turns at least one test red each time. ``` sbt "ComputingUnitManagingService/testOnly org.apache.texera.service.util.ClusterCapacitySpec org.apache.texera.service.util.KubernetesClientSpec org.apache.texera.service.util.ComputingUnitHelpersSpec org.apache.texera.service.resource.AdminComputingUnitResourceSpec" sbt "ComputingUnitManagingService/test" ``` The first command passes 107 tests and the second passes 289. `scalafmtCheckAll` and `scalafixAll --check` ran clean. The ClusterRole was checked by rendering its template alone with `helm template` against the chart's `values.yaml` (rendering the whole chart needs its subcharts downloaded) and asserting it grants exactly `list` on `nodes` and `pods` and is bound to the manager's service account, and `bin/k8s/tests/test_helm_values.sh` passes. Not exercised: a live Kubernetes cluster, since the local stack has none, so the Kubernetes calls are only checked against a stubbed client, and the endpoint over HTTP. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code, Claude Sonnet 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
