gmautner opened a new issue, #14159:
URL: https://github.com/apache/cloudstack/issues/14159
### Problem
On CloudStack 4.22.1.1, a fresh CKS cluster with three separate etcd nodes
can be created and scaled from one to two workers successfully, but scaling
back to one worker fails with API error 530 while updating SSH network rules.
The backend has already removed the worker when it fails. It revokes the
first etcd SSH firewall rule (50000), leaves the control/worker firewall range
(2222–2224) active, and then conflicts when creating the replacement range
(2222–2223). It also removes the control/worker SSH forwards and the first etcd
SSH forward. The cluster becomes Alert with a stale worker count.
This controlled reproduction uses **three sequential native API mutations on
one fresh cluster**, with Kubernetes **1.33.13 throughout**. Creation and
scale-up were allowed to finish and pass node/pod/API readiness checks before
proceeding. There was no Kubernetes upgrade, concurrent lifecycle operation, UI
involvement, autoscaling, manual firewall customization, or per-node offering
override.
### Versions / environment
- CloudStack management server and KVM agent: **4.22.1.1**. Installed
package verification reports only local configuration differences; backend
binaries are unchanged.
- Advanced zone, isolated network, CloudStack virtual router, shared NFS
primary storage.
- SystemVM Template (KVM), Debian 12 guest OS; uniform offering: 2 vCPU, 4
GiB RAM, fixed 80 GiB root disk for all node types.
- Kubernetes 1.33.13, Calico 3.31.5. The ISO is custom-built with official
component releases, includes the native etcd 3.5.24 binaries needed for
separate-etcd bootstrap, and includes Headlamp and an upgrade-readiness hook.
No upgrade was performed in this reproduction.
- `cloud.kubernetes.etcd.node.start.port=50000` (default).
- One control node, three separate etcd VMs, initially one worker. Normal
tenant API credentials.
### Exact API calls / reproduction
The following are the complete application parameters for the three
mutations, submitted via HTTP POST to `/client/api`. Standard `response=json`
and API-key/signature authentication are omitted here. UUIDs identify this
lab's zone, registered 1.33.13 ISO, offering and cluster; substitute equivalent
IDs in another environment. No existing network was supplied: CloudStack
created the isolated network and its rules automatically.
```json
{
"command": "createKubernetesCluster",
"name": "cks-etcd-minimal-20260912",
"zoneid": "5ebb765f-3f58-4574-acde-4df589dd478e",
"kubernetesversionid": "759912e1-1fd1-48c9-9d1c-da3de44834cc",
"serviceofferingid": "db6694af-7532-414d-b5f2-0300d95656f4",
"noderootdisksize": 80,
"clustertype": "CloudManaged",
"hypervisor": "kvm",
"controlnodes": 1,
"etcdnodes": 3,
"size": 1
}
```
```json
{
"command": "scaleKubernetesCluster",
"id": "3b74274b-6554-40c2-b2cf-4329d4308544",
"size": 2
}
```
```json
{
"command": "scaleKubernetesCluster",
"id": "3b74274b-6554-40c2-b2cf-4329d4308544",
"size": 1
}
```
Wait for each async job before proceeding:
| Operation | Job ID | Result / completion (UTC−03) |
|---|---|---|
| Create, size=1 | `821341c9-4fdd-45a5-ab0c-7838790880fb` | jobstatus=1,
2026-09-12 00:27:59; health passed |
| Scale, size=2 | `927ddb07-b482-4b16-a568-926249c79bec` | jobstatus=1,
00:29:52; health passed |
| Scale, size=1 | `b860f7d3-cd52-4396-b9ce-b3ee03eb4d5d` | jobstatus=2,
error 530, 00:30:08 |
`queryAsyncJobResult(jobid=...)` was polled every five seconds. Read-only
`listKubernetesClusters`, `getKubernetesClusterConfig`,
`listPublicIpAddresses`, `listFirewallRules` and `listPortForwardingRules`
calls captured state between steps; kubectl checked nodes, pods and `/readyz`.
No fourth cluster mutation was submitted after the failure. The cluster remains
available for investigation.
### Expected result
Scale-down succeeds, the cluster reports one worker and Running, and SSH
forwarding for the remaining control/worker and all three etcd VMs is preserved.
### Actual results and relevant logs
The complete nested error states that 2222–2223 conflicts with the
still-active 2222–2224 firewall rule. These are verbatim relevant
management-server log lines from the failed job (unrelated lines omitted):
```text
2026-09-12 00:29:57,354 INFO [c.c.k.c.a.KubernetesClusterScaleWorker]
(API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3)
Removing vm VM instance
{"id":519,"instanceName":"i-39-519-VM","state":"Running","type":"User","uuid":"939ddd32-c36f-461b-8771-c3a55d51647f"}
from cluster KubernetesCluster
{"id":79,"name":"cks-etcd-minimal-20260912","uuid":"3b74274b-6554-40c2-b2cf-4329d4308544"}
2026-09-12 00:30:07,904 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker]
(API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3)
The SSH firewall rule null with the id 1600 was revoked
2026-09-12 00:30:07,905 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker]
(API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3)
The Port forwarding rule [null] with the id [1601] was mark as revoked.
2026-09-12 00:30:07,907 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker]
(API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3)
The Port forwarding rule [null] with the id [1607] was mark as revoked.
2026-09-12 00:30:07,907 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker]
(API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3)
The Port forwarding rule [null] with the id [1608] was mark as revoked.
2026-09-12 00:30:08,761 DEBUG [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f, ctx-0f06110e])
(logid:b860f7d3) Rules ([FirewallRule
{"id":1606,"networkId":281,"purpose":"Firewall","state":"Active","uuid":"9439cbc3-faeb-4b6a-b7a6-66fdd167343b"}]
and [FirewallRule
{"id":1610,"networkId":281,"purpose":"Firewall","state":"Staged","uuid":"d2e0ba0c-931d-45a8-a093-770dc73e636e"}])
have conflicting port ranges.
2026-09-12 00:30:08,768 ERROR [c.c.k.c.a.KubernetesClusterScaleWorker]
(API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3)
Scaling failed for Kubernetes cluster KubernetesCluster
{"id":79,"name":"cks-etcd-minimal-20260912","uuid":"3b74274b-6554-40c2-b2cf-4329d4308544"},
unable to update network rules com.cloud.exception.ManagementServerException:
Failed to provision firewall rules for SSH access for the Kubernetes cluster :
cks-etcd-minimal-20260912
Caused by: com.cloud.exception.NetworkRuleConflictException: The range
specified, 2222-2223, conflicts with rule FirewallRule
{"id":1606,"networkId":281,"purpose":"Firewall","state":"Active","uuid":"9439cbc3-faeb-4b6a-b7a6-66fdd167343b"}
which has 2222-2224
at
com.cloud.network.firewall.FirewallManagerImpl.detectRulesConflict(FirewallManagerImpl.java:486)
at
com.cloud.network.firewall.FirewallManagerImpl.lambda$createFirewallRule$0(FirewallManagerImpl.java:265)
at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
at
com.cloud.network.firewall.FirewallManagerImpl.createFirewallRule(FirewallManagerImpl.java:255)
```
Rule identities reconstructed from the rule-application commands and API
snapshots:
| Rule | Purpose / ports | Observed change |
|---|---|---|
| 1595 | Initial control/worker firewall, 2222–2223 | Correctly revoked
during scale-up |
| 1600 | First etcd firewall, 50000 | Incorrectly revoked during scale-down |
| 1602 / 1604 | Other etcd firewalls, 50001 / 50002 | Remain active |
| 1606 | Replacement control/worker firewall, 2222–2224 | Created during
scale-up; incorrectly left active during scale-down |
| 1601 | First etcd port forward, 50000→22 | Removed during scale-down |
| 1607 / 1608 | Control / retained-worker port forwards, 2222→22 / 2223→22 |
Removed during scale-down |
| 1610 | Attempted replacement firewall, 2222–2223 | Rejected due to overlap
with 1606 |
After failure, `listPortForwardingRules` returns only 50001→22 and 50002→22.
`listFirewallRules` returns 2222–2224, 6443, 50001 and 50002. CloudStack
reports Alert and `size=2`, while only one worker VM remains. The five
remaining VMs are Running. kubectl shows the control and one worker Ready on
**v1.33.13**, and `/readyz` still returns `ok`.
### Source analysis / suggested correction
In [4.22.1.1
`removeSshFirewallRule`](https://github.com/apache/cloudstack/blob/4.22.1.1/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorker.java#L546-L558),
the selector accepts either a firewall starting at 2222 **or any corresponding
port forward whose destination starts at 22**, then breaks after the first
match:
```java
if (Objects.equals(firewallRule.getSourcePortStart(),
CLUSTER_NODES_DEFAULT_START_SSH_PORT)
|| (Objects.nonNull(pfRule) && pfRule.getDestinationPortStart() ==
DEFAULT_SSH_PORT)) {
rule = firewallRule;
firewallService.revokeIngressFwRule(firewallRule.getId(), true);
break;
}
```
The first etcd SSH rule also satisfies that condition. The preceding
scale-up recreated the control/worker rule after the etcd rules; the next scale
selected etcd rule 1600 first.
[`scaleKubernetesClusterIsolatedNetworkRules`](https://github.com/apache/cloudstack/blob/4.22.1.1/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java#L134-L155)
then uses the selected rule's end port as the upper bound for port-forward
cleanup, yielding **2222–50000**. This explains the removal of both
regular-node SSH forwards and the first etcd forward. Recreating 2222–2223 then
conflicts with the untouched general rule.
Please distinguish control/worker SSH rules from separate-etcd rules during
selection and bound the forwarding cleanup accordingly. A regression test
covering fresh three-etcd creation → worker scale 1→2→1, including rule
ordering after recreation, should catch this. The partial VM removal and stale
cluster count on a network-rule failure also merit recovery handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
