johannes-engler-mw opened a new issue, #2833:
URL: https://github.com/apache/apisix-ingress-controller/issues/2833
## Current Behavior
When an HTTPRoute's `parentRefs` are repointed from a Gateway managed by
controller A to a
Gateway managed by controller B (a different
`GatewayClass.spec.controllerName`), controller B
picks the route up correctly — but **controller A never removes the
configuration it had already
pushed**. Its data plane keeps serving the route indefinitely.
The chain, on `master` (`be4aaef`) and on the released `2.1.0`:
1. `httproute_controller.go:75` — the reconciler is registered with
`For(&gatewayv1.HTTPRoute{})` and no controller-name predicate, so
**every** controller
instance watches **every** HTTPRoute. Repointing `parentRefs` bumps the
generation, so
`GenerationChangedPredicate` passes and controller A does reconcile the
change.
2. `utils.go:357-359` — `ParseRouteParentRefs` skips parentRefs whose
GatewayClass belongs to
another controller:
```go
if string(gatewayClass.Spec.ControllerName) !=
config.ControllerConfig.ControllerName {
continue
}
```
For controller A the route's only remaining parentRef is now controller
B's Gateway, so the
returned slice is empty.
3. `httproute_controller.go:172-174` — the empty slice returns early with no
cleanup:
```go
if len(gateways) == 0 {
return ctrl.Result{}, nil
}
```
`Provider.Delete` is only reachable from the `IgnoreNotFound` branch at
:147, i.e. when the
object is deleted from the cluster. There is no path for "this object
used to be mine and no
longer is".
4. `adc/client/client.go:290,303` — the periodic sync replays the in-memory
store
(`ConfigManager.List()` → `GetResources(name)`); it never re-derives
state from Kubernetes.
The orphaned service therefore gets re-pushed to the data plane every
sync period, forever.
Restarting the controller pod clears it, because startup rebuilds the store
from the informer's
initial list and the route no longer produces an entry for controller A.
The same early return exists in all five Gateway API route reconcilers:
```
httproute_controller.go:172
grpcroute_controller.go:190
tcproute_controller.go:283
udproute_controller.go:283
tlsroute_controller.go:283
```
### Why this is hard to notice
Every Kubernetes-level signal is correct and green while it happens:
- the HTTPRoute's `status.parents` lists only controller B, with
`Accepted=True`
- controller B's Gateway shows the route in
`status.listeners[].attachedRoutes`
- controller A's Gateway shows the correct (reduced) `attachedRoutes` count
- controller A logs no errors — its sync loop reports success every period
- GitOps tooling reports the manifests as fully synced
The stale configuration is visible **only** in the old data plane's own
control API
(`:9090/v1/services`). In our case the orphaned routes carried public
hostnames, so this failed
*open*: hosts that were supposed to have been taken off the public gateway
stayed publicly
reachable for over 21 hours, with the label metadata and `modifiedIndex`
still showing values
from weeks before the change.
### Prior art
PR #2543 ("fix: residual data issue when updating ingressClassName", merged
2025-09-08) fixed
exactly this failure mode for Ingress, and its description states the
problem in general terms:
> the controller simply ignores the Ingress resource. However, this behavior
is problematic: the
> corresponding configuration is not removed and remains in the data plane,
which may result in
> incorrect routing or stale configurations.
The resulting code in `ingress_controller.go:159-166` calls
`Provider.Delete` when the
IngressClass no longer matches. The Gateway API route reconcilers were never
given the
equivalent.
## Expected Behavior
When a route no longer references any Gateway managed by this controller,
the controller removes
the configuration it previously pushed for that route, so the data plane
stops serving it —
matching the Ingress/IngressClass behaviour from #2543.
## Error Logs
None. That is a significant part of the problem: the controller logs no
error at any log level,
and the ADC sync reports success on every tick while re-pushing the orphaned
configuration.
## Steps to Reproduce
1. Install two APISIX ingress controller instances with distinct controller
names, e.g.
`apisix.apache.org/apisix-ingress-controller` and
`apisix.apache.org/apisix-ingress-controller-internal`, each with its own
GatewayClass and
Gateway (`gateway-a` in namespace `apisix`, `gateway-b` in namespace
`apisix-internal`).
2. Create an HTTPRoute with `parentRefs: [gateway-a]` and a hostname, and
confirm it serves:
```console
$ curl -H 'Host: demo.example.com' http://<gateway-a>/
200
```
3. Change **only** the `parentRefs` to `gateway-b` (keep the same
name/namespace) and apply.
4. Observe that controller B accepts the route and `gateway-b` serves it —
and that `gateway-a`
**still** serves it too:
```console
$ curl -H 'Host: demo.example.com' http://<gateway-a>/
200 # expected: 404
```
5. Confirm the orphan is in controller A's data plane, and that it is never
removed:
```console
$ kubectl port-forward -n apisix pod/<apisix-pod> 9090:9090
$ curl -s localhost:9090/v1/services | jq -r '.[].value |
"\(.labels["k8s/name"]) \(.hosts)"'
demo ["demo.example.com"] # still present, indefinitely
```
6. `kubectl rollout restart deploy/apisix-ingress-controller -n apisix` —
the entry disappears,
confirming the store is the only thing holding it.
The same reproduction applies to GRPCRoute, TCPRoute, UDPRoute and TLSRoute.
## Environment
- APISIX Ingress controller version: `2.1.0` (also reproduces on `master`,
`be4aaef`)
- ADC version: `0.26.0`
- Mode: standalone / ADC (`config_provider: yaml`)
- Kubernetes cluster version: `v1.34.9`
- Gateway API version: `v1.3.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
