Venkat-C-J commented on issue #2822:
URL:
https://github.com/apache/apisix-ingress-controller/issues/2822#issuecomment-5191636505
Thanks for the detailed investigation — you were right, and my original
premise was wrong. APISIX does inherit `hosts` from the service. Digging
further, I found the actual trigger, and it also explains why your reproduction
passed.
**TL;DR:** the regression only appears when a hostname contains an
**uppercase character**. APISIX normalizes host case for route-level hosts, but
not for service-level hosts.
## Root cause
`apisix/router.lua` lowercases hosts, but only on the **route** object:
```lua
-- apisix/router.lua filter() -- applied to routes
if route.value.host then
route.value.host = str_lower(route.value.host)
elseif route.value.hosts then
for i, v in ipairs(route.value.hosts) do
route.value.hosts[i] = str_lower(v)
end
end
```
`apisix/http/service.lua` `filter()` has no host handling at all —
`service.value.hosts` is stored verbatim:
```lua
local function filter(service)
service.has_domain = false
if not service.value then return end
plugin.set_plugins_meta_parent(service.value.plugins, service)
apisix_upstream.filter_upstream(service.value.upstream, service)
end
```
Matching runs against `api_ctx.var.host` (nginx `$host`), which is **always
lowercase**, and `radixtree_host_uri` buckets routes by the reversed host
string, which is case-sensitive:
```lua
for i, host in ipairs(hosts) do
local host_rev = host:reverse()
...
end
```
So a host such as `MixedCase.example.com` arriving via `service.hosts`
produces a bucket keyed on `moc.elpmaxe.esaCdexiM`, which no request can ever
reach. The route lands in an orphaned bucket, `only_uri_router` is consulted
next, and if nothing host-agnostic exists the request returns 404.
## Why 2.0.1 worked and 2.1.0 does not
In 2.0.1, `buildRoute` also set the hosts on the route:
```go
// internal/adc/translator/apisixroute.go
route.Hosts = rule.Match.Hosts // removed in 2.1.0 (PR #2743)
service.Hosts = rule.Match.Hosts // unchanged, still present in
buildService
```
That assignment meant the value passed through `router.lua`'s `str_lower`
normalization, so casing never mattered. After #2743 the constraint travels
only on the service object, where nothing lowercases it.
Neither ADC nor the ingress controller normalizes case (verified by grep),
and the APISIX schema permits uppercase:
```lua
local host_def_pat = "^\\*$|^\\*?[0-9a-zA-Z-._\\[\\]:]+$"
```
so the value passes validation, syncs without error, reports healthy — and
is silently unroutable.
## Why the earlier reproduction passed
The repro used `tenant-a.example.com` / `tenant-b.example.com` — all
lowercase. With lowercase hosts the service-level fallback is byte-identical to
the route-level value and behaves correctly, exactly as observed. The defect is
invisible unless a host contains an uppercase character.
## Minimal reproduction
```yaml
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: route-mixed
spec:
http:
- name: rule-mixed
match:
hosts: ["MixedCase.example.com"]
paths: ["/*"]
backends:
- serviceName: backend-a
servicePort: 80
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: route-lower
spec:
http:
- name: rule-lower
match:
hosts: ["lowercase.example.com"]
paths: ["/*"]
backends:
- serviceName: backend-b
servicePort: 80
```
```bash
curl -H 'Host: MixedCase.example.com' http://<gateway>/ # 2.1.0 -> 404,
2.0.1 -> 200
curl -H 'Host: lowercase.example.com' http://<gateway>/ # 200 on both
```
`GET /apisix/admin/routes` shows no `hosts` on the route under 2.1.0, and
`GET /apisix/admin/services` shows the host with its original casing preserved.
The failure is a blanket 404 across every path and priority for that host —
the signature of an orphaned host bucket, not a route-priority conflict.
## Version matrix
| APISIX | ADC | Ingress Controller | Result |
| --- | --- | --- | --- |
| 3.15 | 0.26 | 2.0.0 | works |
| 3.16 | 0.27 | 2.0.1 | works |
| 3.17 | 0.28 | 2.0.1 | works |
| 3.16 | 0.27 | 2.1.0 | fails |
| 3.17 | 0.28 | 2.1.0 | fails |
The controller version is the only variable that changes the outcome, with
identical CRDs in every run. The normalization asymmetry itself predates 3.16;
2.0.1 simply masked it.
## Suggested fix
**1. Ingress controller** — restore the previous behaviour:
```go
// internal/adc/translator/apisixroute.go, buildRoute()
route.Hosts = rule.Match.Hosts
```
If the motivation behind #2743 was ADC diff noise from duplicated host
values, that may be better solved in the diff/normalization layer, since
route-level hosts are what currently receives APISIX's case normalization.
**2. APISIX** — mirror `router.lua`'s normalization in
`apisix/http/service.lua` so both paths are equivalent:
```lua
local function filter(service)
service.has_domain = false
if not service.value then return end
if service.value.hosts then
for i, v in ipairs(service.value.hosts) do
service.value.hosts[i] = str_lower(v)
end
end
plugin.set_plugins_meta_parent(service.value.plugins, service)
apisix_upstream.filter_upstream(service.value.upstream, service)
end
```
Normalizing in the translator or in ADC would also work, but doing it in
APISIX covers every client, not just the controller.
Happy to open a PR for either or both — just let me know which you would
prefer.
## Questions
1. Do you agree that removing route-level hosts changed observable behaviour
here, given that `route.hosts` and `service.hosts` are not normalized
identically?
2. Should hosts on the service object be lowercased in APISIX to match the
route path? Today an uppercase host passes schema validation and syncs cleanly,
but is unroutable when it appears only on a service — a silent failure mode
that is difficult to diagnose.
For anyone hitting this before a fix lands: lowercasing all `match.hosts`
values resolves it immediately, since hostnames are case-insensitive (RFC 1035
§2.3.3, RFC 9110 §4.2) and nginx `$host` is lowercase anyway. That is what we
did on our side, and all previously failing tests now pass on 2.1.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
