calvix opened a new pull request, #14294:
URL: https://github.com/apache/cloudstack/pull/14294
### Description
With redundant routers, the lease of an expunged VM never gets released from
dnsmasq. If a new VM later gets the same IP, the primary router refuses to give
it out and the VM comes up without an address.
On a redundant router the guest NIC has the router's own IP as primary and
the gateway (VIP) as secondary, and dnsmasq only listens on the VIP
(`listen-address=127.0.0.1,<VIP>`), so the VIP is what it uses as its DHCP
server identifier. On expunge `CsDhcp.py` calls `dhcp_release`, which puts the
first address of the interface into option 54 - the router IP, not the VIP.
dnsmasq ignores a DHCPRELEASE that isn't addressed to its own server
identifier, without logging anything, so the lease just stays in memory.
The fallback from #13194 doesn't help either. It writes a new leases file
over the old one and reloads dnsmasq, but dnsmasq only reads that file on
startup, so the lease is still there, and from then on dnsmasq keeps writing to
the deleted file.
Non-redundant routers aren't affected, there dnsmasq listens on the router
IP, which is what `dhcp_release` sends.
The fixed IP in the steps below is only there to make it reproducible. In
normal use the IP is picked at random, and a freed IP is a normal candidate, so
it happens whenever a new VM happens to land on the IP of an expunged one.
Leases are infinite, so these stuck ones stay until dnsmasq is restarted.
## How to reproduce
Create an isolated network offering with redundant routers and a network
from it:
```
cmk create networkoffering name=redundant displaytext=redundant
guestiptype=Isolated traffictype=GUEST \
supportedservices=Dhcp,Dns,SourceNat \
"serviceproviderlist[0].service=Dhcp"
"serviceproviderlist[0].provider=VirtualRouter" \
"serviceproviderlist[1].service=Dns"
"serviceproviderlist[1].provider=VirtualRouter" \
"serviceproviderlist[2].service=SourceNat"
"serviceproviderlist[2].provider=VirtualRouter" \
"servicecapabilitylist[0].service=SourceNat"
"servicecapabilitylist[0].capabilitytype=RedundantRouter"
"servicecapabilitylist[0].capabilityvalue=true" \
"servicecapabilitylist[1].service=SourceNat"
"servicecapabilitylist[1].capabilitytype=SupportedSourceNatTypes"
"servicecapabilitylist[1].capabilityvalue=peraccount"
cmk update networkoffering id=<offering> state=Enabled
cmk create network name=rnet displaytext=rnet zoneid=<zone>
networkofferingid=<offering>
```
Deploy a VM with a fixed IP and expunge it:
```
cmk deploy virtualmachine zoneid=<zone> serviceofferingid=<so>
networkids=<network> templateid=<template 1> ipaddress=10.1.1.71
cmk destroy virtualmachine id=<A> expunge=true
```
Then deploy another one on the same IP:
```
cmk deploy virtualmachine zoneid=<zone> serviceofferingid=<so>
networkids=<network> templateid=<template 2> ipaddress=10.1.1.71
```
One catch: dnsmasq looks leases up by client identifier first, so if the
second VM sends the same client id as the first one, it just takes over the old
lease and you won't see the problem. Clones of a template with a baked-in
`/etc/machine-id` do exactly that. If your templates regenerate the machine-id
the same template is fine, otherwise use a different one for the second VM.
The second VM gets no IP, and `/var/log/dnsmasq.log` on the primary router
has:
```
dnsmasq-dhcp[3606]: not using configured address 10.1.1.71 because it is
leased to 02:01:00:ce:00:04
dnsmasq-dhcp[3606]: DHCPDISCOVER(eth0) 02:01:00:ce:00:05 no address available
```
There's no DHCPRELEASE line for the first VM anywhere in the log. `ls -l
/proc/$(pidof dnsmasq)/fd` shows `/var/lib/misc/dnsmasq.leases (deleted)`, and
reading that fd still shows the old lease.
You can also see the release being dropped directly on the primary router
with `dhcp_release eth0 <ip> <mac>` for any VM - the lease stays and nothing
shows up in the log.
## What the fix does
- The DHCPRELEASE is now sent with the address dnsmasq actually listens on
in the VM's network, taken from `listen-address` in `cloud.conf` (the VIP on a
redundant router, the router IP otherwise). `CsHelper.send_dhcp_release()`
builds the same packet as `dhcp_release` and sends it the same way, just with
the right server identifier. Guests still see the VIP as the DHCP server, so
#11877 / #13720 stay fixed.
- Before falling back, it waits up to 2 seconds for dnsmasq to drop the
lease from the file. Before, the check ran right after sending.
- The #13194 fallback now removes the line in place instead of replacing the
file, and does `systemctl try-restart dnsmasq` instead of a reload. On the
backup router dnsmasq isn't running, so `try-restart` doesn't do anything there.
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [X] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [X] Major
- [ ] Minor
- [ ] Trivial
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
