GitHub user MitchDrage closed a discussion: Password server on redundant VRs
binds only the VRRP VIP, so guests never receive their password
### problem
SUMMARY
On a redundant VR, the password server is started with both the VRRP VIP and
the router's own guest IP, but passwd_server_ip.py binds only the first of the
two. Guests derive the password server address from the DHCP server-identifier,
which is the router's own guest IP, so their requests to port 8080 are answered
with a TCP RST. Passwords accumulate uncollected in
/var/cache/cloud/passwords-<vip> and password injection silently fails for
every guest on every isolated network with redundant routers.
This is not to be confused with #12161, which changes the DNS list for
redundant VPC VRs only and does not affect this path.
CAUSE
I believe this was introduced in #10183, which made the VR's own guest IP
unconditional in dnsmasq's listen-address (fixing #10182). That deliberately
changed the behaviour established by #3477, where redundant VRs listened on the
VIP only. dnsmasq now hands out the VR's own IP as the DHCP server-identifier,
but the password server is still started as <VIP>,<VR IP> and binds
addresses[0], the VIP, so guests query an address nothing listens on.
In the environment below, the guest network is 192.168.0.0/24, the VRRP VIP
(gateway) is 192.168.0.1, and the primary router's own guest IP is 192.168.0.48.
EXPECTED RESULTS
The guest retrieves its password from the address it was given as the DHCP
server-identifier, and the entry is removed from the password cache.
ACTUAL RESULTS
The password server listens on the VIP only:
root@r-3050-VM:~# ss -lntp | grep 8080
LISTEN 0 5 192.168.0.1:8080 0.0.0.0:* users:(("python",pid=3372,fd=3))
Passwords are delivered by the management server and never collected:
root@r-3050-VM:~# cat /var/cache/cloud/passwords-192.168.0.1
192.168.0.82=<redacted>
192.168.0.188=<redacted>
192.168.0.38=<redacted>
Metadata on port 80 works for every guest, because cloudstack binds all
addresses. Both agents complete a full metadata crawl and only the password
request fails. Note the 61 second stall in the cloudbase-init crawl, which is
the password request timing out:
```
192.168.0.82 - - [.../Jul/2026:03:30:36 +0000] "GET
/latest/meta-data/instance-id HTTP/1.1" 200 291 "-" "python-requests/2.33.1"
192.168.0.82 - - [.../Jul/2026:03:31:37 +0000] "GET
/latest/meta-data/service-offering HTTP/1.1" 200 302 "-"
"python-requests/2.33.1"
192.168.0.38 - - [.../Jul/2026:04:55:56 +0000] "GET
/latest/meta-data/instance-id HTTP/1.1" 200 291 "-"
"Cloud-Init/26.1-0ubuntu1~22.04.1"
192.168.0.38 - - [.../Jul/2026:04:55:56 +0000] "GET /latest/user-data HTTP/1.1"
200 351 "-" "Cloud-Init/26.1-0ubuntu1~22.04.1"
```
The guest requests port 8080 at the router's own guest IP (192.168.0.48), not
at the VIP, and is reset because nothing is bound there:
```
05:03:16.482340 IP 192.168.0.38.43828 > 192.168.0.48.8080: Flags [S], seq
2121724, length 0
05:03:16.482361 IP 192.168.0.48.8080 > 192.168.0.38.43828: Flags [R.], seq 0,
ack 2121725, win 0, length 0
05:03:17.484909 IP 192.168.0.38.43838 > 192.168.0.48.8080: Flags [S], seq
2008568524, length 0
05:03:17.484944 IP 192.168.0.48.8080 > 192.168.0.38.43838: Flags [R.], seq 0,
ack 2008568525, win 0, length 0
```
This is not a firewall problem. The rules are present and counting the SYNs
above as accepted:
root@r-3050-VM:~# iptables-save | grep -E 'dport (80|8080)'
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -s 192.168.0.0/24 -i eth0 -p tcp -m tcp --dport 80 -m state --state
NEW -j ACCEPT
-A INPUT -s 192.168.0.0/24 -i eth0 -p tcp -m tcp --dport 8080 -m state --state
NEW -j ACCEPT
17 204 10640 ACCEPT 6 -- eth0 * 192.168.0.0/24 0.0.0.0/0 tcp dpt:8080
state NEW
Confirmed with a DNAT rule on the primary router, after which every pending
password was collected immediately:
iptables -t nat -I PREROUTING -i eth0 -d 192.168.0.48/32 -p tcp --dport 8080 \
-j DNAT --to-destination 192.168.0.1:8080
### versions
ACS: 4.22.1.0
systemvm template: systemvm-kvm-4.22.0-x86_64
Hypervisor: KVM.
Guests tested: Windows Server 2022 (cloudbase-init) and Ubuntu 22.04
(cloud-init 26.1). Same guest templates work correctly on a shared network
served by a single, non-redundant VR.
CONFIGURATION
Advanced zone, KVM, isolated guest networks using a network offering with
redundant routers enabled. UserData/password service provided by VirtualRouter.
### The steps to reproduce the bug
1. Create an isolated network using an offering with redundant VRs enabled and
UserData provided by VirtualRouter.
2. Deploy a guest from any template with a password-enabled image
(cloudbase-init or cloud-init), or reset the password of an existing guest.
3. Observe the guest never receives its password.
### What to do about it?
passwd_server_ip.py: start a listener for every address supplied rather than
only addresses[0]. allowAddresses already contains both, and savepassword.sh
already POSTs to every local IP, so no other change is needed. Keeping
addresses[0] as listeningAddress preserves the existing
/var/cache/cloud/passwords-<vip> path. This is what I am running locally using
custom userdata, verified across save, fetch and acknowledge.
Something like this??
FROM:
(Starting at line 182 of passwd_server_ip.py)
```
global listeningAddress
global allowAddresses
if len(sys.argv) > 1:
addresses = sys.argv[1].split(",")
if len(addresses) > 0:
listeningAddress = addresses[0]
allowAddresses.append(addresses[0])
if len(addresses) > 1:
allowAddresses.append(addresses[1])
server_address = (listeningAddress, 8080)
passwordServer = ServerClass(server_address, HandlerClass)
passwordServer.allow_reuse_address = True
sa = passwordServer.socket.getsockname()
initToken()
loadPasswordFile()
syslog.syslog('serve_password running on %s:%s' % (sa[0], sa[1]))
try:
passwordServer.serve_forever()
```
TO:
```
global listeningAddress
global allowAddresses
addresses = [listeningAddress]
if len(sys.argv) > 1:
addresses = [a.strip() for a in sys.argv[1].split(",") if a.strip()]
listeningAddress = addresses[0] # Need to keep listeningAddress as-is
on the first address as it's global. Used in getPasswordFile()
for addr in addresses:
if addr not in allowAddresses:
allowAddresses.append(addr)
initToken()
loadPasswordFile()
servers = []
for addr in addresses:
server = ServerClass((addr, 8080), HandlerClass)
server.allow_reuse_address = True
servers.append(server)
syslog.syslog('serve_password running on %s:8080' % addr)
for server in servers[1:]: # Add each subsequent server as a thread.
Primary server starts in the try block.
t = threading.Thread(target=server.serve_forever)
t.daemon = True
t.start()
try:
servers[0].serve_forever()
```
GitHub link: https://github.com/apache/cloudstack/discussions/13732
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
