GitHub user RyanP-Testing added a comment to the discussion: Cloudstack
multiple management server connection across different networks
Hi Arpan,
Thanks for sharing the detailed configuration and logs. We went through the
issue carefully, and based on the behavior you described, we don't believe the
fact that the two Management Servers are on different subnets is, by itself,
the problem. CloudStack can run multiple Management Servers against the same
CloudStack database, even when the servers are on different routed networks.
The important point here is to identify what changes when MGMT-02 moves from
the same subnet to `172.16.18.0/24`.
The `DefaultConfigResources` exception is occurring during CloudStack's
Spring/module initialization, so we would not immediately treat it as a direct
MySQL connectivity error. It may be a secondary error caused by an earlier
configuration or initialization problem.
We recommend checking the following in order:
**1. Verify both Management Servers are using the same CloudStack database**
MGMT-02 should be pointing to the existing CloudStack database used by MGMT-01.
Please make sure the database was not initialized separately on MGMT-02.
Also, please review the `db.properties` configuration on both servers and
compare the database-related entries.
**2. Check the Management Server encryption key**
You mentioned that `/etc/cloudstack/management/key` is different on the two
Management Servers. This is something we would definitely investigate.
Please compare the keys on both servers:
```bash
sha256sum /etc/cloudstack/management/key
```
For security, please don't share the actual key publicly. If the hashes are
different, we recommend correcting the configuration so both Management Servers
use the appropriate shared key for the same CloudStack deployment.
**3. Verify connectivity from MGMT-02 to the database**
>From the `172.16.18.x` Management Server, test:
```bash
nc -vz 172.16.17.x 3306
```
If that succeeds, also verify actual MySQL authentication:
```bash
mysql -h 172.16.17.x -u cloud -p
```
Changing MySQL's `bind-address` to `0.0.0.0` only makes MySQL listen on the
available interfaces; the MySQL user grants and firewall rules must also allow
the remote Management Server.
On MGMT-01, you can verify that MySQL is listening with:
```bash
ss -lntp | grep 3306
```
**4. Verify routing and firewall rules between the two networks**
The change to:
```text
172.16.0.0/16
```
is fine from a CIDR perspective because it covers both `172.16.17.0/24` and
`172.16.18.0/24`.
However, the Management CIDR setting itself does not create routing between the
networks.
Please make sure there is an actual route and that the required traffic is
permitted in both directions between:
```text
172.16.17.0/24
↕
Router / Firewall
↕
172.16.18.0/24
```
**5. Use the same-subnet test as a controlled comparison**
Since you already confirmed that two Management Servers work when both are on
`192.168.1.0/24`, that is actually a very useful test result.
If possible, perform one more controlled test with both Management Servers
temporarily placed on the same `172.16.17.0/24` network while using the same
database and configuration.
If that works, and moving MGMT-02 back to `172.16.18.0/24` causes the failure
again, we can confidently focus on the inter-subnet routing, firewall, MySQL
remote access, or related network configuration rather than CloudStack's basic
multi-Management-Server functionality.
**6. Check the first error in the Management Server log**
The `DefaultConfigResources` exception may be the final result of an earlier
failure.
Please check the Management Server log immediately before this exception and
look for the first occurrence of:
```text
ERROR
Exception
MySQL
database
connection
encryption
key
```
That first error will usually tell us much more about the actual root cause.
### What we would check first
Based on the information currently available, our priority would be:
1. Confirm both Management Servers use the same CloudStack database.
2. Verify the Management Server/encryption keys are consistent.
3. Confirm MGMT-02 can reach MySQL on TCP/3306.
4. Verify MySQL grants and firewall rules.
5. Verify routing between `172.16.17.0/24` and `172.16.18.0/24`.
6. Compare the relevant CloudStack configuration between both nodes.
7. Then investigate the `DefaultConfigResources` exception further if the above
checks are clean.
We would avoid rebuilding the environment or making further configuration
changes until these checks are completed. The fact that the same setup works
when both Management Servers are on the same subnet gives us a good baseline,
and we can use that to isolate the exact point of failure rather than changing
multiple variables at once.
If you can share the **sanitized `db.properties` from both Management
Servers**, the **SHA-256 hashes of the management keys**, and the result of the
TCP/3306 connectivity test from MGMT-02, we can narrow this down considerably.
Regards,
**DevOps Engineer & Technical Support Engineer **
**AccuWeb.Cloud**
GitHub link:
https://github.com/apache/cloudstack/discussions/13919#discussioncomment-18084159
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
