Hi Rohit, Thanks for the suggestions.
Before I saw your reply here to use Advanced Zones, I tried Basic Zone because my CloudStack configuration requirements are so simple (only needing one server). And it turns out that a Basic Zone without SG works as expected... the VM's can communicate with each other. However, for the sake of being consistent with typical deployments, I'll try Advanced Zones now. Unfortunately, I won't have physical access to the server for at least another week so I can't install Ubuntu. I confirmed that physdev is available and bridge-nf-call-iptables is 1. I think that the Basic Zone with SG isn't working as it should, since the default firewall rules created by CloudStack prevent all communication across cloudbr0, and the SG rules appear to be ignored. But maybe it's not so important since, as you said, most people use Advanced Zones and what I've been trying is a bit of an outlier case. Thanks, Dave On 03-20-25 10:07 p.m., Rohit Yadav wrote:
Hi Dave, This shouldn't normally happen, most people using security groups with KVM have been using Ubuntu and Adv zone with security groups. In fact, most people should use Adv zone without SG groups now, as one can create shared network with SG in such zones now. I would suggest that you at least re-deploy zone with SG and try again, my notes are for Adv zone without security groups. Can you also try this: 1. Check if physdev is available: lsmod | grep xt_physdev 1. Enable it when missing: modprobe xt_physdev Also ensure bridge-based firewalling is enabled: modprobe br_netfilter echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables Regards.________________________________ From: Dave Lapointe <da...@island.net> Sent: Tuesday, March 18, 2025 05:57 To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Looking for suggestion on network configuration - followup Hi Rohit, Thanks for the link. I'm running ACS 4.20 on Rocky 8. I did a "basic" setup when first connecting to the CloudStack UI. Hardware is Dell PowerEdge 450 with 64GB RAM. I've tried Windows 10 and Debian Linux VM's. I did some more checking and found that the firewall rules for the bridge are being set in /usr/share/cloudstack-common/scripts/vm/network/security_group.py:default_network_rules where it blocks all traffic between VM's and then fails to define VM specific rules because a table doesn't exist. Here is the log from agent.log: 2025-03-17 07:26:58,267 - Executing command: default_network_rules 2025-03-17 07:26:58,267 - modprobe br_netfilter 2025-03-17 07:26:58,270 - sysctl -w net.bridge.bridge-nf-call-arptables=1 2025-03-17 07:26:58,273 - sysctl -w net.bridge.bridge-nf-call-iptables=1 2025-03-17 07:26:58,276 - sysctl -w net.bridge.bridge-nf-call-ip6tables=1 2025-03-17 07:26:58,278 - iptables-save |grep physdev-is-bridged |grep FORWARD |grep BF |grep '\-o' | grep -w cloudbr0|awk '{print $9}' | head -1 2025-03-17 07:26:58,283 - iptables -L BF-cloudbr0 2025-03-17 07:26:58,286 - iptables -L BF-cloudbr0-OUT 2025-03-17 07:26:58,289 - iptables -L BF-cloudbr0-IN 2025-03-17 07:26:58,292 - ip6tables -L BF-cloudbr0 2025-03-17 07:26:58,295 - ip6tables -L BF-cloudbr0-OUT 2025-03-17 07:26:58,298 - ip6tables -L BF-cloudbr0-IN 2025-03-17 07:26:58,300 - bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1 | cut -d ':' -f1 | cut -d '@' -f1 2025-03-17 07:26:58,304 - iptables -n -L BF-cloudbr0 | awk '/BF-cloudbr0(.*)references/ {gsub(/\(/, "") ;print $3}' 2025-03-17 07:26:58,307 - iptables -n -L BF-cloudbr0-IN | awk '/BF-cloudbr0-IN(.*)references/ {gsub(/\(/, "") ;print $3}' 2025-03-17 07:26:58,311 - iptables -n -L BF-cloudbr0-OUT | awk '/BF-cloudbr0-OUT(.*)references/ {gsub(/\(/, "") ;print $3}' 2025-03-17 07:26:58,314 - ip6tables -n -L BF-cloudbr0 | awk '/BF-cloudbr0(.*)references/ {gsub(/\(/, "") ;print $3}' 2025-03-17 07:26:58,318 - iptables -I FORWARD -i cloudbr0 -j DROP 2025-03-17 07:26:58,320 - iptables -I FORWARD -o cloudbr0 -j DROP 2025-03-17 07:26:58,323 - iptables -I FORWARD -i cloudbr0 -m physdev --physdev-is-bridged -j BF-cloudbr0 2025-03-17 07:26:58,326 - iptables -I FORWARD -o cloudbr0 -m physdev --physdev-is-bridged -j BF-cloudbr0 2025-03-17 07:26:58,329 - iptables -A BF-cloudbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 2025-03-17 07:26:58,332 - iptables -A BF-cloudbr0 -m physdev --physdev-is-bridged --physdev-is-in -j BF-cloudbr0-IN 2025-03-17 07:26:58,335 - iptables -A BF-cloudbr0 -m physdev --physdev-is-bridged --physdev-is-out -j BF-cloudbr0-OUT 2025-03-17 07:26:58,337 - iptables -A BF-cloudbr0 -m physdev --physdev-is-bridged --physdev-out -j ACCEPT 2025-03-17 07:26:58,340 - Command exited non-zero: iptables -A BF-cloudbr0 -m physdev --physdev-is-bridged --physdev-out -j ACCEPT Traceback (most recent call last): File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 53, in execute return check_output(cmd, shell=True).decode() File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output **kwargs).stdout File "/usr/lib64/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command 'iptables -A BF-cloudbr0 -m physdev --physdev-is-bridged --physdev-out -j ACCEPT' returned non-zero exit status 2. 2025-03-17 07:26:58,341 - iptables -F BF-cloudbr0 2025-03-17 07:26:58,344 - ip6tables -F BF-cloudbr0 The exception that's thrown is caused by the error: Bad argument `ACCEPT' Just to confirm that the rules set by default_network_rules were causing the issue after a VM was started I tried adding: iptables -I FORWARD -i cloudbr0 -j ACCEPT iptables -I FORWARD -i cloudbr0 -j ACCEPT and communications between VM's starting working, including DHCP. I also tried setting rules in Security Group but it appears that they're not added to iptables because the VM specific table doesn't exist. I was hoping that the file at /usr/share/cloudstack-common/scripts/vm/network/security_group.py was actually the one that was being invoked but it's not... I guess it's a resource in the agent package. If it was then I thought I could kind of "bludgeon" a solution by adding: iptables -I FORWARD -i cloudbr0 -j ACCEPT iptables -I FORWARD -i cloudbr0 -j ACCEPT in default_network_rules. In your notes on installing CloudStack the suggestion of setting: net.bridge.bridge-nf-call-arptables = 0 net.bridge.bridge-nf-call-iptables = 0 in sysctl.conf doesn't work because default_network_rules always re-enables them. So I think something isn't working right with CloudStack. It appears to be ignoring security group rules, and at least on of the iptables rules appear to have invalid syntax. If it was possible to modify security_group.py then I could at least make it work. But default_network_rules is called quite often, like whenever a VM is started so I can't think of anyway to add firewall rules that won't be replaced by the rules from default_network_rules. Thanks, Dave On 03-16-25 12:35 a.m., Rohit Yadav wrote:Hi Dave, Can you share your OS and other env details? For ACS 4.20 you don't have to seed the systemvmtemplate manually. My notes on installing CloudStack are here - https://rohityadav.cloud/blog/cloudstack-kvm/ if you're trying this on Ubuntu. Regards. ________________________________ From: Dave Lapointe <da...@island.net.INVALID> Sent: Sunday, March 16, 2025 06:12 To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Looking for suggestion on network configuration - followup Hi, I decided to go back to the basic CloudStack installation as described in the Quick Installation Guide, and followed every instruction there (there is an error with /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /export/secondary -u http://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.0.0-x86_64-kvm.qcow2.bz2 -h kvm -F. The SystemVMtemplate should be systemvmtemplate-4.20.0-x86_64-kvm.qcow2.bz2). After completing the install and observing that there were no errors in management-server.log and agent.log I assumed everything was setup correctly. Once the two system VM's started up I ssh'ed into them, (they are bridged through cloudbr0 and cloud0) and found that I couldn't ping on the subnet for cloudbr0 but could ping on the cloud0 subnet. I deleted all of the firewall rules and set the default policies to ACCEPT but it made no difference. The routing tables appeared to be okay, with traffic for the cloudbr0 subnet being routed through the appropriate NIC. On the host I tried tcpdump -i cloudbr0 -w cloudbr0.cap and Wireshark showed that the ICMP packets are arriving on cloudbr0 but there is no response. I tried tcpdump on one of the system VM's and confirmed that there is a response to pinging the cloudbr0 ip address, and that there was no response from the ping target. I've done lots of bridging in different environments, and haven't encountered anything like this before, so I'm at a loss to know how to continue investigating this. I've spent a few days on this now and have made no progress. Please, any suggestions? I'm close to the point now of abandoning CloudStack because I can't get it to work as documented. Thanks, Dave
