Hi Dave,
This shouldn't normally happen, most people using security groups with KVM have
been using Ubuntu and Adv zone with security groups. In fact, most people
should use Adv zone without SG groups now, as one can create shared network
with SG in such zones now. I would suggest that you at least re-deploy zone
with SG and try again, my notes are for Adv zone without security groups.
Can you also try this:
1.
Check if physdev is available:
lsmod | grep xt_physdev
1.
Enable it when missing:
modprobe xt_physdev
Also ensure bridge-based firewalling is enabled:
modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
Regards.
________________________________
From: Dave Lapointe <da...@island.net>
Sent: Tuesday, March 18, 2025 05:57
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: Looking for suggestion on network configuration - followup
Hi Rohit,
Thanks for the link. I'm running ACS 4.20 on Rocky 8. I did a "basic" setup
when first connecting to the CloudStack UI. Hardware is Dell PowerEdge 450
with 64GB RAM. I've tried Windows 10 and Debian Linux VM's.
I did some more checking and found that the firewall rules for the bridge are
being set in
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:default_network_rules
where it blocks all traffic between VM's and then fails to define VM specific
rules because a table doesn't exist. Here is the log from agent.log:
2025-03-17 07:26:58,267 - Executing command: default_network_rules
2025-03-17 07:26:58,267 - modprobe br_netfilter
2025-03-17 07:26:58,270 - sysctl -w net.bridge.bridge-nf-call-arptables=1
2025-03-17 07:26:58,273 - sysctl -w net.bridge.bridge-nf-call-iptables=1
2025-03-17 07:26:58,276 - sysctl -w net.bridge.bridge-nf-call-ip6tables=1
2025-03-17 07:26:58,278 - iptables-save |grep physdev-is-bridged |grep FORWARD
|grep BF |grep '\-o' | grep -w cloudbr0|awk '{print $9}' | head -1
2025-03-17 07:26:58,283 - iptables -L BF-cloudbr0
2025-03-17 07:26:58,286 - iptables -L BF-cloudbr0-OUT
2025-03-17 07:26:58,289 - iptables -L BF-cloudbr0-IN
2025-03-17 07:26:58,292 - ip6tables -L BF-cloudbr0
2025-03-17 07:26:58,295 - ip6tables -L BF-cloudbr0-OUT
2025-03-17 07:26:58,298 - ip6tables -L BF-cloudbr0-IN
2025-03-17 07:26:58,300 - bridge -o link show | awk '/master cloudbr0 / &&
!/^[0-9]+: vnet/ {print $2}' | head -1 | cut -d ':' -f1 | cut -d '@' -f1
2025-03-17 07:26:58,304 - iptables -n -L BF-cloudbr0 | awk
'/BF-cloudbr0(.*)references/ {gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,307 - iptables -n -L BF-cloudbr0-IN | awk
'/BF-cloudbr0-IN(.*)references/ {gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,311 - iptables -n -L BF-cloudbr0-OUT | awk
'/BF-cloudbr0-OUT(.*)references/ {gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,314 - ip6tables -n -L BF-cloudbr0 | awk
'/BF-cloudbr0(.*)references/ {gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,318 - iptables -I FORWARD -i cloudbr0 -j DROP
2025-03-17 07:26:58,320 - iptables -I FORWARD -o cloudbr0 -j DROP
2025-03-17 07:26:58,323 - iptables -I FORWARD -i cloudbr0 -m physdev
--physdev-is-bridged -j BF-cloudbr0
2025-03-17 07:26:58,326 - iptables -I FORWARD -o cloudbr0 -m physdev
--physdev-is-bridged -j BF-cloudbr0
2025-03-17 07:26:58,329 - iptables -A BF-cloudbr0 -m state --state
RELATED,ESTABLISHED -j ACCEPT
2025-03-17 07:26:58,332 - iptables -A BF-cloudbr0 -m physdev
--physdev-is-bridged --physdev-is-in -j BF-cloudbr0-IN
2025-03-17 07:26:58,335 - iptables -A BF-cloudbr0 -m physdev
--physdev-is-bridged --physdev-is-out -j BF-cloudbr0-OUT
2025-03-17 07:26:58,337 - iptables -A BF-cloudbr0 -m physdev
--physdev-is-bridged --physdev-out -j ACCEPT
2025-03-17 07:26:58,340 - Command exited non-zero: iptables -A BF-cloudbr0 -m
physdev --physdev-is-bridged --physdev-out -j ACCEPT
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
line 53, in execute
return check_output(cmd, shell=True).decode()
File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'iptables -A BF-cloudbr0 -m physdev
--physdev-is-bridged --physdev-out -j ACCEPT' returned non-zero exit status 2.
2025-03-17 07:26:58,341 - iptables -F BF-cloudbr0
2025-03-17 07:26:58,344 - ip6tables -F BF-cloudbr0
The exception that's thrown is caused by the error:
Bad argument `ACCEPT'
Just to confirm that the rules set by default_network_rules were causing the
issue after a VM was started I tried adding:
iptables -I FORWARD -i cloudbr0 -j ACCEPT
iptables -I FORWARD -i cloudbr0 -j ACCEPT
and communications between VM's starting working, including DHCP. I also tried
setting rules in Security Group but it appears that they're not added to
iptables because the VM specific table doesn't exist.
I was hoping that the file at
/usr/share/cloudstack-common/scripts/vm/network/security_group.py was actually
the one that was being invoked but it's not... I guess it's a resource in the
agent package. If it was then I thought I could kind of "bludgeon" a solution
by adding:
iptables -I FORWARD -i cloudbr0 -j ACCEPT
iptables -I FORWARD -i cloudbr0 -j ACCEPT
in default_network_rules.
In your notes on installing CloudStack the suggestion of setting:
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
in sysctl.conf doesn't work because default_network_rules always re-enables
them.
So I think something isn't working right with CloudStack. It appears to be
ignoring security group rules, and at least on of the iptables rules appear to
have invalid syntax. If it was possible to modify security_group.py then I
could at least make it work. But default_network_rules is called quite often,
like whenever a VM is started so I can't think of anyway to add firewall rules
that won't be replaced by the rules from default_network_rules.
Thanks,
Dave
On 03-16-25 12:35 a.m., Rohit Yadav wrote:
> Hi Dave,
>
> Can you share your OS and other env details? For ACS 4.20 you don't have to
> seed the systemvmtemplate manually.
>
> My notes on installing CloudStack are here -
> https://rohityadav.cloud/blog/cloudstack-kvm/ if you're trying this on Ubuntu.
>
> Regards.
>
>
>
> ________________________________
> From: Dave Lapointe <da...@island.net.INVALID>
> Sent: Sunday, March 16, 2025 06:12
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Looking for suggestion on network configuration - followup
>
> Hi,
>
> I decided to go back to the basic CloudStack installation as described in the
> Quick Installation Guide, and followed every instruction there (there is an
> error with
> /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
> -m /export/secondary -u
> http://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.0.0-x86_64-kvm.qcow2.bz2
> -h kvm -F. The SystemVMtemplate should be
> systemvmtemplate-4.20.0-x86_64-kvm.qcow2.bz2).
>
> After completing the install and observing that there were no errors in
> management-server.log and agent.log I assumed everything was setup correctly.
> Once the two system VM's started up I ssh'ed into them, (they are bridged
> through cloudbr0 and cloud0) and found that I couldn't ping on the subnet for
> cloudbr0 but could ping on the cloud0 subnet. I deleted all of the firewall
> rules and set the default policies to ACCEPT but it made no difference. The
> routing tables appeared to be okay, with traffic for the cloudbr0 subnet
> being routed through the appropriate NIC.
>
> On the host I tried tcpdump -i cloudbr0 -w cloudbr0.cap and Wireshark showed
> that the ICMP packets are arriving on cloudbr0 but there is no response. I
> tried tcpdump on one of the system VM's and confirmed that there is a
> response to pinging the cloudbr0 ip address, and that there was no response
> from the ping target.
>
> I've done lots of bridging in different environments, and haven't encountered
> anything like this before, so I'm at a loss to know how to continue
> investigating this.
>
> I've spent a few days on this now and have made no progress. Please, any
> suggestions? I'm close to the point now of abandoning CloudStack because I
> can't get it to work as documented.
>
> Thanks,
> Dave
>
