Hi Hean,
I was able to deploy a VM in a shared network in an advanced zone with security
groups enabled and could see the iptables rules getting added for the VM on the
host, so it is probably not a bug that we are seeing. Seems like a setup issue.
Applying the rules can fail because:
* The hypervisor host doesn't have iptables / ebtables command - which is
probably unlikely
* The VM deployed doesn't have an interface - can be verified by dumping
its configuration - virsh dumpxml <domain_name> | grep interface
* or application of the default network rules for the VM itself failed -If
you are up for some debugging, can you try running the script manually in the
hypervisor host:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip <vm_ip>
--vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name> --nicsecips
0; --isFirstNic --check
when you do a virsh dumpxml <vm_domain_name>, look out for an interface section
that looks something like:
<interface type='bridge'>
<mac address='1e:00:86:00:00:ba'/>
<source bridge='breth1-11'/>
<bandwidth>
<inbound average='25600' peak='25600'/>
<outbound average='25600' peak='25600'/>
</bandwidth>
<target dev='vnet0'/>
<model type='virtio'/>
<link state='up'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
the inputs to the script can be found as follows:
domain_name - internal name of your vm (i-x-y-VM); can also be obtained by
"virsh list"
vm_id - the 'y' part of the internal name : i-x-y-VM
vm_ip - Now that debug logs have been enabled, you can search for
"StartCommand" in your logs get the ip from the 'nics' section or you could
choose any ip from the shared network range
vm_mac - from the above bridge section, you will be able to get the MAC address
vif / device name - taget dev name - in the above case - vnet0
bridge_name - again can be obtained from the above snippet (eg, breth1-11)
if there are no secondary ips, then pass 0 for nicsecips otherwise specify the
seconday ips separated by a semicolon (;)
Basically, at the end of it, it should look something like:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac
1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0; --isFirstNic
Hopefully, now you should be able to see a security_group.log file in the
/var/log/cloudstack/agent/ path.
Thanks,
Pearl
________________________________
From: Hean Seng <[email protected]>
Sent: Tuesday, September 29, 2020 10:45 AM
To: [email protected] <[email protected]>
Subject: Re: Cloudstack Advance with Security Group
Hi Pearl,
I had change to debug, following is the error captured:
2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
com.cloud.agent.api.SecurityGroupRulesCmd
2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
connection at: qemu:///system
2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) (logid:388b92e0) Checking default network
rules for vm i-23-77-VM
2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
network rule for nic cloudbr0 for VM i-23-77-VM
2020-09-29 01:13:30,094 WARN
[resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
(agentRequest-Handler-2:null) (logid:388b92e0) Failed to program default
network rules for vm i-23-77-VM
2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:388b92e0) Seq 11-6057904448766738456: {
Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
default network rules failed","wait":0}}] }
On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <[email protected]>
wrote:
> Hi Hean,
>
>
> Could you try enabling debug logging on your hypervisor hosts, by running
> the following:
> *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
>
> And then restart the cloudstack-agent service on the hosts. Post this, try
> deploying a VM in the shared network. This may give a better insight into
> where exactly the issue lies.
>
> Thanks,
> Pearl
>
>
> Software Engineer
> [email protected]
> www.shapeblue.com<http://www.shapeblue.com>
>
>
>
>
>
>
> ------------------------------
> *From:* Ivan Kudryavtsev <[email protected]>
> *Sent:* Monday, September 28, 2020 2:17 PM
> *To:* users <[email protected]>
> *Subject:* Re: Cloudstack Advance with Security Group
>
> This is it.
>
[email protected]
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
> On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <[email protected]> wrote:
>
> > In the log, I cannot see anything much, except a few lines showing above
> > . Not sure if this is a bug on 4.14.
> >
> >
> >
> > 2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
> > network rule for nic cloudbr0 for VM i-2-81-VM
> >
> > 2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
> > network rule for nic cloudbr0 for VM i-2-81-VM
> >
> > 2020-09-28 03:53:36,858 WARN
> > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > (agentRequest-Handler-3:null) (logid:74058678) Failed to program default
> > network rules for vm i-2-81-VM
> >
> >
> >
> >
> > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <[email protected]> wrote:
> >
> > > Hi,
> > > no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu,
> > though,
> > > but for any KVM hypervisor Linux distribution, the logic is the same.
> > >
> > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <[email protected]> wrote:
> > >
> > > > Hi
> > > >
> > > > Are you running on CentOS7 ?
> > > >
> > > > I am running on CentOS7 , ACS 4.14 , and seem there is no log at
> of
> > > > security_group.log
> > > >
> > > > # ls /var/log/cloudstack/agent/
> > > >
> > > > agent.log resizevolume.log setup.log
> > > >
> > > >
> > > > I recheck back the Intall guide, seems no missing anything.
> > > >
> > > >
> > > > Older intallation guide, 4.11 mentioned need , allow
> > > > /usr/lib/sysctl.d/00-system.conf
> > > >
> > > > # Enable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables =
> 1
> > > > net.bridge.bridge-nf-call-iptables = 1
> > > net.bridge.bridge-nf-call-arptables
> > > > = 1
> > > >
> > > > And it has been done too.
> > > >
> > > >
> > > >
> > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <[email protected]>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > No, this is not the issue.
> > > > > It's a normal state of the system, as KVM hooks are a new and
> > optional
> > > > > feature of 4.14.
> > > > >
> > > > > You should find some sort of messages regarding security_groups at
> > > > > /var/log/cloudstack/agent/security_group.log
> > > > >
> > > > >
> > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <[email protected]>
> > wrote:
> > > > >
> > > > > > I not sure where goes wrong, are you running on CentOS 7 ? I
> have
> > > this
> > > > > > error too, do you think is this contribute to the error as well:
> > > > > >
> > > > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
> is
> > > not
> > > > > > available. Transformations will not be applied.
> > > > > >
> > > > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting
> > > engine
> > > > is
> > > > > > not initialized. Data transformation skipped.
> > > > > >
> > > > > > 2020-09-28 03:04:53,083 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is
> not
> > > > > > available. Transformations will not be applied.
> > > > > >
> > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <[email protected]
> >
> > > > wrote:
> > > > > >
> > > > > > > This just means you installed it in the wrong way. Ebtables and
> > > > > Iptables
> > > > > > > must be filled with rules like
> > > > > > >
> > > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j
> > ACCEPT
> > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > > --physdev-is-bridged
> > > > > > -m
> > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
> > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m
> > udp
> > > > > > --dport
> > > > > > > 53 -j RETURN
> > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m
> > tcp
> > > > > > --dport
> > > > > > > 53 -j RETURN
> > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > > --physdev-is-bridged
> > > > > > -m
> > > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
> > > > > --physdev-is-bridged
> > > > > > -j
> > > > > > > i-6242-10304-vm
> > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state
> --state
> > > NEW
> > > > > -j
> > > > > > > ACCEPT
> > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state
> --state
> > > NEW
> > > > > -j
> > > > > > > ACCEPT
> > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> > > > > > > -A i-6242-10304-vm -j DROP
> > > > > > >
> > > > > > >
> > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> > > > > > > -p ARP -j i-4435-8929-vm-in-ips
> > > > > > > -p ARP --arp-op Request -j ACCEPT
> > > > > > > -p ARP --arp-op Reply -j ACCEPT
> > > > > > > -p ARP -j DROP
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <[email protected]>
> > > > wrote:
> > > > > > >
> > > > > > > > I checked the hypervisor , it seems iptables is nothing
> inside
> > ,
> > > > > this
> > > > > > is
> > > > > > > > centos7 , initially i turnoff firewalld , but even i turn
> on
> > it
> > > > now
> > > > > > and
> > > > > > > > try to update the security group rules, it seems empty
> iptable
> > > > rules
> > > > > :
> > > > > > > >
> > > > > > > > [root@kvm03 ~]# iptables -L -v -n
> > > > > > > >
> > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> > > > > > > >
> > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > destination
> > > > > > > >
> > > > > > > >
> > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > > > > > > >
> > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > destination
> > > > > > > >
> > > > > > > >
> > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> > > > > > > >
> > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > destination
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> > > > > > > [email protected]
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Hean,
> > > > > > > > >
> > > > > > > > > In an Advanced Zone with Security Groups enabled, by
> default,
> > > > > egress
> > > > > > > > > traffic from the VM is allowed, while Ingress traffic is
> > > denied.
> > > > > > Hence,
> > > > > > > > as
> > > > > > > > > you rightly mentioned, security group rules are added
> > > > accordingly.
> > > > > > > These
> > > > > > > > > rules get added on the hypervisor host, and you can verify
> > > them,
> > > > by
> > > > > > > going
> > > > > > > > > into the host and searching for iptables rules
> corresponding
> > to
> > > > the
> > > > > > VM
> > > > > > > > > (internal name - i-x-y-VM).
> > > > > > > > > This blog maybe helpful in providing further details:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Pearl
> > > > > > > > > ________________________________
> > > > > > > > > From: Hean Seng <[email protected]>
> > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
> > > > > > > > > To: [email protected] <
> [email protected]
> > >
> > > > > > > > > Subject: Cloudstack Advance with Security Group
> > > > > > > > >
> > > > > > > > > Hi
> > > > > > > > >
> > > > > > > > > I created advance zone with security group, all working
> fine.
> > > > > > > > >
> > > > > > > > > But VMcreated ， seems the default security group that
> > assigned
> > > to
> > > > > the
> > > > > > > VM.
> > > > > > > > > all accept policy , i understand is Default Deny, and once
> > add
> > > > in
> > > > > > the
> > > > > > > > port
> > > > > > > > > in Security Group Ingress and Egress, only is allowed
> > > > > > > > >
> > > > > > > > > Also, is this rules created at VirtualRouter of the
> > > > SharedNetwork,
> > > > > or
> > > > > > > at
> > > > > > > > > the Hypervisor?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > > Hean Seng
> > > > > > > > >
> > > > > > > > > [email protected]
> > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
> > > > > > > > > 3 London Bridge Street, 3rd floor, News Building, London
> > SE1
> > > > > 9SGUK
> > > > > > > > > @shapeblue
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > > Hean Seng
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Hean Seng
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Hean Seng
> > > >
> > >
> >
> >
> > --
> > Regards,
> > Hean Seng
> >
>
--
Regards,
Hean Seng
