Hi
I am running on CentOS 7.8.2003
Cloudstack Agent , and KVM hypervisor:
cloudstack-agent-4.14.0.0-1.el7.x86_64
iptables and ebtables aldy install, iptables -L and ebtables -L show
default rules .
This is dumpxml, seem the bridge is there, the VLAN for Guest is VLAN401,
which the setup seem right :
<interface type='bridge'>
<mac address='02:00:21:d9:00:01'/>
<source bridge='brem1-401'/>
<bandwidth>
<inbound average='1280' peak='1280'/>
<outbound average='1280' peak='1280'/>
</bandwidth>
<target dev='vnet9'/>
<model type='virtio'/>
<link state='up'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
# ifconfig brem1-401
brem1-401: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::3824:b2ff:fe09:873f prefixlen 64 scopeid 0x20<link>
ether 04:7d:7b:60:09:66 txqueuelen 1000 (Ethernet)
RX packets 7191638 bytes 333185781 (317.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 656 (656.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
brctl show
bridge name bridge id STP enabled interfaces
brem1-401 8000.047d7b600966 no em1.401
vnet10
vnet11
vnet12
vnet14
vnet4
vnet8
vnet9
cloud0 8000.fe00a9fe1cea no vnet13
vnet15
vnet2
vnet6
cloudbr0 8000.047d7b600966 yes em1
vnet3
vnet5
vnet7
cloudbr1 8000.000000000000 yes
I always have /var/log/cloudstack/agent/ this path , just inside
ls /var/log/cloudstack/agent/
agent.log agent.log.2020-09-26.gz agent.log.2020-09-27.gz
agent.log.2020-09-28.gz resizevolume.log setup.log
it doesnt have securitygroup folder/ log, all the log ip pasted is at
agent.log
On Tue, Sep 29, 2020 at 2:44 PM Pearl d'Silva <[email protected]>
wrote:
> Hi Hean,
>
> I was able to deploy a VM in a shared network in an advanced zone with
> security groups enabled and could see the iptables rules getting added for
> the VM on the host, so it is probably not a bug that we are seeing. Seems
> like a setup issue.
> Applying the rules can fail because:
>
> * The hypervisor host doesn't have iptables / ebtables command - which
> is probably unlikely
> * The VM deployed doesn't have an interface - can be verified by
> dumping its configuration - virsh dumpxml <domain_name> | grep interface
> * or application of the default network rules for the VM itself failed
> -If you are up for some debugging, can you try running the script manually
> in the hypervisor host:
>
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip <vm_ip>
> --vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name>
> --nicsecips 0; --isFirstNic --check
>
> when you do a virsh dumpxml <vm_domain_name>, look out for an interface
> section that looks something like:
>
> <interface type='bridge'>
> <mac address='1e:00:86:00:00:ba'/>
> <source bridge='breth1-11'/>
> <bandwidth>
> <inbound average='25600' peak='25600'/>
> <outbound average='25600' peak='25600'/>
> </bandwidth>
> <target dev='vnet0'/>
> <model type='virtio'/>
> <link state='up'/>
> <alias name='net0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> </interface>
>
>
> the inputs to the script can be found as follows:
> domain_name - internal name of your vm (i-x-y-VM); can also be obtained by
> "virsh list"
> vm_id - the 'y' part of the internal name : i-x-y-VM
> vm_ip - Now that debug logs have been enabled, you can search for
> "StartCommand" in your logs get the ip from the 'nics' section or you could
> choose any ip from the shared network range
> vm_mac - from the above bridge section, you will be able to get the MAC
> address
> vif / device name - taget dev name - in the above case - vnet0
> bridge_name - again can be obtained from the above snippet (eg, breth1-11)
> if there are no secondary ips, then pass 0 for nicsecips otherwise specify
> the seconday ips separated by a semicolon (;)
>
> Basically, at the end of it, it should look something like:
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac
> 1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0; --isFirstNic
>
> Hopefully, now you should be able to see a security_group.log file in the
> /var/log/cloudstack/agent/ path.
>
> Thanks,
> Pearl
>
>
>
>
>
> ________________________________
> From: Hean Seng <[email protected]>
> Sent: Tuesday, September 29, 2020 10:45 AM
> To: [email protected] <[email protected]>
> Subject: Re: Cloudstack Advance with Security Group
>
> Hi Pearl,
>
> I had change to debug, following is the error captured:
>
>
> 2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
> com.cloud.agent.api.SecurityGroupRulesCmd
>
> 2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
> connection at: qemu:///system
>
> 2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) (logid:388b92e0) Checking default network
> rules for vm i-23-77-VM
>
> 2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
> network rule for nic cloudbr0 for VM i-23-77-VM
>
> 2020-09-29 01:13:30,094 WARN
> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> (agentRequest-Handler-2:null) (logid:388b92e0) Failed to program default
> network rules for vm i-23-77-VM
>
> 2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) (logid:388b92e0) Seq
> 11-6057904448766738456: {
> Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
>
> [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
> default network rules failed","wait":0}}] }
>
>
>
>
>
> On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <[email protected]
> >
> wrote:
>
> > Hi Hean,
> >
> >
> > Could you try enabling debug logging on your hypervisor hosts, by running
> > the following:
> > *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
> >
> > And then restart the cloudstack-agent service on the hosts. Post this,
> try
> > deploying a VM in the shared network. This may give a better insight into
> > where exactly the issue lies.
> >
> > Thanks,
> > Pearl
> >
> >
> > Software Engineer
> > [email protected]
> > www.shapeblue.com<http://www.shapeblue.com>
> >
> >
> >
> >
> >
> >
> > ------------------------------
> > *From:* Ivan Kudryavtsev <[email protected]>
> > *Sent:* Monday, September 28, 2020 2:17 PM
> > *To:* users <[email protected]>
> > *Subject:* Re: Cloudstack Advance with Security Group
> >
> > This is it.
> >
>
> [email protected]
> www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
> > On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <[email protected]> wrote:
> >
> > > In the log, I cannot see anything much, except a few lines showing
> above
> > > . Not sure if this is a bug on 4.14.
> > >
> > >
> > >
> > > 2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
> > > network rule for nic cloudbr0 for VM i-2-81-VM
> > >
> > > 2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
> > > network rule for nic cloudbr0 for VM i-2-81-VM
> > >
> > > 2020-09-28 03:53:36,858 WARN
> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > > (agentRequest-Handler-3:null) (logid:74058678) Failed to program
> default
> > > network rules for vm i-2-81-VM
> > >
> > >
> > >
> > >
> > > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <[email protected]>
> wrote:
> > >
> > > > Hi,
> > > > no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu,
> > > though,
> > > > but for any KVM hypervisor Linux distribution, the logic is the same.
> > > >
> > > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <[email protected]>
> wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > Are you running on CentOS7 ?
> > > > >
> > > > > I am running on CentOS7 , ACS 4.14 , and seem there is no log at
> > of
> > > > > security_group.log
> > > > >
> > > > > # ls /var/log/cloudstack/agent/
> > > > >
> > > > > agent.log resizevolume.log setup.log
> > > > >
> > > > >
> > > > > I recheck back the Intall guide, seems no missing anything.
> > > > >
> > > > >
> > > > > Older intallation guide, 4.11 mentioned need , allow
> > > > > /usr/lib/sysctl.d/00-system.conf
> > > > >
> > > > > # Enable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables
> =
> > 1
> > > > > net.bridge.bridge-nf-call-iptables = 1
> > > > net.bridge.bridge-nf-call-arptables
> > > > > = 1
> > > > >
> > > > > And it has been done too.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <[email protected]>
> > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > No, this is not the issue.
> > > > > > It's a normal state of the system, as KVM hooks are a new and
> > > optional
> > > > > > feature of 4.14.
> > > > > >
> > > > > > You should find some sort of messages regarding security_groups
> at
> > > > > > /var/log/cloudstack/agent/security_group.log
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <[email protected]>
> > > wrote:
> > > > > >
> > > > > > > I not sure where goes wrong, are you running on CentOS 7 ? I
> > have
> > > > this
> > > > > > > error too, do you think is this contribute to the error as
> well:
> > > > > > >
> > > > > > > 2020-09-28 03:04:52,762 WARN
> [kvm.resource.LibvirtKvmAgentHook]
> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
> > is
> > > > not
> > > > > > > available. Transformations will not be applied.
> > > > > > >
> > > > > > > 2020-09-28 03:04:52,762 WARN
> [kvm.resource.LibvirtKvmAgentHook]
> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting
> > > > engine
> > > > > is
> > > > > > > not initialized. Data transformation skipped.
> > > > > > >
> > > > > > > 2020-09-28 03:04:53,083 WARN
> [kvm.resource.LibvirtKvmAgentHook]
> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is
> > not
> > > > > > > available. Transformations will not be applied.
> > > > > > >
> > > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <
> [email protected]
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > This just means you installed it in the wrong way. Ebtables
> and
> > > > > > Iptables
> > > > > > > > must be filled with rules like
> > > > > > > >
> > > > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j
> > > ACCEPT
> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> > > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > > > --physdev-is-bridged
> > > > > > > -m
> > > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
> -m
> > > udp
> > > > > > > --dport
> > > > > > > > 53 -j RETURN
> > > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
> -m
> > > tcp
> > > > > > > --dport
> > > > > > > > 53 -j RETURN
> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > > > --physdev-is-bridged
> > > > > > > -m
> > > > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> > > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
> > > > > > --physdev-is-bridged
> > > > > > > -j
> > > > > > > > i-6242-10304-vm
> > > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state
> > --state
> > > > NEW
> > > > > > -j
> > > > > > > > ACCEPT
> > > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state
> > --state
> > > > NEW
> > > > > > -j
> > > > > > > > ACCEPT
> > > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> > > > > > > > -A i-6242-10304-vm -j DROP
> > > > > > > >
> > > > > > > >
> > > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> > > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> > > > > > > > -p ARP -j i-4435-8929-vm-in-ips
> > > > > > > > -p ARP --arp-op Request -j ACCEPT
> > > > > > > > -p ARP --arp-op Reply -j ACCEPT
> > > > > > > > -p ARP -j DROP
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <
> [email protected]>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > I checked the hypervisor , it seems iptables is nothing
> > inside
> > > ,
> > > > > > this
> > > > > > > is
> > > > > > > > > centos7 , initially i turnoff firewalld , but even i turn
> > on
> > > it
> > > > > now
> > > > > > > and
> > > > > > > > > try to update the security group rules, it seems empty
> > iptable
> > > > > rules
> > > > > > :
> > > > > > > > >
> > > > > > > > > [root@kvm03 ~]# iptables -L -v -n
> > > > > > > > >
> > > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> > > > > > > > >
> > > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > > destination
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > > > > > > > >
> > > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > > destination
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> > > > > > > > >
> > > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > > destination
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> > > > > > > > [email protected]
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Hean,
> > > > > > > > > >
> > > > > > > > > > In an Advanced Zone with Security Groups enabled, by
> > default,
> > > > > > egress
> > > > > > > > > > traffic from the VM is allowed, while Ingress traffic is
> > > > denied.
> > > > > > > Hence,
> > > > > > > > > as
> > > > > > > > > > you rightly mentioned, security group rules are added
> > > > > accordingly.
> > > > > > > > These
> > > > > > > > > > rules get added on the hypervisor host, and you can
> verify
> > > > them,
> > > > > by
> > > > > > > > going
> > > > > > > > > > into the host and searching for iptables rules
> > corresponding
> > > to
> > > > > the
> > > > > > > VM
> > > > > > > > > > (internal name - i-x-y-VM).
> > > > > > > > > > This blog maybe helpful in providing further details:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Pearl
> > > > > > > > > > ________________________________
> > > > > > > > > > From: Hean Seng <[email protected]>
> > > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
> > > > > > > > > > To: [email protected] <
> > [email protected]
> > > >
> > > > > > > > > > Subject: Cloudstack Advance with Security Group
> > > > > > > > > >
> > > > > > > > > > Hi
> > > > > > > > > >
> > > > > > > > > > I created advance zone with security group, all working
> > fine.
> > > > > > > > > >
> > > > > > > > > > But VMcreated ， seems the default security group that
> > > assigned
> > > > to
> > > > > > the
> > > > > > > > VM.
> > > > > > > > > > all accept policy , i understand is Default Deny, and
> once
> > > add
> > > > > in
> > > > > > > the
> > > > > > > > > port
> > > > > > > > > > in Security Group Ingress and Egress, only is allowed
> > > > > > > > > >
> > > > > > > > > > Also, is this rules created at VirtualRouter of the
> > > > > SharedNetwork,
> > > > > > or
> > > > > > > > at
> > > > > > > > > > the Hypervisor?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Regards,
> > > > > > > > > > Hean Seng
> > > > > > > > > >
> > > > > > > > > > [email protected]
> > > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
> > > > > > > > > > 3 London Bridge Street, 3rd floor, News Building, London
> > > SE1
> > > > > > 9SGUK
> > > > > > > > > > @shapeblue
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > > Hean Seng
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Hean Seng
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Hean Seng
> > > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > Hean Seng
> > >
> >
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
