[ https://issues.apache.org/jira/browse/CLOUDSTACK-8428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aleksandr updated CLOUDSTACK-8428: ---------------------------------- Description: Clean install, Cloudstack 4.4.2 on ubuntu 14.04 from .deb pkg repo. KVM, Advanced zone, GRE - OVS, 1 nic and 3 bridges/3vlans ( like in official manual ) - mgmt0, cloudbr0, cloudbr1 ( and parent bridge cloudbr ) I'm adding new instances ( from iso for example ) so the VR starts for this default nework - Offering for Isolated networks with Source Nat service enabled ( everything by default, no custom configuration ) And just after VR goes up the host comes in and add 2nd public nic > Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip addr add dev eth3 185.22.174.12/24 brd + So the right public nic is eth2 and fake duplicate is eth3 ########################################################## Logs from VR root@r-33-VM:/var/log# grep -R "eth3" . Binary file ./sysstat/sa29 matches ./cloud.log:Wed Apr 29 09:17:38 UTC 2015 : VR config: executing: /opt/cloud/bin/ipassoc.sh -A -s -f -l 185.22.174.12/24 -c eth3 -g 185.22.174.1 -n ./cloud.log:ARPING 185.22.174.12 from 185.22.174.12 eth3 ./cloud.log:ARPING 185.22.174.12 from 185.22.174.12 eth3 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip link show eth3 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip addr add dev eth3 185.22.174.12/24 brd + ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -D FORWARD -i eth3 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -D FORWARD -i eth0 -o eth3 -j FW_OUTBOUND ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -A FORWARD -i eth3 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -A FORWARD -i eth0 -o eth3 -j FW_OUTBOUND ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -t nat -D POSTROUTING -j SNAT -o eth3 --to-source 185.22.174.12 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -t nat -A POSTROUTING -j SNAT -o eth3 --to-source 185.22.174.12 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip link set eth3 up ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/arping -c 1 -I eth3 -A -U -s 185.22.174.12 185.22.174.12 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/arping -c 1 -I eth3 -A -U -s 185.22.174.12 185.22.174.12 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/echo 3 Table_eth3 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip route add throw 172.17.150.0/24 table Table_eth3 proto static ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip route add throw 169.254.0.0/16 table Table_eth3 proto static ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip route add throw 185.22.174.0/24 table Table_eth3 proto static ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip route add default via 185.22.174.1 table Table_eth3 proto static ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip rule add from 185.22.174.0/24 table Table_eth3 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip rule add fwmark 3 table Table_eth3 ./messages:Apr 29 09:17:38 r-33-VM cloud: VR config: executing: /opt/cloud/bin/ipassoc.sh -A -s -f -l 185.22.174.12/24 -c eth3 -g 185.22.174.1 -n ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Adding first ip 185.22.174.12/24 on interface eth3 ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Added SourceNAT 185.22.174.12/24 on interface eth3 ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Added first ip 185.22.174.12/24 on interface eth3 ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Add routing 185.22.174.12/24 on interface eth3 ########################################################### Host has no logs about this " r-33-VM " VR ########################################################### Mgmt server : ./management-server.log:2015-04-29 12:16:28,550 DEBUG [c.c.a.t.Request] (Work-Job-Executor-38:ctx-01f0beeb job-260/job-263 ctx-ab6ac568) Seq 1-3349552222856808115: Sending { Cmd , MgmtId: 115129176880998, via: 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":33,"name":"r-33-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" template=domP name=r-33-VM eth2ip=185.22.174.12 eth2mask=255.255.255.0 gateway=185.22.174.1 eth0ip=172.17.150.1 eth0mask=255.255.255.0 domain=cs2cloud.internal cidrsize=24 dhcprange=172.17.150.1 eth1ip=169.254.0.58 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=8.8.8.8 dns2=8.8.4.4","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"b54e615a272b5f45","params":{},"uuid":"8d1c0a71-1cd2-4639-97f3-13ae9fb28b6d","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"cd2d4a78-c807-42bd-a499-14b32b424925","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary","port":0,"url":"SharedMountPoint://localhost/mnt/primary/?ROLE=Primary&STOREUUID=cd2d4a78-c807-42bd-a499-14b32b424925"}},"name":"ROOT-33","size":308822528,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeId":35,"vmName":"r-33-VM","accountId":2,"format":"QCOW2","id":35,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"localhost","volumeSize":"308822528"}}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"e852e031-11b7-4b63-be7d-03d1229541cd","ip":"185.22.174.12","netmask":"255.255.255.0","gateway":"185.22.174.1","mac":"06:8d:dc:00:00:34","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1700","isolationUri":"vlan://1700","isSecurityGroupEnabled":false,"name":"cloudbr0"},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"082b65b6-24ed-4af0-aede-34ea2bc2003e","ip":"172.17.150.1","netmask":"255.255.255.0","mac":"02:00:25:23:00:07","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1731","isolationUri":"vlan://1731","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"842f837f-0c34-42db-b860-6c4628a91f2c","ip":"169.254.0.58","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:3a","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"172.17.100.4","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.58","port":3922,"interval":6,"retries":100,"name":"r-33-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}},{},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[{"accountId":2,"publicIp":"185.22.174.12","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"broadcastUri":"vlan://1700","vlanGateway":"185.22.174.1","vlanNetmask":"255.255.255.0","vifMacAddress":"06:51:da:00:00:34","networkRate":200,"trafficType":"Public","networkName":"cloudbr0","newNic":false}],"accessDetails":{"zone.network.type":"Advanced","router.name":"r-33-V ","router.ip":"169.254.0.58","router.guest.ip":"172.17.150.1"},"wait":0}},{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"services":[{"id":0,"service":"dhcp","processname":"dnsmasq","serviceName":"dnsmasq","servicePath":"/var/run/dnsmasq/dnsmasq.pid","pidFile":"/var/run/dnsmasq/dnsmasq.pid","isDefault":false},{"id":0,"service":"loadbalancing","processname":"haproxy","serviceName":"haproxy","servicePath":"/var/run/haproxy.pid","pidFile":"/var/run/haproxy.pid","isDefault":false},{"id":0,"service":"ssh","processname":"sshd","serviceName":"ssh","servicePath":"/var/run/sshd.pid","pidFile":"/var/run/sshd.pid","isDefault":true},{"id":0,"service":"webserver","processname":"apache2","serviceName":"apache2","servicePath":"/var/run/apache2.pid","pidFile":"/var/run/apache2.pid","isDefault":true}],"accessDetails":{"router.name":"r-33-VM","router.ip":"169.254.0.58","router.guest.ip":"172.17.150.1"},"wait":0}},{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:2e:7b:00:01","vmIpAddress":"172.17.150.190","vmName":"testvps","defaultRouter":"172.17.150.1","defaultDns":"172.17.150.1","duid":"00:03:00:01:02:00:2e:7b:00:01","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"172.17.150.1","router.ip":"169.254.0.58","router.name":"r-33-VM"},"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"172.17.150.190","vmName":"testvps","executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.name":"r-33-VM","router.ip":"169.254.0.58","router.guest.ip":"172.17.150.1"},"wait":0}},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Finish","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] } ./management-server.log:2015-04-29 12:17:39,420 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null) Seq 1-3349552222856808115: Processing: { Ans: , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":33,"name":"r-33-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" template=domP name=r-33-VM eth2ip=185.22.174.12 eth2mask=255.255.255.0 gateway=185.22.174.1 eth0ip=172.17.150.1 eth0mask=255.255.255.0 domain=cs2cloud.internal cidrsize=24 dhcprange=172.17.150.1 eth1ip=169.254.0.58 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=8.8.8.8 dns2=8.8.4.4","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"b54e615a272b5f45","vncAddr":"172.17.100.4","params":{},"uuid":"8d1c0a71-1cd2-4639-97f3-13ae9fb28b6d","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"cd2d4a78-c807-42bd-a499-14b32b424925","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary","port":0,"url":"SharedMountPoint://localhost/mnt/primary/?ROLE=Primary&STOREUUID=cd2d4a78-c807-42bd-a499-14b32b424925"}},"name":"ROOT-33","size":308822528,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeId":35,"vmName":"r-33-VM","accountId":2,"format":"QCOW2","id":35,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"localhost","volumeSize":"308822528"}}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"e852e031-11b7-4b63-be7d-03d1229541cd","ip":"185.22.174.12","netmask":"255.255.255.0","gateway":"185.22.174.1","mac":"06:8d:dc:00:00:34","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1700","isolationUri":"vlan://1700","isSecurityGroupEnabled":false,"name":"cloudbr0"},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"082b65b6-24ed-4af0-aede-34ea2bc2003e","ip":"172.17.150.1","netmask":"255.255.255.0","mac":"02:00:25:23:00:07","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1731","isolationUri":"vlan://1731","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"842f837f-0c34-42db-b860-6c4628a91f2c","ip":"169.254.0.58","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:3a","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"result":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack Release 4.4.1 Mon Sep 29 14:29:20 UTC 2014","scriptsVersion":"5bccd9c9d4b8d0b6ae66c0128d771789\n","result":true,"details":"Cloudstack Release 4.4.1 Mon Sep 29 14:29:20 UTC 2014&5bccd9c9d4b8d0b6ae66c0128d771789\n","wait":0}},{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":0,"bytesReceived":0,"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] } ./management-server.log:2015-04-29 12:17:39,905 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-38:ctx-01f0beeb job-260/job-263 ctx-ab6ac568) Start completed for VM VM[DomainRouter|r-33-VM] ./management-server.log:2015-04-29 12:17:40,417 DEBUG [c.c.a.t.Request] (API-Job-Executor-58:ctx-b2a553ab job-260 ctx-fe610701) Seq 1-3349552222856808119: Sending { Cmd , MgmtId: 115129176880998, via: 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] } ./management-server.log:2015-04-29 12:17:40,463 DEBUG [c.c.a.t.Request] (API-Job-Executor-58:ctx-b2a553ab job-260 ctx-fe610701) Seq 1-3349552222856808120: Sending { Cmd , MgmtId: 115129176880998, via: 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Finish","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] } ./management-server.log:2015-04-29 12:17:40,802 DEBUG [c.c.a.t.Request] (API-Job-Executor-58:ctx-b2a553ab job-260 ctx-fe610701) Seq 1-3349552222856808121: Sending { Cmd , MgmtId: 115129176880998, via: 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Cleanup","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] } ./management-server.log:2015-04-29 12:17:42,238 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) Seq 1-3349552222856808122: Processing: { Ans: , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":336,"bytesReceived":0,"result":true,"details":"","wait":0}}] } ./management-server.log:2015-04-29 12:22:42,190 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:null) Seq 1-3349552222856808138: Processing: { Ans: , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":25368,"bytesReceived":0,"result":true,"details":"","wait":0}}] } ./management-server.log:2015-04-29 12:27:42,187 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:null) Seq 1-3349552222856808154: Processing: { Ans: , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":50400,"bytesReceived":0,"result":true,"details":"","wait":0}}] } ######################################################### Iptables from VR root@r-33-VM:~# iptables -L -nv -t nat Chain PREROUTING (policy ACCEPT 14 packets, 951 bytes) pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 14 packets, 951 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 SNAT all -- * eth3 0.0.0.0/0 0.0.0.0/0 to:185.22.174.12 root@r-33-VM:~# iptables -L -nv Chain INPUT (policy DROP 19 packets, 1444 bytes) pkts bytes target prot opt in out source destination 891 77029 NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * * 0.0.0.0/0 224.0.0.18 0 0 ACCEPT all -- * * 0.0.0.0/0 225.0.0.50 0 0 ACCEPT all -- eth0 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 835 76520 ACCEPT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 74 6112 ACCEPT all -- eth2 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 8 672 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 1 93 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67 3 195 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 14 840 ACCEPT tcp -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3922 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80 0 0 ACCEPT tcp -- eth0 * 172.17.150.0/24 0.0.0.0/0 state NEW tcp dpt:8080 Chain FORWARD (policy DROP 886 packets, 74424 bytes) pkts bytes target prot opt in out source destination 887 74508 NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- eth0 eth1 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state NEW 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 895 75180 FW_OUTBOUND all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- eth3 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 FW_OUTBOUND all -- eth0 eth3 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT 929 packets, 204K bytes) pkts bytes target prot opt in out source destination 986 214K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 Chain FW_OUTBOUND (2 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED Chain NETWORK_STATS (3 references) pkts bytes target prot opt in out source destination 887 74508 all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0 0 0 all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 0 0 tcp -- !eth0 eth2 0.0.0.0/0 0.0.0.0/0 0 0 tcp -- eth2 !eth0 0.0.0.0/0 0.0.0.0/0 0 0 all -- eth0 eth3 0.0.0.0/0 0.0.0.0/0 0 0 all -- eth3 eth0 0.0.0.0/0 0.0.0.0/0 0 0 tcp -- !eth0 eth3 0.0.0.0/0 0.0.0.0/0 0 0 tcp -- eth3 !eth0 0.0.0.0/0 0.0.0.0/0 root@r-33-VM:~# iptables -L -nv -t nat Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 SNAT all -- * eth3 0.0.0.0/0 0.0.0.0/0 to:185.22.174.12 p.s. i think something wrong with a mechanism which is propagating rules ( firewall rules ) to VR was: Clean install, Cloudstack 4.4.2 on ubuntu 14.04 from .deb pkg repo. KVM, Advanced zone, GRE - OVS, 1 nic and 3 bridges/3vlans ( like in official manual ) - mgmt0, cloudbr0, cloudbr1 ( and parent bridge cloudbr ) I'm adding new instances ( from iso for example ) so the VR starts for this default nework - Offering for Isolated networks with Source Nat service enabled ( everything by default, no custom configuration ) And just after VR goes up the host comes in and add 2nd public nic > Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip addr add dev eth3 185.22.174.12/24 brd + So the right public nic is eth2 and fake duplicate is eth3 ########################################################## Logs from VR root@r-33-VM:/var/log# grep -R "eth3" . Binary file ./sysstat/sa29 matches ./cloud.log:Wed Apr 29 09:17:38 UTC 2015 : VR config: executing: /opt/cloud/bin/ipassoc.sh -A -s -f -l 185.22.174.12/24 -c eth3 -g 185.22.174.1 -n ./cloud.log:ARPING 185.22.174.12 from 185.22.174.12 eth3 ./cloud.log:ARPING 185.22.174.12 from 185.22.174.12 eth3 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip link show eth3 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip addr add dev eth3 185.22.174.12/24 brd + ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -D FORWARD -i eth3 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -D FORWARD -i eth0 -o eth3 -j FW_OUTBOUND ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -A FORWARD -i eth3 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -A FORWARD -i eth0 -o eth3 -j FW_OUTBOUND ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -t nat -D POSTROUTING -j SNAT -o eth3 --to-source 185.22.174.12 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/iptables -t nat -A POSTROUTING -j SNAT -o eth3 --to-source 185.22.174.12 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip link set eth3 up ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/arping -c 1 -I eth3 -A -U -s 185.22.174.12 185.22.174.12 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/arping -c 1 -I eth3 -A -U -s 185.22.174.12 185.22.174.12 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/bin/echo 3 Table_eth3 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip route add throw 172.17.150.0/24 table Table_eth3 proto static ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip route add throw 169.254.0.0/16 table Table_eth3 proto static ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip route add throw 185.22.174.0/24 table Table_eth3 proto static ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip route add default via 185.22.174.1 table Table_eth3 proto static ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip rule add from 185.22.174.0/24 table Table_eth3 ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/sbin/ip rule add fwmark 3 table Table_eth3 ./messages:Apr 29 09:17:38 r-33-VM cloud: VR config: executing: /opt/cloud/bin/ipassoc.sh -A -s -f -l 185.22.174.12/24 -c eth3 -g 185.22.174.1 -n ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Adding first ip 185.22.174.12/24 on interface eth3 ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Added SourceNAT 185.22.174.12/24 on interface eth3 ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Added first ip 185.22.174.12/24 on interface eth3 ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Add routing 185.22.174.12/24 on interface eth3 ########################################################### Host has no logs about this " r-33-VM " VR ########################################################### Mgmt server : ./management-server.log:2015-04-29 12:16:28,550 DEBUG [c.c.a.t.Request] (Work-Job-Executor-38:ctx-01f0beeb job-260/job-263 ctx-ab6ac568) Seq 1-3349552222856808115: Sending { Cmd , MgmtId: 115129176880998, via: 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":33,"name":"r-33-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" template=domP name=r-33-VM eth2ip=185.22.174.12 eth2mask=255.255.255.0 gateway=185.22.174.1 eth0ip=172.17.150.1 eth0mask=255.255.255.0 domain=cs2cloud.internal cidrsize=24 dhcprange=172.17.150.1 eth1ip=169.254.0.58 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=8.8.8.8 dns2=8.8.4.4","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"b54e615a272b5f45","params":{},"uuid":"8d1c0a71-1cd2-4639-97f3-13ae9fb28b6d","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"cd2d4a78-c807-42bd-a499-14b32b424925","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary","port":0,"url":"SharedMountPoint://localhost/mnt/primary/?ROLE=Primary&STOREUUID=cd2d4a78-c807-42bd-a499-14b32b424925"}},"name":"ROOT-33","size":308822528,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeId":35,"vmName":"r-33-VM","accountId":2,"format":"QCOW2","id":35,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"localhost","volumeSize":"308822528"}}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"e852e031-11b7-4b63-be7d-03d1229541cd","ip":"185.22.174.12","netmask":"255.255.255.0","gateway":"185.22.174.1","mac":"06:8d:dc:00:00:34","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1700","isolationUri":"vlan://1700","isSecurityGroupEnabled":false,"name":"cloudbr0"},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"082b65b6-24ed-4af0-aede-34ea2bc2003e","ip":"172.17.150.1","netmask":"255.255.255.0","mac":"02:00:25:23:00:07","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1731","isolationUri":"vlan://1731","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"842f837f-0c34-42db-b860-6c4628a91f2c","ip":"169.254.0.58","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:3a","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"172.17.100.4","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.58","port":3922,"interval":6,"retries":100,"name":"r-33-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}},{},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[{"accountId":2,"publicIp":"185.22.174.12","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"broadcastUri":"vlan://1700","vlanGateway":"185.22.174.1","vlanNetmask":"255.255.255.0","vifMacAddress":"06:51:da:00:00:34","networkRate":200,"trafficType":"Public","networkName":"cloudbr0","newNic":false}],"accessDetails":{"zone.network.type":"Advanced","router.name":"r-33-V ","router.ip":"169.254.0.58","router.guest.ip":"172.17.150.1"},"wait":0}},{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"services":[{"id":0,"service":"dhcp","processname":"dnsmasq","serviceName":"dnsmasq","servicePath":"/var/run/dnsmasq/dnsmasq.pid","pidFile":"/var/run/dnsmasq/dnsmasq.pid","isDefault":false},{"id":0,"service":"loadbalancing","processname":"haproxy","serviceName":"haproxy","servicePath":"/var/run/haproxy.pid","pidFile":"/var/run/haproxy.pid","isDefault":false},{"id":0,"service":"ssh","processname":"sshd","serviceName":"ssh","servicePath":"/var/run/sshd.pid","pidFile":"/var/run/sshd.pid","isDefault":true},{"id":0,"service":"webserver","processname":"apache2","serviceName":"apache2","servicePath":"/var/run/apache2.pid","pidFile":"/var/run/apache2.pid","isDefault":true}],"accessDetails":{"router.name":"r-33-VM","router.ip":"169.254.0.58","router.guest.ip":"172.17.150.1"},"wait":0}},{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:2e:7b:00:01","vmIpAddress":"172.17.150.190","vmName":"testvps","defaultRouter":"172.17.150.1","defaultDns":"172.17.150.1","duid":"00:03:00:01:02:00:2e:7b:00:01","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"172.17.150.1","router.ip":"169.254.0.58","router.name":"r-33-VM"},"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"172.17.150.190","vmName":"testvps","executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.name":"r-33-VM","router.ip":"169.254.0.58","router.guest.ip":"172.17.150.1"},"wait":0}},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Finish","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] } ./management-server.log:2015-04-29 12:17:39,420 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:null) Seq 1-3349552222856808115: Processing: { Ans: , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":33,"name":"r-33-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" template=domP name=r-33-VM eth2ip=185.22.174.12 eth2mask=255.255.255.0 gateway=185.22.174.1 eth0ip=172.17.150.1 eth0mask=255.255.255.0 domain=cs2cloud.internal cidrsize=24 dhcprange=172.17.150.1 eth1ip=169.254.0.58 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=8.8.8.8 dns2=8.8.4.4","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"b54e615a272b5f45","vncAddr":"172.17.100.4","params":{},"uuid":"8d1c0a71-1cd2-4639-97f3-13ae9fb28b6d","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"cd2d4a78-c807-42bd-a499-14b32b424925","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary","port":0,"url":"SharedMountPoint://localhost/mnt/primary/?ROLE=Primary&STOREUUID=cd2d4a78-c807-42bd-a499-14b32b424925"}},"name":"ROOT-33","size":308822528,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeId":35,"vmName":"r-33-VM","accountId":2,"format":"QCOW2","id":35,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"localhost","volumeSize":"308822528"}}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"e852e031-11b7-4b63-be7d-03d1229541cd","ip":"185.22.174.12","netmask":"255.255.255.0","gateway":"185.22.174.1","mac":"06:8d:dc:00:00:34","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1700","isolationUri":"vlan://1700","isSecurityGroupEnabled":false,"name":"cloudbr0"},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"082b65b6-24ed-4af0-aede-34ea2bc2003e","ip":"172.17.150.1","netmask":"255.255.255.0","mac":"02:00:25:23:00:07","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1731","isolationUri":"vlan://1731","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"842f837f-0c34-42db-b860-6c4628a91f2c","ip":"169.254.0.58","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:3a","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"result":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack Release 4.4.1 Mon Sep 29 14:29:20 UTC 2014","scriptsVersion":"5bccd9c9d4b8d0b6ae66c0128d771789\n","result":true,"details":"Cloudstack Release 4.4.1 Mon Sep 29 14:29:20 UTC 2014&5bccd9c9d4b8d0b6ae66c0128d771789\n","wait":0}},{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":0,"bytesReceived":0,"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] } ./management-server.log:2015-04-29 12:17:39,905 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-38:ctx-01f0beeb job-260/job-263 ctx-ab6ac568) Start completed for VM VM[DomainRouter|r-33-VM] ./management-server.log:2015-04-29 12:17:40,417 DEBUG [c.c.a.t.Request] (API-Job-Executor-58:ctx-b2a553ab job-260 ctx-fe610701) Seq 1-3349552222856808119: Sending { Cmd , MgmtId: 115129176880998, via: 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] } ./management-server.log:2015-04-29 12:17:40,463 DEBUG [c.c.a.t.Request] (API-Job-Executor-58:ctx-b2a553ab job-260 ctx-fe610701) Seq 1-3349552222856808120: Sending { Cmd , MgmtId: 115129176880998, via: 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Finish","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] } ./management-server.log:2015-04-29 12:17:40,802 DEBUG [c.c.a.t.Request] (API-Job-Executor-58:ctx-b2a553ab job-260 ctx-fe610701) Seq 1-3349552222856808121: Sending { Cmd , MgmtId: 115129176880998, via: 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Cleanup","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] } ./management-server.log:2015-04-29 12:17:42,238 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) Seq 1-3349552222856808122: Processing: { Ans: , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":336,"bytesReceived":0,"result":true,"details":"","wait":0}}] } ./management-server.log:2015-04-29 12:22:42,190 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:null) Seq 1-3349552222856808138: Processing: { Ans: , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":25368,"bytesReceived":0,"result":true,"details":"","wait":0}}] } ./management-server.log:2015-04-29 12:27:42,187 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:null) Seq 1-3349552222856808154: Processing: { Ans: , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":50400,"bytesReceived":0,"result":true,"details":"","wait":0}}] } ######################################################### Iptables from VR root@r-33-VM:~# iptables -L -nv -t nat Chain PREROUTING (policy ACCEPT 14 packets, 951 bytes) pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 14 packets, 951 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 SNAT all -- * eth3 0.0.0.0/0 0.0.0.0/0 to:185.22.174.12 root@r-33-VM:~# iptables -L -nv Chain INPUT (policy DROP 19 packets, 1444 bytes) pkts bytes target prot opt in out source destination 891 77029 NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * * 0.0.0.0/0 224.0.0.18 0 0 ACCEPT all -- * * 0.0.0.0/0 225.0.0.50 0 0 ACCEPT all -- eth0 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 835 76520 ACCEPT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 74 6112 ACCEPT all -- eth2 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 8 672 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 1 93 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67 3 195 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 14 840 ACCEPT tcp -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3922 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80 0 0 ACCEPT tcp -- eth0 * 172.17.150.0/24 0.0.0.0/0 state NEW tcp dpt:8080 Chain FORWARD (policy DROP 886 packets, 74424 bytes) pkts bytes target prot opt in out source destination 887 74508 NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- eth0 eth1 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state NEW 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 895 75180 FW_OUTBOUND all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- eth3 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 FW_OUTBOUND all -- eth0 eth3 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT 929 packets, 204K bytes) pkts bytes target prot opt in out source destination 986 214K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 Chain FW_OUTBOUND (2 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED Chain NETWORK_STATS (3 references) pkts bytes target prot opt in out source destination 887 74508 all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0 0 0 all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 0 0 tcp -- !eth0 eth2 0.0.0.0/0 0.0.0.0/0 0 0 tcp -- eth2 !eth0 0.0.0.0/0 0.0.0.0/0 0 0 all -- eth0 eth3 0.0.0.0/0 0.0.0.0/0 0 0 all -- eth3 eth0 0.0.0.0/0 0.0.0.0/0 0 0 tcp -- !eth0 eth3 0.0.0.0/0 0.0.0.0/0 0 0 tcp -- eth3 !eth0 0.0.0.0/0 0.0.0.0/0 root@r-33-VM:~# iptables -L -nv -t nat Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 SNAT all -- * eth3 0.0.0.0/0 0.0.0.0/0 to:185.22.174.12 p.s. i think something wrong with a mechanism which is propagating rules ( firewall rules ) to VM > VR can't provide services to instances due to wrong interface configuration ( > duplicate public interface on VR) > --------------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-8428 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8428 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Virtual Router > Affects Versions: 4.4.2 > Environment: MGMT - Ubuntu 14.04, Host - Centos 6.6 > Cloudstack - 4.4.2, VR - 4.4.1 > Reporter: Aleksandr > Priority: Blocker > > Clean install, Cloudstack 4.4.2 on ubuntu 14.04 from .deb pkg repo. > KVM, Advanced zone, GRE - OVS, 1 nic and 3 bridges/3vlans ( like in official > manual ) - mgmt0, cloudbr0, cloudbr1 ( and parent bridge cloudbr ) > I'm adding new instances ( from iso for example ) so the VR starts for this > default nework - Offering for Isolated networks with Source Nat service > enabled ( everything by default, no custom configuration ) > And just after VR goes up the host comes in and add 2nd public nic > > Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; USER=root > ; COMMAND=/sbin/ip addr add dev eth3 185.22.174.12/24 brd + > So the right public nic is eth2 and fake duplicate is eth3 > ########################################################## > Logs from VR > root@r-33-VM:/var/log# grep -R "eth3" . > Binary file ./sysstat/sa29 matches > ./cloud.log:Wed Apr 29 09:17:38 UTC 2015 : VR config: executing: > /opt/cloud/bin/ipassoc.sh -A -s -f -l 185.22.174.12/24 -c eth3 -g > 185.22.174.1 -n > ./cloud.log:ARPING 185.22.174.12 from 185.22.174.12 eth3 > ./cloud.log:ARPING 185.22.174.12 from 185.22.174.12 eth3 > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/ip link show eth3 > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/ip addr add dev eth3 185.22.174.12/24 brd + > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/iptables -D FORWARD -i eth3 -o eth0 -m state > --state RELATED,ESTABLISHED -j ACCEPT > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/iptables -D FORWARD -i eth0 -o eth3 -j FW_OUTBOUND > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/iptables -A FORWARD -i eth3 -o eth0 -m state > --state RELATED,ESTABLISHED -j ACCEPT > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/iptables -A FORWARD -i eth0 -o eth3 -j FW_OUTBOUND > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/iptables -t nat -D POSTROUTING -j SNAT -o eth3 > --to-source 185.22.174.12 > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/iptables -t nat -A POSTROUTING -j SNAT -o eth3 > --to-source 185.22.174.12 > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/ip link set eth3 up > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/usr/bin/arping -c 1 -I eth3 -A -U -s 185.22.174.12 > 185.22.174.12 > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/usr/bin/arping -c 1 -I eth3 -A -U -s 185.22.174.12 > 185.22.174.12 > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/bin/echo 3 Table_eth3 > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/ip route add throw 172.17.150.0/24 table Table_eth3 > proto static > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/ip route add throw 169.254.0.0/16 table Table_eth3 > proto static > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/ip route add throw 185.22.174.0/24 table Table_eth3 > proto static > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/ip route add default via 185.22.174.1 table > Table_eth3 proto static > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/ip rule add from 185.22.174.0/24 table Table_eth3 > ./auth.log:Apr 29 09:17:38 r-33-VM sudo: root : TTY=unknown ; PWD=/root ; > USER=root ; COMMAND=/sbin/ip rule add fwmark 3 table Table_eth3 > ./messages:Apr 29 09:17:38 r-33-VM cloud: VR config: executing: > /opt/cloud/bin/ipassoc.sh -A -s -f -l 185.22.174.12/24 -c eth3 -g > 185.22.174.1 -n > ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Adding first ip > 185.22.174.12/24 on interface eth3 > ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Added SourceNAT > 185.22.174.12/24 on interface eth3 > ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Added first ip > 185.22.174.12/24 on interface eth3 > ./messages:Apr 29 09:17:38 r-33-VM cloud: ipassoc.sh:Add routing > 185.22.174.12/24 on interface eth3 > ########################################################### > Host has no logs about this " r-33-VM " VR > ########################################################### > Mgmt server : > ./management-server.log:2015-04-29 12:16:28,550 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-38:ctx-01f0beeb job-260/job-263 ctx-ab6ac568) Seq > 1-3349552222856808115: Sending { Cmd , MgmtId: 115129176880998, via: > 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StartCommand":{"vm":{"id":33,"name":"r-33-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian > GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" > template=domP name=r-33-VM eth2ip=185.22.174.12 eth2mask=255.255.255.0 > gateway=185.22.174.1 eth0ip=172.17.150.1 eth0mask=255.255.255.0 > domain=cs2cloud.internal cidrsize=24 dhcprange=172.17.150.1 > eth1ip=169.254.0.58 eth1mask=255.255.0.0 type=router disable_rp_filter=true > dns1=8.8.8.8 > dns2=8.8.4.4","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"b54e615a272b5f45","params":{},"uuid":"8d1c0a71-1cd2-4639-97f3-13ae9fb28b6d","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"cd2d4a78-c807-42bd-a499-14b32b424925","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary","port":0,"url":"SharedMountPoint://localhost/mnt/primary/?ROLE=Primary&STOREUUID=cd2d4a78-c807-42bd-a499-14b32b424925"}},"name":"ROOT-33","size":308822528,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeId":35,"vmName":"r-33-VM","accountId":2,"format":"QCOW2","id":35,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"localhost","volumeSize":"308822528"}}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"e852e031-11b7-4b63-be7d-03d1229541cd","ip":"185.22.174.12","netmask":"255.255.255.0","gateway":"185.22.174.1","mac":"06:8d:dc:00:00:34","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1700","isolationUri":"vlan://1700","isSecurityGroupEnabled":false,"name":"cloudbr0"},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"082b65b6-24ed-4af0-aede-34ea2bc2003e","ip":"172.17.150.1","netmask":"255.255.255.0","mac":"02:00:25:23:00:07","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1731","isolationUri":"vlan://1731","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"842f837f-0c34-42db-b860-6c4628a91f2c","ip":"169.254.0.58","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:3a","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"172.17.100.4","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.58","port":3922,"interval":6,"retries":100,"name":"r-33-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}},{},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[{"accountId":2,"publicIp":"185.22.174.12","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"broadcastUri":"vlan://1700","vlanGateway":"185.22.174.1","vlanNetmask":"255.255.255.0","vifMacAddress":"06:51:da:00:00:34","networkRate":200,"trafficType":"Public","networkName":"cloudbr0","newNic":false}],"accessDetails":{"zone.network.type":"Advanced","router.name":"r-33-V > > ","router.ip":"169.254.0.58","router.guest.ip":"172.17.150.1"},"wait":0}},{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"services":[{"id":0,"service":"dhcp","processname":"dnsmasq","serviceName":"dnsmasq","servicePath":"/var/run/dnsmasq/dnsmasq.pid","pidFile":"/var/run/dnsmasq/dnsmasq.pid","isDefault":false},{"id":0,"service":"loadbalancing","processname":"haproxy","serviceName":"haproxy","servicePath":"/var/run/haproxy.pid","pidFile":"/var/run/haproxy.pid","isDefault":false},{"id":0,"service":"ssh","processname":"sshd","serviceName":"ssh","servicePath":"/var/run/sshd.pid","pidFile":"/var/run/sshd.pid","isDefault":true},{"id":0,"service":"webserver","processname":"apache2","serviceName":"apache2","servicePath":"/var/run/apache2.pid","pidFile":"/var/run/apache2.pid","isDefault":true}],"accessDetails":{"router.name":"r-33-VM","router.ip":"169.254.0.58","router.guest.ip":"172.17.150.1"},"wait":0}},{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:2e:7b:00:01","vmIpAddress":"172.17.150.190","vmName":"testvps","defaultRouter":"172.17.150.1","defaultDns":"172.17.150.1","duid":"00:03:00:01:02:00:2e:7b:00:01","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"172.17.150.1","router.ip":"169.254.0.58","router.name":"r-33-VM"},"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"172.17.150.190","vmName":"testvps","executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.name":"r-33-VM","router.ip":"169.254.0.58","router.guest.ip":"172.17.150.1"},"wait":0}},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Finish","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] > } > ./management-server.log:2015-04-29 12:17:39,420 DEBUG [c.c.a.t.Request] > (AgentManager-Handler-8:null) Seq 1-3349552222856808115: Processing: { Ans: > , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":33,"name":"r-33-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian > GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" > template=domP name=r-33-VM eth2ip=185.22.174.12 eth2mask=255.255.255.0 > gateway=185.22.174.1 eth0ip=172.17.150.1 eth0mask=255.255.255.0 > domain=cs2cloud.internal cidrsize=24 dhcprange=172.17.150.1 > eth1ip=169.254.0.58 eth1mask=255.255.0.0 type=router disable_rp_filter=true > dns1=8.8.8.8 > dns2=8.8.4.4","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"b54e615a272b5f45","vncAddr":"172.17.100.4","params":{},"uuid":"8d1c0a71-1cd2-4639-97f3-13ae9fb28b6d","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"cd2d4a78-c807-42bd-a499-14b32b424925","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary","port":0,"url":"SharedMountPoint://localhost/mnt/primary/?ROLE=Primary&STOREUUID=cd2d4a78-c807-42bd-a499-14b32b424925"}},"name":"ROOT-33","size":308822528,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","volumeId":35,"vmName":"r-33-VM","accountId":2,"format":"QCOW2","id":35,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"d53c29bb-149b-416e-9303-5b1a3588fbc3","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"localhost","volumeSize":"308822528"}}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"e852e031-11b7-4b63-be7d-03d1229541cd","ip":"185.22.174.12","netmask":"255.255.255.0","gateway":"185.22.174.1","mac":"06:8d:dc:00:00:34","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1700","isolationUri":"vlan://1700","isSecurityGroupEnabled":false,"name":"cloudbr0"},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"082b65b6-24ed-4af0-aede-34ea2bc2003e","ip":"172.17.150.1","netmask":"255.255.255.0","mac":"02:00:25:23:00:07","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1731","isolationUri":"vlan://1731","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"842f837f-0c34-42db-b860-6c4628a91f2c","ip":"169.254.0.58","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:3a","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"result":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack > Release 4.4.1 Mon Sep 29 14:29:20 UTC > 2014","scriptsVersion":"5bccd9c9d4b8d0b6ae66c0128d771789\n","result":true,"details":"Cloudstack > Release 4.4.1 Mon Sep 29 14:29:20 UTC > 2014&5bccd9c9d4b8d0b6ae66c0128d771789\n","wait":0}},{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":0,"bytesReceived":0,"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] > } > ./management-server.log:2015-04-29 12:17:39,905 DEBUG > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-38:ctx-01f0beeb > job-260/job-263 ctx-ab6ac568) Start completed for VM VM[DomainRouter|r-33-VM] > ./management-server.log:2015-04-29 12:17:40,417 DEBUG [c.c.a.t.Request] > (API-Job-Executor-58:ctx-b2a553ab job-260 ctx-fe610701) Seq > 1-3349552222856808119: Sending { Cmd , MgmtId: 115129176880998, via: > 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] > } > ./management-server.log:2015-04-29 12:17:40,463 DEBUG [c.c.a.t.Request] > (API-Job-Executor-58:ctx-b2a553ab job-260 ctx-fe610701) Seq > 1-3349552222856808120: Sending { Cmd , MgmtId: 115129176880998, via: > 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Finish","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] > } > ./management-server.log:2015-04-29 12:17:40,802 DEBUG [c.c.a.t.Request] > (API-Job-Executor-58:ctx-b2a553ab job-260 ctx-fe610701) Seq > 1-3349552222856808121: Sending { Cmd , MgmtId: 115129176880998, via: > 1(node2.cloud.vstoike.ru), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Cleanup","accessDetails":{"router.guest.ip":"172.17.150.1","router.name":"r-33-VM","router.ip":"169.254.0.58"},"wait":0}}] > } > ./management-server.log:2015-04-29 12:17:42,238 DEBUG [c.c.a.t.Request] > (AgentManager-Handler-12:null) Seq 1-3349552222856808122: Processing: { Ans: > , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":336,"bytesReceived":0,"result":true,"details":"","wait":0}}] > } > ./management-server.log:2015-04-29 12:22:42,190 DEBUG [c.c.a.t.Request] > (AgentManager-Handler-6:null) Seq 1-3349552222856808138: Processing: { Ans: > , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":25368,"bytesReceived":0,"result":true,"details":"","wait":0}}] > } > ./management-server.log:2015-04-29 12:27:42,187 DEBUG [c.c.a.t.Request] > (AgentManager-Handler-5:null) Seq 1-3349552222856808154: Processing: { Ans: > , MgmtId: 115129176880998, via: 1, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-33-VM","bytesSent":50400,"bytesReceived":0,"result":true,"details":"","wait":0}}] > } > ######################################################### > Iptables from VR > root@r-33-VM:~# iptables -L -nv -t nat > Chain PREROUTING (policy ACCEPT 14 packets, 951 bytes) > pkts bytes target prot opt in out source > destination > Chain INPUT (policy ACCEPT 14 packets, 951 bytes) > pkts bytes target prot opt in out source > destination > Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) > pkts bytes target prot opt in out source > destination > Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) > pkts bytes target prot opt in out source > destination > 0 0 SNAT all -- * eth3 0.0.0.0/0 0.0.0.0/0 > to:185.22.174.12 > root@r-33-VM:~# iptables -L -nv > Chain INPUT (policy DROP 19 packets, 1444 bytes) > pkts bytes target prot opt in out source > destination > 891 77029 NETWORK_STATS all -- * * 0.0.0.0/0 > 0.0.0.0/0 > 0 0 ACCEPT all -- * * 0.0.0.0/0 224.0.0.18 > 0 0 ACCEPT all -- * * 0.0.0.0/0 225.0.0.50 > 0 0 ACCEPT all -- eth0 * 0.0.0.0/0 0.0.0.0/0 > state RELATED,ESTABLISHED > 835 76520 ACCEPT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 > state RELATED,ESTABLISHED > 74 6112 ACCEPT all -- eth2 * 0.0.0.0/0 0.0.0.0/0 > state RELATED,ESTABLISHED > 8 672 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 > 1 93 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 > 0 0 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 > udp dpt:67 > 3 195 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 > udp dpt:53 > 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 > tcp dpt:53 > 14 840 ACCEPT tcp -- eth1 * 0.0.0.0/0 0.0.0.0/0 > state NEW tcp dpt:3922 > 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 > state NEW tcp dpt:80 > 0 0 ACCEPT tcp -- eth0 * 172.17.150.0/24 0.0.0.0/0 > state NEW tcp dpt:8080 > Chain FORWARD (policy DROP 886 packets, 74424 bytes) > pkts bytes target prot opt in out source > destination > 887 74508 NETWORK_STATS all -- * * 0.0.0.0/0 > 0.0.0.0/0 > 0 0 ACCEPT all -- eth0 eth1 0.0.0.0/0 0.0.0.0/0 > state RELATED,ESTABLISHED > 0 0 ACCEPT all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 > state RELATED,ESTABLISHED > 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 > state NEW > 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 > state RELATED,ESTABLISHED > 895 75180 FW_OUTBOUND all -- eth0 eth2 0.0.0.0/0 > 0.0.0.0/0 > 0 0 ACCEPT all -- eth3 eth0 0.0.0.0/0 0.0.0.0/0 > state RELATED,ESTABLISHED > 0 0 FW_OUTBOUND all -- eth0 eth3 0.0.0.0/0 > 0.0.0.0/0 > Chain OUTPUT (policy ACCEPT 929 packets, 204K bytes) > pkts bytes target prot opt in out source > destination > 986 214K NETWORK_STATS all -- * * 0.0.0.0/0 > 0.0.0.0/0 > Chain FW_OUTBOUND (2 references) > pkts bytes target prot opt in out source > destination > 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 > state RELATED,ESTABLISHED > Chain NETWORK_STATS (3 references) > pkts bytes target prot opt in out source > destination > 887 74508 all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0 > 0 0 all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 > 0 0 tcp -- !eth0 eth2 0.0.0.0/0 0.0.0.0/0 > 0 0 tcp -- eth2 !eth0 0.0.0.0/0 0.0.0.0/0 > 0 0 all -- eth0 eth3 0.0.0.0/0 0.0.0.0/0 > 0 0 all -- eth3 eth0 0.0.0.0/0 0.0.0.0/0 > 0 0 tcp -- !eth0 eth3 0.0.0.0/0 0.0.0.0/0 > 0 0 tcp -- eth3 !eth0 0.0.0.0/0 0.0.0.0/0 > root@r-33-VM:~# iptables -L -nv -t nat > Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) > pkts bytes target prot opt in out source > destination > 0 0 SNAT all -- * eth3 0.0.0.0/0 0.0.0.0/0 > to:185.22.174.12 > p.s. i think something wrong with a mechanism which is propagating rules ( > firewall rules ) to VR -- This message was sent by Atlassian JIRA (v6.3.4#6332)