Hi Dag, When I tried your method and created a NIC-less bridge, following are the contents of my ifcfg-* network files:
*ifcfg-cloudbr0:* TYPE=Bridge PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=no NAME=cloudbr0 UUID=25aabe73-8e11-408f-a4ec-c03b26d3aa6e DEVICE=cloudbr0 ONBOOT=yes IPADDR=172.16.20.13 PREFIX=16 GATEWAY=172.16.0.1 DNS1=8.8.8.8 DNS2=172.16.0.1 NM_CONTROLLED=no *ifcfg-cloudbr1:* TYPE=Bridge IPV4_FAILURE_FATAL=no NAME=cloudbr1 UUID=25aabe73-8e11-408f-a4ec-c03b26d3aa6e DEVICE=cloudbr1 ONBOOT=yes NM_CONTROLLED=no *ifcfg-eno1:* TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=no NAME=eno1 UUID=25aabe73-8e11-408f-a4ec-c03b26d3aa6e DEVICE=eno1 ONBOOT=yes IPADDR=172.16.20.13 PREFIX=16 GATEWAY=172.16.0.1 DNS1=8.8.8.8 DNS2=172.16.0.1 NM_CONTROLLED=no BRIDGE=cloudbr0 *brctl show output:* [root@srvr3 ~]# brctl show bridge name bridge id STP enabled interfaces cloud0 8000.000000000000 no cloudbr0 8000.3464a92a09f3 no eno1 cloudbr1 8000.000000000000 no virbr0 8000.5254002dabdb yes virbr0-nic *when adding a host in advanced zone it shows the following error: Could not find network 'cloudbr1'* 2018-04-04 02:03:11,887 DEBUG [c.c.u.s.SSHCmdHelper] (qtp510113906-14:ctx-707b53e5 ctx-8d49ccb3) (logid:dff92f23) Executing cmd: /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/ 2018-04-04 02:03:15,686 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp510113906-14:ctx-707b53e5 ctx-8d49ccb3) (logid:dff92f23) Succeeded to import certificate in the keystore for agent on the KVM host: 172.16.20.13. Agent secured and trusted. 2018-04-04 02:03:15,688 DEBUG [c.c.u.s.SSHCmdHelper] (qtp510113906-14:ctx-707b53e5 ctx-8d49ccb3) (logid:dff92f23) Executing cmd: cloudstack-setup-agent -m 172.16.20.13 -z 1 -p 1 -c 1 -g 1fd67886-c5d9-3464-ac73-46689258b34e -a --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr1 --hypervisor=kvm 2018-04-04 02:03:19,674 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-af4b26a6) (logid:4c5c40d4) Begin cleanup expired async-jobs 2018-04-04 02:03:19,683 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-af4b26a6) (logid:4c5c40d4) End cleanup expired async-jobs 2018-04-04 02:03:20,022 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-f1d46df0) (logid:a021b44c) Found 0 routers to update status. 2018-04-04 02:03:20,025 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-f1d46df0) (logid:a021b44c) Found 0 VPC networks to update Redundant State. 2018-04-04 02:03:20,029 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-f1d46df0) (logid:a021b44c) Found 0 networks to update RvR status. 2018-04-04 02:03:22,518 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-9d047585) (logid:3b4da034) Skip capacity scan as there is no Primary Storage in 'Up' state 2018-04-04 02:03:29,677 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-4378508d) (logid:9de3110e) Begin cleanup expired async-jobs 2018-04-04 02:03:29,695 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-4378508d) (logid:9de3110e) End cleanup expired async-jobs 2018-04-04 02:03:35,796 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-a6998a7a) (logid:174df0f0) AutoScaling Monitor is running... 2018-04-04 02:03:35,812 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-51d8b17f) (logid:8d7cca2e) HostStatsCollector is running... 2018-04-04 02:03:35,812 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-db3df7da) (logid:c38b9f19) StorageCollector is running... 2018-04-04 02:03:39,675 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-26aea785) (logid:48c9a5ba) Begin cleanup expired async-jobs 2018-04-04 02:03:39,686 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-26aea785) (logid:48c9a5ba) End cleanup expired async-jobs 2018-04-04 02:03:49,677 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-ec993f07) (logid:98d10d6c) Begin cleanup expired async-jobs 2018-04-04 02:03:49,692 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-ec993f07) (logid:98d10d6c) End cleanup expired async-jobs 2018-04-04 02:03:50,021 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-592f1702) (logid:39d0b341) Found 0 routers to update status. 2018-04-04 02:03:50,024 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-592f1702) (logid:39d0b341) Found 0 VPC networks to update Redundant State. 2018-04-04 02:03:50,027 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-592f1702) (logid:39d0b341) Found 0 networks to update RvR status. 2018-04-04 02:03:52,519 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-099c555a) (logid:1e1eb23a) Skip capacity scan as there is no Primary Storage in 'Up' state 2018-04-04 02:03:59,674 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-33034d0a) (logid:1c504485) Begin cleanup expired async-jobs 2018-04-04 02:03:59,684 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-33034d0a) (logid:1c504485) End cleanup expired async-jobs 2018-04-04 02:04:09,675 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-c5079da9) (logid:8b320bb8) Begin cleanup expired async-jobs 2018-04-04 02:04:09,689 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-c5079da9) (logid:8b320bb8) End cleanup expired async-jobs 2018-04-04 02:04:19,505 DEBUG [c.c.u.s.SSHCmdHelper] (qtp510113906-14:ctx-707b53e5 ctx-8d49ccb3) (logid:dff92f23) SSH command: cloudstack-setup-agent -m 172.16.20.13 -z 1 -p 1 -c 1 -g 1fd67886-c5d9-3464-ac73-46689258b34e -a --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr1 --hypervisor=kvm SSH command output:Starting to configure your system: Configure SElinux ... [OK] Configure Network ... [OK] Configure Libvirt ... [OK] Configure Firewall ... [OK] Configure Nfs ... [OK] Configure cloudAgent ... [OK] CloudStack Agent setup is done! 2018-04-04 02:04:19,670 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-3953edaa) (logid:223c6612) Begin cleanup expired async-jobs 2018-04-04 02:04:19,678 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-3953edaa) (logid:223c6612) End cleanup expired async-jobs 2018-04-04 02:04:20,020 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-d51c2a8c) (logid:31404001) Found 0 routers to update status. 2018-04-04 02:04:20,022 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-d51c2a8c) (logid:31404001) Found 0 VPC networks to update Redundant State. 2018-04-04 02:04:20,023 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-d51c2a8c) (logid:31404001) Found 0 networks to update RvR status. 2018-04-04 02:04:22,514 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-03cb5ba3) (logid:b63f8dea) Skip capacity scan as there is no Primary Storage in 'Up' state 2018-04-04 02:04:22,597 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-7-thread-1:null) (logid:) A client/agent attempting connection from address=172.16.20.13 has presented these certificate(s): Certificate [1] : Serial: da4b61927e0b90f4 Not Before:Tue Apr 03 14:03:11 IST 2018 Not After:Thu Apr 04 02:03:11 IST 2019 Signature Algorithm:SHA256withRSA Version:3 Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=srvr3.cloud.priv Issuer DN:CN=ca.cloudstack.apache.org Alternative Names:[[7, 172.16.20.13], [2, 172.16.20.13]] Certificate [2] : Serial: ca8575145c21dcc5 Not Before:Tue Apr 03 13:55:06 IST 2018 Not After:Fri Mar 27 01:55:06 IST 2048 Signature Algorithm:SHA256withRSA Version:3 Subject DN:CN=ca.cloudstack.apache.org Issuer DN:CN=ca.cloudstack.apache.org Alternative Names:null 2018-04-04 02:04:22,600 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-7-thread-1:null) (logid:) Client/agent connection from ip=172.16.20.13 has been validated and trusted. 2018-04-04 02:04:23,304 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null) (logid:) Seq 5-0: Scheduling the first command { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":1,"cpus":4,"speed":2000,"memory":11202912256,"dom0MinMemory":1073741824,"poolSync":false,"supportsClonedVolumes":false,"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"3.10.0-693.el7.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"7.4.1708","Host.OS":"CentOS"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":"1fd67886-c5d9-3464-ac73-46689258b34e-LibvirtComputingResource","name":"srvr3.cloud.priv","id":5,"version":"4.11.0.0","iqn":"iqn.1994-05.com.redhat:895c24ab79d","publicIpAddress":"172.16.20.13","publicNetmask":"255.255.0.0","publicMacAddress":"34:64:a9:2a:09:f3","privateIpAddress":"172.16.20.13","privateMacAddress":"34:64:a9:2a:09:f3","privateNetmask":"255.255.0.0","storageIpAddress":"172.16.20.13","storageNetmask":"255.255.0.0","storageMacAddress":"34:64:a9:2a:09:f3","resourceName":"LibvirtComputingResource","gatewayIpAddress":"172.16.0.1","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"b0c95dc8-bd46-4d3d-a4ee-90cb869d035e","host":"172.16.20.13","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":486035005440,"availableBytes":478874988544},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"1fd67886-c5d9-3464-ac73-46689258b34e-LibvirtComputingResource","name":"srvr3.cloud.priv","id":5,"version":"4.11.0.0","resourceName":"LibvirtComputingResource","wait":0}}] } 2018-04-04 02:04:23,315 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Seq 5-0: Processing the first command { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":1,"cpus":4,"speed":2000,"memory":11202912256,"dom0MinMemory":1073741824,"poolSync":false,"supportsClonedVolumes":false,"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"3.10.0-693.el7.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"7.4.1708","Host.OS":"CentOS"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"1","pod":"1","cluster":"1","guid":"1fd67886-c5d9-3464-ac73-46689258b34e-LibvirtComputingResource","name":"srvr3.cloud.priv","id":5,"version":"4.11.0.0","iqn":"iqn.1994-05.com.redhat:895c24ab79d","publicIpAddress":"172.16.20.13","publicNetmask":"255.255.0.0","publicMacAddress":"34:64:a9:2a:09:f3","privateIpAddress":"172.16.20.13","privateMacAddress":"34:64:a9:2a:09:f3","privateNetmask":"255.255.0.0","storageIpAddress":"172.16.20.13","storageNetmask":"255.255.0.0","storageMacAddress":"34:64:a9:2a:09:f3","resourceName":"LibvirtComputingResource","gatewayIpAddress":"172.16.0.1","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"b0c95dc8-bd46-4d3d-a4ee-90cb869d035e","host":"172.16.20.13","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":486035005440,"availableBytes":478874988544},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"1","pod":"1","guid":"1fd67886-c5d9-3464-ac73-46689258b34e-LibvirtComputingResource","name":"srvr3.cloud.priv","id":5,"version":"4.11.0.0","resourceName":"LibvirtComputingResource","wait":0}}] } 2018-04-04 02:04:23,325 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BareMetalDiscoverer 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetscalerElement 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to HypervServerDiscoverer 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to XcpServerDiscoverer 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NiciraNvp 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BrocadeVcsElement 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovm3Discoverer 2018-04-04 02:04:23,326 DEBUG [c.c.h.o.r.Ovm3Discoverer] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) createHostVOForConnectedAgent: Host[-0-null] 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to LxcServerDiscoverer 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetworkUsageManagerImpl 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to PremiumSecondaryStorageManagerImpl 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovs 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to ConsoleProxyManagerImpl 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to OvmDiscoverer 2018-04-04 02:04:23,326 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to KvmServerDiscoverer 2018-04-04 02:04:23,486 DEBUG [c.c.r.ResourceState] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Resource state update: [id = 1; name = srvr3.cloud.priv; old state = Creating; event = InternalCreated; new state = Enabled] 2018-04-04 02:04:23,487 DEBUG [c.c.h.Status] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Transition:[Resource state = Enabled, Agent event = AgentConnected, Host id = 1, name = srvr3.cloud.priv] 2018-04-04 02:04:23,512 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) create ClusteredAgentAttache for 1 2018-04-04 02:04:23,516 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Connect to listener: XcpServerDiscoverer 2018-04-04 02:04:23,517 DEBUG [c.c.h.x.d.XcpServerDiscoverer] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Not XenServer so moving on. 2018-04-04 02:04:23,517 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Connect to listener: HypervServerDiscoverer 2018-04-04 02:04:23,517 DEBUG [c.c.h.h.d.HypervServerDiscoverer] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Not Hyper-V hypervisor, so moving on. 2018-04-04 02:04:23,517 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Connect to listener: ClusteredVirtualMachineManagerImpl 2018-04-04 02:04:23,517 DEBUG [c.c.v.VirtualMachineManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Received startup command from hypervisor host. host id: 1 2018-04-04 02:04:23,517 INFO [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Reset VM power state sync for host: 1 2018-04-04 02:04:23,547 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Connect to listener: SecurityGroupListener 2018-04-04 02:04:23,547 INFO [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Received a host startup notification 2018-04-04 02:04:23,552 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Seq 1-1865897620614938625: Sending { Cmd , MgmtId: 55884536686396, via: 1(srvr3.cloud.priv), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CleanupNetworkRulesCmd":{"interval":1951,"wait":0}}] } 2018-04-04 02:04:23,555 INFO [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Scheduled network rules cleanup, interval=1951 2018-04-04 02:04:23,556 INFO [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Received a host startup notification 2018-04-04 02:04:23,556 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Connect to listener: NetworkOrchestrator 2018-04-04 02:04:23,563 DEBUG [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Host's hypervisorType is: KVM 2018-04-04 02:04:23,580 DEBUG [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending CheckNetworkCommand to check the Network is setup correctly on Agent 2018-04-04 02:04:23,600 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Seq 1-1865897620614938626: Sending { Cmd , MgmtId: 55884536686396, via: 1(srvr3.cloud.priv), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.CheckNetworkCommand":{"networkInfoList":[{"physicalNetworkId":200,"privateNetworkName":"cloudbr0","publicNetworkName":"cloudbr0","guestNetworkName":"cloudbr1"}],"wait":0}}] } 2018-04-04 02:04:23,607 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null) (logid:) Seq 1-1865897620614938625: Processing: { Ans: , MgmtId: 55884536686396, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] } 2018-04-04 02:04:23,730 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Ping from 1(srvr3.cloud.priv) 2018-04-04 02:04:23,731 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:null) (logid:) Process host VM state report from ping process. host: 1 2018-04-04 02:04:23,731 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:null) (logid:) Process VM state report. host: 1, number of records in report: 0 2018-04-04 02:04:23,734 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:null) (logid:) Done with process of VM state report. host: 1 2018-04-04 02:04:23,741 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-4:null) (logid:) PingMap for agent: 1 will not be updated because agent is no longer in the PingMap 2018-04-04 02:04:23,745 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Not processing PingRoutingCommand for agent id=0; can't find the host in the DB 2018-04-04 02:04:23,953 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:null) (logid:) Seq 1-1865897620614938626: Processing: { Ans: , MgmtId: 55884536686396, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.CheckNetworkAnswer":{"_reconnect":false,"result":false,"details":"Can not find network: cloudbr1","wait":0}}] } 2018-04-04 02:04:23,953 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Seq 1-1865897620614938626: Received: { Ans: , MgmtId: 55884536686396, via: 1(srvr3.cloud.priv), Ver: v1, Flags: 110, { CheckNetworkAnswer } } 2018-04-04 02:04:23,953 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Details from executing class com.cloud.agent.api.CheckNetworkCommand: Can not find network: cloudbr1 2018-04-04 02:04:23,954 WARN [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Unable to setup agent 1 due to Can not find network: cloudbr1 2018-04-04 02:04:23,956 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-5:null) (logid:) Seq 1-1865897620614938626: No more commands found 2018-04-04 02:04:23,957 WARN [o.a.c.alerts] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) AlertType:: 7 | dataCenterId:: 1 | podId:: 1 | clusterId:: null | message:: Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: cloudbr1 2018-04-04 02:04:23,997 INFO [c.c.u.e.CSExceptionErrorCode] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Could not find exception: com.cloud.exception.ConnectionException in error code list for exceptions 2018-04-04 02:04:23,998 WARN [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Monitor NetworkOrchestrator says there is an error in the connect process for 1 due to Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: cloudbr1 2018-04-04 02:04:23,998 INFO [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Host 1 is disconnecting with event AgentDisconnected 2018-04-04 02:04:24,000 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) The next status of agent 1is Alert, current status is Connecting 2018-04-04 02:04:24,001 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Deregistering link for 1 with state Alert 2018-04-04 02:04:24,001 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Remove Agent : 1 2018-04-04 02:04:24,001 DEBUG [c.c.a.m.ConnectedAgentAttache] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Processing Disconnect. 2018-04-04 02:04:24,001 DEBUG [c.c.a.m.AgentAttache] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Seq 1-1865897620614938625: Sending disconnect to class com.cloud.network.security.SecurityGroupListener 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: org.apache.cloudstack.engine.orchestration.NetworkOrchestrator 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.deploy.DeploymentPlanningManagerImpl 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl 2018-04-04 02:04:24,002 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener 2018-04-04 02:04:24,005 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor 2018-04-04 02:04:24,005 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.network.router.VirtualNetworkApplianceManagerImpl 2018-04-04 02:04:24,005 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener 2018-04-04 02:04:24,005 DEBUG [c.c.n.NetworkUsageManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Disconnected called on 1 with status Alert 2018-04-04 02:04:24,006 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener 2018-04-04 02:04:24,006 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener 2018-04-04 02:04:24,006 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener 2018-04-04 02:04:24,006 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.agent.manager.AgentManagerImpl$BehindOnPingListener 2018-04-04 02:04:24,006 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.agent.manager.AgentManagerImpl$SetHostParamsListener 2018-04-04 02:04:24,006 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener 2018-04-04 02:04:24,007 DEBUG [c.c.h.Status] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvr3.cloud.priv] 2018-04-04 02:04:24,028 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Notifying other nodes of to disconnect 2018-04-04 02:04:24,034 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Failed to handle host connection: com.cloud.exception.ConnectionException: Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: cloudbr1 2018-04-04 02:04:24,046 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-1:ctx-0bf7b058) (logid:f7a4d646) Can not send command com.cloud.agent.api.ReadyCommand due to Host 1 is not up 2018-04-04 02:04:25,767 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-f7869282) (logid:9ff5ad9c) Resetting hosts suitable for reconnect 2018-04-04 02:04:25,770 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-f7869282) (logid:9ff5ad9c) Completed resetting hosts suitable for reconnect 2018-04-04 02:04:25,770 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-f7869282) (logid:9ff5ad9c) Acquiring hosts for clusters already owned by this management server 2018-04-04 02:04:25,771 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-f7869282) (logid:9ff5ad9c) Completed acquiring hosts for clusters already owned by this management server 2018-04-04 02:04:25,772 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-f7869282) (logid:9ff5ad9c) Acquiring hosts for clusters not owned by any management server 2018-04-04 02:04:25,773 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-f7869282) (logid:9ff5ad9c) Completed acquiring hosts for clusters not owned by any management server 2018-04-04 02:04:29,219 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-8-thread-1:null) (logid:) A client/agent attempting connection from address=172.16.20.13 has presented these certificate(s): Certificate [1] : Please help me understand what am I doing wrong here. I have tried several approaches but none seem to work (like creating the bridge through virt-manager GUI or typing the IP configuration of targeted guest network in the cloudbr1 file). Also, the method you suggested here where L2 network does not leave the host, that would basically mean we do not need to have a physical network in our LAN? like I currently have 192.168.20.0/24 and 172.16.20.0/16 in my physical network, but I could possibly create a isolated guest network of 10.0.2.0/24 ? I am also attaching the management-server logs if it would not be stripped by mailing system. The logs of agent had nothing interesting or related to network configuration errors, so not attaching them here. Thanks, Parth Patel On Thu, 29 Mar 2018 at 16:18 Parth Patel <parthpatel2...@gmail.com> wrote: > Thanks Dag. Appreciate it. Will try this out. > > On Thu, 29 Mar 2018 at 16:02 Dag Sonstebo <dag.sonst...@shapeblue.com> > wrote: > >> Hi Parth, >> >> If you want a KVM networking introduction take a look at my blog post >> from a couple of years back – this is still valid: >> http://www.shapeblue.com/networking-kvm-for-cloudstack/ >> >> In short – you don’t set up VLAN tagging for isolated networks on the KVM >> host – you set up the bridge and then specify your VLAN range when you set >> up your zone in CloudStack. CloudStack then takes care of creating the >> isolated VLAN isolated networks on the host. So in short – you create your >> bridges, then use the bridge names in the advanced zone setup. >> >> Virtual bridge – yes this is similar to the cloud0 bridge, and yes you >> create the bridge without a physical interface. >> >> Regards, >> Dag Sonstebo >> Cloud Architect >> ShapeBlue >> >> >> dag.sonst...@shapeblue.com >> www.shapeblue.com >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK >> @shapeblue >> >> >> >> On 29/03/2018, 11:14, "Parth Patel" <parthpatel2...@gmail.com> wrote: >> >> Hi Dag, >> >> Thanks for helping me understand the requirement of advanced >> networking. >> Sorry if I have missed something obvious or my question seems stupid, >> but I >> am just starting to learn. Can you help me out on how to setup VLAN >> "tagging" on one machine? I have tried several methods and tutorials I >> could find on the internet for VLANs, but none mention "tagging". >> >> Also, I do not fully understand private virtual bridge..... Means I >> create >> an interface file for bridge but mention no physical interface >> device? Is >> it similar to how cloud0 is configured for link local network of >> System >> VMs? I could probably do that, but I don't know much about >> configuring VLAN >> tagging. I would appreciate if you could give me some guidance or >> point me >> where you think some good documentation is given for CentOS/RHEL >> hosts for >> configuring bridges with VLAN tagging (I have tried but failed to >> understand most of them). I am especially stuck at understanding this >> "tagging" of VLANs. >> >> Thanks, >> Parth Patel >> >> On Thu, 29 Mar 2018 at 15:17 Dag Sonstebo <dag.sonst...@shapeblue.com >> > >> wrote: >> >> > Hi Parth, >> > >> > Yes and no. >> > >> > No – you cannot do advanced zones with *all three* KVM hosts and >> advanced >> > networking without using VLANs (or another isolation mechanism) and >> still >> > expect traffic to flow between VMs/VRs on different KVM hosts. >> > >> > Yes – you can probably do this *on a single KVM host* – but you >> will have >> > to use VLAN tagging internally – this can however be done on a >> virtual >> > bridge interface, i.e. the L2 traffic doesn’t ever leave that host. >> > >> > Without deep diving into this I think it would look like this: >> > >> > Physical eth0 -> cloudbr0 > handles management and public >> > No nic -> private virtual bridge cloudbr1 > handles isolated guest >> traffic >> > but allows for isolated VLANs internally on the host >> > >> > Regards, >> > Dag Sonstebo >> > Cloud Architect >> > ShapeBlue >> > >> > >> > dag.sonst...@shapeblue.com >> > www.shapeblue.com >> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK >> > @shapeblue >> > >> > >> > >> > On 29/03/2018, 09:25, "Parth Patel" <parthpatel2...@gmail.com> >> wrote: >> > >> > Hi Dag, >> > >> > Thanks for the reply. I am trying to use Shapeblue CCS >> (Container as a >> > Service) with ACS, but for that Isolated networks are required >> which >> > are >> > only available in Advanced Zone. Further, I want to explore >> Cloudstack >> > further and am also aiming to test and configure other advanced >> > features >> > such as load balancing and auto scaling without netscaler >> device. For >> > that >> > I badly need Advanced Zone networking (especially isolated >> networks >> > offerings). I just want to know if Advanced Zone can succesfully >> > function >> > with two networks, one physcial NIC and no VLAN tagging. >> > >> > Thanks, >> > Parth Patel >> > >> > On Thu, 29 Mar 2018 at 13:48 Dag Sonstebo < >> dag.sonst...@shapeblue.com> >> > wrote: >> > >> > > Hi Parth, >> > > >> > > Not sure if I follow. Generally, your management network is >> untagged, >> > > whilst your public and isolated networks tagged. The >> underlying idea >> > of >> > > advanced zones is you must have network isolation between >> multiple >> > guest >> > > networks, otherwise you have no privacy/security. You can do >> this >> > either at >> > > L2 with VLAN tagging, which is the simplest, or with L3 using >> > various SDN >> > > overlay network solutions (more complicated and comes at a >> cost). >> > > >> > > If you don’t want to tag anything you’re probably better off >> using >> > basic >> > > networks, where I believe you could use a single flat subnet >> (happy >> > to be >> > > proven wrong). >> > > >> > > Regards, >> > > Dag Sonstebo >> > > Cloud Architect >> > > ShapeBlue >> > > >> > > >> > > dag.sonst...@shapeblue.com >> > > www.shapeblue.com >> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK >> > > @shapeblue >> > > >> > > >> > > >> > > On 29/03/2018, 08:48, "Parth Patel" <parthpatel2...@gmail.com >> > >> > wrote: >> > > >> > > Hi all, >> > > >> > > After banging my head with different network configuration >> > > permutations, I >> > > don't understand what is the issue with Network Guru here >> and >> > why it >> > > can't >> > > implement the isolated guest network. I just want to know >> if >> > Advanced >> > > Zone >> > > can be successfully setup or has someone configured an >> advanced >> > zone >> > > using >> > > untagged VLAN traffic? >> > > >> > > I have the following configuration of components: >> > > - I have 3 (16 GB Ram and 4 Cores) machines each with 1 >> physical >> > NIC. >> > > - I have two networks: 192.168.20.0/24 (using this for >> isolated >> > guest >> > > network) and 172.16.20.0/16 (management server and NFS >> servers >> > > network) >> > > - I am using KVM hypervisor and NFS for storage. >> > > - Currently, the output of brctl show is (when the >> Cloudstack is >> > not >> > > running, other wise the interface are populated with >> three vnets >> > for >> > > cloud0 >> > > and 4-5 vnets for cloudbr0): >> > > bridge name bridge id STP enabled >> > interfaces >> > > cloud0 8000.000000000000 no >> > > cloudbr0 8000.3464a92a083a no >> > eno1 >> > > virbr0 8000.525400daae23 yes >> > virbr0-nic >> > > >> > > My earlier doubt was if I can configure advanced zone >> with one >> > physical >> > > interface available in each host, but that was resolved >> when I >> > read >> > > this >> > > post of ShankerBalan: >> > > >> > > >> > >> https://shankerbalan.net/blog/cloudstack-simple-advanced-network-example/ >> > > >> > > ACS throws InsufficientVirtualNetworkCapacity exception >> and >> > lines like: >> > > "NetworkGuru can't implement network [275||15]" are >> printed in >> > > management >> > > server logs when I try to create a simple CentOS 5.5 >> NoGUI KVM >> > instance >> > > after a complete and fresh install of ACS (even of >> CentOS). >> > > >> > > My main doubt here is if I can successfully configure an >> > advanced zone >> > > with >> > > two networks but with untagged VLAN traffic ? I can't >> currently >> > > configure >> > > the router or switches to allow tagged VLAN networking as >> I am >> > doing >> > > this >> > > project in my university. But, I have requested and gained >> > access to >> > > the >> > > mentioned two networks: 192.168.20.0/24 and >> 172.16.20.0/16 and >> > both >> > > networks are pingable and have internet access across all >> three >> > > machines. >> > > Can anyone help me with this please? >> > > >> > > Thanks, >> > > Parth Patel >> > > >> > > >> > > >> > >> > >> > >> >> >>