Failed to create the 33rd static nat
I have a weird problem in CloudStack 4.11.2. And want to see if there is any suggestion on how to debug / fix this issue. I created a VPC network, with one guest network. I have added 32 VMs, requested 32 public IPs, and setup static NAT to associate the IP to each of the VM. These works without any issue. However, then I added one more VM, and request 1 public IP. When I try to associate the public IP with the VM, I get the error that it failed to enable the static NAT: 2019-02-07 00:52:41,474 WARN [c.c.n.r.RulesManagerImpl] (qtp788117692-1109:ctx-5b6f9e33 ctx-351aca5a) (logid:56d8d140) Failed to create static nat rule due to com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Unable to apply static nat rules on router at org.apache.cloudstack.network.topology.BasicNetworkTopology.applyRules(BasicNetworkTopology.java:403) at org.apache.cloudstack.network.topology.BasicNetworkTopology.applyStaticNats(BasicNetworkTopology.java:245) at com.cloud.network.element.VirtualRouterElement.applyStaticNats(VirtualRouterElement.java:599) at com.cloud.network.IpAddressManagerImpl.applyStaticNats(IpAddressManagerImpl.java:1899) at com.cloud.network.rules.RulesManagerImpl.applyStaticNatForIp(RulesManagerImpl.java:1338) at com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:598) It seems that the virtual route has problem associating the 33rd static NAT. We have tried CloudStack 4.9.3 and does not have this problem. Is there any suggestion on what I should do? Thanks Ivan
Re: Unable to schedule async job
Dag, That's good idea. How can I update log to debug / trace mode? Thanks Ivan From: Dag Sonstebo To: "users@cloudstack.apache.org" Date: 2019/01/10 11:40 AM Subject:Re: Unable to schedule async job Hi Ivan, Probably a good idea to bump your logging up to debug or trace – these show the SQL queries being prepared in the logs. Regards, Dag Sonstebo Cloud Architect ShapeBlue From: Ivan X Yue Reply-To: "users@cloudstack.apache.org" Date: Thursday, 10 January 2019 at 16:18 To: "users@cloudstack.apache.org" Subject: Re: Unable to schedule async job Hi, Dag, Thank for the reply. Below is my async_job table. Today, I try to delete some VM. Interestingly, I can delete some VM and some are failed withthis error: Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '387' for key 'PRIMARY' But in the async_job table, I don't see id=387 there. I have attached part of the Management Server log wrt the delete VM action. One minor correction. I am actually using cloudstack 4.11.1 instead of 4.9.2. MariaDB [cloud]> select id, job_cmd from async_job order by id; +-+-+ | id | job_cmd | +-+-+ | 275 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 276 | NULL| | 281 | NULL| | 287 | NULL| | 288 | NULL| | 292 | NULL| | 305 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 307 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 309 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 313 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd | | 314 | NULL| | 315 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd | | 316 | com.cloud.vm.VmWorkStart| | 317 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 319 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 321 | NULL| | 322 | com.cloud.vm.VmWorkStop | | 323 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 327 | com.cloud.vm.VmWorkStart| | 328 | com.cloud.vm.VmWorkStart| | 329 | NULL| | 330 | com.cloud.vm.VmWorkStop | | 332 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 334 | com.cloud.vm.VmWorkStop | | 335 | com.cloud.vm.VmWorkStart| | 336 | com.cloud.vm.VmWorkStart| | 338 | org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin | | 340 | com.cloud.vm.VmWorkStop | | 341 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 342 | com.cloud.vm.VmWorkStop | | 343 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd| | 344 | com.cloud.vm.VmWorkStart| | 345 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 346 | com.cloud.vm.VmWorkStop | | 347 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd | | 348 | com.cloud.vm.VmWorkStop | | 349 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd| | 350 | com.cloud.vm.VmWorkStart| | 351 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 352 | com.cloud.vm.VmWorkStop | | 354 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 355 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 356 | com.cloud.vm.VmWorkStop | | 357 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots| | 358 | com.cloud.vm.VmWorkStop | | 359 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 360 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 361 | com.cloud.vm.VmWorkStop | | 362 | com.cloud.vm.sn
Re: Unable to schedule async job
370 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 372 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 374 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 376 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd | | 377 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 378 | com.cloud.vm.VmWorkStop | | 380 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 381 | com.cloud.vm.VmWorkStop | | 383 | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin| | 384 | com.cloud.vm.VmWorkStart| | 385 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin | | 386 | com.cloud.vm.VmWorkStop | +-+-+ Thanks Ivan Yue Development Manager - CLM SaaS Operations Persistent Systems Limited Email: ivan...@ca.ibm.com Email: ivan_...@persistent.com From: Dag Sonstebo To: "users@cloudstack.apache.org" Date: 2019/01/10 05:01 AM Subject:Re: Unable to schedule async job What does your async_job table say around id>330? Regards, Dag Sonstebo Cloud Architect ShapeBlue On 10/01/2019, 05:34, "Ivan X Yue" wrote: Hi, I am using CloudStack 4.9.2 with KVM hypervisors. Today, I find that the hypervisor is not responding, and therefore I restart it. After that, I find that virtual routers are stopped. When I try to start them, I keep getting "Unable to schedule async job" error. From the management-server.log, I see some exception related to MySQL: Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '333' for key 'PRIMARY' Is there any cleanup that I need to do in the database? How can I do that? Here is the full stacktrace of the exception that I get: 2019-01-09 12:06:31,417 WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-3248f525 job-332 ctx-d797e8bd) (logid:d37b7ec3) Unable to schedule async job for command com.cloud.vm.VmWorkStop, unexpected exception. javax.persistence.EntityExistsException: Entity already exists: at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231) at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50) at com.cloud.utils.db.Transaction.execute(Transaction.java:40) at com.cloud.utils.db.Transaction.execute(Transaction.java:47) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231) at com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4498) at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600) at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513) at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:502) at com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:491) at com.cloud.network.router.NetworkHelperImpl.destroyRouter(NetworkHelperImpl.java:253) at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.destroyRouter(VirtualNetworkApplianceManagerImpl.java:350) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop
Unable to schedule async job
Hi, I am using CloudStack 4.9.2 with KVM hypervisors. Today, I find that the hypervisor is not responding, and therefore I restart it. After that, I find that virtual routers are stopped. When I try to start them, I keep getting "Unable to schedule async job" error. >From the management-server.log, I see some exception related to MySQL: Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '333' for key 'PRIMARY' Is there any cleanup that I need to do in the database? How can I do that? Here is the full stacktrace of the exception that I get: 2019-01-09 12:06:31,417 WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-3248f525 job-332 ctx-d797e8bd) (logid:d37b7ec3) Unable to schedule async job for command com.cloud.vm.VmWorkStop, unexpected exception. javax.persistence.EntityExistsException: Entity already exists: at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231) at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50) at com.cloud.utils.db.Transaction.execute(Transaction.java:40) at com.cloud.utils.db.Transaction.execute(Transaction.java:47) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231) at com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4498) at com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600) at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513) at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:502) at com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:491) at com.cloud.network.router.NetworkHelperImpl.destroyRouter(NetworkHelperImpl.java:253) at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.destroyRouter(VirtualNetworkApplianceManagerImpl.java:350) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy239.destroyRouter(Unknown Source) at org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd.execute(DestroyRouterCmd.java:103) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '333' for key 'PRIMARY' at
Re: AW:DNS in virtual router not stable
Hi, Ben, Thanks for the reply. We have been thinking about updating /etc/hosts of each of the VM to add the entries for other VMs. Then we don't have dependency to the DNS in the virtual router. But there's lots of work as VMs are being added / removed continuously in our case. I just hope there will be easier solution. Thanks Ivan From: Benjamin Naber To: "users@cloudstack.apache.org" Date: 2018/12/13 09:39 AM Subject:AW:DNS in virtual router not stable Hi Ivan, I've seen this problem to. The solution is to handle your ip address by the host youre adding static by api or gui. The router VM uses the old host entries set in /etc/hosts if you cleanup vpc network. The host file will be cleared. Kind Regards Ben Von meinem Huawei-Mobiltelefon gesendet Originalnachricht Betreff: DNS in virtual router not stable Von: Ivan X Yue An: users@cloudstack.apache.org Cc: Hi, all, We are using Cloudstack 4.9.2 with KVM and advance networking We are having VPC network, and need to continuously adding / removing VM. One issue we found is, from time-to-time, on the VM deployed, it gets error that the hostname of the other VM is not resolved. We dig into the VR, and it seems that every time a VM is added or removed, the dnsmasq needs to be restarted. And based on /var/log/dnsmasq.log, there is 2 second delay before dnsmasq is running again. Quite likely while restart is happening, the DNS is not working and therefore VM will not be able to resolve hostname. Dec 12 15:22:55 dnsmasq[6479]: exiting on receipt of SIGTERM Dec 12 15:22:57 dnsmasq[6747]: started, version 2.62 cachesize 150 Dec 12 15:26:36 dnsmasq[6747]: exiting on receipt of SIGTERM Dec 12 15:26:38 dnsmasq[28921]: started, version 2.62 cachesize 150 We don't see this issue in CloudStack 4.11. In 4.11, we also see dnsmasq being restarted in dnsmasq.log, but there is no delay between exiting and started. Is there anything we can do in 4.9.2 to have the DNS more stable. e.g. can dnsmasq not being restarted even new VM info is added? Or can it just reload the VM info? Thanks Ivan
DNS in virtual router not stable
Hi, all, We are using Cloudstack 4.9.2 with KVM and advance networking We are having VPC network, and need to continuously adding / removing VM. One issue we found is, from time-to-time, on the VM deployed, it gets error that the hostname of the other VM is not resolved. We dig into the VR, and it seems that every time a VM is added or removed, the dnsmasq needs to be restarted. And based on /var/log/dnsmasq.log, there is 2 second delay before dnsmasq is running again. Quite likely while restart is happening, the DNS is not working and therefore VM will not be able to resolve hostname. Dec 12 15:22:55 dnsmasq[6479]: exiting on receipt of SIGTERM Dec 12 15:22:57 dnsmasq[6747]: started, version 2.62 cachesize 150 Dec 12 15:26:36 dnsmasq[6747]: exiting on receipt of SIGTERM Dec 12 15:26:38 dnsmasq[28921]: started, version 2.62 cachesize 150 We don't see this issue in CloudStack 4.11. In 4.11, we also see dnsmasq being restarted in dnsmasq.log, but there is no delay between exiting and started. Is there anything we can do in 4.9.2 to have the DNS more stable. e.g. can dnsmasq not being restarted even new VM info is added? Or can it just reload the VM info? Thanks Ivan
Re: VR corrupted when adding many guest network to VPC
Hi, Andrija, Our problem is exactly the one that you found. After we apply the patch, it works very well. Thanks a lot for the help. Thanks Ivan From: Andrija Panic To: users Date: 2018/11/09 03:55 AM Subject:Re: VR corrupted when adding many guest network to VPC afaik, if ti's the same stuff - then it's here: https://issues.apache.org/jira/browse/CLOUDSTACK- cheers On Fri, 9 Nov 2018 at 06:00, Ivan X Yue wrote: > We are using CloudStack 4.9.3.1. We are using advance networking, and > create VPC network with multiple guest networks. Each guest network has a > few VMs and a few public IPs. Everything works well until we add the 9th > guest network to the VPC. After creating it and adding a VM there, the > Virtual Router is corrupted. All the public IPs that works before stop > working. > > When we check out the virtual router, it seems that when it setup the 9th > network, it creates eth10. However, instead of adding the inet info to > eth10, it is added to eth1 instead. We observe the same issue in > /etc/dnsmasq.d/cloud.conf. The DNS info is added to eth1 instead of > eth10. As the result, the cloud.conf file is corrupted and dnsmasq fails > to start. > > Is it a known issue? Is there any patch or work around? > > > root@r-514-VM:/etc/dnsmasq.d# ip addr > 1: lo: mtu 16436 qdisc noqueue state UNKNOWN > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > inet 127.0.0.1/8 scope host lo > 2: eth0: mtu 1500 qdisc pfifo_fast state > UP qlen 1000 > link/ether 0e:00:a9:fe:01:72 brd ff:ff:ff:ff:ff:ff > inet 169.254.1.114/16 brd 169.254.255.255 scope global eth0 > 3: eth1: mtu 1500 qdisc pfifo_fast state > UP qlen 1000 > link/ether 06:76:18:00:00:0a brd ff:ff:ff:ff:ff:ff > inet 10.125.18.81/26 brd 10.125.18.127 scope global eth1 > inet 172.16.8.1/24 brd 172.16.8.255 scope global eth1 > inet 10.125.18.82/26 brd 10.125.18.127 scope global secondary eth1 > inet 10.125.18.92/26 brd 10.125.18.127 scope global secondary eth1 > inet 10.125.18.99/26 brd 10.125.18.127 scope global secondary eth1 > inet 10.125.18.91/26 brd 10.125.18.127 scope global secondary eth1 > inet 10.125.18.101/26 brd 10.125.18.127 scope global secondary eth1 > inet 10.125.18.102/26 brd 10.125.18.127 scope global secondary eth1 > inet 10.125.18.115/26 brd 10.125.18.127 scope global secondary eth1 > inet 10.125.18.98/26 brd 10.125.18.127 scope global secondary eth1 > 4: eth2: mtu 1500 qdisc pfifo_fast state > UP qlen 1000 > link/ether 02:00:67:af:00:02 brd ff:ff:ff:ff:ff:ff > inet 172.16.0.1/24 brd 172.16.0.255 scope global eth2 > 5: eth3: mtu 1500 qdisc pfifo_fast state > UP qlen 1000 > link/ether 02:00:5e:46:00:02 brd ff:ff:ff:ff:ff:ff > inet 172.16.1.1/24 brd 172.16.1.255 scope global eth3 > 6: eth4: mtu 1500 qdisc pfifo_fast state > UP qlen 1000 > link/ether 02:00:27:b8:00:02 brd ff:ff:ff:ff:ff:ff > inet 172.16.2.1/24 brd 172.16.2.255 scope global eth4 > 7: eth5: mtu 1500 qdisc pfifo_fast state > UP qlen 1000 > link/ether 02:00:27:4b:00:02 brd ff:ff:ff:ff:ff:ff > inet 172.16.3.1/24 brd 172.16.3.255 scope global eth5 > 8: eth6: mtu 1500 qdisc pfifo_fast state > UP qlen 1000 > link/ether 02:00:79:f7:00:02 brd ff:ff:ff:ff:ff:ff > inet 172.16.4.1/24 brd 172.16.4.255 scope global eth6 > 9: eth7: mtu 1500 qdisc pfifo_fast state > UP qlen 1000 > link/ether 02:00:67:91:00:02 brd ff:ff:ff:ff:ff:ff > inet 172.16.5.1/24 brd 172.16.5.255 scope global eth7 > 10: eth8: mtu 1500 qdisc pfifo_fast > state UP qlen 1000 > link/ether 02:00:4b:40:00:02 brd ff:ff:ff:ff:ff:ff > inet 172.16.6.1/24 brd 172.16.6.255 scope global eth8 > 11: eth9: mtu 1500 qdisc pfifo_fast > state UP qlen 1000 > link/ether 02:00:44:40:00:02 brd ff:ff:ff:ff:ff:ff > inet 172.16.7.1/24 brd 172.16.7.255 scope global eth9 > 12: eth10: mtu 1500 qdisc noop state DOWN qlen 1000 > link/ether 02:00:1d:61:00:02 brd ff:ff:ff:ff:ff:ff > > > Please let me know if there is any suggestion. Thx for any help in > advance. > > Thanks, > Ivan > > > -- Andrija Panić
VR corrupted when adding many guest network to VPC
We are using CloudStack 4.9.3.1. We are using advance networking, and create VPC network with multiple guest networks. Each guest network has a few VMs and a few public IPs. Everything works well until we add the 9th guest network to the VPC. After creating it and adding a VM there, the Virtual Router is corrupted. All the public IPs that works before stop working. When we check out the virtual router, it seems that when it setup the 9th network, it creates eth10. However, instead of adding the inet info to eth10, it is added to eth1 instead. We observe the same issue in /etc/dnsmasq.d/cloud.conf. The DNS info is added to eth1 instead of eth10. As the result, the cloud.conf file is corrupted and dnsmasq fails to start. Is it a known issue? Is there any patch or work around? root@r-514-VM:/etc/dnsmasq.d# ip addr 1: lo: mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo 2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 0e:00:a9:fe:01:72 brd ff:ff:ff:ff:ff:ff inet 169.254.1.114/16 brd 169.254.255.255 scope global eth0 3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 06:76:18:00:00:0a brd ff:ff:ff:ff:ff:ff inet 10.125.18.81/26 brd 10.125.18.127 scope global eth1 inet 172.16.8.1/24 brd 172.16.8.255 scope global eth1 inet 10.125.18.82/26 brd 10.125.18.127 scope global secondary eth1 inet 10.125.18.92/26 brd 10.125.18.127 scope global secondary eth1 inet 10.125.18.99/26 brd 10.125.18.127 scope global secondary eth1 inet 10.125.18.91/26 brd 10.125.18.127 scope global secondary eth1 inet 10.125.18.101/26 brd 10.125.18.127 scope global secondary eth1 inet 10.125.18.102/26 brd 10.125.18.127 scope global secondary eth1 inet 10.125.18.115/26 brd 10.125.18.127 scope global secondary eth1 inet 10.125.18.98/26 brd 10.125.18.127 scope global secondary eth1 4: eth2: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:00:67:af:00:02 brd ff:ff:ff:ff:ff:ff inet 172.16.0.1/24 brd 172.16.0.255 scope global eth2 5: eth3: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:00:5e:46:00:02 brd ff:ff:ff:ff:ff:ff inet 172.16.1.1/24 brd 172.16.1.255 scope global eth3 6: eth4: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:00:27:b8:00:02 brd ff:ff:ff:ff:ff:ff inet 172.16.2.1/24 brd 172.16.2.255 scope global eth4 7: eth5: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:00:27:4b:00:02 brd ff:ff:ff:ff:ff:ff inet 172.16.3.1/24 brd 172.16.3.255 scope global eth5 8: eth6: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:00:79:f7:00:02 brd ff:ff:ff:ff:ff:ff inet 172.16.4.1/24 brd 172.16.4.255 scope global eth6 9: eth7: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:00:67:91:00:02 brd ff:ff:ff:ff:ff:ff inet 172.16.5.1/24 brd 172.16.5.255 scope global eth7 10: eth8: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:00:4b:40:00:02 brd ff:ff:ff:ff:ff:ff inet 172.16.6.1/24 brd 172.16.6.255 scope global eth8 11: eth9: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:00:44:40:00:02 brd ff:ff:ff:ff:ff:ff inet 172.16.7.1/24 brd 172.16.7.255 scope global eth9 12: eth10: mtu 1500 qdisc noop state DOWN qlen 1000 link/ether 02:00:1d:61:00:02 brd ff:ff:ff:ff:ff:ff Please let me know if there is any suggestion. Thx for any help in advance. Thanks, Ivan
