[
https://issues.apache.org/jira/browse/CLOUDSTACK-7742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangeetha Hariharan updated CLOUDSTACK-7742:
--------------------------------------------
Attachment: ssvm-fail.rar
> Xenserver HA - SSVM failing to start since it is running out of management ip
> address
> --------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-7742
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7742
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.5.0
> Environment: Build from master
> Reporter: Sangeetha Hariharan
> Attachments: ssvm-fail.rar
>
>
> HA - SSVM failing to start since it is running out of management ip address
> Set up:
> Cluster with 3 Xenserver hosts.
> I am executing host HA scenarios where host is being brought down ( or
> simulating contol path network failure / storage network failure).
> After couple of such scenarios , i see that the SSVM fails to start as part
> of HA the reason being running out of management nic:
> management server logs:
> 014-10-16 12:15:44,311 DEBUG [c.c.u.d.T.Transaction]
> (Work-Job-Executor-106:ctx-323991ca job-771/job-943 ctx-3a2e9ed6) Rolling
> back the transaction: Time = 1 Name = Work-Job-Executor-106; called by
> -TransactionLegacy.rollback:902-DataCenterIpAddressDaoImpl.takeIpAddress:61-GeneratedMethodAccessor493.invoke:-1-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:606-AopUtils.invokeJoinpointUsingReflection:317-ReflectiveMethodInvocation.invokeJoinpoint:183-ReflectiveMethodInvocation.proceed:150-TransactionContextInterceptor.invoke:34-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172
> 2014-10-16 12:15:44,312 INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-106:ctx-323991ca job-771/job-943 ctx-3a2e9ed6)
> Insufficient capacity
> com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
> management ip addressScope=interface com.cloud.dc.Pod; id=1
> at
> com.cloud.network.guru.PodBasedNetworkGuru.reserve(PodBasedNetworkGuru.java:123)
> at
> com.cloud.network.guru.StorageNetworkGuru.reserve(StorageNetworkGuru.java:122)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1338)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1309)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:970)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4590)
> at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4746)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2014-10-16 12:15:44,324 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-106:ctx-323991ca job-771/job-943 ctx-3a2e9ed6) Cleaning up
> resources for the vm VM[SecondaryStorageVm|s-115-VM] in Starting state
> There are 2 issues here:
> 1. Some of the SSVMs that are in destroyed state still have not released the
> management Ips back to the freepool.
>
> 2. Some of these destroyed SSVMs have 2 management ip addresses associated
> with it . why is this the case?
> 3. I still see 1 management ip address that is free , but SSVM is still not
> able to come up.
>
> mysql> select id,name,state from vm_instance where id in (1,7,18,71);
> +----+---------+-----------+
> | id | name | state |
> +----+---------+-----------+
> | 1 | v-1-VM | Running |
> | 7 | s-7-VM | Destroyed |
> | 18 | s-18-VM | Destroyed |
> | 71 | s-71-VM | Destroyed |
> +----+---------+-----------+
> 4 rows in set (0.00 sec)
> mysql> select instance_id from nics where id in (select nic_id from
> op_dc_ip_address_alloc where taken is not null);
> +-------------+
> | instance_id |
> +-------------+
> | 1 |
> | 7 |
> | 7 |
> | 18 |
> | 18 |
> | 71 |
> | 71 |
> +-------------+
> 7 rows in set (0.00 sec)
> mysql> select * from op_dc_ip_address_alloc ;
> +----+--------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+
> | id | ip_address | data_center_id | pod_id | nic_id | reservation_id
> | taken | mac_address |
> +----+--------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+
> | 1 | 10.223.59.69 | 1 | 1 | 261 |
> 5cf7268d-cf6c-421d-86d7-fe124b055bd5 | 2014-10-15 00:20:44 | 1 |
> | 2 | 10.223.59.70 | 1 | 1 | 3 |
> a5254254-b384-4783-adab-05bb7d6355be | 2014-10-16 17:34:42 | 2 |
> | 3 | 10.223.59.71 | 1 | 1 | 261 |
> a89646ab-ce75-4518-b685-5fa49e98faca | 2014-10-13 22:05:14 | 3 |
> | 4 | 10.223.59.72 | 1 | 1 | 26 |
> 4c197653-f794-438b-b743-f80cc7348be0 | 2014-10-10 18:47:44 | 4 |
> | 5 | 10.223.59.73 | 1 | 1 | 3 |
> 2c15e101-b534-4b4e-bebb-f580ea284c01 | 2014-10-16 18:31:42 | 5 |
> | 6 | 10.223.59.74 | 1 | 1 | 27 |
> 4c197653-f794-438b-b743-f80cc7348be0 | 2014-10-10 18:47:44 | 6 |
> | 7 | 10.223.59.75 | 1 | 1 | 49 |
> 680321c2-9290-4915-aec4-1b37dc5aefbd | 2014-10-11 00:24:44 | 7 |
> | 8 | 10.223.59.76 | 1 | 1 | 48 |
> 680321c2-9290-4915-aec4-1b37dc5aefbd | 2014-10-11 00:24:44 | 8 |
> | 9 | 10.223.59.77 | 1 | 1 | NULL | NULL
> | NULL | 9 |
> | 10 | 10.223.59.78 | 1 | 1 | 3 |
> 9980f72c-4bc8-40ed-87bf-99fd375bd13d | 2014-10-10 18:48:12 | 10 |
> | 11 | 10.223.59.79 | 1 | 1 | 260 |
> 5cf7268d-cf6c-421d-86d7-fe124b055bd5 | 2014-10-15 00:20:44 | 11 |
> | 12 | 10.223.59.80 | 1 | 1 | 260 |
> a89646ab-ce75-4518-b685-5fa49e98faca | 2014-10-13 22:05:14 | 12 |
> +----+--------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+
> 12 rows in set (0.00 sec)
> mysql>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
