FYI: From the GUI, I noticed that the cluster for the only host has an
undefined State.
FYI: From the GUI, the only host has a State of Up.
First ...
I stopped the cloud-management service.
I dumped the cloud DB:
[root@cumulus management]# mysqldump -u root -p cloud > /tmp/cloud.sql
Enter password:
[root@cumulus management]# ls -alg /tmp/cloud.sql
-rw-r--r--. 1 root root 401072 Jul 16 19:35 /tmp/cloud.sql
[root@cumulus management]#
Got the ids for the VMs ...
mysql> Select * from vm_instance where state in ('Running', 'Starting',
'Stopping') AND host_id in (select id from host where removed is not null);
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+
| id | name | uuid | instance_name | state
| vm_template_id | guest_os_id | private_mac_address | private_ip_address |
pod_id | data_center_id | host_id | last_host_id | proxy_id | proxy_assign_time
| vnc_password | ha_enabled | limit_cpu_use |
update_count | update_time | created | removed | type
| vm_type | account_id | domain_id | service_offering_id |
reservation_id | hypervisor_type |
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+
| 1 | s-1-VM | 4780c7ba-dd55-45a2-a2e5-ed2974c070b5 | s-1-VM | Running
| 1 | 15 | 06:9e:ec:00:00:07 | 10.217.5.199 |
1 | 1 | 1 | 1 | NULL | NULL |
caqFANzAxnvRMnVMJeajvIY4x3W8NMjFUkj/RkQmhSU= | 0 | 0 |
3 | 2012-07-14 00:02:27 | 2012-07-13 23:59:42 | NULL |
SecondaryStorageVm | SecondaryStorageVm | 1 | 1 |
6 | e62a93bd-9bb0-4640-8a89-dc1049f300fa | XenServer |
| 2 | v-2-VM | 79924f6e-66e2-429f-b38a-9d3bd3f6a575 | v-2-VM | Running
| 1 | 15 | 06:31:80:00:00:06 | 10.217.5.198 |
1 | 1 | 1 | 1 | NULL | NULL |
cF8VnXqcjTMS8dUfURYlShHJhV55iUFKMyyEKA/SNhw= | 0 | 0 |
3 | 2012-07-14 00:04:19 | 2012-07-13 23:59:42 | NULL | ConsoleProxy
| ConsoleProxy | 1 | 1 | 8 |
c1e4d053-95c6-4804-b73f-c38e7e4e887a | XenServer |
| 4 | r-4-VM | 593d5b00-9c73-485f-b091-121b69171e65 | r-4-VM | Running
| 1 | 15 | 0e:00:a9:fe:00:88 | 169.254.0.136 |
1 | 1 | 1 | 1 | NULL | NULL |
AZ69gd1Mp6IO4gt0AWAeAQawEXZtjh9mA7cQ6ZCDcJg= | 1 | 0 |
3 | 2012-07-14 01:53:39 | 2012-07-14 01:52:48 | NULL | DomainRouter
| DomainRouter | 1 | 1 | 7 |
4f69471a-c0f0-48d2-a39e-3dda20b20c56 | XenServer |
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+
3 rows in set (0.00 sec)
mysql> Select id, name, state from vm_instance where state in ('Running',
'Starting', 'Stopping') AND host_id in (select id from host where removed is
not null);
+----+--------+---------+
| id | name | state |
+----+--------+---------+
| 1 | s-1-VM | Running |
| 2 | v-2-VM | Running |
| 4 | r-4-VM | Running |
+----+--------+---------+
3 rows in set (0.00 sec)
mysql>
Next ...
Update vm_instance set state='Stopped', host_id=null where id in('1', '2', '4');
mysql> Update vm_instance set state='Stopped', host_id=null where id in('1',
'2', '4');
Query OK, 3 rows affected (0.06 sec)
Rows matched: 3 Changed: 3 Warnings: 0
mysql>
Restarted CSMS ...
[root@cumulus management]# service cloud-management start
Starting cloud-management: [ OK ]
[root@cumulus management]#
>From the GUI, started the two system VMs ...
>From the API (no other way), started the Router ...
http://10.217.5.192:8080/client/api?apikey=iFl88lw1Pk6gKqUIFPN8vzZbJNsUVdYGIJKBTEXtrymcIH5UWp9VHjgnpP_zCmaucmi8XmwK75TR70z-2ayjGA&command=startR
outer&id=593d5b00-9c73-485f-b091-121b69171e65&response=json&signature=sx8etg9c1W5o%2BV7L2JUr1Bn%2BfEk%3D
{ "listroutersresponse" : { "count":1 ,"router" : [
{"id":"593d5b00-9c73-485f-b091-121b69171e65","zoneid":"9bf663f5-167a-4152-b80c-a1fb905e01ef
... etc} ] } }
>From the API, verified the Router had started ...
http://10.217.5.192:8080/client/api?apikey=iFl88lw1Pk6gKqUIFPN8vzZbJNsUVdYGIJKBTEXtrymcIH5UWp9VHjgnpP_zCmaucmi8XmwK75TR70z-2ayjGA&command=listRouters&listall=true&response=json&signature=aQWLi1pJXK2wKF5D2ljFR6rbsDA%3D
{ "listroutersresponse" : { "count":1 ,"router" : [
{"id":"593d5b00-9c73-485f-b091-121b69171e65","zoneid":"9bf663f5-167a-4152-b80c-a1fb905e01ef","zonename":"LS_ZONE1","dns1":"10.217.120.2","dns2":"10.217.120.3","networkdomain":"cs1cloud.internal","name":"r-4-VM","podid":"3d6a0b64-94c7-40e1-865b-c62469ca2bda","hostid":"7b966f69-e792-4fb5-a5d7-2c3acb294f79","hostname":"xenserver1-cs","linklocalip":"169.254.2.164","linklocalmacaddress":"0e:00:a9:fe:02:a4","linklocalnetmask":"255.255.0.0","linklocalnetworkid":"83214711-3a39-46ac-a482-95368cbd6950","guestipaddress":"10.217.5.214","guestmacaddress":"06:0e:d6:00:00:0d","guestnetmask":"255.255.255.0","guestnetworkid":"003cef18-fb0c-4b5b-8f4d-1b89eda55580","templateid":"d6260123-7192-4122-beda-d810a7c596b0","created":"2012-07-13T18:52:48-0700","state":"Running","account":"system","domainid":"b8d53f75-99be-432f-9a49-853509c461a3","domain":"ROOT","serviceofferingid":"43713ed5-61fe-4dbc-b676-f21f0d0ebbb0","serviceofferingname":"System
Offering For Software
Router","isredundantrouter":false,"redundantstate":"UNKNOWN"} ] } }
The three VMs are now Running ...
>From the GUI, successfully added a VM to the only host.
Thanks.
Could you please give the sequence of API steps to deleting a host?
deleteHost reports success when devices other than the host have been adversely
affected.
Regards,
Evan
-----Original Message-----
From: Alena Prokharchyk [mailto:alena.prokharc...@citrix.com]
Sent: Monday, July 16, 2012 5:26 PM
To: cloudstack-users@incubator.apache.org
Subject: Re: Problem: Unable to create a deployment for VM[User|i-2-8-VM]
How did you remove the host id=1, through the API or UI? It should have been
resulted with error because it has vms running on it - your Virtual router
r-4-VM.
Looks like the system is in a pretty bad state. I can think of one way to
recover the vms:
1) stop the Management server, save the cloud DB dump.
2) In the DB find all Running vms referring the Removed hosts:
Select * from vm_instance where state in (Running,Starting, Stopping) and
host_id in (select id from host where removed is not null).
3) update all these vms with the following info:
Update vm_instance set state='Stopped', host_id=null where id in (<ids found on
prev step>)
4) start the management server
5) start the vms updated on step 3).
-Alena.
On 7/16/12 5:16 PM, "Evan Miller" <evan.mil...@citrix.com> wrote:
>Hi Alena:
>
>It looks like the host table in the database needs to be cleaned up:
>
>mysql> select id, name, uuid, status from host;
>+----+-----------------------------------------------------+-----------
>+----+-----------------------------------------------------+---
>------------------------+--------------+
>| id | name | uuid
> | status |
>+----+-----------------------------------------------------+-----------
>+----+-----------------------------------------------------+---
>------------------------+--------------+
>| 1 | xenserver1-cs |
>5da2e723-c41a-45bb-9f87-6048c5487ea2 | Removed |
>| 2 | nfs://cumulus.eng.citrite.net/home/export/secondary |
>7bf58f58-0cf5-4487-9346-9481fa28ad7a | Alert |
>| 3 | s-1-VM |
>780d68d6-f6d7-45de-bbd6-766d8e17ee48 | Disconnected |
>| 4 | v-2-VM |
>ca98385f-90f5-49ba-8d88-50af5587e7a9 | Disconnected |
>| 5 | xenserver1-cs |
>dff5e16e-7d8b-47d1-a9a3-8c8e18ca8d23 | Removed |
>| 6 | xenserver1-cs |
>7b966f69-e792-4fb5-a5d7-2c3acb294f79 | Up |
>+----+-----------------------------------------------------+-----------
>+----+-----------------------------------------------------+---
>------------------------+--------------+
>6 rows in set (0.00 sec)
>
>mysql>
>
>id 1 and id 5 should be deleted.
>id 6 has the correct state and uuid.
>id 2, 3 and 4 have the wrong state.
>
>I can fix directly with SQL.
>
>Any other tables that need fixing besides host?
>
>Regards,
>Evan
>
>
>
>-----Original Message-----
>From: Alena Prokharchyk [mailto:alena.prokharc...@citrix.com]
>Sent: Monday, July 16, 2012 4:55 PM
>To: cloudstack-users@incubator.apache.org
>Subject: Re: Problem: Unable to create a deployment for
>VM[User|i-2-8-VM]
>
>The log states that the host doesn't exist:
>
>com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>unreachable: Host 1: Can't find the host
>
>
>
>Can you check the database for host id=1? Make sure its not removed.
>
>
>On 7/16/12 4:45 PM, "Evan Miller" <evan.mil...@citrix.com> wrote:
>
>>From CSMS GUI, the State is Up for the host, xenserver1-cs.
>>
>>Regards,
>>Evan
>>
>>
>>-----Original Message-----
>>From: Alena Prokharchyk [mailto:alena.prokharc...@citrix.com]
>>Sent: Monday, July 16, 2012 4:45 PM
>>To: cloudstack-users@incubator.apache.org
>>Subject: Re: Problem: Unable to create a deployment for
>>VM[User|i-2-8-VM]
>>
>>Looks like the Xen host your Virtual Router r-4-VM is running on, is
>>in not in UP state. Can you please confirm. If this is the case, then
>>the failure is expected. The Xen host should be in UP state.
>>
>>
>>On 7/16/12 4:16 PM, "Evan Miller" <evan.mil...@citrix.com> wrote:
>>
>>>Running CloudStack Management Server:
>>> v3.0.2.1
>>>On:
>>> [root@cumulus management]# uname -a
>>> Linux cumulus.eng.citrite.net 2.6.32-220.el6.x86_64 #1 SMP Tue Dec
>>>6
>>>19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
>>> [root@cumulus management]#
>>>Hypervisor :
>>> XenServer v6.02
>>>
>>>Hi:
>>>
>>>Can¹t add VM instance.
>>>Getting this error message from the GUI:
>>> Unable to create a deployment for VM[User|i-2-8-VM]
>>>
>>>In the attached log file, there are a few ³apparent² issues when
>>>trying to create the instance :
>>>
>>>1)
>>>2012-07-16 15:56:19,928 DEBUG
>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>(Job-Executor-3:job-33) Applying dhcp entry in network
>>>Ntwk[204|Guest|5]
>>>2012-07-16 15:56:19,943 WARN
>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>(Job-Executor-3:job-33) Unable to apply dhcp entry on disconnected
>>>router r-4-VM
>>>com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>>unreachable: Host 1: Can't find the host
>>>
>>>This one occurred at the beginning of the async job. Not sure if it
>>>is an error or not.
>>>
>>>2)
>>>2012-07-16 15:56:20,521 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>(Job-Executor-3:job-33) VM state transitted from :Stopped to Error
>>>with
>>>event: OperationFailedToErrorvm's original host id: null new host id:
>>>null host id before state transition:
>>> null
>>>2012-07-16 15:56:21,192 INFO [api.commands.DeployVMCmd]
>>>(Job-Executor-3:job-33)
>>>com.cloud.exception.InsufficientServerCapacityException: Unable to
>>>create a deployment for VM[User|i-2-8-VM]Scope=interface
>>>com.cloud.dc.DataCenter; id=1
>>>2012-07-16 15:56:21,192 WARN [cloud.api.ApiDispatcher]
>>>(Job-Executor-3:job-33) class com.cloud.api.ServerApiException :
>>>Unable to create a deployment for VM[User|i-2-8-VM]
>>>
>>>That one doesn¹t look good.
>>>
>>>3)
>>>2012-07-16 15:56:30,263 DEBUG [cloud.server.StatsCollector]
>>>(StatsCollector-1:null) There is no secondary storage VM for
>>>secondary storage host
>>>nfs://cumulus.eng.citrite.net/home/export/secondary
>>>
>>>
>>>From CloudStack GUI, secondary storage is defined and enabled.
>>>
>>>What¹s important, which needs to be fixed and how to fix?
>>>
>>>Thanks.
>>>
>>>Regards,
>>>
>>>Evan Miller
>>>
>>>Citrix Systems. Inc.
>>>Desktop and Cloud Engineering Infrastructure
>>>4988 Great America Parkway
>>>Santa Clara, CA 95054
>>>
>>>
>>>
>>
>>
>>
>
>
>
