Hi Daan :)
Primary and Secondary storage are on the MS. Primary is on a different
subnet and it is currently mounted (and visible as an NFS SR from
XenCenter) and working fine from the xen machines. I have also been able
to mount Secondary without a problem.
Here are my current system VMs:
mysql> select
id,name,instance_name,state,host_id,type,vm_type,desired_state from
vm_instance where type!='User';
+----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
| id | name | instance_name | state | host_id | type |
vm_type | desired_state |
+----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
| 2 | v-2-VM | v-2-VM | Stopped | 1 | ConsoleProxy |
ConsoleProxy | NULL |
| 5 | r-5-VM | r-5-VM | Stopped | 2 | DomainRouter |
DomainRouter | NULL |
| 1 | s-1-VM | s-1-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 3 | s-3-VM | s-3-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 52 | s-52-VM | s-52-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 53 | s-53-VM | s-53-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 54 | s-54-VM | s-54-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 55 | s-55-VM | s-55-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
| 56 | s-56-VM | s-56-VM | Expunging | NULL | SecondaryStorageVm |
SecondaryStorageVm | NULL |
+----+---------+---------------+-----------+---------+--------------------+--------------------+---------------+
I have been looking through the DB to see if anything sticks out. I notice
that the mshost and mshost_peer table have 2 entries but they are the same
server. Is that ok? Looks like one is from the install of 4.1.0 an the
other from after I upgraded to 4.1.1:
mysql> select * from mshost;
+----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
| id | msid | runid | name | state | version |
service_ip | service_port | last_update | removed | alert_count |
+----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
| 1 | 233845174730255 | 1375383759777 | isbustor01 | Up | 4.1.0 |
172.30.45.2 | 9090 | 2013-10-24 21:13:10 | NULL | 0 |
| 2 | 233845174730253 | 1383331508626 | isbustor01 | Up | 4.1.1 |
172.30.45.2 | 9090 | 2013-11-01 20:00:04 | NULL | 0 |
+----+-----------------+---------------+------------+-------+---------+-------------+--------------+---------------------+---------+-------------+
2 rows in set (0.00 sec)
mysql> select * from mshost_peer;
+----+--------------+-------------+---------------+------------+---------------------+
| id | owner_mshost | peer_mshost | peer_runid | peer_state |
last_update |
+----+--------------+-------------+---------------+------------+---------------------+
| 17 | 1 | 1 | 1375383759777 | Up | 2013-08-01
19:02:49 |
| 34 | 2 | 2 | 1383331508626 | Up | 2013-11-01
18:45:19 |
+----+--------------+-------------+---------------+------------+---------------------+
thanks,
Carlos
On Fri, Nov 1, 2013 at 12:17 PM, Daan Hoogland <daan.hoogl...@gmail.com>wrote:
> h Carlos,
>
> the states of virtual machines, especially the entries for the ssvm.
> if it is not Stopped. set it to stopped and check if your storage,
> both primary and secondary, are reachable botjh from the ms and the
> hosts.
>
> Daan (double a!)
>
> On Fri, Nov 1, 2013 at 7:52 PM, Carlos Reategui <create...@gmail.com>
> wrote:
> > Hi Dan,
> > The host value is correct and I am able to connect on ports 22,80 and
> 8080
> > from the Xen hosts to that IP (same subnet). Port 8787 is not listening
> on
> > my MS.
> >
> > What can I look for in the DB?
> >
> > thanks
> > Carlos
> >
> >
> >
> >
> > On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <daan.hoogl...@gmail.com>
> > wrote:
> >>
> >> H Carlox,
> >>
> >> It still seems to me you have a communication problem albeit not at
> >> the network level. ( I agree this doesn't seem likely) But even so the
> >> ms can't reach the xen hosts. The next thing to look at would be
> >> configuration. I presume you did check the ip addresses for the hosts.
> >> check the value 'host' in the global vars and try to telnet it on port
> >> 22 80 8080 8787 or what else from the xen hosts . I am sure the
> >> problem is hidden somewhere in the mysql database.
> >>
> >> regards,
> >> Daan
> >>
> >> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <car...@reategui.com>
> >> wrote:
> >> > Hi Dan,
> >> > Finally getting around to trying this. I am not able to start any vm
> >> > (mine
> >> > or system).
> >> >
> >> > I am in the process of trying Ahmad's suggestion of removing one of
> the
> >> > hosts, reinstalling it and then adding it back in to see if that
> helps.
> >> > However I am not able to put it in maintenance mode. I am getting the
> >> > following when I try:
> >> >
> >> >
> >> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
> >> > (catalina-exec-5:null)
> >> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
> >> >
> >> >
> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> >> > 2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> > (catalina-exec-5:null) submit async job-167, details: AsyncJobVO
> {id:16
> >> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType: Host,
> >> > instanceId: 1, cmd: org.apache.cloudstack.api.command.admin.host.Prep
> >> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
> >> >
> >> >
> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
> >> >
> >> >
> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
> >> > cmdVersion: 0, callbackTy
> >> > pe: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
> >> > 0,
> >> > result: null, initMsid: 233845174730253, completeMsid: null,
> >> > lastUpdated: null, lastPolled: null, created: null}
> >> > 2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
> >> > (Job-Executor-1:job-167) Executing
> org.apache.cloudstack.api.command.ad
> >> > min.host.PrepareForMaintenanceCmd for job-167
> >> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
> >> > (catalina-exec-5:null)
> >> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
> >> >
> >> >
> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> >> > 2013-10-31 15:19:59,207 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Job-Executor-1:job-167) Propagating agent change request event:AdminA
> >> > skMaintenace to agent:1
> >> > 2013-10-31 15:19:59,208 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1 [{"Propa
> >> >
> >> >
> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> >> > 2013-10-31 15:19:59,213 INFO
> [cloud.cluster.ClusterServiceServletImpl]
> >> > (Cluster-Worker-1:null) Setup cluster service servlet. service
> >> > url: http://172.30.45.2:9090/clusterservice, request timeout: 300
> >> > seconds
> >> > 2013-10-31 15:19:59,214 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
> >> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
> >> >
> >> >
> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
> >> > },"wait":0}}]
> >> > 2013-10-31 15:19:59,343 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Cluster-Worker-7:null) Dispatch ->1, json: [{"PropagateResourceEventC
> >> >
> >> >
> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> >> > 2013-10-31 15:19:59,347 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Cluster-Worker-7:null) Intercepting command to propagate event AdminA
> >> > skMaintenace for host 1
> >> > 2013-10-31 15:19:59,351 DEBUG
> [cloud.cluster.ClusterServiceServletImpl]
> >> > (Cluster-Worker-1:null) POST http://172.30.45.2:9090/clusterser
> >> > vice response :true, responding time: 80 ms
> >> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterManagerImpl]
> >> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
> >> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0, json:
> >> > [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
> >> > ntenace","contextMap":{},"wait":0}}]
> >> > 2013-10-31 15:19:59,356 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq 1-102760483
> >> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> > 2013-10-31 15:19:59,360 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq 1-102
> >> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> > 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> > 2013-10-31 15:19:59,365 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq 1-10
> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> > 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> > 2013-10-31 15:19:59,369 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq 1-10
> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> > 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> > 2013-10-31 15:19:59,372 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq 1-10
> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> > 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> > 2013-10-31 15:19:59,376 DEBUG [agent.manager.ClusteredAgentAttache]
> >> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq 1-10
> >> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> > 100111,
> >> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >> >
> >> > These MaintainCommand lines are repeated endlessly and very fast (1GB
> of
> >> > logs in about 20 mins) so I currently have the MS stopped.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <car...@reategui.com
> >
> >> > wrote:
> >> >>
> >> >> Hi Dan,
> >> >> Network is definitely not a problem. iptables is disabled on MS and
> >> >> both
> >> >> hosts. I am able to ssh from MS to both hosts. I am also able to
> >> >> connect
> >> >> to port 80 from the MS.
> >> >>
> >> >> I need to head out now and will try what you suggested in a couple
> >> >> hours.
> >> >>
> >> >> thanks,
> >> >> Carlos
> >> >>
> >> >>
> >> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland
> >> >> <daan.hoogl...@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> Carlos,
> >> >>>
> >> >>> can the ms reach the xs hosts?
> >> >>> if so try this, Stop the ms, set the state for the vm_instances for
> >> >>> cpvm and vr from expunging to stopped. and restart.
> >> >>> if not you need to troubleshoot your network connectivity. It seems
> >> >>> this would not be the issue as you only stopped and started it. Did
> >> >>> you
> >> >>>
> >> >>> Daan
> >> >>>
> >> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui
> >> >>> <car...@reategui.com>
> >> >>> wrote:
> >> >>> > Hi Dan,
> >> >>> >
> >> >>> > Unfortunately not. Still trying to solve this. See my last email
> >> >>> > to
> >> >>> > Ahmad
> >> >>> > in the recover cs thread about next steps I am trying to figure
> out.
> >> >>> >
> >> >>> > The CPVM and VR are not running. They are currently in expunging
> >> >>> > state.
> >> >>> >
> >> >>> > If I try to start an instance the MS is unable to communicate the
> >> >>> > start
> >> >>> > vm
> >> >>> > commands to my XS hosts and then starts writing to the logs at an
> >> >>> > amazingly
> >> >>> > high rate. You can see one of my logs here:
> >> >>> > http://reategui.com/cloudstack/management-server.log.new
> >> >>> >
> >> >>> > thank you
> >> >>> > Carlos
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
> >> >>> > <daan.hoogl...@gmail.com>
> >> >>> > wrote:
> >> >>> >>
> >> >>> >> H Carlos,
> >> >>> >>
> >> >>> >> Did you solve this problem yet?
> >> >>> >>
> >> >>> >> You can try removing the cpvm and vr at the hypervisor and change
> >> >>> >> their state to 'Stopped' in the db.
> >> >>> >>
> >> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui
> >> >>> >> <car...@reategui.com>
> >> >>> >> wrote:
> >> >>> >> > In the last 15 minutes management log file it grew 500MB..
> >> >>> >> >
> >> >>> >> > Any ideas what I should look at to figure out what is
> happening?
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
> >> >>> >> > <create...@gmail.com>wrote:
> >> >>> >> >
> >> >>> >> >> I am trying to recover my CloudStack installation (CS 4.1.1 on
> >> >>> >> >> ubuntu +
> >> >>> >> >> XS
> >> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
> >> >>> >> >>
> >> >>> >> >> I tried to destroy the CPVM and the VR and all I am seeing in
> >> >>> >> >> the
> >> >>> >> >> logs
> >> >>> >> >> is
> >> >>> >> >> the following and they are filling up fast (management log is
> >> >>> >> >> almost
> >> >>> >> >> 1GB).
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq
> >> >>> >> >> 1-201919186:
> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
> v1,
> >> >>> >> >> Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq
> >> >>> >> >> 1-201919186:
> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
> v1,
> >> >>> >> >> Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq
> >> >>> >> >> 1-201919186:
> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
> v1,
> >> >>> >> >> Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq
> >> >>> >> >> 1-201919186:
> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding
> Seq
> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
> v1,
> >> >>> >> >> Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq
> >> >>> >> >> 1-201919186:
> >> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
> >> >>> >> >> [agent.manager.ClusteredAgentAttache]
> >> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
> >> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver:
> v1,
> >> >>> >> >> Flags:
> >> >>> >> >> 100111,
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >> >>> >> >> } to 233845174730255
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >
> >> >>> >
> >> >>
> >> >>
> >> >
> >
> >
>
