Hi Dan,
The host value is correct and I am able to connect on ports 22,80 and 8080
from the Xen hosts to that IP (same subnet). Port 8787 is not listening on
my MS.
What can I look for in the DB?
thanks
Carlos
On Fri, Nov 1, 2013 at 2:47 AM, Daan Hoogland <daan.hoogl...@gmail.com>wrote:
> H Carlox,
>
> It still seems to me you have a communication problem albeit not at
> the network level. ( I agree this doesn't seem likely) But even so the
> ms can't reach the xen hosts. The next thing to look at would be
> configuration. I presume you did check the ip addresses for the hosts.
> check the value 'host' in the global vars and try to telnet it on port
> 22 80 8080 8787 or what else from the xen hosts . I am sure the
> problem is hidden somewhere in the mysql database.
>
> regards,
> Daan
>
> On Fri, Nov 1, 2013 at 6:23 AM, Carlos Reategui <car...@reategui.com>
> wrote:
> > Hi Dan,
> > Finally getting around to trying this. I am not able to start any vm
> (mine
> > or system).
> >
> > I am in the process of trying Ahmad's suggestion of removing one of the
> > hosts, reinstalling it and then adding it back in to see if that helps.
> > However I am not able to put it in maintenance mode. I am getting the
> > following when I try:
> >
> >
> > 2013-10-31 15:19:59,120 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-5:null)
> > ===START=== 172.30.40.135 -- GET command=prepareHostForMa
> >
> intenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> > 2013-10-31 15:19:59,181 DEBUG [cloud.async.AsyncJobManagerImpl]
> > (catalina-exec-5:null) submit async job-167, details: AsyncJobVO {id:16
> > 7, userId: 2, accountId: 2, sessionKey: null, instanceType: Host,
> > instanceId: 1, cmd: org.apache.cloudstack.api.command.admin.host.Prep
> > areForMaintenanceCmd, cmdOriginator: null, cmdInfo:
> >
> {"id":"c2f3c416-f4ba-4f73-b414-6fd6efc734e3","response":"json","sessionkey":"Yjye5q
> >
> mcLtUBW0/P4XCAcx2ZTAI\u003d","ctxUserId":"2","_":"1383258001952","ctxAccountId":"2","ctxStartEventId":"699"},
> > cmdVersion: 0, callbackTy
> > pe: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 233845174730253, completeMsid: null,
> > lastUpdated: null, lastPolled: null, created: null}
> > 2013-10-31 15:19:59,185 DEBUG [cloud.async.AsyncJobManagerImpl]
> > (Job-Executor-1:job-167) Executing org.apache.cloudstack.api.command.ad
> > min.host.PrepareForMaintenanceCmd for job-167
> > 2013-10-31 15:19:59,188 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-5:null)
> > ===END=== 172.30.40.135 -- GET command=prepareHostForMain
> >
> tenance&id=c2f3c416-f4ba-4f73-b414-6fd6efc734e3&response=json&sessionkey=Yjye5qmcLtUBW0%2FP4XCAcx2ZTAI%3D&_=1383258001952
> > 2013-10-31 15:19:59,207 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Job-Executor-1:job-167) Propagating agent change request event:AdminA
> > skMaintenace to agent:1
> > 2013-10-31 15:19:59,208 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Job-Executor-1:job-167) 233845174730253 -> 233845174730255.1 [{"Propa
> >
> gateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> > 2013-10-31 15:19:59,213 INFO [cloud.cluster.ClusterServiceServletImpl]
> > (Cluster-Worker-1:null) Setup cluster service servlet. service
> > url: http://172.30.45.2:9090/clusterservice, request timeout: 300
> seconds
> > 2013-10-31 15:19:59,214 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
> > . agent: 1, pdu seq: 1, pdu ack seq: 0, json:
> >
> [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{
> > },"wait":0}}]
> > 2013-10-31 15:19:59,343 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Cluster-Worker-7:null) Dispatch ->1, json: [{"PropagateResourceEventC
> >
> ommand":{"hostId":1,"event":"AdminAskMaintenace","contextMap":{},"wait":0}}]
> > 2013-10-31 15:19:59,347 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Cluster-Worker-7:null) Intercepting command to propagate event AdminA
> > skMaintenace for host 1
> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterServiceServletImpl]
> > (Cluster-Worker-1:null) POST http://172.30.45.2:9090/clusterser
> > vice response :true, responding time: 80 ms
> > 2013-10-31 15:19:59,351 DEBUG [cloud.cluster.ClusterManagerImpl]
> > (Cluster-Worker-1:null) Cluster PDU 233845174730253 -> 233845174730255
> > completed. time: 137ms. agent: 1, pdu seq: 1, pdu ack seq: 0, json:
> > [{"PropagateResourceEventCommand":{"hostId":1,"event":"AdminAskMai
> > ntenace","contextMap":{},"wait":0}}]
> > 2013-10-31 15:19:59,356 DEBUG [agent.manager.ClusteredAgentAttache]
> > (Cluster-Worker-7:null) Seq 1-102760483: Forwarding Seq 1-102760483
> > : { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> > 2013-10-31 15:19:59,360 DEBUG [agent.manager.ClusteredAgentAttache]
> > (AgentManager-Handler-1:null) Seq 1-102760483: Forwarding Seq 1-102
> > 760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> > 2013-10-31 15:19:59,365 DEBUG [agent.manager.ClusteredAgentAttache]
> > (AgentManager-Handler-13:null) Seq 1-102760483: Forwarding Seq 1-10
> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> > 2013-10-31 15:19:59,369 DEBUG [agent.manager.ClusteredAgentAttache]
> > (AgentManager-Handler-15:null) Seq 1-102760483: Forwarding Seq 1-10
> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> > 2013-10-31 15:19:59,372 DEBUG [agent.manager.ClusteredAgentAttache]
> > (AgentManager-Handler-10:null) Seq 1-102760483: Forwarding Seq 1-10
> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> > 2013-10-31 15:19:59,376 DEBUG [agent.manager.ClusteredAgentAttache]
> > (AgentManager-Handler-11:null) Seq 1-102760483: Forwarding Seq 1-10
> > 2760483: { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags:
> 100111,
> > [{"MaintainCommand":{"wait":0}}] } to 233845174730255
> >
> > These MaintainCommand lines are repeated endlessly and very fast (1GB of
> > logs in about 20 mins) so I currently have the MS stopped.
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Oct 31, 2013 at 3:43 PM, Carlos Reategui <car...@reategui.com>
> > wrote:
> >>
> >> Hi Dan,
> >> Network is definitely not a problem. iptables is disabled on MS and
> both
> >> hosts. I am able to ssh from MS to both hosts. I am also able to
> connect
> >> to port 80 from the MS.
> >>
> >> I need to head out now and will try what you suggested in a couple
> hours.
> >>
> >> thanks,
> >> Carlos
> >>
> >>
> >> On Thu, Oct 31, 2013 at 3:19 PM, Daan Hoogland <daan.hoogl...@gmail.com
> >
> >> wrote:
> >>>
> >>> Carlos,
> >>>
> >>> can the ms reach the xs hosts?
> >>> if so try this, Stop the ms, set the state for the vm_instances for
> >>> cpvm and vr from expunging to stopped. and restart.
> >>> if not you need to troubleshoot your network connectivity. It seems
> >>> this would not be the issue as you only stopped and started it. Did
> >>> you
> >>>
> >>> Daan
> >>>
> >>> On Thu, Oct 31, 2013 at 10:19 PM, Carlos Reategui <car...@reategui.com
> >
> >>> wrote:
> >>> > Hi Dan,
> >>> >
> >>> > Unfortunately not. Still trying to solve this. See my last email to
> >>> > Ahmad
> >>> > in the recover cs thread about next steps I am trying to figure out.
> >>> >
> >>> > The CPVM and VR are not running. They are currently in expunging
> >>> > state.
> >>> >
> >>> > If I try to start an instance the MS is unable to communicate the
> start
> >>> > vm
> >>> > commands to my XS hosts and then starts writing to the logs at an
> >>> > amazingly
> >>> > high rate. You can see one of my logs here:
> >>> > http://reategui.com/cloudstack/management-server.log.new
> >>> >
> >>> > thank you
> >>> > Carlos
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Thu, Oct 31, 2013 at 1:50 PM, Daan Hoogland
> >>> > <daan.hoogl...@gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> H Carlos,
> >>> >>
> >>> >> Did you solve this problem yet?
> >>> >>
> >>> >> You can try removing the cpvm and vr at the hypervisor and change
> >>> >> their state to 'Stopped' in the db.
> >>> >>
> >>> >> On Tue, Oct 29, 2013 at 8:03 PM, Carlos Reategui <
> car...@reategui.com>
> >>> >> wrote:
> >>> >> > In the last 15 minutes management log file it grew 500MB..
> >>> >> >
> >>> >> > Any ideas what I should look at to figure out what is happening?
> >>> >> >
> >>> >> >
> >>> >> > On Tue, Oct 29, 2013 at 11:53 AM, Carlos Reategui
> >>> >> > <create...@gmail.com>wrote:
> >>> >> >
> >>> >> >> I am trying to recover my CloudStack installation (CS 4.1.1 on
> >>> >> >> ubuntu +
> >>> >> >> XS
> >>> >> >> 6.0.2) that I shutdown and cannot bring back to life.
> >>> >> >>
> >>> >> >> I tried to destroy the CPVM and the VR and all I am seeing in the
> >>> >> >> logs
> >>> >> >> is
> >>> >> >> the following and they are filling up fast (management log is
> >>> >> >> almost
> >>> >> >> 1GB).
> >>> >> >>
> >>> >> >>
> >>> >> >> 2013-10-29 11:51:08,291 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-14:null) Seq 1-201919186: Forwarding Seq
> >>> >> >> 1-201919186:
> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-4:null) Seq 2-1168706613: Forwarding Seq
> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> >>> >> >> Flags:
> >>> >> >> 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,293 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-2:null) Seq 1-201919186: Forwarding Seq
> >>> >> >> 1-201919186:
> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-8:null) Seq 2-1168706613: Forwarding Seq
> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> >>> >> >> Flags:
> >>> >> >> 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,294 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-9:null) Seq 1-201919186: Forwarding Seq
> >>> >> >> 1-201919186:
> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-6:null) Seq 2-1168706613: Forwarding Seq
> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> >>> >> >> Flags:
> >>> >> >> 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,296 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-15:null) Seq 1-201919186: Forwarding Seq
> >>> >> >> 1-201919186:
> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-13:null) Seq 2-1168706613: Forwarding Seq
> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> >>> >> >> Flags:
> >>> >> >> 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,297 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-10:null) Seq 1-201919186: Forwarding Seq
> >>> >> >> 1-201919186:
> >>> >> >> { Cmd , MgmtId: 233845174730253, via: 1, Ver: v1, Flags: 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"v-2-VM","volume":{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"9aa8e81d-edb8-4284-a422-a89e9fd4237c","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >> 2013-10-29 11:51:08,299 DEBUG
> [agent.manager.ClusteredAgentAttache]
> >>> >> >> (AgentManager-Handler-3:null) Seq 2-1168706613: Forwarding Seq
> >>> >> >> 2-1168706613: { Cmd , MgmtId: 233845174730253, via: 2, Ver: v1,
> >>> >> >> Flags:
> >>> >> >> 100111,
> >>> >> >>
> >>> >> >>
> >>> >> >>
> [{"storage.DestroyCommand":{"vmName":"r-5-VM","volume":{"id":6,"name":"ROOT-5","mountPoint":"/export/primary","path":"56b215e0-bbb0-455f-9896-620ce22d28ad","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"17b0a8a5-2376-3d11-b60e-31eebeafb217","deviceId":0},"wait":0}}]
> >>> >> >> } to 233845174730255
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >
> >>> >
> >>
> >>
> >
>
