Hi, I restartd All hosts (one mgr and xenserver) again.
Following is the error log.
2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
(StatsCollector-3:ctx-692a5392) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for
exceptions
2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed out on null
2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Cancelling.
2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
(StatsCollector-3:ctx-692a5392) Unable to send storage pool command to
Pool[1|NetworkFilesystem] via 1
com.cloud.exception.OperationTimedoutException: Commands
3176445112179752972 to Host 1 timed out after 3600
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:433)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1000)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
at
com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.lang.Thread.run(Thread.java:745)
2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
(StatsCollector-3:ctx-692a5392) Unable to reach Pool[1|NetworkFilesystem]
com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:1] is unreachable: Unable to send command to the pool
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1010)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
at
com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
-----------
Cao Tong
On 07/20/2015 02:52 PM, tony_caot...@163.com wrote:
No, no one's IP was changed.
1. In xenserver I can not login systemvms using the internal IP like
'169.254.1.112', There shoud be a bridge network for this right? it
is gone.
2. I try to delete xenserver host from CS on web, it also failed with
lots of logs like following, then memory is full and mangement down...
2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-15:null) Seq 1-7282039122481381399: Forwarding
Seq 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to 192405008094602
2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-5:null) Seq 1-7282039122481381399: Forwarding
Seq 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to 192405008094602
2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-1:null) Seq 1-7282039122481381399: Forwarding
Seq 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to 192405008094602
2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-14:null) Seq 1-7282039122481381399: Forwarding
Seq 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to 192405008094602
I guess that, is there some service or daemons working for CS is not
up on Xenserver ?
-----------
Cao Tong
On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
Did the management server ip change?
management server ip in the configuration table is used my systemvms.
select * from configuration where name like 'host';
If it changed, correct the value in db and restart systemvms.
~Rajani
On Mon, Jul 20, 2015 at 11:56 AM,<tony_caot...@163.com> wrote:
Hello,
I shutdown my cs-manager and xenserver last weekend, And now the ssvm
and cpvm is disconnect, thost two was runing on xenserver. so What
should i do right now ?
Please anybody help me and thanks.
In xenserver I found that the three system VMs are not running.
my xenserver seems can not reconnect to CS-manager. and it seams not
under control of CS.
What is the right steps of shutdown all CS group machines and resume them?
How can i let my xenserver reconnected ?
Thanks,
--
-----------
Cao Tong
