Hi,
One of our agents cannot connect to the server anymore... The manager
complains about active pools already created. We rebooted the agent, but
same result, no success.
Any hints about how to solve this without reinstalling the agent?
** Agent:
[root@c11-38 ~]# virsh pool-list
Name State Autostart
-------------------------------------------
691f4f70-2724-4f9f-8537-6e0cc006dc41 active no
c69fb76e-f2f9-359e-bda0-1259786717af active no
2017-01-03 09:04:44,654 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Connected to the server
2017-01-03 09:04:50,132 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Proccess agent startup answer, agent id = 0
2017-01-03 09:04:50,133 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Set agent id 0
2017-01-03 09:04:50,133 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Startup Response Received: agent id = 0
2017-01-03 09:04:50,487 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:0fee19e8) Attempting to create
storage pool c69fb76e-f2f9-359e-bda0-1259786717af (Gluster) in libvirt
2017-01-03 09:04:50,488 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:0fee19e8) Found existing defined
storage pool c69fb76e-f2f9-359e-bda0-1259786717af, using it.
2017-01-03 09:04:50,488 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:0fee19e8) Trying to fetch storage
pool c69fb76e-f2f9-359e-bda0-1259786717af from libvirt
2017-01-03 09:04:55,674 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Lost connection to the server. Dealing with the remaining
commands...
2017-01-03 09:04:55,675 INFO [utils.nio.NioClient]
(Agent-Handler-2:null) (logid:) NioClient connection closed
2017-01-03 09:04:55,675 INFO [cloud.agent.Agent] (Agent-Handler-2:null)
(logid:) Reconnecting...
2017-01-03 09:04:55,675 INFO [utils.nio.NioClient]
(Agent-Handler-2:null) (logid:) Connecting to 10.112.0.100:8250
2017-01-03 09:04:55,762 INFO [utils.nio.NioClient]
(Agent-Handler-2:null) (logid:) SSL: Handshake done
2017-01-03 09:04:55,762 INFO [utils.nio.NioClient]
(Agent-Handler-2:null) (logid:) Connected to 10.112.0.100:8250
2017-01-03 09:04:55,846 INFO [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) (logid:) Attempting to create storage pool
691f4f70-2724-4f9f-8537-6e0cc006dc41 (Filesystem) in libvirt
2017-01-03 09:04:55,846 INFO [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) (logid:) Found existing defined storage pool
691f4f70-2724-4f9f-8537-6e0cc006dc41, using it.
2017-01-03 09:04:55,846 INFO [kvm.storage.LibvirtStorageAdaptor]
(Agent-Handler-1:null) (logid:) Trying to fetch storage pool
691f4f70-2724-4f9f-8537-6e0cc006dc41 from libvirt
2017-01-03 09:04:57,721 INFO [cloud.agent.Agent]
(AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
** Manager
2017-01-02 23:16:01,613 DEBUG [c.c.s.StorageManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Adding pool
null to host 1
2017-01-02 23:16:01,616 DEBUG [c.c.a.t.Request]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Seq
1-5971210155939856387: Sending { Cmd , MgmtId: 2199198367844, via:
1(c11-38.lan.cesga.es), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":5,"uuid":"c69fb76e-f2f9-359e-bda0-1259786717af","host":"cloudstack-storage.lan.cesga.es","path":"/primary","port":24007,"type":"Gluster"},"localPath":"/mnt//c69fb76e-f2f9-359e-bda0-1259786717af","wait":0}}]
}
2017-01-02 23:16:01,691 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-10:null) (logid:) Seq 1-5971210155939856386:
Processing: { Ans: , MgmtId: 2199198367844, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2017-01-02 23:16:01,691 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-10:null) (logid:) Seq 1-5971210155939856387:
Processing: { Ans: , MgmtId: 2199198367844, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.ModifyStoragePoolAnswer":{"poolInfo":{"host":"cloudstack-storage.lan.cesga.es","localPath":"/mnt//c69fb76e-f2f9-359e-bda0-1259786717af","hostPath":"/primary","poolType":"Gluster","capacityBytes":7188486029312,"availableBytes":7161281118208},"templateInfo":{},"result":true,"wait":0}}]
}
2017-01-02 23:16:01,691 DEBUG [c.c.a.t.Request]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Seq
1-5971210155939856387: Received: { Ans: , MgmtId: 2199198367844, via:
1(c11-38.lan.cesga.es), Ver: v1, Flags: 10, { ModifyStoragePoolAnswer } }
2017-01-02 23:16:01,702 INFO [o.a.c.s.d.p.DefaultHostListener]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Connection
established between storage pool
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@2dee5c66
and host 1
2017-01-02 23:16:01,703 DEBUG [c.c.s.StorageManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Found storage
pool almacenamiento-primario-gluster of type Gluster
2017-01-02 23:16:01,703 DEBUG [c.c.s.StorageManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Total over
provisioned capacity of the pool almacenamiento-primario-gluster id: 5
is 7188486029312
2017-01-02 23:16:01,703 DEBUG [c.c.s.StorageManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Successfully
set Capacity - 7188486029312 for capacity type - 3 , DataCenterId - 1,
HostOrPoolId - 5, PodId null
2017-01-02 23:16:01,703 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Sending
Connect to listener: ClusteredVirtualMachineManagerImpl
2017-01-02 23:16:01,703 DEBUG [c.c.v.VirtualMachineManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Received
startup command from hypervisor host. host id: 1
2017-01-02 23:16:01,703 INFO [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Reset VM power
state sync for host: 1
2017-01-02 23:16:01,747 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-2:null) (logid:) Seq 1-5971210155939856386:
Processing: { Ans: , MgmtId: 2199198367844, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"details":"","wait":0}}] }
2017-01-02 23:16:01,751 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Sending
Connect to listener: ConsoleProxyListener
2017-01-02 23:16:01,752 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Sending
Connect to listener: StorageCapacityListener
2017-01-02 23:16:01,752 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Sending
Connect to listener: ComputeCapacityListener
2017-01-02 23:16:01,755 DEBUG [c.c.c.CapacityManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Found 1 VMs on
host 1
2017-01-02 23:16:01,759 DEBUG [c.c.c.CapacityManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Found 0 VM,
not running on host 1
2017-01-02 23:16:01,760 DEBUG [c.c.c.CapacityManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) No need to
calibrate cpu capacity, host:1 usedCpu: 500 reservedCpu: 0
2017-01-02 23:16:01,760 DEBUG [c.c.c.CapacityManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) No need to
calibrate memory capacity, host:1 usedMem: 1073741824 reservedMem: 0
2017-01-02 23:16:01,791 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-23b32966) (logid:c300fd4b) Zone 1 is ready to launch
console proxy
2017-01-02 23:16:01,809 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Sending
Connect to listener: LocalStoragePoolListener
2017-01-02 23:16:01,811 DEBUG
[o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) createPool
Params @ scheme - Filesystem storageHost - 10.112.11.38 hostPath -
/var/lib/libvirt/images port - -1
2017-01-02 23:16:01,811 DEBUG
[o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Another active
pool with the same uuid already exists
2017-01-02 23:16:01,812 WARN [c.c.s.StorageManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Unable to
setup the local storage pool for Host[-1-Routing]
com.cloud.utils.exception.CloudRuntimeException: Another active pool
with the same uuid already exists
at
org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.initialize(CloudStackPrimaryDataStoreLifeCycleImpl.java:341)
at
com.cloud.storage.StorageManagerImpl.createLocalStorage(StorageManagerImpl.java:607)
at
com.cloud.storage.LocalStoragePoolListener.processConnect(LocalStoragePoolListener.java:88)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:564)
at
com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1087)
at
com.cloud.agent.manager.AgentManagerImpl.access$000(AgentManagerImpl.java:120)
at
com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1171)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-01-02 23:16:01,812 INFO [c.c.u.e.CSExceptionErrorCode]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Could not find
exception: com.cloud.exception.ConnectionException in error code list
for exceptions
2017-01-02 23:16:01,812 WARN [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-1208:ctx-b95fb6f1) (logid:3c2b146e) Monitor
LocalStoragePoolListener says there is an error in the connect process
for 1 due to Unable to setup the local storage pool for Host[-1-Routing]
--
====================================================
Natalia Costas Lago
Senior Communications Technician
Galicia Supercomputing Centre (CESGA)
(CESGA on Twitter | CESGA on Facebook)
Avenida de Vigo, s/n (Campus Vida)
15705 Santiago de Compostela - SPAIN
E-mail: nata...@cesga.es
Cell: +34 981 56 98 10 (ext. 237)
Fax: +34 981 59 46 16
Web: https://www.cesga.es/
====================================================
[IMPORTANTE] La información contenida en este mensaje y
sus posibles documentos adjuntos es privada y confidencial
y está dirigida únicamente a su destinatario/a. Si usted no
es el/la destinatario/a original de este mensaje, por favor
elimínelo. La distribución o copia de este mensaje no está
autorizada.
