I had built a new XenServer (licensed expired), I had copied vhd-util from
download site to /opt/xensource/bin and removed the one in /usr/sbin, but looks
like it referenced by hardcoded path, so it needs to be found via, /usr/sbin.
[17802] 2013-06-27 09:15:08.707337 ['/usr/sbin/vhd-util', 'scan', '-f',
'-c', '-m', '/var/run/sr-mount/476d133b-5c8c-a674-6544-89e261cd18cb/*.vhd']
I fixed this problem and the resolved the problem with mounting the primary
storage.
-Soheil
________________________________________
From: Soheil Eizadi [seiz...@infoblox.com]
Sent: Thursday, June 27, 2013 3:53 PM
To: dev@cloudstack.apache.org
Subject: Failed to mount primary NFS storage
This is a new problem I have not seen before, I failed to mount primary NFS
storage but it looks like it is mounted on XenServer and operational. After
this problem it is in a loop where you can not fix the problem from wizard, as
every time you try to mount it you get an exception since it is already in the
SR list.
I manually removed the primary storage on XenServer and tried again and it
fails again with the same error:
Caused by: SR_BACKEND_FAILURE_40The SR scan failed [opterr=error VHD-scanning
path /var/run/sr-mount/14499ec3-000f-bb8a-a33d-eba5a98f69e6 (2)]
Is this a known problem?
-Soheil
--------
INFO [network.security.SecurityGroupListener] (2004014557@qtp-1799775865-8:)
Received a host startup notification
INFO [network.security.SecurityGroupListener] (2004014557@qtp-1799775865-8:)
Scheduled network rules cleanup, interval=2118
INFO [network.security.SecurityGroupListener] (2004014557@qtp-1799775865-8:)
Received a host startup notification
INFO [cloud.resource.ResourceManagerImpl] (2004014557@qtp-1799775865-8:)
server resources successfully discovered by XCP Agent
WARN [xen.resource.CitrixResourceBase] (DirectAgent-1:) Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed due
to com.cloud.utils.exception.CloudRuntimeException: Unable to create NFS SR
Pool[1|172.16.197.134:2049|/opt/storage/primary] on
host:846b5000-897d-4eee-8497-3b7992bdc40e pool:
172.16.197.134/opt/storage/primary
com.cloud.utils.exception.CloudRuntimeException: Unable to create NFS SR
Pool[1|172.16.197.134:2049|/opt/storage/primary]
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getNfsSR(CitrixResourceBase.java:6532)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:5541)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:560)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:680)
Caused by: SR_BACKEND_FAILURE_40The SR scan failed [opterr=error VHD-scanning
path /var/run/sr-mount/66f0805b-da95-872c-2a6e-d5c46c219c2a (2)]
at com.xensource.xenapi.Types.checkResponse(Types.java:1936)
at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
at com.xensource.xenapi.SR.scan(SR.java:1241)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getNfsSR(CitrixResourceBase.java:6529)
... 12 more
WARN [datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl]
(2004014557@qtp-1799775865-8:) Can not create storage pool through host 1 due
to Catch Exception com.cloud.utils.exception.CloudRuntimeException, create
StoragePool failed due to com.cloud.utils.exception.CloudRuntimeException:
Unable to create NFS SR Pool[1|172.16.197.134:2049|/opt/storage/primary] on
host:846b5000-897d-4eee-8497-3b7992bdc40e pool:
172.16.197.134/opt/storage/primary
ERROR [cloud.api.ApiServer] (2004014557@qtp-1799775865-8:) unhandled exception
executing api command: createStoragePool
com.cloud.utils.exception.CloudRuntimeException: Failed to add data store
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:752)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:183)
at
org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:144)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:528)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:371)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Can not create
storage pool through host 1 due to Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed due
to com.cloud.utils.exception.CloudRuntimeException: Unable to create NFS SR
Pool[1|172.16.197.134:2049|/opt/storage/primary] on
host:846b5000-897d-4eee-8497-3b7992bdc40e pool:
172.16.197.134/opt/storage/primary
at
org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.createStoragePool(CloudStackPrimaryDataStoreLifeCycleImpl.java:355)
at
org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:378)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:745)
... 27 more
WARN [xen.resource.CitrixResourceBase] (DirectAgent-3:) Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed due
to com.cloud.utils.exception.CloudRuntimeException: There is a SR using the
same configuration server:172.16.197.134, serverpath:/opt/storage/primary for
pool 75719dd8-db6b-398f-924a-c6d088e442d2on
host:846b5000-897d-4eee-8497-3b7992bdc40e on
host:846b5000-897d-4eee-8497-3b7992bdc40e pool:
172.16.197.134/opt/storage/primary
com.cloud.utils.exception.CloudRuntimeException: There is a SR using the same
configuration server:172.16.197.134, serverpath:/opt/storage/primary for pool
75719dd8-db6b-398f-924a-c6d088e442d2on host:846b5000-897d-4eee-8497-3b7992bdc40e
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getNfsSR(CitrixResourceBase.java:6519)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:5541)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:560)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:680)
--------
[root@xenserver1-cloudstack ~]# xe sr-list
....
uuid ( RO) : 66f0805b-da95-872c-2a6e-d5c46c219c2a
name-label ( RW): 75719dd8-db6b-398f-924a-c6d088e442d2
name-description ( RW): 1
host ( RO): xenserver1-cloudstack
type ( RO): nfs
content-type ( RO): user
.....
[root@xenserver1-cloudstack ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
.....
172.16.197.134:/opt/storage/primary
19856384 2560128 16300160 14%
/var/run/sr-mount/66f0805b-da95-872c-2a6e-d5c46c219c2a
[root@xenserver1-cloudstack ~]# xe pbd-list
sr-uuid=66f0805b-da95-872c-2a6e-d5c46c219c2a
uuid ( RO) : 2e2aced7-f27f-917f-a017-68f3504967f0
..
[root@xenserver1-cloudstack ~]# xe pbd-unplug
uuid=2e2aced7-f27f-917f-a017-68f3504967f0
[root@xenserver1-cloudstack ~]# xe sr-forget
uuid=66f0805b-da95-872c-2a6e-d5c46c219c2a
