Github user borisstoyanov commented on the issue:
https://github.com/apache/cloudstack/pull/1854
Hi @mike-tutkowski @syed,
I've been investigating the test failure from test_primary_storage.py:
test_01_primary_storage_iscsi, and we came to a conclusion that iscsi attaches
but it's not writeable.
By any chance do you recall any change that might cause this?
Here's the management log failing to create a hearbeat volume:
```
2016-12-23 11:30:58,826 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-404:ctx-c95759e3) (logid:f96038aa) Setting up the heartbeat sr for
host 10.2.2.33 and sr ca748d82-3668-9559-07ba-19ff94844ed0
2016-12-23 11:30:59,104 WARN
[c.c.h.x.r.w.x.CitrixModifyStoragePoolCommandWrapper]
(DirectAgent-404:ctx-c95759e3) (logid:f96038aa) ModifyStoragePoolCommand add
XenAPIException:Unable to setup heartbeat
sr on SR ca748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create
heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06d822
host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool: 10.5.0.4/iqn.2016-10.l
ocal.marvin:tgt1/0
com.cloud.utils.exception.CloudRuntimeException: Unable to setup heartbeat
sr on SR ca748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create
heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06d
822
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.setupHeartbeatSr(CitrixResourceBase.java:4596)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixModifyStoragePoolCommandWrapper.execute(CitrixModifyStoragePoolCommandWrapper.java:54)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixModifyStoragePoolCommandWrapper.execute(CitrixModifyStoragePoolCommandWrapper.java:40)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1687)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-12-23 11:30:59,111 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-404:ctx-c95759e3) (logid:f96038aa) Seq 1-4225502350380368681:
Response Received:
2016-12-23 11:30:59,111 DEBUG [c.c.a.t.Request]
(DirectAgent-404:ctx-c95759e3) (logid:f96038aa) Seq 1-4225502350380368681:
Processing: { Ans: , MgmtId: 7003044906821, via: 1(pr1854-t722-xenserver-65sp1
-xs1), Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"ModifyStoragePoolCommand
add XenAPIException:Unable to setup heartbeat sr on SR
ca748d82-3668-9559-07ba-19ff94844ed0
due to #6# Unable to create heartbeat volume
hb-db2a4468-ef57-4936-9ccd-7b8f4c06d822
host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool:
10.5.0.4/iqn.2016-10.local.marvin:tgt1/0","wait":0}}] }
2016-12-23 11:30:59,111 DEBUG [c.c.a.t.Request]
(catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Seq
1-4225502350380368681: Received: { Ans: , MgmtId: 7003044906821, via: 1(pr
1854-t722-xenserver-65sp1-xs1), Ver: v1, Flags: 10, { Answer } }
2016-12-23 11:30:59,111 DEBUG [c.c.a.m.AgentManagerImpl]
(catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa)
Details from executing class com.cloud.agent.api.ModifyStoragePoolComm
and: ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat
sr on SR ca748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create
heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06
d822 host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool:
10.5.0.4/iqn.2016-10.local.marvin:tgt1/0
2016-12-23 11:30:59,112 WARN [o.a.c.alerts] (catalina-exec-18:ctx-ecf415b7
ctx-29dead47 ctx-1fd26575) (logid:f96038aa) alertType:: 7 // dataCenterId:: 1
// podId:: 1 // clusterId:: null // message:: U
nable to attach storage pool11 to the host1
2016-12-23 11:30:59,120 WARN
[o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
(catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa)
Unable to establish a connection between Ho
st[-1-Routing] and
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@70894d68
com.cloud.utils.exception.CloudRuntimeException: Unable establish
connection from storage head to storage pool 11 due to ModifyStoragePoolCommand
add XenAPIException:Unable to setup heartbeat sr on SR c
a748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create heartbeat
volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06d822
host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool:
10.5.0.4/iqn.2016-10.local.marvin
:tgt1/011
at
org.apache.cloudstack.storage.datastore.provider.DefaultHostListener.hostConnect(DefaultHostListener.java:75)
at
com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:955)
at
org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:414)
at
com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:709)
at
com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:199)
at
org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:709)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:533)
at
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:299)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:129)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:126)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:299)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:732)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-12-23 11:30:59,121 WARN
[o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
(catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) No
host can access storage pool org.apache.
cloudstack.storage.datastore.PrimaryDataStoreImpl@70894d68 on cluster 1
2016-12-23 11:30:59,127 DEBUG [c.c.s.StorageManagerImpl]
(catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa)
Failed to add data store: Failed to access storage pool
com.cloud.utils.exception.CloudRuntimeException: Failed to access storage
pool
at
org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:427)
at
com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:709)
at
com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:199)
at
org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:709)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:533)
at
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:299)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:129)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:126)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:299)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:732)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-12-23 11:30:59,129 DEBUG [c.c.s.StorageManagerImpl]
(catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa)
Failed to clean up storage pool: null
2016-12-23 11:30:59,130 INFO [c.c.a.ApiServer]
(catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa)
Failed to add data store: Failed to access storage pool
2016-12-23 11:30:59,130 DEBUG [c.c.a.ApiServlet]
(catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa)
===END=== 10.5.0.4 -- GET
apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&name=Test+iSCSI&url=iscsi%3A%2F%2F10.5.0.4%2Fiqn.2016-10.local.marvin%3Atgt1%2F0&podid=8b409e63-f298-4113-a23b-0700a8dfc76d&clusterid=4b2458e0-9da9-4a99-987e-3d2cd6f2f0a9&zoneid=8d414040-93d1-47d9-8706-7a15f3c1dfb6&command=createStoragePool&signature=eLjbko60Wfjj4iny2w0ktjvKcK4%3D&response=json
2016-12-23 11:30:59,143 DEBUG [c.c.a.ApiServlet]
(catalina-exec-16:ctx-9fd311e8) (logid:2d0d3d40) ===START=== 10.5.0.4 -- GET
apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5
```
Marvin test failure:
`2016-12-23 11:30:58,520 - CRITICAL - EXCEPTION:
test_01_primary_storage_iscsi: ['Traceback (most recent call last):\n', ' File
"/usr/lib64/python2.7/unittest/case.py", line 369, in run\n testMethod()\n',
' File "/marvin/tests/smoke/test_primary_storage.py", line 199, in
test_01_primary_storage_iscsi\n podid=self.pod.id\n', ' File
"/usr/lib/python2.7/site-packages/marvin/lib/base.py", line 2732, in create\n
return StoragePool(apiclient.createStoragePool(cmd).__dict__)\n', ' File
"/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py",
line 2537, in createStoragePool\n response =
self.connection.marvinRequest(command, response_type=response,
method=method)\n', ' File
"/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in
marvinRequest\n raise e\n', 'CloudstackAPIException: Execute cmd:
createstoragepool failed, due to: errorCode: 530, errorText:Failed to add data
store: Failed to access storage pool\n']`
CC: @rhtyd @PaulAngus
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
