[
https://issues.apache.org/jira/browse/CLOUDSTACK-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
manasaveloori updated CLOUDSTACK-5849:
--------------------------------------
Description:
Steps:
1. Deployed CS 3.0.6 Patch E with Xen 6.0.2 HV.
2. Created the service offering for netscaler device.
3. Added incompatible version of Netscaler. added NS10.1: Build 120.1316 to
CS 3.0.6 build.
it was success.
4. Now created the network using the service offering created in step 2.
5. Observed that the network went into shut down state as the netscaler failed
to implement the network.
6. Removed the netscaler device from CS and disabled the service offering.
7. Now there were no VMs associated to that network.Tried to delete the
network.But it failed as the network was not in correct state.
Note: issue existed in 3.0.6 that network is not deleted if it is in shutdown
state.
2014-01-07 21:32:09,720 DEBUG [agent.manager.AgentManagerImpl]
(RouterMonitor-1:null) Details from executing class
com.cloud.agent.api.NetworkUsageCommand:
2014-01-07 21:32:09,720 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:null)
Recieved and Sent bytes are both 0. Not updating user_statistics
2014-01-07 21:32:11,498 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-5:null) Ping from 8
2014-01-07 21:32:11,917 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-6:null) Ping from 5
2014-01-07 21:32:12,164 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-19:null) submit async job-75, job: AsyncJobVO {id:75, userId: 2,
accountId: 2, sessionKey: null, instanceType: null, instanceId: null, cmd:
com.cloud.api.commands.DeleteNetworkCmd, cmdOriginator: null, cmdVersion: 0,
callbackType: 0, callbackAddress: null, status: 0, processStatus: 0,
resultCode: 0, result: null, initMsid: 6642334695485, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null}
2014-01-07 21:32:12,169 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-69:job-75) Executing com.cloud.api.commands.DeleteNetworkCmd for
job-75
2014-01-07 21:32:12,179 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-69:job-75) Sync job-75 execution on object network.216
2014-01-07 21:32:12,188 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-69:job-75) job com.cloud.api.commands.DeleteNetworkCmd for job-75
was queued, processing the queue.
2014-01-07 21:32:12,197 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-69:job-75) Executing sync queue item: SyncQueueItemVO {id:17,
queueId: 17, contentType: AsyncJob, contentId: 75, lastProcessMsid:
6642334695485, lastprocessNumber: 1, lastProcessTime: Tue Jan 07 21:32:12 IST
2014, created: Tue Jan 07 21:32:12 IST 2014}
2014-01-07 21:32:12,199 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-69:job-75) Schedule queued job-75
2014-01-07 21:32:12,207 DEBUG [cloud.async.SyncQueueManagerImpl]
(Job-Executor-69:job-75) There is a pending process in sync queue(id: 17)
2014-01-07 21:32:12,210 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-70:job-75) Executing com.cloud.api.commands.DeleteNetworkCmd for
job-75
2014-01-07 21:32:12,233 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-70:job-75) Network is not implemented: Ntwk[216|Guest|17]
2014-01-07 21:32:12,233 DEBUG [db.Transaction.Transaction]
(Job-Executor-70:job-75) Rolling back the transaction: Time = 1 Name =
-AsyncJobManagerImpl$1.run:396-Executors$RunnableAdapter.call:471-FutureTask$Sync.innerRun:334-FutureTask.run:166-ThreadPoolExecutor.runWorker:1110-ThreadPoolExecutor$Worker.run:603-Thread.run:679;
called by
-Transaction.rollback:854-Transaction.removeUpTo:797-Transaction.close:621-DatabaseCallback.interceptComplete:67-DatabaseCallback.intercept:32-NetworkManagerImpl.destroyNetwork:3799-DatabaseCallback.intercept:30-NetworkManagerImpl.deleteNetwork:3656-ActionEventCallback.intercept:32-DeleteNetworkCmd.execute:65-ApiDispatcher.dispatch:263-AsyncJobManagerImpl$1.run:430
2014-01-07 21:32:12,235 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-70:job-75) Network is not not in the correct state to be
destroyed: Shutdown
2014-01-07 21:32:12,242 WARN [cloud.api.ApiDispatcher]
(Job-Executor-70:job-75) class com.cloud.api.ServerApiException : Failed to
delete network
2014-01-07 21:32:12,243 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-70:job-75) Complete async job-75, jobStatus: 2, resultCode: 530,
result: Error Code: 530 Error text: Failed to delete network
8. Now upgraded the CS to 4.3.
9. The network state changed to implemented.
10.Tried to delete the network now which failed.Following Error is observed:
2014-01-09 22:51:18,420 DEBUG [c.c.a.ApiServlet]
(catalina-exec-25:ctx-20a7144b) ===START=== 10.252.192.34 -- GET
command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
2014-01-09 22:51:18,459 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(catalina-exec-25:ctx-20a7144b ctx-bbdb1ebe) submit async job-217, details:
AsyncJobVO {id:217, userId: 2, accountId: 2, instanceType: None, instanceId:
null, cmd: org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
cmdInfo:
{"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2014-01-09 22:51:18,461 DEBUG [c.c.a.ApiServlet] (catalina-exec-25:ctx-20a7144b
ctx-bbdb1ebe) ===END=== 10.252.192.34 -- GET
command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
2014-01-09 22:51:18,463 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Job-Executor-39:ctx-a33dc0eb) Add job-217 into job monitoring
2014-01-09 22:51:18,463 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-39:ctx-a33dc0eb) Executing AsyncJobVO {id:217, userId: 2,
accountId: 2, instanceType: None, instanceId: null, cmd:
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
{"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2014-01-09 22:51:18,504 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Lock is acquired for network
Ntwk[217|Guest|17] as a part of network shutdown
2014-01-09 22:51:18,516 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 port forwarding rules
for network id=217 as a part of shutdownNetworkRules
2014-01-09 22:51:18,516 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
the network elements
2014-01-09 22:51:18,518 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 static nat rules for
network id=217 as a part of shutdownNetworkRules
2014-01-09 22:51:18,519 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
the network elements
2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Revoking 0 Public load balancing
rules for network id=217
2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no Load Balancing Rules
to forward to the network elements
2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Revoking 0 Internal load balancing
rules for network id=217
2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no Load Balancing Rules
to forward to the network elements
2014-01-09 22:51:18,525 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 firewall ingress rules
for network id=217 as a part of shutdownNetworkRules
2014-01-09 22:51:18,525 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
the network elements
2014-01-09 22:51:18,527 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 firewall egress rules
for network id=217 as a part of shutdownNetworkRules
2014-01-09 22:51:18,533 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) applying default firewall egress
rules
2014-01-09 22:51:18,553 DEBUG [c.c.n.e.VirtualRouterElement]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Virtual router elemnt doesn't need
to apply firewall rules on the backend; virtual router doesn't exist in the
network 217
2014-01-09 22:51:18,553 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
the network elements
2014-01-09 22:51:18,555 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Found 0 static nat rules to apply
for network id 217
2014-01-09 22:51:18,575 DEBUG [c.c.n.e.VirtualRouterElement]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Virtual router elemnt doesn't need
to associate ip addresses on the backend; virtual router doesn't exist in the
network 217
2014-01-09 22:51:18,583 WARN [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Unable to complete shutdown of the
network elements due to element: Netscaler either doesn't exist or not enabled
in the physical network 200
2014-01-09 22:51:18,583 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Sending network shutdown to
Netscaler
2014-01-09 22:51:18,588 WARN [c.c.n.ExternalLoadBalancerDeviceManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Network shutdwon requested on
external load balancer element, which did not implement the network. Either
network implement failed half way through or already network shutdown is
completed. So just returning.
2014-01-09 22:51:18,593 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Sending network shutdown to
VirtualRouter
2014-01-09 22:51:18,604 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Lock is released for network
Ntwk[217|Guest|17] as a part of network shutdown
2014-01-09 22:51:18,607 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Network is not not in the correct
state to be destroyed: Implemented
2014-01-09 22:51:18,619 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-39:ctx-a33dc0eb) Complete async job-217, jobStatus: FAILED,
resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to delete network"}
2014-01-09 22:51:18,630 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-39:ctx-a33dc0eb) Done executing
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd for job-217
Attaching the MS ,DB dumps.
was:
Steps:
1. Deployed CS 3.0.6 Patch E with Xen 6.0.2 HV.
2. Created the service offering for netscaler device.
3. Added incompatible version of Netscaler. added NS10.1: Build 120.1316 to
CS 3.0.6 build.
it was success.
4. Now created the network using the service offering created in step 2.
5. Observed that the network went into shut down state as the netscaler failed
to implement the network.
6. Removed the netscaler device from CS and disabled the service offering.
7. Now there were no VMs associated to that network.Tried to delete the
network.But it failed as the network was not in correct state.
Note: issue existed in 3.0.6 that network is not deleted if it is in shutdown
state.
8. Now upgraded the CS to 4.3.
9. The network state changed to implemented.
10.Tried to delete the network now which failed.Following Error is observed:
2014-01-09 22:51:18,420 DEBUG [c.c.a.ApiServlet]
(catalina-exec-25:ctx-20a7144b) ===START=== 10.252.192.34 -- GET
command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
2014-01-09 22:51:18,459 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(catalina-exec-25:ctx-20a7144b ctx-bbdb1ebe) submit async job-217, details:
AsyncJobVO {id:217, userId: 2, accountId: 2, instanceType: None, instanceId:
null, cmd: org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
cmdInfo:
{"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2014-01-09 22:51:18,461 DEBUG [c.c.a.ApiServlet] (catalina-exec-25:ctx-20a7144b
ctx-bbdb1ebe) ===END=== 10.252.192.34 -- GET
command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
2014-01-09 22:51:18,463 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Job-Executor-39:ctx-a33dc0eb) Add job-217 into job monitoring
2014-01-09 22:51:18,463 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-39:ctx-a33dc0eb) Executing AsyncJobVO {id:217, userId: 2,
accountId: 2, instanceType: None, instanceId: null, cmd:
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
{"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2014-01-09 22:51:18,504 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Lock is acquired for network
Ntwk[217|Guest|17] as a part of network shutdown
2014-01-09 22:51:18,516 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 port forwarding rules
for network id=217 as a part of shutdownNetworkRules
2014-01-09 22:51:18,516 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
the network elements
2014-01-09 22:51:18,518 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 static nat rules for
network id=217 as a part of shutdownNetworkRules
2014-01-09 22:51:18,519 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
the network elements
2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Revoking 0 Public load balancing
rules for network id=217
2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no Load Balancing Rules
to forward to the network elements
2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Revoking 0 Internal load balancing
rules for network id=217
2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no Load Balancing Rules
to forward to the network elements
2014-01-09 22:51:18,525 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 firewall ingress rules
for network id=217 as a part of shutdownNetworkRules
2014-01-09 22:51:18,525 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
the network elements
2014-01-09 22:51:18,527 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 firewall egress rules
for network id=217 as a part of shutdownNetworkRules
2014-01-09 22:51:18,533 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) applying default firewall egress
rules
2014-01-09 22:51:18,553 DEBUG [c.c.n.e.VirtualRouterElement]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Virtual router elemnt doesn't need
to apply firewall rules on the backend; virtual router doesn't exist in the
network 217
2014-01-09 22:51:18,553 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
the network elements
2014-01-09 22:51:18,555 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Found 0 static nat rules to apply
for network id 217
2014-01-09 22:51:18,575 DEBUG [c.c.n.e.VirtualRouterElement]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Virtual router elemnt doesn't need
to associate ip addresses on the backend; virtual router doesn't exist in the
network 217
2014-01-09 22:51:18,583 WARN [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Unable to complete shutdown of the
network elements due to element: Netscaler either doesn't exist or not enabled
in the physical network 200
2014-01-09 22:51:18,583 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Sending network shutdown to
Netscaler
2014-01-09 22:51:18,588 WARN [c.c.n.ExternalLoadBalancerDeviceManagerImpl]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Network shutdwon requested on
external load balancer element, which did not implement the network. Either
network implement failed half way through or already network shutdown is
completed. So just returning.
2014-01-09 22:51:18,593 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Sending network shutdown to
VirtualRouter
2014-01-09 22:51:18,604 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Lock is released for network
Ntwk[217|Guest|17] as a part of network shutdown
2014-01-09 22:51:18,607 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Network is not not in the correct
state to be destroyed: Implemented
2014-01-09 22:51:18,619 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-39:ctx-a33dc0eb) Complete async job-217, jobStatus: FAILED,
resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to delete network"}
2014-01-09 22:51:18,630 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-39:ctx-a33dc0eb) Done executing
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd for job-217
Attaching the MS ,DB dumps.
> Unable to delete the network after upgrading from 3.0.6 to 4.3
> --------------------------------------------------------------
>
> Key: CLOUDSTACK-5849
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5849
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Network Controller, Network Devices
> Affects Versions: 4.3.0
> Environment: upgraded from 3.0.6 patch E to 4.3
> Reporter: manasaveloori
> Priority: Critical
> Fix For: 4.3.0
>
>
> Steps:
> 1. Deployed CS 3.0.6 Patch E with Xen 6.0.2 HV.
> 2. Created the service offering for netscaler device.
> 3. Added incompatible version of Netscaler. added NS10.1: Build 120.1316
> to CS 3.0.6 build.
> it was success.
> 4. Now created the network using the service offering created in step 2.
> 5. Observed that the network went into shut down state as the netscaler
> failed to implement the network.
> 6. Removed the netscaler device from CS and disabled the service offering.
> 7. Now there were no VMs associated to that network.Tried to delete the
> network.But it failed as the network was not in correct state.
> Note: issue existed in 3.0.6 that network is not deleted if it is in shutdown
> state.
> 2014-01-07 21:32:09,720 DEBUG [agent.manager.AgentManagerImpl]
> (RouterMonitor-1:null) Details from executing class
> com.cloud.agent.api.NetworkUsageCommand:
> 2014-01-07 21:32:09,720 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:null)
> Recieved and Sent bytes are both 0. Not updating user_statistics
> 2014-01-07 21:32:11,498 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-5:null) Ping from 8
> 2014-01-07 21:32:11,917 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-6:null) Ping from 5
> 2014-01-07 21:32:12,164 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-19:null) submit async job-75, job: AsyncJobVO {id:75, userId:
> 2, accountId: 2, sessionKey: null, instanceType: null, instanceId: null, cmd:
> com.cloud.api.commands.DeleteNetworkCmd, cmdOriginator: null, cmdVersion: 0,
> callbackType: 0, callbackAddress: null, status: 0, processStatus: 0,
> resultCode: 0, result: null, initMsid: 6642334695485, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null}
> 2014-01-07 21:32:12,169 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-69:job-75) Executing com.cloud.api.commands.DeleteNetworkCmd
> for job-75
> 2014-01-07 21:32:12,179 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-69:job-75) Sync job-75 execution on object network.216
> 2014-01-07 21:32:12,188 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-69:job-75) job com.cloud.api.commands.DeleteNetworkCmd for
> job-75 was queued, processing the queue.
> 2014-01-07 21:32:12,197 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-69:job-75) Executing sync queue item: SyncQueueItemVO {id:17,
> queueId: 17, contentType: AsyncJob, contentId: 75, lastProcessMsid:
> 6642334695485, lastprocessNumber: 1, lastProcessTime: Tue Jan 07 21:32:12 IST
> 2014, created: Tue Jan 07 21:32:12 IST 2014}
> 2014-01-07 21:32:12,199 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-69:job-75) Schedule queued job-75
> 2014-01-07 21:32:12,207 DEBUG [cloud.async.SyncQueueManagerImpl]
> (Job-Executor-69:job-75) There is a pending process in sync queue(id: 17)
> 2014-01-07 21:32:12,210 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-70:job-75) Executing com.cloud.api.commands.DeleteNetworkCmd
> for job-75
> 2014-01-07 21:32:12,233 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-70:job-75) Network is not implemented: Ntwk[216|Guest|17]
> 2014-01-07 21:32:12,233 DEBUG [db.Transaction.Transaction]
> (Job-Executor-70:job-75) Rolling back the transaction: Time = 1 Name =
> -AsyncJobManagerImpl$1.run:396-Executors$RunnableAdapter.call:471-FutureTask$Sync.innerRun:334-FutureTask.run:166-ThreadPoolExecutor.runWorker:1110-ThreadPoolExecutor$Worker.run:603-Thread.run:679;
> called by
> -Transaction.rollback:854-Transaction.removeUpTo:797-Transaction.close:621-DatabaseCallback.interceptComplete:67-DatabaseCallback.intercept:32-NetworkManagerImpl.destroyNetwork:3799-DatabaseCallback.intercept:30-NetworkManagerImpl.deleteNetwork:3656-ActionEventCallback.intercept:32-DeleteNetworkCmd.execute:65-ApiDispatcher.dispatch:263-AsyncJobManagerImpl$1.run:430
> 2014-01-07 21:32:12,235 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-70:job-75) Network is not not in the correct state to be
> destroyed: Shutdown
> 2014-01-07 21:32:12,242 WARN [cloud.api.ApiDispatcher]
> (Job-Executor-70:job-75) class com.cloud.api.ServerApiException : Failed to
> delete network
> 2014-01-07 21:32:12,243 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-70:job-75) Complete async job-75, jobStatus: 2, resultCode:
> 530, result: Error Code: 530 Error text: Failed to delete network
> 8. Now upgraded the CS to 4.3.
> 9. The network state changed to implemented.
> 10.Tried to delete the network now which failed.Following Error is observed:
> 2014-01-09 22:51:18,420 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-25:ctx-20a7144b) ===START=== 10.252.192.34 -- GET
> command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
> 2014-01-09 22:51:18,459 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (catalina-exec-25:ctx-20a7144b ctx-bbdb1ebe) submit async job-217, details:
> AsyncJobVO {id:217, userId: 2, accountId: 2, instanceType: None, instanceId:
> null, cmd: org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> cmdInfo:
> {"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: null}
> 2014-01-09 22:51:18,461 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-25:ctx-20a7144b ctx-bbdb1ebe) ===END=== 10.252.192.34 -- GET
> command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
> 2014-01-09 22:51:18,463 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Job-Executor-39:ctx-a33dc0eb) Add job-217 into job monitoring
> 2014-01-09 22:51:18,463 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb) Executing AsyncJobVO {id:217, userId: 2,
> accountId: 2, instanceType: None, instanceId: null, cmd:
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
> {"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: null}
> 2014-01-09 22:51:18,504 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Lock is acquired for network
> Ntwk[217|Guest|17] as a part of network shutdown
> 2014-01-09 22:51:18,516 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 port forwarding rules
> for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,516 DEBUG [c.c.n.f.FirewallManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
> the network elements
> 2014-01-09 22:51:18,518 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 static nat rules for
> network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,519 DEBUG [c.c.n.f.FirewallManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
> the network elements
> 2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Revoking 0 Public load balancing
> rules for network id=217
> 2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no Load Balancing Rules
> to forward to the network elements
> 2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Revoking 0 Internal load
> balancing rules for network id=217
> 2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no Load Balancing Rules
> to forward to the network elements
> 2014-01-09 22:51:18,525 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 firewall ingress
> rules for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,525 DEBUG [c.c.n.f.FirewallManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
> the network elements
> 2014-01-09 22:51:18,527 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 firewall egress rules
> for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,533 DEBUG [c.c.n.f.FirewallManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) applying default firewall egress
> rules
> 2014-01-09 22:51:18,553 DEBUG [c.c.n.e.VirtualRouterElement]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Virtual router elemnt doesn't
> need to apply firewall rules on the backend; virtual router doesn't exist in
> the network 217
> 2014-01-09 22:51:18,553 DEBUG [c.c.n.f.FirewallManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
> the network elements
> 2014-01-09 22:51:18,555 DEBUG [c.c.n.r.RulesManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Found 0 static nat rules to apply
> for network id 217
> 2014-01-09 22:51:18,575 DEBUG [c.c.n.e.VirtualRouterElement]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Virtual router elemnt doesn't
> need to associate ip addresses on the backend; virtual router doesn't exist
> in the network 217
> 2014-01-09 22:51:18,583 WARN [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Unable to complete shutdown of
> the network elements due to element: Netscaler either doesn't exist or not
> enabled in the physical network 200
> 2014-01-09 22:51:18,583 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Sending network shutdown to
> Netscaler
> 2014-01-09 22:51:18,588 WARN [c.c.n.ExternalLoadBalancerDeviceManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Network shutdwon requested on
> external load balancer element, which did not implement the network. Either
> network implement failed half way through or already network shutdown is
> completed. So just returning.
> 2014-01-09 22:51:18,593 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Sending network shutdown to
> VirtualRouter
> 2014-01-09 22:51:18,604 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Lock is released for network
> Ntwk[217|Guest|17] as a part of network shutdown
> 2014-01-09 22:51:18,607 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Network is not not in the correct
> state to be destroyed: Implemented
> 2014-01-09 22:51:18,619 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb) Complete async job-217, jobStatus: FAILED,
> resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to delete network"}
> 2014-01-09 22:51:18,630 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb) Done executing
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd for job-217
> Attaching the MS ,DB dumps.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
