[ https://issues.apache.org/jira/browse/CLOUDSTACK-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731661#comment-13731661 ]
Min Chen commented on CLOUDSTACK-4125: -------------------------------------- In 4.1, two new columns (state and update_count) are added to template_spool_ref table, we need to set those two columns to make sure state transition work properly. > [Upgrade] after upgrade 3.0.7 GA -> 4.2 deploy guest VM fail > ------------------------------------------------------------ > > Key: CLOUDSTACK-4125 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4125 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.2.0 > Environment: MS 10.223.195.62 campo 4.2 > host 10.223.51.3 XS 6.1 > Reporter: angeline shen > Assignee: Min Chen > Priority: Blocker > Fix For: 4.2.0 > > Attachments: management-server.log.gz > > > MS 10.223.195.62 build 3.0.7 GA upgrade 4.2 build host 10.223.51.3 XS 6.1 > Anthony says this bug is related to Upgrade: > 1. MS install 3.0.7 GA, > 2. create advanced zone with SG > 3. create several networks with different subnets and same vlan > vlan GW start IP end IP > ---------------------------------------------------------------------------------------------------------- > guest network 1250 10.223.125.1 10.223.125.13 10.223.125.18 > > 1250-66to68 1250 10.223.125.65 10.223.125.66 10.223.125.68 > 1250-130to132 1250 10.223.125.129 10.223.125.130 10.223.125.132 > 4. start VMs on each of these networks > 125.18 125.16 125.14 125.68 125.67 125.131 125.132 > no ingress or egress rules : > VMs can ssh + ping other VMs in all subnets > VMs can egress internet > add ingress TCP icmp egress TCP egress > 5. make sure VM work well > 6. upgrade to 4.2 > 7. existing guest VM works welll > 8. check multiple subnets > - add new IP range to subnet in network > - start new VM in network in second subnet - FAILed > 2013-08-06 17:00:35,556 DEBUG [storage.volume.VolumeServiceImpl] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Acquire > lock on VMTemplateStoragePool 2 with timeout 3600 seconds > 2013-08-06 17:00:35,557 INFO [storage.volume.VolumeServiceImpl] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) lock is > acquired for VMTemplateStoragePool 2 > 2013-08-06 17:00:35,559 DEBUG [storage.volume.VolumeServiceImpl] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) failed to > create template on storage > com.cloud.utils.exception.CloudRuntimeException: Failed to update state > at > org.apache.cloudstack.storage.image.store.TemplateObject.processEvent(TemplateObject.java:164) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:433) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:538) > at > com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2488) > at > com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624) > at > org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227) > at > org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3408) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2968) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > Caused by: com.cloud.utils.fsm.NoTransitionException: Unable to transition to > a new state from null via CreateOnlyRequested > at > com.cloud.utils.fsm.StateMachine2.getNextState(StateMachine2.java:83) > at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:100) > at > org.apache.cloudstack.storage.datastore.ObjectInDataStoreManagerImpl.update(ObjectInDataStoreManagerImpl.java:258) > at > org.apache.cloudstack.storage.image.store.TemplateObject.processEvent(TemplateObject.java:162) > ... 25 more > 2013-08-06 17:00:35,579 INFO [storage.volume.VolumeServiceImpl] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) releasing > lock for VMTemplateStoragePool 2 > 2013-08-06 17:00:35,579 ERROR [cloud.vm.VirtualMachineManagerImpl] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Failed to > start instance VM[User|z1upgrade1250-130to132] > com.cloud.utils.exception.CloudRuntimeException: Failed to update state > at > org.apache.cloudstack.storage.image.store.TemplateObject.processEvent(TemplateObject.java:164) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:437) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:538) > at > com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2488) > at > com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624) > at > org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227) > at > org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3408) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2968) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > Caused by: com.cloud.utils.fsm.NoTransitionException: Unable to transition to > a new state from null via OperationFailed > at > com.cloud.utils.fsm.StateMachine2.getNextState(StateMachine2.java:83) > at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:100) > at > org.apache.cloudstack.storage.datastore.ObjectInDataStoreManagerImpl.update(ObjectInDataStoreManagerImpl.java:258) > at > org.apache.cloudstack.storage.image.store.TemplateObject.processEvent(TemplateObject.java:162) > 2013-08-06 17:00:35,584 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Cleaning > up resources for the vm VM[User|z1upgrade1250-130to132] in Starting state > 2013-08-06 17:00:35,585 DEBUG [agent.transport.Request] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Seq > 1-1201344676: Sending { Cmd , MgmtId: 7264031213276, via: 1, Ver: v1, Flags: > 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-12-VM","wait":0}}] > } > 2013-08-06 17:00:35,585 DEBUG [agent.transport.Request] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Seq 1-120 > 2013-08-06 17:00:35,852 DEBUG [cloud.vm.UserVmManagerImpl] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Destroying > vm VM[User|z1upgrade1250-130to132] as it failed to create on Host with Id:null > 2013-08-06 17:00:35,866 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) VM state > transitted from :Stopped to Error with event: OperationFailedToErrorvm's > original host id: null new host id: null host id before state transition: null > 2013-08-06 17:00:36,131 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) > ===START=== 10.216.133.70 -- GET > command=queryAsyncJobResult&jobId=f2595277-039b-4166-a8c3-01044c1b4cec&response=json&sessionkey=C2NtUAEudxOaSrj3%2B%2ByPxrOyXMo%3D&_=1375834266650 > 2013-08-06 17:00:36,158 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) > ===END=== 10.216.133.70 -- GET > command=queryAsyncJobResult&jobId=f2595277-039b-4166-a8c3-01044c1b4cec&response=json&sessionkey=C2NtUAEudxOaSrj3%2B%2ByPxrOyXMo%3D&_=1375834266650 > 2013-08-06 17:00:36,176 WARN [apache.cloudstack.alerts] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) > alertType:: 8 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // > message:: Failed to deploy Vm with Id: 12, on Host with Id: null > 2013-08-06 17:00:36,209 INFO [user.vm.DeployVMCmd] (Job-Executor-8:job-21 = > [ f2595277-039b-4166-a8c3-01044c1b4cec ]) > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|z1upgrade1250-130to132]Scope=interface > com.cloud.dc.DataCenter; id=1 > 2013-08-06 17:00:36,209 INFO [user.vm.DeployVMCmd] (Job-Executor-8:job-21 = > [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Unable to create a deployment for > VM[User|z1upgrade1250-130to132] > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|z1upgrade1250-130to132]Scope=interface > com.cloud.dc.DataCenter; id=1 > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624) > at > org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:237) > at > org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3408) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2968) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-08-06 17:00:36,210 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Complete > async job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ], jobStatus: 2, > resultCode: 530, result: Error Code: 533 Error text: Unable to create a > deployment for VM[User|z1upgrade1250-130to132] > 2013-08-06 17:00:39,128 DEBUG [cloud.api.ApiServlet] (catalina-exec-23:null) > ===START=== 10.216.133.70 -- GET > command=queryAsyncJobResult&jobId=f2595277-039b-4166-a8c3-01044c1b4cec&response=json&sessionkey=C2NtUAEudxOaSrj3%2B%2ByPxrOyXMo%3D&_=1375834269647 > 2013-08-06 17:00:39,141 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-23:null) Async job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec > ] completed > ------------------------------------------------------------------------------------------ > 9. destroy 1 guest VM. Deploy guest VM : VM remain in STOPPED state > 2013-08-06 17:24:57,230 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) No > suitable hosts found > 2013-08-06 17:24:57,230 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) No > suitable hosts found under this Cluster: 1 > 2013-08-06 17:24:57,232 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Could not > find suitable Deployment Destinatio > n for this VM under any clusters, returning. > 2013-08-06 17:24:57,232 DEBUG [cloud.deploy.FirstFitPlanner] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Searching > resources only under specified Pod: 1 > 2013-08-06 17:24:57,232 DEBUG [cloud.deploy.FirstFitPlanner] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Listing > clusters in order of aggregate capacity, that have (atleast one host with) > enough CPU and RAM capacity under this Pod: 1 > 2013-08-06 17:24:57,235 DEBUG [cloud.deploy.FirstFitPlanner] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Removing > from the clusterId list these clusters from avoid set: [1] > 2013-08-06 17:24:57,235 DEBUG [cloud.deploy.FirstFitPlanner] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) No > clusters found after removing disabled clusters and clusters in avoid list, > returning. > 2013-08-06 17:24:57,280 DEBUG [cloud.api.ApiServlet] (catalina-exec-14:null) > ===START=== 10.216.133.64 -- GET > command=queryAsyncJobResult&jobId=85b0ca6d-7586-4225-a9aa-b772ea99e3a8&response=json&sessionkey=kxfo3DkYEzBv44N8ynxFwuMn6bE%3D&_=1375835530701 > 2013-08-06 17:24:57,441 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) VM state > transitted from :Starting to Stopped with event: OperationFailedvm's original > host id: null new host id: null host id before state transition: null > 2013-08-06 17:24:57,447 DEBUG [cloud.api.ApiServlet] (catalina-exec-14:null) > ===END=== 10.216.133.64 -- GET > command=queryAsyncJobResult&jobId=85b0ca6d-7586-4225-a9aa-b772ea99e3a8&response=json&sessionkey=kxfo3DkYEzBv44N8ynxFwuMn6bE%3D&_=1375835530701 > 2013-08-06 17:24:57,460 DEBUG [cloud.vm.UserVmManagerImpl] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Destroying > vm VM[User|187ff69f-e0d9-4364-bef6-86b59690b152] as it failed to create on > Host with Id:null > 2013-08-06 17:24:58,427 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) VM state > transitted from :Stopped to Error with event: OperationFailedToErrorvm's > original host id: null new host id: null host id before state transition: null > 2013-08-06 17:24:58,446 WARN [apache.cloudstack.alerts] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) > alertType:: 8 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // > message:: Failed to deploy Vm with Id: 14, on Host with Id: null > 2013-08-06 17:24:58,476 INFO [user.vm.DeployVMCmd] (Job-Executor-1:job-24 = > [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|187ff69f-e0d9-4364-bef6-86b59690b152]Scope=interface > com.cloud.dc.DataCenter; id=1 > 2013-08-06 17:24:58,476 INFO [user.vm.DeployVMCmd] (Job-Executor-1:job-24 = > [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Unable to create a deployment for > VM[User|187ff69f-e0d9-4364-bef6-86b59690b152] > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|187ff69f-e0d9-4364-bef6-86b59690b152]Scope=interface > com.cloud.dc.DataCenter; id=1 > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624) > at > org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:237) > at > org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3408) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2968) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-08-06 17:24:58,477 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Complete > async job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ], jobStatus: 2, > resultCode: 530, result: Error Code: 533 Error text: Unable to create a > deployment for VM[User|187ff69f-e0d9-4364-bef6-86b59690b152] > 2013-08-06 17:24:59,563 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-9:null) SeqA 3-788: Processing Seq 3-788: { Cmd , > MgmtId: -1, via: 3, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n > \"connections\": []\n}","wait":0}}] } > 2013-08-06 17:24:59,765 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-9:null) SeqA 3-788: Sending Seq 3-788: { Ans: , > MgmtId: 7264031213276, via: 3, Ver: v1, Flags: 100010, > [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } > ............ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira