[ https://issues.apache.org/jira/browse/CLOUDSTACK-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Animesh Chaturvedi resolved CLOUDSTACK-5747. -------------------------------------------- Resolution: Fixed > [Upgrade]Network restart failed after upgarding from 2.2.16 to 4.3 with > External Firewall SRX added to CS. > ---------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-5747 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5747 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Network Controller, Network Devices, Upgrade > Affects Versions: 4.3.0 > Environment: upgraded the CS2.2.16 with SRX to 4.3 > Reporter: manasaveloori > Assignee: Jayapal Reddy > Fix For: 4.3.0 > > Attachments: management-server.log.rar, mysqldump2216.dmp, > mysqldump4.3.dmp > > > Steps: > 1. Deploy CS 2.2 X.16 using Xen5.6 sp2 HV. > 2. Add the External firewall SRX to CS. > 3. Set the GC parameter firewall.rule.ui.enabled to "true." > 4. Now acquire the IP and configure firewall and PF rules. > 5. Upgrade the CS to 4.3. > 6. Stop and start all the System VMs and router VMs so that the new template > is upgraded. > 7. Now perform Network restart on which the firwall and PF rules are > configured. > Observation : > Observed the follwoing exceptions in Ms logs and Network restart failed. > 2014-01-03 17:43:32,329 DEBUG [c.c.n.r.JuniperSrxResource] > (DirectAgent-76:ctx-2128a4a2) Added Egress firewall rule for guest network 965 > 2014-01-03 17:43:32,329 DEBUG [c.c.n.r.JuniperSrxResource] > (DirectAgent-76:ctx-2128a4a2) Sending request: <!--Licensed to the Apache > Software Foundation (ASF) under oneor more contributor license agreements. > See the NOTICE filedistributed with this work for additional > informationregarding copyright ownership. The ASF licenses this fileto you > under the Apache License, Version 2.0 (the"License"); you may not use this > file except in compliancewith the License. You may obtain a copy of the > License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by > applicable law or agreed to in writing,software distributed under the License > is distributed on an"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF > ANYKIND, either express or implied. See the License for thespecific language > governing permissions and limitationsunder the > License.--><rpc><commit-configuration></commit-configuration></rpc> > 2014-01-03 17:43:33,966 DEBUG [c.c.n.r.JuniperSrxResource] > (DirectAgent-76:ctx-2128a4a2) Checking response: <rpc-reply > xmlns:junos="http://xml.juniper.net/junos/10.4R6/junos"><commit-results><load-success/><xnm:error > xmlns="http://xml.juniper.net/xnm/1.1/xnm" > xmlns:xnm="http://xml.juniper.net/xnm/1.1/xnm"><source-daemon>mgd</source-daemon><edit-path>[edit > security policies from-zone trust to-zone untrust policy > egress-trust-untrust-965]</edit-path><statement>match</statement><message>Missing > mandatory statement: 'source-address'</message></xnm:error><xnm:error > xmlns="http://xml.juniper.net/xnm/1.1/xnm" > xmlns:xnm="http://xml.juniper.net/xnm/1.1/xnm"><message>commit failed: > (missing statements)</message></xnm:error></commit-results></rpc-reply> > 2014-01-03 17:43:33,966 ERROR [c.c.n.r.JuniperSrxResource] > (DirectAgent-76:ctx-2128a4a2) Request failed due to: Missing mandatory > statement: 'source-address' > 2014-01-03 17:43:33,967 ERROR [c.c.n.r.JuniperSrxResource] > (DirectAgent-76:ctx-2128a4a2) com.cloud.utils.exception.ExecutionException: > Failed to commit to global configuration. > 2014-01-03 17:43:33,967 DEBUG [c.c.n.r.JuniperSrxResource] > (DirectAgent-76:ctx-2128a4a2) Sending request: <!--Licensed to the Apache > Software Foundation (ASF) under oneor more contributor license agreements. > See the NOTICE filedistributed with this work for additional > informationregarding copyright ownership. The ASF licenses this fileto you > under the Apache License, Version 2.0 (the"License"); you may not use this > file except in compliancewith the License. You may obtain a copy of the > License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by > applicable law or agreed to in writing,software distributed under the License > is distributed on an"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF > ANYKIND, either express or implied. See the License for thespecific language > governing permissions and limitationsunder the > License.--><rpc><close-configuration/></rpc> > 2014-01-03 17:43:34,012 DEBUG [c.c.n.r.JuniperSrxResource] > (DirectAgent-76:ctx-2128a4a2) Checking response: <rpc-reply > xmlns:junos="http://xml.juniper.net/junos/10.4R6/junos"></rpc-reply> > 2014-01-03 17:43:34,012 DEBUG [c.c.n.r.JuniperSrxResource] > (DirectAgent-76:ctx-2128a4a2) Closed private configuration. > 2014-01-03 17:43:34,013 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-76:ctx-2128a4a2) Seq 5-1877934113: Response Received: > 2014-01-03 17:43:34,014 DEBUG [c.c.a.t.Request] (DirectAgent-76:ctx-2128a4a2) > Seq 5-1877934113: Processing: { Ans: , MgmtId: 7588401905746, via: 5, Ver: > v1, Flags: 10, > [{"com.cloud.agent.api.Answer":{"result":false,"details":"Exception: > com.cloud.utils.exception.ExecutionException\nMessage: Failed to commit to > global configuration.\nStack: com.cloud.utils.exception.ExecutionException: > Failed to commit to global configuration.\n\tat > com.cloud.network.resource.JuniperSrxResource.commitConfiguration(JuniperSrxResource.java:654)\n\tat > > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:881)\n\tat > > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912)\n\tat > > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912)\n\tat > > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912)\n\tat > > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:830)\n\tat > > com.cloud.network.resource.JuniperSrxResource.executeRequest(JuniperSrxResource.java:353)\n\tat > > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)\n\tat > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)\n\tat > java.util.concurrent.FutureTask.run(FutureTask.java:166)\n\tat > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)\n\tat > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)\n\tat > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat > java.lang.Thread.run(Thread.java:636)\n","wait":0}}] } > 2014-01-03 17:43:34,014 DEBUG [c.c.a.t.Request] (Job-Executor-53:ctx-4d95c752 > ctx-99ce704c) Seq 5-1877934113: Received: { Ans: , MgmtId: 7588401905746, > via: 5, Ver: v1, Flags: 10, { Answer } } > 2014-01-03 17:43:34,015 DEBUG [c.c.a.m.AgentManagerImpl] > (Job-Executor-53:ctx-4d95c752 ctx-99ce704c) Details from executing class > com.cloud.agent.api.routing.SetFirewallRulesCommand: Exception: > com.cloud.utils.exception.ExecutionException > Message: Failed to commit to global configuration. > Stack: com.cloud.utils.exception.ExecutionException: Failed to commit to > global configuration. > at > com.cloud.network.resource.JuniperSrxResource.commitConfiguration(JuniperSrxResource.java:654) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:881) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:830) > at > com.cloud.network.resource.JuniperSrxResource.executeRequest(JuniperSrxResource.java:353) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > 2014-01-03 17:43:34,015 ERROR [c.c.n.ExternalFirewallDeviceManagerImpl] > (Job-Executor-53:ctx-4d95c752 ctx-99ce704c) External firewall was unable to > apply static nat rules to the SRX appliance in zone zonexen due to: > Exception: com.cloud.utils.exception.ExecutionException > Message: Failed to commit to global configuration. > Stack: com.cloud.utils.exception.ExecutionException: Failed to commit to > global configuration. > at > com.cloud.network.resource.JuniperSrxResource.commitConfiguration(JuniperSrxResource.java:654) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:881) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:830) > at > com.cloud.network.resource.JuniperSrxResource.executeRequest(JuniperSrxResource.java:353) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > . > 2014-01-03 17:43:34,015 WARN [c.c.n.f.FirewallManagerImpl] > (Job-Executor-53:ctx-4d95c752 ctx-99ce704c) Failed to apply firewall rules > due to > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is > unreachable: External firewall was unable to apply static nat rules to the > SRX appliance in zone zonexen due to: Exception: > com.cloud.utils.exception.ExecutionException > Message: Failed to commit to global configuration. > Stack: com.cloud.utils.exception.ExecutionException: Failed to commit to > global configuration. > at > com.cloud.network.resource.JuniperSrxResource.commitConfiguration(JuniperSrxResource.java:654) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:881) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:912) > at > com.cloud.network.resource.JuniperSrxResource.execute(JuniperSrxResource.java:830) > at > com.cloud.network.resource.JuniperSrxResource.executeRequest(JuniperSrxResource.java:353) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > . > at > com.cloud.network.ExternalFirewallDeviceManagerImpl.sendFirewallRules(ExternalFirewallDeviceManagerImpl.java:616) > at > com.cloud.network.ExternalFirewallDeviceManagerImpl.applyFirewallRules(ExternalFirewallDeviceManagerImpl.java:573) > at > com.cloud.network.element.JuniperSRXExternalFirewallElement.applyFWRules(JuniperSRXExternalFirewallElement.java:233) > at > com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:569) > at > com.cloud.network.IpAddressManagerImpl.applyRules(IpAddressManagerImpl.java:502) > at > com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:523) > at > com.cloud.network.firewall.FirewallManagerImpl.applyFirewallRules(FirewallManagerImpl.java:643) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy174.applyFirewallRules(Unknown Source) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.reprogramNetworkRules(NetworkOrchestrator.java:1106) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1063) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2370) > at > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1839) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy199.restartNetwork(Unknown Source) > at > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > at > com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109) > at > com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManage > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManaged > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:4 > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > 2014-01-03 17:43:34,021 WARN [o.a.c.e.o.NetworkOrchestrator] > (Job-Executor-53:ctx-4d95c752 ctx-99ce704c) > 2014-01-03 17:43:34,055 DEBUG [c.c.a.t.Request] (Job-Executor-53:ctx-4d95c752 > ctx-99ce704c) Seq 5-1877934 > v1, Flags: 100011, > [{"com.cloud.agent.api.routing.SetPortForwardingRulesCommand":{"rules":[{"dstIp":"10.0 > > > "protocol":"tcp","srcPortRange":[22,22],"revoked":false,"alreadyAdded":true,"purpose":"PortForwarding","d > > > VlanTag":"untagged","srcIp":"10.147.47.6","protocol":"tcp","srcPortRange":[22,22],"revoked":false,"alread > > ":{},"wait":0}}] } > Attaching the DB dumps and MSlogs: -- This message was sent by Atlassian JIRA (v6.1.5#6160)