Hi Adrija, From the logs, the public subnet is untagged. I think this issue is coming for the untagged public vlan in 4.3.
1. {"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid":"e6b734d4-3302-4113-8ec7-5c205c90959a","ip":"46.232.180.248","netmask":"255.255.255.0","gateway":"46.232.180.1","mac":"06:5e:e8:00:00:27","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"}, 2. 3. "instanceName":"r-779-VM","vmType":"DomainRouter","wait":0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses":[{"accountId":2,"publicIp":"46.232.180.248","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":false,"broadcastUri":"untagged","vlanGateway":"46.232.180.1","vlanNetmask":"255.255.255.0","vifMacAddress":"06:5e:e8:00:00:27","networkRate":99999,"trafficType":"Public","networkName":"breth1-500"}],"accessDetails": From the logs VR logs, the ipassoc script got the interface id as null. May 28 12:37:33 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 0 seconds Thanks, Jayapal On 29-May-2014, at 1:08 AM, Andrija Panic <andrija.pa...@gmail.com<mailto:andrija.pa...@gmail.com>> wrote: Thanks Daan, my problem is that I'm in production for 3rd day now, and restoring DB and downgrading back to 4.2.1 doesn't seem as option for me at the moment, since I would loose new acounts and single VMs, etc... Thanks, Andrija On 28 May 2014 21:34, Daan Hoogland <daan.hoogl...@gmail.com<mailto:daan.hoogl...@gmail.com>> wrote: Andrija, nevertheless it sounds familiar. I will be back in the office on monday and ask around. On Wed, May 28, 2014 at 9:23 PM, Andrija Panic <andrija.pa...@gmail.com<mailto:andrija.pa...@gmail.com>> wrote: Hi Daan, I don't think this is my issue, at least I don't make use of private gateway - this is just simple as: create new VPC from scratch - Public IP is not assigned to VR eth1 interface inside VR... I have filed the bug: https://issues.apache.org/jira/browse/CLOUDSTACK-6801 This same thing happened previously to Andrei Mikhailovsky: http://mail-archives.apache.org/mod_mbox/cloudstack-users/201405.mbox/%3C33347835.250.1399336340785.JavaMail.andrei@tuchka%3Eand it is not resolved Thanks, Andrija On 28 May 2014 21:01, Daan Hoogland <daan.hoogl...@gmail.com> wrote: Andrija, this sound like something we seen as well. can you check if this is it : https://issues.apache.org/jira/browse/CLOUDSTACK-6485 thanks, Daan On Wed, May 28, 2014 at 3:30 PM, Andrija Panic <andrija.pa...@gmail.com wrote: Hi there, I'm having big time problems with Public IP missing from VPC VR's eth1, after upgrade to ACS 4.3.1 - did not found this filed as bug so far...and it worked all fine on ACS 4.2.1. No help so far from user mailing list... Below is a detailed explanation, and logs from inside VR, and from management (all fine with management logs...) If anybody can help, I would very much appriciate this, since now I have bunch fo VPC unoperational... Thanks ---------- Forwarded message ---------- From: Andrija Panic <andrija.pa...@gmail.com> Date: 28 May 2014 14:50 Subject: Re: VPC's VR missing public NIC eth1 To: us...@cloudstack.apache.org and as I said eth1 is present: root@r-794-VM:~# cat /proc/net/dev Inter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed eth3: 11484 131 0 0 0 0 0 0 11590 131 0 0 0 0 0 0 lo: 214 2 0 0 0 0 0 0 214 2 0 0 0 0 0 0 eth2: 32970 544 0 0 0 0 0 0 2084 24 0 0 0 0 0 0 eth1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 eth0: 150207 1319 0 0 0 0 0 0 264232 1180 0 0 0 0 0 0 On 28 May 2014 14:47, Andrija Panic <andrija.pa...@gmail.com> wrote: Also, from /var/log/messages/ inside VR: This is a major show stopper - all our VPCs are unusable complete. Anybody... ? May 28 12:37:33 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 0 seconds May 28 12:37:34 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 1 seconds May 28 12:37:35 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 2 seconds May 28 12:37:36 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 3 seconds May 28 12:37:37 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 4 seconds May 28 12:37:38 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 5 seconds May 28 12:37:39 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 6 seconds May 28 12:37:40 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 7 seconds May 28 12:37:41 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 8 seconds May 28 12:37:42 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 9 seconds May 28 12:37:43 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 10 seconds May 28 12:37:44 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 11 seconds May 28 12:37:45 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 12 seconds May 28 12:37:46 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 13 seconds May 28 12:37:47 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 14 seconds May 28 12:37:48 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 15 seconds May 28 12:37:49 r-794-VM cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 16 seconds May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:interface ethnull never appeared May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Adding ip 46.232.180.246 on interface ethnull May 28 12:37:50 r-794-VM cloud: vpc_ipassoc.sh:Add routing 46.232.180.246 on interface ethnull May 28 12:37:50 r-794-VM cloud: vpc_privateGateway.sh:Added SourceNAT 46.232.180.246 on interface ethnull May 28 12:37:50 r-794-VM cloud: vpc_snat.sh:Added SourceNAT 46.232.180.246 on interface eth1 On 28 May 2014 12:59, Andrija Panic <andrija.pa...@gmail.com> wrote: Defined eth1 manually inside /etc/network/interfaces inside VPC's VR. iface eth1 inet static address 46.232.180.246 netmask 255.255.255.0 ifup eth1 ip route add default via 46.232.180.1 so now VR works fine (have access to internet) But again, adding new IP to VR, and enabling static NAT is failing... That is, geting new IP works fine (just associated with account) But enabling static NAT fails, due to "resource unavailable" Here are management logs: 2014-05-28 12:57:00,716 WARN [c.c.n.r.RulesManagerImpl] (catalina-exec-22:ctx-537ac57b ctx-8c44c786) Failed to create static nat rule due to com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Unable to apply static nat rules on router at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3915) at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyStaticNats(VirtualNetworkApplianceManagerImpl.java:3963) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:622) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy240.applyStaticNats(Unknown Source) at com.cloud.network.element.VirtualRouterElement.applyStaticNats(VirtualRouterElement.java:650) at com.cloud.network.IpAddressManagerImpl.applyStaticNats(IpAddressManagerImpl.java:1762) at com.cloud.network.rules.RulesManagerImpl.applyStaticNatForIp(RulesManagerImpl.java:1324) at com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:602) at com.cloud.network.rules.RulesManagerImpl.enableStaticNat(RulesManagerImpl.java:446) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:622) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy88.enableStaticNat(Unknown Source) at org.apache.cloudstack.api.command.user.nat.EnableStaticNatCmd.execute(EnableStaticNatCmd.java:129) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) at com.cloud.api.ApiServer.queueCommand(ApiServer.java:531) at com.cloud.api.ApiServer.handleRequest(ApiServer.java:374) at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:323) at com.cloud.api.ApiServlet.access$000(ApiServlet.java:53) at com.cloud.api.ApiServlet$1.run(ApiServlet.java:115) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:112) at com.cloud.api.ApiServlet.doGet(ApiServlet.java:74) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:701) On 28 May 2014 00:58, Andrija Panic <andrija.pa...@gmail.com> wrote: Hi Jayapal, eth1 seems present (lspci and virsh comfirmed), but not started inside VPC's VR - (VR used for Shared Network is fine)... I could confirm by virsh that is is plugged inside appropriate bridge breth1-500 (check management logs...) management log while createing new VPC (VR) - http://pastebin.com/s77nu5Ei The public IP is there, so command is fine for creating it I guess... VR's /var/log/cloud.log after rebooting VR from CS GUI Tue May 27 22:46:58 UTC 2014 Executing cloud-early-config Tue May 27 22:46:58 UTC 2014 Detected that we are running inside kvm guest Tue May 27 22:46:59 UTC 2014 Patching cloud service Tue May 27 22:47:00 UTC 2014 Updating log4j-cloud.xml Tue May 27 22:47:00 UTC 2014 Setting up VPC virtual router system vm Tue May 27 22:47:00 UTC 2014 checking that eth0 has IP Tue May 27 22:47:00 UTC 2014 Setting up apache web server for VPC Tue May 27 22:47:00 UTC 2014 Enable service dnsmasq = 1 Tue May 27 22:47:00 UTC 2014 Enable service haproxy = 1 Tue May 27 22:47:00 UTC 2014 Processors = 1 Enable service = 0 Tue May 27 22:47:00 UTC 2014 Enable service cloud = 0 Tue May 27 22:47:00 UTC 2014 cloud: disable rp_filter Tue May 27 22:47:00 UTC 2014 disable rpfilter Tue May 27 22:47:00 UTC 2014 cloud: enable_fwding = 1 Tue May 27 22:47:00 UTC 2014 enable_fwding = 1 ifconfig (no eth1 shown) eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:5c inet addr:169.254.3.92 Bcast:169.254.255.255 Mask:255.255.0.0 eth2 Link encap:Ethernet HWaddr 02:00:7d:92:00:10 inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0 eth3 Link encap:Ethernet HWaddr 02:00:78:e9:00:05 inet addr:10.0.3.1 Bcast:10.0.3.255 Mask:255.255.255.0 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 cat /etc/network/interfaces auto lo eth0 iface lo inet loopback iface eth0 inet static address 169.254.3.92 netmask 255.255.0.0 lspci - shows 4 ehternet addapters ethtool eth1 = no link detected virsh - confirmed that eth1 is plugged to correct bridge (breth1-500) as indicated by management logs, and shows good MAC address as shown in managemetn log on pastebin.. This is completely makeing VPCs unusable... :( Cheers On 27 May 2014 16:36, Jayapal Reddy Uradi < jayapalreddy.ur...@citrix.com wrote: Hi, Can you please share management server and router logs in pastebin.comto understand the issue ? Thanks, Jayapal On 27-May-2014, at 6:21 PM, Andrija Panic < andrija.pa...@gmail.com> wrote: Hi, after the upgrade to ACS 4.3 (from 4.2.1) existing VRs for VPC lost their eth1 which is public NIC. VR got eth0(control nic) and eth2 and eth3 (bith belonging to Tiers). From CS GUI, it is reported that the VR has eth1 with Public network attached, but from inside (ssh to VR) there is no eth1 with public IP... Even after destroying those VR, they are recreated again, but without eth1. Anybody experienced same situtation ? Thanks, -- Andrija Panić -------------------------------------- -- -- Daan -- Andrija Panić -------------------------------------- http://admintweets.com -------------------------------------- -- Daan -- Andrija Panić -------------------------------------- http://admintweets.com --------------------------------------