Suggest that you stop and start (not reboot) the router from the Admin GUI.
On 9/16/13 5:26 AM, "Noel Kendall" <noeldkend...@hotmail.com> wrote: >Jayapal, I did a ping test and traced as you suggested. tcpdump >monitoring was done on the public facing interface of the VR. >From within the VR, ping to public IP functions correctly, source address >is the public IP assigned to the VR. >From within the guest, ping to same public IP, does not function, source >address is (as you suspected) the IP of guest on the guest network of VR. >Therefore, it must be: the SNAT rule in iptables in the VR is being >bypassed... that is, the packets are being forwarded without SNAT being >performed on them correctly. >Noel > >> From: jayapalreddy.ur...@citrix.com >> To: users@cloudstack.apache.org >> Subject: Re: Advanced Network - SNAT not working >> Date: Mon, 16 Sep 2013 05:14:53 +0000 >> >> Hi, >> >> I think when the packets are going out the packets are NATed with >>private ip, that can't reach back to router. >> From the VR when you ping public network observe with what source ip >>address the packet is going out and >> From the guest VM when you access public n/w observe on VR with what >>source ip the packet is going out. >> In later case I think the source ip address is different. >> >> Thanks, >> Jayapal >> >> >> On 16-Sep-2013, at 2:30 AM, Noel Kendall <noeldkend...@hotmail.com> >>wrote: >> >> > No other NAT. There is nothing but copper between the KVM host >>machine and the ISP router.There is an L2/L3 switch that the packets >>travel through. However, there is no forwarding in the switch,just >>straight through. I've had a well-functioning V4.0.1 environment running >>on this same configurationin the past. What is new is the conversion to >>4.1 (which was a clean install). >> > It's very mysterious, I have never seen anything like this before. >>There are two other VRs, both having same issue. >> > I will try your suggestion. >> > Noel >> >> Date: Sun, 15 Sep 2013 21:20:41 +0100 >> >> Subject: Re: Advanced Network - SNAT not working >> >> From: msweet....@gmail.com >> >> To: users@cloudstack.apache.org >> >> >> >> This is mostly confusing that the packets are not seen on the VR >>public >> >> interface, seeing as other services are working. >> >> If it was a local NAT issue then the packet would atleast get into >>that >> >> interface. Do you have any upstream devices providing NAT? Or any >>other VR >> >> with the issue? >> >> >> >> It may be worth recreating the VR, by stopping and destroying it and >> >> creating another guest to start a fresh. >> >> >> >> Marty >> >> >> >> >> >> On Sun, Sep 15, 2013 at 8:12 PM, Noel Kendall >><noeldkend...@hotmail.com>wrote: >> >> >> >>> Marty, if I run a telnet <www.xyz.com> 80 from a shell in the guest, >> >>> while running a tcpdumpon the public i/f of the VR: >> >>> - I can see the outbound packets going out- I do not see a response >>packet >> >>> coming back in >> >>> FYI there are no firewalls outbound from the KVM host. The host >>bridges vi >> >>> CS networkingdirectly out on to the internet via a switch. >> >>> Note that traffic from outside (ssh, web) can happily traverse the >>VR to >> >>> the guest. I get the usualits working html page from the guest. >>This tells >> >>> me that there is nothing outbound from the VR thatis filtering >>packets. >> >>> Am truly stumped. This is mysterious indeed. >> >>> From within the VR, can happily telnet to <www.xyz.com> 80 and >>receive >> >>> response.Only if packet came from guest and was forwarded does the >>response >> >>> not show up. >> >>> In short: >> >>> wget from VR to www.xyz.com works, response received and saved >> >>> wget from guest to www.xyz.com does not work, network not available >> >>> displayed on guest, response packets not seen on the public i/f of >>VR at all >> >>> Noel >> >>> >> >>>> Date: Sun, 15 Sep 2013 18:16:17 +0100 >> >>>> Subject: Re: Advanced Network - SNAT not working >> >>>> From: msweet....@gmail.com >> >>>> To: users@cloudstack.apache.org >> >>>> >> >>>> Hi Noel, >> >>>> >> >>>> Can you answer: Does the traffic come back on the public >>interface? and >> >>>> then onto the Guest interface? >> >>>> >> >>>> Thanks, >> >>>> Marty >> >>>> >> >>>> >> >>>> On Sun, Sep 15, 2013 at 2:05 PM, Noel Kendall >><noeldkend...@hotmail.com >> >>>> wrote: >> >>>> >> >>>>> Indeed, yes, a wget executed on the VR to a public website works >>just >> >>> fine. >> >>>>> Noel >> >>>>> >> >>>>>> Date: Sun, 15 Sep 2013 13:15:20 +0100 >> >>>>>> Subject: Re: Advanced Network - SNAT not working >> >>>>>> From: msweet....@gmail.com >> >>>>>> To: users@cloudstack.apache.org >> >>>>>> >> >>>>>> Hi Noel, >> >>>>>> >> >>>>>> Does the traffic come back on the public interface? and then >>onto the >> >>>>> Guest >> >>>>>> interface? >> >>>>>> >> >>>>>> Does a wget on the VR work? >> >>>>>> >> >>>>>> Marty >> >>>>>> >> >>>>>> >> >>>>>> On Sat, Sep 14, 2013 at 8:19 PM, Noel Kendall < >> >>> noeldkend...@hotmail.com >> >>>>>> wrote: >> >>>>>> >> >>>>>>> I have that Marty. I see the http outbound request coming in on >>the >> >>>>> guest >> >>>>>>> interface of the VR,and see the http request being sent out on >>the >> >>>>> public >> >>>>>>> interface of the VR. >> >>>>>>> The traffic is flowing fine from guest to the outbound i/f of >>the >> >>> VR. >> >>>>>>> This is tcpdump on the public i/f while guest is doing wget to >> >>>>>>> 6x.xxx.xxx.xxx >> >>>>>>> >> >>>>>>> 19:17:58.834932 06:e3:3a:00:01:0a > 00:0c:86:4e:fe:00, ethertype >> >>> IPv4 >> >>>>>>> (0x0800), length 74: 10.11.79.178.39074 > 6x.xxx.xxx.xx.80: >>Flags >> >>> [S], >> >>>>> seq >> >>>>>>> 1859313238, win 14600, options [mss 1460,sackOK,TS val 27489348 >>ecr >> >>>>>>> 0,nop,wscale 4], length 0 0x0000: 4500 003c ad1d 4000 3f06 >>2d13 >> >>> 0a0b >> >>>>> 4fb2 >> >>>>>>> 0x0010: 416e c660 98a2 0050 6ed2 de56 0000 0000 >> >>> 0x0020: >> >>>>>>> a002 3908 516c 0000 0204 05b4 0402 080a 0x0030: 01a3 >>7444 >> >>> 0000 >> >>>>>>> 0000 0103 0304 >> >>>>>>> >> >>>>>>> >> >>>>>>>> Date: Sat, 14 Sep 2013 19:29:53 +0100 >> >>>>>>>> Subject: Re: Advanced Network - SNAT not working >> >>>>>>>> From: msweet....@gmail.com >> >>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>> >> >>>>>>>> Hi Noel, >> >>>>>>>> >> >>>>>>>> Can you run a tcpdump on both VR interfaces, this should make >>it >> >>>>> apparent >> >>>>>>>> what is happening? >> >>>>>>>> >> >>>>>>>> Thanks, >> >>>>>>>> Marty >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Sat, Sep 14, 2013 at 6:41 PM, Noel Kendall < >> >>>>> noeldkend...@hotmail.com >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> http://pastebin.com/3FZmFnvZ >> >>>>>>>>> Many thanks Marty. >> >>>>>>>>> Noel >> >>>>>>>>>> Date: Sat, 14 Sep 2013 18:07:55 +0100 >> >>>>>>>>>> Subject: Re: Advanced Network - SNAT not working >> >>>>>>>>>> From: msweet....@gmail.com >> >>>>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>>>> >> >>>>>>>>>> Hi Noel, >> >>>>>>>>>> >> >>>>>>>>>> Could you put the IP tables on pastebin? GMail has collapsed >> >>> the >> >>>>>>> lines >> >>>>>>>>>> horrifically. >> >>>>>>>>>> Have you also tried a tcpdump on both interfaces on the VR? >> >>>>>>>>>> tcpdump -i eth0 <--- Or whatever it may be called >> >>>>>>>>>> >> >>>>>>>>>> I would expect worse connectivity if it was a pure NAT issue, >> >>>>> but I >> >>>>>>> will >> >>>>>>>>>> review the tables later. >> >>>>>>>>>> >> >>>>>>>>>> Thanks, >> >>>>>>>>>> Marty >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Sat, Sep 14, 2013 at 5:55 PM, Noel Kendall < >> >>>>>>> noeldkend...@hotmail.com >> >>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Not seeing return packets on VR. Suspect, therefore, that >> >>> SNAT >> >>>>> is >> >>>>>>>>> fouled >> >>>>>>>>>>> up in some way.I have been doing wget to from guest, can >> >>> see >> >>>>> the >> >>>>>>>>> outgoing >> >>>>>>>>>>> request fine, both in the guest andthe VR. >> >>>>>>>>>>> Could it be that the SNAT table entries from the >> >>>>> 10.11.0.0/16subnet >> >>>>>>>>> to >> >>>>>>>>>>> dpt www are interfering withthe SNAT to public ip?? (wild >> >>>>> guess) - >> >>>>>>> not >> >>>>>>>>> an >> >>>>>>>>>>> iptables expert by any stretch of the imagination >> >>>>>>>>>>> 67.xxx.xxx.56 is the guest public IP10.11.79.178 is the >> >>> guest >> >>>>> IP on >> >>>>>>>>> guest >> >>>>>>>>>>> network >> >>>>>>>>>>> iptables _L -t nat on the VR shows... >> >>>>>>>>>>> Chain PREROUTING (policy ACCEPT)target prot opt source >> >>>>>>>>>>> destination DNAT tcp -- anywhere >> >>>>>>> anywhere >> >>>>>>>>>>> tcp dpt:domain to:10.11.0.1 DNAT tcp -- >> >>> anywhere >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:www to:10.11.79.178:80 DNAT >> >>>>>>> tcp -- >> >>>>>>>>>>> anywhere 67.xxx.xxx.56 tcp dpt:www >> >>>>>>>>> to:10.11.79.178:80DNAT tcp -- anywhere >> >>>>>>> 67.xxx.xxx.56 >> >>>>>>>>> tcp dpt:https >> >>>>>>>>>>> to:10.11.79.178:443 DNAT tcp -- anywhere >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:https to:10.11.79.178:443DNAT >> >>>>>>> tcp >> >>>>>>>>> -- >> >>>>>>>>>>> anywhere 67.xxx.xxx.56 tcp dpt:ssh >> >>>>>>>>> to:10.11.79.178:22DNAT tcp -- anywhere >> >>>>>>> 67.xxx.xxx.56 >> >>>>>>>>> tcp dpt:ssh >> >>>>>>>>>>> to:10.11.79.178:22 DNAT tcp -- anywhere >> >>>>>>>>> 67.xxx.xxx.56 >> >>>>>>>>>>> tcp dpt:ftp to:10.11.79.178:21 DNAT tcp -- >> >>>>> anywhere >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:ftp to:10.11.79.178:21DNAT >> >>>>>>>>> tcp >> >>>>>>>>>>> -- anywhere 67.xxx.xxx.56 tcp >> >>> dpt:5901 to: >> >>>>>>>>>>> 10.11.79.178:5901 DNAT tcp -- anywhere >> >>>>>>>>> 67.xxx.xxx.56 >> >>>>>>>>>>> tcp dpt:5901 to:10.11.79.178:5901 >> >>>>>>>>>>> Chain POSTROUTING (policy ACCEPT)target prot opt source >> >>>>>>>>>>> destination SNAT all -- anywhere >> >>>>>>> anywhere >> >>>>>>>>>>> to:67.xxx.xxx.56 SNAT all -- anywhere >> >>>>>>>>> anywhere >> >>>>>>>>>>> to:67.xxx.xxx.56 SNAT all -- anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all -- >> >>>>> anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all -- >> >>>>> anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56SNAT all -- >> >>>>>>> anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all >> >>> -- >> >>>>>>> anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT >> >>> all -- >> >>>>>>>>> anywhere >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT >> >>> tcp >> >>>>> -- >> >>>>>>>>>>> 10.11.0.0/16 myguest tcp dpt:www >> >>>>> to:10.11.0.1 >> >>>>>>> SNAT >> >>>>>>>>>>> tcp -- 10.11.0.0/16 myguest tcp >> >>>>>>> dpt:https >> >>>>>>>>>>> to:10.11.0.1 SNAT tcp -- 10.11.0.0/16 >> >>> myguest >> >>>>>>>>>>> tcp dpt:ssh to:10.11.0.1 SNAT tcp -- 10.11.0.0/16 >> >>>>>>>>> myguest >> >>>>>>>>>>> tcp dpt:ftp to:10.11.0.1 SNAT tcp -- >> >>>>>>> 10.11.0.0/16 >> >>>>>>>>>>> myguest tcp dpt:5901 to:10.11.0.1 SNAT >> >>>>> all >> >>>>>>> -- >> >>>>>>>>>>> anywhere anywhere to:67.xxx.xxx.56 >> >>>>>>>>>>> Chain OUTPUT (policy ACCEPT)target prot opt source >> >>>>>>>>>>> destination DNAT tcp -- anywhere >> >>>>>>>>> 67.xxx.xxx.56 >> >>>>>>>>>>> tcp dpt:www to:10.11.79.178:80 DNAT tcp -- >> >>>>> anywhere >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:https to:10.11.79.178:443DNAT >> >>>>>>>>> tcp >> >>>>>>>>>>> -- anywhere 67.xxx.xxx.56 tcp dpt:ssh >> >>> to: >> >>>>>>>>>>> 10.11.79.178:22 DNAT tcp -- anywhere >> >>>>>>> 67.xxx.xxx.56 >> >>>>>>>>>>> tcp dpt:ftp to:10.11.79.178:21 DNAT tcp -- >> >>>>> anywhere >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:5901 to:10.11.79.178:5901 >> >>>>>>>>>>> >> >>>>>>>>>>>> Date: Sat, 14 Sep 2013 17:25:14 +0100 >> >>>>>>>>>>>> Subject: Re: Advanced Network - SNAT not working >> >>>>>>>>>>>> From: msweet....@gmail.com >> >>>>>>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>>>>>> >> >>>>>>>>>>>> Hi Noel, >> >>>>>>>>>>>> >> >>>>>>>>>>>> Can you try using telnet to connect to an external >> >>> webserver? >> >>>>>>> telnet >> >>>>>>>>>>>> www.google.com 80 >> >>>>>>>>>>>> Can you also clarify: do you see the response packets >> >>> reach >> >>>>> the >> >>>>>>> VR >> >>>>>>>>> and/or >> >>>>>>>>>>>> on what interfaces? >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks, >> >>>>>>>>>>>> Marty >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Saturday, September 14, 2013, Noel Kendall wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> Guest OS cannot receive responses to http GETs from >> >>>>> resources >> >>>>>>> on >> >>>>>>>>> the >> >>>>>>>>>>>>> Internet. >> >>>>>>>>>>>>> Network is advanced, VLAN isolated. >> >>>>>>>>>>>>> What is working: >> >>>>>>>>>>>>> - can browse guest website from internet- can ssh to >> >>> guest >> >>>>> from >> >>>>>>>>>>> internet- >> >>>>>>>>>>>>> can VPN to guest network from internet >> >>>>>>>>>>>>> - network VR can access internet sites no problem >> >>>>>>>>>>>>> What is not working: >> >>>>>>>>>>>>> - guest http traffic to external website gets to VR on >> >>>>> internal >> >>>>>>>>> NIC, >> >>>>>>>>>>>>> packets forwarded to external site via external NIC >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Response traffic is not seen. Appears to be dropped. >> >>>>>>>>>>>>> Have been looking hard at IPTABLES rules, doing >> >>> tcpdumps, >> >>>>> etc. >> >>>>>>>>>>>>> Am at this point stumped. >> >>>>>>>>>>>>> Any ideas on what could be wrong, or how to determine >> >>> what >> >>>>>>> could be >> >>>>>>>>>>> wrong? >> >>>>>>>>>>>>> Thanks in advance everyone who tries to help! >> >>>>>>>>>>>>> N. >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>> >> >>>>> >> >>> >> >>> >> > >> >
