My theory was that by logging into the VR and running commands, you might have inadvertently changed something. What is the output of iptables-save after the restart?
On 9/16/13 12:04 PM, "Noel Kendall" <noeldkend...@hotmail.com> wrote: >That has worked Chiradeep. What could have caused this problem? Is it >somethingthat should be fixed? >Thanks for the simple and rather effective suggestion! >Noel > >> From: chiradeep.vit...@citrix.com >> To: users@cloudstack.apache.org >> Subject: Re: Advanced Network - SNAT not working >> Date: Mon, 16 Sep 2013 18:00:29 +0000 >> >> Suggest that you stop and start (not reboot) the router from the Admin >>GUI. >> >> On 9/16/13 5:26 AM, "Noel Kendall" <noeldkend...@hotmail.com> wrote: >> >> >Jayapal, I did a ping test and traced as you suggested. tcpdump >> >monitoring was done on the public facing interface of the VR. >> >From within the VR, ping to public IP functions correctly, source >>address >> >is the public IP assigned to the VR. >> >From within the guest, ping to same public IP, does not function, >>source >> >address is (as you suspected) the IP of guest on the guest network of >>VR. >> >Therefore, it must be: the SNAT rule in iptables in the VR is being >> >bypassed... that is, the packets are being forwarded without SNAT being >> >performed on them correctly. >> >Noel >> > >> >> From: jayapalreddy.ur...@citrix.com >> >> To: users@cloudstack.apache.org >> >> Subject: Re: Advanced Network - SNAT not working >> >> Date: Mon, 16 Sep 2013 05:14:53 +0000 >> >> >> >> Hi, >> >> >> >> I think when the packets are going out the packets are NATed with >> >>private ip, that can't reach back to router. >> >> From the VR when you ping public network observe with what source ip >> >>address the packet is going out and >> >> From the guest VM when you access public n/w observe on VR with what >> >>source ip the packet is going out. >> >> In later case I think the source ip address is different. >> >> >> >> Thanks, >> >> Jayapal >> >> >> >> >> >> On 16-Sep-2013, at 2:30 AM, Noel Kendall <noeldkend...@hotmail.com> >> >>wrote: >> >> >> >> > No other NAT. There is nothing but copper between the KVM host >> >>machine and the ISP router.There is an L2/L3 switch that the packets >> >>travel through. However, there is no forwarding in the switch,just >> >>straight through. I've had a well-functioning V4.0.1 environment >>running >> >>on this same configurationin the past. What is new is the conversion >>to >> >>4.1 (which was a clean install). >> >> > It's very mysterious, I have never seen anything like this before. >> >>There are two other VRs, both having same issue. >> >> > I will try your suggestion. >> >> > Noel >> >> >> Date: Sun, 15 Sep 2013 21:20:41 +0100 >> >> >> Subject: Re: Advanced Network - SNAT not working >> >> >> From: msweet....@gmail.com >> >> >> To: users@cloudstack.apache.org >> >> >> >> >> >> This is mostly confusing that the packets are not seen on the VR >> >>public >> >> >> interface, seeing as other services are working. >> >> >> If it was a local NAT issue then the packet would atleast get into >> >>that >> >> >> interface. Do you have any upstream devices providing NAT? Or any >> >>other VR >> >> >> with the issue? >> >> >> >> >> >> It may be worth recreating the VR, by stopping and destroying it >>and >> >> >> creating another guest to start a fresh. >> >> >> >> >> >> Marty >> >> >> >> >> >> >> >> >> On Sun, Sep 15, 2013 at 8:12 PM, Noel Kendall >> >><noeldkend...@hotmail.com>wrote: >> >> >> >> >> >>> Marty, if I run a telnet <www.xyz.com> 80 from a shell in the >>guest, >> >> >>> while running a tcpdumpon the public i/f of the VR: >> >> >>> - I can see the outbound packets going out- I do not see a >>response >> >>packet >> >> >>> coming back in >> >> >>> FYI there are no firewalls outbound from the KVM host. The host >> >>bridges vi >> >> >>> CS networkingdirectly out on to the internet via a switch. >> >> >>> Note that traffic from outside (ssh, web) can happily traverse >>the >> >>VR to >> >> >>> the guest. I get the usualits working html page from the guest. >> >>This tells >> >> >>> me that there is nothing outbound from the VR thatis filtering >> >>packets. >> >> >>> Am truly stumped. This is mysterious indeed. >> >> >>> From within the VR, can happily telnet to <www.xyz.com> 80 and >> >>receive >> >> >>> response.Only if packet came from guest and was forwarded does >>the >> >>response >> >> >>> not show up. >> >> >>> In short: >> >> >>> wget from VR to www.xyz.com works, response received and saved >> >> >>> wget from guest to www.xyz.com does not work, network not >>available >> >> >>> displayed on guest, response packets not seen on the public i/f >>of >> >>VR at all >> >> >>> Noel >> >> >>> >> >> >>>> Date: Sun, 15 Sep 2013 18:16:17 +0100 >> >> >>>> Subject: Re: Advanced Network - SNAT not working >> >> >>>> From: msweet....@gmail.com >> >> >>>> To: users@cloudstack.apache.org >> >> >>>> >> >> >>>> Hi Noel, >> >> >>>> >> >> >>>> Can you answer: Does the traffic come back on the public >> >>interface? and >> >> >>>> then onto the Guest interface? >> >> >>>> >> >> >>>> Thanks, >> >> >>>> Marty >> >> >>>> >> >> >>>> >> >> >>>> On Sun, Sep 15, 2013 at 2:05 PM, Noel Kendall >> >><noeldkend...@hotmail.com >> >> >>>> wrote: >> >> >>>> >> >> >>>>> Indeed, yes, a wget executed on the VR to a public website >>works >> >>just >> >> >>> fine. >> >> >>>>> Noel >> >> >>>>> >> >> >>>>>> Date: Sun, 15 Sep 2013 13:15:20 +0100 >> >> >>>>>> Subject: Re: Advanced Network - SNAT not working >> >> >>>>>> From: msweet....@gmail.com >> >> >>>>>> To: users@cloudstack.apache.org >> >> >>>>>> >> >> >>>>>> Hi Noel, >> >> >>>>>> >> >> >>>>>> Does the traffic come back on the public interface? and then >> >>onto the >> >> >>>>> Guest >> >> >>>>>> interface? >> >> >>>>>> >> >> >>>>>> Does a wget on the VR work? >> >> >>>>>> >> >> >>>>>> Marty >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> On Sat, Sep 14, 2013 at 8:19 PM, Noel Kendall < >> >> >>> noeldkend...@hotmail.com >> >> >>>>>> wrote: >> >> >>>>>> >> >> >>>>>>> I have that Marty. I see the http outbound request coming in >>on >> >>the >> >> >>>>> guest >> >> >>>>>>> interface of the VR,and see the http request being sent out >>on >> >>the >> >> >>>>> public >> >> >>>>>>> interface of the VR. >> >> >>>>>>> The traffic is flowing fine from guest to the outbound i/f of >> >>the >> >> >>> VR. >> >> >>>>>>> This is tcpdump on the public i/f while guest is doing wget >>to >> >> >>>>>>> 6x.xxx.xxx.xxx >> >> >>>>>>> >> >> >>>>>>> 19:17:58.834932 06:e3:3a:00:01:0a > 00:0c:86:4e:fe:00, >>ethertype >> >> >>> IPv4 >> >> >>>>>>> (0x0800), length 74: 10.11.79.178.39074 > 6x.xxx.xxx.xx.80: >> >>Flags >> >> >>> [S], >> >> >>>>> seq >> >> >>>>>>> 1859313238, win 14600, options [mss 1460,sackOK,TS val >>27489348 >> >>ecr >> >> >>>>>>> 0,nop,wscale 4], length 0 0x0000: 4500 003c ad1d 4000 3f06 >> >>2d13 >> >> >>> 0a0b >> >> >>>>> 4fb2 >> >> >>>>>>> 0x0010: 416e c660 98a2 0050 6ed2 de56 0000 0000 >> >> >>> 0x0020: >> >> >>>>>>> a002 3908 516c 0000 0204 05b4 0402 080a 0x0030: 01a3 >> >>7444 >> >> >>> 0000 >> >> >>>>>>> 0000 0103 0304 >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>>> Date: Sat, 14 Sep 2013 19:29:53 +0100 >> >> >>>>>>>> Subject: Re: Advanced Network - SNAT not working >> >> >>>>>>>> From: msweet....@gmail.com >> >> >>>>>>>> To: users@cloudstack.apache.org >> >> >>>>>>>> >> >> >>>>>>>> Hi Noel, >> >> >>>>>>>> >> >> >>>>>>>> Can you run a tcpdump on both VR interfaces, this should >>make >> >>it >> >> >>>>> apparent >> >> >>>>>>>> what is happening? >> >> >>>>>>>> >> >> >>>>>>>> Thanks, >> >> >>>>>>>> Marty >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> On Sat, Sep 14, 2013 at 6:41 PM, Noel Kendall < >> >> >>>>> noeldkend...@hotmail.com >> >> >>>>>>>> wrote: >> >> >>>>>>>> >> >> >>>>>>>>> http://pastebin.com/3FZmFnvZ >> >> >>>>>>>>> Many thanks Marty. >> >> >>>>>>>>> Noel >> >> >>>>>>>>>> Date: Sat, 14 Sep 2013 18:07:55 +0100 >> >> >>>>>>>>>> Subject: Re: Advanced Network - SNAT not working >> >> >>>>>>>>>> From: msweet....@gmail.com >> >> >>>>>>>>>> To: users@cloudstack.apache.org >> >> >>>>>>>>>> >> >> >>>>>>>>>> Hi Noel, >> >> >>>>>>>>>> >> >> >>>>>>>>>> Could you put the IP tables on pastebin? GMail has >>collapsed >> >> >>> the >> >> >>>>>>> lines >> >> >>>>>>>>>> horrifically. >> >> >>>>>>>>>> Have you also tried a tcpdump on both interfaces on the >>VR? >> >> >>>>>>>>>> tcpdump -i eth0 <--- Or whatever it may be called >> >> >>>>>>>>>> >> >> >>>>>>>>>> I would expect worse connectivity if it was a pure NAT >>issue, >> >> >>>>> but I >> >> >>>>>>> will >> >> >>>>>>>>>> review the tables later. >> >> >>>>>>>>>> >> >> >>>>>>>>>> Thanks, >> >> >>>>>>>>>> Marty >> >> >>>>>>>>>> >> >> >>>>>>>>>> >> >> >>>>>>>>>> On Sat, Sep 14, 2013 at 5:55 PM, Noel Kendall < >> >> >>>>>>> noeldkend...@hotmail.com >> >> >>>>>>>>>> wrote: >> >> >>>>>>>>>> >> >> >>>>>>>>>>> Not seeing return packets on VR. Suspect, therefore, that >> >> >>> SNAT >> >> >>>>> is >> >> >>>>>>>>> fouled >> >> >>>>>>>>>>> up in some way.I have been doing wget to from guest, can >> >> >>> see >> >> >>>>> the >> >> >>>>>>>>> outgoing >> >> >>>>>>>>>>> request fine, both in the guest andthe VR. >> >> >>>>>>>>>>> Could it be that the SNAT table entries from the >> >> >>>>> 10.11.0.0/16subnet >> >> >>>>>>>>> to >> >> >>>>>>>>>>> dpt www are interfering withthe SNAT to public ip?? (wild >> >> >>>>> guess) - >> >> >>>>>>> not >> >> >>>>>>>>> an >> >> >>>>>>>>>>> iptables expert by any stretch of the imagination >> >> >>>>>>>>>>> 67.xxx.xxx.56 is the guest public IP10.11.79.178 is the >> >> >>> guest >> >> >>>>> IP on >> >> >>>>>>>>> guest >> >> >>>>>>>>>>> network >> >> >>>>>>>>>>> iptables _L -t nat on the VR shows... >> >> >>>>>>>>>>> Chain PREROUTING (policy ACCEPT)target prot opt >>source >> >> >>>>>>>>>>> destination DNAT tcp -- anywhere >> >> >>>>>>> anywhere >> >> >>>>>>>>>>> tcp dpt:domain to:10.11.0.1 DNAT tcp -- >> >> >>> anywhere >> >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:www to:10.11.79.178:80 DNAT >> >> >>>>>>> tcp -- >> >> >>>>>>>>>>> anywhere 67.xxx.xxx.56 tcp dpt:www >> >> >>>>>>>>> to:10.11.79.178:80DNAT tcp -- anywhere >> >> >>>>>>> 67.xxx.xxx.56 >> >> >>>>>>>>> tcp dpt:https >> >> >>>>>>>>>>> to:10.11.79.178:443 DNAT tcp -- anywhere >> >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:https >>to:10.11.79.178:443DNAT >> >> >>>>>>> tcp >> >> >>>>>>>>> -- >> >> >>>>>>>>>>> anywhere 67.xxx.xxx.56 tcp dpt:ssh >> >> >>>>>>>>> to:10.11.79.178:22DNAT tcp -- anywhere >> >> >>>>>>> 67.xxx.xxx.56 >> >> >>>>>>>>> tcp dpt:ssh >> >> >>>>>>>>>>> to:10.11.79.178:22 DNAT tcp -- anywhere >> >> >>>>>>>>> 67.xxx.xxx.56 >> >> >>>>>>>>>>> tcp dpt:ftp to:10.11.79.178:21 DNAT tcp -- >> >> >>>>> anywhere >> >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:ftp >>to:10.11.79.178:21DNAT >> >> >>>>>>>>> tcp >> >> >>>>>>>>>>> -- anywhere 67.xxx.xxx.56 tcp >> >> >>> dpt:5901 to: >> >> >>>>>>>>>>> 10.11.79.178:5901 DNAT tcp -- anywhere >> >> >>>>>>>>> 67.xxx.xxx.56 >> >> >>>>>>>>>>> tcp dpt:5901 to:10.11.79.178:5901 >> >> >>>>>>>>>>> Chain POSTROUTING (policy ACCEPT)target prot opt >>source >> >> >>>>>>>>>>> destination SNAT all -- anywhere >> >> >>>>>>> anywhere >> >> >>>>>>>>>>> to:67.xxx.xxx.56 SNAT all -- anywhere >> >> >>>>>>>>> anywhere >> >> >>>>>>>>>>> to:67.xxx.xxx.56 SNAT all -- anywhere >> >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all -- >> >> >>>>> anywhere >> >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all -- >> >> >>>>> anywhere >> >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56SNAT all -- >> >> >>>>>>> anywhere >> >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all >> >> >>> -- >> >> >>>>>>> anywhere >> >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT >> >> >>> all -- >> >> >>>>>>>>> anywhere >> >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT >> >> >>> tcp >> >> >>>>> -- >> >> >>>>>>>>>>> 10.11.0.0/16 myguest tcp dpt:www >> >> >>>>> to:10.11.0.1 >> >> >>>>>>> SNAT >> >> >>>>>>>>>>> tcp -- 10.11.0.0/16 myguest tcp >> >> >>>>>>> dpt:https >> >> >>>>>>>>>>> to:10.11.0.1 SNAT tcp -- 10.11.0.0/16 >> >> >>> myguest >> >> >>>>>>>>>>> tcp dpt:ssh to:10.11.0.1 SNAT tcp -- 10.11.0.0/16 >> >> >>>>>>>>> myguest >> >> >>>>>>>>>>> tcp dpt:ftp to:10.11.0.1 SNAT tcp -- >> >> >>>>>>> 10.11.0.0/16 >> >> >>>>>>>>>>> myguest tcp dpt:5901 to:10.11.0.1 SNAT >> >> >>>>> all >> >> >>>>>>> -- >> >> >>>>>>>>>>> anywhere anywhere to:67.xxx.xxx.56 >> >> >>>>>>>>>>> Chain OUTPUT (policy ACCEPT)target prot opt source >> >> >>>>>>>>>>> destination DNAT tcp -- anywhere >> >> >>>>>>>>> 67.xxx.xxx.56 >> >> >>>>>>>>>>> tcp dpt:www to:10.11.79.178:80 DNAT tcp -- >> >> >>>>> anywhere >> >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:https >>to:10.11.79.178:443DNAT >> >> >>>>>>>>> tcp >> >> >>>>>>>>>>> -- anywhere 67.xxx.xxx.56 tcp dpt:ssh >> >> >>> to: >> >> >>>>>>>>>>> 10.11.79.178:22 DNAT tcp -- anywhere >> >> >>>>>>> 67.xxx.xxx.56 >> >> >>>>>>>>>>> tcp dpt:ftp to:10.11.79.178:21 DNAT tcp -- >> >> >>>>> anywhere >> >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:5901 to:10.11.79.178:5901 >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> Date: Sat, 14 Sep 2013 17:25:14 +0100 >> >> >>>>>>>>>>>> Subject: Re: Advanced Network - SNAT not working >> >> >>>>>>>>>>>> From: msweet....@gmail.com >> >> >>>>>>>>>>>> To: users@cloudstack.apache.org >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> Hi Noel, >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> Can you try using telnet to connect to an external >> >> >>> webserver? >> >> >>>>>>> telnet >> >> >>>>>>>>>>>> www.google.com 80 >> >> >>>>>>>>>>>> Can you also clarify: do you see the response packets >> >> >>> reach >> >> >>>>> the >> >> >>>>>>> VR >> >> >>>>>>>>> and/or >> >> >>>>>>>>>>>> on what interfaces? >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> Thanks, >> >> >>>>>>>>>>>> Marty >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> On Saturday, September 14, 2013, Noel Kendall wrote: >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>>> Guest OS cannot receive responses to http GETs from >> >> >>>>> resources >> >> >>>>>>> on >> >> >>>>>>>>> the >> >> >>>>>>>>>>>>> Internet. >> >> >>>>>>>>>>>>> Network is advanced, VLAN isolated. >> >> >>>>>>>>>>>>> What is working: >> >> >>>>>>>>>>>>> - can browse guest website from internet- can ssh to >> >> >>> guest >> >> >>>>> from >> >> >>>>>>>>>>> internet- >> >> >>>>>>>>>>>>> can VPN to guest network from internet >> >> >>>>>>>>>>>>> - network VR can access internet sites no problem >> >> >>>>>>>>>>>>> What is not working: >> >> >>>>>>>>>>>>> - guest http traffic to external website gets to VR on >> >> >>>>> internal >> >> >>>>>>>>> NIC, >> >> >>>>>>>>>>>>> packets forwarded to external site via external NIC >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> Response traffic is not seen. Appears to be dropped. >> >> >>>>>>>>>>>>> Have been looking hard at IPTABLES rules, doing >> >> >>> tcpdumps, >> >> >>>>> etc. >> >> >>>>>>>>>>>>> Am at this point stumped. >> >> >>>>>>>>>>>>> Any ideas on what could be wrong, or how to determine >> >> >>> what >> >> >>>>>>> could be >> >> >>>>>>>>>>> wrong? >> >> >>>>>>>>>>>>> Thanks in advance everyone who tries to help! >> >> >>>>>>>>>>>>> N. >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>> >> >> >>>>> >> >> >>> >> >> >>> >> >> > >> >> >> > >> >
