Reply, sorry, spoke too soon. Stop and start did not resolve the issue. Noel
> From: chiradeep.vit...@citrix.com > To: users@cloudstack.apache.org > Subject: Re: Advanced Network - SNAT not working > Date: Mon, 16 Sep 2013 18:00:29 +0000 > > Suggest that you stop and start (not reboot) the router from the Admin GUI. > > On 9/16/13 5:26 AM, "Noel Kendall" <noeldkend...@hotmail.com> wrote: > > >Jayapal, I did a ping test and traced as you suggested. tcpdump > >monitoring was done on the public facing interface of the VR. > >From within the VR, ping to public IP functions correctly, source address > >is the public IP assigned to the VR. > >From within the guest, ping to same public IP, does not function, source > >address is (as you suspected) the IP of guest on the guest network of VR. > >Therefore, it must be: the SNAT rule in iptables in the VR is being > >bypassed... that is, the packets are being forwarded without SNAT being > >performed on them correctly. > >Noel > > > >> From: jayapalreddy.ur...@citrix.com > >> To: users@cloudstack.apache.org > >> Subject: Re: Advanced Network - SNAT not working > >> Date: Mon, 16 Sep 2013 05:14:53 +0000 > >> > >> Hi, > >> > >> I think when the packets are going out the packets are NATed with > >>private ip, that can't reach back to router. > >> From the VR when you ping public network observe with what source ip > >>address the packet is going out and > >> From the guest VM when you access public n/w observe on VR with what > >>source ip the packet is going out. > >> In later case I think the source ip address is different. > >> > >> Thanks, > >> Jayapal > >> > >> > >> On 16-Sep-2013, at 2:30 AM, Noel Kendall <noeldkend...@hotmail.com> > >>wrote: > >> > >> > No other NAT. There is nothing but copper between the KVM host > >>machine and the ISP router.There is an L2/L3 switch that the packets > >>travel through. However, there is no forwarding in the switch,just > >>straight through. I've had a well-functioning V4.0.1 environment running > >>on this same configurationin the past. What is new is the conversion to > >>4.1 (which was a clean install). > >> > It's very mysterious, I have never seen anything like this before. > >>There are two other VRs, both having same issue. > >> > I will try your suggestion. > >> > Noel > >> >> Date: Sun, 15 Sep 2013 21:20:41 +0100 > >> >> Subject: Re: Advanced Network - SNAT not working > >> >> From: msweet....@gmail.com > >> >> To: users@cloudstack.apache.org > >> >> > >> >> This is mostly confusing that the packets are not seen on the VR > >>public > >> >> interface, seeing as other services are working. > >> >> If it was a local NAT issue then the packet would atleast get into > >>that > >> >> interface. Do you have any upstream devices providing NAT? Or any > >>other VR > >> >> with the issue? > >> >> > >> >> It may be worth recreating the VR, by stopping and destroying it and > >> >> creating another guest to start a fresh. > >> >> > >> >> Marty > >> >> > >> >> > >> >> On Sun, Sep 15, 2013 at 8:12 PM, Noel Kendall > >><noeldkend...@hotmail.com>wrote: > >> >> > >> >>> Marty, if I run a telnet <www.xyz.com> 80 from a shell in the guest, > >> >>> while running a tcpdumpon the public i/f of the VR: > >> >>> - I can see the outbound packets going out- I do not see a response > >>packet > >> >>> coming back in > >> >>> FYI there are no firewalls outbound from the KVM host. The host > >>bridges vi > >> >>> CS networkingdirectly out on to the internet via a switch. > >> >>> Note that traffic from outside (ssh, web) can happily traverse the > >>VR to > >> >>> the guest. I get the usualits working html page from the guest. > >>This tells > >> >>> me that there is nothing outbound from the VR thatis filtering > >>packets. > >> >>> Am truly stumped. This is mysterious indeed. > >> >>> From within the VR, can happily telnet to <www.xyz.com> 80 and > >>receive > >> >>> response.Only if packet came from guest and was forwarded does the > >>response > >> >>> not show up. > >> >>> In short: > >> >>> wget from VR to www.xyz.com works, response received and saved > >> >>> wget from guest to www.xyz.com does not work, network not available > >> >>> displayed on guest, response packets not seen on the public i/f of > >>VR at all > >> >>> Noel > >> >>> > >> >>>> Date: Sun, 15 Sep 2013 18:16:17 +0100 > >> >>>> Subject: Re: Advanced Network - SNAT not working > >> >>>> From: msweet....@gmail.com > >> >>>> To: users@cloudstack.apache.org > >> >>>> > >> >>>> Hi Noel, > >> >>>> > >> >>>> Can you answer: Does the traffic come back on the public > >>interface? and > >> >>>> then onto the Guest interface? > >> >>>> > >> >>>> Thanks, > >> >>>> Marty > >> >>>> > >> >>>> > >> >>>> On Sun, Sep 15, 2013 at 2:05 PM, Noel Kendall > >><noeldkend...@hotmail.com > >> >>>> wrote: > >> >>>> > >> >>>>> Indeed, yes, a wget executed on the VR to a public website works > >>just > >> >>> fine. > >> >>>>> Noel > >> >>>>> > >> >>>>>> Date: Sun, 15 Sep 2013 13:15:20 +0100 > >> >>>>>> Subject: Re: Advanced Network - SNAT not working > >> >>>>>> From: msweet....@gmail.com > >> >>>>>> To: users@cloudstack.apache.org > >> >>>>>> > >> >>>>>> Hi Noel, > >> >>>>>> > >> >>>>>> Does the traffic come back on the public interface? and then > >>onto the > >> >>>>> Guest > >> >>>>>> interface? > >> >>>>>> > >> >>>>>> Does a wget on the VR work? > >> >>>>>> > >> >>>>>> Marty > >> >>>>>> > >> >>>>>> > >> >>>>>> On Sat, Sep 14, 2013 at 8:19 PM, Noel Kendall < > >> >>> noeldkend...@hotmail.com > >> >>>>>> wrote: > >> >>>>>> > >> >>>>>>> I have that Marty. I see the http outbound request coming in on > >>the > >> >>>>> guest > >> >>>>>>> interface of the VR,and see the http request being sent out on > >>the > >> >>>>> public > >> >>>>>>> interface of the VR. > >> >>>>>>> The traffic is flowing fine from guest to the outbound i/f of > >>the > >> >>> VR. > >> >>>>>>> This is tcpdump on the public i/f while guest is doing wget to > >> >>>>>>> 6x.xxx.xxx.xxx > >> >>>>>>> > >> >>>>>>> 19:17:58.834932 06:e3:3a:00:01:0a > 00:0c:86:4e:fe:00, ethertype > >> >>> IPv4 > >> >>>>>>> (0x0800), length 74: 10.11.79.178.39074 > 6x.xxx.xxx.xx.80: > >>Flags > >> >>> [S], > >> >>>>> seq > >> >>>>>>> 1859313238, win 14600, options [mss 1460,sackOK,TS val 27489348 > >>ecr > >> >>>>>>> 0,nop,wscale 4], length 0 0x0000: 4500 003c ad1d 4000 3f06 > >>2d13 > >> >>> 0a0b > >> >>>>> 4fb2 > >> >>>>>>> 0x0010: 416e c660 98a2 0050 6ed2 de56 0000 0000 > >> >>> 0x0020: > >> >>>>>>> a002 3908 516c 0000 0204 05b4 0402 080a 0x0030: 01a3 > >>7444 > >> >>> 0000 > >> >>>>>>> 0000 0103 0304 > >> >>>>>>> > >> >>>>>>> > >> >>>>>>>> Date: Sat, 14 Sep 2013 19:29:53 +0100 > >> >>>>>>>> Subject: Re: Advanced Network - SNAT not working > >> >>>>>>>> From: msweet....@gmail.com > >> >>>>>>>> To: users@cloudstack.apache.org > >> >>>>>>>> > >> >>>>>>>> Hi Noel, > >> >>>>>>>> > >> >>>>>>>> Can you run a tcpdump on both VR interfaces, this should make > >>it > >> >>>>> apparent > >> >>>>>>>> what is happening? > >> >>>>>>>> > >> >>>>>>>> Thanks, > >> >>>>>>>> Marty > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> On Sat, Sep 14, 2013 at 6:41 PM, Noel Kendall < > >> >>>>> noeldkend...@hotmail.com > >> >>>>>>>> wrote: > >> >>>>>>>> > >> >>>>>>>>> http://pastebin.com/3FZmFnvZ > >> >>>>>>>>> Many thanks Marty. > >> >>>>>>>>> Noel > >> >>>>>>>>>> Date: Sat, 14 Sep 2013 18:07:55 +0100 > >> >>>>>>>>>> Subject: Re: Advanced Network - SNAT not working > >> >>>>>>>>>> From: msweet....@gmail.com > >> >>>>>>>>>> To: users@cloudstack.apache.org > >> >>>>>>>>>> > >> >>>>>>>>>> Hi Noel, > >> >>>>>>>>>> > >> >>>>>>>>>> Could you put the IP tables on pastebin? GMail has collapsed > >> >>> the > >> >>>>>>> lines > >> >>>>>>>>>> horrifically. > >> >>>>>>>>>> Have you also tried a tcpdump on both interfaces on the VR? > >> >>>>>>>>>> tcpdump -i eth0 <--- Or whatever it may be called > >> >>>>>>>>>> > >> >>>>>>>>>> I would expect worse connectivity if it was a pure NAT issue, > >> >>>>> but I > >> >>>>>>> will > >> >>>>>>>>>> review the tables later. > >> >>>>>>>>>> > >> >>>>>>>>>> Thanks, > >> >>>>>>>>>> Marty > >> >>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> On Sat, Sep 14, 2013 at 5:55 PM, Noel Kendall < > >> >>>>>>> noeldkend...@hotmail.com > >> >>>>>>>>>> wrote: > >> >>>>>>>>>> > >> >>>>>>>>>>> Not seeing return packets on VR. Suspect, therefore, that > >> >>> SNAT > >> >>>>> is > >> >>>>>>>>> fouled > >> >>>>>>>>>>> up in some way.I have been doing wget to from guest, can > >> >>> see > >> >>>>> the > >> >>>>>>>>> outgoing > >> >>>>>>>>>>> request fine, both in the guest andthe VR. > >> >>>>>>>>>>> Could it be that the SNAT table entries from the > >> >>>>> 10.11.0.0/16subnet > >> >>>>>>>>> to > >> >>>>>>>>>>> dpt www are interfering withthe SNAT to public ip?? (wild > >> >>>>> guess) - > >> >>>>>>> not > >> >>>>>>>>> an > >> >>>>>>>>>>> iptables expert by any stretch of the imagination > >> >>>>>>>>>>> 67.xxx.xxx.56 is the guest public IP10.11.79.178 is the > >> >>> guest > >> >>>>> IP on > >> >>>>>>>>> guest > >> >>>>>>>>>>> network > >> >>>>>>>>>>> iptables _L -t nat on the VR shows... > >> >>>>>>>>>>> Chain PREROUTING (policy ACCEPT)target prot opt source > >> >>>>>>>>>>> destination DNAT tcp -- anywhere > >> >>>>>>> anywhere > >> >>>>>>>>>>> tcp dpt:domain to:10.11.0.1 DNAT tcp -- > >> >>> anywhere > >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:www to:10.11.79.178:80 DNAT > >> >>>>>>> tcp -- > >> >>>>>>>>>>> anywhere 67.xxx.xxx.56 tcp dpt:www > >> >>>>>>>>> to:10.11.79.178:80DNAT tcp -- anywhere > >> >>>>>>> 67.xxx.xxx.56 > >> >>>>>>>>> tcp dpt:https > >> >>>>>>>>>>> to:10.11.79.178:443 DNAT tcp -- anywhere > >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:https to:10.11.79.178:443DNAT > >> >>>>>>> tcp > >> >>>>>>>>> -- > >> >>>>>>>>>>> anywhere 67.xxx.xxx.56 tcp dpt:ssh > >> >>>>>>>>> to:10.11.79.178:22DNAT tcp -- anywhere > >> >>>>>>> 67.xxx.xxx.56 > >> >>>>>>>>> tcp dpt:ssh > >> >>>>>>>>>>> to:10.11.79.178:22 DNAT tcp -- anywhere > >> >>>>>>>>> 67.xxx.xxx.56 > >> >>>>>>>>>>> tcp dpt:ftp to:10.11.79.178:21 DNAT tcp -- > >> >>>>> anywhere > >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:ftp to:10.11.79.178:21DNAT > >> >>>>>>>>> tcp > >> >>>>>>>>>>> -- anywhere 67.xxx.xxx.56 tcp > >> >>> dpt:5901 to: > >> >>>>>>>>>>> 10.11.79.178:5901 DNAT tcp -- anywhere > >> >>>>>>>>> 67.xxx.xxx.56 > >> >>>>>>>>>>> tcp dpt:5901 to:10.11.79.178:5901 > >> >>>>>>>>>>> Chain POSTROUTING (policy ACCEPT)target prot opt source > >> >>>>>>>>>>> destination SNAT all -- anywhere > >> >>>>>>> anywhere > >> >>>>>>>>>>> to:67.xxx.xxx.56 SNAT all -- anywhere > >> >>>>>>>>> anywhere > >> >>>>>>>>>>> to:67.xxx.xxx.56 SNAT all -- anywhere > >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all -- > >> >>>>> anywhere > >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all -- > >> >>>>> anywhere > >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56SNAT all -- > >> >>>>>>> anywhere > >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT all > >> >>> -- > >> >>>>>>> anywhere > >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT > >> >>> all -- > >> >>>>>>>>> anywhere > >> >>>>>>>>>>> anywhere to:67.xxx.xxx.56 SNAT > >> >>> tcp > >> >>>>> -- > >> >>>>>>>>>>> 10.11.0.0/16 myguest tcp dpt:www > >> >>>>> to:10.11.0.1 > >> >>>>>>> SNAT > >> >>>>>>>>>>> tcp -- 10.11.0.0/16 myguest tcp > >> >>>>>>> dpt:https > >> >>>>>>>>>>> to:10.11.0.1 SNAT tcp -- 10.11.0.0/16 > >> >>> myguest > >> >>>>>>>>>>> tcp dpt:ssh to:10.11.0.1 SNAT tcp -- 10.11.0.0/16 > >> >>>>>>>>> myguest > >> >>>>>>>>>>> tcp dpt:ftp to:10.11.0.1 SNAT tcp -- > >> >>>>>>> 10.11.0.0/16 > >> >>>>>>>>>>> myguest tcp dpt:5901 to:10.11.0.1 SNAT > >> >>>>> all > >> >>>>>>> -- > >> >>>>>>>>>>> anywhere anywhere to:67.xxx.xxx.56 > >> >>>>>>>>>>> Chain OUTPUT (policy ACCEPT)target prot opt source > >> >>>>>>>>>>> destination DNAT tcp -- anywhere > >> >>>>>>>>> 67.xxx.xxx.56 > >> >>>>>>>>>>> tcp dpt:www to:10.11.79.178:80 DNAT tcp -- > >> >>>>> anywhere > >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:https to:10.11.79.178:443DNAT > >> >>>>>>>>> tcp > >> >>>>>>>>>>> -- anywhere 67.xxx.xxx.56 tcp dpt:ssh > >> >>> to: > >> >>>>>>>>>>> 10.11.79.178:22 DNAT tcp -- anywhere > >> >>>>>>> 67.xxx.xxx.56 > >> >>>>>>>>>>> tcp dpt:ftp to:10.11.79.178:21 DNAT tcp -- > >> >>>>> anywhere > >> >>>>>>>>>>> 67.xxx.xxx.56 tcp dpt:5901 to:10.11.79.178:5901 > >> >>>>>>>>>>> > >> >>>>>>>>>>>> Date: Sat, 14 Sep 2013 17:25:14 +0100 > >> >>>>>>>>>>>> Subject: Re: Advanced Network - SNAT not working > >> >>>>>>>>>>>> From: msweet....@gmail.com > >> >>>>>>>>>>>> To: users@cloudstack.apache.org > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Hi Noel, > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Can you try using telnet to connect to an external > >> >>> webserver? > >> >>>>>>> telnet > >> >>>>>>>>>>>> www.google.com 80 > >> >>>>>>>>>>>> Can you also clarify: do you see the response packets > >> >>> reach > >> >>>>> the > >> >>>>>>> VR > >> >>>>>>>>> and/or > >> >>>>>>>>>>>> on what interfaces? > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Thanks, > >> >>>>>>>>>>>> Marty > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> On Saturday, September 14, 2013, Noel Kendall wrote: > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> Guest OS cannot receive responses to http GETs from > >> >>>>> resources > >> >>>>>>> on > >> >>>>>>>>> the > >> >>>>>>>>>>>>> Internet. > >> >>>>>>>>>>>>> Network is advanced, VLAN isolated. > >> >>>>>>>>>>>>> What is working: > >> >>>>>>>>>>>>> - can browse guest website from internet- can ssh to > >> >>> guest > >> >>>>> from > >> >>>>>>>>>>> internet- > >> >>>>>>>>>>>>> can VPN to guest network from internet > >> >>>>>>>>>>>>> - network VR can access internet sites no problem > >> >>>>>>>>>>>>> What is not working: > >> >>>>>>>>>>>>> - guest http traffic to external website gets to VR on > >> >>>>> internal > >> >>>>>>>>> NIC, > >> >>>>>>>>>>>>> packets forwarded to external site via external NIC > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> Response traffic is not seen. Appears to be dropped. > >> >>>>>>>>>>>>> Have been looking hard at IPTABLES rules, doing > >> >>> tcpdumps, > >> >>>>> etc. > >> >>>>>>>>>>>>> Am at this point stumped. > >> >>>>>>>>>>>>> Any ideas on what could be wrong, or how to determine > >> >>> what > >> >>>>>>> could be > >> >>>>>>>>>>> wrong? > >> >>>>>>>>>>>>> Thanks in advance everyone who tries to help! > >> >>>>>>>>>>>>> N. > >> >>>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>> > >> >>>>> > >> >>> > >> >>> > >> > > >> > > >
