No other NAT. There is nothing but copper between the KVM host machine and the
ISP router.There is an L2/L3 switch that the packets travel through. However,
there is no forwarding in the switch,just straight through. I've had a
well-functioning V4.0.1 environment running on this same configurationin the
past. What is new is the conversion to 4.1 (which was a clean install).
It's very mysterious, I have never seen anything like this before. There are
two other VRs, both having same issue.
I will try your suggestion.
Noel
> Date: Sun, 15 Sep 2013 21:20:41 +0100
> Subject: Re: Advanced Network - SNAT not working
> From: msweet....@gmail.com
> To: users@cloudstack.apache.org
>
> This is mostly confusing that the packets are not seen on the VR public
> interface, seeing as other services are working.
> If it was a local NAT issue then the packet would atleast get into that
> interface. Do you have any upstream devices providing NAT? Or any other VR
> with the issue?
>
> It may be worth recreating the VR, by stopping and destroying it and
> creating another guest to start a fresh.
>
> Marty
>
>
> On Sun, Sep 15, 2013 at 8:12 PM, Noel Kendall <noeldkend...@hotmail.com>wrote:
>
> > Marty, if I run a telnet <www.xyz.com> 80 from a shell in the guest,
> > while running a tcpdumpon the public i/f of the VR:
> > - I can see the outbound packets going out- I do not see a response packet
> > coming back in
> > FYI there are no firewalls outbound from the KVM host. The host bridges vi
> > CS networkingdirectly out on to the internet via a switch.
> > Note that traffic from outside (ssh, web) can happily traverse the VR to
> > the guest. I get the usualits working html page from the guest. This tells
> > me that there is nothing outbound from the VR thatis filtering packets.
> > Am truly stumped. This is mysterious indeed.
> > From within the VR, can happily telnet to <www.xyz.com> 80 and receive
> > response.Only if packet came from guest and was forwarded does the response
> > not show up.
> > In short:
> > wget from VR to www.xyz.com works, response received and saved
> > wget from guest to www.xyz.com does not work, network not available
> > displayed on guest, response packets not seen on the public i/f of VR at all
> > Noel
> >
> > > Date: Sun, 15 Sep 2013 18:16:17 +0100
> > > Subject: Re: Advanced Network - SNAT not working
> > > From: msweet....@gmail.com
> > > To: users@cloudstack.apache.org
> > >
> > > Hi Noel,
> > >
> > > Can you answer: Does the traffic come back on the public interface? and
> > > then onto the Guest interface?
> > >
> > > Thanks,
> > > Marty
> > >
> > >
> > > On Sun, Sep 15, 2013 at 2:05 PM, Noel Kendall <noeldkend...@hotmail.com
> > >wrote:
> > >
> > > > Indeed, yes, a wget executed on the VR to a public website works just
> > fine.
> > > > Noel
> > > >
> > > > > Date: Sun, 15 Sep 2013 13:15:20 +0100
> > > > > Subject: Re: Advanced Network - SNAT not working
> > > > > From: msweet....@gmail.com
> > > > > To: users@cloudstack.apache.org
> > > > >
> > > > > Hi Noel,
> > > > >
> > > > > Does the traffic come back on the public interface? and then onto the
> > > > Guest
> > > > > interface?
> > > > >
> > > > > Does a wget on the VR work?
> > > > >
> > > > > Marty
> > > > >
> > > > >
> > > > > On Sat, Sep 14, 2013 at 8:19 PM, Noel Kendall <
> > noeldkend...@hotmail.com
> > > > >wrote:
> > > > >
> > > > > > I have that Marty. I see the http outbound request coming in on the
> > > > guest
> > > > > > interface of the VR,and see the http request being sent out on the
> > > > public
> > > > > > interface of the VR.
> > > > > > The traffic is flowing fine from guest to the outbound i/f of the
> > VR.
> > > > > > This is tcpdump on the public i/f while guest is doing wget to
> > > > > > 6x.xxx.xxx.xxx
> > > > > >
> > > > > > 19:17:58.834932 06:e3:3a:00:01:0a > 00:0c:86:4e:fe:00, ethertype
> > IPv4
> > > > > > (0x0800), length 74: 10.11.79.178.39074 > 6x.xxx.xxx.xx.80: Flags
> > [S],
> > > > seq
> > > > > > 1859313238, win 14600, options [mss 1460,sackOK,TS val 27489348 ecr
> > > > > > 0,nop,wscale 4], length 0 0x0000: 4500 003c ad1d 4000 3f06 2d13
> > 0a0b
> > > > 4fb2
> > > > > > 0x0010: 416e c660 98a2 0050 6ed2 de56 0000 0000
> > 0x0020:
> > > > > > a002 3908 516c 0000 0204 05b4 0402 080a 0x0030: 01a3 7444
> > 0000
> > > > > > 0000 0103 0304
> > > > > >
> > > > > >
> > > > > > > Date: Sat, 14 Sep 2013 19:29:53 +0100
> > > > > > > Subject: Re: Advanced Network - SNAT not working
> > > > > > > From: msweet....@gmail.com
> > > > > > > To: users@cloudstack.apache.org
> > > > > > >
> > > > > > > Hi Noel,
> > > > > > >
> > > > > > > Can you run a tcpdump on both VR interfaces, this should make it
> > > > apparent
> > > > > > > what is happening?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Marty
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Sep 14, 2013 at 6:41 PM, Noel Kendall <
> > > > noeldkend...@hotmail.com
> > > > > > >wrote:
> > > > > > >
> > > > > > > > http://pastebin.com/3FZmFnvZ
> > > > > > > > Many thanks Marty.
> > > > > > > > Noel
> > > > > > > > > Date: Sat, 14 Sep 2013 18:07:55 +0100
> > > > > > > > > Subject: Re: Advanced Network - SNAT not working
> > > > > > > > > From: msweet....@gmail.com
> > > > > > > > > To: users@cloudstack.apache.org
> > > > > > > > >
> > > > > > > > > Hi Noel,
> > > > > > > > >
> > > > > > > > > Could you put the IP tables on pastebin? GMail has collapsed
> > the
> > > > > > lines
> > > > > > > > > horrifically.
> > > > > > > > > Have you also tried a tcpdump on both interfaces on the VR?
> > > > > > > > > tcpdump -i eth0 <--- Or whatever it may be called
> > > > > > > > >
> > > > > > > > > I would expect worse connectivity if it was a pure NAT issue,
> > > > but I
> > > > > > will
> > > > > > > > > review the tables later.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Marty
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sat, Sep 14, 2013 at 5:55 PM, Noel Kendall <
> > > > > > noeldkend...@hotmail.com
> > > > > > > > >wrote:
> > > > > > > > >
> > > > > > > > > > Not seeing return packets on VR. Suspect, therefore, that
> > SNAT
> > > > is
> > > > > > > > fouled
> > > > > > > > > > up in some way.I have been doing wget to from guest, can
> > see
> > > > the
> > > > > > > > outgoing
> > > > > > > > > > request fine, both in the guest andthe VR.
> > > > > > > > > > Could it be that the SNAT table entries from the
> > > > 10.11.0.0/16subnet
> > > > > > > > to
> > > > > > > > > > dpt www are interfering withthe SNAT to public ip?? (wild
> > > > guess) -
> > > > > > not
> > > > > > > > an
> > > > > > > > > > iptables expert by any stretch of the imagination
> > > > > > > > > > 67.xxx.xxx.56 is the guest public IP10.11.79.178 is the
> > guest
> > > > IP on
> > > > > > > > guest
> > > > > > > > > > network
> > > > > > > > > > iptables _L -t nat on the VR shows...
> > > > > > > > > > Chain PREROUTING (policy ACCEPT)target prot opt source
> > > > > > > > > > destination DNAT tcp -- anywhere
> > > > > > anywhere
> > > > > > > > > > tcp dpt:domain to:10.11.0.1 DNAT tcp --
> > anywhere
> > > > > > > > > > 67.xxx.xxx.56 tcp dpt:www to:10.11.79.178:80 DNAT
> > > > > > tcp --
> > > > > > > > > > anywhere 67.xxx.xxx.56 tcp dpt:www
> > > > > > > > to:10.11.79.178:80DNAT tcp -- anywhere
> > > > > > 67.xxx.xxx.56
> > > > > > > > tcp dpt:https
> > > > > > > > > > to:10.11.79.178:443 DNAT tcp -- anywhere
> > > > > > > > > > 67.xxx.xxx.56 tcp dpt:https to:10.11.79.178:443DNAT
> > > > > > tcp
> > > > > > > > --
> > > > > > > > > > anywhere 67.xxx.xxx.56 tcp dpt:ssh
> > > > > > > > to:10.11.79.178:22DNAT tcp -- anywhere
> > > > > > 67.xxx.xxx.56
> > > > > > > > tcp dpt:ssh
> > > > > > > > > > to:10.11.79.178:22 DNAT tcp -- anywhere
> > > > > > > > 67.xxx.xxx.56
> > > > > > > > > > tcp dpt:ftp to:10.11.79.178:21 DNAT tcp --
> > > > anywhere
> > > > > > > > > > 67.xxx.xxx.56 tcp dpt:ftp
> > > > > > > > > > to:10.11.79.178:21DNAT
> > > > > > > > tcp
> > > > > > > > > > -- anywhere 67.xxx.xxx.56 tcp
> > dpt:5901 to:
> > > > > > > > > > 10.11.79.178:5901 DNAT tcp -- anywhere
> > > > > > > > 67.xxx.xxx.56
> > > > > > > > > > tcp dpt:5901 to:10.11.79.178:5901
> > > > > > > > > > Chain POSTROUTING (policy ACCEPT)target prot opt source
> > > > > > > > > > destination SNAT all -- anywhere
> > > > > > anywhere
> > > > > > > > > > to:67.xxx.xxx.56 SNAT all -- anywhere
> > > > > > > > anywhere
> > > > > > > > > > to:67.xxx.xxx.56 SNAT all -- anywhere
> > > > > > > > > > anywhere to:67.xxx.xxx.56 SNAT all --
> > > > anywhere
> > > > > > > > > > anywhere to:67.xxx.xxx.56 SNAT all --
> > > > anywhere
> > > > > > > > > > anywhere to:67.xxx.xxx.56SNAT all --
> > > > > > anywhere
> > > > > > > > > > anywhere to:67.xxx.xxx.56 SNAT all
> > --
> > > > > > anywhere
> > > > > > > > > > anywhere to:67.xxx.xxx.56 SNAT
> > all --
> > > > > > > > anywhere
> > > > > > > > > > anywhere to:67.xxx.xxx.56 SNAT
> > tcp
> > > > --
> > > > > > > > > > 10.11.0.0/16 myguest tcp dpt:www
> > > > to:10.11.0.1
> > > > > > SNAT
> > > > > > > > > > tcp -- 10.11.0.0/16 myguest tcp
> > > > > > dpt:https
> > > > > > > > > > to:10.11.0.1 SNAT tcp -- 10.11.0.0/16
> > myguest
> > > > > > > > > > tcp dpt:ssh to:10.11.0.1 SNAT tcp -- 10.11.0.0/16
> > > > > > > > myguest
> > > > > > > > > > tcp dpt:ftp to:10.11.0.1 SNAT tcp --
> > > > > > 10.11.0.0/16
> > > > > > > > > > myguest tcp dpt:5901 to:10.11.0.1 SNAT
> > > > all
> > > > > > --
> > > > > > > > > > anywhere anywhere to:67.xxx.xxx.56
> > > > > > > > > > Chain OUTPUT (policy ACCEPT)target prot opt source
> > > > > > > > > > destination DNAT tcp -- anywhere
> > > > > > > > 67.xxx.xxx.56
> > > > > > > > > > tcp dpt:www to:10.11.79.178:80 DNAT tcp --
> > > > anywhere
> > > > > > > > > > 67.xxx.xxx.56 tcp dpt:https
> > > > > > > > > > to:10.11.79.178:443DNAT
> > > > > > > > tcp
> > > > > > > > > > -- anywhere 67.xxx.xxx.56 tcp dpt:ssh
> > to:
> > > > > > > > > > 10.11.79.178:22 DNAT tcp -- anywhere
> > > > > > 67.xxx.xxx.56
> > > > > > > > > > tcp dpt:ftp to:10.11.79.178:21 DNAT tcp --
> > > > anywhere
> > > > > > > > > > 67.xxx.xxx.56 tcp dpt:5901 to:10.11.79.178:5901
> > > > > > > > > >
> > > > > > > > > > > Date: Sat, 14 Sep 2013 17:25:14 +0100
> > > > > > > > > > > Subject: Re: Advanced Network - SNAT not working
> > > > > > > > > > > From: msweet....@gmail.com
> > > > > > > > > > > To: users@cloudstack.apache.org
> > > > > > > > > > >
> > > > > > > > > > > Hi Noel,
> > > > > > > > > > >
> > > > > > > > > > > Can you try using telnet to connect to an external
> > webserver?
> > > > > > telnet
> > > > > > > > > > > www.google.com 80
> > > > > > > > > > > Can you also clarify: do you see the response packets
> > reach
> > > > the
> > > > > > VR
> > > > > > > > and/or
> > > > > > > > > > > on what interfaces?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Marty
> > > > > > > > > > >
> > > > > > > > > > > On Saturday, September 14, 2013, Noel Kendall wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Guest OS cannot receive responses to http GETs from
> > > > resources
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > > Internet.
> > > > > > > > > > > > Network is advanced, VLAN isolated.
> > > > > > > > > > > > What is working:
> > > > > > > > > > > > - can browse guest website from internet- can ssh to
> > guest
> > > > from
> > > > > > > > > > internet-
> > > > > > > > > > > > can VPN to guest network from internet
> > > > > > > > > > > > - network VR can access internet sites no problem
> > > > > > > > > > > > What is not working:
> > > > > > > > > > > > - guest http traffic to external website gets to VR on
> > > > internal
> > > > > > > > NIC,
> > > > > > > > > > > > packets forwarded to external site via external NIC
> > > > > > > > > > > >
> > > > > > > > > > > > Response traffic is not seen. Appears to be dropped.
> > > > > > > > > > > > Have been looking hard at IPTABLES rules, doing
> > tcpdumps,
> > > > etc.
> > > > > > > > > > > > Am at this point stumped.
> > > > > > > > > > > > Any ideas on what could be wrong, or how to determine
> > what
> > > > > > could be
> > > > > > > > > > wrong?
> > > > > > > > > > > > Thanks in advance everyone who tries to help!
> > > > > > > > > > > > N.
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > >
> > > >
> >
> >
