Hi David, >From the initial VM, where you able to ping a public IP such as 8.8.8.8 or 74.125.225.128 ? just in case it would be a DNS issue.
>From the VR, can you telnet <your cloudstack MAnagement server>:8250, Maybe the VR was unable to contact CloudStack MGTport? Pierre-Luc Dion Architecte de Solution Cloud | Cloud Solutions Architect 855-OK-CLOUD (855-652-5683) x1101 - - - *CloudOps*420 rue Guy Montréal QC H3J 1S6 www.cloudops.com @CloudOps_ On Wed, May 21, 2014 at 1:18 PM, David <d...@ntso.com> wrote: > Hello > > Cloudstack VM Issue, no outgoing access > > There is an issue in cloudstack where a VM cannot gain outgoing access to > the internet. This is a > blocking issue since Cloudstack users cannot gain Internet access to do > patch updates, download > software, etc. > > Environment: > > Cloudstack version 4.3 > System VM from January 2014 > CentOS6.5 for all servers > 4 servers (management server, 2 compute nodes, and a server that serves > both > primary and secondary > storage) > Advanced networking zone with 1 zone, 1 pod, 1 cluster, 2 hosts > 2 NICs on the compute node hosts > - 1 NIC hosts the management and storage > - 1 NIC hosts the public and guest networks > Primary and Secondard storage use NFS > Management network is 172.16.10.x/24 > Guest networks are 10.0.0.x/24 > Public network uses a /24 public range > > > Here is how to reproduce the prove the issue. > > 1. Create a new VM using a new isolated network > 2. Create the egress rules to allow 10.0.0.0/24 out for everything > 3. Create a port map rule for SSH to the new VM and also allow external > acess to SSH > 4. Login to the console on the new VM > 5. Ping google.com, see 100% packet loss. > 6. Login to the console on the virtual router for this network and type > this > command: > > Ping results > [root@NewTestVM ~]# ping google.com > PING google.com (74.125.225.128) 56(84) bytes of data. > ^C > --- google.com ping statistics --- > 7 packets transmitted, 0 received, 100% packet loss, time > 6127ms > [root@NewTestVM ~]# > > > > All the iptables rules look as they should, especially look at the SNAT > rule > in the nat table for > POSTROUTING. But, still can't ping. How do we know there isn't some other > problem in the > network. Well, first we were able to SSH to the new VM externally from the > Internet to do the ping. > > > Second, we found that if we add a second identical SNAT rule, now > everything > works. > > Okay so now add the second SNAT rule like this; > > iptables -t nat -A POSTROUTING -j SNAT –t-source 68.70.175.6 > > Yes, pinging google.com still does not work, losing 100% of packets.. > > > Now, Pinging from the new VM works like a charm. What is going on?? > [root@NewTestVM ~]# ping google.com > PING google.com (74.125.225.132) 56(84) bytes of data. > 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=1 ttl=52 > time=2.12 ms > 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=2 ttl=52 > time=2.09 ms > 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=3 ttl=52 > time=2.22 ms > 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=4 ttl=52 > time=2.24 ms > ^C > --- google.com ping statistics --- > 4 packets transmitted, 4 received, 0% packet loss, time 3556ms > rtt min/avg/max/mdev = 2.095/2.172/2.249/0.064 ms > [root@NewTestVM ~]# > > > The first SNAT rule should have worked but it doesn't. WE've proven, I > think, that everything else is > setup correctly by being able to access the VM externally via SSH and by > simply adding the SNAT rule > to the virtual router. Why doesn't the first rule work. > > > It seems cloudstack is doing exactly what it is supposed to do, adding the > SNAT rule. So, it seems the > actual problem is with either iptables or with the debian kernel for the > system VM. I think we are using > the most recent system VM. > > Note, neither restarting the new network nor cleaning it makes any > difference. It just seems having the > one SNAT rule will not work. I've done all the tcpdump debugging, that is > how I found this issue to > begin with. Tcpdump shows that the ping/icmp packets go through the virtual > router unaltered using > the private IP address of the VM (10.0.0.x). After the second SNAT rule is > added tcpdump shows then > that the ping/icmp packets do get translated. > > I am totally floored by these results. How can iptables or the debian > kernel > not recognize the first > SNAT rule and then recognize the second one?? > > > Okay, so before someone tells me that this is not a cloudstack issue and to > find a resolution else, > cloudstack depends on this technique for isolated networks. If guest users > can't make their Vms work > (note, they don't have access to the virtual routers like I do), isolated > networks are broken and > cloudstack is not usable. > > > I have screen shots i can send if it will help. > >
