Hi David, Sorry for late response! I would be surprise there is issue with the VR, I would suspect a communication issue between Cloudstack management-server or a misconfiguration in the zone.
- Does the Global setting "host" is an IP or FQDN that can be reach by the VR ? the VR must contact the port 8250 of the management-server. - does public DNS configured on the zone are reachable? were you able to ping IP not FQDN from VM's ? Pierre-Luc Dion Architecte de Solution Cloud | Cloud Solutions Architect 855-OK-CLOUD (855-652-5683) x1101 - - - *CloudOps*420 rue Guy Montréal QC H3J 1S6 www.cloudops.com @CloudOps_ On Fri, May 23, 2014 at 12:43 PM, David Gil <d...@netsource.com> wrote: > Hello Pierre > > I know how to fix this problem with adding the iptables rule on the > virtual router for the network even though the rule shows up as installed > already. While I can do this as i am an admin, the end user cannot since > they cant get to the virtual router to make the change. Do you know what is > the cause for this and how to fix it? > > Thank you > > David Gil > d...@ntso.com > > > Hi David, > > >From the initial VM, where you able to ping a public IP such as 8.8.8.8 > or 74.125.225.128 ? just in case it would be a DNS issue. >From the VR, can > you telnet <your cloudstack MAnagement server>:8250, Maybe the VR was > unable to contact CloudStack MGTport? > > Pierre-Luc Dion > Architecte de Solution Cloud | Cloud Solutions Architect > 855-OK-CLOUD (855-652-5683) x1101 - - - > *CloudOps*420 rue Guy > Montréal QC H3J 1S6 > www.cloudops.com > @CloudOps_ > > On 5/20/2014 4:29 PM, David Gil wrote: > > Hello > > Cloudstack VM Issue, no outgoing access > > There is an issue in cloudstack where a VM cannot gain outgoing access to > the internet. This is a > blocking issue since Cloudstack users cannot gain Internet access to do > patch updates, download > software, etc. > > Environment: > > Cloudstack version 4.3 > System VM from January 2014 > CentOS6.5 for all servers > 4 servers (management server, 2 compute nodes, and a server that serves > both primary and secondary > storage) > Advanced networking zone with 1 zone, 1 pod, 1 cluster, 2 hosts > 2 NICs on the compute node hosts > - 1 NIC hosts the management and storage > - 1 NIC hosts the public and guest networks > Primary and Secondard storage use NFS > Management network is 172.16.10.x/24 > Guest networks are 10.0.0.x/24 > Public network uses a /24 public range > > > Here is how to reproduce the prove the issue. > > 1. Create a new VM using a new isolated network > 2. Create the egress rules to allow 10.0.0.0/24 out for everything > 3. Create a port map rule for SSH to the new VM and also allow external > acess to SSH > 4. Login to the console on the new VM > 5. Ping google.com, see 100% packet loss. > 6. Login to the console on the virtual router for this network and type > this command: > iptables -t nat -L and > iptables -L > > The results are in the screen print below. > Ping results > [root@NewTestVM ~]# ping google.com > PING google.com (74.125.225.128) 56(84) bytes of data. > ^C > --- google.com ping statistics --- > 7 packets transmitted, 0 received, 100% packet loss, time > 6127ms > [root@NewTestVM ~]# > > > iptables -t nat -L > > iptables -L > > > > > > > > All the iptables rules look as they should, especially look at the SNAT > rule in the nat table for > POSTROUTING. But, still can't ping. How do we know there isn't some other > problem in the > network. Well, first we were able to SSH to the new VM externally from the > Internet to do the ping. > > > Second, we found that if we add a second identical SNAT rule, now > everything works. > > Okay so now add the second SNAT rule like this; > > iptables -t nat -A POSTROUTING -j SNAT –t-source 68.70.175.6 > > Yes, pinging google.com still does not work, losing 100% of packets. Now > here is the screen print of > the virtual router iptables setup after the command above. > > > Now, Pinging from the new VM works like a charm. What is going on?? > [root@NewTestVM ~]# ping google.com > PING google.com (74.125.225.132) 56(84) bytes of data. > 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=1 > ttl=52 time=2.12 ms > 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=2 > ttl=52 time=2.09 ms > 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=3 > ttl=52 time=2.22 ms > 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=4 > ttl=52 time=2.24 ms > ^C > --- google.com ping statistics --- > 4 packets transmitted, 4 received, 0% packet loss, time 3556ms > rtt min/avg/max/mdev = 2.095/2.172/2.249/0.064 ms > [root@NewTestVM ~]# > > > The first SNAT rule should have worked but it doesn't. WE've proven, I > think, that everything else is > setup correctly by being able to access the VM externally via SSH and by > simply adding the SNAT rule > to the virtual router. Why doesn't the first rule work. > > > It seems cloudstack is doing exactly what it is supposed to do, adding the > SNAT rule. So, it seems the > actual problem is with either iptables or with the debian kernel for the > system VM. I think we are using > the most recent system VM. > > Note, neither restarting the new network nor cleaning it makes any > difference. It just seems having the > one SNAT rule will not work. I've done all the tcpdump debugging, that is > how I found this issue to > begin with. Tcpdump shows that the ping/icmp packets go through the > virtual router unaltered using > the private IP address of the VM (10.0.0.x). After the second SNAT rule is > added tcpdump shows then > that the ping/icmp packets do get translated. > > I am totally floored by these results. How can iptables or the debian > kernel not recognize the first > SNAT rule and then recognize the second one?? > > > Okay, so before someone tells me that this is not a cloudstack issue and > to find a resolution else, > cloudstack depends on this technique for isolated networks. If guest users > can't make their Vms work > (note, they don't have access to the virtual routers like I do), isolated > networks are broken and > cloudstack is not usable. > > > Now, one more thought... I have only been using cloudstack for about 2 > weeks and I can't believe that I > am the first one to have this issue. So, I am open to something I have > done wrong if someone has any > ideas. I can give access to someone that wants to be able to reproduce > this issue and look for a solution > if no one else can reproduce this – it is a prototype environment so what > happens to it doesn't matter > > Please help. > > Thank you! > > > -- > David Gil > NetSource Communications Inc. > 2368 Corporate Lane Suite 112 > Naperville, IL 60563 > Phone: 630-778-1212 > Fax: 630-778-0459support@netsource.comhttp://www.netsource.com > >
