Hi David, Thanks for the reply.
I just did some more troubleshooting and I can definitely see using dhcpdump on the VR that it only receives a DHCPREQUEST when the Guest VM is on the same KVM Host. If I migrate the guest to any other KVM Host that request never even makes it to the VR. I monitored the physical KVM host running the VR, but it never registered the DHCPREQUEST regardless of where the guest VM was running. I even turned off the firewall on both KVM Hosts and still no DHCPREQUEST. There actually is another physical DHCP server on the same network. *10.97.38.1 Router & DHCP* However we (corporate IT Admin) have blocked off a range of IPs {10.97.38.[110 -170]} from the 10.97.38.1 Router & DHCP and filtered out any DHCPREQUESTS made by a '06' MAC address per these instructions (written by you I believe): http://open.citrix.com/blog/42-tip-of-the-month-external-dhcp-server-and-cloudstack-on-the-same-network.html So that tells me that we probably did something wrong on the router and it's got to be the 10.97.38.1 Router/Switch/DHCP server that's stopping those requests from ever reaching the built in VR, but I have no visibility into that unfortunately so I cannot continue to test. I need to be able to give very precise instructions to my IT Admin on what to check. Thoughts? *dhcpdump from VR when Guest VM is on same KVM Host:* root@r-21-VM:~# dhcpdump -i eth0 TIME: 2013-11-18 08:24:46.997 IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff) OP: 1 (BOOTPREQUEST) HTYPE: 1 (Ethernet) HLEN: 6 HOPS: 0 XID: 060d4c79 SECS: 0 FLAGS: 0 CIADDR: 0.0.0.0 YIADDR: 0.0.0.0 SIADDR: 0.0.0.0 GIADDR: 0.0.0.0 CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 SNAME: . FNAME: . OPTION: 53 ( 1) DHCP message type 3 (DHCPREQUEST) OPTION: 50 ( 4) Request IP address 10.97.38.116 OPTION: 12 ( 16) Host name centos6-template OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask) 28 (Broadcast address) 2 (Time offset) 121 (Classless Static Route) 15 (Domainname) 6 (DNS server) 12 (Host name) 40 (NIS domain) 41 (NIS servers) 42 (NTP servers) 26 (Interface MTU) 119 (Domain Search) 3 (Routers) 121 (Classless Static Route) 249 (MSFT - Classless route) 42 (NTP servers) --------------------------------------------------------------------------- TIME: 2013-11-18 08:24:53.998 IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff) OP: 1 (BOOTPREQUEST) HTYPE: 1 (Ethernet) HLEN: 6 HOPS: 0 XID: 060d4c79 SECS: 7 FLAGS: 0 CIADDR: 0.0.0.0 YIADDR: 0.0.0.0 SIADDR: 0.0.0.0 GIADDR: 0.0.0.0 CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 SNAME: . FNAME: . OPTION: 53 ( 1) DHCP message type 3 (DHCPREQUEST) OPTION: 50 ( 4) Request IP address 10.97.38.116 OPTION: 12 ( 16) Host name centos6-template OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask) 28 (Broadcast address) 2 (Time offset) 121 (Classless Static Route) 15 (Domainname) 6 (DNS server) 12 (Host name) 40 (NIS domain) 41 (NIS servers) 42 (NTP servers) 26 (Interface MTU) 119 (Domain Search) 3 (Routers) 121 (Classless Static Route) 249 (MSFT - Classless route) 42 (NTP servers) --------------------------------------------------------------------------- TIME: 2013-11-18 08:25:02.003 IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff) OP: 1 (BOOTPREQUEST) HTYPE: 1 (Ethernet) HLEN: 6 HOPS: 0 XID: 91cdcd74 SECS: 0 FLAGS: 0 CIADDR: 0.0.0.0 YIADDR: 0.0.0.0 SIADDR: 0.0.0.0 GIADDR: 0.0.0.0 CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 SNAME: . FNAME: . OPTION: 53 ( 1) DHCP message type 1 (DHCPDISCOVER) OPTION: 50 ( 4) Request IP address 10.97.38.116 OPTION: 12 ( 16) Host name centos6-template OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask) 28 (Broadcast address) 2 (Time offset) 121 (Classless Static Route) 15 (Domainname) 6 (DNS server) 12 (Host name) 40 (NIS domain) 41 (NIS servers) 42 (NTP servers) 26 (Interface MTU) 119 (Domain Search) 3 (Routers) 121 (Classless Static Route) 249 (MSFT - Classless route) 42 (NTP servers) --------------------------------------------------------------------------- TIME: 2013-11-18 08:25:02.014 IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c) OP: 2 (BOOTPREPLY) HTYPE: 1 (Ethernet) HLEN: 6 HOPS: 0 XID: 91cdcd74 SECS: 0 FLAGS: 0 CIADDR: 0.0.0.0 YIADDR: 10.97.38.116 SIADDR: 10.97.38.113 GIADDR: 0.0.0.0 CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 SNAME: . FNAME: . OPTION: 53 ( 1) DHCP message type 2 (DHCPOFFER) OPTION: 54 ( 4) Server identifier 10.97.38.113 OPTION: 51 ( 4) IP address leasetime -1 () OPTION: 1 ( 4) Subnet mask 255.255.255.0 OPTION: 28 ( 4) Broadcast address 10.97.38.255 OPTION: 12 ( 16) Host name centos6-template OPTION: 6 ( 8) DNS server 10.97.38.113,10.97.32.20 OPTION: 3 ( 4) Routers 10.97.38.1 OPTION: 15 ( 17) Domainname cs1cloud.internal --------------------------------------------------------------------------- TIME: 2013-11-18 08:25:02.015 IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff) OP: 1 (BOOTPREQUEST) HTYPE: 1 (Ethernet) HLEN: 6 HOPS: 0 XID: 91cdcd74 SECS: 0 FLAGS: 0 CIADDR: 0.0.0.0 YIADDR: 0.0.0.0 SIADDR: 0.0.0.0 GIADDR: 0.0.0.0 CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 SNAME: . FNAME: . OPTION: 53 ( 1) DHCP message type 3 (DHCPREQUEST) OPTION: 54 ( 4) Server identifier 10.97.38.113 OPTION: 50 ( 4) Request IP address 10.97.38.116 OPTION: 12 ( 16) Host name centos6-template OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask) 28 (Broadcast address) 2 (Time offset) 121 (Classless Static Route) 15 (Domainname) 6 (DNS server) 12 (Host name) 40 (NIS domain) 41 (NIS servers) 42 (NTP servers) 26 (Interface MTU) 119 (Domain Search) 3 (Routers) 121 (Classless Static Route) 249 (MSFT - Classless route) 42 (NTP servers) --------------------------------------------------------------------------- TIME: 2013-11-18 08:25:02.016 IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c) OP: 2 (BOOTPREPLY) HTYPE: 1 (Ethernet) HLEN: 6 HOPS: 0 XID: 91cdcd74 SECS: 0 FLAGS: 0 CIADDR: 0.0.0.0 YIADDR: 10.97.38.116 SIADDR: 10.97.38.113 GIADDR: 0.0.0.0 CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 SNAME: . FNAME: . OPTION: 53 ( 1) DHCP message type 5 (DHCPACK) OPTION: 54 ( 4) Server identifier 10.97.38.113 OPTION: 51 ( 4) IP address leasetime -1 () OPTION: 1 ( 4) Subnet mask 255.255.255.0 OPTION: 28 ( 4) Broadcast address 10.97.38.255 OPTION: 12 ( 16) Host name centos6-template OPTION: 6 ( 8) DNS server 10.97.38.113,10.97.32.20 OPTION: 3 ( 4) Routers 10.97.38.1 OPTION: 15 ( 17) Domainname cs1cloud.internal --------------------------------------------------------------------------- root@r-21-VM:~# Best Regards, Adam Scarcella On Sun, Nov 17, 2013 at 2:30 PM, David Nalley <da...@gnsa.us> wrote: > So having seen this specific problem scores of times, it's almost > always network config related. Several potential problems to look out > for: > > 1. Another DHCP server on the same network. (There are ways of > mitigating this, but for the time being, lets just say that > CloudStack's VR should be the only DHCP server on the network. > > 2. Shell into the physical machine that the VR is running on. Using > tcpdump (with filters of course) Do you see the broadcast asking for a > DHCP address coming from the VM on the other physical machine? Do you > see the VR answering the request? If so, move to step 3. If not (for > either of those questions) - you aren't communicating between the two > physical hosts - your switch config is suspect. > > 3. Shell into the physical machine that the VM is running on - You've > seen the response with an IP address assigned go out - so see if it is > seen by the physical host NIC? If not, you've again. If you see the > response (and are sure it's from the VR and not another DHCP server) > come back and tell us - something funky is going on. > > --David > > On Sun, Nov 17, 2013 at 10:11 AM, Adam <adam.scarce...@gmail.com> wrote: > > Yes, I've disabled the firewall on both KVM hosts in question and still > no > > dice. I can't even ping the VR from my guest VM, but when I set the eth0 > > device on the guest to static everything works fine, which makes no sense > > at all to me. Simply setting the NIC to static allows me see the VR. > > Switching back to DHCP kills it again. I do not understand what is > required > > for the guests to acquire a DHCP lease from the VR. I know that the VR is > > running dnsmasq, and I've tailed the /var/log/dnsmasq.log for more info, > > but only see a DHCP request when the guest is on the same KVM host as the > > VR. > > > > Does anyone know exactly how to troubleshoot this scenario? I'm not even > > sure what to look for. > > > > -Adam > > > > Best Regards, > > > > > > > > Adam Scarcella > > > > > > On Sat, Nov 16, 2013 at 5:43 PM, Carlos Reátegui <create...@gmail.com > >wrote: > > > >> Did you also check the host firewall? Try disabling. > >> > >> > On Nov 16, 2013, at 1:51 PM, Adam <adam.scarce...@gmail.com> wrote: > >> > > >> > The hosts have static IPs (10.97.38.[10-14]) and can all see and talk > to > >> > each other via IP and hostname. I'm only using a basic zone so no VLAN > >> > tagging or anything funky like that. > >> > > >> > Best Regards, > >> > > >> > > >> > > >> > Adam Scarcella > >> > > >> > > >> > On Sat, Nov 16, 2013 at 4:17 PM, Andrei Mikhailovsky < > and...@arhont.com > >> >wrote: > >> > > >> >> > >> >> > >> >> Adam, it sounds like a networking issue, check that all hosts can > talk > >> to > >> >> each other on the same vlan that is used for guest network. I had the > >> same > >> >> issue with advanced networking when my vlans were not properly setup > on > >> the > >> >> switches. > >> >> > >> >> Andrei > >> >> > >> >> ----- Original Message ----- > >> >> > >> >> From: "Adam" <adam.scarce...@gmail.com> > >> >> To: users@cloudstack.apache.org > >> >> Sent: Saturday, 16 November, 2013 7:08:50 PM > >> >> Subject: Guest VMs not able to acquire DHCP IP from Virtual Router > >> unless > >> >> Guest and VR are on the same KVM host > >> >> > >> >> Hi All, > >> >> > >> >> I have a new and very strange issue that for the life of me I cannot > >> seem > >> >> to track down and fix. > >> >> > >> >> I have CS 4.2 running on 5 hosts in a simple basic zone. All is > working > >> >> fine, except that my Guest VMs cannot seem to get a DHCP lease from > the > >> >> Virtual Router unless I migrate the Guest VM to the same physical KVM > >> >> Host running the VR. That would seem to indicate a firewall issue, > but > >> I've > >> >> tested by turning off both the firewalls (VR KVM Host iptables & > Guest > >> VM > >> >> KVM Host iptables). It didn't help. The only way to fix it is to > migrate > >> >> the Guest VM to the same KVM Host that's running the Virtual Router. > >> >> > >> >> NOTE: The Console Proxy has worked flawlessly this whole time. > >> >> > >> >> So, if a Guest VM starts on a different physical KVM Host, it will > not > >> get > >> >> an internal IP of 169.254.x.x. All along the Console Proxy works > fine. > >> Then > >> >> if I migrate the Guest VM to the same KVM host that's running the VR, > >> DHCP > >> >> automatically starts working and the Guest VM receives a proper IP > >> address > >> >> of 10.97.38.x. Then I can migrate the Guest VM back to any other > >> physical > >> >> KVM Host and believe it or not, it continues to work flawlessly, > until I > >> >> either reboot the VM or restart the network services. Then it cannot > see > >> >> the VR again and instead receives an internal IP of 169.254.x.x. If I > >> set a > >> >> static IP address & DNS everything works fine, no matter where the > >> Guest VM > >> >> is running. > >> >> > >> >> Setting static IPs is not an option > >> >> Running all the Guest VMs on the same physical KVM host is not an > option > >> >> > >> >> I desperately need to track down the root cause of this issue so I > can > >> >> release this cloud to my entire department by Monday morning. Someone > >> >> please help! > >> >> > >> >> Best Regards, > >> >> > >> >> > >> >> > >> >> Adam Scarcella > >> >> > >> >> > >> >