Thanks Travis. I agree and will see if Corporate IT will comply with that request. Will post results/response soon.
Best Regards, Adam Scarcella On Tue, Nov 19, 2013 at 8:36 AM, Travis Graham <tgra...@tgraham.us> wrote: > Since DHCP leases aren't something that immediately go away you could > temporarily stop DHCP services on your default router and see if that > allows CS instances to pull IPs correctly then enable your default DHCP > server again once you know more. > > That would eliminate your default router from the possible causes list if > nothing changes or confirm it if things start working correctly in your CS > environment. > > Travis > > On Nov 19, 2013, at 8:19 AM, Adam <adam.scarce...@gmail.com> wrote: > > > Good Morning David, > > > > I'm not sure if you saw my reply from yesterday, but I'm now 99% sure the > > problem is on our default router/dhcp server on 10.97.38.1. I think we > did > > everything we're supposed to to mitigate the normal issues you would > expect > > from having two competing DHCP servers on the same network. We went by > your > > instructions > > > http://open.citrix.com/blog/42-tip-of-the-month-external-dhcp-server-and-cloudstack-on-the-same-network.html > > > > Perhaps we implemented the filtering incorrectly? My bosses are getting > > anxious with me and I don't have an answer for them yet. > > > > Any further help you could provide would be greatly appreciated. Thanks. > > > > Best Regards, > > > > > > > > Adam Scarcella > > > > > > On Mon, Nov 18, 2013 at 12:40 PM, Adam <adam.scarce...@gmail.com> wrote: > > > >> Hi David, > >> > >> Thanks for the reply. > >> > >> I just did some more troubleshooting and I can definitely see using > >> dhcpdump on the VR that it only receives a DHCPREQUEST when the Guest VM > >> is on the same KVM Host. If I migrate the guest to any other KVM Host > that > >> request never even makes it to the VR. I monitored the physical KVM host > >> running the VR, but it never registered the DHCPREQUEST regardless of > where > >> the guest VM was running. > >> > >> I even turned off the firewall on both KVM Hosts and still no > DHCPREQUEST. > >> > >> There actually is another physical DHCP server on the same network. > *10.97.38.1 > >> Router & DHCP* > >> > >> However we (corporate IT Admin) have blocked off a range of IPs > >> {10.97.38.[110 -170]} from the 10.97.38.1 Router & DHCP and filtered out > >> any DHCPREQUESTS made by a '06' MAC address per these instructions > (written > >> by you I believe): > >> > >> > >> > http://open.citrix.com/blog/42-tip-of-the-month-external-dhcp-server-and-cloudstack-on-the-same-network.html > >> > >> So that tells me that we probably did something wrong on the router and > >> it's got to be the 10.97.38.1 Router/Switch/DHCP server that's stopping > >> those requests from ever reaching the built in VR, but I have no > visibility > >> into that unfortunately so I cannot continue to test. I need to be able > to > >> give very precise instructions to my IT Admin on what to check. > >> > >> Thoughts? > >> > >> *dhcpdump from VR when Guest VM is on same KVM Host:* > >> > >> root@r-21-VM:~# dhcpdump -i eth0 > >> TIME: 2013-11-18 08:24:46.997 > >> IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff) > >> OP: 1 (BOOTPREQUEST) > >> HTYPE: 1 (Ethernet) > >> HLEN: 6 > >> HOPS: 0 > >> XID: 060d4c79 > >> SECS: 0 > >> FLAGS: 0 > >> CIADDR: 0.0.0.0 > >> YIADDR: 0.0.0.0 > >> SIADDR: 0.0.0.0 > >> GIADDR: 0.0.0.0 > >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 > >> SNAME: . > >> FNAME: . > >> OPTION: 53 ( 1) DHCP message type 3 (DHCPREQUEST) > >> OPTION: 50 ( 4) Request IP address 10.97.38.116 > >> OPTION: 12 ( 16) Host name centos6-template > >> OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask) > >> 28 (Broadcast address) > >> 2 (Time offset) > >> 121 (Classless Static Route) > >> 15 (Domainname) > >> 6 (DNS server) > >> 12 (Host name) > >> 40 (NIS domain) > >> 41 (NIS servers) > >> 42 (NTP servers) > >> 26 (Interface MTU) > >> 119 (Domain Search) > >> 3 (Routers) > >> 121 (Classless Static Route) > >> 249 (MSFT - Classless route) > >> 42 (NTP servers) > >> > >> > --------------------------------------------------------------------------- > >> > >> TIME: 2013-11-18 08:24:53.998 > >> IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff) > >> OP: 1 (BOOTPREQUEST) > >> HTYPE: 1 (Ethernet) > >> HLEN: 6 > >> HOPS: 0 > >> XID: 060d4c79 > >> SECS: 7 > >> FLAGS: 0 > >> CIADDR: 0.0.0.0 > >> YIADDR: 0.0.0.0 > >> SIADDR: 0.0.0.0 > >> GIADDR: 0.0.0.0 > >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 > >> SNAME: . > >> FNAME: . > >> OPTION: 53 ( 1) DHCP message type 3 (DHCPREQUEST) > >> OPTION: 50 ( 4) Request IP address 10.97.38.116 > >> OPTION: 12 ( 16) Host name centos6-template > >> OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask) > >> 28 (Broadcast address) > >> 2 (Time offset) > >> 121 (Classless Static Route) > >> 15 (Domainname) > >> 6 (DNS server) > >> 12 (Host name) > >> 40 (NIS domain) > >> 41 (NIS servers) > >> 42 (NTP servers) > >> 26 (Interface MTU) > >> 119 (Domain Search) > >> 3 (Routers) > >> 121 (Classless Static Route) > >> 249 (MSFT - Classless route) > >> 42 (NTP servers) > >> > >> > --------------------------------------------------------------------------- > >> > >> TIME: 2013-11-18 08:25:02.003 > >> IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff) > >> OP: 1 (BOOTPREQUEST) > >> HTYPE: 1 (Ethernet) > >> HLEN: 6 > >> HOPS: 0 > >> XID: 91cdcd74 > >> SECS: 0 > >> FLAGS: 0 > >> CIADDR: 0.0.0.0 > >> YIADDR: 0.0.0.0 > >> SIADDR: 0.0.0.0 > >> GIADDR: 0.0.0.0 > >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 > >> SNAME: . > >> FNAME: . > >> OPTION: 53 ( 1) DHCP message type 1 (DHCPDISCOVER) > >> OPTION: 50 ( 4) Request IP address 10.97.38.116 > >> OPTION: 12 ( 16) Host name centos6-template > >> OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask) > >> 28 (Broadcast address) > >> 2 (Time offset) > >> 121 (Classless Static Route) > >> 15 (Domainname) > >> 6 (DNS server) > >> 12 (Host name) > >> 40 (NIS domain) > >> 41 (NIS servers) > >> 42 (NTP servers) > >> 26 (Interface MTU) > >> 119 (Domain Search) > >> 3 (Routers) > >> 121 (Classless Static Route) > >> 249 (MSFT - Classless route) > >> 42 (NTP servers) > >> > >> > --------------------------------------------------------------------------- > >> > >> TIME: 2013-11-18 08:25:02.014 > >> IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c) > >> OP: 2 (BOOTPREPLY) > >> HTYPE: 1 (Ethernet) > >> HLEN: 6 > >> HOPS: 0 > >> XID: 91cdcd74 > >> SECS: 0 > >> FLAGS: 0 > >> CIADDR: 0.0.0.0 > >> YIADDR: 10.97.38.116 > >> SIADDR: 10.97.38.113 > >> GIADDR: 0.0.0.0 > >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 > >> SNAME: . > >> FNAME: . > >> OPTION: 53 ( 1) DHCP message type 2 (DHCPOFFER) > >> OPTION: 54 ( 4) Server identifier 10.97.38.113 > >> OPTION: 51 ( 4) IP address leasetime -1 () > >> OPTION: 1 ( 4) Subnet mask 255.255.255.0 > >> OPTION: 28 ( 4) Broadcast address 10.97.38.255 > >> OPTION: 12 ( 16) Host name centos6-template > >> OPTION: 6 ( 8) DNS server 10.97.38.113,10.97.32.20 > >> OPTION: 3 ( 4) Routers 10.97.38.1 > >> OPTION: 15 ( 17) Domainname cs1cloud.internal > >> > --------------------------------------------------------------------------- > >> > >> TIME: 2013-11-18 08:25:02.015 > >> IP: 0.0.0.0 (6:cc:20:0:0:c) > 255.255.255.255 (ff:ff:ff:ff:ff:ff) > >> OP: 1 (BOOTPREQUEST) > >> HTYPE: 1 (Ethernet) > >> HLEN: 6 > >> HOPS: 0 > >> XID: 91cdcd74 > >> SECS: 0 > >> FLAGS: 0 > >> CIADDR: 0.0.0.0 > >> YIADDR: 0.0.0.0 > >> SIADDR: 0.0.0.0 > >> GIADDR: 0.0.0.0 > >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 > >> SNAME: . > >> FNAME: . > >> OPTION: 53 ( 1) DHCP message type 3 (DHCPREQUEST) > >> OPTION: 54 ( 4) Server identifier 10.97.38.113 > >> OPTION: 50 ( 4) Request IP address 10.97.38.116 > >> OPTION: 12 ( 16) Host name centos6-template > >> OPTION: 55 ( 16) Parameter Request List 1 (Subnet mask) > >> 28 (Broadcast address) > >> 2 (Time offset) > >> 121 (Classless Static Route) > >> 15 (Domainname) > >> 6 (DNS server) > >> 12 (Host name) > >> 40 (NIS domain) > >> 41 (NIS servers) > >> 42 (NTP servers) > >> 26 (Interface MTU) > >> 119 (Domain Search) > >> 3 (Routers) > >> 121 (Classless Static Route) > >> 249 (MSFT - Classless route) > >> 42 (NTP servers) > >> > >> > --------------------------------------------------------------------------- > >> > >> TIME: 2013-11-18 08:25:02.016 > >> IP: 10.97.38.113 (6:5d:1e:0:0:9) > 10.97.38.116 (6:cc:20:0:0:c) > >> OP: 2 (BOOTPREPLY) > >> HTYPE: 1 (Ethernet) > >> HLEN: 6 > >> HOPS: 0 > >> XID: 91cdcd74 > >> SECS: 0 > >> FLAGS: 0 > >> CIADDR: 0.0.0.0 > >> YIADDR: 10.97.38.116 > >> SIADDR: 10.97.38.113 > >> GIADDR: 0.0.0.0 > >> CHADDR: 06:cc:20:00:00:0c:00:00:00:00:00:00:00:00:00:00 > >> SNAME: . > >> FNAME: . > >> OPTION: 53 ( 1) DHCP message type 5 (DHCPACK) > >> OPTION: 54 ( 4) Server identifier 10.97.38.113 > >> OPTION: 51 ( 4) IP address leasetime -1 () > >> OPTION: 1 ( 4) Subnet mask 255.255.255.0 > >> OPTION: 28 ( 4) Broadcast address 10.97.38.255 > >> OPTION: 12 ( 16) Host name centos6-template > >> OPTION: 6 ( 8) DNS server 10.97.38.113,10.97.32.20 > >> OPTION: 3 ( 4) Routers 10.97.38.1 > >> OPTION: 15 ( 17) Domainname cs1cloud.internal > >> > --------------------------------------------------------------------------- > >> > >> root@r-21-VM:~# > >> > >> > >> Best Regards, > >> > >> > >> > >> Adam Scarcella > >> > >> > >> On Sun, Nov 17, 2013 at 2:30 PM, David Nalley <da...@gnsa.us> wrote: > >> > >>> So having seen this specific problem scores of times, it's almost > >>> always network config related. Several potential problems to look out > >>> for: > >>> > >>> 1. Another DHCP server on the same network. (There are ways of > >>> mitigating this, but for the time being, lets just say that > >>> CloudStack's VR should be the only DHCP server on the network. > >>> > >>> 2. Shell into the physical machine that the VR is running on. Using > >>> tcpdump (with filters of course) Do you see the broadcast asking for a > >>> DHCP address coming from the VM on the other physical machine? Do you > >>> see the VR answering the request? If so, move to step 3. If not (for > >>> either of those questions) - you aren't communicating between the two > >>> physical hosts - your switch config is suspect. > >>> > >>> 3. Shell into the physical machine that the VM is running on - You've > >>> seen the response with an IP address assigned go out - so see if it is > >>> seen by the physical host NIC? If not, you've again. If you see the > >>> response (and are sure it's from the VR and not another DHCP server) > >>> come back and tell us - something funky is going on. > >>> > >>> --David > >>> > >>> On Sun, Nov 17, 2013 at 10:11 AM, Adam <adam.scarce...@gmail.com> > wrote: > >>>> Yes, I've disabled the firewall on both KVM hosts in question and > still > >>> no > >>>> dice. I can't even ping the VR from my guest VM, but when I set the > eth0 > >>>> device on the guest to static everything works fine, which makes no > >>> sense > >>>> at all to me. Simply setting the NIC to static allows me see the VR. > >>>> Switching back to DHCP kills it again. I do not understand what is > >>> required > >>>> for the guests to acquire a DHCP lease from the VR. I know that the VR > >>> is > >>>> running dnsmasq, and I've tailed the /var/log/dnsmasq.log for more > info, > >>>> but only see a DHCP request when the guest is on the same KVM host as > >>> the > >>>> VR. > >>>> > >>>> Does anyone know exactly how to troubleshoot this scenario? I'm not > even > >>>> sure what to look for. > >>>> > >>>> -Adam > >>>> > >>>> Best Regards, > >>>> > >>>> > >>>> > >>>> Adam Scarcella > >>>> > >>>> > >>>> On Sat, Nov 16, 2013 at 5:43 PM, Carlos Reátegui <create...@gmail.com > >>>> wrote: > >>>> > >>>>> Did you also check the host firewall? Try disabling. > >>>>> > >>>>>> On Nov 16, 2013, at 1:51 PM, Adam <adam.scarce...@gmail.com> wrote: > >>>>>> > >>>>>> The hosts have static IPs (10.97.38.[10-14]) and can all see and > >>> talk to > >>>>>> each other via IP and hostname. I'm only using a basic zone so no > >>> VLAN > >>>>>> tagging or anything funky like that. > >>>>>> > >>>>>> Best Regards, > >>>>>> > >>>>>> > >>>>>> > >>>>>> Adam Scarcella > >>>>>> > >>>>>> > >>>>>> On Sat, Nov 16, 2013 at 4:17 PM, Andrei Mikhailovsky < > >>> and...@arhont.com > >>>>>> wrote: > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Adam, it sounds like a networking issue, check that all hosts can > >>> talk > >>>>> to > >>>>>>> each other on the same vlan that is used for guest network. I had > >>> the > >>>>> same > >>>>>>> issue with advanced networking when my vlans were not properly > >>> setup on > >>>>> the > >>>>>>> switches. > >>>>>>> > >>>>>>> Andrei > >>>>>>> > >>>>>>> ----- Original Message ----- > >>>>>>> > >>>>>>> From: "Adam" <adam.scarce...@gmail.com> > >>>>>>> To: users@cloudstack.apache.org > >>>>>>> Sent: Saturday, 16 November, 2013 7:08:50 PM > >>>>>>> Subject: Guest VMs not able to acquire DHCP IP from Virtual Router > >>>>> unless > >>>>>>> Guest and VR are on the same KVM host > >>>>>>> > >>>>>>> Hi All, > >>>>>>> > >>>>>>> I have a new and very strange issue that for the life of me I > cannot > >>>>> seem > >>>>>>> to track down and fix. > >>>>>>> > >>>>>>> I have CS 4.2 running on 5 hosts in a simple basic zone. All is > >>> working > >>>>>>> fine, except that my Guest VMs cannot seem to get a DHCP lease from > >>> the > >>>>>>> Virtual Router unless I migrate the Guest VM to the same physical > >>> KVM > >>>>>>> Host running the VR. That would seem to indicate a firewall issue, > >>> but > >>>>> I've > >>>>>>> tested by turning off both the firewalls (VR KVM Host iptables & > >>> Guest > >>>>> VM > >>>>>>> KVM Host iptables). It didn't help. The only way to fix it is to > >>> migrate > >>>>>>> the Guest VM to the same KVM Host that's running the Virtual > Router. > >>>>>>> > >>>>>>> NOTE: The Console Proxy has worked flawlessly this whole time. > >>>>>>> > >>>>>>> So, if a Guest VM starts on a different physical KVM Host, it will > >>> not > >>>>> get > >>>>>>> an internal IP of 169.254.x.x. All along the Console Proxy works > >>> fine. > >>>>> Then > >>>>>>> if I migrate the Guest VM to the same KVM host that's running the > >>> VR, > >>>>> DHCP > >>>>>>> automatically starts working and the Guest VM receives a proper IP > >>>>> address > >>>>>>> of 10.97.38.x. Then I can migrate the Guest VM back to any other > >>>>> physical > >>>>>>> KVM Host and believe it or not, it continues to work flawlessly, > >>> until I > >>>>>>> either reboot the VM or restart the network services. Then it > >>> cannot see > >>>>>>> the VR again and instead receives an internal IP of 169.254.x.x. If > >>> I > >>>>> set a > >>>>>>> static IP address & DNS everything works fine, no matter where the > >>>>> Guest VM > >>>>>>> is running. > >>>>>>> > >>>>>>> Setting static IPs is not an option > >>>>>>> Running all the Guest VMs on the same physical KVM host is not an > >>> option > >>>>>>> > >>>>>>> I desperately need to track down the root cause of this issue so I > >>> can > >>>>>>> release this cloud to my entire department by Monday morning. > >>> Someone > >>>>>>> please help! > >>>>>>> > >>>>>>> Best Regards, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Adam Scarcella > >>>>>>> > >>>>>>> > >>>>> > >>> > >> > >> > >
