Ok, just watch for those createdhcpentry mgmt server logs. Perhaps they're just triggered by you trying to fix the situation by migrating, but the original issue was something else entirely.
On Mon, Mar 16, 2015 at 8:44 AM, Andrija Panic <andrija.pa...@gmail.com> wrote: > I did migrate and also changed accounts, unsucessfully, so some bugs > definitively or my specific setup... > > Thanks, I' fixing this now and will let you know. > > On 16 March 2015 at 16:42, Marcus <shadow...@gmail.com> wrote: > >> Yes, each VM should have at least one default nic, so if there's only >> one nic it should be set to default. Take a db backup first, of >> course, before messing with it. Any idea how it may have happened? Do >> you migrate VMs between networks ever? >> >> On Mon, Mar 16, 2015 at 8:39 AM, Andrija Panic <andrija.pa...@gmail.com> >> wrote: >> > Ok, so if the VM has only 1 VM - and default_nic=0, then I need to change >> > all of them to default_nic=1... ? >> > >> > >> > On 16 March 2015 at 16:38, Marcus <shadow...@gmail.com> wrote: >> > >> >> VMs can have multiple nics and be on multiple networks. If you set a >> >> nic as default, it becomes the network that the vm has its default >> >> route on. Every VM should have a default nic, and if it doesn't I >> >> wonder how it might have happened (maybe a specific combination of >> >> add/delete nic triggered a bug?). You should set a default nic for >> >> every VM that might be missing one, and see if that gets your router >> >> up. >> >> >> >> On Mon, Mar 16, 2015 at 8:34 AM, Andrija Panic <andrija.pa...@gmail.com >> > >> >> wrote: >> >> > Hi Marcus, >> >> > >> >> > Thanks a lot fot hint >> >> > >> >> > True, I have the 0 as the value for some reason in database, for >> couple >> >> of >> >> > NICs.... >> >> > select * from nics where ip4_address like "46.232%" and broadcast_uri >> = >> >> > "vlan://500" and default_nic = 0; >> >> > >> >> > results: http://pastebin.com/rDAe2RY9 >> >> > >> >> > or down there... >> >> > >> >> > This Techvee-FileServer server is already running (still not dead) >> and I >> >> > can see 1 NIC from UI... >> >> > >> >> > Should I reset all of these to 1 ? >> >> > What is the purpose of this field default_nic = 0. >> >> > >> >> > vlan://500 in my case limits results only to the network for this VR >> that >> >> > is having problems... >> >> > >> >> > Any suggestions ? >> >> > >> >> > "id" "uuid" "instance_id" "mac_address" "ip4_address" "netmask" >> "gateway" >> >> > "ip_type" "broadcast_uri" "network_id" "mode" "state" "strategy" >> >> > "reserver_name" "reservation_id" "device_id" "update_time" >> >> "isolation_uri" >> >> > "ip6_address" "default_nic" "vm_type" "created" "removed" >> "ip6_gateway" >> >> > "ip6_cidr" "secondary_ip" "display_nic" >> >> > "2816" "5066bc3a-dbec-4789-aa42-3b9eb8f50bb4" "1795" >> "06:70:0a:00:00:ac" >> >> > "46.232.180.101" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-04 >> >> > 23:06:23" "vlan://500" \N "0" "User" "2015-02-04 20:41:05" "2015-02-04 >> >> > 22:06:23" \N \N "0" "1" >> >> > "3132" "c8a5f98e-5663-40e3-ac03-1ac3545eaa83" "1958" >> "06:fc:c2:00:00:ad" >> >> > "46.232.180.102" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-03 >> >> > 15:45:47" "vlan://500" \N "0" "User" "2015-02-18 15:50:35" "2015-03-03 >> >> > 14:45:47" \N \N "0" "1" >> >> > "3139" "f5a41229-2267-4615-9128-63fbce69bb01" "1962" >> "06:d7:ac:00:00:ae" >> >> > "46.232.180.103" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-02-19 >> >> > 03:10:45" "vlan://500" \N "0" "User" "2015-02-19 00:09:02" "2015-02-19 >> >> > 02:10:45" \N \N "0" "1" >> >> > "707" "99afa70a-39d5-4685-8fc0-9857fdc77c90" "511" "06:b5:72:00:00:72" >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "0" "2014-01-27 >> >> > 14:38:52" "vlan://500" \N "0" "User" "2014-01-27 11:29:08" "2014-01-27 >> >> > 13:38:52" \N \N "0" "1" >> >> > "1580" "bf56315e-b4c3-4338-88d9-3013ab2e2c37" "1088" >> "06:1d:90:00:00:72" >> >> > "46.232.180.144" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23 >> >> > 10:15:18" "vlan://500" \N "0" "User" "2014-07-17 19:14:06" "2014-07-23 >> >> > 08:15:18" \N \N "0" "1" >> >> > "3799" "712cbcb6-097f-4555-a73b-e8c2a5bd557f" "2306" >> "06:33:ac:00:00:77" >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16 >> >> > 12:41:11" "vlan://500" \N "0" "User" "2015-03-16 09:50:25" "2015-03-16 >> >> > 11:41:11" \N \N "0" "1" >> >> > "3817" "3599d144-6cdc-488b-8c9d-5837c7f612ac" "2311" >> "06:8a:ac:00:00:77" >> >> > "46.232.180.149" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16 >> >> > 13:02:59" "vlan://500" \N "0" "User" "2015-03-16 11:49:31" "2015-03-16 >> >> > 12:02:59" \N \N "0" "1" >> >> > "1581" "47cfe113-218c-4a60-a108-584d64cd16ed" "1089" >> "06:c2:c4:00:00:7a" >> >> > "46.232.180.152" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23 >> >> > 10:16:12" "vlan://500" \N "0" "User" "2014-07-17 19:14:43" "2014-07-23 >> >> > 08:16:12" \N \N "0" "1" >> >> > "1582" "dae4b3cf-e2b2-4cbc-960e-77d2e854fffa" "1090" >> "06:1c:78:00:00:7c" >> >> > "46.232.180.154" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23 >> >> > 10:16:32" "vlan://500" \N "0" "User" "2014-07-17 19:30:44" "2014-07-23 >> >> > 08:16:32" \N \N "0" "1" >> >> > "435" "5c370395-2b70-4c8c-b710-eab4151b14ab" "252" "06:96:4e:00:00:8e" >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-01-13 >> >> > 23:25:44" "vlan://500" \N "0" "User" "2013-10-24 23:16:11" "2014-01-13 >> >> > 22:25:44" \N \N "0" "1" >> >> > "2820" "feee7c33-c723-41e3-8583-bcbfc4e257b1" "1797" >> "06:be:24:00:00:8e" >> >> > "46.232.180.172" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-04 >> >> 22:54:29" >> >> > "vlan://500" \N "0" "User" "2015-02-04 21:54:29" \N \N \N "0" "1" >> >> > "1813" "df725af8-12ed-4c27-b521-51d1d465326d" "1213" >> "06:cd:d6:00:00:97" >> >> > "46.232.180.181" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-09-02 >> >> > 11:06:56" "vlan://500" \N "0" "User" "2014-09-02 09:03:11" "2014-09-02 >> >> > 09:06:56" \N \N "0" "1" >> >> > "2897" "aab5d282-e9d5-4c4f-91c9-bfae88f404d6" "1304" >> "06:6e:0c:00:00:9c" >> >> > "46.232.180.186" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "0" "2015-02-11 >> >> 14:32:00" >> >> > "vlan://500" \N "0" "User" "2015-02-11 13:31:59" \N \N \N "0" "1" >> >> > "2235" "30865e2e-083e-4e8a-bccf-c24bf403dece" "1488" >> "06:b8:b4:00:00:a1" >> >> > "46.232.180.191" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-12-02 >> >> > 14:58:11" "vlan://500" \N "0" "User" "2014-12-02 13:31:08" "2014-12-02 >> >> > 13:58:11" \N \N "0" "1" >> >> > "2642" "233d03e4-360f-4bbe-8b9c-882468bc9c7b" "1706" >> "06:f8:36:00:00:a7" >> >> > "46.232.180.197" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-01-21 >> >> > 19:46:10" "vlan://500" \N "0" "User" "2015-01-21 17:27:13" "2015-01-21 >> >> > 18:46:10" \N \N "0" "1" >> >> > "1631" "54ee2677-c502-489d-bdd2-44e7d4937020" "1102" >> "06:ac:4e:00:00:54" >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23 >> >> > 12:33:59" "vlan://500" \N "0" "User" "2014-07-23 09:47:24" "2014-07-23 >> >> > 10:33:59" \N \N "0" "1" >> >> > "1634" "db10636e-458a-40b2-8789-2bb1778ff50a" "1108" >> "06:46:d8:00:00:54" >> >> > "46.232.180.205" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2014-07-23 >> >> > 16:54:11" "vlan://500" \N "0" "User" "2014-07-23 10:41:03" "2014-07-23 >> >> > 14:54:11" \N \N "0" "1" >> >> > "204" "8bd10db8-cb45-4e43-8d0a-9319be5b29be" "102" "06:7c:54:00:00:35" >> >> > "46.232.180.222" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Allocated" "Create" "DirectNetworkGuru" \N "2" "2015-03-16 >> >> 12:25:41" >> >> > "vlan://500" \N "0" "User" "2013-03-08 14:54:59" \N \N \N "0" "1" >> >> > "3814" "4267a186-2908-4fbe-9ca4-dbc654c0d37c" "2312" >> "06:c9:e0:00:00:3a" >> >> > "46.232.180.227" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2015-03-16 >> >> > 13:03:42" "vlan://500" \N "0" "User" "2015-03-16 11:46:38" "2015-03-16 >> >> > 12:03:42" \N \N "0" "1" >> >> > "29" "35f48fcb-722f-49f5-9a6d-584f92cb197b" "14" "06:b1:30:00:00:2a" >> >> > "46.232.180.241" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "206" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11 >> >> > 02:11:18" "vlan://500" \N "0" "User" "2013-01-11 01:06:41" "2013-01-11 >> >> > 01:11:18" \N \N "0" "1" >> >> > "25" "674ded1f-f705-45cc-a538-611aa2b1ea48" "12" "06:12:88:00:00:2b" >> >> > "46.232.180.242" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "206" >> >> > "Dhcp" "Deallocating" "Create" "DirectNetworkGuru" \N "1" "2013-01-11 >> >> > 01:35:20" "vlan://500" \N "0" "User" "2013-01-11 00:28:40" "2013-01-11 >> >> > 00:35:20" \N \N "0" "1" >> >> > "2953" "51002a53-0500-4732-85db-111804ac5f40" "1871" >> "06:02:a8:00:00:e2" >> >> > "46.232.180.39" "255.255.255.0" "46.232.180.1" "Ip4" "vlan://500" >> "212" >> >> > "Dhcp" "Reserved" "Create" "DirectNetworkGuru" \N "1" "2015-02-12 >> >> 14:16:23" >> >> > "vlan://500" \N "0" "User" "2015-02-12 13:16:20" \N \N \N "0" "1" >> >> > >> >> > >> >> > >> >> > >> >> > On 16 March 2015 at 16:11, Marcus <shadow...@gmail.com> wrote: >> >> > >> >> >> FWIW, if your 4.3.2 is the same as what's in the source tree for the >> >> >> router code, the null pointer indicates that there's no default nic >> >> >> for one of your guests "Techvee-FileServer". I'd guess that if you >> >> >> delete/move just that host you may be able to start the router, or at >> >> >> least get past this. You can also look into the db and see if you can >> >> >> find its nics in the cloud.nics table and see if there is in fact one >> >> >> marked default. >> >> >> >> >> >> On Mon, Mar 16, 2015 at 8:05 AM, Marcus <shadow...@gmail.com> wrote: >> >> >> > Looks like the issue is that null pointer in CreateDhcpEntry for >> >> >> > either "Techvee-FileServer" or the DHCP entry immediately after >> that. >> >> >> > It would suggest some inconsistent/unexpected data when creating a >> >> >> > DHCP entry for one of the guests serviced by this router. It's too >> bad >> >> >> > that one bad entry is fatal for the whole router. >> >> >> > >> >> >> > >> >> >> > On Mon, Mar 16, 2015 at 7:16 AM, Andrija Panic < >> >> andrija.pa...@gmail.com> >> >> >> wrote: >> >> >> >> Not really - we are painfully migrating stopped VMs, from VPS >> network >> >> >> >> (Guest Shared netwotk) to VPCs... >> >> >> >> >> >> >> >> MGMT server sends the STOP command to agent, even though the VM >> was >> >> >> never >> >> >> >> started, BUT the storage provisioning from template to volume is >> >> done... >> >> >> >> >> >> >> >> We are also looking into some external help as we speak... >> >> >> >> >> >> >> >> On 16 March 2015 at 14:52, Nux! <n...@li.nux.ro> wrote: >> >> >> >> >> >> >> >>> Hi, >> >> >> >>> >> >> >> >>> Have you managed to get to the bottom of this? >> >> >> >>> >> >> >> >>> -- >> >> >> >>> Sent from the Delta quadrant using Borg technology! >> >> >> >>> >> >> >> >>> Nux! >> >> >> >>> www.nux.ro >> >> >> >>> >> >> >> >>> ----- Original Message ----- >> >> >> >>> > From: "Andrija Panic" <andrija.pa...@gmail.com> >> >> >> >>> > To: us...@cloudstack.apache.org, dev@cloudstack.apache.org >> >> >> >>> > Sent: Sunday, 15 March, 2015 16:19:07 >> >> >> >>> > Subject: [URGENT - HELP NEEDED] >> >> >> >>> >> >> >> >>> > Hi guys, >> >> >> >>> > >> >> >> >>> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated >> >> right >> >> >> >>> before >> >> >> >>> > that, from CentOS 6.5 to CentOS 6.6) >> >> >> >>> > >> >> >> >>> > And now I can not start SYSTEM VR - that is used for SHARED >> GUEST >> >> >> network >> >> >> >>> > anymore. >> >> >> >>> > And some VMs are down - and cant be started because they >> depend on >> >> >> this >> >> >> >>> > VR... >> >> >> >>> > >> >> >> >>> > VPC VRs are created fine, so new VR for VPC are created fine, >> but >> >> >> this >> >> >> >>> one >> >> >> >>> > fro Guest network fails to start: >> >> >> >>> > >> >> >> >>> > Here you can see, after agent copies template from secondary >> >> >> storage, to >> >> >> >>> > primary local storage, it created base image, and backing file >> - >> >> so >> >> >> >>> storage >> >> >> >>> > setup seems completed. >> >> >> >>> > >> >> >> >>> > Than all out of sudden we have errors: >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> >> >> >> Andrija Panić >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > >> >> > Andrija Panić >> >> >> > >> > >> > >> > -- >> > >> > Andrija Panić >> > > > > -- > > Andrija Panić