I've seen issues where the quota_use table in the nova database gets out of sync with the resources actually used.
try this query and see if the fixed_ips matches what is in use: mysql nova -e 'select * from quota_usages where project_id="60d776fe573f44a4810cb294b95e09d6" and resource="fixed_ips"; if it doesn't you'll need to manually update it and check the other resources for this tenant (if one is wrong usually they all are) I was just ranting privately after running into this issue today (not for the first time) about how having this in its own table is bad for these reasons & that I'd rather have the right answer slowly than the wrong answer quickly. Though this may not be your problem & I haven't yet reproduced the issue to see if I can file a proper bug (a previous bug related to this was fixed a long time ago) -Jon On Wed, Dec 11, 2013 at 2:11 PM, JR <botem...@gmail.com> wrote: > Hi all, > > I'm having an odd issue where I can't spin up new VMs; the error is below. > I've confirmed that the tenant starting the instance has sufficient > capacity. > I have another tenant who is running a similar number of VMs and it's not > having the problem. > > This is ubuntu 12.04 running grizzly. > > Any thoughts? I assume that something in the DB is wedged ... > > Thanks > JR > > root@nebula01:/mnt/gdm_source# su - dd_test -c 'nova quota-show' > +-----------------------------+--------+ > | Property | Value | > +-----------------------------+--------+ > | metadata_items | 128 | > | injected_file_content_bytes | 10240 | > | ram | 256000 | > | floating_ips | 128 | > | key_pairs | 100 | > | instances | 128 | > | security_group_rules | 20 | > | injected_files | 5 | > | cores | 128 | > | fixed_ips | 128 | > | injected_file_path_bytes | 255 | > | security_groups | 50 | > +-----------------------------+--------+ > > root@nebula01:/mnt/gdm_source# su - dd_test -c 'nova show gdm-worker17-t' > +-----------------------------+---------------------------------------------------------------------------------+ > | Property | Value > | > +-----------------------------+---------------------------------------------------------------------------------+ > | status | ERROR > | > | updated | 2013-12-11T19:05:15Z > | > | OS-EXT-STS:task_state | None > | > | key_name | dd_test > | > | image | gdm-worker > (fe0147b8-d355-4d7e-826f-195ba97d7594) | > | hostId | > 5f1a4bd11ee027f10f8fc37b0e2ce091904b94befdf5abeb196c7f67 > | > | OS-EXT-STS:vm_state | error > | > | public network | > | > | flavor | i3.small_d10g > (7d1037f1-dfd0-4a0e-8da7-097258c9934d) | > | id | d7b9499b-7cc4-4d2d-80f1-50b1e4a7fc85 > | > | security_groups | [{u'name': u'dd_test'}] > | > | user_id | dcc3374a58984a92981d09f27c76a869 > | > | name | gdm-worker17-t > | > | created | 2013-12-11T19:05:10Z > | > | fault | {u'message': u'NoValidHost', u'code': 500, > u'created': u'2013-12-11T19:05:15Z'} | > | OS-DCF:diskConfig | MANUAL > | > | metadata | {} > | > | accessIPv4 | > | > | accessIPv6 | > | > | tenant_id | 60d776fe573f44a4810cb294b95e09d6 > | > | OS-EXT-STS:power_state | 0 > | > | OS-EXT-AZ:availability_zone | nova > | > | config_drive | > | > +-----------------------------+---------------------------------------------------------------------------------+ > root@nebula01:/mnt/gdm_source# keystone tenant-list|grep 60d > | 60d776fe573f44a4810cb294b95e09d6 | dd_test | True | > > > Error: > > 2013-12-11 12:05:15.153 ERROR nova.compute.manager > [req-2363d68e-a947-41af-9359-2a6e6720a0ee dcc3374a58984a92981d09f27c76a869 > 60d776fe573f44a4810cb294b95e09d6] [instance: > d7b9499b-7cc4-4d2d-80f1-50b1e4a7fc85] Error: ['Traceback (most recent call > last):\n', ' File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 847, in > _run_instance\n requested_networks, macs, security_groups)\n', ' File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1091, in > _allocate_network\n instance=instance)\n', ' File > "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n > self.gen.next()\n', ' File > "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1087, in > _allocate_network\n security_groups=security_groups)\n', ' File > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 102, in > wrapped\n return func(self, context, *args, **kwargs)\n', ' File > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 47, in > wrapper\n res = f(self, context, *args, **kwargs)\n', ' File > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 283, in > allocate_for_instance\n nw_info = > self.network_rpcapi.allocate_for_instance(context, **args)\n', ' File > "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 171, in > allocate_for_instance\n topic=topic, version=\'1.9\')\n', ' File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line > 80, in call\n return rpc.call(context, self._get_topic(topic), msg, > timeout)\n', ' File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", > line 140, in call\n return _get_impl().call(CONF, context, topic, msg, > timeout)\n', ' File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", > line 798, in call\n rpc_amqp.get_connection_pool(conf, Connection))\n', ' > File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", > line 612, in call\n rv = list(rv)\n', ' File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line > 561, in __iter__\n raise result\n', 'FixedIpLimitExceeded_Remote: Maximum > number of fixed ips exceeded\nTraceback (most recent call last):\n\n File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line > 430, in _process_data\n rval = self.proxy.dispatch(ctxt, version, method, > **args)\n\n File > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", > line 133, in dispatch\n return getattr(proxyobj, method)(ctxt, > **kwargs)\n\n File > "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 116, > in allocate_for_instance\n **kwargs)\n\n File > "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 513, in > allocate_for_instance\n requested_networks=requested_networks)\n\n File > "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 217, in > _allocate_fixed_ips\n vpn=vpn, address=address)\n\n File > "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 836, in > allocate_fixed_ip\n raise > exception.FixedIpLimitExceeded()\n\nFixedIpLimitExceeded: Maximum number of > fixed ips exceeded\n\n'] > > -- > Your electronic communications are being monitored; strong encryption is an > answer. My public key > > _______________________________________________ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack