[Openstack] After deleted an instance, nobody can create any instance.

[Sam Su]

Hi,

I have an Essex cluster with 6 compute nodes and one control node, and it
worked fine in the past several months. Last week, since someone deleted an
instance in a project, no new instance can be launched in that project.

When create a new instance in the project, it displays:
*Error: *An error occurred. Please try again.

the following error info will come up in the file nova-api.log:
2012-08-28 19:14:23 ERROR nova.api.openstack
[req-6fb211c1-60cf-4a69-83e1-f9e7da30b458 6023cea36f784448b667922894ca7102
afdee06258774b2d9e768c08b62dbbf2] Caught error: Remote error:
InstanceNotFound Instance 603 could not be found.

Here is the info of nova-api.log :
http://pastebin.com/evuDMdrU
.

Actually this instance with id 603 has been deleted, it's very strange.
Does anyone has any idea about this issue?

By the way, if an existing instance in another project is terminated, the
same problem occurs in that project.

Thanks ahead,
Sam
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] After deleted an instance, nobody can create any instance.

[Emilien Macchi]

Hi,

I think the best way is to delete the instance from the database :
Look at the "nova" database and "fixed_ips" table.

If you have the old instance number here, try put the "instance_id" filed
into NULL where instance_id is 603.

If you have still this issue, find every field corresponding to 603 into
the database and delete it.

Restart nova-api service, and that should be good.


Good luck !


Regards

On Wed, Aug 29, 2012 at 4:44 AM, Sam Su  wrote:

> Hi,
>
> I have an Essex cluster with 6 compute nodes and one control node, and it
> worked fine in the past several months. Last week, since someone deleted an
> instance in a project, no new instance can be launched in that project.
>
> When create a new instance in the project, it displays:
> *Error: *An error occurred. Please try again.
>
> the following error info will come up in the file nova-api.log:
> 2012-08-28 19:14:23 ERROR nova.api.openstack
> [req-6fb211c1-60cf-4a69-83e1-f9e7da30b458 6023cea36f784448b667922894ca7102
> afdee06258774b2d9e768c08b62dbbf2] Caught error: Remote error:
> InstanceNotFound Instance 603 could not be found.
>
> Here is the info of nova-api.log :
> http://pastebin.com/evuDMdrU
> .
>
> Actually this instance with id 603 has been deleted, it's very strange.
> Does anyone has any idea about this issue?
>
> By the way, if an existing instance in another project is terminated, the
> same problem occurs in that project.
>
> Thanks ahead,
> Sam
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Emilien Macchi
*System Engineer*
*www.stackops.com

| *emilien.mac...@stackops.com**  *|* skype:emilien.macchi*
* 
*

*

 ADVERTENCIA LEGAL 
Le informamos, como destinatario de este mensaje, que el correo electrónico
y las comunicaciones por medio de Internet no permiten asegurar ni
garantizar la confidencialidad de los mensajes transmitidos, así como
tampoco su integridad o su correcta recepción, por lo que STACKOPS
TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias.
Si no consintiese en la utilización del correo electrónico o de las
comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro
conocimiento de manera inmediata. Este mensaje va dirigido, de manera
exclusiva, a su destinatario y contiene información confidencial y sujeta
al secreto profesional, cuya divulgación no está permitida por la ley. En
caso de haber recibido este mensaje por error, le rogamos que, de forma
inmediata, nos lo comunique mediante correo electrónico remitido a nuestra
atención y proceda a su eliminación, así como a la de cualquier documento
adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o
utilización de este mensaje, o de cualquier documento adjunto al mismo,
cualquiera que fuera su finalidad, están prohibidas por la ley.

* PRIVILEGED AND CONFIDENTIAL 
We hereby inform you, as addressee of this message, that e-mail and
Internet do not guarantee the confidentiality, nor the completeness or
proper reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L.
does not assume any liability for those circumstances. Should you not agree
to the use of e-mail or to communications via Internet, you are kindly
requested to notify us immediately. This message is intended exclusively
for the person to whom it is addressed and contains privileged and
confidential information protected from disclosure by law. If you are not
the addressee indicated in this message, you should immediately delete it
and any attachments and notify the sender by reply e-mail. In such case,
you are hereby notified that any dissemination, distribution, copying or
use of this message or any attachments, for any purpose, is strictly
prohibited by law.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] After deleted an instance, nobody can create any instance.

[Trinath Somanchi]

Hi-

A solution would be deleting the instance table record.

Instances table in database has two relational tables.

[1] security_group_instance_association
[2] instance_info_caches
along with the tuples from instances table.

-
TNS


On Wed, Aug 29, 2012 at 11:00 AM, Emilien Macchi <
emilien.mac...@stackops.com> wrote:

> Hi,
>
> I think the best way is to delete the instance from the database :
> Look at the "nova" database and "fixed_ips" table.
>
> If you have the old instance number here, try put the "instance_id" filed
> into NULL where instance_id is 603.
>
> If you have still this issue, find every field corresponding to 603 into
> the database and delete it.
>
> Restart nova-api service, and that should be good.
>
>
> Good luck !
>
>
> Regards
>
> On Wed, Aug 29, 2012 at 4:44 AM, Sam Su  wrote:
>
>> Hi,
>>
>> I have an Essex cluster with 6 compute nodes and one control node, and it
>> worked fine in the past several months. Last week, since someone deleted an
>> instance in a project, no new instance can be launched in that project.
>>
>> When create a new instance in the project, it displays:
>> *Error: *An error occurred. Please try again.
>>
>> the following error info will come up in the file nova-api.log:
>> 2012-08-28 19:14:23 ERROR nova.api.openstack
>> [req-6fb211c1-60cf-4a69-83e1-f9e7da30b458 6023cea36f784448b667922894ca7102
>> afdee06258774b2d9e768c08b62dbbf2] Caught error: Remote error:
>> InstanceNotFound Instance 603 could not be found.
>>
>> Here is the info of nova-api.log :
>> http://pastebin.com/evuDMdrU
>> .
>>
>> Actually this instance with id 603 has been deleted, it's very strange.
>> Does anyone has any idea about this issue?
>>
>> By the way, if an existing instance in another project is terminated, the
>> same problem occurs in that project.
>>
>> Thanks ahead,
>> Sam
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Emilien Macchi
> *System Engineer*
> *www.stackops.com
>
> | *emilien.mac...@stackops.com**  *|* skype:emilien.macchi*
> * 
> *
>
> *
>
>  ADVERTENCIA LEGAL 
> Le informamos, como destinatario de este mensaje, que el correo
> electrónico y las comunicaciones por medio de Internet no permiten asegurar
> ni garantizar la confidencialidad de los mensajes transmitidos, así como
> tampoco su integridad o su correcta recepción, por lo que STACKOPS
> TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias.
> Si no consintiese en la utilización del correo electrónico o de las
> comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro
> conocimiento de manera inmediata. Este mensaje va dirigido, de manera
> exclusiva, a su destinatario y contiene información confidencial y sujeta
> al secreto profesional, cuya divulgación no está permitida por la ley. En
> caso de haber recibido este mensaje por error, le rogamos que, de forma
> inmediata, nos lo comunique mediante correo electrónico remitido a nuestra
> atención y proceda a su eliminación, así como a la de cualquier documento
> adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o
> utilización de este mensaje, o de cualquier documento adjunto al mismo,
> cualquiera que fuera su finalidad, están prohibidas por la ley.
>
> * PRIVILEGED AND CONFIDENTIAL 
> We hereby inform you, as addressee of this message, that e-mail and
> Internet do not guarantee the confidentiality, nor the completeness or
> proper reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L.
> does not assume any liability for those circumstances. Should you not agree
> to the use of e-mail or to communications via Internet, you are kindly
> requested to notify us immediately. This message is intended exclusively
> for the person to whom it is addressed and contains privileged and
> confidential information protected from disclosure by law. If you are not
> the addressee indicated in this message, you should immediately delete it
> and any attachments and notify the sender by reply e-mail. In such case,
> you are hereby notified that any dissemination, distribution, copying or
> use of this message or any attachments, for any purpose, is strictly
> prohibited by law.
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Regards,
--
Trinath Somanchi,
+91 9866 235 130
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launc

Re: [Openstack] After deleted an instance, nobody can create any instance.

[Sam Su]

Thank you so much for you guys help.
I will do these later.

Actually this problem can be reproduced in my Essex environment, and I have
no idea about how to figure out why it happened, but I'd like to try it. It
is highly appreciated if someone can give me some suggestions.

Thanks,
Sam

On Tue, Aug 28, 2012 at 11:08 PM, Trinath Somanchi <
trinath.soman...@gmail.com> wrote:

> Hi-
>
> A solution would be deleting the instance table record.
>
> Instances table in database has two relational tables.
>
> [1] security_group_instance_association
> [2] instance_info_caches
> along with the tuples from instances table.
>
> -
> TNS
>
>
> On Wed, Aug 29, 2012 at 11:00 AM, Emilien Macchi <
> emilien.mac...@stackops.com> wrote:
>
>> Hi,
>>
>> I think the best way is to delete the instance from the database :
>> Look at the "nova" database and "fixed_ips" table.
>>
>> If you have the old instance number here, try put the "instance_id" filed
>> into NULL where instance_id is 603.
>>
>> If you have still this issue, find every field corresponding to 603 into
>> the database and delete it.
>>
>> Restart nova-api service, and that should be good.
>>
>>
>> Good luck !
>>
>>
>> Regards
>>
>> On Wed, Aug 29, 2012 at 4:44 AM, Sam Su  wrote:
>>
>>> Hi,
>>>
>>> I have an Essex cluster with 6 compute nodes and one control node, and
>>> it worked fine in the past several months. Last week, since someone deleted
>>> an instance in a project, no new instance can be launched in that project.
>>>
>>> When create a new instance in the project, it displays:
>>> *Error: *An error occurred. Please try again.
>>>
>>> the following error info will come up in the file nova-api.log:
>>> 2012-08-28 19:14:23 ERROR nova.api.openstack
>>> [req-6fb211c1-60cf-4a69-83e1-f9e7da30b458 6023cea36f784448b667922894ca7102
>>> afdee06258774b2d9e768c08b62dbbf2] Caught error: Remote error:
>>> InstanceNotFound Instance 603 could not be found.
>>>
>>> Here is the info of nova-api.log :
>>> http://pastebin.com/evuDMdrU
>>> .
>>>
>>> Actually this instance with id 603 has been deleted, it's very strange.
>>> Does anyone has any idea about this issue?
>>>
>>> By the way, if an existing instance in another project is terminated,
>>> the same problem occurs in that project.
>>>
>>> Thanks ahead,
>>> Sam
>>>
>>> ___
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to : openstack@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>>
>> --
>> Emilien Macchi
>> *System Engineer*
>> *www.stackops.com
>>
>> | *emilien.mac...@stackops.com**  *|* skype:emilien.macchi*
>> * 
>> *
>>
>> *
>>
>>  ADVERTENCIA LEGAL 
>> Le informamos, como destinatario de este mensaje, que el correo
>> electrónico y las comunicaciones por medio de Internet no permiten asegurar
>> ni garantizar la confidencialidad de los mensajes transmitidos, así como
>> tampoco su integridad o su correcta recepción, por lo que STACKOPS
>> TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias.
>> Si no consintiese en la utilización del correo electrónico o de las
>> comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro
>> conocimiento de manera inmediata. Este mensaje va dirigido, de manera
>> exclusiva, a su destinatario y contiene información confidencial y sujeta
>> al secreto profesional, cuya divulgación no está permitida por la ley. En
>> caso de haber recibido este mensaje por error, le rogamos que, de forma
>> inmediata, nos lo comunique mediante correo electrónico remitido a nuestra
>> atención y proceda a su eliminación, así como a la de cualquier documento
>> adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o
>> utilización de este mensaje, o de cualquier documento adjunto al mismo,
>> cualquiera que fuera su finalidad, están prohibidas por la ley.
>>
>> * PRIVILEGED AND CONFIDENTIAL 
>> We hereby inform you, as addressee of this message, that e-mail and
>> Internet do not guarantee the confidentiality, nor the completeness or
>> proper reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L.
>> does not assume any liability for those circumstances. Should you not agree
>> to the use of e-mail or to communications via Internet, you are kindly
>> requested to notify us immediately. This message is intended exclusively
>> for the person to whom it is addressed and contains privileged and
>> confidential information protected from disclosure by law. If you are not
>> the addressee indicated in this message, you should immediately delete it
>> and any attachments and notify the sender by reply e-mail. In such case,
>> you are hereby notified that any dissemination, distribution, copying or
>> use of this message or any attachments, for any purpose, is strictly
>> prohibited by law.
>>
>>
>> _

Re: [Openstack] After deleted an instance, nobody can create any instance.

[Sam Su]

Hi,

Thank you all for you suggestions.
After updated the table fixed_ips and floating_ips, this problem was
eventually fixed.

I finally figure out how to reproduce this problem:
If an instance associated with a floating IP is terminated, the instance
can be successfully terminated, but the floating IP is also associated with
the deleted instance in database.

Also this is a known issue:https://bugs.launchpad.net/nova/+bug/997763

If someone can make a backport for Essex, that will be great.

Thanks,
Sam



On Tue, Aug 28, 2012 at 10:30 PM, Emilien Macchi <
emilien.mac...@stackops.com> wrote:

> Hi,
>
> I think the best way is to delete the instance from the database :
> Look at the "nova" database and "fixed_ips" table.
>
> If you have the old instance number here, try put the "instance_id" filed
> into NULL where instance_id is 603.
>
> If you have still this issue, find every field corresponding to 603 into
> the database and delete it.
>
> Restart nova-api service, and that should be good.
>
>
> Good luck !
>
>
> Regards
>
>  On Wed, Aug 29, 2012 at 4:44 AM, Sam Su  wrote:
>
>>  Hi,
>>
>> I have an Essex cluster with 6 compute nodes and one control node, and it
>> worked fine in the past several months. Last week, since someone deleted an
>> instance in a project, no new instance can be launched in that project.
>>
>> When create a new instance in the project, it displays:
>> *Error: *An error occurred. Please try again.
>>
>> the following error info will come up in the file nova-api.log:
>> 2012-08-28 19:14:23 ERROR nova.api.openstack
>> [req-6fb211c1-60cf-4a69-83e1-f9e7da30b458 6023cea36f784448b667922894ca7102
>> afdee06258774b2d9e768c08b62dbbf2] Caught error: Remote error:
>> InstanceNotFound Instance 603 could not be found.
>>
>> Here is the info of nova-api.log :
>> http://pastebin.com/evuDMdrU
>> .
>>
>> Actually this instance with id 603 has been deleted, it's very strange.
>> Does anyone has any idea about this issue?
>>
>> By the way, if an existing instance in another project is terminated, the
>> same problem occurs in that project.
>>
>> Thanks ahead,
>> Sam
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Emilien Macchi
> *System Engineer*
> *www.stackops.com
>
> | *emilien.mac...@stackops.com**  *|* skype:emilien.macchi*
> * 
> *
>
> *
>
>  ADVERTENCIA LEGAL 
> Le informamos, como destinatario de este mensaje, que el correo
> electrónico y las comunicaciones por medio de Internet no permiten asegurar
> ni garantizar la confidencialidad de los mensajes transmitidos, así como
> tampoco su integridad o su correcta recepción, por lo que STACKOPS
> TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias.
> Si no consintiese en la utilización del correo electrónico o de las
> comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro
> conocimiento de manera inmediata. Este mensaje va dirigido, de manera
> exclusiva, a su destinatario y contiene información confidencial y sujeta
> al secreto profesional, cuya divulgación no está permitida por la ley. En
> caso de haber recibido este mensaje por error, le rogamos que, de forma
> inmediata, nos lo comunique mediante correo electrónico remitido a nuestra
> atención y proceda a su eliminación, así como a la de cualquier documento
> adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o
> utilización de este mensaje, o de cualquier documento adjunto al mismo,
> cualquiera que fuera su finalidad, están prohibidas por la ley.
>
> * PRIVILEGED AND CONFIDENTIAL 
> We hereby inform you, as addressee of this message, that e-mail and
> Internet do not guarantee the confidentiality, nor the completeness or
> proper reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L.
> does not assume any liability for those circumstances. Should you not agree
> to the use of e-mail or to communications via Internet, you are kindly
> requested to notify us immediately. This message is intended exclusively
> for the person to whom it is addressed and contains privileged and
> confidential information protected from disclosure by law. If you are not
> the addressee indicated in this message, you should immediately delete it
> and any attachments and notify the sender by reply e-mail. In such case,
> you are hereby notified that any dissemination, distribution, copying or
> use of this message or any attachments, for any purpose, is strictly
> prohibited by law.
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHe