Public bug reported:
1. Initially there were no spare VMs since the “spare_amphora_pool_size =
0”
.
[house_keeping]
# Pool size for the spare pool
spare_amphora_pool_size = 0
stack@hlm:~/scratch/ansible/next/hos/ansible$ nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option
will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| ID | Name
| Tenant ID | Status | Task State | Power State |
Networks |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running |
n1=4.5.6.5 |
| 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running |
n1=4.5.6.6 |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
2. Change the spare pool size to 1 and restart Octavia-housekeeping
service. Spare Amphora VM gets created as below.
stack@hlm:~/scratch/ansible/next/hos/ansible$ nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option
will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| ID | Name
| Tenant ID | Status | Task State | Power State |
Networks |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| 6a1101cd-d9d3-4c8e-aa1d-0790f7f4ac8b |
amphora-18f4d90f-fe6e-4085-851e-7571cba0c65a | a5e6e87d402847e7b4210e035a0fceec
| ACTIVE | - | Running | OCTAVIA-MGMT-NET=100.74.25.13 |
| 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running |
n1=4.5.6.5 |
| 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running |
n1=4.5.6.6 |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
3. Now change the spare pool size to 0 and restart Octavia-
housekeeping service. Spare Amphora VM does not get deleted.
stack@hlm:~/scratch/ansible/next/hos/ansible$ nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option
will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| ID | Name
| Tenant ID | Status | Task State | Power State |
Networks |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| 6a1101cd-d9d3-4c8e-aa1d-0790f7f4ac8b |
amphora-18f4d90f-fe6e-4085-851e-7571cba0c65a | a5e6e87d402847e7b4210e035a0fceec
| ACTIVE | - | Running | OCTAVIA-MGMT-NET=100.74.25.13 |
| 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running |
n1=4.5.6.5 |
| 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running |
n1=4.5.6.6 |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
stack@hlm:~/scratch/ansible/next/hos/ansible$
4. I see the requirement is not satisfied as per the config so
hosue keeping should take care of cleaning up the spare amphora VM.But
in the logs I see the statement “ requirement is satisfied “.
In the logs
2016-08-22 06:37:56.901 63471 DEBUG octavia.cmd.house_keeping [-] Initiating
the cleanup of old amphora... db_cleanup
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:64
2016-08-22 06:37:57.194 63471 DEBUG octavia.cmd.house_keeping [-] Initiating
spare amphora check... spare_amphora_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:49
2016-08-22 06:37:57.203 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Required Spare Amphora count
: 0 spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:45
2016-08-22 06:37:57.204 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Current Spare Amphora count :
1 spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:46
2016-08-22 06:37:57.204 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Current spare amphora count
satisfies the requirement spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:60
2016-08-22 06:37:56.901 63471 DEBUG octavia.cmd.house_keeping [-] Initiating
the cleanup of old amphora... db_cleanup
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:64
2016-08-22 06:37:57.194 63471 DEBUG octavia.cmd.house_keeping [-] Initiating
spare amphora check... spare_amphora_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:49
2016-08-22 06:37:57.203 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Required Spare Amphora count
: 0 spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:45
2016-08-22 06:37:57.204 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Current Spare Amphora count :
1 spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:46
2016-08-22 06:37:57.204 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Current spare amphora count
satisfies the requirement spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:60
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1615502
Title:
LBAAS - housekeeping serive does not cleanup stale amphora VMs
Status in neutron:
New
Bug description:
1. Initially there were no spare VMs since the “spare_amphora_pool_size =
0”
.
[house_keeping]
# Pool size for the spare pool
spare_amphora_pool_size = 0
stack@hlm:~/scratch/ansible/next/hos/ansible$ nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this
option will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| ID | Name
| Tenant ID | Status | Task State | Power State
| Networks |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running
| n1=4.5.6.5 |
| 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running
| n1=4.5.6.6 |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
2. Change the spare pool size to 1 and restart Octavia-
housekeeping service. Spare Amphora VM gets created as below.
stack@hlm:~/scratch/ansible/next/hos/ansible$ nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this
option will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| ID | Name
| Tenant ID | Status | Task State | Power State
| Networks |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| 6a1101cd-d9d3-4c8e-aa1d-0790f7f4ac8b |
amphora-18f4d90f-fe6e-4085-851e-7571cba0c65a | a5e6e87d402847e7b4210e035a0fceec
| ACTIVE | - | Running | OCTAVIA-MGMT-NET=100.74.25.13 |
| 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running
| n1=4.5.6.5 |
| 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running
| n1=4.5.6.6 |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
3. Now change the spare pool size to 0 and restart Octavia-
housekeeping service. Spare Amphora VM does not get deleted.
stack@hlm:~/scratch/ansible/next/hos/ansible$ nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this
option will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| ID | Name
| Tenant ID | Status | Task State | Power State
| Networks |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| 6a1101cd-d9d3-4c8e-aa1d-0790f7f4ac8b |
amphora-18f4d90f-fe6e-4085-851e-7571cba0c65a | a5e6e87d402847e7b4210e035a0fceec
| ACTIVE | - | Running | OCTAVIA-MGMT-NET=100.74.25.13 |
| 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running
| n1=4.5.6.5 |
| 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2
| d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running
| n1=4.5.6.6 |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
stack@hlm:~/scratch/ansible/next/hos/ansible$
4. I see the requirement is not satisfied as per the config so
hosue keeping should take care of cleaning up the spare amphora VM.But
in the logs I see the statement “ requirement is satisfied “.
In the logs
2016-08-22 06:37:56.901 63471 DEBUG octavia.cmd.house_keeping [-] Initiating
the cleanup of old amphora... db_cleanup
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:64
2016-08-22 06:37:57.194 63471 DEBUG octavia.cmd.house_keeping [-] Initiating
spare amphora check... spare_amphora_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:49
2016-08-22 06:37:57.203 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Required Spare Amphora count
: 0 spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:45
2016-08-22 06:37:57.204 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Current Spare Amphora count :
1 spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:46
2016-08-22 06:37:57.204 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Current spare amphora count
satisfies the requirement spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:60
2016-08-22 06:37:56.901 63471 DEBUG octavia.cmd.house_keeping [-] Initiating
the cleanup of old amphora... db_cleanup
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:64
2016-08-22 06:37:57.194 63471 DEBUG octavia.cmd.house_keeping [-] Initiating
spare amphora check... spare_amphora_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:49
2016-08-22 06:37:57.203 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Required Spare Amphora count
: 0 spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:45
2016-08-22 06:37:57.204 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Current Spare Amphora count :
1 spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:46
2016-08-22 06:37:57.204 63471 DEBUG
octavia.controller.housekeeping.house_keeping [-] Current spare amphora count
satisfies the requirement spare_check
/opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:60
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1615502/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
