[openstack-dev] [neutron][ovs] OVS drop issue

[Ajay Kalambur (akalambu)]

Hi
We are seeing a very weird issue with OVS 2.9 which is not seen in OVS 2.6. 
Basically the end symptom is from the neutron L3 agent  router namespace we 
cant ping the gateway in this case 10.86.67.1
Now on performing tcpdump tests we see the gateway responding to ARP the reply 
comes into the control/network node and is dropped by OVS in the qg- interface
We observed that when the ARP reply came back to OVS port it added the 
following entry
recirc_id(0),in_port(2),eth(src=f0:25:72:ab:d4:c1,dst=fa:16:3e:65:85:ad),eth_type(0x8100),vlan(vid=0),encap(eth_type(0x0806)),
 packets:217, bytes:13888, used:0.329s, actions:drop

That drop rule states src mac=gateway mac(f0:25:72:ab:d4:c1), destination 
mac=qg-xxx interface drop the packet. We were first not sure why this was 
happening but when we inspected the arp response packet from gateway we noticed 
in this setup the packet from gatway was sent with cos /tos bit set to priority 
5.

When we rewrote the packet on TOR to set cos/tos priority to 0 it worked fine.

Question is why does OVS 2.9 add a drop rule when it sees an ARP response with 
cos/tos priority set to 5.

Has anyone seen this before 2.6 with newton works for this use case and 2.9 
with queens fails

Some info below



L3 Namespace
 ip netns exec qrouter-eee032f4-670f-4e43-8e83-e04cb23f00ae ip addr
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
   valid_lft forever preferred_lft forever
46: ha-df4ad8aa-18:  mtu 1500 qdisc 
noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:eb:0c:21 brd ff:ff:ff:ff:ff:ff
inet 169.254.192.5/18 brd 
169.254.255.255 scope global ha-df4ad8aa-18
   valid_lft forever preferred_lft forever
inet 169.254.0.1/24 scope global ha-df4ad8aa-18
   valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:feeb:c21/64 scope link
   valid_lft forever preferred_lft forever
47: qg-e5541f70-a5:  mtu 1500 qdisc 
noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:65:85:ad brd ff:ff:ff:ff:ff:ff
inet 10.86.67.78/24 scope global qg-e5541f70-a5
   valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe65:85ad/64 scope link nodad
   valid_lft forever preferred_lft forever


Appctl Trace output
ovs_vswitch_15520 [root@BXB-AIO-1 /]# ovs-appctl ofproto/trace br-int 
in_port=2,dl_src=f0:25:72:ab:d4:c1,dl_dst=fa:16:3e:65:85:ad
Flow: 
in_port=2,vlan_tci=0x,dl_src=f0:25:72:ab:d4:c1,dl_dst=fa:16:3e:65:85:ad,dl_type=0x

bridge("br-int")

 0. in_port=2,vlan_tci=0x/0x1fff, priority 3, cookie 0x21589cb48848e7fb
push_vlan:0x8100
set_field:4102->vlan_vid
goto_table:60
60. priority 3, cookie 0x21589cb48848e7fb
NORMAL
 -> no learned MAC for destination, flooding

bridge("br-prov")
-
 0. in_port=2, priority 2, cookie 0x7faae4a30960716f
drop

bridge("br-inst")
-
 0. in_port=2, priority 2, cookie 0x8b9b0311aedc1b0c
drop

Final flow: 
in_port=2,dl_vlan=6,dl_vlan_pcp=0,vlan_tci1=0x,dl_src=f0:25:72:ab:d4:c1,dl_dst=fa:16:3e:65:85:ad,dl_type=0x
Megaflow: 
recirc_id=0,eth,in_port=2,vlan_tci=0x/0x1fff,dl_src=f0:25:72:ab:d4:c1,dl_dst=fa:16:3e:65:85:ad,dl_type=0x
Datapath actions: 9,push_vlan(vid=6,pcp=0),7
ovs_vswitch_15520 [root@BXB-AIO-1 /]#

ovs-dpctl dump-flows
ovs_vswitch_15520 [root@BXB-AIO-1 /]# ovs-dpctl dump-flows | grep 
f0:25:72:ab:d4:c1
recirc_id(0),in_port(2),eth(src=f0:25:72:ab:d4:c1,dst=ff:ff:ff:ff:ff:ff),eth_type(0x8100),vlan(vid=0),encap(eth_type(0x0806),arp(sip=10.86.67.1,tip=10.86.67.77,op=1/0xff)),
 packets:3, bytes:192, used:8.199s, actions:1
recirc_id(0),in_port(2),eth(src=f0:25:72:ab:d4:c1,dst=fa:16:3e:65:85:ad),eth_type(0x8100),vlan(vid=0),encap(eth_type(0x0806)),
 packets:217, bytes:13888, used:0.329s, actions:drop
recirc_id(0),in_port(2),eth(src=f0:25:72:ab:d4:c1,dst=ff:ff:ff:ff:ff:ff),eth_type(0x8100),vlan(vid=0),encap(eth_type(0x0806),arp(sip=10.86.67.1,tip=10.86.67.45,op=1/0xff)),
 packets:3, bytes:192, used:9.778s, actions:1

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Neutron L3 agent/Keepalived

[Ajay Kalambur (akalambu)]

I noticed that this happens when the router HA interface shows status: Down 
what can cause the ha interface to do down

Ajay


From: Ajay Kalambur >
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
>
Date: Wednesday, November 29, 2017 at 4:14 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
>
Subject: [openstack-dev] [neutron] Neutron L3 agent/Keepalived

Hi
I have a case where after running the system for a while I see that floating ip 
association API gets accepted and no Errors in neutron l3 logs
But when I go into the qrouter namespace and check the qg- interface the 
floating ip is not added
Also the keepalived.conf is not updated and SIGUP of the keepalived process is 
not done

So whats the place to look in this case.  What is the flow in neutron l3 agent 
who adds the floating ip to the qg- interface
I see the router update notification received
Key log snippet attached. As you can see SIGUP of keepalived is missing and 
also I confirmed keepalived configs are not updated


2017-11-29 14:32:08.883 78 DEBUG oslo_messaging._drivers.amqpdriver [-] 
received message with unique_id: b3886b998c3049e799170bb5351300d1 __call__ 
/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:269
2017-11-29 14:32:08.885 78 DEBUG neutron.agent.l3.agent 
[req-6c505d32-2d1a-4392-8ea3-0bb888397b9e e759eebecc4648b4964d4ecf439dc0ff 
13b4f6fb188048ba9bbf211344e3342f - - -] Got routers updated notification 
:[u'a1a59e5f-d74e-404d-a3aa-1667c4442aed'] routers_updated 
/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:413
2017-11-29 14:32:08.886 78 DEBUG neutron.agent.l3.agent [-] Starting router 
update for a1a59e5f-d74e-404d-a3aa-1667c4442aed, action None, priority 0 
_process_router_update 
/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:487
2017-11-29 14:32:08.886 78 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL 
msg_id: 92eb015c8c84489681b1efbe949fab8b exchange 'neutron' topic 'q-l3-plugin' 
_send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568

2017-11-29 14:32:09.571 78 DEBUG oslo_messaging._drivers.amqpdriver [-] 
received reply msg_id: 92eb015c8c84489681b1efbe949fab8b __call__ 
/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416
2017-11-29 14:32:09.571 78 DEBUG neutron.callbacks.manager [-] Notify callbacks 
[] for router, before_update _notify_loop 
/usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:142
2017-11-29 14:32:09.572 78 DEBUG neutron.agent.l3.router_info [-] process 
router updates process 
/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:1105
2017-11-29 14:32:09.572 78 DEBUG neutron.agent.linux.utils [-] Running command 
(rootwrap daemon): ['ip', 'netns', 'exec', 
'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', 
'-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.709 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 
execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock 
"l3-agent-pd" acquired by "neutron.agent.linux.pd.sync_router" :: waited 0.000s 
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock 
"l3-agent-pd" released by "neutron.agent.linux.pd.sync_router" :: held 0.000s 
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-11-29 14:32:09.710 78 DEBUG neutron.agent.linux.utils [-] Running command 
(rootwrap daemon): ['ip', 'netns', 'exec', 
'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', 
'-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.845 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 
execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.846 78 DEBUG oslo_concurrency.lockutils [-] Acquired 
semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock 
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-11-29 14:32:09.846 78 DEBUG neutron.agent.linux.utils [-] Running command 
(rootwrap daemon): ['ip', 'netns', 'exec', 
'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-save'] 
execute_rootwrap_daemon 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.983 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 
execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.986 78 DEBUG neutron.agent.linux.utils [-] Running 

[openstack-dev] [neutron] Neutron L3 agent/Keepalived

[Ajay Kalambur (akalambu)]

Hi
I have a case where after running the system for a while I see that floating ip 
association API gets accepted and no Errors in neutron l3 logs
But when I go into the qrouter namespace and check the qg- interface the 
floating ip is not added
Also the keepalived.conf is not updated and SIGUP of the keepalived process is 
not done

So whats the place to look in this case.  What is the flow in neutron l3 agent 
who adds the floating ip to the qg- interface
I see the router update notification received
Key log snippet attached. As you can see SIGUP of keepalived is missing and 
also I confirmed keepalived configs are not updated


2017-11-29 14:32:08.883 78 DEBUG oslo_messaging._drivers.amqpdriver [-] 
received message with unique_id: b3886b998c3049e799170bb5351300d1 __call__ 
/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:269
2017-11-29 14:32:08.885 78 DEBUG neutron.agent.l3.agent 
[req-6c505d32-2d1a-4392-8ea3-0bb888397b9e e759eebecc4648b4964d4ecf439dc0ff 
13b4f6fb188048ba9bbf211344e3342f - - -] Got routers updated notification 
:[u'a1a59e5f-d74e-404d-a3aa-1667c4442aed'] routers_updated 
/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:413
2017-11-29 14:32:08.886 78 DEBUG neutron.agent.l3.agent [-] Starting router 
update for a1a59e5f-d74e-404d-a3aa-1667c4442aed, action None, priority 0 
_process_router_update 
/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:487
2017-11-29 14:32:08.886 78 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL 
msg_id: 92eb015c8c84489681b1efbe949fab8b exchange 'neutron' topic 'q-l3-plugin' 
_send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568

2017-11-29 14:32:09.571 78 DEBUG oslo_messaging._drivers.amqpdriver [-] 
received reply msg_id: 92eb015c8c84489681b1efbe949fab8b __call__ 
/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416
2017-11-29 14:32:09.571 78 DEBUG neutron.callbacks.manager [-] Notify callbacks 
[] for router, before_update _notify_loop 
/usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:142
2017-11-29 14:32:09.572 78 DEBUG neutron.agent.l3.router_info [-] process 
router updates process 
/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:1105
2017-11-29 14:32:09.572 78 DEBUG neutron.agent.linux.utils [-] Running command 
(rootwrap daemon): ['ip', 'netns', 'exec', 
'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', 
'-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.709 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 
execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock 
"l3-agent-pd" acquired by "neutron.agent.linux.pd.sync_router" :: waited 0.000s 
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock 
"l3-agent-pd" released by "neutron.agent.linux.pd.sync_router" :: held 0.000s 
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-11-29 14:32:09.710 78 DEBUG neutron.agent.linux.utils [-] Running command 
(rootwrap daemon): ['ip', 'netns', 'exec', 
'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', 
'-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.845 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 
execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.846 78 DEBUG oslo_concurrency.lockutils [-] Acquired 
semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock 
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-11-29 14:32:09.846 78 DEBUG neutron.agent.linux.utils [-] Running command 
(rootwrap daemon): ['ip', 'netns', 'exec', 
'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-save'] 
execute_rootwrap_daemon 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.983 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 
execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.986 78 DEBUG neutron.agent.linux.utils [-] Running command 
(rootwrap daemon): ['ip', 'netns', 'exec', 
'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-restore', '-n'] 
execute_rootwrap_daemon 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:10.120 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 
execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:10.120 78 DEBUG neutron.agent.linux.iptables_manager [-] 
IPTablesManager.apply completed with success. 7 iptables commands were issued 
_apply_synchronized 

Re: [openstack-dev] [neutron]OVS connection tracking cleanup

[Ajay Kalambur (akalambu)]

set, write_set, [])
  File "/usr/lib/python2.7/site-packages/eventlet/green/select.py", line 86, in 
select
return hub.switch()
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in 
switch
return self.greenlet.switch()
Timeout: 5 seconds

2017-09-12 09:23:03.860 35 INFO oslo_rootwrap.client [-] Stopping rootwrap 
daemon process with pid=95


Ajay



From: Kevin Benton <ke...@benton.pub<mailto:ke...@benton.pub>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Monday, September 11, 2017 at 1:12 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Cc: "Ian Wells (iawells)" <iawe...@cisco.com<mailto:iawe...@cisco.com>>
Subject: Re: [openstack-dev] [neutron]OVS connection tracking cleanup

Can you start a bug on launchpad and upload the conntrack attachment to the bug?

Switching to the rootwrap daemon should also help significantly.

On Mon, Sep 11, 2017 at 12:32 PM, Ajay Kalambur (akalambu) 
<akala...@cisco.com<mailto:akala...@cisco.com>> wrote:
Hi Kevin
The information you asked for
For 1 compute node with 45 Vms here is the number of connection tracking 
entries getting deleted
cat conntrack.file  | wc -l
   38528

The file with output is 14MB so ill email it to Ian and he can share it if 
needed

Security group rules
DirectionEther TypeIP ProtocolPort RangeRemote IP PrefixRemote Security 
GroupActions
EgressIPv4AnyAny0.0.0.0/0<http://0.0.0.0/0>
IngressIPv6AnyAny-default
EgressIPv6AnyAny::/0-
IngressIPv4AnyAny-

Please let me know if u need the dump of conntrack entries if so I can email it 
to email address of your choice


Ajay



From: Ajay Kalambur <akala...@cisco.com<mailto:akala...@cisco.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Monday, September 11, 2017 at 10:02 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [neutron]OVS connection tracking cleanup

Hi Kevin
Thanks for your response it was about 50 vms
Ajay



On Sep 11, 2017, at 9:49 AM, Kevin Benton 
<ke...@benton.pub<mailto:ke...@benton.pub>> wrote:

The biggest improvement will be switching to native netlink calls: 
https://review.openstack.org/#/c/470912/

How many VMs were on a single compute node?

On Mon, Sep 11, 2017 at 9:15 AM, Ajay Kalambur (akalambu) 
<akala...@cisco.com<mailto:akala...@cisco.com>> wrote:
Hi
I am performing a scale test and I see that after creating 500 Vms with ping 
traffic between them it took almost 1 hr for the connection tracking
To clean up and ovs agent was busy doing this and unable to service any new 
port bind requests on some computes for almost an hr
It took that long for conntrack clean up to complete


I see the following bug
https://bugs.launchpad.net/neutron/+bug/1513765

And I also have the fix below
https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7aeb8dd4b1d122e17eef8687192cd122b79fd6e


Still see really long times for conntrack cleanup

What is the solution to this problem in scale scenarios?
Ajay


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org<mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron]OVS connection tracking cleanup

[Ajay Kalambur (akalambu)]

Also the weird part with this conntrack deletion I perform a conntrack –L to 
view the table I see no entry for any of the entries its trying to delete. 
Those entries are all removed anyways when Vms are cleaned up from the look of 
it. So it looks like all those conntrack deletions were pretty much no-ops
Ajay


From: Ajay Kalambur <akala...@cisco.com<mailto:akala...@cisco.com>>
Date: Tuesday, September 12, 2017 at 9:30 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Cc: "Ian Wells (iawells)" <iawe...@cisco.com<mailto:iawe...@cisco.com>>
Subject: Re: [openstack-dev] [neutron]OVS connection tracking cleanup

Hi Kevin
Sure will log a bug
Also does the config change involve having both these lines in the neutron.conf 
file?
[agent]
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
root_helper_daemon = sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf

If I have only the second line I see the exception below on neutron openvswitch 
agent bring up:

2017-09-12 09:23:03.633 35 DEBUG neutron.agent.linux.utils 
[req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] Running command: ['ps', 
'--ppid', '103', '-o', 'pid='] create_process 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:89
2017-09-12 09:23:03.762 35 ERROR ryu.lib.hub 
[req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] hub: uncaught exception: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
return func(*args, **kwargs)
  File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py",
 line 42, in agent_main_wrapper
ovs_agent.main(bridge_classes)
  File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 2184, in main
agent.daemon_loop()
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in 
wrapper
return f(*args, **kwargs)
  File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 2100, in daemon_loop
self.ovsdb_monitor_respawn_interval) as pm:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 
35, in get_polling_manager
pm.start()
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 
57, in start
while not self.is_active():
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", 
line 100, in is_active
self.pid, self.cmd_without_namespace)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", 
line 159, in pid
run_as_root=self.run_as_root)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 
297, in get_root_helper_child_pid
pid = find_child_pids(pid)[0]
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 
179, in find_child_pids
log_fail_as_error=False)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 
128, in execute
_stdout, _stderr = obj.communicate(_process_input)
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1403, in _communicate
stdout, stderr = self._communicate_with_select(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1504, in 
_communicate_with_select
rlist, wlist, xlist = select.select(read_set, write_set, [])
  File "/usr/lib/python2.7/site-packages/eventlet/green/select.py", line 86, in 
select
return hub.switch()
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in 
switch
return self.greenlet.switch()
Timeout: 5 seconds

2017-09-12 09:23:03.860 35 INFO oslo_rootwrap.client [-] Stopping rootwrap 
daemon process with pid=95


Ajay



From: Kevin Benton <ke...@benton.pub<mailto:ke...@benton.pub>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Monday, September 11, 2017 at 1:12 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Cc: "Ian Wells (iawells)" <iawe...@cisco.com<mailto:iawe...@cisco.com>>
Subject: Re: [openstack-dev] [neutron]OVS connection tracking cleanup

Can you start a bug on launchpad and upload the conntrack attachment to the bug?

Switchi

Re: [openstack-dev] [neutron] MTU native ovs firewall driver

[Ajay Kalambur (akalambu)]

So I was forced to explicitly set the MTU on br-int
ovs-vsctl set int br-int mtu_request=9000


Without this the tap device added to br-int would get MTU 1500

Would this be something the ovs l2 agent can handle since it creates the bridge?

Ajay




On 9/20/17, 7:28 AM, "Ajay Kalambur (akalambu)" <akala...@cisco.com> wrote:

>Hi
>i am using large mtu setting of 9000. with the hybrid firewall driver i see 
>large mtu set on both tap devices and the linuxbridges
>While i switch from ovs hybrid firewall driver to native ovs firewall driver i 
>now see the tap device gets the default 1500 mtu
>i have the right setting for mtu in global physnet mtu in neutron.conf and 
>path mtu in ml2 conf
>Do i need to do anything different to get large mtu working with native ovs 
>firewall driver
>Ajay
>
>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] MTU native ovs firewall driver

[Ajay Kalambur (akalambu)]

Hi
i am using large mtu setting of 9000. with the hybrid firewall driver i see 
large mtu set on both tap devices and the linuxbridges
While i switch from ovs hybrid firewall driver to native ovs firewall driver i 
now see the tap device gets the default 1500 mtu
i have the right setting for mtu in global physnet mtu in neutron.conf and path 
mtu in ml2 conf
Do i need to do anything different to get large mtu working with native ovs 
firewall driver
Ajay



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron]OVS connection tracking cleanup

[Ajay Kalambur (akalambu)]

Looks like I need both lines. I now see conntrack deletes happening with 
rootwrap daemon, Kevin yes the performance seems much better but still lots of 
entries to be deleted and hence port bind blocks. Any way we can perform 
conntrack cleanup in separate thread

Ajay


From: Ajay Kalambur <akala...@cisco.com<mailto:akala...@cisco.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Tuesday, September 12, 2017 at 9:30 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Cc: "Ian Wells (iawells)" <iawe...@cisco.com<mailto:iawe...@cisco.com>>
Subject: Re: [openstack-dev] [neutron]OVS connection tracking cleanup

Hi Kevin
Sure will log a bug
Also does the config change involve having both these lines in the neutron.conf 
file?
[agent]
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
root_helper_daemon = sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf

If I have only the second line I see the exception below on neutron openvswitch 
agent bring up:

2017-09-12 09:23:03.633 35 DEBUG neutron.agent.linux.utils 
[req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] Running command: ['ps', 
'--ppid', '103', '-o', 'pid='] create_process 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:89
2017-09-12 09:23:03.762 35 ERROR ryu.lib.hub 
[req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] hub: uncaught exception: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
return func(*args, **kwargs)
  File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py",
 line 42, in agent_main_wrapper
ovs_agent.main(bridge_classes)
  File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 2184, in main
agent.daemon_loop()
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in 
wrapper
return f(*args, **kwargs)
  File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 2100, in daemon_loop
self.ovsdb_monitor_respawn_interval) as pm:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 
35, in get_polling_manager
pm.start()
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 
57, in start
while not self.is_active():
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", 
line 100, in is_active
self.pid, self.cmd_without_namespace)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", 
line 159, in pid
run_as_root=self.run_as_root)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 
297, in get_root_helper_child_pid
pid = find_child_pids(pid)[0]
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 
179, in find_child_pids
log_fail_as_error=False)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 
128, in execute
_stdout, _stderr = obj.communicate(_process_input)
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1403, in _communicate
stdout, stderr = self._communicate_with_select(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1504, in 
_communicate_with_select
rlist, wlist, xlist = select.select(read_set, write_set, [])
  File "/usr/lib/python2.7/site-packages/eventlet/green/select.py", line 86, in 
select
return hub.switch()
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in 
switch
return self.greenlet.switch()
Timeout: 5 seconds

2017-09-12 09:23:03.860 35 INFO oslo_rootwrap.client [-] Stopping rootwrap 
daemon process with pid=95


Ajay



From: Kevin Benton <ke...@benton.pub<mailto:ke...@benton.pub>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Monday, September 11, 2017 at 1:12 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Cc: "Ian Wells (iawells)" <iawe...@cisco.com<mailto:iawe...@cisco.com>>
Subject: Re: [openstack-dev] [ne

Re: [openstack-dev] [neutron]OVS connection tracking cleanup

[Ajay Kalambur (akalambu)]

Hi Kevin
Sure will log a bug
Also does the config change involve having both these lines in the neutron.conf 
file?
[agent]
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
root_helper_daemon = sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf

If I have only the second line I see the exception below on neutron openvswitch 
agent bring up:

2017-09-12 09:23:03.633 35 DEBUG neutron.agent.linux.utils 
[req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] Running command: ['ps', 
'--ppid', '103', '-o', 'pid='] create_process 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:89
2017-09-12 09:23:03.762 35 ERROR ryu.lib.hub 
[req-0f8fe685-66bd-44d7-beac-bb4c24f0ccfa - - - - -] hub: uncaught exception: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
return func(*args, **kwargs)
  File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py",
 line 42, in agent_main_wrapper
ovs_agent.main(bridge_classes)
  File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 2184, in main
agent.daemon_loop()
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in 
wrapper
return f(*args, **kwargs)
  File 
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py",
 line 2100, in daemon_loop
self.ovsdb_monitor_respawn_interval) as pm:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 
35, in get_polling_manager
pm.start()
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 
57, in start
while not self.is_active():
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", 
line 100, in is_active
self.pid, self.cmd_without_namespace)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", 
line 159, in pid
run_as_root=self.run_as_root)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 
297, in get_root_helper_child_pid
pid = find_child_pids(pid)[0]
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 
179, in find_child_pids
log_fail_as_error=False)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 
128, in execute
_stdout, _stderr = obj.communicate(_process_input)
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1403, in _communicate
stdout, stderr = self._communicate_with_select(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1504, in 
_communicate_with_select
rlist, wlist, xlist = select.select(read_set, write_set, [])
  File "/usr/lib/python2.7/site-packages/eventlet/green/select.py", line 86, in 
select
return hub.switch()
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in 
switch
return self.greenlet.switch()
Timeout: 5 seconds

2017-09-12 09:23:03.860 35 INFO oslo_rootwrap.client [-] Stopping rootwrap 
daemon process with pid=95


Ajay



From: Kevin Benton <ke...@benton.pub<mailto:ke...@benton.pub>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Monday, September 11, 2017 at 1:12 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Cc: "Ian Wells (iawells)" <iawe...@cisco.com<mailto:iawe...@cisco.com>>
Subject: Re: [openstack-dev] [neutron]OVS connection tracking cleanup

Can you start a bug on launchpad and upload the conntrack attachment to the bug?

Switching to the rootwrap daemon should also help significantly.

On Mon, Sep 11, 2017 at 12:32 PM, Ajay Kalambur (akalambu) 
<akala...@cisco.com<mailto:akala...@cisco.com>> wrote:
Hi Kevin
The information you asked for
For 1 compute node with 45 Vms here is the number of connection tracking 
entries getting deleted
cat conntrack.file  | wc -l
   38528

The file with output is 14MB so ill email it to Ian and he can share it if 
needed

Security group rules
DirectionEther TypeIP ProtocolPort RangeRemote IP PrefixRemote Security 
GroupActions
EgressIPv4AnyAny0.0.0.0/0<http://0.0.0.0/0>
IngressIPv6AnyAny-default
EgressIPv6AnyAny::/0-
IngressIPv4AnyAny-

Please let me know if u need the dump of conntrack entries if so I can email it 
to email address of your ch

Re: [openstack-dev] [neutron]OVS connection tracking cleanup

[Ajay Kalambur (akalambu)]

Hi Kevin
The information you asked for
For 1 compute node with 45 Vms here is the number of connection tracking 
entries getting deleted
cat conntrack.file  | wc -l
   38528

The file with output is 14MB so ill email it to Ian and he can share it if 
needed

Security group rules
Direction Ether Type IP Protocol Port Range Remote IP Prefix Remote Security 
Group Actions
Egress IPv4 Any Any 0.0.0.0/0
Ingress IPv6 Any Any - default
Egress IPv6 Any Any ::/0 -
Ingress IPv4 Any Any -

Please let me know if u need the dump of conntrack entries if so I can email it 
to email address of your choice


Ajay



From: Ajay Kalambur <akala...@cisco.com<mailto:akala...@cisco.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Monday, September 11, 2017 at 10:02 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [neutron]OVS connection tracking cleanup

Hi Kevin
Thanks for your response it was about 50 vms
Ajay



On Sep 11, 2017, at 9:49 AM, Kevin Benton 
<ke...@benton.pub<mailto:ke...@benton.pub>> wrote:

The biggest improvement will be switching to native netlink calls: 
https://review.openstack.org/#/c/470912/

How many VMs were on a single compute node?

On Mon, Sep 11, 2017 at 9:15 AM, Ajay Kalambur (akalambu) 
<akala...@cisco.com<mailto:akala...@cisco.com>> wrote:
Hi
I am performing a scale test and I see that after creating 500 Vms with ping 
traffic between them it took almost 1 hr for the connection tracking
To clean up and ovs agent was busy doing this and unable to service any new 
port bind requests on some computes for almost an hr
It took that long for conntrack clean up to complete


I see the following bug
https://bugs.launchpad.net/neutron/+bug/1513765

And I also have the fix below
https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7aeb8dd4b1d122e17eef8687192cd122b79fd6e


Still see really long times for conntrack cleanup

What is the solution to this problem in scale scenarios?
Ajay


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org<mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron]OVS connection tracking cleanup

[Ajay Kalambur (akalambu)]

Hi Kevin
Thanks for your response it was about 50 vms
Ajay



On Sep 11, 2017, at 9:49 AM, Kevin Benton 
<ke...@benton.pub<mailto:ke...@benton.pub>> wrote:

The biggest improvement will be switching to native netlink calls: 
https://review.openstack.org/#/c/470912/

How many VMs were on a single compute node?

On Mon, Sep 11, 2017 at 9:15 AM, Ajay Kalambur (akalambu) 
<akala...@cisco.com<mailto:akala...@cisco.com>> wrote:
Hi
I am performing a scale test and I see that after creating 500 Vms with ping 
traffic between them it took almost 1 hr for the connection tracking
To clean up and ovs agent was busy doing this and unable to service any new 
port bind requests on some computes for almost an hr
It took that long for conntrack clean up to complete


I see the following bug
https://bugs.launchpad.net/neutron/+bug/1513765

And I also have the fix below
https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7aeb8dd4b1d122e17eef8687192cd122b79fd6e


Still see really long times for conntrack cleanup

What is the solution to this problem in scale scenarios?
Ajay


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org<mailto:openstack-dev-requ...@lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron]OVS connection tracking cleanup

[Ajay Kalambur (akalambu)]

Hi
I am performing a scale test and I see that after creating 500 Vms with ping 
traffic between them it took almost 1 hr for the connection tracking
To clean up and ovs agent was busy doing this and unable to service any new 
port bind requests on some computes for almost an hr
It took that long for conntrack clean up to complete


I see the following bug
https://bugs.launchpad.net/neutron/+bug/1513765

And I also have the fix below
https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7aeb8dd4b1d122e17eef8687192cd122b79fd6e


Still see really long times for conntrack cleanup

What is the solution to this problem in scale scenarios?
Ajay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Compute Node restart

[Ajay Kalambur (akalambu)]

I am just issuing a reboot command on the compute node

Not a reboot –f

From: Mark Mielke <mark.mie...@gmail.com<mailto:mark.mie...@gmail.com>>
Date: Wednesday, April 26, 2017 at 8:42 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>, 
Ajay Kalambur <akala...@cisco.com<mailto:akala...@cisco.com>>
Subject: Re: [openstack-dev] [nova] Compute Node restart

On Apr 25, 2017 2:45 AM, "Ajay Kalambur (akalambu)" 
<akala...@cisco.com<mailto:akala...@cisco.com>> wrote:
I see that when a host is gracefully rebooted nova-compute receives a lifecycle 
event to shutdown the instance and it updates the database with the state set 
to SHUTOFF.
Now when compute node reboot and libvirt brings the VM back up nova checks its 
database and issues stop api and hence shuts down the VM

So even though resume guest state on reboot is set it does not work. Why does 
nova compute ignore the lifecycle event from libvirt saying bring up the VM and 
reconciles with database

How are you defining "graceful reboot"?

I am thinking that you may be defining it in a way that implies prior safe 
shutdown of the guests including setting their state to "power off", in which 
case restoring them to the prior state before reboot will correctly leave them 
"power off". I believe this feature is only intended to work with a shutdown of 
the hypervisor such as occurs when you "shutdown -r now" on the hypervisor 
without first shutting down the guests.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Compute Node restart

[Ajay Kalambur (akalambu)]

Hi
I see that when a host is gracefully rebooted nova-compute receives a lifecycle 
event to shutdown the instance and it updates the database with the state set 
to SHUTOFF.
Now when compute node reboot and libvirt brings the VM back up nova checks its 
database and issues stop api and hence shuts down the VM

So even though resume guest state on reboot is set it does not work. Why does 
nova compute ignore the lifecycle event from libvirt saying bring up the VM and 
reconciles with database


How do we ensure Vms come back up on a graceful reboot

Note on a non graceful reboot libvirt does not send any lifecycle event to nova 
and hence nova database always shows the VM in Running State

This is with Newton code base of nova


Ajay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Unit tests for a review

[Ajay Kalambur (akalambu)]

Hi
I am looking into writing unit tests for a review
https://review.openstack.org/#/c/387687/

Basically it modifies the deallocate for instance function to only deallocate 
ports to which this instance is bound to

So this is more like a scenario test where the port is unbound outside of nova.

So what would be the closest test to start with and modify to get this scenario 
going.

Would this be better of as a functional test vs a unit test?

Ajay


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upgrade skip release

[Ajay Kalambur (akalambu)]

Hi Chris
Thanks for the response. What were the specific problem areas you saw when 
upgrading from Kilo to Mitaka which were introduced by skipping Liberty?
Ajay




On 12/19/16, 12:36 PM, "Chris Friesen" <chris.frie...@windriver.com> wrote:

>On 12/19/2016 01:56 PM, Ajay Kalambur (akalambu) wrote:
>> Hi
>> Does openstack support skipping releases if one is performing a maintanance
>> upgrade. Would alembic migrations work for instance from Liberty to Newton
>
>OpenStack as a whole only supports N to N+1 upgrades, though it's possible 
>some 
>services may support more than that.
>
>We did Kilo->Mitaka for various reasons and had to resolve a bunch of problems.
>
>Chris
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Upgrade skip release

[Ajay Kalambur (akalambu)]

Hi
Does openstack support skipping releases if one is performing a maintanance 
upgrade. Would alembic migrations work for instance from Liberty to Newton

I think we would have to stop all existing services since API backward 
compatiblity is only 1 release front and back right?

Ajay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] rabbitmq timeout

[Ajay Kalambur (akalambu)]

Hi
We see an issue where if we boot 10 instances on 1 compute node at a time we 
see that sometimes a few instances error out on compute node with following 
Traceback
It looks like the RPC message call from compute saying set Instance to Active 
is timing out.  It seems to help to set rpc response timeout to 180 seconds. 
But what we see from TCP dump is that the message reaches the conductor and 
conductor replies almost instantly. It looks like the compute node in question 
somehow takes a long time to process it. Any pointers. Also we had to increase 
heartbeat timeout to 120 from 60 to make this scenario work better. Any inputs 
on what maybe going on here?




2016-11-14 04:44:19.518 6 INFO nova.virt.libvirt.driver [-] [instance: 
31e67591-6bd4-4df9-bba1-7a1d6f785be0] Instance spawned successfully.

2016-11-14 04:44:55.692 6 INFO nova.compute.manager 
[req-25dc826e-54bc-4c7c-b58f-c32c8e024624 - - - - -] [instance: 
31e67591-6bd4-4df9-bba1-7a1d6f785be0] During sync_power_state the instance has 
a pending task (spawning). Skip.

2016-11-14 04:44:55.693 6 INFO nova.compute.manager 
[req-25dc826e-54bc-4c7c-b58f-c32c8e024624 - - - - -] [instance: 
31e67591-6bd4-4df9-bba1-7a1d6f785be0] VM Started (Lifecycle Event)

2016-11-14 04:45:04.544 6 INFO nova.virt.libvirt.driver [-] [instance: 
5040fdad-2dbe-49a2-aa3d-ccd8e7725b5d] Instance spawned successfully.

2016-11-14 04:45:04.558 6 INFO nova.virt.libvirt.driver [-] [instance: 
ba4b3c89-0f57-4c46-91dd-44434012d289] Instance spawned successfully.

2016-11-14 04:45:04.607 6 INFO nova.compute.manager 
[req-25dc826e-54bc-4c7c-b58f-c32c8e024624 - - - - -] [instance: 
5040fdad-2dbe-49a2-aa3d-ccd8e7725b5d] VM Resumed (Lifecycle Event)

2016-11-14 04:45:16.669 6 INFO nova.virt.libvirt.driver [-] [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c] Instance spawned successfully.

2016-11-14 04:45:16.710 6 INFO nova.compute.manager 
[req-25dc826e-54bc-4c7c-b58f-c32c8e024624 - - - - -] [instance: 
5040fdad-2dbe-49a2-aa3d-ccd8e7725b5d] VM Started (Lifecycle Event)

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager 
[req-20ec8e07-d57c-4679-8122-4cfa990b6d6f 168b323822284084b1c1393faeb5b9e1 
aa39823c34e4496793250c0bc5cf7a31 - - -] [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c] Unexpected build failure, not 
rescheduling build.

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c] Traceback (most recent call last):

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c]   File 
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1905, in 
_do_build_and_run_instance

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c] filter_properties)

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c]   File 
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2082, in 
_build_and_run_instance

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c] 
instance.save(expected_task_state=task_states.SPAWNING)

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c]   File 
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 197, in 
wrapper

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c] ctxt, self, fn.__name__, args, kwargs)

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c]   File 
"/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 242, in 
object_action

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c] objmethod=objmethod, args=args, 
kwargs=kwargs)

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c]   File 
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in 
call

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c] retry=self.retry)

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c]   File 
"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in 
_send

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c] timeout=timeout, retry=retry)

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c]   File 
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 
431, in send

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c] retry=retry)

2016-11-14 04:46:43.765 6 ERROR nova.compute.manager [instance: 
3794b761-f623-4fb9-a91a-f2dffca42c4c]   File 

Re: [openstack-dev] [openstack][nova] Port unbound from active VM

[Ajay Kalambur (akalambu)]

Hi
I logged the following bug
https://bugs.launchpad.net/nova/+bug/1634269

I sent out review below please let me know if I am on right track

https://review.openstack.org/#/c/387687/

Ajay




From: Kevin Benton <ke...@benton.pub<mailto:ke...@benton.pub>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Friday, October 14, 2016 at 1:23 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [openstack][nova] Port unbound from active VM

I suggest filing a bug against nova for this.

On Thu, Oct 13, 2016 at 9:44 AM, Ajay Kalambur (akalambu) 
<akala...@cisco.com<mailto:akala...@cisco.com>> wrote:
Any comments/input on this?
Ajay


From: Ajay Kalambur <akala...@cisco.com<mailto:akala...@cisco.com>>
Date: Monday, October 10, 2016 at 6:31 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: [openstack][nova] Port unbound from active VM

Hi
There seems to be a corner case bug in nova code. Steps to reproduce it are

  1.  Create a neutron port
  2.  Create a VM and launch instance with this port
  3.  Shutdown nova compute and network agent on compute node
  4.  Unbind port from VM and delete the VM (offline delete)
  5.  Now create a VM with same port but on a different compute node
  6.  Bring up nova compute on old node

It basically runs the reap for deleted instances and cleanes up VM from 
libvirt. In the process it unbinds the pre-existing ports and ends up unbinding 
the port from an active VM on a different compute node
Reason nova simply sends a blind port-update with binding_host: “” even if that 
port is bound to a different instance


So following fix seemed to help any suggestions on a better fix

In nova/network/neutronv2/api.py. So basically when neutron sees no ports for 
this instance don’t attempt an unbind
In this case
data = neutron.list_ports(**search_opts)
Call in deallocate_for_instance returns empty ports




  # Reset device_id and device_owner for the ports that are skipped

if data.get('ports', []):

self._unbind_ports(context, ports_to_skip, neutron)

else:

LOG.debug("Neutron sees a different view of this port hence 
skipping unbind”)



Ajay


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] Heartbeat timeout

[Ajay Kalambur (akalambu)]

Hi
In our setup we see the following message every second or 2
This cant be normal right?
This means 30 heartbeats were missed since interval is 2 seconds. So in 60 
seconds no heartbeat was sent. Is that how I interpret this message. I don’t 
see any Errors or Tracebacks related to oslo


2016-10-14 10:48:59.354 6 INFO oslo.messaging._drivers.impl_rabbit [-] A
recoverable connection/channel error occurred, trying to reconnect: Too
many heartbeats missed
2016-10-14 10:49:14.354 6 INFO oslo.messaging._drivers.impl_rabbit [-] A
recoverable connection/channel error occurred, trying to reconnect: Too
many heartbeats missed
2016-10-14 10:49:29.355 6 INFO oslo.messaging._drivers.impl_rabbit [-] A
recoverable connection/channel error occurred, trying to reconnect: Too
many heartbeats missed
2016-10-14 10:49:44.356 6 INFO oslo.messaging._drivers.impl_rabbit [-] A
recoverable connection/channel error occurred, trying to reconnect: Too
many heartbeats missed


Has anyone got a working cloud where they see these messages every so often
Typically when do you see these messages?



Ajay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack][nova] Port unbound from active VM

[Ajay Kalambur (akalambu)]

Any comments/input on this?
Ajay


From: Ajay Kalambur >
Date: Monday, October 10, 2016 at 6:31 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
>
Subject: [openstack][nova] Port unbound from active VM

Hi
There seems to be a corner case bug in nova code. Steps to reproduce it are

  1.  Create a neutron port
  2.  Create a VM and launch instance with this port
  3.  Shutdown nova compute and network agent on compute node
  4.  Unbind port from VM and delete the VM (offline delete)
  5.  Now create a VM with same port but on a different compute node
  6.  Bring up nova compute on old node

It basically runs the reap for deleted instances and cleanes up VM from 
libvirt. In the process it unbinds the pre-existing ports and ends up unbinding 
the port from an active VM on a different compute node
Reason nova simply sends a blind port-update with binding_host: “” even if that 
port is bound to a different instance


So following fix seemed to help any suggestions on a better fix

In nova/network/neutronv2/api.py. So basically when neutron sees no ports for 
this instance don’t attempt an unbind
In this case
data = neutron.list_ports(**search_opts)
Call in deallocate_for_instance returns empty ports




  # Reset device_id and device_owner for the ports that are skipped

if data.get('ports', []):

self._unbind_ports(context, ports_to_skip, neutron)

else:

LOG.debug("Neutron sees a different view of this port hence 
skipping unbind”)



Ajay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [openstack][nova] Port unbound from active VM

[Ajay Kalambur (akalambu)]

Hi
There seems to be a corner case bug in nova code. Steps to reproduce it are

  1.  Create a neutron port
  2.  Create a VM and launch instance with this port
  3.  Shutdown nova compute and network agent on compute node
  4.  Unbind port from VM and delete the VM (offline delete)
  5.  Now create a VM with same port but on a different VM
  6.  Bring up nova compute on old node

It basically runs the reap for deleted instances and cleanes up VM from 
libvirt. In the process it unbinds the pre-existing ports and ends up unbinding 
the port from an active VM
Reason nova simply sends a blind port-update with binding_host: “” even if that 
port is bound to a different instance


So following fix seemed to help any suggestions on a better fix

In nova/network/neutronv2/api.py. So basically when neutron sees no ports for 
this instance don’t attempt an unbind
In this case
data = neutron.list_ports(**search_opts)
Call in deallocate_for_instance returns empty ports




  # Reset device_id and device_owner for the ports that are skipped

if data.get('ports', []):

self._unbind_ports(context, ports_to_skip, neutron)

else:

LOG.debug("Neutron sees a different view of this port hence 
skipping unbind”)



Ajay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] VIF plugin issue _get_neutron_events

[Ajay Kalambur (akalambu)]

Hi Dan
Thanks for the response. This is with one of our custom neutron ML2 mechanism 
drivers and that code is not yet upstream
Nothing has changed in nova code though. We just use the Liberty version
If I provide the nova and neutron logs would it be possible to provide some 
insights.
I can log a bug for this
Ajay




On 10/8/16, 5:41 PM, "Dan Smith"  wrote:

>> Basically the issue is seen in the following three lines of nova compute
>> log. For that port even though it received the vif plugging event 2 mins
>> before it waits for it and blocks and times out
>> Is there a race condition in the code that basically gets the events to
>> wait for and the one where it registers for this callback 
>> Any comments?
>
>This shouldn't be possible because the point at which we call
>plug_vifs() is when we should trigger neutron to fire the vif-plugged
>event, and that is after where we have already setup to receive those
>events. In other words, your log lines should never be able to be in the
>order you showed. If it's happening in that order (especially two
>minutes early) I would be highly suspect of some modifications or
>something else weird going on.
>
>It'd be much better to handle this in the context of a bug, especially
>providing the versions and components (i.e. neutron driver), etc. Can
>you open one and provide all the usual details? I'll be happy to look.
>
>--Dan
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] VIF plugin issue _get_neutron_events

[Ajay Kalambur (akalambu)]

Basically the issue is seen in the following three lines of nova compute log. 
For that port even though it received the vif plugging event 2 mins before it 
waits for it and blocks and times out
Is there a race condition in the code that basically gets the events to wait 
for and the one where it registers for this callback
Any comments?

2016-10-09 07:35:24.869 6 DEBUG nova.compute.manager 
[req-afbc9c31-757a-478e-af8f-a4937e787a57 53a2bd71435c43e3953c37ffb9b38e21 
56b04dfbb802430face997791412a62a - - -] [instance: 
59b4f72b-5d1d-41ee-bc57-c57a8204383a] Received event 
network-vif-plugged-4cf63ac9-3848-4c0e-a666-25dca7fadda0 
external_instance_event 
/usr/lib/python2.7/site-packages/nova/compute/manager.py:6509
2016-10-09 07:35:24.871 6 DEBUG nova.compute.manager 
[req-afbc9c31-757a-478e-af8f-a4937e787a57 53a2bd71435c43e3953c37ffb9b38e21 
56b04dfbb802430face997791412a62a - - -] [instance: 
59b4f72b-5d1d-41ee-bc57-c57a8204383a] No waiting events found dispatching 
network-vif-plugged-4cf63ac9-3848-4c0e-a666-25dca7fadda0 pop_instance_event 
/usr/lib/python2.7/site-packages/nova/compute/manager.py:545
2016-10-09 07:37:36.668 6 DEBUG nova.compute.manager 
[req-31f88ed2-ab3d-4b1d-a75b-b4e7b80b9202 28450924ed064558948f98eaec39fe6e 
54e6da5850104d3682ef3500f3eb7a57 - - -] [instance: 
59b4f72b-5d1d-41ee-bc57-c57a8204383a] Preparing to wait for external event 
network-vif-plugged-4cf63ac9-3848-4c0e-a666-25dca7fadda0 
prepare_for_instance_event 
/usr/lib/python2.7/site-packages/nova/compute/manager.py:507

From: Ajay Kalambur >
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
>
Date: Tuesday, October 4, 2016 at 10:56 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
>
Subject: [openstack-dev] [nova] VIF plugin issue _get_neutron_events

Hi
When booting Vms with 3 interfaces and when port binding happens quick enough I 
sometimes see that the first VM booted on a compute node gets stuck waiting for 
VIF plugging event but looking at the neutron logs the notification is sent and 
also nova responded with a 200 OK.

When looking at the code in nova/virt/libvirt/driver.py we see in spawn() there 
is a call to _create_domain_and_network which calls _get_neutron_events

So in what scenarios can this notification happen to soon or is there a race 
condition somewhere in this code wherby even though neutron sends the 
notification and nova responds the code here blocks for 300 seconds since it 
missed the event.

Any clues on what maybe going on here. Its almost like the neutron notification 
happened too quickly while nova code was busy performing create_image

This happens only with the first VM on a compute node

Any ideas on how to fix this

Ajay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] VIF plugin issue _get_neutron_events

[Ajay Kalambur (akalambu)]

Hi
When booting Vms with 3 interfaces and when port binding happens quick enough I 
sometimes see that the first VM booted on a compute node gets stuck waiting for 
VIF plugging event but looking at the neutron logs the notification is sent and 
also nova responded with a 200 OK.

When looking at the code in nova/virt/libvirt/driver.py we see in spawn() there 
is a call to _create_domain_and_network which calls _get_neutron_events

So in what scenarios can this notification happen to soon or is there a race 
condition somewhere in this code wherby even though neutron sends the 
notification and nova responds the code here blocks for 300 seconds since it 
missed the event.

Any clues on what maybe going on here. Its almost like the neutron notification 
happened too quickly while nova code was busy performing create_image

This happens only with the first VM on a compute node

Any ideas on how to fix this

Ajay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] port delete allowed on VM

[Ajay Kalambur (akalambu)]

Hi
Today when we create a VM on a port and delete that port I don’t get a message 
saying Port in Use

Is there a plan to fix this or is this expected behavior in neutron

Is there a plan to fix this and if so is there a bug tracking this?

Ajay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [manuals] Training guide issue

[Ajay Kalambur (akalambu)]

Hi
I am trying to just get started with  openstack commits and wanted to start by 
fixing some documentation bugs. I assigned 3 bugs which seem to be in the same 
file/area

https://bugs.launchpad.net/openstack-training-guides/+bug/1380153
https://bugs.launchpad.net/openstack-training-guides/+bug/1380155
https://bugs.launchpad.net/openstack-training-guides/+bug/1380156


The file seems to be located under the openstack-manuals branch since I found 
this xml file there
But the bug seems to be under Openstack Training guides which seems to be a 
different git repo with this file not present there

Can someone help me understand whats going on here?
Ajay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [rally] Rally scenario for network scale with VMs

[Ajay Kalambur (akalambu)]

I created a new scenario which scales on network and VM at same time.
If you have no objection I would like to send out a review this week
I actually have following reviews to do next week

  1.  Scenario for network stress I.e VM+ network with subnet with unique cidr. 
So in future we can add a router to this and do apt-get update from all Vms and 
test network scale
  2.  Scenario for booting Vms on every compute node in the cloud….This has a 
dependency right now this is admin access. So for this I need Item 3
  3.  Provide an ability to make rally created users have admin access for 
things like forced host scheduling . Planning to add this to user context
  4.  Iperf scenario we discussed

If you have objection to these I can submit reviews for these. Have the code 
need to write unit tests for the scenarios since looking at other reviews that 
seems to be the case

Ajay


From: Boris Pavlovic bo...@pavlovic.memailto:bo...@pavlovic.me
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Thursday, December 18, 2014 at 2:16 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [rally] Rally scenario for network scale with VMs

Ajay,

Sorry for long reply.

At this point Rally supports only benchmarking from temporary created users and 
tenants.

Fortunately today we merged this Network context class: 
https://review.openstack.org/#/c/103306/96
it creates any amount of networks for each rally temporary tenant.

So basically you can use it and extend current benchmark scenarios in 
rally/benchmark/scenarios/nova/ or add new one that will attach N networks to 
created VM (which is just few lines of code). So task is quite easy resolvable 
now.


Best regards,
Boris Pavlovic


On Wed, Nov 26, 2014 at 9:54 PM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
Is there  a Rally scenario under works where we create N networks and associate 
N Vms with each network.
This would be a decent stress tests of neutron
Is there any such scale scenario in works
I see scenario for N networks, subnet creation and a separate one for N VM 
bootups
I am looking for an integration of these 2



Ajay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [rally] Rally scenario for network scale with VMs

[Ajay Kalambur (akalambu)]

Ok ill get them out right away than . You should see some reviews coming in 
next week
I will try the admin=True option and see if that works for me


Ajay


From: Boris Pavlovic bpavlo...@mirantis.commailto:bpavlo...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Thursday, December 18, 2014 at 4:40 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [rally] Rally scenario for network scale with VMs

Ajay,

Oh looks you are working hard on Rally. I am excited about your patches.

By the way, one small advice is to share results and ideas that you have with 
community ASAP.
it's perfectly to publish not ideal patches on review even if they don't have 
unit tests and doesn't work at all.
Because community can help with advices and safe a lot of your time.


2. Scenario for booting Vms on every compute node in the cloud….This has a 
dependency right now this is admin access. So for this I need Item 3
3. Provide an ability to make rally created users have admin access for things 
like forced host scheduling . Planning to add this to user context


Not quite sure that you need point 3. If you put on scenario: 
@validation.required_openstack(admin=True, users=True).
You'll have admin in scenario. So you can execute from him commands. E.g.:

self.admin_clients(neutron) (it's similar to self.clients but with admin 
access)

That's how live_migrate benchmark is implemented:
https://github.com/stackforge/rally/blob/master/rally/benchmark/scenarios/nova/servers.py#L253

 Does this make sense?


4. Iperf scenario we discussed

Nice!


Best regards,
Boris Pavlovic






On Fri, Dec 19, 2014 at 4:06 AM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
I created a new scenario which scales on network and VM at same time.
If you have no objection I would like to send out a review this week
I actually have following reviews to do next week

  1.  Scenario for network stress I.e VM+ network with subnet with unique cidr. 
So in future we can add a router to this and do apt-get update from all Vms and 
test network scale
  2.  Scenario for booting Vms on every compute node in the cloud….This has a 
dependency right now this is admin access. So for this I need Item 3
  3.  Provide an ability to make rally created users have admin access for 
things like forced host scheduling . Planning to add this to user context
  4.  Iperf scenario we discussed

If you have objection to these I can submit reviews for these. Have the code 
need to write unit tests for the scenarios since looking at other reviews that 
seems to be the case

Ajay


From: Boris Pavlovic bo...@pavlovic.memailto:bo...@pavlovic.me
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Thursday, December 18, 2014 at 2:16 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [rally] Rally scenario for network scale with VMs

Ajay,

Sorry for long reply.

At this point Rally supports only benchmarking from temporary created users and 
tenants.

Fortunately today we merged this Network context class: 
https://review.openstack.org/#/c/103306/96
it creates any amount of networks for each rally temporary tenant.

So basically you can use it and extend current benchmark scenarios in 
rally/benchmark/scenarios/nova/ or add new one that will attach N networks to 
created VM (which is just few lines of code). So task is quite easy resolvable 
now.


Best regards,
Boris Pavlovic


On Wed, Nov 26, 2014 at 9:54 PM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
Is there  a Rally scenario under works where we create N networks and associate 
N Vms with each network.
This would be a decent stress tests of neutron
Is there any such scale scenario in works
I see scenario for N networks, subnet creation and a separate one for N VM 
bootups
I am looking for an integration of these 2



Ajay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [rally] Rally scenario for network scale with VMs

[Ajay Kalambur (akalambu)]

Hi
Is there  a Rally scenario under works where we create N networks and associate 
N Vms with each network.
This would be a decent stress tests of neutron
Is there any such scale scenario in works
I see scenario for N networks, subnet creation and a separate one for N VM 
bootups
I am looking for an integration of these 2



Ajay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Rally] Question on periodic task

[Ajay Kalambur (akalambu)]

Hi Boris
Looks like this would require changes in key portions of Rally infra. Need some 
more time getting a hang of rally by committing a few scenarios before I make 
infra changes


Ajay


From: Boris Pavlovic bo...@pavlovic.memailto:bo...@pavlovic.me
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, November 21, 2014 at 7:02 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Rally] Question on periodic task

Ajay,


We have this in our RoadMap:
https://github.com/stackforge/rally/blob/master/doc/feature_request/multi_scenarios_load_gen.rst

So, it's not yet supported out of box, but we really would like to have it in 
upstream.

Are you interested in work on this direction?


Best regards,
Boris Pavlovic


On Fri, Nov 21, 2014 at 8:22 AM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Ok the action I wanted to perform was for HA I.e execute a scenario like VM 
boot and in parallel in a separate process , ssh and restart controller node 
for instance
I thought periodic task would be useful for that. I guess I need to look at 
some other way of performing this
Ajay


From: Boris Pavlovic bpavlo...@mirantis.commailto:bpavlo...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Thursday, November 20, 2014 at 7:03 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Rally] Question on periodic task

Hi Ajay,


I am not sure why you are looking that part at all.
everything in openstack/common/* is oslo-incubator code.
Actually that method is not used in Rally yet, except Rally as a Service part 
that doesn't work yet.

As a scenario developer I think you should be able to find everything here:
https://github.com/stackforge/rally/tree/master/rally/benchmark

So I really don't the case when you need to pass something to periodic task.. 
It's not that task


Best regards,
Boris Pavlovic






On Fri, Nov 21, 2014 at 3:36 AM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
I have a question on
/rally/openstack/common/periodic_task.py

It looks like if I have a method decorated with @periodic_task my method would 
get scheduled in separate process every N seconds
Now let us say we have a scenario and this periodic_task how does it work when 
concurrency=2 for instance

Is the periodic task also scheduled in 2 separate process. I actually want only 
one periodic task process irrespective of concurrency count in scenario
Also as a scenario developer how can I pass arguments into the periodic task


Ajay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Rally] Question on periodic task

[Ajay Kalambur (akalambu)]

Hi
I have a question on
/rally/openstack/common/periodic_task.py

It looks like if I have a method decorated with @periodic_task my method would 
get scheduled in separate process every N seconds
Now let us say we have a scenario and this periodic_task how does it work when 
concurrency=2 for instance

Is the periodic task also scheduled in 2 separate process. I actually want only 
one periodic task process irrespective of concurrency count in scenario
Also as a scenario developer how can I pass arguments into the periodic task


Ajay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Rally] Question on periodic task

[Ajay Kalambur (akalambu)]

Ok the action I wanted to perform was for HA I.e execute a scenario like VM 
boot and in parallel in a separate process , ssh and restart controller node 
for instance
I thought periodic task would be useful for that. I guess I need to look at 
some other way of performing this
Ajay


From: Boris Pavlovic bpavlo...@mirantis.commailto:bpavlo...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Thursday, November 20, 2014 at 7:03 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Rally] Question on periodic task

Hi Ajay,


I am not sure why you are looking that part at all.
everything in openstack/common/* is oslo-incubator code.
Actually that method is not used in Rally yet, except Rally as a Service part 
that doesn't work yet.

As a scenario developer I think you should be able to find everything here:
https://github.com/stackforge/rally/tree/master/rally/benchmark

So I really don't the case when you need to pass something to periodic task.. 
It's not that task


Best regards,
Boris Pavlovic






On Fri, Nov 21, 2014 at 3:36 AM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
I have a question on
/rally/openstack/common/periodic_task.py

It looks like if I have a method decorated with @periodic_task my method would 
get scheduled in separate process every N seconds
Now let us say we have a scenario and this periodic_task how does it work when 
concurrency=2 for instance

Is the periodic task also scheduled in 2 separate process. I actually want only 
one periodic task process irrespective of concurrency count in scenario
Also as a scenario developer how can I pass arguments into the periodic task


Ajay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [rally]Rally created tenant and users

[Ajay Kalambur (akalambu)]

Hi
Is there a way to specify that the Rally created tenants and users and created 
with admin privileges
Currently its created using member role and hence some admin operations are not 
allowed
I want to specify that the account created has admin access
Ajay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Rally] Rally scenario question

[Ajay Kalambur (akalambu)]

Hi
In rally I have 2 questions. When you use a context like example
users: 2, tenant:2 I want the users and tenants to not be cleaned up but 
accumulated and cleaned up at end.
Example create 1 user 1 tenant run tests create second user tenant run test and 
finally clean up user and tenant

Similarly when you specify
times: 100
concurrency: 10
I want it to launch 10 at a time but not clean them up immediately and only 
clean up at end so 100 instances are active at end before clean up

What changes would be needed to Rally to support this. Is there also a 
temporary way to do this if not present

Ajay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [rally][users]

[Ajay Kalambur (akalambu)]

Hi Boris
Existing users is one thing but according to Rally page it says admin account 
benchmarking is already supported

Rally is on its way to support of benchmarking OpenStack clouds using ordinary 
user accounts that already exist. Rally lacked such functionality (it only 
supported benchmarking either from an admin account or from a bunch of 
temporarily created users), which posed a problem since some deployments don't 
allow temporary users creation. There have been 
twohttps://review.openstack.org/#/c/116766/ 
patcheshttps://review.openstack.org/#/c/119344/ that prepare the code for 
this new functionality. It is going to come very soon - stay tuned.


Ajay

From: Boris Pavlovic bpavlo...@mirantis.commailto:bpavlo...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Wednesday, September 24, 2014 at 6:13 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [rally][users]

Ajay,

I am working that feature. It's almost ready.
I'll let you know when I finish.


Best regards,
Boris Pavlovic

On Thu, Sep 25, 2014 at 5:02 AM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
Our default mode of execution of rally is allowing Rally to create a new user 
and tenant. Is there a way to have rally use the existing admin tenant and user.
I need to use Rally for some tests which would need a admin access so I would 
like Rally to use existing admin tenant and admin user for tests
Ajay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [rally][iperf] Benchmarking network performance

[Ajay Kalambur (akalambu)]

The following context worked for me.


context: {
neutron_network: {
network_cidr: 10.%s.0.0/16,
},
users: {
tenants: 1,
users_per_tenant: 2
}
}



From: masoom alam masoom.a...@gmail.commailto:masoom.a...@gmail.com
Date: Sunday, September 7, 2014 at 5:27 AM
To: akalambu akala...@cisco.commailto:akala...@cisco.com
Cc: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [rally][iperf] Benchmarking network performance

The problem lies in this patch:

https://review.openstack.org/#/c/96300

Even If I apply it, i get an error that Unknown Neutron context. The patch is 
correctly applied - some 20 times :)

Task and output is as follows:


{

VMTasks.boot_runcommand_delete: [

{

args: {

flavor: {

name: m1.tiny

},

image: {

name: cirros-0.3.2-x86_64-uec

},

fixed_network: net04,

floating_network: net04_ext,

use_floatingip: true,

script: 
/home/alam/Desktop/rally/doc/samples/tasks/support/instance_dd_test.sh,

interpreter: /bin/sh,

username: cirros

},

runner: {

type: constant,

times: 2,

concurrency: 1

},

context: {

users: {

tenants: 1,

users_per_tenant: 1

},
neutron_network: {
network_cidr: 10.%s.0.0/16
}

}



}

]

}




$rally -v task start 
/home/alam/Desktop/rally/doc/samples/tasks/scenarios/vm/boot-runcommand-delete.json

Task  193a4b11-ec2d-4e36-ba53-23819e9d6bcf is started

2014-09-07 17:23:00.680 2845 INFO rally.orchestrator.api [-] Benchmark Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf on Deployment 
3cba9ee5-ef47-42f9-95bc-91107009a348
2014-09-07 17:23:00.680 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Starting:  Check cloud.
2014-09-07 17:23:09.083 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Completed: Check cloud.
2014-09-07 17:23:09.084 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Starting:  Task validation.
2014-09-07 17:23:09.134 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Starting:  Task validation of scenarios 
names.
2014-09-07 17:23:09.137 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Completed: Task validation of scenarios 
names.
2014-09-07 17:23:09.138 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Starting:  Task validation of syntax.


Task 193a4b11-ec2d-4e36-ba53-23819e9d6bcf is failed.

class 'rally.exceptions.InvalidBenchmarkConfig'
Task config is invalid.
Benchmark %(name)s has wrong configuration at position %(pos)s
Reason: %(reason)s
Benchmark configuration: %(config)s





On Fri, Sep 5, 2014 at 7:46 PM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi mason
What is the task you want to perform run commands after vm boot or run 
performance
Based on that I can help with correct pointer
Ajay

Sent from my iPhone

On Sep 5, 2014, at 2:28 AM, masoom alam 
masoom.a...@gmail.commailto:masoom.a...@gmail.com wrote:

Please forward ur vmtasks.py file

On Friday, September 5, 2014, masoom alam 
masoom.a...@gmail.commailto:masoom.a...@gmail.com wrote:
http://paste.openstack.org/show/106297/


On Fri, Sep 5, 2014 at 1:12 PM, masoom alam masoom.a...@gmail.com wrote:
Thanks Ajay

I corrected this earlier. But facing another problem. Will forward paste in a 
while.



On Friday, September 5, 2014, Ajay Kalambur (akalambu) akala...@cisco.com 
wrote:
Sorry there was  typo in the patch should be @validation and not @(validation
Please change that in vm_perf.py

Sent from my iPhone

On Sep 4, 2014, at 7:51 PM, masoom alam masoom.a...@gmail.com wrote:

Why this is so when I patched with your sent patch:

http://paste.openstack.org/show/106196/


On Thu, Sep 4, 2014 at 8:58 PM, Rick Jones rick.jon...@hp.com wrote:
On 09/03/2014 11:47 AM, Ajay Kalambur (akalambu) wrote:
Hi
Looking into the following blueprint which requires that network
performance tests be done as part of a scenario
I plan

Re: [openstack-dev] [rally][iperf] Benchmarking network performance

[Ajay Kalambur (akalambu)]

Hi Boris
It worked for me see no error

Ajay


Sent from my iPhone

On Sep 7, 2014, at 12:11 PM, Boris Pavlovic 
bo...@pavlovic.memailto:bo...@pavlovic.me wrote:

Ajay,

Seems like you are using old rally.
Because it should show detailed information about error.
Recently we merged this: https://review.openstack.org/#/c/118169/ that shows 
full information.


Could you send the set of commands that you run to apply neutron context?


Best regards,
Boris Pavlovic


On Sun, Sep 7, 2014 at 10:57 PM, Boris Pavlovic 
bo...@pavlovic.memailto:bo...@pavlovic.me wrote:
Massoom,

Seems like you are using old rally code.


On Sun, Sep 7, 2014 at 10:33 PM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
The following context worked for me.


context: {
neutron_network: {
network_cidr: 10.%s.0.0/16,
},
users: {
tenants: 1,
users_per_tenant: 2
}
}



From: masoom alam masoom.a...@gmail.commailto:masoom.a...@gmail.com
Date: Sunday, September 7, 2014 at 5:27 AM
To: akalambu akala...@cisco.commailto:akala...@cisco.com
Cc: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [rally][iperf] Benchmarking network performance

The problem lies in this patch:

https://review.openstack.org/#/c/96300

Even If I apply it, i get an error that Unknown Neutron context. The patch is 
correctly applied - some 20 times :)

Task and output is as follows:


{

VMTasks.boot_runcommand_delete: [

{

args: {

flavor: {

name: m1.tiny

},

image: {

name: cirros-0.3.2-x86_64-uec

},

fixed_network: net04,

floating_network: net04_ext,

use_floatingip: true,

script: 
/home/alam/Desktop/rally/doc/samples/tasks/support/instance_dd_test.sh,

interpreter: /bin/sh,

username: cirros

},

runner: {

type: constant,

times: 2,

concurrency: 1

},

context: {

users: {

tenants: 1,

users_per_tenant: 1

},
neutron_network: {
network_cidr: 10.%s.0.0/16
}

}



}

]

}




$rally -v task start 
/home/alam/Desktop/rally/doc/samples/tasks/scenarios/vm/boot-runcommand-delete.json

Task  193a4b11-ec2d-4e36-ba53-23819e9d6bcf is started

2014-09-07 17:23:00.680 2845 INFO rally.orchestrator.api [-] Benchmark Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf on Deployment 
3cba9ee5-ef47-42f9-95bc-91107009a348
2014-09-07 17:23:00.680 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Starting:  Check cloud.
2014-09-07 17:23:09.083 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Completed: Check cloud.
2014-09-07 17:23:09.084 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Starting:  Task validation.
2014-09-07 17:23:09.134 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Starting:  Task validation of scenarios 
names.
2014-09-07 17:23:09.137 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Completed: Task validation of scenarios 
names.
2014-09-07 17:23:09.138 2845 INFO rally.benchmark.engine [-] Task 
193a4b11-ec2d-4e36-ba53-23819e9d6bcf | Starting:  Task validation of syntax.


Task 193a4b11-ec2d-4e36-ba53-23819e9d6bcf is failed.

class 'rally.exceptions.InvalidBenchmarkConfig'
Task config is invalid.
Benchmark %(name)s has wrong configuration at position %(pos)s
Reason: %(reason)s
Benchmark configuration: %(config)s





On Fri, Sep 5, 2014 at 7:46 PM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi mason
What is the task you want to perform run commands after vm boot or run 
performance
Based on that I can help with correct pointer
Ajay

Sent from my iPhone

On Sep 5, 2014, at 2:28 AM, masoom alam 
masoom.a...@gmail.commailto:masoom.a...@gmail.com wrote:

Please forward ur vmtasks.py file

On Friday, September 5, 2014, masoom alam 
masoom.a...@gmail.commailto:masoom.a...@gmail.com wrote:
http://paste.openstack.org/show/106297/


On Fri, Sep 5, 2014 at 1:12 PM, masoom alam masoom.a...@gmail.com wrote

Re: [openstack-dev] [rally][iperf] Benchmarking network performance

[Ajay Kalambur (akalambu)]

You need to apply the other 2 network context patches from the reviews I sent 
out
Ajay


Sent from my iPhone

On Sep 5, 2014, at 2:25 AM, masoom alam 
masoom.a...@gmail.commailto:masoom.a...@gmail.com wrote:

http://paste.openstack.org/show/106297/


On Fri, Sep 5, 2014 at 1:12 PM, masoom alam 
masoom.a...@gmail.commailto:masoom.a...@gmail.com wrote:
Thanks Ajay

I corrected this earlier. But facing another problem. Will forward paste in a 
while.



On Friday, September 5, 2014, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Sorry there was  typo in the patch should be @validation and not @(validation
Please change that in vm_perf.py

Sent from my iPhone

On Sep 4, 2014, at 7:51 PM, masoom alam masoom.a...@gmail.com wrote:

Why this is so when I patched with your sent patch:

http://paste.openstack.org/show/106196/


On Thu, Sep 4, 2014 at 8:58 PM, Rick Jones rick.jon...@hp.com wrote:
On 09/03/2014 11:47 AM, Ajay Kalambur (akalambu) wrote:
Hi
Looking into the following blueprint which requires that network
performance tests be done as part of a scenario
I plan to implement this using iperf and basically a scenario which
includes a client/server VM pair

My experience with netperf over the years has taught me that when there is just 
the single stream and pair of systems one won't actually know if the 
performance was limited by inbound, or outbound.  That is why the likes of

http://www.netperf.org/svn/netperf2/trunk/doc/examples/netperf_by_flavor.py

and

http://www.netperf.org/svn/netperf2/trunk/doc/examples/netperf_by_quantum.py

apart from being poorly written python :)  Will launch several instances of a 
given flavor and then run aggregate tests on the Instance Under Test.  Those 
aggregate tests will include inbound, outbound, bidirectional, aggregate small 
packet and then a latency test.

happy benchmarking,

rick jones


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Sent from noir

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [rally] Parallel scenarios

[Ajay Kalambur (akalambu)]

Hi
In Rally is there a way to run parallel scenarios?
I know the scenario can support concurrency but I am talking about running 
multiple such scenarios themselves in parallel
Ajay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [rally][iperf] Benchmarking network performance

[Ajay Kalambur (akalambu)]

Sorry there was  typo in the patch should be @validation and not @(validation
Please change that in vm_perf.py

Sent from my iPhone

On Sep 4, 2014, at 7:51 PM, masoom alam 
masoom.a...@gmail.commailto:masoom.a...@gmail.com wrote:

Why this is so when I patched with your sent patch:

http://paste.openstack.org/show/106196/


On Thu, Sep 4, 2014 at 8:58 PM, Rick Jones 
rick.jon...@hp.commailto:rick.jon...@hp.com wrote:
On 09/03/2014 11:47 AM, Ajay Kalambur (akalambu) wrote:
Hi
Looking into the following blueprint which requires that network
performance tests be done as part of a scenario
I plan to implement this using iperf and basically a scenario which
includes a client/server VM pair

My experience with netperf over the years has taught me that when there is just 
the single stream and pair of systems one won't actually know if the 
performance was limited by inbound, or outbound.  That is why the likes of

http://www.netperf.org/svn/netperf2/trunk/doc/examples/netperf_by_flavor.py

and

http://www.netperf.org/svn/netperf2/trunk/doc/examples/netperf_by_quantum.py

apart from being poorly written python :)  Will launch several instances of a 
given flavor and then run aggregate tests on the Instance Under Test.  Those 
aggregate tests will include inbound, outbound, bidirectional, aggregate small 
packet and then a latency test.

happy benchmarking,

rick jones


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Rally scenario Issue

[Ajay Kalambur (akalambu)]

The reason this is failing is you are specifying a fixed network and at the 
same time asking for a new network to be created using context. What happens is 
the VM gets attached to your fixed network instead of the created Rally network
So you need to modify vm_tasks.py for this case and basically if not fixed and 
using floating ip just skip the network check and the fixed ip code and just 
directly use floating ip to access.
I think some changes are needed to vm_tasks.py to make it work for both fixed 
and floating and support for network context


If you need a local change I can pass along a local change for vm_tasks.py that 
only support floating ip. Let me know do you have floating ips available ?
Ajay


From: masoom alam masoom.a...@gmail.commailto:masoom.a...@gmail.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Tuesday, September 2, 2014 at 11:12 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] Rally scenario Issue

Hi Ajay,

We are testing the same scenario that you are working one, but getting the 
follow error:

http://paste.openstack.org/show/105029/

Could you be of any help here?

Thanks




On Wed, Sep 3, 2014 at 4:16 AM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi Guys
For the throughput tests I need to be able to install iperf on the cloud image. 
For this DNS server needs to be set. But the current network context should 
also support DNS name server setting
Should we add that into network context?
Ajay



From: Boris Pavlovic bo...@pavlovic.memailto:bo...@pavlovic.me

Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, August 29, 2014 at 2:08 PM

To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Cc: Harshil Shah (harsshah) harss...@cisco.commailto:harss...@cisco.com
Subject: Re: [openstack-dev] Rally scenario Issue

Timur,

Thanks for pointing Ajay.

Ajay,

 Also I cannot see this failure unless I run rally with –v –d object.

Actually rally is sotring information about all failures. To get information 
about them you can run next command:

rally task results --pprint

It will display all information about all iterations (including exceptions)


Second when most of the steps in the scenario failed like attaching to network, 
ssh and run command why bother reporting the results

Because, bad results are better then nothing...


Best regards,
Boris Pavlovic


On Sat, Aug 30, 2014 at 12:54 AM, Timur Nurlygayanov 
tnurlygaya...@mirantis.commailto:tnurlygaya...@mirantis.com wrote:
Hi Ajay,

looks like you need to use NeutronContext feature to configure Neutron Networks 
during the benchmarks execution.
We now working on merge of two different comits with NeutronContext 
implementation:
https://review.openstack.org/#/c/96300  and 
https://review.openstack.org/#/c/103306

could you please apply commit https://review.openstack.org/#/c/96300 and run 
your benchmarks? Neutron Network with subnetworks and routers will be 
automatically created for each created tenant and you should have the ability 
to connect to VMs. Please, note, that you should add the following part to your 
task JSON to enable Neutron context:
...
context: {
...
neutron_network: {
network_cidr: 10.%s.0.0/16,
}
}
...

Hope this will help.



On Fri, Aug 29, 2014 at 11:42 PM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
I am trying to run the Rally scenario boot-runcommand-delete. This scenario has 
the following code
 def boot_runcommand_delete(self, image, flavor,
   script, interpreter, username,
   fixed_network=private,
   floating_network=public,
   ip_version=4, port=22,
   use_floatingip=True, **kwargs):
  server = None
floating_ip = None
try:
print fixed network:%s floating network:%s 
%(fixed_network,floating_network)
server = self._boot_server(
self._generate_random_name(rally_novaserver_),
image, flavor, key_name='rally_ssh_key', **kwargs)

self.check_network(server, fixed_network)

The question I have is the instance is created with a call to boot_server but 
no networks are attached to this server instance. Next step it goes and checks 
if the fixed network is attached to the instance and sure enough it fails
At the step highlighted in bold. Also I cannot see this failure unless I run 
rally with –v –d object. So it actually reports benchmark scenario numbers in a 
table

[openstack-dev] [rally][iperf] Benchmarking network performance

[Ajay Kalambur (akalambu)]

Hi
Looking into the following blueprint which requires that network performance 
tests be done as part of a scenario
I plan to implement this using iperf and basically a scenario which includes a 
client/server VM pair

The client than sends out TCP traffic using iperf to server and the VM 
throughput is recorded
I have a WIP patch attached to this email

The patch has a dependency on following 2 review
https://review.openstack.org/#/c/103306/
 https://review.openstack.org/#/c/96300https://review.openstack.org/#/c/96300




On top of this it creates a new VM performance scenario and uses floating ips 
to access the VM and download iperf to the VM and than run throughout tests
The code will  be made more modular but this patch will give you a good idea of 
whats in store.
We also need to handle the case next where no floating ip is available and we 
assume direct access. We need to have ssh to install the tool and drive the 
tests

Please look at the attached diff and let me know if overall the flow looks fine
If it does I can make the code more modular and proceed. Note this is Work in 
progress still


Ajay



rally_perf.diff
Description: rally_perf.diff
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [rally]Rally scenario Issue

[Ajay Kalambur (akalambu)]

Does this look right.


{

VMTasks.boot_runcommand_delete: [

{

args: {

flavor: {

name: m1.small

},

image: {

name: Ubuntu Server 14.04

},

fixed_network: net04,

floating_network: net04_ext,

use_floatingip: true,

script: doc/samples/tasks/support/instance_dd_test.sh,

interpreter: /bin/sh,

username: ubuntu

},

runner: {

type: constant,

times: 10,

concurrency: 2

},

context: {

users: {

tenants: 1,

users_per_tenant: 1

}

}

neutron_network: {

network_cidr: 10.%s.0.0/16,

}

}

]

}

Because I see this error
2014-08-29 14:56:23.769 5495 TRACE rally expected ',' or '}', but got %r 
% token.id, token.start_mark)
2014-08-29 14:56:23.769 5495 TRACE rally ParserError: while parsing a flow 
mapping
2014-08-29 14:56:23.769 5495 TRACE rally   in string, line 3, column 9:
2014-08-29 14:56:23.769 5495 TRACE rally {
2014-08-29 14:56:23.769 5495 TRACE rally ^
2014-08-29 14:56:23.769 5495 TRACE rally expected ',' or '}', but got 'scalar'
2014-08-29 14:56:23.769 5495 TRACE rally   in string, line 29, column 13:
2014-08-29 14:56:23.769 5495 TRACE rally neutron_network: {
2014-08-29 14:56:23.769 5495 TRACE rally ^
2014-08-29 14:56:23.769 5495 TRACE rally


Ajay


From: Timur Nurlygayanov 
tnurlygaya...@mirantis.commailto:tnurlygaya...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, August 29, 2014 at 1:54 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Cc: Harshil Shah (harsshah) harss...@cisco.commailto:harss...@cisco.com
Subject: Re: [openstack-dev] Rally scenario Issue

Hi Ajay,

looks like you need to use NeutronContext feature to configure Neutron Networks 
during the benchmarks execution.
We now working on merge of two different comits with NeutronContext 
implementation:
https://review.openstack.org/#/c/96300  and 
https://review.openstack.org/#/c/103306

could you please apply commit https://review.openstack.org/#/c/96300 and run 
your benchmarks? Neutron Network with subnetworks and routers will be 
automatically created for each created tenant and you should have the ability 
to connect to VMs. Please, note, that you should add the following part to your 
task JSON to enable Neutron context:
...
context: {
...
neutron_network: {
network_cidr: 10.%s.0.0/16,
}
}
...

Hope this will help.



On Fri, Aug 29, 2014 at 11:42 PM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
I am trying to run the Rally scenario boot-runcommand-delete. This scenario has 
the following code
 def boot_runcommand_delete(self, image, flavor,
   script, interpreter, username,
   fixed_network=private,
   floating_network=public,
   ip_version=4, port=22,
   use_floatingip=True, **kwargs):
  server = None
floating_ip = None
try:
print fixed network:%s floating network:%s 
%(fixed_network,floating_network)
server = self._boot_server(
self._generate_random_name(rally_novaserver_),
image, flavor, key_name='rally_ssh_key', **kwargs)

self.check_network(server, fixed_network)

The question I have is the instance is created with a call to boot_server but 
no networks are attached to this server instance. Next step it goes and checks 
if the fixed network is attached to the instance and sure enough it fails
At the step highlighted in bold. Also I cannot see this failure unless I run 
rally with –v –d object. So it actually reports benchmark scenario numbers in a 
table with no errors when I run with
rally task start boot-and-delete.json

And reports results. First what am I missing in this case. Thing is I am using 
neutron not nova-network
Second when most of the steps in the scenario failed like attaching to network, 
ssh and run command why bother reporting the results

Ajay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--

Timur,
QA Engineer
OpenStack Projects
Mirantis Inc

[http://www.openstacksv.com/]http://www.openstacksv.com

Re: [openstack-dev] [rally]Rally scenario Issue

[Ajay Kalambur (akalambu)]

Sorry here is the context


context: {

users: {

tenants: 1,

users_per_tenant: 1

}

neutron_network: {

network_cidr: 10.%s.0.0/16,

}

}

Because I see this error
2014-08-29 14:56:23.769 5495 TRACE rally expected ',' or '}', but got %r 
% token.id, token.start_mark)
2014-08-29 14:56:23.769 5495 TRACE rally ParserError: while parsing a flow 
mapping
2014-08-29 14:56:23.769 5495 TRACE rally   in string, line 3, column 9:
2014-08-29 14:56:23.769 5495 TRACE rally {
2014-08-29 14:56:23.769 5495 TRACE rally ^
2014-08-29 14:56:23.769 5495 TRACE rally expected ',' or '}', but got 'scalar'
2014-08-29 14:56:23.769 5495 TRACE rally   in string, line 29, column 13:
2014-08-29 14:56:23.769 5495 TRACE rally neutron_network: {
2014-08-29 14:56:23.769 5495 TRACE rally ^
2014-08-29 14:56:23.769 5495 TRACE rally


Ajay


From: Timur Nurlygayanov 
tnurlygaya...@mirantis.commailto:tnurlygaya...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, August 29, 2014 at 1:54 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Cc: Harshil Shah (harsshah) harss...@cisco.commailto:harss...@cisco.com
Subject: Re: [openstack-dev] Rally scenario Issue

Hi Ajay,

looks like you need to use NeutronContext feature to configure Neutron Networks 
during the benchmarks execution.
We now working on merge of two different comits with NeutronContext 
implementation:
https://review.openstack.org/#/c/96300  and 
https://review.openstack.org/#/c/103306

could you please apply commit https://review.openstack.org/#/c/96300 and run 
your benchmarks? Neutron Network with subnetworks and routers will be 
automatically created for each created tenant and you should have the ability 
to connect to VMs. Please, note, that you should add the following part to your 
task JSON to enable Neutron context:
...
context: {
...
neutron_network: {
network_cidr: 10.%s.0.0/16,
}
}
...

Hope this will help.



On Fri, Aug 29, 2014 at 11:42 PM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
I am trying to run the Rally scenario boot-runcommand-delete. This scenario has 
the following code
 def boot_runcommand_delete(self, image, flavor,
   script, interpreter, username,
   fixed_network=private,
   floating_network=public,
   ip_version=4, port=22,
   use_floatingip=True, **kwargs):
  server = None
floating_ip = None
try:
print fixed network:%s floating network:%s 
%(fixed_network,floating_network)
server = self._boot_server(
self._generate_random_name(rally_novaserver_),
image, flavor, key_name='rally_ssh_key', **kwargs)

self.check_network(server, fixed_network)

The question I have is the instance is created with a call to boot_server but 
no networks are attached to this server instance. Next step it goes and checks 
if the fixed network is attached to the instance and sure enough it fails
At the step highlighted in bold. Also I cannot see this failure unless I run 
rally with –v –d object. So it actually reports benchmark scenario numbers in a 
table with no errors when I run with
rally task start boot-and-delete.json

And reports results. First what am I missing in this case. Thing is I am using 
neutron not nova-network
Second when most of the steps in the scenario failed like attaching to network, 
ssh and run command why bother reporting the results

Ajay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--

Timur,
QA Engineer
OpenStack Projects
Mirantis Inc

[http://www.openstacksv.com/]http://www.openstacksv.com/
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [rally]Rally scenario Issue

[Ajay Kalambur (akalambu)]

Issue fixed small syntax mistake in scenario file
Will now look into more details
Thx


From: akalambu akala...@cisco.commailto:akala...@cisco.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, August 29, 2014 at 3:03 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [rally]Rally scenario Issue

Sorry here is the context


context: {

users: {

tenants: 1,

users_per_tenant: 1

}

neutron_network: {

network_cidr: 10.%s.0.0/16,

}

}

Because I see this error
2014-08-29 14:56:23.769 5495 TRACE rally expected ',' or '}', but got %r 
% token.id, token.start_mark)
2014-08-29 14:56:23.769 5495 TRACE rally ParserError: while parsing a flow 
mapping
2014-08-29 14:56:23.769 5495 TRACE rally   in string, line 3, column 9:
2014-08-29 14:56:23.769 5495 TRACE rally {
2014-08-29 14:56:23.769 5495 TRACE rally ^
2014-08-29 14:56:23.769 5495 TRACE rally expected ',' or '}', but got 'scalar'
2014-08-29 14:56:23.769 5495 TRACE rally   in string, line 29, column 13:
2014-08-29 14:56:23.769 5495 TRACE rally neutron_network: {
2014-08-29 14:56:23.769 5495 TRACE rally ^
2014-08-29 14:56:23.769 5495 TRACE rally


Ajay


From: Timur Nurlygayanov 
tnurlygaya...@mirantis.commailto:tnurlygaya...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, August 29, 2014 at 1:54 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Cc: Harshil Shah (harsshah) harss...@cisco.commailto:harss...@cisco.com
Subject: Re: [openstack-dev] Rally scenario Issue

Hi Ajay,

looks like you need to use NeutronContext feature to configure Neutron Networks 
during the benchmarks execution.
We now working on merge of two different comits with NeutronContext 
implementation:
https://review.openstack.org/#/c/96300  and 
https://review.openstack.org/#/c/103306

could you please apply commit https://review.openstack.org/#/c/96300 and run 
your benchmarks? Neutron Network with subnetworks and routers will be 
automatically created for each created tenant and you should have the ability 
to connect to VMs. Please, note, that you should add the following part to your 
task JSON to enable Neutron context:
...
context: {
...
neutron_network: {
network_cidr: 10.%s.0.0/16,
}
}
...

Hope this will help.



On Fri, Aug 29, 2014 at 11:42 PM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
I am trying to run the Rally scenario boot-runcommand-delete. This scenario has 
the following code
 def boot_runcommand_delete(self, image, flavor,
   script, interpreter, username,
   fixed_network=private,
   floating_network=public,
   ip_version=4, port=22,
   use_floatingip=True, **kwargs):
  server = None
floating_ip = None
try:
print fixed network:%s floating network:%s 
%(fixed_network,floating_network)
server = self._boot_server(
self._generate_random_name(rally_novaserver_),
image, flavor, key_name='rally_ssh_key', **kwargs)

self.check_network(server, fixed_network)

The question I have is the instance is created with a call to boot_server but 
no networks are attached to this server instance. Next step it goes and checks 
if the fixed network is attached to the instance and sure enough it fails
At the step highlighted in bold. Also I cannot see this failure unless I run 
rally with –v –d object. So it actually reports benchmark scenario numbers in a 
table with no errors when I run with
rally task start boot-and-delete.json

And reports results. First what am I missing in this case. Thing is I am using 
neutron not nova-network
Second when most of the steps in the scenario failed like attaching to network, 
ssh and run command why bother reporting the results

Ajay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--

Timur,
QA Engineer
OpenStack Projects
Mirantis Inc

[http://www.openstacksv.com/]http://www.openstacksv.com/
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Rally scenario Issue

[Ajay Kalambur (akalambu)]

Hi Timur
With this I was able to create networks and attach VM to those networks. Would 
I now be able to ssh to this and run command what I am looking for is a 
unification of boot-runcommand-delete and this neutron_network so create 
network attach to router associate floating ip and ssh to it
Ajay


From: Timur Nurlygayanov 
tnurlygaya...@mirantis.commailto:tnurlygaya...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, August 29, 2014 at 1:54 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Cc: Harshil Shah (harsshah) harss...@cisco.commailto:harss...@cisco.com
Subject: Re: [openstack-dev] Rally scenario Issue

Hi Ajay,

looks like you need to use NeutronContext feature to configure Neutron Networks 
during the benchmarks execution.
We now working on merge of two different comits with NeutronContext 
implementation:
https://review.openstack.org/#/c/96300  and 
https://review.openstack.org/#/c/103306

could you please apply commit https://review.openstack.org/#/c/96300 and run 
your benchmarks? Neutron Network with subnetworks and routers will be 
automatically created for each created tenant and you should have the ability 
to connect to VMs. Please, note, that you should add the following part to your 
task JSON to enable Neutron context:
...
context: {
...
neutron_network: {
network_cidr: 10.%s.0.0/16,
}
}
...

Hope this will help.



On Fri, Aug 29, 2014 at 11:42 PM, Ajay Kalambur (akalambu) 
akala...@cisco.commailto:akala...@cisco.com wrote:
Hi
I am trying to run the Rally scenario boot-runcommand-delete. This scenario has 
the following code
 def boot_runcommand_delete(self, image, flavor,
   script, interpreter, username,
   fixed_network=private,
   floating_network=public,
   ip_version=4, port=22,
   use_floatingip=True, **kwargs):
  server = None
floating_ip = None
try:
print fixed network:%s floating network:%s 
%(fixed_network,floating_network)
server = self._boot_server(
self._generate_random_name(rally_novaserver_),
image, flavor, key_name='rally_ssh_key', **kwargs)

self.check_network(server, fixed_network)

The question I have is the instance is created with a call to boot_server but 
no networks are attached to this server instance. Next step it goes and checks 
if the fixed network is attached to the instance and sure enough it fails
At the step highlighted in bold. Also I cannot see this failure unless I run 
rally with –v –d object. So it actually reports benchmark scenario numbers in a 
table with no errors when I run with
rally task start boot-and-delete.json

And reports results. First what am I missing in this case. Thing is I am using 
neutron not nova-network
Second when most of the steps in the scenario failed like attaching to network, 
ssh and run command why bother reporting the results

Ajay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--

Timur,
QA Engineer
OpenStack Projects
Mirantis Inc

[http://www.openstacksv.com/]http://www.openstacksv.com/
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Neutron permission issue

[Ajay Kalambur (akalambu)]

Hi
As a tenant when I try to create a router and associate a gateway with the 
router as a two step process in Horizon things work fine.
Now when I want to do the same thing through a create router API call with 
request below I get permission denied to create router

{
router:
{
name: another_router,
admin_state_up: true,
external_gateway_info: {
network_id: 3c5bcddd-6af9-4e6b-9c3e-c153e521cab8,
enable_snat: false}
}
}
The network id in both cases is the same. This does not make sense to me


Traceback (most recent call last):

  File vm-tp.py, line 54, in setUp

ext_router = self.net.create_router(CONF.ROUTER_NAME, ext_net['id'])

  File /Users/akalambu/python_venv/latest_code/pns/network.py, line 121, in 
create_router

router = self.neutron_client.create_router(body)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 101, in with_params

ret = self.function(instance, *args, **kwargs)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 398, in create_router

return self.post(self.routers_path, body=body)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 1320, in post

headers=headers, params=params)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 1243, in do_request

self._handle_fault_response(status_code, replybody)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 1211, in _handle_fault_response

exception_handler_v20(status_code, des_error_body)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 68, in exception_handler_v20

status_code=status_code)

Forbidden: Policy doesn't allow create_router to be performed.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Neutron permission issue

[Ajay Kalambur (akalambu)]

Hi
The issue seems to be the following default config in Neutron policy

   create_router:external_gateway_info:enable_snat: rule:admin_only,

update_router:external_gateway_info:enable_snat: rule:admin_only,


Puzzling part is from horizon when I set an external gateway for a router is it 
not the same thing as above. How does it allow it from horizon than?


Ajay


From: Ian Wells (iawells) iawe...@cisco.commailto:iawe...@cisco.com
Date: Friday, July 11, 2014 at 10:56 AM
To: akalambu akala...@cisco.commailto:akala...@cisco.com, OpenStack 
Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Cc: openstack-systems-group(mailer list) 
openstack-systems-gr...@cisco.commailto:openstack-systems-gr...@cisco.com
Subject: Re: Neutron permission issue

Check /etc/neutron/policy.json, but I agree that's weird...
--
Ian.


From: Ajay Kalambur (akalambu) akala...@cisco.commailto:akala...@cisco.com
Date: Friday, 11 July 2014 10:05
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Cc: openstack-systems-group(mailer list) 
openstack-systems-gr...@cisco.commailto:openstack-systems-gr...@cisco.com
Subject: Neutron permission issue

Hi
As a tenant when I try to create a router and associate a gateway with the 
router as a two step process in Horizon things work fine.
Now when I want to do the same thing through a create router API call with 
request below I get permission denied to create router

{
router:
{
name: another_router,
admin_state_up: true,
external_gateway_info: {
network_id: 3c5bcddd-6af9-4e6b-9c3e-c153e521cab8,
enable_snat: false}
}
}
The network id in both cases is the same. This does not make sense to me


Traceback (most recent call last):

  File vm-tp.py, line 54, in setUp

ext_router = self.net.create_router(CONF.ROUTER_NAME, ext_net['id'])

  File /Users/akalambu/python_venv/latest_code/pns/network.py, line 121, in 
create_router

router = self.neutron_client.create_router(body)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 101, in with_params

ret = self.function(instance, *args, **kwargs)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 398, in create_router

return self.post(self.routers_path, body=body)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 1320, in post

headers=headers, params=params)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 1243, in do_request

self._handle_fault_response(status_code, replybody)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 1211, in _handle_fault_response

exception_handler_v20(status_code, des_error_body)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 68, in exception_handler_v20

status_code=status_code)

Forbidden: Policy doesn't allow create_router to be performed.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Neutron permission issue

[Ajay Kalambur (akalambu)]

Never mind figured it out the rule is on enable_snat inside external gateway 
info that was the issue
But I think there is an issue with update because the message is misleading 
when I try to update with external gateway info and enable_snat. I get a 
message that Resource not found when in reality its a permission issue

I got this exception on update router

/v2_0/client.py, line 1212, in _handle_fault_response

exception_handler_v20(status_code, des_error_body)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 74, in exception_handler_v20

message=error_dict)

NeutronClientException: The resource could not be found.


When I had following

 body = {

router:

{

name : pns-router,

external_gateway_info:

{

network_id: net_id,

 enable_snat : False

 }

}

   }


It should have thrown a policy error and not this


From: akalambu akala...@cisco.commailto:akala...@cisco.com
Date: Friday, July 11, 2014 at 11:09 AM
To: Ian Wells (iawells) iawe...@cisco.commailto:iawe...@cisco.com, 
OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: Neutron permission issue

Hi
The issue seems to be the following default config in Neutron policy

   create_router:external_gateway_info:enable_snat: rule:admin_only,

update_router:external_gateway_info:enable_snat: rule:admin_only,


Puzzling part is from horizon when I set an external gateway for a router is it 
not the same thing as above. How does it allow it from horizon than?


Ajay


From: Ian Wells (iawells) iawe...@cisco.commailto:iawe...@cisco.com
Date: Friday, July 11, 2014 at 10:56 AM
To: akalambu akala...@cisco.commailto:akala...@cisco.com, OpenStack 
Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Cc: openstack-systems-group(mailer list) 
openstack-systems-gr...@cisco.commailto:openstack-systems-gr...@cisco.com
Subject: Re: Neutron permission issue

Check /etc/neutron/policy.json, but I agree that's weird...
--
Ian.


From: Ajay Kalambur (akalambu) akala...@cisco.commailto:akala...@cisco.com
Date: Friday, 11 July 2014 10:05
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Cc: openstack-systems-group(mailer list) 
openstack-systems-gr...@cisco.commailto:openstack-systems-gr...@cisco.com
Subject: Neutron permission issue

Hi
As a tenant when I try to create a router and associate a gateway with the 
router as a two step process in Horizon things work fine.
Now when I want to do the same thing through a create router API call with 
request below I get permission denied to create router

{
router:
{
name: another_router,
admin_state_up: true,
external_gateway_info: {
network_id: 3c5bcddd-6af9-4e6b-9c3e-c153e521cab8,
enable_snat: false}
}
}
The network id in both cases is the same. This does not make sense to me


Traceback (most recent call last):

  File vm-tp.py, line 54, in setUp

ext_router = self.net.create_router(CONF.ROUTER_NAME, ext_net['id'])

  File /Users/akalambu/python_venv/latest_code/pns/network.py, line 121, in 
create_router

router = self.neutron_client.create_router(body)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 101, in with_params

ret = self.function(instance, *args, **kwargs)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 398, in create_router

return self.post(self.routers_path, body=body)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 1320, in post

headers=headers, params=params)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 1243, in do_request

self._handle_fault_response(status_code, replybody)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 1211, in _handle_fault_response

exception_handler_v20(status_code, des_error_body)

  File 
/Users/akalambu/python_venv/venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py,
 line 68, in exception_handler_v20

status_code=status_code)

Forbidden: Policy doesn't allow create_router to be performed.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Tempest ipv6 tests

[Ajay Kalambur (akalambu)]

Hi
Are there any tests available for ipv6 in Tempest. Also whats the road map for 
addition of these tests.

Ajay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev