[openstack-dev] [neutron]Performance of security group

[shihanzhang]

Hello all,


Now in neutron, it use iptable implementing security group, but the performance 
of this  implementation is very poor, there is a 
bug:https://bugs.launchpad.net/neutron/+bug/1302272 to reflect this problem. In 
his test, with default security groups(which has remote security group), beyond 
250-300 VMs, there were around 6k Iptable rules on evry compute node, although 
his patch can reduce the processing time, but it don't solve this problem 
fundamentally. I have commit a BP to solve this 
problem:https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security 
There are other people interested in this it?___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron]Performance of security group

[Miguel Angel Ajo Pelayo]

I couldn't resist making a little benchmark test of the new RPC implementation
shihanzhang wrote:

http://www.ajo.es/post/95269040924/neutron-security-group-rules-for-devices-rpc-rewrite

The results are awesome :-)

We yet need to polish the tests a bit, and it's ready.

Best regards,
Miguel Ángel.

- Original Message -
> On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang  wrote:
> >
> > With the deployment 'nova + neutron + openvswitch', when we bulk create
> > about 500 VM with a default security group, the CPU usage of neutron-server
> > and openvswitch agent is very high, especially the CPU usage of openvswitch
> > agent will be 100%, this will cause creating VMs failed.
> >
> > With the method discussed in mailist:
> >
> > 1) ipset optimization   (https://review.openstack.org/#/c/100761/)
> >
> > 3) sg rpc optimization (with fanout)
> > (https://review.openstack.org/#/c/104522/)
> >
> > I have implement  these two scheme in my deployment,  when we again bulk
> > create about 500 VM with a default security group, the CPU usage of
> > openvswitch agent will reduce to 10%, even lower than 10%, so I think the
> > iprovement of these two options are very efficient.
> >
> > Who can help us to review our spec?
> >
> This is great work! These are on my list of things to review in detail
> soon, but given the Neutron sprint this week, I haven't had time yet.
> I'll try to remedy that by the weekend.
> 
> Thanks!
> Kyle
> 
> >Best regards,
> > shihanzhang
> >
> >
> >
> >
> >
> > At 2014-07-03 10:08:21, "Ihar Hrachyshka"  wrote:
> >>-BEGIN PGP SIGNED MESSAGE-
> >>Hash: SHA512
> >>
> >>Oh, so you have the enhancement implemented? Great! Any numbers that
> >>shows how much we gain from that?
> >>
> >>/Ihar
> >>
> >>On 03/07/14 02:49, shihanzhang wrote:
> >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
> >>> I will modify my spec, when the spec is approved, I will commit the
> >>> codes as soon as possilbe!
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" 
> >>> wrote:
> 
>  Nice Shihanzhang,
> 
>  Do you mean the ipset implementation is ready, or just the
>  spec?.
> 
> 
>  For the SG group refactor, I don't worry about who does it, or
>  who takes the credit, but I believe it's important we address
>  this bottleneck during Juno trying to match nova's scalability.
> 
>  Best regards, Miguel Ángel.
> 
> 
>  On 07/02/2014 02:50 PM, shihanzhang wrote:
> > hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
> > split  the work in several specs, I have finished the work (
> > ipset optimization), you can do 'sg rpc optimization (without
> > fanout)'. as the third part(sg rpc optimization (with fanout)),
> > I think we need talk about it, because just using ipset to
> > optimize security group agent codes does not bring the best
> > results!
> >
> > Best regards, shihanzhang.
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2014-07-02 04:43:24, "Ihar Hrachyshka" 
> > wrote:
> >>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
> >>>
>  Shihazhang,
> >>>
>  I really believe we need the RPC refactor done for this cycle,
>  and given the close deadlines we have (July 10 for spec
>  submission and July 20 for spec approval).
> >>>
>  Don't you think it's going to be better to split the work in
>  several specs?
> >>>
>  1) ipset optimization   (you) 2) sg rpc optimization (without
>  fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
>  , me)
> >>>
> >>>
>  This way we increase the chances of having part of this for the
>  Juno cycle. If we go for something too complicated is going to
>  take more time for approval.
> >>>
> >>>
> >>> I agree. And it not only increases chances to get at least some of
> >>> those highly demanded performance enhancements to get into Juno,
> >>> it's also "the right thing to do" (c). It's counterproductive to
> >>> put multiple vaguely related enhancements in single spec. This
> >>> would dim review focus and put us into position of getting
> >>> 'all-or-nothing'. We can't afford that.
> >>>
> >>> Let's leave one spec per enhancement. @Shihazhang, what do you
> >>> think?
> >>>
> >>>
>  Also, I proposed the details of "2", trying to bring awareness
>  on the topic, as I have been working with the scale lab in Red
>  Hat to find and understand those issues, I have a very good
>  knowledge of the problem and I believe I could make a very fast
>  advance on the issue at the RPC level.
> >>>
>  Given that, I'd like to work on this specific part, whether or
>  not we split the specs, as it's something we believe critical
>  for neutron scalability and thus, *nova parity*.
> >>>
>  I will start a separate spec for "2", later on, if you find it
>  ok, we keep them as separate ones, if you believe having just

Re: [openstack-dev] [neutron]Performance of security group

[Jay Pipes]

On 08/20/2014 07:34 AM, Miguel Angel Ajo Pelayo wrote:

I couldn't resist making a little benchmark test of the new RPC implementation
shihanzhang wrote:

http://www.ajo.es/post/95269040924/neutron-security-group-rules-for-devices-rpc-rewrite

The results are awesome :-)


Indeed, fantastic news. ++

-jay


We yet need to polish the tests a bit, and it's ready.

Best regards,
Miguel Ángel.

- Original Message -

On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang  wrote:


With the deployment 'nova + neutron + openvswitch', when we bulk create
about 500 VM with a default security group, the CPU usage of neutron-server
and openvswitch agent is very high, especially the CPU usage of openvswitch
agent will be 100%, this will cause creating VMs failed.

With the method discussed in mailist:

1) ipset optimization   (https://review.openstack.org/#/c/100761/)

3) sg rpc optimization (with fanout)
(https://review.openstack.org/#/c/104522/)

I have implement  these two scheme in my deployment,  when we again bulk
create about 500 VM with a default security group, the CPU usage of
openvswitch agent will reduce to 10%, even lower than 10%, so I think the
iprovement of these two options are very efficient.

Who can help us to review our spec?


This is great work! These are on my list of things to review in detail
soon, but given the Neutron sprint this week, I haven't had time yet.
I'll try to remedy that by the weekend.

Thanks!
Kyle


Best regards,
shihanzhang





At 2014-07-03 10:08:21, "Ihar Hrachyshka"  wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Oh, so you have the enhancement implemented? Great! Any numbers that
shows how much we gain from that?

/Ihar

On 03/07/14 02:49, shihanzhang wrote:

Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
I will modify my spec, when the spec is approved, I will commit the
codes as soon as possilbe!





At 2014-07-02 10:12:34, "Miguel Angel Ajo" 
wrote:


Nice Shihanzhang,

Do you mean the ipset implementation is ready, or just the
spec?.


For the SG group refactor, I don't worry about who does it, or
who takes the credit, but I believe it's important we address
this bottleneck during Juno trying to match nova's scalability.

Best regards, Miguel Ángel.


On 07/02/2014 02:50 PM, shihanzhang wrote:

hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
split  the work in several specs, I have finished the work (
ipset optimization), you can do 'sg rpc optimization (without
fanout)'. as the third part(sg rpc optimization (with fanout)),
I think we need talk about it, because just using ipset to
optimize security group agent codes does not bring the best
results!

Best regards, shihanzhang.








At 2014-07-02 04:43:24, "Ihar Hrachyshka" 
wrote:

On 02/07/14 10:12, Miguel Angel Ajo wrote:


Shihazhang,



I really believe we need the RPC refactor done for this cycle,
and given the close deadlines we have (July 10 for spec
submission and July 20 for spec approval).



Don't you think it's going to be better to split the work in
several specs?



1) ipset optimization   (you) 2) sg rpc optimization (without
fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
, me)




This way we increase the chances of having part of this for the
Juno cycle. If we go for something too complicated is going to
take more time for approval.



I agree. And it not only increases chances to get at least some of
those highly demanded performance enhancements to get into Juno,
it's also "the right thing to do" (c). It's counterproductive to
put multiple vaguely related enhancements in single spec. This
would dim review focus and put us into position of getting
'all-or-nothing'. We can't afford that.

Let's leave one spec per enhancement. @Shihazhang, what do you
think?



Also, I proposed the details of "2", trying to bring awareness
on the topic, as I have been working with the scale lab in Red
Hat to find and understand those issues, I have a very good
knowledge of the problem and I believe I could make a very fast
advance on the issue at the RPC level.



Given that, I'd like to work on this specific part, whether or
not we split the specs, as it's something we believe critical
for neutron scalability and thus, *nova parity*.



I will start a separate spec for "2", later on, if you find it
ok, we keep them as separate ones, if you believe having just 1
spec (for 1 & 2) is going be safer for juno-* approval, then we
can incorporate my spec in yours, but then
"add-ipset-to-security" is not a good spec title to put all this
together.




Best regards, Miguel Ángel.




On 07/02/2014 03:37 AM, shihanzhang wrote:


hi Miguel Angel Ajo Pelayo! I agree with you and modify my
spes, but I will also optimization the RPC from security group
agent to neutron server. Now the modle is
'port[rule1,rule2...], port...', I will change it to 'port[sg1,
sg2..]', this can reduce the size of RPC respose message from
neutron server to security group agent.

At 201

Re: [openstack-dev] [neutron]Performance of security group

[Baohua Yang]

Great！
We met similar problems.
The current mechanisms produce too many iptables rules, and it's hard to
debug.
Really look forward to seeing a more efficient security group design.


On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery 
wrote:

> On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang 
> wrote:
> >
> > With the deployment 'nova + neutron + openvswitch', when we bulk create
> > about 500 VM with a default security group, the CPU usage of
> neutron-server
> > and openvswitch agent is very high, especially the CPU usage of
> openvswitch
> > agent will be 100%, this will cause creating VMs failed.
> >
> > With the method discussed in mailist:
> >
> > 1) ipset optimization   (https://review.openstack.org/#/c/100761/)
> >
> > 3) sg rpc optimization (with fanout)
> > (https://review.openstack.org/#/c/104522/)
> >
> > I have implement  these two scheme in my deployment,  when we again bulk
> > create about 500 VM with a default security group, the CPU usage of
> > openvswitch agent will reduce to 10%, even lower than 10%, so I think the
> > iprovement of these two options are very efficient.
> >
> > Who can help us to review our spec?
> >
> This is great work! These are on my list of things to review in detail
> soon, but given the Neutron sprint this week, I haven't had time yet.
> I'll try to remedy that by the weekend.
>
> Thanks!
> Kyle
>
> >Best regards,
> > shihanzhang
> >
> >
> >
> >
> >
> > At 2014-07-03 10:08:21, "Ihar Hrachyshka"  wrote:
> >>-BEGIN PGP SIGNED MESSAGE-
> >>Hash: SHA512
> >>
> >>Oh, so you have the enhancement implemented? Great! Any numbers that
> >>shows how much we gain from that?
> >>
> >>/Ihar
> >>
> >>On 03/07/14 02:49, shihanzhang wrote:
> >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
> >>> I will modify my spec, when the spec is approved, I will commit the
> >>> codes as soon as possilbe!
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" 
> >>> wrote:
> 
>  Nice Shihanzhang,
> 
>  Do you mean the ipset implementation is ready, or just the
>  spec?.
> 
> 
>  For the SG group refactor, I don't worry about who does it, or
>  who takes the credit, but I believe it's important we address
>  this bottleneck during Juno trying to match nova's scalability.
> 
>  Best regards, Miguel Ángel.
> 
> 
>  On 07/02/2014 02:50 PM, shihanzhang wrote:
> > hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
> > split  the work in several specs, I have finished the work (
> > ipset optimization), you can do 'sg rpc optimization (without
> > fanout)'. as the third part(sg rpc optimization (with fanout)),
> > I think we need talk about it, because just using ipset to
> > optimize security group agent codes does not bring the best
> > results!
> >
> > Best regards, shihanzhang.
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2014-07-02 04:43:24, "Ihar Hrachyshka" 
> > wrote:
> >>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
> >>>
>  Shihazhang,
> >>>
>  I really believe we need the RPC refactor done for this cycle,
>  and given the close deadlines we have (July 10 for spec
>  submission and July 20 for spec approval).
> >>>
>  Don't you think it's going to be better to split the work in
>  several specs?
> >>>
>  1) ipset optimization   (you) 2) sg rpc optimization (without
>  fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
>  , me)
> >>>
> >>>
>  This way we increase the chances of having part of this for the
>  Juno cycle. If we go for something too complicated is going to
>  take more time for approval.
> >>>
> >>>
> >>> I agree. And it not only increases chances to get at least some of
> >>> those highly demanded performance enhancements to get into Juno,
> >>> it's also "the right thing to do" (c). It's counterproductive to
> >>> put multiple vaguely related enhancements in single spec. This
> >>> would dim review focus and put us into position of getting
> >>> 'all-or-nothing'. We can't afford that.
> >>>
> >>> Let's leave one spec per enhancement. @Shihazhang, what do you
> >>> think?
> >>>
> >>>
>  Also, I proposed the details of "2", trying to bring awareness
>  on the topic, as I have been working with the scale lab in Red
>  Hat to find and understand those issues, I have a very good
>  knowledge of the problem and I believe I could make a very fast
>  advance on the issue at the RPC level.
> >>>
>  Given that, I'd like to work on this specific part, whether or
>  not we split the specs, as it's something we believe critical
>  for neutron scalability and thus, *nova parity*.
> >>>
>  I will start a separate spec for "2", later on, if you find it
>  ok, we keep them as separate ones, if you believe having just 1
>  spec (for 1 & 2) is going be safer for juno-* approval, then we
>  can in

Re: [openstack-dev] [neutron]Performance of security group

[Martinx - ジェームズ]

+1 "NFTablesDriver"!

Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example:
https://home.regit.org/2014/02/suricata-and-nftables/

Then, I'm wondering here... What benefits might come for OpenStack Nova /
Neutron, if it comes with a NFTables driver, instead of the current
IPTables?!

* Efficient Security Group design?
* Better FWaaS, maybe with NAT(44/66) support?
* Native support for IPv6, with the defamed NAT66 built-in, simpler
"Floating IP" implementation, for both v4 and v6 networks under a single
implementation (*I don't like NAT66, I prefer a `routed Floating IPv6`
version*)?
* Metadata over IPv6 still using NAT(66) (*I don't like NAT66*), single
implementation?
* Suricata-as-a-Service?!

It sounds pretty cool!   :-)


On 20 August 2014 23:16, Baohua Yang  wrote:

> Great！
> We met similar problems.
> The current mechanisms produce too many iptables rules, and it's hard to
> debug.
> Really look forward to seeing a more efficient security group design.
>
>
> On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery 
> wrote:
>
>> On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang 
>> wrote:
>> >
>> > With the deployment 'nova + neutron + openvswitch', when we bulk create
>> > about 500 VM with a default security group, the CPU usage of
>> neutron-server
>> > and openvswitch agent is very high, especially the CPU usage of
>> openvswitch
>> > agent will be 100%, this will cause creating VMs failed.
>> >
>> > With the method discussed in mailist:
>> >
>> > 1) ipset optimization   (https://review.openstack.org/#/c/100761/)
>> >
>> > 3) sg rpc optimization (with fanout)
>> > (https://review.openstack.org/#/c/104522/)
>> >
>> > I have implement  these two scheme in my deployment,  when we again bulk
>> > create about 500 VM with a default security group, the CPU usage of
>> > openvswitch agent will reduce to 10%, even lower than 10%, so I think
>> the
>> > iprovement of these two options are very efficient.
>> >
>> > Who can help us to review our spec?
>> >
>> This is great work! These are on my list of things to review in detail
>> soon, but given the Neutron sprint this week, I haven't had time yet.
>> I'll try to remedy that by the weekend.
>>
>> Thanks!
>> Kyle
>>
>> >Best regards,
>> > shihanzhang
>> >
>> >
>> >
>> >
>> >
>> > At 2014-07-03 10:08:21, "Ihar Hrachyshka"  wrote:
>> >>-BEGIN PGP SIGNED MESSAGE-
>> >>Hash: SHA512
>> >>
>> >>Oh, so you have the enhancement implemented? Great! Any numbers that
>> >>shows how much we gain from that?
>> >>
>> >>/Ihar
>> >>
>> >>On 03/07/14 02:49, shihanzhang wrote:
>> >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
>> >>> I will modify my spec, when the spec is approved, I will commit the
>> >>> codes as soon as possilbe!
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" 
>> >>> wrote:
>> 
>>  Nice Shihanzhang,
>> 
>>  Do you mean the ipset implementation is ready, or just the
>>  spec?.
>> 
>> 
>>  For the SG group refactor, I don't worry about who does it, or
>>  who takes the credit, but I believe it's important we address
>>  this bottleneck during Juno trying to match nova's scalability.
>> 
>>  Best regards, Miguel Ángel.
>> 
>> 
>>  On 07/02/2014 02:50 PM, shihanzhang wrote:
>> > hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
>> > split  the work in several specs, I have finished the work (
>> > ipset optimization), you can do 'sg rpc optimization (without
>> > fanout)'. as the third part(sg rpc optimization (with fanout)),
>> > I think we need talk about it, because just using ipset to
>> > optimize security group agent codes does not bring the best
>> > results!
>> >
>> > Best regards, shihanzhang.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > At 2014-07-02 04:43:24, "Ihar Hrachyshka" 
>> > wrote:
>> >>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>> >>>
>>  Shihazhang,
>> >>>
>>  I really believe we need the RPC refactor done for this cycle,
>>  and given the close deadlines we have (July 10 for spec
>>  submission and July 20 for spec approval).
>> >>>
>>  Don't you think it's going to be better to split the work in
>>  several specs?
>> >>>
>>  1) ipset optimization   (you) 2) sg rpc optimization (without
>>  fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
>>  , me)
>> >>>
>> >>>
>>  This way we increase the chances of having part of this for the
>>  Juno cycle. If we go for something too complicated is going to
>>  take more time for approval.
>> >>>
>> >>>
>> >>> I agree. And it not only increases chances to get at least some of
>> >>> those highly demanded performance enhancements to get into Juno,
>> >>> it's also "the right thing to do" (c). It's counterproductive to
>> >>> put multiple vaguely related enhancements in single spec. This
>> >>> would dim review focus

Re: [openstack-dev] [neutron]Performance of security group

[shihanzhang]

hi neutroner!
my patch about 
BP:https://blueprints.launchpad.net/openstack/?searchtext=add-ipset-to-security 
need install ipset in devstack, I have commit the 
patch:https://review.openstack.org/#/c/113453/, who can help me review it, 
thanks very much!


 Best regards,
shihanzhang





At 2014-08-21 10:47:59, "Martinx - ジェームズ"  wrote:

+1 "NFTablesDriver"!


Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example: 
https://home.regit.org/2014/02/suricata-and-nftables/


Then, I'm wondering here... What benefits might come for OpenStack Nova / 
Neutron, if it comes with a NFTables driver, instead of the current IPTables?!


* Efficient Security Group design?
* Better FWaaS, maybe with NAT(44/66) support?
* Native support for IPv6, with the defamed NAT66 built-in, simpler "Floating 
IP" implementation, for both v4 and v6 networks under a single implementation 
(I don't like NAT66, I prefer a `routed Floating IPv6` version)?
* Metadata over IPv6 still using NAT(66) (I don't like NAT66), single 
implementation?
* Suricata-as-a-Service?!


It sounds pretty cool!   :-)



On 20 August 2014 23:16, Baohua Yang  wrote:

Great！
We met similar problems.
The current mechanisms produce too many iptables rules, and it's hard to debug.
Really look forward to seeing a more efficient security group design.



On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery  
wrote:

On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang  wrote:
>
> With the deployment 'nova + neutron + openvswitch', when we bulk create
> about 500 VM with a default security group, the CPU usage of neutron-server
> and openvswitch agent is very high, especially the CPU usage of openvswitch
> agent will be 100%, this will cause creating VMs failed.
>
> With the method discussed in mailist:
>
> 1) ipset optimization   (https://review.openstack.org/#/c/100761/)
>
> 3) sg rpc optimization (with fanout)
> (https://review.openstack.org/#/c/104522/)
>
> I have implement  these two scheme in my deployment,  when we again bulk
> create about 500 VM with a default security group, the CPU usage of
> openvswitch agent will reduce to 10%, even lower than 10%, so I think the
> iprovement of these two options are very efficient.
>
> Who can help us to review our spec?
>

This is great work! These are on my list of things to review in detail
soon, but given the Neutron sprint this week, I haven't had time yet.
I'll try to remedy that by the weekend.

Thanks!
Kyle


>Best regards,
> shihanzhang
>
>
>
>
>
> At 2014-07-03 10:08:21, "Ihar Hrachyshka"  wrote:
>>-BEGIN PGP SIGNED MESSAGE-
>>Hash: SHA512
>>
>>Oh, so you have the enhancement implemented? Great! Any numbers that
>>shows how much we gain from that?
>>
>>/Ihar
>>
>>On 03/07/14 02:49, shihanzhang wrote:
>>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
>>> I will modify my spec, when the spec is approved, I will commit the
>>> codes as soon as possilbe!
>>>
>>>
>>>
>>>
>>>
>>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" 
>>> wrote:

 Nice Shihanzhang,

 Do you mean the ipset implementation is ready, or just the
 spec?.


 For the SG group refactor, I don't worry about who does it, or
 who takes the credit, but I believe it's important we address
 this bottleneck during Juno trying to match nova's scalability.

 Best regards, Miguel Ángel.


 On 07/02/2014 02:50 PM, shihanzhang wrote:
> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
> split  the work in several specs, I have finished the work (
> ipset optimization), you can do 'sg rpc optimization (without
> fanout)'. as the third part(sg rpc optimization (with fanout)),
> I think we need talk about it, because just using ipset to
> optimize security group agent codes does not bring the best
> results!
>
> Best regards, shihanzhang.
>
>
>
>
>
>
>
>
> At 2014-07-02 04:43:24, "Ihar Hrachyshka" 
> wrote:
>>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>>>
 Shihazhang,
>>>
 I really believe we need the RPC refactor done for this cycle,
 and given the close deadlines we have (July 10 for spec
 submission and July 20 for spec approval).
>>>
 Don't you think it's going to be better to split the work in
 several specs?
>>>
 1) ipset optimization   (you) 2) sg rpc optimization (without
 fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
 , me)
>>>
>>>
 This way we increase the chances of having part of this for the
 Juno cycle. If we go for something too complicated is going to
 take more time for approval.
>>>
>>>
>>> I agree. And it not only increases chances to get at least some of
>>> those highly demanded performance enhancements to get into Juno,
>>> it's also "the right thing to do" (c). It's counterproductive to
>>> put multiple vaguely related enhancements in single spec. This
>>> would dim review focus and put us i

Re: [openstack-dev] [neutron]Performance of security group

[Miguel Angel Ajo Pelayo]

Thank you shihanzhang!,

I can't believe I didn't realize the ipset part spec was accepted I live
on my own bubble... I will be reviewing and testing/helping on that part 
too during the next few days,  I was too concentrated in the RPC part.


Best regards,

- Original Message -
> hi neutroner!
> my patch about BP:
> https://blueprints.launchpad.net/openstack/?searchtext=add-ipset-to-security
> need install ipset in devstack, I have commit the patch:
> https://review.openstack.org/#/c/113453/, who can help me review it, thanks
> very much!
> 
> Best regards,
> shihanzhang
> 
> 
> 
> 
> At 2014-08-21 10:47:59, "Martinx - ジェームズ"  wrote:
> 
> 
> 
> +1 "NFTablesDriver"!
> 
> Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example:
> https://home.regit.org/2014/02/suricata-and-nftables/
> 
> Then, I'm wondering here... What benefits might come for OpenStack Nova /
> Neutron, if it comes with a NFTables driver, instead of the current
> IPTables?!
> 
> * E fficient Security Group design?
> * Better FWaaS, maybe with NAT(44/66) support?
> * Native support for IPv6, with the defamed NAT66 built-in, simpler "Floating
> IP" implementation, for both v4 and v6 networks under a single
> implementation ( I don't like NAT66, I prefer a `routed Floating IPv6`
> version ) ?
> * Metadata over IPv6 still using NAT(66) ( I don't like NAT66 ), single
> implementation?
> * Suricata-as-a-Service?!
> 
> It sounds pretty cool! :-)
> 
> 
> On 20 August 2014 23:16, Baohua Yang < yangbao...@gmail.com > wrote:
> 
> 
> 
> Great！
> We met similar problems.
> The current mechanisms produce too many iptables rules, and it's hard to
> debug.
> Really look forward to seeing a more efficient security group design.
> 
> 
> On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery < mest...@noironetworks.com >
> wrote:
> 
> 
> 
> On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang < ayshihanzh...@126.com > wrote:
> > 
> > With the deployment 'nova + neutron + openvswitch', when we bulk create
> > about 500 VM with a default security group, the CPU usage of neutron-server
> > and openvswitch agent is very high, especially the CPU usage of openvswitch
> > agent will be 100%, this will cause creating VMs failed.
> > 
> > With the method discussed in mailist:
> > 
> > 1) ipset optimization ( https://review.openstack.org/#/c/100761/ )
> > 
> > 3) sg rpc optimization (with fanout)
> > ( https://review.openstack.org/#/c/104522/ )
> > 
> > I have implement these two scheme in my deployment, when we again bulk
> > create about 500 VM with a default security group, the CPU usage of
> > openvswitch agent will reduce to 10%, even lower than 10%, so I think the
> > iprovement of these two options are very efficient.
> > 
> > Who can help us to review our spec?
> > 
> This is great work! These are on my list of things to review in detail
> soon, but given the Neutron sprint this week, I haven't had time yet.
> I'll try to remedy that by the weekend.
> 
> Thanks!
> Kyle
> 
> > Best regards,
> > shihanzhang
> > 
> > 
> > 
> > 
> > 
> > At 2014-07-03 10:08:21, "Ihar Hrachyshka" < ihrac...@redhat.com > wrote:
> >>-BEGIN PGP SIGNED MESSAGE-
> >>Hash: SHA512
> >> 
> >>Oh, so you have the enhancement implemented? Great! Any numbers that
> >>shows how much we gain from that?
> >> 
> >>/Ihar
> >> 
> >>On 03/07/14 02:49, shihanzhang wrote:
> >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
> >>> I will modify my spec, when the spec is approved, I will commit the
> >>> codes as soon as possilbe!
> >>> 
> >>> 
> >>> 
> >>> 
> >>> 
> >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" < majop...@redhat.com >
> >>> wrote:
>  
>  Nice Shihanzhang,
>  
>  Do you mean the ipset implementation is ready, or just the
>  spec?.
>  
>  
>  For the SG group refactor, I don't worry about who does it, or
>  who takes the credit, but I believe it's important we address
>  this bottleneck during Juno trying to match nova's scalability.
>  
>  Best regards, Miguel Ángel.
>  
>  
>  On 07/02/2014 02:50 PM, shihanzhang wrote:
> > hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
> > split the work in several specs, I have finished the work (
> > ipset optimization), you can do 'sg rpc optimization (without
> > fanout)'. as the third part(sg rpc optimization (with fanout)),
> > I think we need talk about it, because just using ipset to
> > optimize security group agent codes does not bring the best
> > results!
> > 
> > Best regards, shihanzhang.
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > At 2014-07-02 04:43:24, "Ihar Hrachyshka" < ihrac...@redhat.com >
> > wrote:
> >>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
> >>> 
>  Shihazhang,
> >>> 
>  I really believe we need the RPC refactor done for this cycle,
>  and given the close deadlines we have (July 10 for spec
>  submission and July 20 for spec

Re: [openstack-dev] [neutron]Performance of security group

[Édouard Thuleau]

Nice job! That's awesome.

Thanks,
Édouard.


On Thu, Aug 21, 2014 at 8:02 AM, Miguel Angel Ajo Pelayo <
mangel...@redhat.com> wrote:

> Thank you shihanzhang!,
>
> I can't believe I didn't realize the ipset part spec was accepted I live
> on my own bubble... I will be reviewing and testing/helping on that part
> too during the next few days,  I was too concentrated in the RPC part.
>
>
> Best regards,
>
> - Original Message -
> > hi neutroner!
> > my patch about BP:
> >
> https://blueprints.launchpad.net/openstack/?searchtext=add-ipset-to-security
> > need install ipset in devstack, I have commit the patch:
> > https://review.openstack.org/#/c/113453/, who can help me review it,
> thanks
> > very much!
> >
> > Best regards,
> > shihanzhang
> >
> >
> >
> >
> > At 2014-08-21 10:47:59, "Martinx - ジェームズ" 
> wrote:
> >
> >
> >
> > +1 "NFTablesDriver"!
> >
> > Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example:
> > https://home.regit.org/2014/02/suricata-and-nftables/
> >
> > Then, I'm wondering here... What benefits might come for OpenStack Nova /
> > Neutron, if it comes with a NFTables driver, instead of the current
> > IPTables?!
> >
> > * E fficient Security Group design?
> > * Better FWaaS, maybe with NAT(44/66) support?
> > * Native support for IPv6, with the defamed NAT66 built-in, simpler
> "Floating
> > IP" implementation, for both v4 and v6 networks under a single
> > implementation ( I don't like NAT66, I prefer a `routed Floating IPv6`
> > version ) ?
> > * Metadata over IPv6 still using NAT(66) ( I don't like NAT66 ), single
> > implementation?
> > * Suricata-as-a-Service?!
> >
> > It sounds pretty cool! :-)
> >
> >
> > On 20 August 2014 23:16, Baohua Yang < yangbao...@gmail.com > wrote:
> >
> >
> >
> > Great！
> > We met similar problems.
> > The current mechanisms produce too many iptables rules, and it's hard to
> > debug.
> > Really look forward to seeing a more efficient security group design.
> >
> >
> > On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery <
> mest...@noironetworks.com >
> > wrote:
> >
> >
> >
> > On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang < ayshihanzh...@126.com >
> wrote:
> > >
> > > With the deployment 'nova + neutron + openvswitch', when we bulk create
> > > about 500 VM with a default security group, the CPU usage of
> neutron-server
> > > and openvswitch agent is very high, especially the CPU usage of
> openvswitch
> > > agent will be 100%, this will cause creating VMs failed.
> > >
> > > With the method discussed in mailist:
> > >
> > > 1) ipset optimization ( https://review.openstack.org/#/c/100761/ )
> > >
> > > 3) sg rpc optimization (with fanout)
> > > ( https://review.openstack.org/#/c/104522/ )
> > >
> > > I have implement these two scheme in my deployment, when we again bulk
> > > create about 500 VM with a default security group, the CPU usage of
> > > openvswitch agent will reduce to 10%, even lower than 10%, so I think
> the
> > > iprovement of these two options are very efficient.
> > >
> > > Who can help us to review our spec?
> > >
> > This is great work! These are on my list of things to review in detail
> > soon, but given the Neutron sprint this week, I haven't had time yet.
> > I'll try to remedy that by the weekend.
> >
> > Thanks!
> > Kyle
> >
> > > Best regards,
> > > shihanzhang
> > >
> > >
> > >
> > >
> > >
> > > At 2014-07-03 10:08:21, "Ihar Hrachyshka" < ihrac...@redhat.com >
> wrote:
> > >>-BEGIN PGP SIGNED MESSAGE-
> > >>Hash: SHA512
> > >>
> > >>Oh, so you have the enhancement implemented? Great! Any numbers that
> > >>shows how much we gain from that?
> > >>
> > >>/Ihar
> > >>
> > >>On 03/07/14 02:49, shihanzhang wrote:
> > >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
> > >>> I will modify my spec, when the spec is approved, I will commit the
> > >>> codes as soon as possilbe!
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" < majop...@redhat.com >
> > >>> wrote:
> > 
> >  Nice Shihanzhang,
> > 
> >  Do you mean the ipset implementation is ready, or just the
> >  spec?.
> > 
> > 
> >  For the SG group refactor, I don't worry about who does it, or
> >  who takes the credit, but I believe it's important we address
> >  this bottleneck during Juno trying to match nova's scalability.
> > 
> >  Best regards, Miguel Ángel.
> > 
> > 
> >  On 07/02/2014 02:50 PM, shihanzhang wrote:
> > > hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
> > > split the work in several specs, I have finished the work (
> > > ipset optimization), you can do 'sg rpc optimization (without
> > > fanout)'. as the third part(sg rpc optimization (with fanout)),
> > > I think we need talk about it, because just using ipset to
> > > optimize security group agent codes does not bring the best
> > > results!
> > >
> > > Best regards, shihanzhang.
> > >
> > >
> > 

Re: [openstack-dev] [neutron]Performance of security group

[Kyle Mestery]

This is great work! Looking forward to seeing this get reviewed and
merged in Juno!

Kyle

On Thu, Aug 21, 2014 at 6:49 AM, Édouard Thuleau  wrote:
> Nice job! That's awesome.
>
> Thanks,
> Édouard.
>
>
> On Thu, Aug 21, 2014 at 8:02 AM, Miguel Angel Ajo Pelayo
>  wrote:
>>
>> Thank you shihanzhang!,
>>
>> I can't believe I didn't realize the ipset part spec was accepted I live
>> on my own bubble... I will be reviewing and testing/helping on that part
>> too during the next few days,  I was too concentrated in the RPC part.
>>
>>
>> Best regards,
>>
>> - Original Message -
>> > hi neutroner!
>> > my patch about BP:
>> >
>> > https://blueprints.launchpad.net/openstack/?searchtext=add-ipset-to-security
>> > need install ipset in devstack, I have commit the patch:
>> > https://review.openstack.org/#/c/113453/, who can help me review it,
>> > thanks
>> > very much!
>> >
>> > Best regards,
>> > shihanzhang
>> >
>> >
>> >
>> >
>> > At 2014-08-21 10:47:59, "Martinx - ジェームズ" 
>> > wrote:
>> >
>> >
>> >
>> > +1 "NFTablesDriver"!
>> >
>> > Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example:
>> > https://home.regit.org/2014/02/suricata-and-nftables/
>> >
>> > Then, I'm wondering here... What benefits might come for OpenStack Nova
>> > /
>> > Neutron, if it comes with a NFTables driver, instead of the current
>> > IPTables?!
>> >
>> > * E fficient Security Group design?
>> > * Better FWaaS, maybe with NAT(44/66) support?
>> > * Native support for IPv6, with the defamed NAT66 built-in, simpler
>> > "Floating
>> > IP" implementation, for both v4 and v6 networks under a single
>> > implementation ( I don't like NAT66, I prefer a `routed Floating IPv6`
>> > version ) ?
>> > * Metadata over IPv6 still using NAT(66) ( I don't like NAT66 ), single
>> > implementation?
>> > * Suricata-as-a-Service?!
>> >
>> > It sounds pretty cool! :-)
>> >
>> >
>> > On 20 August 2014 23:16, Baohua Yang < yangbao...@gmail.com > wrote:
>> >
>> >
>> >
>> > Great！
>> > We met similar problems.
>> > The current mechanisms produce too many iptables rules, and it's hard to
>> > debug.
>> > Really look forward to seeing a more efficient security group design.
>> >
>> >
>> > On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery <
>> > mest...@noironetworks.com >
>> > wrote:
>> >
>> >
>> >
>> > On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang < ayshihanzh...@126.com >
>> > wrote:
>> > >
>> > > With the deployment 'nova + neutron + openvswitch', when we bulk
>> > > create
>> > > about 500 VM with a default security group, the CPU usage of
>> > > neutron-server
>> > > and openvswitch agent is very high, especially the CPU usage of
>> > > openvswitch
>> > > agent will be 100%, this will cause creating VMs failed.
>> > >
>> > > With the method discussed in mailist:
>> > >
>> > > 1) ipset optimization ( https://review.openstack.org/#/c/100761/ )
>> > >
>> > > 3) sg rpc optimization (with fanout)
>> > > ( https://review.openstack.org/#/c/104522/ )
>> > >
>> > > I have implement these two scheme in my deployment, when we again bulk
>> > > create about 500 VM with a default security group, the CPU usage of
>> > > openvswitch agent will reduce to 10%, even lower than 10%, so I think
>> > > the
>> > > iprovement of these two options are very efficient.
>> > >
>> > > Who can help us to review our spec?
>> > >
>> > This is great work! These are on my list of things to review in detail
>> > soon, but given the Neutron sprint this week, I haven't had time yet.
>> > I'll try to remedy that by the weekend.
>> >
>> > Thanks!
>> > Kyle
>> >
>> > > Best regards,
>> > > shihanzhang
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > At 2014-07-03 10:08:21, "Ihar Hrachyshka" < ihrac...@redhat.com >
>> > > wrote:
>> > >>-BEGIN PGP SIGNED MESSAGE-
>> > >>Hash: SHA512
>> > >>
>> > >>Oh, so you have the enhancement implemented? Great! Any numbers that
>> > >>shows how much we gain from that?
>> > >>
>> > >>/Ihar
>> > >>
>> > >>On 03/07/14 02:49, shihanzhang wrote:
>> > >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
>> > >>> I will modify my spec, when the spec is approved, I will commit the
>> > >>> codes as soon as possilbe!
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" < majop...@redhat.com >
>> > >>> wrote:
>> > 
>> >  Nice Shihanzhang,
>> > 
>> >  Do you mean the ipset implementation is ready, or just the
>> >  spec?.
>> > 
>> > 
>> >  For the SG group refactor, I don't worry about who does it, or
>> >  who takes the credit, but I believe it's important we address
>> >  this bottleneck during Juno trying to match nova's scalability.
>> > 
>> >  Best regards, Miguel Ángel.
>> > 
>> > 
>> >  On 07/02/2014 02:50 PM, shihanzhang wrote:
>> > > hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
>> > > split the work in several specs, I have finished the work (
>> > > ipset optimization), you can do 'sg rpc o

Re: [openstack-dev] [neutron]Performance of security group

[Kevin Benton]

This sounds like a good idea to handle some of the performance issues until
the ovs firewall can be implemented down the the line.
Do you have any performance comparisons?
On Jun 18, 2014 7:46 PM, "shihanzhang"  wrote:

> Hello all,
>
> Now in neutron, it use iptable implementing security group, but the
> performance of this  implementation is very poor, there is a bug:
> https://bugs.launchpad.net/neutron/+bug/1302272 to reflect this problem.
> In his test, with default security groups(which has remote security
> group), beyond 250-300 VMs, there were around 6k Iptable rules on evry
> compute node, although his patch can reduce the processing time, but it
> don't solve this problem fundamentally. I have commit a BP to solve this
> problem:
> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
> 
> There are other people interested in this it?
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron]Performance of security group

[shihanzhang]

Now I have not get accurate test data, but I  can confirm the following points：
1. In compute node, the iptable's chain of a VM is liner, iptable filter it one 
by one, if a VM in default security group and this default security group have 
many members, but ipset chain is set, the time ipset filter one and many member 
is not much difference.
2. when the iptable rule is very large, the probability of  failure  that  
iptable-save save the iptable rule  is very large.






At 2014-06-19 10:55:56, "Kevin Benton"  wrote:


This sounds like a good idea to handle some of the performance issues until the 
ovs firewall can be implemented down the the line.
Do you have any performance comparisons?

On Jun 18, 2014 7:46 PM, "shihanzhang"  wrote:

Hello all,


Now in neutron, it use iptable implementing security group, but the performance 
of this  implementation is very poor, there is a 
bug:https://bugs.launchpad.net/neutron/+bug/1302272 to reflect this problem. In 
his test, with default security groups(which has remote security group), beyond 
250-300 VMs, there were around 6k Iptable rules on evry compute node, although 
his patch can reduce the processing time, but it don't solve this problem 
fundamentally. I have commit a BP to solve this 
problem:https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security 
There are other people interested in this it?



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron]Performance of security group

[henry hly]

we have done some tests, but have different result: the performance is
nearly the same for empty and 5k rules in iptable, but huge gap between
enable/disable iptable hook on linux bridge


On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang  wrote:

> Now I have not get accurate test data, but I  can confirm the following
> points：
> 1. In compute node, the iptable's chain of a VM is liner, iptable filter
> it one by one, if a VM in default security group and this default security
> group have many members, but ipset chain is set, the time ipset filter one
> and many member is not much difference.
> 2. when the iptable rule is very large, the probability of  failure  that  
> iptable-save
> save the iptable rule  is very large.
>
>
>
>
>
> At 2014-06-19 10:55:56, "Kevin Benton"  wrote:
>
> This sounds like a good idea to handle some of the performance issues
> until the ovs firewall can be implemented down the the line.
> Do you have any performance comparisons?
> On Jun 18, 2014 7:46 PM, "shihanzhang"  wrote:
>
>> Hello all,
>>
>> Now in neutron, it use iptable implementing security group, but the
>> performance of this  implementation is very poor, there is a bug:
>> https://bugs.launchpad.net/neutron/+bug/1302272 to reflect this problem.
>> In his test, with default security groups(which has remote security
>> group), beyond 250-300 VMs, there were around 6k Iptable rules on evry
>> compute node, although his patch can reduce the processing time, but it
>> don't solve this problem fundamentally. I have commit a BP to solve this
>> problem:
>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>> 
>> There are other people interested in this it?
>>
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron]Performance of security group

[Édouard Thuleau]

Do you though about nftables that will replace {ip,ip6,arp,eb}tables?
It also based on the rule set mechanism.
The issue in that proposition, it's only stable since the begin of the year
and on Linux kernel 3.13.
But there lot of pros I don't list here (leverage iptables limitation,
efficient update rule, rule set, standardization of netfilter commands...).

Édouard.


On Thu, Jun 19, 2014 at 8:25 AM, henry hly  wrote:

> we have done some tests, but have different result: the performance is
> nearly the same for empty and 5k rules in iptable, but huge gap between
> enable/disable iptable hook on linux bridge
>
>
> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang 
> wrote:
>
>> Now I have not get accurate test data, but I  can confirm the following
>> points：
>> 1. In compute node, the iptable's chain of a VM is liner, iptable filter
>> it one by one, if a VM in default security group and this default security
>> group have many members, but ipset chain is set, the time ipset filter one
>> and many member is not much difference.
>> 2. when the iptable rule is very large, the probability of  failure  that  
>> iptable-save
>> save the iptable rule  is very large.
>>
>>
>>
>>
>>
>> At 2014-06-19 10:55:56, "Kevin Benton"  wrote:
>>
>> This sounds like a good idea to handle some of the performance issues
>> until the ovs firewall can be implemented down the the line.
>> Do you have any performance comparisons?
>> On Jun 18, 2014 7:46 PM, "shihanzhang"  wrote:
>>
>>> Hello all,
>>>
>>> Now in neutron, it use iptable implementing security group, but the
>>> performance of this  implementation is very poor, there is a bug:
>>> https://bugs.launchpad.net/neutron/+bug/1302272 to reflect this
>>> problem. In his test, with default security groups(which has remote
>>> security group), beyond 250-300 VMs, there were around 6k Iptable rules on
>>> evry compute node, although his patch can reduce the processing time, but
>>> it don't solve this problem fundamentally. I have commit a BP to solve
>>> this problem:
>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>>> 
>>> There are other people interested in this it?
>>>
>>>
>>>
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron]Performance of security group

[Miguel Angel Ajo Pelayo]

  Hi it's a very interesting topic, I was getting ready to raise
the same concerns about our security groups implementation, shihanzhang
thank you for starting this topic.

  Not only at low level where (with our default security group
rules -allow all incoming from 'default' sg- the iptable rules
will grow in ~X^2 for a tenant, and, the "security_group_rules_for_devices"
rpc call from ovs-agent to neutron-server grows to message sizes of >100MB,
generating serious scalability issues or timeouts/retries that 
totally break neutron service.

   (example trace of that RPC call with a few instances
http://www.fpaste.org/104401/14008522/)

  I believe that we also need to review the RPC calling mechanism
for the OVS agent here, there are several possible approaches to breaking
down (or/and CIDR compressing) the information we return via this api call.


   So we have to look at two things here:

  * physical implementation on the hosts (ipsets, nftables, ... )
  * rpc communication mechanisms.

   Best regards,
Miguel Ángel.

- Mensaje original - 

> Do you though about nftables that will replace {ip,ip6,arp,eb}tables?
> It also based on the rule set mechanism.
> The issue in that proposition, it's only stable since the begin of the year
> and on Linux kernel 3.13.
> But there lot of pros I don't list here (leverage iptables limitation,
> efficient update rule, rule set, standardization of netfilter commands...).

> Édouard.

> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < henry4...@gmail.com > wrote:

> > we have done some tests, but have different result: the performance is
> > nearly
> > the same for empty and 5k rules in iptable, but huge gap between
> > enable/disable iptable hook on linux bridge
> 

> > On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < ayshihanzh...@126.com >
> > wrote:
> 

> > > Now I have not get accurate test data, but I can confirm the following
> > > points：
> > 
> 
> > > 1. In compute node, the iptable's chain of a VM is liner, iptable filter
> > > it
> > > one by one, if a VM in default security group and this default security
> > > group have many members, but ipset chain is set, the time ipset filter
> > > one
> > > and many member is not much difference.
> > 
> 
> > > 2. when the iptable rule is very large, the probability of failure that
> > > iptable-save save the iptable rule is very large.
> > 
> 

> > > At 2014-06-19 10:55:56, "Kevin Benton" < blak...@gmail.com > wrote:
> > 
> 

> > > > This sounds like a good idea to handle some of the performance issues
> > > > until
> > > > the ovs firewall can be implemented down the the line.
> > > 
> > 
> 
> > > > Do you have any performance comparisons?
> > > 
> > 
> 
> > > > On Jun 18, 2014 7:46 PM, "shihanzhang" < ayshihanzh...@126.com > wrote:
> > > 
> > 
> 

> > > > > Hello all,
> > > > 
> > > 
> > 
> 

> > > > > Now in neutron, it use iptable implementing security group, but the
> > > > > performance of this implementation is very poor, there is a bug:
> > > > > https://bugs.launchpad.net/neutron/+bug/1302272 to reflect this
> > > > > problem.
> > > > > In
> > > > > his test, w ith default security groups(which has remote security
> > > > > group),
> > > > > beyond 250-300 VMs, there were around 6k Iptable rules on evry
> > > > > compute
> > > > > node,
> > > > > although his patch can reduce the processing time, but it don't solve
> > > > > this
> > > > > problem fundamentally. I have commit a BP to solve this problem:
> > > > > https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
> > > > 
> > > 
> > 
> 
> > > > > There are other people interested in this it?
> > > > 
> > > 
> > 
> 

> > > > > ___
> > > > 
> > > 
> > 
> 
> > > > > OpenStack-dev mailing list
> > > > 
> > > 
> > 
> 
> > > > > OpenStack-dev@lists.openstack.org
> > > > 
> > > 
> > 
> 
> > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > > 
> > > 
> > 
> 

> > > ___
> > 
> 
> > > OpenStack-dev mailing list
> > 
> 
> > > OpenStack-dev@lists.openstack.org
> > 
> 
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 

> > ___
> 
> > OpenStack-dev mailing list
> 
> > OpenStack-dev@lists.openstack.org
> 
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron]Performance of security group

[Nachi Ueno]

Hi folks

Thank you for your starting this topic.
Let me share some of my ideas

(1) Improve security_group_rules_for_devices

In current implementation, we are generating rules per port in server side.
It is something like this.

port1[SG_Rule1, SG_Rule2, SG_Rule3] .. port2, port3

This can be improved by using this payload data model.

SG_LIST [ SG1, SG2]
SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
port[SG_ID1, SG_ID2], port2 , port3


(2) Applying security group for network.
I have bp for applying security group for network.

IMO, this usecase1 and usecase2 can cover most of usecases.
(Usecase1) default group for all network
(Usecase2) security group per network

so if we can specify security group for network, we can optimize many payload
especially for default security group.
https://blueprints.launchpad.net/neutron/+spec/network-security-group
spec
http://git.openstack.org/cgit/openstack/neutron-specs/tree/specs/juno/security_group_for_network.rst#n156

Best
Nachi

2014-06-19 7:11 GMT-07:00 Miguel Angel Ajo Pelayo :
>
>   Hi it's a very interesting topic, I was getting ready to raise
> the same concerns about our security groups implementation, shihanzhang
> thank you for starting this topic.
>
>   Not only at low level where (with our default security group
> rules -allow all incoming from 'default' sg- the iptable rules
> will grow in ~X^2 for a tenant, and, the "security_group_rules_for_devices"
> rpc call from ovs-agent to neutron-server grows to message sizes of >100MB,
> generating serious scalability issues or timeouts/retries that
> totally break neutron service.
>
>(example trace of that RPC call with a few instances
> http://www.fpaste.org/104401/14008522/)
>
>   I believe that we also need to review the RPC calling mechanism
> for the OVS agent here, there are several possible approaches to breaking
> down (or/and CIDR compressing) the information we return via this api call.
>
>
>So we have to look at two things here:
>
>   * physical implementation on the hosts (ipsets, nftables, ... )
>   * rpc communication mechanisms.
>
>Best regards,
> Miguel Ángel.
>
> - Mensaje original -
>
>> Do you though about nftables that will replace {ip,ip6,arp,eb}tables?
>> It also based on the rule set mechanism.
>> The issue in that proposition, it's only stable since the begin of the year
>> and on Linux kernel 3.13.
>> But there lot of pros I don't list here (leverage iptables limitation,
>> efficient update rule, rule set, standardization of netfilter commands...).
>
>> Édouard.
>
>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < henry4...@gmail.com > wrote:
>
>> > we have done some tests, but have different result: the performance is
>> > nearly
>> > the same for empty and 5k rules in iptable, but huge gap between
>> > enable/disable iptable hook on linux bridge
>>
>
>> > On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < ayshihanzh...@126.com >
>> > wrote:
>>
>
>> > > Now I have not get accurate test data, but I can confirm the following
>> > > points：
>> >
>>
>> > > 1. In compute node, the iptable's chain of a VM is liner, iptable filter
>> > > it
>> > > one by one, if a VM in default security group and this default security
>> > > group have many members, but ipset chain is set, the time ipset filter
>> > > one
>> > > and many member is not much difference.
>> >
>>
>> > > 2. when the iptable rule is very large, the probability of failure that
>> > > iptable-save save the iptable rule is very large.
>> >
>>
>
>> > > At 2014-06-19 10:55:56, "Kevin Benton" < blak...@gmail.com > wrote:
>> >
>>
>
>> > > > This sounds like a good idea to handle some of the performance issues
>> > > > until
>> > > > the ovs firewall can be implemented down the the line.
>> > >
>> >
>>
>> > > > Do you have any performance comparisons?
>> > >
>> >
>>
>> > > > On Jun 18, 2014 7:46 PM, "shihanzhang" < ayshihanzh...@126.com > wrote:
>> > >
>> >
>>
>
>> > > > > Hello all,
>> > > >
>> > >
>> >
>>
>
>> > > > > Now in neutron, it use iptable implementing security group, but the
>> > > > > performance of this implementation is very poor, there is a bug:
>> > > > > https://bugs.launchpad.net/neutron/+bug/1302272 to reflect this
>> > > > > problem.
>> > > > > In
>> > > > > his test, w ith default security groups(which has remote security
>> > > > > group),
>> > > > > beyond 250-300 VMs, there were around 6k Iptable rules on evry
>> > > > > compute
>> > > > > node,
>> > > > > although his patch can reduce the processing time, but it don't solve
>> > > > > this
>> > > > > problem fundamentally. I have commit a BP to solve this problem:
>> > > > > https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>> > > >
>> > >
>> >
>>
>> > > > > There are other people interested in this it?
>> > > >
>> > >
>> >
>>
>
>> > > > > ___
>> > > >
>> > >
>> >
>>
>> > > > > OpenStack-dev mailing list
>> > > >
>> > >
>> >
>>
>> > > > > OpenStack-dev@lists.openstack.org
>> > > >
>> > >

Re: [openstack-dev] [neutron]Performance of security group

[shihanzhang]

hi Miguel Ángel,
I am very agree with you about the following point:
>  * physical implementation on the hosts (ipsets, nftables, ... )
--this can reduce the load of compute node.
>  * rpc communication mechanisms.
  --this can reduce the load of neutron server
can you help me to review my BP specs?










At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo"  wrote:
>
>  Hi it's a very interesting topic, I was getting ready to raise
>the same concerns about our security groups implementation, shihanzhang
>thank you for starting this topic.
>
>  Not only at low level where (with our default security group
>rules -allow all incoming from 'default' sg- the iptable rules
>will grow in ~X^2 for a tenant, and, the "security_group_rules_for_devices"
>rpc call from ovs-agent to neutron-server grows to message sizes of >100MB,
>generating serious scalability issues or timeouts/retries that 
>totally break neutron service.
>
>   (example trace of that RPC call with a few instances
>http://www.fpaste.org/104401/14008522/)
>
>  I believe that we also need to review the RPC calling mechanism
>for the OVS agent here, there are several possible approaches to breaking
>down (or/and CIDR compressing) the information we return via this api call.
>
>
>   So we have to look at two things here:
>
>  * physical implementation on the hosts (ipsets, nftables, ... )
>  * rpc communication mechanisms.
>
>   Best regards,
>Miguel Ángel.
>
>- Mensaje original - 
>
>> Do you though about nftables that will replace {ip,ip6,arp,eb}tables?
>> It also based on the rule set mechanism.
>> The issue in that proposition, it's only stable since the begin of the year
>> and on Linux kernel 3.13.
>> But there lot of pros I don't list here (leverage iptables limitation,
>> efficient update rule, rule set, standardization of netfilter commands...).
>
>> Édouard.
>
>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < henry4...@gmail.com > wrote:
>
>> > we have done some tests, but have different result: the performance is
>> > nearly
>> > the same for empty and 5k rules in iptable, but huge gap between
>> > enable/disable iptable hook on linux bridge
>> 
>
>> > On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < ayshihanzh...@126.com >
>> > wrote:
>> 
>
>> > > Now I have not get accurate test data, but I can confirm the following
>> > > points：
>> > 
>> 
>> > > 1. In compute node, the iptable's chain of a VM is liner, iptable filter
>> > > it
>> > > one by one, if a VM in default security group and this default security
>> > > group have many members, but ipset chain is set, the time ipset filter
>> > > one
>> > > and many member is not much difference.
>> > 
>> 
>> > > 2. when the iptable rule is very large, the probability of failure that
>> > > iptable-save save the iptable rule is very large.
>> > 
>> 
>
>> > > At 2014-06-19 10:55:56, "Kevin Benton" < blak...@gmail.com > wrote:
>> > 
>> 
>
>> > > > This sounds like a good idea to handle some of the performance issues
>> > > > until
>> > > > the ovs firewall can be implemented down the the line.
>> > > 
>> > 
>> 
>> > > > Do you have any performance comparisons?
>> > > 
>> > 
>> 
>> > > > On Jun 18, 2014 7:46 PM, "shihanzhang" < ayshihanzh...@126.com > wrote:
>> > > 
>> > 
>> 
>
>> > > > > Hello all,
>> > > > 
>> > > 
>> > 
>> 
>
>> > > > > Now in neutron, it use iptable implementing security group, but the
>> > > > > performance of this implementation is very poor, there is a bug:
>> > > > > https://bugs.launchpad.net/neutron/+bug/1302272 to reflect this
>> > > > > problem.
>> > > > > In
>> > > > > his test, w ith default security groups(which has remote security
>> > > > > group),
>> > > > > beyond 250-300 VMs, there were around 6k Iptable rules on evry
>> > > > > compute
>> > > > > node,
>> > > > > although his patch can reduce the processing time, but it don't solve
>> > > > > this
>> > > > > problem fundamentally. I have commit a BP to solve this problem:
>> > > > > https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>> > > > 
>> > > 
>> > 
>> 
>> > > > > There are other people interested in this it?
>> > > > 
>> > > 
>> > 
>> 
>
>> > > > > ___
>> > > > 
>> > > 
>> > 
>> 
>> > > > > OpenStack-dev mailing list
>> > > > 
>> > > 
>> > 
>> 
>> > > > > OpenStack-dev@lists.openstack.org
>> > > > 
>> > > 
>> > 
>> 
>> > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > > > 
>> > > 
>> > 
>> 
>
>> > > ___
>> > 
>> 
>> > > OpenStack-dev mailing list
>> > 
>> 
>> > > OpenStack-dev@lists.openstack.org
>> > 
>> 
>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > 
>> 
>
>> > ___
>> 
>> > OpenStack-dev mailing list
>> 
>> > OpenStack-dev@lists.openstack.org
>> 
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>
>> ___
>> OpenStac

Re: [openstack-dev] [neutron]Performance of security group

[Édouard Thuleau]

@Nachi: Yes that could a good improvement to factorize the RPC mechanism.

Another idea:
What about creating a RPC topic per security group (quid of the RPC topic
scalability) on which an agent subscribes if one of its ports is associated
to the security group?

Regards,
Édouard.



On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang  wrote:

> hi Miguel Ángel,
> I am very agree with you about the following point:
>
> >  * physical implementation on the hosts (ipsets, nftables, ... )
>
> --this can reduce the load of compute node.
> >  * rpc communication mechanisms.
>
>   --this can reduce the load of neutron server
>
> can you help me to review my BP specs?
>
>
>
>
>
>
>
>
> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo"  
> wrote:
> >
> >  Hi it's a very interesting topic, I was getting ready to raise
> >the same concerns about our security groups implementation, shihanzhang
> >thank you for starting this topic.
> >
> >  Not only at low level where (with our default security group
> >rules -allow all incoming from 'default' sg- the iptable rules
> >will grow in ~X^2 for a tenant, and, the "security_group_rules_for_devices"
> >rpc call from ovs-agent to neutron-server grows to message sizes of >100MB,
> >generating serious scalability issues or timeouts/retries that
> >totally break neutron service.
> >
> >   (example trace of that RPC call with a few instances
> >http://www.fpaste.org/104401/14008522/)
> >
> >  I believe that we also need to review the RPC calling mechanism
> >for the OVS agent here, there are several possible approaches to breaking
> >down (or/and CIDR compressing) the information we return via this api call.
> >
> >
> >   So we have to look at two things here:
> >
> >  * physical implementation on the hosts (ipsets, nftables, ... )
> >  * rpc communication mechanisms.
> >
> >   Best regards,
> >Miguel Ángel.
> >
> >- Mensaje original -
> >
> >> Do you though about nftables that will replace {ip,ip6,arp,eb}tables?
> >> It also based on the rule set mechanism.
> >> The issue in that proposition, it's only stable since the begin of the year
> >> and on Linux kernel 3.13.
> >> But there lot of pros I don't list here (leverage iptables limitation,
> >> efficient update rule, rule set, standardization of netfilter commands...).
> >
> >> Édouard.
> >
> >> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < henry4...@gmail.com > wrote:
> >
> >> > we have done some tests, but have different result: the performance is
> >> > nearly
> >> > the same for empty and 5k rules in iptable, but huge gap between
> >> > enable/disable iptable hook on linux bridge
> >>
> >
> >> > On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < ayshihanzh...@126.com >
> >> > wrote:
> >>
> >
> >> > > Now I have not get accurate test data, but I can confirm the following
> >> > > points：
> >> >
> >>
> >> > > 1. In compute node, the iptable's chain of a VM is liner, iptable 
> >> > > filter
> >> > > it
> >> > > one by one, if a VM in default security group and this default security
> >> > > group have many members, but ipset chain is set, the time ipset filter
> >> > > one
> >> > > and many member is not much difference.
> >> >
> >>
> >> > > 2. when the iptable rule is very large, the probability of failure that
> >> > > iptable-save save the iptable rule is very large.
> >> >
> >>
> >
> >> > > At 2014-06-19 10:55:56, "Kevin Benton" < blak...@gmail.com > wrote:
> >> >
> >>
> >
> >> > > > This sounds like a good idea to handle some of the performance issues
> >> > > > until
> >> > > > the ovs firewall can be implemented down the the line.
> >> > >
> >> >
> >>
> >> > > > Do you have any performance comparisons?
> >> > >
> >> >
> >>
> >> > > > On Jun 18, 2014 7:46 PM, "shihanzhang" < ayshihanzh...@126.com > 
> >> > > > wrote:
> >> > >
> >> >
> >>
> >
> >> > > > > Hello all,
> >> > > >
> >> > >
> >> >
> >>
> >
> >> > > > > Now in neutron, it use iptable implementing security group, but the
> >> > > > > performance of this implementation is very poor, there is a bug:
> >> > > > > https://bugs.launchpad.net/neutron/+bug/1302272 to reflect this
> >> > > > > problem.
> >> > > > > In
> >> > > > > his test, w ith default security groups(which has remote security
> >> > > > > group),
> >> > > > > beyond 250-300 VMs, there were around 6k Iptable rules on evry
> >> > > > > compute
> >> > > > > node,
> >> > > > > although his patch can reduce the processing time, but it don't 
> >> > > > > solve
> >> > > > > this
> >> > > > > problem fundamentally. I have commit a BP to solve this problem:
> >> > > > > https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
> >> > > >
> >> > >
> >> >
> >>
> >> > > > > There are other people interested in this it?
> >> > > >
> >> > >
> >> >
> >>
> >
> >> > > > > ___
> >> > > >
> >> > >
> >> >
> >>
> >> > > > > OpenStack-dev mailing list
> >> > > >
> >> > >
> >> >
> >>
> >> > > > > OpenStack-dev@lists.openstack.org
> >> > > >
> >> > >
> >> >
> >>
>

Re: [openstack-dev] [neutron]Performance of security group

[Miguel Angel Ajo Pelayo]

- Original Message -
> @Nachi: Yes that could a good improvement to factorize the RPC mechanism.
> 
> Another idea:
> What about creating a RPC topic per security group (quid of the RPC topic
> scalability) on which an agent subscribes if one of its ports is associated
> to the security group?
> 
> Regards,
> Édouard.
> 
> 


Hmm, Interesting,

@Nachi, I'm not sure I fully understood:


SG_LIST [ SG1, SG2]
SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
port[SG_ID1, SG_ID2], port2 , port3


Probably we may need to include also the 
SG_IP_LIST = [SG_IP1, SG_IP2] ...


and let the agent do all the combination work.

Something like this could make sense?

Security_Groups = {SG1:{IPs:[],RULES:[],
   SG2:{IPs:[],RULES:[]} 
  }

Ports = {Port1:[SG1, SG2], Port2: [SG1]  }


@Edouard, actually I like the idea of having the agent subscribed
to security groups they have ports on... That would remove the need to include
all the security groups information on every call...

But would need another call to get the full information of a set of security 
groups
at start/resync if we don't already have any. 


> 
> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang < ayshihanzh...@126.com > wrote:
> 
> 
> 
> hi Miguel Ángel,
> I am very agree with you about the following point:
> >  * physical implementation on the hosts (ipsets, nftables, ... )
> --this can reduce the load of compute node.
> >  * rpc communication mechanisms.
> -- this can reduce the load of neutron server
> can you help me to review my BP specs?
> 
> 
> 
> 
> 
> 
> 
> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" < mangel...@redhat.com >
> wrote:
> >
> >  Hi it's a very interesting topic, I was getting ready to raise
> >the same concerns about our security groups implementation, shihanzhang
> >thank you for starting this topic.
> >
> >  Not only at low level where (with our default security group
> >rules -allow all incoming from 'default' sg- the iptable rules
> >will grow in ~X^2 for a tenant, and, the "security_group_rules_for_devices"
> >rpc call from ovs-agent to neutron-server grows to message sizes of >100MB,
> >generating serious scalability issues or timeouts/retries that
> >totally break neutron service.
> >
> >   (example trace of that RPC call with a few instances
> > http://www.fpaste.org/104401/14008522/ )
> >
> >  I believe that we also need to review the RPC calling mechanism
> >for the OVS agent here, there are several possible approaches to breaking
> >down (or/and CIDR compressing) the information we return via this api call.
> >
> >
> >   So we have to look at two things here:
> >
> >  * physical implementation on the hosts (ipsets, nftables, ... )
> >  * rpc communication mechanisms.
> >
> >   Best regards,
> >Miguel Ángel.
> >
> >- Mensaje original -
> >
> >> Do you though about nftables that will replace {ip,ip6,arp,eb}tables?
> >> It also based on the rule set mechanism.
> >> The issue in that proposition, it's only stable since the begin of the
> >> year
> >> and on Linux kernel 3.13.
> >> But there lot of pros I don't list here (leverage iptables limitation,
> >> efficient update rule, rule set, standardization of netfilter
> >> commands...).
> >
> >> Édouard.
> >
> >> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < henry4...@gmail.com > wrote:
> >
> >> > we have done some tests, but have different result: the performance is
> >> > nearly
> >> > the same for empty and 5k rules in iptable, but huge gap between
> >> > enable/disable iptable hook on linux bridge
> >> 
> >
> >> > On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < ayshihanzh...@126.com >
> >> > wrote:
> >> 
> >
> >> > > Now I have not get accurate test data, but I can confirm the following
> >> > > points：
> >> > 
> >> 
> >> > > 1. In compute node, the iptable's chain of a VM is liner, iptable
> >> > > filter
> >> > > it
> >> > > one by one, if a VM in default security group and this default
> >> > > security
> >> > > group have many members, but ipset chain is set, the time ipset filter
> >> > > one
> >> > > and many member is not much difference.
> >> > 
> >> 
> >> > > 2. when the iptable rule is very large, the probability of failure
> >> > > that
> >> > > iptable-save save the iptable rule is very large.
> >> > 
> >> 
> >
> >> > > At 2014-06-19 10:55:56, "Kevin Benton" < blak...@gmail.com > wrote:
> >> > 
> >> 
> >
> >> > > > This sounds like a good idea to handle some of the performance
> >> > > > issues
> >> > > > until
> >> > > > the ovs firewall can be implemented down the the line.
> >> > > 
> >> > 
> >> 
> >> > > > Do you have any performance comparisons?
> >> > > 
> >> > 
> >> 
> >> > > > On Jun 18, 2014 7:46 PM, "shihanzhang" < ayshihanzh...@126.com >
> >> > > > wrote:
> >> > > 
> >> > 
> >> 
> >
> >> > > > > Hello all,
> >> > > > 
> >> > > 
> >> > 
> >> 
> >
> >> > > > > Now in neutron, it use iptable implementing security group, but
> >> > > > > the
> >> > > > > performance of this implementation is ver

Re: [openstack-dev] [neutron]Performance of security group

[Miguel Angel Ajo Pelayo]

Ok, I was talking with Édouard @ IRC, and as I have time to work 
into this problem, I could file an specific spec for the security
group RPC optimization, a masterplan in two steps:

1) Refactor the current RPC communication for security_groups_for_devices,
   which could be used for full syncs, etc..

2) Benchmark && make use of a fanout queue per security group to make
   sure only the hosts with instances on a certain security group get
   the updates as they happen.

@shihanzhang do you find it reasonable?



- Original Message -
> - Original Message -
> > @Nachi: Yes that could a good improvement to factorize the RPC mechanism.
> > 
> > Another idea:
> > What about creating a RPC topic per security group (quid of the RPC topic
> > scalability) on which an agent subscribes if one of its ports is associated
> > to the security group?
> > 
> > Regards,
> > Édouard.
> > 
> > 
> 
> 
> Hmm, Interesting,
> 
> @Nachi, I'm not sure I fully understood:
> 
> 
> SG_LIST [ SG1, SG2]
> SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
> port[SG_ID1, SG_ID2], port2 , port3
> 
> 
> Probably we may need to include also the
> SG_IP_LIST = [SG_IP1, SG_IP2] ...
> 
> 
> and let the agent do all the combination work.
> 
> Something like this could make sense?
> 
> Security_Groups = {SG1:{IPs:[],RULES:[],
>SG2:{IPs:[],RULES:[]}
>   }
> 
> Ports = {Port1:[SG1, SG2], Port2: [SG1]  }
> 
> 
> @Edouard, actually I like the idea of having the agent subscribed
> to security groups they have ports on... That would remove the need to
> include
> all the security groups information on every call...
> 
> But would need another call to get the full information of a set of security
> groups
> at start/resync if we don't already have any.
> 
> 
> > 
> > On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang < ayshihanzh...@126.com >
> > wrote:
> > 
> > 
> > 
> > hi Miguel Ángel,
> > I am very agree with you about the following point:
> > >  * physical implementation on the hosts (ipsets, nftables, ... )
> > --this can reduce the load of compute node.
> > >  * rpc communication mechanisms.
> > -- this can reduce the load of neutron server
> > can you help me to review my BP specs?
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" < mangel...@redhat.com >
> > wrote:
> > >
> > >  Hi it's a very interesting topic, I was getting ready to raise
> > >the same concerns about our security groups implementation, shihanzhang
> > >thank you for starting this topic.
> > >
> > >  Not only at low level where (with our default security group
> > >rules -allow all incoming from 'default' sg- the iptable rules
> > >will grow in ~X^2 for a tenant, and, the
> > >"security_group_rules_for_devices"
> > >rpc call from ovs-agent to neutron-server grows to message sizes of
> > >>100MB,
> > >generating serious scalability issues or timeouts/retries that
> > >totally break neutron service.
> > >
> > >   (example trace of that RPC call with a few instances
> > > http://www.fpaste.org/104401/14008522/ )
> > >
> > >  I believe that we also need to review the RPC calling mechanism
> > >for the OVS agent here, there are several possible approaches to breaking
> > >down (or/and CIDR compressing) the information we return via this api
> > >call.
> > >
> > >
> > >   So we have to look at two things here:
> > >
> > >  * physical implementation on the hosts (ipsets, nftables, ... )
> > >  * rpc communication mechanisms.
> > >
> > >   Best regards,
> > >Miguel Ángel.
> > >
> > >- Mensaje original -
> > >
> > >> Do you though about nftables that will replace {ip,ip6,arp,eb}tables?
> > >> It also based on the rule set mechanism.
> > >> The issue in that proposition, it's only stable since the begin of the
> > >> year
> > >> and on Linux kernel 3.13.
> > >> But there lot of pros I don't list here (leverage iptables limitation,
> > >> efficient update rule, rule set, standardization of netfilter
> > >> commands...).
> > >
> > >> Édouard.
> > >
> > >> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < henry4...@gmail.com >
> > >> wrote:
> > >
> > >> > we have done some tests, but have different result: the performance is
> > >> > nearly
> > >> > the same for empty and 5k rules in iptable, but huge gap between
> > >> > enable/disable iptable hook on linux bridge
> > >> 
> > >
> > >> > On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < ayshihanzh...@126.com
> > >> > >
> > >> > wrote:
> > >> 
> > >
> > >> > > Now I have not get accurate test data, but I can confirm the
> > >> > > following
> > >> > > points：
> > >> > 
> > >> 
> > >> > > 1. In compute node, the iptable's chain of a VM is liner, iptable
> > >> > > filter
> > >> > > it
> > >> > > one by one, if a VM in default security group and this default
> > >> > > security
> > >> > > group have many members, but ipset chain is set, the time ipset
> > >> > > filter
> > >> > > one
> > >> > > and many member is not much difference.
> > >> > 
>

Re: [openstack-dev] [neutron]Performance of security group

[shihanzhang]

hi Miguel Angel Ajo Pelayo!
I agree with you and modify my spes, but I will also optimization the RPC from 
security group agent to neutron server. 
Now the modle is 'port[rule1,rule2...], port...', I will change it to 
'port[sg1, sg2..]', this can reduce the size of RPC respose message from 
neutron server to security group agent. 
At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"  wrote:
>
>
>Ok, I was talking with Édouard @ IRC, and as I have time to work 
>into this problem, I could file an specific spec for the security
>group RPC optimization, a masterplan in two steps:
>
>1) Refactor the current RPC communication for security_groups_for_devices,
>   which could be used for full syncs, etc..
>
>2) Benchmark && make use of a fanout queue per security group to make
>   sure only the hosts with instances on a certain security group get
>   the updates as they happen.
>
>@shihanzhang do you find it reasonable?
>
>
>
>- Original Message -
>> - Original Message -
>> > @Nachi: Yes that could a good improvement to factorize the RPC mechanism.
>> > 
>> > Another idea:
>> > What about creating a RPC topic per security group (quid of the RPC topic
>> > scalability) on which an agent subscribes if one of its ports is associated
>> > to the security group?
>> > 
>> > Regards,
>> > Édouard.
>> > 
>> > 
>> 
>> 
>> Hmm, Interesting,
>> 
>> @Nachi, I'm not sure I fully understood:
>> 
>> 
>> SG_LIST [ SG1, SG2]
>> SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
>> port[SG_ID1, SG_ID2], port2 , port3
>> 
>> 
>> Probably we may need to include also the
>> SG_IP_LIST = [SG_IP1, SG_IP2] ...
>> 
>> 
>> and let the agent do all the combination work.
>> 
>> Something like this could make sense?
>> 
>> Security_Groups = {SG1:{IPs:[],RULES:[],
>>SG2:{IPs:[],RULES:[]}
>>   }
>> 
>> Ports = {Port1:[SG1, SG2], Port2: [SG1]  }
>> 
>> 
>> @Edouard, actually I like the idea of having the agent subscribed
>> to security groups they have ports on... That would remove the need to
>> include
>> all the security groups information on every call...
>> 
>> But would need another call to get the full information of a set of security
>> groups
>> at start/resync if we don't already have any.
>> 
>> 
>> > 
>> > On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang < ayshihanzh...@126.com >
>> > wrote:
>> > 
>> > 
>> > 
>> > hi Miguel Ángel,
>> > I am very agree with you about the following point:
>> > >  * physical implementation on the hosts (ipsets, nftables, ... )
>> > --this can reduce the load of compute node.
>> > >  * rpc communication mechanisms.
>> > -- this can reduce the load of neutron server
>> > can you help me to review my BP specs?
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" < mangel...@redhat.com >
>> > wrote:
>> > >
>> > >  Hi it's a very interesting topic, I was getting ready to raise
>> > >the same concerns about our security groups implementation, shihanzhang
>> > >thank you for starting this topic.
>> > >
>> > >  Not only at low level where (with our default security group
>> > >rules -allow all incoming from 'default' sg- the iptable rules
>> > >will grow in ~X^2 for a tenant, and, the
>> > >"security_group_rules_for_devices"
>> > >rpc call from ovs-agent to neutron-server grows to message sizes of
>> > >>100MB,
>> > >generating serious scalability issues or timeouts/retries that
>> > >totally break neutron service.
>> > >
>> > >   (example trace of that RPC call with a few instances
>> > > http://www.fpaste.org/104401/14008522/ )
>> > >
>> > >  I believe that we also need to review the RPC calling mechanism
>> > >for the OVS agent here, there are several possible approaches to breaking
>> > >down (or/and CIDR compressing) the information we return via this api
>> > >call.
>> > >
>> > >
>> > >   So we have to look at two things here:
>> > >
>> > >  * physical implementation on the hosts (ipsets, nftables, ... )
>> > >  * rpc communication mechanisms.
>> > >
>> > >   Best regards,
>> > >Miguel Ángel.
>> > >
>> > >- Mensaje original -
>> > >
>> > >> Do you though about nftables that will replace {ip,ip6,arp,eb}tables?
>> > >> It also based on the rule set mechanism.
>> > >> The issue in that proposition, it's only stable since the begin of the
>> > >> year
>> > >> and on Linux kernel 3.13.
>> > >> But there lot of pros I don't list here (leverage iptables limitation,
>> > >> efficient update rule, rule set, standardization of netfilter
>> > >> commands...).
>> > >
>> > >> Édouard.
>> > >
>> > >> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < henry4...@gmail.com >
>> > >> wrote:
>> > >
>> > >> > we have done some tests, but have different result: the performance is
>> > >> > nearly
>> > >> > the same for empty and 5k rules in iptable, but huge gap between
>> > >> > enable/disable iptable hook on linux bridge
>> > >> 
>> > >
>> > >> > On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < ayshihanzh...@126.com
>> > >

Re: [openstack-dev] [neutron]Performance of security group

[Miguel Angel Ajo]

Shihazhang,

I really believe we need the RPC refactor done for this cycle,
and given the close deadlines we have (July 10 for spec submission and 
July 20 for spec approval).


Don't you think it's going to be better to split the work in several
specs?

1) ipset optimization   (you)
2) sg rpc optimization (without fanout) (me)
3) sg rpc optimization (with fanout) (edouard, you , me)


   This way we increase the chances of having part of this for the Juno
cycle. If we go for something too complicated is going to take more time 
for approval.



  Also, I proposed the details of "2", trying to bring awareness on the
topic, as I have been working with the scale lab in Red Hat to find
and understand those issues, I have a very good knowledge of the problem
and I believe I could make a very fast advance on the issue at the
RPC level.

  Given that, I'd like to work on this specific part, whether or
not we split the specs, as it's something we believe critical for
neutron scalability and thus, *nova parity*.

   I will start a separate spec for "2", later on, if you find it ok,
we keep them as separate ones, if you believe having just 1 spec (for 1
 & 2) is going be safer for juno-* approval, then we can incorporate my 
spec in yours, but then "add-ipset-to-security" is not a good spec title 
to put all this together.



   Best regards,
Miguel Ángel.


On 07/02/2014 03:37 AM, shihanzhang wrote:


hi Miguel Angel Ajo Pelayo!
I agree with you and modify my spes, but I will also optimization the
RPC from security group agent to neutron server.
Now the modle is 'port[rule1,rule2...], port...', I will change it to
'port[sg1, sg2..]', this can reduce the size of RPC respose message from
neutron server to security group agent.

At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"  wrote:



Ok, I was talking with Édouard @ IRC, and as I have time to work
into this problem, I could file an specific spec for the security
group RPC optimization, a masterplan in two steps:

1) Refactor the current RPC communication for security_groups_for_devices,
  which could be used for full syncs, etc..

2) Benchmark && make use of a fanout queue per security group to make
  sure only the hosts with instances on a certain security group get
  the updates as they happen.

@shihanzhang do you find it reasonable?



- Original Message -

- Original Message -
> @Nachi: Yes that could a good improvement to factorize the RPC mechanism.
>
> Another idea:
> What about creating a RPC topic per security group (quid of the RPC topic
> scalability) on which an agent subscribes if one of its ports is associated
> to the security group?
>
> Regards,
> Édouard.
>
>


Hmm, Interesting,

@Nachi, I'm not sure I fully understood:


SG_LIST [ SG1, SG2]
SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
port[SG_ID1, SG_ID2], port2 , port3


Probably we may need to include also the
SG_IP_LIST = [SG_IP1, SG_IP2] ...


and let the agent do all the combination work.

Something like this could make sense?

Security_Groups = {SG1:{IPs:[],RULES:[],
   SG2:{IPs:[],RULES:[]}
  }

Ports = {Port1:[SG1, SG2], Port2: [SG1]  }


@Edouard, actually I like the idea of having the agent subscribed
to security groups they have ports on... That would remove the need to
include
all the security groups information on every call...

But would need another call to get the full information of a set of security
groups
at start/resync if we don't already have any.


>
> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang < ayshihanzh...@126.com >
> wrote:
>
>
>
> hi Miguel Ángel,
> I am very agree with you about the following point:
> >  * physical implementation on the hosts (ipsets, nftables, ... )
> --this can reduce the load of compute node.
> >  * rpc communication mechanisms.
> -- this can reduce the load of neutron server
> can you help me to review my BP specs?
>
>
>
>
>
>
>
> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" < mangel...@redhat.com >
> wrote:
> >
> >  Hi it's a very interesting topic, I was getting ready to raise
> >the same concerns about our security groups implementation, shihanzhang
> >thank you for starting this topic.
> >
> >  Not only at low level where (with our default security group
> >rules -allow all incoming from 'default' sg- the iptable rules
> >will grow in ~X^2 for a tenant, and, the
> >"security_group_rules_for_devices"
> >rpc call from ovs-agent to neutron-server grows to message sizes of
> >>100MB,
> >generating serious scalability issues or timeouts/retries that
> >totally break neutron service.
> >
> >   (example trace of that RPC call with a few instances
> > http://www.fpaste.org/104401/14008522/ )
> >
> >  I believe that we also need to review the RPC calling mechanism
> >for the OVS agent here, there are several possible approaches to breaking
> >down (or/and CIDR compressing) the information we return via this api
> >call.
> >
> >
> >   So we have to look at two things here

Re: [openstack-dev] [neutron]Performance of security group

[Ihar Hrachyshka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 02/07/14 10:12, Miguel Angel Ajo wrote:
> 
> Shihazhang,
> 
> I really believe we need the RPC refactor done for this cycle, and
> given the close deadlines we have (July 10 for spec submission and 
> July 20 for spec approval).
> 
> Don't you think it's going to be better to split the work in
> several specs?
> 
> 1) ipset optimization   (you) 2) sg rpc optimization (without
> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you ,
> me)
> 
> 
> This way we increase the chances of having part of this for the
> Juno cycle. If we go for something too complicated is going to take
> more time for approval.
> 

I agree. And it not only increases chances to get at least some of
those highly demanded performance enhancements to get into Juno, it's
also "the right thing to do" (c). It's counterproductive to put
multiple vaguely related enhancements in single spec. This would dim
review focus and put us into position of getting 'all-or-nothing'. We
can't afford that.

Let's leave one spec per enhancement. @Shihazhang, what do you think?

> 
> Also, I proposed the details of "2", trying to bring awareness on
> the topic, as I have been working with the scale lab in Red Hat to
> find and understand those issues, I have a very good knowledge of
> the problem and I believe I could make a very fast advance on the
> issue at the RPC level.
> 
> Given that, I'd like to work on this specific part, whether or not
> we split the specs, as it's something we believe critical for 
> neutron scalability and thus, *nova parity*.
> 
> I will start a separate spec for "2", later on, if you find it ok, 
> we keep them as separate ones, if you believe having just 1 spec
> (for 1 & 2) is going be safer for juno-* approval, then we can
> incorporate my spec in yours, but then "add-ipset-to-security" is
> not a good spec title to put all this together.
> 
> 
> Best regards, Miguel Ángel.
> 
> 
> On 07/02/2014 03:37 AM, shihanzhang wrote:
>> 
>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my spes,
>> but I will also optimization the RPC from security group agent to
>> neutron server. Now the modle is 'port[rule1,rule2...], port...',
>> I will change it to 'port[sg1, sg2..]', this can reduce the size
>> of RPC respose message from neutron server to security group
>> agent.
>> 
>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo" 
>>  wrote:
>>> 
>>> 
>>> Ok, I was talking with Édouard @ IRC, and as I have time to
>>> work into this problem, I could file an specific spec for the
>>> security group RPC optimization, a masterplan in two steps:
>>> 
>>> 1) Refactor the current RPC communication for 
>>> security_groups_for_devices, which could be used for full
>>> syncs, etc..
>>> 
>>> 2) Benchmark && make use of a fanout queue per security group
>>> to make sure only the hosts with instances on a certain
>>> security group get the updates as they happen.
>>> 
>>> @shihanzhang do you find it reasonable?
>>> 
>>> 
>>> 
>>> - Original Message -
 - Original Message -
> @Nachi: Yes that could a good improvement to factorize the
> RPC
 mechanism.
> 
> Another idea: What about creating a RPC topic per security
> group (quid of the
 RPC topic
> scalability) on which an agent subscribes if one of its
> ports is
 associated
> to the security group?
> 
> Regards, Édouard.
> 
> 
 
 
 Hmm, Interesting,
 
 @Nachi, I'm not sure I fully understood:
 
 
 SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] .. 
 port[SG_ID1, SG_ID2], port2 , port3
 
 
 Probably we may need to include also the SG_IP_LIST =
 [SG_IP1, SG_IP2] ...
 
 
 and let the agent do all the combination work.
 
 Something like this could make sense?
 
 Security_Groups = {SG1:{IPs:[],RULES:[], 
 SG2:{IPs:[],RULES:[]} }
 
 Ports = {Port1:[SG1, SG2], Port2: [SG1]  }
 
 
 @Edouard, actually I like the idea of having the agent
 subscribed to security groups they have ports on... That
 would remove the need to include all the security groups
 information on every call...
 
 But would need another call to get the full information of a
 set of security groups at start/resync if we don't already
 have any.
 
 
> 
> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
 ayshihanzh...@126.com >
> wrote:
> 
> 
> 
> hi Miguel Ángel, I am very agree with you about the
> following point:
>> * physical implementation on the hosts (ipsets, nftables,
>> ... )
> --this can reduce the load of compute node.
>> * rpc communication mechanisms.
> -- this can reduce the load of neutron server can you help
> me to review my BP specs?
> 
> 
> 
> 
> 
> 
> 
> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
 mangel..

Re: [openstack-dev] [neutron]Performance of security group

[shihanzhang]

hi Miguel Ángel and Ihar Hrachyshka,
I agree with you that split  the work in several specs, I have finished the 
work ( ipset optimization), you can do 'sg rpc optimization (without fanout)'.
as the third part(sg rpc optimization (with fanout)),  I think we need talk 
about it, because just using ipset to optimize security group agent codes does 
not bring the best results!


Best regards,
shihanzhang.










At 2014-07-02 04:43:24, "Ihar Hrachyshka"  wrote:
>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA512
>
>On 02/07/14 10:12, Miguel Angel Ajo wrote:
>> 
>> Shihazhang,
>> 
>> I really believe we need the RPC refactor done for this cycle, and
>> given the close deadlines we have (July 10 for spec submission and 
>> July 20 for spec approval).
>> 
>> Don't you think it's going to be better to split the work in
>> several specs?
>> 
>> 1) ipset optimization   (you) 2) sg rpc optimization (without
>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you ,
>> me)
>> 
>> 
>> This way we increase the chances of having part of this for the
>> Juno cycle. If we go for something too complicated is going to take
>> more time for approval.
>> 
>
>I agree. And it not only increases chances to get at least some of
>those highly demanded performance enhancements to get into Juno, it's
>also "the right thing to do" (c). It's counterproductive to put
>multiple vaguely related enhancements in single spec. This would dim
>review focus and put us into position of getting 'all-or-nothing'. We
>can't afford that.
>
>Let's leave one spec per enhancement. @Shihazhang, what do you think?
>
>> 
>> Also, I proposed the details of "2", trying to bring awareness on
>> the topic, as I have been working with the scale lab in Red Hat to
>> find and understand those issues, I have a very good knowledge of
>> the problem and I believe I could make a very fast advance on the
>> issue at the RPC level.
>> 
>> Given that, I'd like to work on this specific part, whether or not
>> we split the specs, as it's something we believe critical for 
>> neutron scalability and thus, *nova parity*.
>> 
>> I will start a separate spec for "2", later on, if you find it ok, 
>> we keep them as separate ones, if you believe having just 1 spec
>> (for 1 & 2) is going be safer for juno-* approval, then we can
>> incorporate my spec in yours, but then "add-ipset-to-security" is
>> not a good spec title to put all this together.
>> 
>> 
>> Best regards, Miguel Ángel.
>> 
>> 
>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>>> 
>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my spes,
>>> but I will also optimization the RPC from security group agent to
>>> neutron server. Now the modle is 'port[rule1,rule2...], port...',
>>> I will change it to 'port[sg1, sg2..]', this can reduce the size
>>> of RPC respose message from neutron server to security group
>>> agent.
>>> 
>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo" 
>>>  wrote:
 
 
 Ok, I was talking with Édouard @ IRC, and as I have time to
 work into this problem, I could file an specific spec for the
 security group RPC optimization, a masterplan in two steps:
 
 1) Refactor the current RPC communication for 
 security_groups_for_devices, which could be used for full
 syncs, etc..
 
 2) Benchmark && make use of a fanout queue per security group
 to make sure only the hosts with instances on a certain
 security group get the updates as they happen.
 
 @shihanzhang do you find it reasonable?
 
 
 
 - Original Message -
> - Original Message -
>> @Nachi: Yes that could a good improvement to factorize the
>> RPC
> mechanism.
>> 
>> Another idea: What about creating a RPC topic per security
>> group (quid of the
> RPC topic
>> scalability) on which an agent subscribes if one of its
>> ports is
> associated
>> to the security group?
>> 
>> Regards, Édouard.
>> 
>> 
> 
> 
> Hmm, Interesting,
> 
> @Nachi, I'm not sure I fully understood:
> 
> 
> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] .. 
> port[SG_ID1, SG_ID2], port2 , port3
> 
> 
> Probably we may need to include also the SG_IP_LIST =
> [SG_IP1, SG_IP2] ...
> 
> 
> and let the agent do all the combination work.
> 
> Something like this could make sense?
> 
> Security_Groups = {SG1:{IPs:[],RULES:[], 
> SG2:{IPs:[],RULES:[]} }
> 
> Ports = {Port1:[SG1, SG2], Port2: [SG1]  }
> 
> 
> @Edouard, actually I like the idea of having the agent
> subscribed to security groups they have ports on... That
> would remove the need to include all the security groups
> information on every call...
> 
> But would need another call to get the full information of a
> set of security groups at start/resync if we don't already
> have a

Re: [openstack-dev] [neutron]Performance of security group

[Miguel Angel Ajo]

Nice Shihanzhang,

  Do you mean the ipset implementation is ready, or just the spec?.


  For the SG group refactor, I don't worry about who does it, or who
takes the credit, but I believe it's important we address this 
bottleneck during Juno trying to match nova's scalability.


Best regards,
Miguel Ángel.


On 07/02/2014 02:50 PM, shihanzhang wrote:

hi Miguel Ángel and Ihar Hrachyshka,
I agree with you that split  the work in several specs, I have finished
the work ( ipset optimization), you can do 'sg rpc optimization (without
fanout)'.
as the third part(sg rpc optimization (with fanout)),  I think we need
talk about it, because just using ipset to optimize security group agent
codes does not bring the best results!

 Best regards,
shihanzhang.








At 2014-07-02 04:43:24, "Ihar Hrachyshka"  wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 02/07/14 10:12, Miguel Angel Ajo wrote:


Shihazhang,

I really believe we need the RPC refactor done for this cycle, and
given the close deadlines we have (July 10 for spec submission and
July 20 for spec approval).

Don't you think it's going to be better to split the work in
several specs?

1) ipset optimization   (you) 2) sg rpc optimization (without
fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you ,
me)


This way we increase the chances of having part of this for the
Juno cycle. If we go for something too complicated is going to take
more time for approval.



I agree. And it not only increases chances to get at least some of
those highly demanded performance enhancements to get into Juno, it's
also "the right thing to do" (c). It's counterproductive to put
multiple vaguely related enhancements in single spec. This would dim
review focus and put us into position of getting 'all-or-nothing'. We
can't afford that.

Let's leave one spec per enhancement. @Shihazhang, what do you think?



Also, I proposed the details of "2", trying to bring awareness on
the topic, as I have been working with the scale lab in Red Hat to
find and understand those issues, I have a very good knowledge of
the problem and I believe I could make a very fast advance on the
issue at the RPC level.

Given that, I'd like to work on this specific part, whether or not
we split the specs, as it's something we believe critical for
neutron scalability and thus, *nova parity*.

I will start a separate spec for "2", later on, if you find it ok,
we keep them as separate ones, if you believe having just 1 spec
(for 1 & 2) is going be safer for juno-* approval, then we can
incorporate my spec in yours, but then "add-ipset-to-security" is
not a good spec title to put all this together.


Best regards, Miguel Ángel.


On 07/02/2014 03:37 AM, shihanzhang wrote:


hi Miguel Angel Ajo Pelayo! I agree with you and modify my spes,
but I will also optimization the RPC from security group agent to
neutron server. Now the modle is 'port[rule1,rule2...], port...',
I will change it to 'port[sg1, sg2..]', this can reduce the size
of RPC respose message from neutron server to security group
agent.

At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
 wrote:



Ok, I was talking with Édouard @ IRC, and as I have time to
work into this problem, I could file an specific spec for the
security group RPC optimization, a masterplan in two steps:

1) Refactor the current RPC communication for
security_groups_for_devices, which could be used for full
syncs, etc..

2) Benchmark && make use of a fanout queue per security group
to make sure only the hosts with instances on a certain
security group get the updates as they happen.

@shihanzhang do you find it reasonable?



- Original Message -

- Original Message -

@Nachi: Yes that could a good improvement to factorize the
RPC

mechanism.


Another idea: What about creating a RPC topic per security
group (quid of the

RPC topic

scalability) on which an agent subscribes if one of its
ports is

associated

to the security group?

Regards, Édouard.





Hmm, Interesting,

@Nachi, I'm not sure I fully understood:


SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
port[SG_ID1, SG_ID2], port2 , port3


Probably we may need to include also the SG_IP_LIST =
[SG_IP1, SG_IP2] ...


and let the agent do all the combination work.

Something like this could make sense?

Security_Groups = {SG1:{IPs:[],RULES:[],
SG2:{IPs:[],RULES:[]} }

Ports = {Port1:[SG1, SG2], Port2: [SG1]  }


@Edouard, actually I like the idea of having the agent
subscribed to security groups they have ports on... That
would remove the need to include all the security groups
information on every call...

But would need another call to get the full information of a
set of security groups at start/resync if we don't already
have any.




On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <

ayshihanzh...@126.com >

wrote:



hi Miguel Ángel, I am very agree with you about the
following point:

* physical implementation on the hosts (ipsets, nftable

Re: [openstack-dev] [neutron]Performance of security group

[Kyle Mestery]

On Wed, Jul 2, 2014 at 3:43 AM, Ihar Hrachyshka  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>>
>> Shihazhang,
>>
>> I really believe we need the RPC refactor done for this cycle, and
>> given the close deadlines we have (July 10 for spec submission and
>> July 20 for spec approval).
>>
>> Don't you think it's going to be better to split the work in
>> several specs?
>>
>> 1) ipset optimization   (you) 2) sg rpc optimization (without
>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you ,
>> me)
>>
>>
>> This way we increase the chances of having part of this for the
>> Juno cycle. If we go for something too complicated is going to take
>> more time for approval.
>>
>
> I agree. And it not only increases chances to get at least some of
> those highly demanded performance enhancements to get into Juno, it's
> also "the right thing to do" (c). It's counterproductive to put
> multiple vaguely related enhancements in single spec. This would dim
> review focus and put us into position of getting 'all-or-nothing'. We
> can't afford that.
>
> Let's leave one spec per enhancement. @Shihazhang, what do you think?
>
+100

File these as separate specs, and lets see how much of this we can get
into Juno.

Thanks for taking this enhancement and performance improvement on everyone!

Kyle

>>
>> Also, I proposed the details of "2", trying to bring awareness on
>> the topic, as I have been working with the scale lab in Red Hat to
>> find and understand those issues, I have a very good knowledge of
>> the problem and I believe I could make a very fast advance on the
>> issue at the RPC level.
>>
>> Given that, I'd like to work on this specific part, whether or not
>> we split the specs, as it's something we believe critical for
>> neutron scalability and thus, *nova parity*.
>>
>> I will start a separate spec for "2", later on, if you find it ok,
>> we keep them as separate ones, if you believe having just 1 spec
>> (for 1 & 2) is going be safer for juno-* approval, then we can
>> incorporate my spec in yours, but then "add-ipset-to-security" is
>> not a good spec title to put all this together.
>>
>>
>> Best regards, Miguel Ángel.
>>
>>
>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>>>
>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my spes,
>>> but I will also optimization the RPC from security group agent to
>>> neutron server. Now the modle is 'port[rule1,rule2...], port...',
>>> I will change it to 'port[sg1, sg2..]', this can reduce the size
>>> of RPC respose message from neutron server to security group
>>> agent.
>>>
>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
>>>  wrote:


 Ok, I was talking with Édouard @ IRC, and as I have time to
 work into this problem, I could file an specific spec for the
 security group RPC optimization, a masterplan in two steps:

 1) Refactor the current RPC communication for
 security_groups_for_devices, which could be used for full
 syncs, etc..

 2) Benchmark && make use of a fanout queue per security group
 to make sure only the hosts with instances on a certain
 security group get the updates as they happen.

 @shihanzhang do you find it reasonable?



 - Original Message -
> - Original Message -
>> @Nachi: Yes that could a good improvement to factorize the
>> RPC
> mechanism.
>>
>> Another idea: What about creating a RPC topic per security
>> group (quid of the
> RPC topic
>> scalability) on which an agent subscribes if one of its
>> ports is
> associated
>> to the security group?
>>
>> Regards, Édouard.
>>
>>
>
>
> Hmm, Interesting,
>
> @Nachi, I'm not sure I fully understood:
>
>
> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
> port[SG_ID1, SG_ID2], port2 , port3
>
>
> Probably we may need to include also the SG_IP_LIST =
> [SG_IP1, SG_IP2] ...
>
>
> and let the agent do all the combination work.
>
> Something like this could make sense?
>
> Security_Groups = {SG1:{IPs:[],RULES:[],
> SG2:{IPs:[],RULES:[]} }
>
> Ports = {Port1:[SG1, SG2], Port2: [SG1]  }
>
>
> @Edouard, actually I like the idea of having the agent
> subscribed to security groups they have ports on... That
> would remove the need to include all the security groups
> information on every call...
>
> But would need another call to get the full information of a
> set of security groups at start/resync if we don't already
> have any.
>
>
>>
>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
> ayshihanzh...@126.com >
>> wrote:
>>
>>
>>
>> hi Miguel Ángel, I am very agree with you about the
>> following point:
>>> * physical implementation on the hosts (ipsets, nf

Re: [openstack-dev] [neutron]Performance of security group

[Miguel Angel Ajo]

I have created a separate spec for the RPC part.

https://review.openstack.org/104522


On 07/02/2014 05:52 PM, Kyle Mestery wrote:

On Wed, Jul 2, 2014 at 3:43 AM, Ihar Hrachyshka  wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 02/07/14 10:12, Miguel Angel Ajo wrote:


Shihazhang,

I really believe we need the RPC refactor done for this cycle, and
given the close deadlines we have (July 10 for spec submission and
July 20 for spec approval).

Don't you think it's going to be better to split the work in
several specs?

1) ipset optimization   (you) 2) sg rpc optimization (without
fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you ,
me)


This way we increase the chances of having part of this for the
Juno cycle. If we go for something too complicated is going to take
more time for approval.



I agree. And it not only increases chances to get at least some of
those highly demanded performance enhancements to get into Juno, it's
also "the right thing to do" (c). It's counterproductive to put
multiple vaguely related enhancements in single spec. This would dim
review focus and put us into position of getting 'all-or-nothing'. We
can't afford that.

Let's leave one spec per enhancement. @Shihazhang, what do you think?


+100

File these as separate specs, and lets see how much of this we can get
into Juno.

Thanks for taking this enhancement and performance improvement on everyone!

Kyle



Also, I proposed the details of "2", trying to bring awareness on
the topic, as I have been working with the scale lab in Red Hat to
find and understand those issues, I have a very good knowledge of
the problem and I believe I could make a very fast advance on the
issue at the RPC level.

Given that, I'd like to work on this specific part, whether or not
we split the specs, as it's something we believe critical for
neutron scalability and thus, *nova parity*.

I will start a separate spec for "2", later on, if you find it ok,
we keep them as separate ones, if you believe having just 1 spec
(for 1 & 2) is going be safer for juno-* approval, then we can
incorporate my spec in yours, but then "add-ipset-to-security" is
not a good spec title to put all this together.


Best regards, Miguel Ángel.


On 07/02/2014 03:37 AM, shihanzhang wrote:


hi Miguel Angel Ajo Pelayo! I agree with you and modify my spes,
but I will also optimization the RPC from security group agent to
neutron server. Now the modle is 'port[rule1,rule2...], port...',
I will change it to 'port[sg1, sg2..]', this can reduce the size
of RPC respose message from neutron server to security group
agent.

At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
 wrote:



Ok, I was talking with Édouard @ IRC, and as I have time to
work into this problem, I could file an specific spec for the
security group RPC optimization, a masterplan in two steps:

1) Refactor the current RPC communication for
security_groups_for_devices, which could be used for full
syncs, etc..

2) Benchmark && make use of a fanout queue per security group
to make sure only the hosts with instances on a certain
security group get the updates as they happen.

@shihanzhang do you find it reasonable?



- Original Message -

- Original Message -

@Nachi: Yes that could a good improvement to factorize the
RPC

mechanism.


Another idea: What about creating a RPC topic per security
group (quid of the

RPC topic

scalability) on which an agent subscribes if one of its
ports is

associated

to the security group?

Regards, Édouard.





Hmm, Interesting,

@Nachi, I'm not sure I fully understood:


SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
port[SG_ID1, SG_ID2], port2 , port3


Probably we may need to include also the SG_IP_LIST =
[SG_IP1, SG_IP2] ...


and let the agent do all the combination work.

Something like this could make sense?

Security_Groups = {SG1:{IPs:[],RULES:[],
SG2:{IPs:[],RULES:[]} }

Ports = {Port1:[SG1, SG2], Port2: [SG1]  }


@Edouard, actually I like the idea of having the agent
subscribed to security groups they have ports on... That
would remove the need to include all the security groups
information on every call...

But would need another call to get the full information of a
set of security groups at start/resync if we don't already
have any.




On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <

ayshihanzh...@126.com >

wrote:



hi Miguel Ángel, I am very agree with you about the
following point:

* physical implementation on the hosts (ipsets, nftables,
... )

--this can reduce the load of compute node.

* rpc communication mechanisms.

-- this can reduce the load of neutron server can you help
me to review my BP specs?







At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <

mangel...@redhat.com >

wrote:


Hi it's a very interesting topic, I was getting ready to
raise the same concerns about our security groups
implementation,

shihanzhang

thank you for starting this topic.

Not only at low level w

Re: [openstack-dev] [neutron]Performance of security group

[Ihar Hrachyshka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Oh, so you have the enhancement implemented? Great! Any numbers that
shows how much we gain from that?

/Ihar

On 03/07/14 02:49, shihanzhang wrote:
> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
> I will modify my spec, when the spec is approved, I will commit the
> codes as soon as possilbe!
> 
> 
> 
> 
> 
> At 2014-07-02 10:12:34, "Miguel Angel Ajo" 
> wrote:
>> 
>> Nice Shihanzhang,
>> 
>> Do you mean the ipset implementation is ready, or just the
>> spec?.
>> 
>> 
>> For the SG group refactor, I don't worry about who does it, or
>> who takes the credit, but I believe it's important we address
>> this bottleneck during Juno trying to match nova's scalability.
>> 
>> Best regards, Miguel Ángel.
>> 
>> 
>> On 07/02/2014 02:50 PM, shihanzhang wrote:
>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
>>> split  the work in several specs, I have finished the work (
>>> ipset optimization), you can do 'sg rpc optimization (without 
>>> fanout)'. as the third part(sg rpc optimization (with fanout)),
>>> I think we need talk about it, because just using ipset to
>>> optimize security group agent codes does not bring the best
>>> results!
>>> 
>>> Best regards, shihanzhang.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" 
>>> wrote:
> On 02/07/14 10:12, Miguel Angel Ajo wrote:
> 
>> Shihazhang,
> 
>> I really believe we need the RPC refactor done for this cycle,
>> and given the close deadlines we have (July 10 for spec
>> submission and July 20 for spec approval).
> 
>> Don't you think it's going to be better to split the work in 
>> several specs?
> 
>> 1) ipset optimization   (you) 2) sg rpc optimization (without 
>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
>> , me)
> 
> 
>> This way we increase the chances of having part of this for the 
>> Juno cycle. If we go for something too complicated is going to
>> take more time for approval.
> 
> 
> I agree. And it not only increases chances to get at least some of 
> those highly demanded performance enhancements to get into Juno,
> it's also "the right thing to do" (c). It's counterproductive to
> put multiple vaguely related enhancements in single spec. This
> would dim review focus and put us into position of getting
> 'all-or-nothing'. We can't afford that.
> 
> Let's leave one spec per enhancement. @Shihazhang, what do you
> think?
> 
> 
>> Also, I proposed the details of "2", trying to bring awareness
>> on the topic, as I have been working with the scale lab in Red
>> Hat to find and understand those issues, I have a very good
>> knowledge of the problem and I believe I could make a very fast
>> advance on the issue at the RPC level.
> 
>> Given that, I'd like to work on this specific part, whether or
>> not we split the specs, as it's something we believe critical
>> for neutron scalability and thus, *nova parity*.
> 
>> I will start a separate spec for "2", later on, if you find it
>> ok, we keep them as separate ones, if you believe having just 1
>> spec (for 1 & 2) is going be safer for juno-* approval, then we
>> can incorporate my spec in yours, but then
>> "add-ipset-to-security" is not a good spec title to put all this
>> together.
> 
> 
>> Best regards, Miguel Ángel.
> 
> 
>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>>> 
>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
>>> spes, but I will also optimization the RPC from security group
>>> agent to neutron server. Now the modle is
>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
>>> sg2..]', this can reduce the size of RPC respose message from
>>> neutron server to security group agent.
>>> 
>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo" 
>>>  wrote:
 
 
 Ok, I was talking with Édouard @ IRC, and as I have time to 
 work into this problem, I could file an specific spec for
 the security group RPC optimization, a masterplan in two
 steps:
 
 1) Refactor the current RPC communication for 
 security_groups_for_devices, which could be used for full 
 syncs, etc..
 
 2) Benchmark && make use of a fanout queue per security
 group to make sure only the hosts with instances on a
 certain security group get the updates as they happen.
 
 @shihanzhang do you find it reasonable?
 
 
 
 - Original Message -
> - Original Message -
>> @Nachi: Yes that could a good improvement to factorize
>> the RPC
> mechanism.
>> 
>> Another idea: What about creating a RPC topic per
>> security group (quid of the
> RPC topic
>> scalability) on which an agent subscribes if one of its 
>> ports is
> associated
>> to the security group?
>> 
>> Regards, Édouard.
>> 
>> 
> 
> 
> Hmm, Interesting,
> 
> @Nachi, I'm not sure I fully understood:
> 
> 
> SG_LIST [ SG1, SG2] SG_RULE_LIS

Re: [openstack-dev] [neutron]Performance of security group

[Miguel Angel Ajo Pelayo]

Wow Shihanzhang, truly awesome!,

  Is the current implementation for https://review.openstack.org/#/c/104522/ 
also ready?, could you make it available?.


Good work!,


- Original Message -
> 
> With the deployment 'nova + neutron + openvswitch', when we bulk create about
> 500 VM with a default security group, the CPU usage of neutron-server and
> openvswitch agent is very high, especially the CPU usage of openvswitch
> agent will be 100%, this will cause creating VMs failed.
> 
> With the method discussed in mailist:
> 1) ipset optimization   (https://review.openstack.org/#/c/100761/)
> 3) sg rpc optimization (with fanout)  (
> https://review.openstack.org/#/c/104522/)
> I have implement  these two scheme in my deployment, when we again bulk
> create about 500 VM with a default security group, the CPU usage of
>  openvswitch agent will reduce to 10%, e ven lower than 10%, so I think the i
> provement of these two options are very efficient.
> Who can help us to review our spec?
> Best regards,
> shihanzhang
> 
> 
> 
> 
> 
> At 2014-07-03 10:08:21, "Ihar Hrachyshka"  wrote:
> >-BEGIN PGP SIGNED MESSAGE-
> >Hash: SHA512
> >
> >Oh, so you have the enhancement implemented? Great! Any numbers that
> >shows how much we gain from that?
> >
> >/Ihar
> >
> >On 03/07/14 02:49, shihanzhang wrote:
> >> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
> >> I will modify my spec, when the spec is approved, I will commit the
> >> codes as soon as possilbe!
> >> 
> >> 
> >> 
> >> 
> >> 
> >> At 2014-07-02 10:12:34, "Miguel Angel Ajo" 
> >> wrote:
> >>> 
> >>> Nice Shihanzhang,
> >>> 
> >>> Do you mean the ipset implementation is ready, or just the
> >>> spec?.
> >>> 
> >>> 
> >>> For the SG group refactor, I don't worry about who does it, or
> >>> who takes the credit, but I believe it's important we address
> >>> this bottleneck during Juno trying to match nova's scalability.
> >>> 
> >>> Best regards, Miguel Ángel.
> >>> 
> >>> 
> >>> On 07/02/2014 02:50 PM, shihanzhang wrote:
>  hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
>  split  the work in several specs, I have finished the work (
>  ipset optimization), you can do 'sg rpc optimization (without
>  fanout)'. as the third part(sg rpc optimization (with fanout)),
>  I think we need talk about it, because just using ipset to
>  optimize security group agent codes does not bring the best
>  results!
>  
>  Best regards, shihanzhang.
>  
>  
>  
>  
>  
>  
>  
>  
>  At 2014-07-02 04:43:24, "Ihar Hrachyshka" 
>  wrote:
> >> On 02/07/14 10:12, Miguel Angel Ajo wrote:
> >> 
> >>> Shihazhang,
> >> 
> >>> I really believe we need the RPC refactor done for this cycle,
> >>> and given the close deadlines we have (July 10 for spec
> >>> submission and July 20 for spec approval).
> >> 
> >>> Don't you think it's going to be better to split the work in
> >>> several specs?
> >> 
> >>> 1) ipset optimization   (you) 2) sg rpc optimization (without
> >>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
> >>> , me)
> >> 
> >> 
> >>> This way we increase the chances of having part of this for the
> >>> Juno cycle. If we go for something too complicated is going to
> >>> take more time for approval.
> >> 
> >> 
> >> I agree. And it not only increases chances to get at least some of
> >> those highly demanded performance enhancements to get into Juno,
> >> it's also "the right thing to do" (c). It's counterproductive to
> >> put multiple vaguely related enhancements in single spec. This
> >> would dim review focus and put us into position of getting
> >> 'all-or-nothing'. We can't afford that.
> >> 
> >> Let's leave one spec per enhancement. @Shihazhang, what do you
> >> think?
> >> 
> >> 
> >>> Also, I proposed the details of "2", trying to bring awareness
> >>> on the topic, as I have been working with the scale lab in Red
> >>> Hat to find and understand those issues, I have a very good
> >>> knowledge of the problem and I believe I could make a very fast
> >>> advance on the issue at the RPC level.
> >> 
> >>> Given that, I'd like to work on this specific part, whether or
> >>> not we split the specs, as it's something we believe critical
> >>> for neutron scalability and thus, *nova parity*.
> >> 
> >>> I will start a separate spec for "2", later on, if you find it
> >>> ok, we keep them as separate ones, if you believe having just 1
> >>> spec (for 1 & 2) is going be safer for juno-* approval, then we
> >>> can incorporate my spec in yours, but then
> >>> "add-ipset-to-security" is not a good spec title to put all this
> >>> together.
> >> 
> >> 
> >>> Best regards, Miguel Ángel.
> >> 
> >> 
> >>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>  
>  hi Miguel Angel Ajo Pelayo! I agree with you and modify my
>  spes, but I will also optimization the RPC from security group
>  agent to neutron server. Now the modle is
>  '

Re: [openstack-dev] [neutron]Performance of security group

[Kyle Mestery]

On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang  wrote:
>
> With the deployment 'nova + neutron + openvswitch', when we bulk create
> about 500 VM with a default security group, the CPU usage of neutron-server
> and openvswitch agent is very high, especially the CPU usage of openvswitch
> agent will be 100%, this will cause creating VMs failed.
>
> With the method discussed in mailist:
>
> 1) ipset optimization   (https://review.openstack.org/#/c/100761/)
>
> 3) sg rpc optimization (with fanout)
> (https://review.openstack.org/#/c/104522/)
>
> I have implement  these two scheme in my deployment,  when we again bulk
> create about 500 VM with a default security group, the CPU usage of
> openvswitch agent will reduce to 10%, even lower than 10%, so I think the
> iprovement of these two options are very efficient.
>
> Who can help us to review our spec?
>
This is great work! These are on my list of things to review in detail
soon, but given the Neutron sprint this week, I haven't had time yet.
I'll try to remedy that by the weekend.

Thanks!
Kyle

>Best regards,
> shihanzhang
>
>
>
>
>
> At 2014-07-03 10:08:21, "Ihar Hrachyshka"  wrote:
>>-BEGIN PGP SIGNED MESSAGE-
>>Hash: SHA512
>>
>>Oh, so you have the enhancement implemented? Great! Any numbers that
>>shows how much we gain from that?
>>
>>/Ihar
>>
>>On 03/07/14 02:49, shihanzhang wrote:
>>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
>>> I will modify my spec, when the spec is approved, I will commit the
>>> codes as soon as possilbe!
>>>
>>>
>>>
>>>
>>>
>>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" 
>>> wrote:

 Nice Shihanzhang,

 Do you mean the ipset implementation is ready, or just the
 spec?.


 For the SG group refactor, I don't worry about who does it, or
 who takes the credit, but I believe it's important we address
 this bottleneck during Juno trying to match nova's scalability.

 Best regards, Miguel Ángel.


 On 07/02/2014 02:50 PM, shihanzhang wrote:
> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
> split  the work in several specs, I have finished the work (
> ipset optimization), you can do 'sg rpc optimization (without
> fanout)'. as the third part(sg rpc optimization (with fanout)),
> I think we need talk about it, because just using ipset to
> optimize security group agent codes does not bring the best
> results!
>
> Best regards, shihanzhang.
>
>
>
>
>
>
>
>
> At 2014-07-02 04:43:24, "Ihar Hrachyshka" 
> wrote:
>>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>>>
 Shihazhang,
>>>
 I really believe we need the RPC refactor done for this cycle,
 and given the close deadlines we have (July 10 for spec
 submission and July 20 for spec approval).
>>>
 Don't you think it's going to be better to split the work in
 several specs?
>>>
 1) ipset optimization   (you) 2) sg rpc optimization (without
 fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
 , me)
>>>
>>>
 This way we increase the chances of having part of this for the
 Juno cycle. If we go for something too complicated is going to
 take more time for approval.
>>>
>>>
>>> I agree. And it not only increases chances to get at least some of
>>> those highly demanded performance enhancements to get into Juno,
>>> it's also "the right thing to do" (c). It's counterproductive to
>>> put multiple vaguely related enhancements in single spec. This
>>> would dim review focus and put us into position of getting
>>> 'all-or-nothing'. We can't afford that.
>>>
>>> Let's leave one spec per enhancement. @Shihazhang, what do you
>>> think?
>>>
>>>
 Also, I proposed the details of "2", trying to bring awareness
 on the topic, as I have been working with the scale lab in Red
 Hat to find and understand those issues, I have a very good
 knowledge of the problem and I believe I could make a very fast
 advance on the issue at the RPC level.
>>>
 Given that, I'd like to work on this specific part, whether or
 not we split the specs, as it's something we believe critical
 for neutron scalability and thus, *nova parity*.
>>>
 I will start a separate spec for "2", later on, if you find it
 ok, we keep them as separate ones, if you believe having just 1
 spec (for 1 & 2) is going be safer for juno-* approval, then we
 can incorporate my spec in yours, but then
 "add-ipset-to-security" is not a good spec title to put all this
 together.
>>>
>>>
 Best regards, Miguel Ángel.
>>>
>>>
 On 07/02/2014 03:37 AM, shihanzhang wrote:
>
> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
> spes, but I will also optimization the RPC from security group
> agent to neutron server. Now the modle is
> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
> sg2..]', this can reduce the si