On Fri, Feb 16, 2018 at 12:12 PM, Daniel Alvarez Sanchez <
dalva...@redhat.com> wrote:
> I've found out more about what is running slow in this scenario.
> I've profiled the processing of the update2 messages and here you can
> see the sequence of calls to __process_update2 (idl.py) when I'm
> creating a new port via OpenStack on a system loaded with 800 ports
> on the same Logical Switch:
>
>
> 1. Logical_Switch_Port 'insert'
> 2. Address_Set 'modify' (to add the new address, it takes ~ takes around
> 0.2 seconds
> 3. Logical_Switch_Port 'modify' (to add the new 8 ACLs) <- This takes > 2
> seconds
>
Sorry this is ^ Logical_Switch table.
> 4. ACL 'insert' x8 (one per ACL)
> 5. Logical_Switch_Port 'modify' ('up' = False)
> 6. Logical_Switch_Port 'insert' (this is exactly the same as 1, so it'll
> be skipped)
> 7. Address_Set 'modify' (this is exactly the same as 2, so it'll be
> skipped) still takes ~0.05-0.01 s
> 8. Logical_Switch 'modify' (to add the new 8 ACLs, same as 3) still takes
> ~0.5 seconds
>
This too ^
> 9. ACL 'insert' x8 (one per ACL, same as 4)
> 10. Logical_Switch_Port 'modify' ('up' = False) same as 5
> 11. Port_Binding (SB) 'insert'
> 12. Port_Binding (SB) 'insert' (same as 11)
>
>
> Half of those are dups and even they are noop, they consume times.
> The most expensive operation is adding the new 8 ACLs to the acls set in
> LS table.
> (800 ports with 8 ACLs each makes that set to have 6400 elements).
>
> NOTE: As you can see, we're trying to insert the address again into
> Address_Sets so we should
> bear this in mind if we go ahead with Lucas' suggestion about allowing
> dups here.
>
> It's obvious that we'll gain a lot by using Ports Sets for ACLs but, we'll
> also need
> to invest some time in finding out why we're getting dups and also trying
> to optimize
> the process_update2 method and its callees to make it faster. With those
> last
> two things I guess we can improve the performance a lot.
> Also, creating a python C binding for this module could also help but that
> seems like
> a lot of work and still we would need to convert to/from C structures to
> Python
> objects. However, inserting or identifying dups on large sets would be way
> faster.
>
> I'm going to try out Ben's suggestions for optimizing the process_update*
> methods,
> and will also try to dig further about the dups. As process_update* seems
> a bit
> expensive, looks to me that calling it 26 times for a single port is a lot.
> 26 calls = 2*(1 (LS insert) + 1 (AS modify) + 1(LSP modify) + 8 (ACL
> insert) + 1(LSP modify) + 1(PB insert))
>
26 calls = 2*(1 (LS insert) + 1 (AS modify) + 1(LS modify) + 8 (ACL
insert) + 1(LSP modify) + 1(PB insert))
> Thoughts?
> Thanks!
> Daniel
>
>
> On Thu, Feb 15, 2018 at 10:56 PM, Daniel Alvarez Sanchez <
> dalva...@redhat.com> wrote:
>
>>
>>
>> On Wed, Feb 14, 2018 at 9:34 PM, Han Zhou <zhou...@gmail.com> wrote:
>>
>>>
>>>
>>> On Wed, Feb 14, 2018 at 9:45 AM, Ben Pfaff <b...@ovn.org> wrote:
>>> >
>>> > On Wed, Feb 14, 2018 at 11:27:11AM +0100, Daniel Alvarez Sanchez wrote:
>>> > > Thanks for your inputs. I need to look more carefully into the patch
>>> you
>>> > > submitted but it looks like, at least, we'll be reducing the number
>>> of
>>> > > calls to Datum.__cmp__ which should be good.
>>> >
>>> > Thanks. Please do take a look. It's a micro-optimization but maybe
>>> > it'll help?
>>> >
>>> > > I probably didn't explain it very well. Right now we have N processes
>>> > > for Neutron server (in every node). Each of those opens a connection
>>> > > to NB db and they subscribe to updates from certain tables. Each time
>>> > > a change happens, ovsdb-server will send N update2 messages that has
>>> > > to be processed in this "expensive" way by each of those N
>>> > > processes. My proposal (yet to be refined) would be to now open N+1
>>> > > connections to ovsdb-server and only subscribe to notifications from
>>> 1
>>> > > of those. So every time a new change happens, ovsdb-server will send
>>> 1
>>> > > update2 message. This message will be processed (using Py IDL as we
>>> do
>>> > > now) and once processed, send it (mcast maybe?) to the rest N
>>> > > processes. This msg could be simply a Python object serialized and
>>> > > we'd be saving all this Datum, Atom, etc. processing by doing it just
>>> > > once.
>>> >
>>> Daniel, I understand that the update2 messages sending would consume NB
>>> ovsdb-server CPU and processing those update would consume neutron server
>>> process CPU. However, are we sure it is the bottleneck for port creation?
>>>
>>> From ovsdb-server point of view, sending updates to tens of clients
>>> should not be the bottleneck, considering that we have a lot more clients
>>> on HVs for SB ovsdb-server.
>>>
>>> From clients point of view, I think it is more of memory overhead than
>>> CPU, and it also depends on how many neutron processes are running on the
>>> same node. I didn't find neutron process CPU in your charts. I am hesitate
>>> for such big change before we are clear about the bottleneck. The chart of
>>> port creation time is very nice, but do we know which part of code
>>> contributed to the linear growth? Do we have profiling for the time spent
>>> in ovn_client.add_acls()?
>>>
>>
>> Here we are [0]. We see some spikes which are larger as the amount of
>> ports increases
>> but looks like the actual bottleneck is going to be when we're actually
>> commiting the
>> transaction [1]. I'll dig further though.
>>
>> [0 https://imgur.com/a/TmwbC
>> [1] https://github.com/openvswitch/ovs/blob/master/python/ovs/
>> db/idl.py#L1158
>>
>>>
>>> > OK. It's an optimization that does the work in one place rather than N
>>> > places, so definitely a win from a CPU cost point of view, but it
>>> trades
>>> > performance for increased complexity. It sounds like performance is
>>> > really important so maybe the increased complexity is a fair trade.
>>> >
>>> > We might also be able to improve performance by using native code for
>>> > some of the work. Were these tests done with the native code JSON
>>> > parser that comes with OVS? It is dramatically faster than the Python
>>> > code.
>>> >
>>> > > On Tue, Feb 13, 2018 at 8:32 PM, Ben Pfaff <b...@ovn.org> wrote:
>>> > >
>>> > > > Can you sketch the rows that are being inserted or modified when a
>>> port
>>> > > > is added? I would expect something like this as a minimum:
>>> > > >
>>> > > > * Insert one Logical_Switch_Port row.
>>> > > >
>>> > > > * Add pointer to Logical_Switch_Port to ports column in
>>> one row
>>> > > > in Logical_Switch.
>>> > > >
>>> > > > In addition it sounds like currently we're seeing:
>>> > > >
>>> > > > * Add one ACL row per security group rule.
>>> > > >
>>> > > > * Add pointers to ACL rows to acls column in one row in
>>> > > > Logical_Switch.
>>> > > >
>>> > > This is what happens when we create a port in OpenStack (without
>>> > > binding it) which belongs to a SG which allows ICMP and SSH traffic
>>> > > and drops the rest [0]
>>> > >
>>> > > Basically, you were right and only thing missing was adding the new
>>> > > address to the Address_Set table.
>>> >
>>> > OK.
>>> >
>>> > It sounds like the real scaling problem here is that for R security
>>> > group rules and P ports, we have R*P rows in the ACL table. Is that
>>> > correct? Should we aim to solve that problem?
>>>
>>> I think this might be the most valuable point to optimize for the
>>> create_port scenario from Neutron.
>>> I remember there was a patch for ACL group in OVN, so that instead of
>>> R*P rows we will have only R + P rows, but didn't see it went through.
>>> Is this also a good use case of conjuncture?
>>>
>>> > _______________________________________________
>>> > discuss mailing list
>>> > disc...@openvswitch.org
>>> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>>
>>>
>>
>
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
