Re: [ovs-dev] Reproducing ovn-scale-test results

[Valentine Sinitsyn]

Hi Han,

On 17.03.2017 23:36, Han Zhou wrote:


On Fri, Mar 17, 2017 at 2:50 AM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>> wrote:

Did you restart controllers or ovn-northd after running full tests

before binding more ports. It would be interesting to learn how long
does it take warm up IDL in controllers/northd in your setup.



No need to restart. It will cool down when test ends. On top of the
scale, we can just test creating & binding 500 lports and then tear down
the 500 lports before next test, which takes much less time than the
full run from empty.



Moreover, in current ovs-scale-test code, the step "wait 100 lport up"
is updated utilizing a new feature (wait for HVs to catch up) that was


Are you referring to ovn-nbctl --wait-until, or something else? If you

were not using it for the talk, what exactly does "create + bind" mean
on the graph?



--wait-until has been used always, to wait lport state become "up" in
NB, which means port binding is reflected in NB. The new change [1] was
using the new feature of ovn-nbctl: "sync --wait=hv" which will wait
until the port binding to be processed on all HVs. This feature was not
there yet when we had the talk, and no reasonable alternative to achieve
the same.
Got you know. This may affect the test negatovely indeed, as OVSDB seems 
to scale poorly with the number of clients, as our tests suggest.


Thanks you again.

Valentine


[1]
https://github.com/openvswitch/ovn-scale-test/commit/0ece1038de45f05f461b45162b21a8bde2793010

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] Reproducing ovn-scale-test results

[Han Zhou]

On Fri, Mar 17, 2017 at 2:50 AM, Valentine Sinitsyn <
valentine.sinit...@gmail.com> wrote:
> Did you restart controllers or ovn-northd after running full tests before
binding more ports. It would be interesting to learn how long does it take
warm up IDL in controllers/northd in your setup.
>
No need to restart. It will cool down when test ends. On top of the scale,
we can just test creating & binding 500 lports and then tear down the 500
lports before next test, which takes much less time than the full run from
empty.

>>
>> Moreover, in current ovs-scale-test code, the step "wait 100 lport up"
>> is updated utilizing a new feature (wait for HVs to catch up) that was
>
> Are you referring to ovn-nbctl --wait-until, or something else? If you
were not using it for the talk, what exactly does "create + bind" mean on
the graph?
>
--wait-until has been used always, to wait lport state become "up" in NB,
which means port binding is reflected in NB. The new change [1] was using
the new feature of ovn-nbctl: "sync --wait=hv" which will wait until the
port binding to be processed on all HVs. This feature was not there yet
when we had the talk, and no reasonable alternative to achieve the same.

[1]
https://github.com/openvswitch/ovn-scale-test/commit/0ece1038de45f05f461b45162b21a8bde2793010
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] Reproducing ovn-scale-test results

[Valentine Sinitsyn]

Hi,

On 17.03.2017 02:24, Han Zhou wrote:



On Thu, Mar 16, 2017 at 1:06 PM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>> wrote:


Hi Han,

Thanks for the quick answer.

On 17.03.2017 00:34, Han Zhou wrote:


On Thu, Mar 16, 2017 at 3:58 AM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>

>> wrote:



Hi all,

We are doing some stress testing on OVN 2.7, and wanted to reproduce


results from the talk [1]. Looking at ovn-scale-test sources, I have two
questions:





Hi Valentine,

Thanks for picking this up.



- Do I get correctly that the benchmark always starts with the empty


northbound db. Then lswitches are added, then you add ports to each

lswitch?





Yes, the test result shown in the talk was started from empty to
gradually reach 20k lports on 200 lswitches.


- What is the batch size in port_create_args?



I remember it was 100. In addition, there were 5 jobs running in

parallel.

+Lei to confirm.


Could you recall how long (approximately) does it take to create and

bind 20K ports with these settings? This would be really helpful.




I don't have the raw data now, but it took around 1 - 2 hours.
We don't always run the full test, but after the full test is completed,
we can just run another task to create and bind 1k more lports to
evaluate the optimizations in each iteration on top of the existing scale.
Did you restart controllers or ovn-northd after running full tests 
before binding more ports. It would be interesting to learn how long 
does it take warm up IDL in controllers/northd in your setup.




One more thing, the graph shared also involved sandbox (simulated HV)
creation and lswitch creation. They were all created gradually during
the test run.
The flow was like:
1. create 50 sandboxes
2. (5 jobs in parallel) create 1 lswitch, create 100 lports, bind 100
lports, wait 100 lport up
3. if there are 100 sandboxes already on the BM, switch to another BM
4. goto step1, until it is done for all 20 BMs.

Moreover, in current ovs-scale-test code, the step "wait 100 lport up"
is updated utilizing a new feature (wait for HVs to catch up) that was
Are you referring to ovn-nbctl --wait-until, or something else? If you 
were not using it for the talk, what exactly does "create + bind" mean 
on the graph?


Many thanks again.

Valentine


added after the report, and we didn't run the test again yet with this
change. I would expect it impact the test result slightly negatively,
but it would more accurate.





In short: is it true that for the setup involving (say) 1 ports


spanned over 100 lswitches in the aforementioned test, a Rally task
would look like this?



{
"version": 2,
"title": "Create and bind port",
"subtasks": [{
"title": "Create and bind port",
"workloads": [{
"name": "OvnNetwork.create_and_bind_ports",
"args": {
"network_create_args": {
"amount": 100,
"batch": 1,
"start_cidr": "172.16.1.0/24

 ",

"physical_network": "providernet"
},
"port_create_args" : {"batch": 2},
"ports_per_network": 100,
"port_bind_args": {"wait_up": true}
},
"runner": {
"type": "serial","times": 1},
"context": {
   "ovn_multihost" : {
"controller": "ovn-controller-node"
},
"sandbox":{ "tag": "ToR1"}
}
}]
}]
}

1. https://youtu.be/okralc7LrZo?t=1185

Thanks,
Valentine





--
С уважением,
Синицын Валентин



___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] Reproducing ovn-scale-test results

[Han Zhou]

On Thu, Mar 16, 2017 at 1:06 PM, Valentine Sinitsyn <
valentine.sinit...@gmail.com> wrote:
>
> Hi Han,
>
> Thanks for the quick answer.
>
> On 17.03.2017 00:34, Han Zhou wrote:
>>
>> On Thu, Mar 16, 2017 at 3:58 AM, Valentine Sinitsyn
>> mailto:valentine.sinit...@gmail.com>>
wrote:
>>>
>>>
>>> Hi all,
>>>
>>> We are doing some stress testing on OVN 2.7, and wanted to reproduce
>>
>> results from the talk [1]. Looking at ovn-scale-test sources, I have two
>> questions:
>>>
>>>
>>
>> Hi Valentine,
>>
>> Thanks for picking this up.
>>
>>
>>> - Do I get correctly that the benchmark always starts with the empty
>>
>> northbound db. Then lswitches are added, then you add ports to each
lswitch?
>>>
>>>
>>
>> Yes, the test result shown in the talk was started from empty to
>> gradually reach 20k lports on 200 lswitches.
>>
>>> - What is the batch size in port_create_args?
>>
>>
>> I remember it was 100. In addition, there were 5 jobs running in
parallel.
>> +Lei to confirm.
>
> Could you recall how long (approximately) does it take to create and bind
20K ports with these settings? This would be really helpful.
>

I don't have the raw data now, but it took around 1 - 2 hours.
We don't always run the full test, but after the full test is completed, we
can just run another task to create and bind 1k more lports to evaluate the
optimizations in each iteration on top of the existing scale.

One more thing, the graph shared also involved sandbox (simulated HV)
creation and lswitch creation. They were all created gradually during the
test run.
The flow was like:
1. create 50 sandboxes
2. (5 jobs in parallel) create 1 lswitch, create 100 lports, bind 100
lports, wait 100 lport up
3. if there are 100 sandboxes already on the BM, switch to another BM
4. goto step1, until it is done for all 20 BMs.

Moreover, in current ovs-scale-test code, the step "wait 100 lport up" is
updated utilizing a new feature (wait for HVs to catch up) that was added
after the report, and we didn't run the test again yet with this change. I
would expect it impact the test result slightly negatively, but it would
more accurate.

>>
>>>
>>> In short: is it true that for the setup involving (say) 1 ports
>>
>> spanned over 100 lswitches in the aforementioned test, a Rally task
>> would look like this?
>>>
>>>
>>> {
>>> "version": 2,
>>> "title": "Create and bind port",
>>> "subtasks": [{
>>> "title": "Create and bind port",
>>> "workloads": [{
>>> "name": "OvnNetwork.create_and_bind_ports",
>>> "args": {
>>> "network_create_args": {
>>> "amount": 100,
>>> "batch": 1,
>>> "start_cidr": "172.16.1.0/24 ",
>>> "physical_network": "providernet"
>>> },
>>> "port_create_args" : {"batch": 2},
>>> "ports_per_network": 100,
>>> "port_bind_args": {"wait_up": true}
>>> },
>>> "runner": {
>>> "type": "serial","times": 1},
>>> "context": {
>>>"ovn_multihost" : {
>>> "controller": "ovn-controller-node"
>>> },
>>> "sandbox":{ "tag": "ToR1"}
>>> }
>>> }]
>>> }]
>>> }
>>>
>>> 1. https://youtu.be/okralc7LrZo?t=1185
>>>
>>> Thanks,
>>> Valentine
>>
>>
>
> --
> С уважением,
> Синицын Валентин
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] Reproducing ovn-scale-test results

[Valentine Sinitsyn]

Hi Han,

Thanks for the quick answer.

On 17.03.2017 00:34, Han Zhou wrote:

On Thu, Mar 16, 2017 at 3:58 AM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>> wrote:


Hi all,

We are doing some stress testing on OVN 2.7, and wanted to reproduce

results from the talk [1]. Looking at ovn-scale-test sources, I have two
questions:




Hi Valentine,

Thanks for picking this up.



- Do I get correctly that the benchmark always starts with the empty

northbound db. Then lswitches are added, then you add ports to each lswitch?




Yes, the test result shown in the talk was started from empty to
gradually reach 20k lports on 200 lswitches.


- What is the batch size in port_create_args?


I remember it was 100. In addition, there were 5 jobs running in parallel.
+Lei to confirm.
Could you recall how long (approximately) does it take to create and 
bind 20K ports with these settings? This would be really helpful.


Thanks,
Valentine





In short: is it true that for the setup involving (say) 1 ports

spanned over 100 lswitches in the aforementioned test, a Rally task
would look like this?


{
"version": 2,
"title": "Create and bind port",
"subtasks": [{
"title": "Create and bind port",
"workloads": [{
"name": "OvnNetwork.create_and_bind_ports",
"args": {
"network_create_args": {
"amount": 100,
"batch": 1,
"start_cidr": "172.16.1.0/24 ",
"physical_network": "providernet"
},
"port_create_args" : {"batch": 2},
"ports_per_network": 100,
"port_bind_args": {"wait_up": true}
},
"runner": {
"type": "serial","times": 1},
"context": {
   "ovn_multihost" : {
"controller": "ovn-controller-node"
},
"sandbox":{ "tag": "ToR1"}
}
}]
}]
}

1. https://youtu.be/okralc7LrZo?t=1185

Thanks,
Valentine




--
С уважением,
Синицын Валентин
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] Reproducing ovn-scale-test results

[Han Zhou]

On Thu, Mar 16, 2017 at 3:58 AM, Valentine Sinitsyn <
valentine.sinit...@gmail.com> wrote:
>
> Hi all,
>
> We are doing some stress testing on OVN 2.7, and wanted to reproduce
results from the talk [1]. Looking at ovn-scale-test sources, I have two
questions:
>

Hi Valentine,

Thanks for picking this up.


> - Do I get correctly that the benchmark always starts with the empty
northbound db. Then lswitches are added, then you add ports to each lswitch?
>

Yes, the test result shown in the talk was started from empty to gradually
reach 20k lports on 200 lswitches.

> - What is the batch size in port_create_args?

I remember it was 100. In addition, there were 5 jobs running in parallel.
+Lei to confirm.

>
> In short: is it true that for the setup involving (say) 1 ports
spanned over 100 lswitches in the aforementioned test, a Rally task would
look like this?
>
> {
> "version": 2,
> "title": "Create and bind port",
> "subtasks": [{
> "title": "Create and bind port",
> "workloads": [{
> "name": "OvnNetwork.create_and_bind_ports",
> "args": {
> "network_create_args": {
> "amount": 100,
> "batch": 1,
> "start_cidr": "172.16.1.0/24",
> "physical_network": "providernet"
> },
> "port_create_args" : {"batch": 2},
> "ports_per_network": 100,
> "port_bind_args": {"wait_up": true}
> },
> "runner": {
> "type": "serial","times": 1},
> "context": {
>"ovn_multihost" : {
> "controller": "ovn-controller-node"
> },
> "sandbox":{ "tag": "ToR1"}
> }
> }]
> }]
> }
>
> 1. https://youtu.be/okralc7LrZo?t=1185
>
> Thanks,
> Valentine
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] Reproducing ovn-scale-test results

[Valentine Sinitsyn]

Hi all,

We are doing some stress testing on OVN 2.7, and wanted to reproduce 
results from the talk [1]. Looking at ovn-scale-test sources, I have two 
questions:


- Do I get correctly that the benchmark always starts with the empty 
northbound db. Then lswitches are added, then you add ports to each lswitch?


- What is the batch size in port_create_args?

In short: is it true that for the setup involving (say) 1 ports 
spanned over 100 lswitches in the aforementioned test, a Rally task 
would look like this?


{
"version": 2,
"title": "Create and bind port",
"subtasks": [{
"title": "Create and bind port",
"workloads": [{
"name": "OvnNetwork.create_and_bind_ports",
"args": {
"network_create_args": {
"amount": 100,
"batch": 1,
"start_cidr": "172.16.1.0/24",
"physical_network": "providernet"
},
"port_create_args" : {"batch": 2},
"ports_per_network": 100,
"port_bind_args": {"wait_up": true}
},
"runner": {
"type": "serial","times": 1},
"context": {
   "ovn_multihost" : {
"controller": "ovn-controller-node"
},
"sandbox":{ "tag": "ToR1"}
}
}]
}]
}

1. https://youtu.be/okralc7LrZo?t=1185

Thanks,
Valentine
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev