Re: [opnfv-tech-discuss] #nfvbench - Looking for line rate performances using NFVbench - CONTINUED

[Pierrick Louin via Lists.Opnfv.Org]

Thanks for your message that I cannot study right now, sorry!

I am going on holidays today and be back on August the 26th.

Just small answers:


-   It seems that when defined (!=0) the size of the cache silently limits 
the actual number of possible flows.

As soon as the cache size it not zero the transmission goes faster.



-   I noticed that starting the T-Rex into my host lead to a ~10 sec for 
initializing before the bound/configured message is displayed.

No such delay when launched within the NFVbench container

Maybe I wrongly shared the OFED binary needed for initializing the mellanox NIC.

I will look into T-Rex logs and other stuff when I come back.



See you later,
Pierrick



[cid:image001.gif@01CF4791.BEC5FC70]

orange labs

ORANGE/IMT/OLN/CISS/IIE/AMI


Pierrick LOUIN


4, rue du Clos Courtel  -  BP 91226

35512  CESSON SEVIGNÉ  cédex
tél  : +33 2 99 12 48 23
mob: +33 6 43 33 01 04
e-mail : pierrick.lo...@orange.com


De : opnfv-tech-discuss@lists.opnfv.org 
[mailto:opnfv-tech-discuss@lists.opnfv.org] De la part de Michael S. Pedersen
Envoyé : vendredi 2 août 2019 15:04
À : opnfv-tech-discuss@lists.opnfv.org
Objet : Re: [opnfv-tech-discuss] #nfvbench - Looking for line rate performances 
using NFVbench - CONTINUED

Hi Pierrick, Alec,

These findings are quite interesting indeed. I have only been using 
Connectx3-4, and always in a setting where line-rates like this hasn't been 
possible and/or relevant.

The increased performance after starting and stopping TRex is strange. I wonder 
if it could be something with the memory allocation on the initial startup?

The cache increase leading to higher performance makes more sense. I wonder if 
there is any correlation between number of flows and the cache size? If that is 
the case then it would make sense to use the value for number of flows as the 
cache size too (instead of having the additional input argument).

BR,
Michael

_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#23427): 
https://lists.opnfv.org/g/opnfv-tech-discuss/message/23427
Mute This Topic: https://lists.opnfv.org/mt/32688323/21656
Mute #nfvbench: https://lists.opnfv.org/mk?hashtag=nfvbench&subid=2783016
Group Owner: opnfv-tech-discuss+ow...@lists.opnfv.org
Unsubscribe: https://lists.opnfv.org/g/opnfv-tech-discuss/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [opnfv-tech-discuss] [opnfv-tsc] Can installers use CircleCI?

[Manuel Buil]

Hello Trevor,

Cool! Thanks.

If you want we work together to try to understand why it fails for XCI. That 
way we will probably find gaps.

Note that you are doing a "virtual" deployment, where controller and computes 
are in VMs. When you were reading the documentation, did you find a way to 
deploy baremetal?

Thanks,
Manuel


From: opnfv-...@lists.opnfv.org  on behalf of Trevor 
Bramwell 
Sent: Tuesday, July 30, 2019 12:10 AM
To: Manuel Buil 
Cc: ahot...@cisco.com ; opnfv-tech-discuss@lists.opnfv.org 
; opnfv-...@lists.opnfv.org 

Subject: Re: [opnfv-tsc] Can installers use CircleCI?

Hi Manuel, Alec, et al.

I finished up a guide[1] for setting up repos on CircleCI, Gitlab-CI,
and Azure Pipelines for testing these proof-of-concepts (PoCs).

Hopefully this will help anyone who has the cycles to dig into these
platforms and find if they'll meet our needs.

It was pretty easy to get a machine from LaaS connected up the Gitlab-CI
and attempt to run XCI[2] (though I've yet to successfully deploy it),
and I don't think I'll have any issues trying to connect it to Azure
Pipelines. From what I know of CircleCI it will take a bit more work
though as it can only SSH out, and that would require first setting up
the VPN connection.

Regards,
Trevor Bramwell

[1] https://wiki.opnfv.org/display/INF/PoC+Setup
[2] https://gitlab.com/bramweltci/releng-xci/pipelines

On Tue, Jul 09, 2019 at 04:49:42PM +, Manuel Buil wrote:
> Thanks for sharing the details Alec. It sounds like an interesting PoC and 
> will give us a lot of insights 🙂.
>
> I also think those baremetal features will be hard to get but that needs to 
> be investigated. By your information, I am also realizing that multi-distro 
> is not supported and we are tight to the images they offer, which are not 
> that many, just ubuntu-1604. For example, by looking at Airship's CI, they 
> use ubuntu-1804 for OpenStack Stein or later, so we would not be able to 
> deploy it in CircleCI. Not sure how much influence we could have over 
> CircleCI to get multi-distro support 😉.
>
>
> Regards,
> Manuel
> 
> From: opnfv-...@lists.opnfv.org  on behalf of Alec 
> via Lists.Opnfv.Org 
> Sent: Tuesday, July 9, 2019 5:53 PM
> To: Manuel Buil
> Cc: opnfv-...@lists.opnfv.org
> Subject: Re: [opnfv-tsc] Can installers use CircleCI?
>
>
> Hi Manuel,
>
>
>
> I doubt circleci can do any of the features you describe below other than 
> perhaps nested virtualization (VM in VM).
>
> Circle ci is great to build software and do unit testing of it, what you need 
> for the below is a bare metal cloud such as packet.net or OPNFV LaaS.
>
> You can chose between a few flavors of VMs or docker containers to run your 
> workload (https://circleci.com/docs/2.0/configuration-reference/#machine)
>
> I don’t see how they can provide anything closer to bare metal.
>
>
>
> I am planning to test circleci to do the following with nfvbench:
>
>   *   Build VM images and push them to a VM image repo
>   *   Build docker containers and push them to docker hub
>
> Unit testing that does not require any HW dependencies
>
> Nothing really extraordinary…
>
> My project is a good example of tool that is highly dependent on NIC hardware 
> and kernel settings. If I can’t control those by API I’m pretty much limited 
> to SW unit testing.
>
>
>
> The only way to test an installer is to run it on a set of “friendly” bare 
> metal servers where you can
>
>   *   Select the NIC to use (or be sure you’re landing on a server that has 
> proper NIC)
>   *   control by API the bare metal SW setup (linux boot)
>   *   control by API the switch where your server is wired
>
>
>
> The devil is in the detail especially when it comes to mapping openstack to 
> the underlying networking layer.
>
> You can get away with nested virtualization but that is hardly comparable to 
> the real installation process in production 😉
>
> The level of details required for production deployers of openstack is 
> excruciatingly difficult.
>
>
>
> HTH
>
>
>
>   Alec
>
>
>
>
>
> From:  on behalf of Manuel Buil 
> Date: Tuesday, July 9, 2019 at 8:09 AM
> To: TSC OPNFV 
> Subject: [opnfv-tsc] Can installers use CircleCI?
>
>
>
> Hey guys,
>
>
>
> Unfortunately, we ran out of time so I could not ask. I think we all agree by 
> saying that installers are key projects in OPNFV and they are the biggest 
> consumers of our current jenkins CI, so we should probably try one of those 
> in the PoC. In fact, the usual way of deploying a scenario is through an 
> installer, right? So most projects depend on them.
>
>
>
> There are some installer requirements that I am not sure whether CircleCI 
> supports:
>
>
>
> 1 - Access to hosts that support IOMMU virtualization
>
> 2 - Access to hosts that have NICs that support DPDK
>
> 3 - Access to hosts with NICs that support SR-IOV
>
> 4 - Access to hosts with CPUs that support NUMA
>
> 5 - Support of multiple distros (laas now supports Cen

Re: [opnfv-tech-discuss] #nfvbench - Looking for line rate performances using NFVbench - CONTINUED

[Michael S. Pedersen]

Hi Pierrick, Alec,

These findings are quite interesting indeed. I have only been using 
Connectx3-4, and always in a setting where line-rates like this hasn't been 
possible and/or relevant.

The increased performance after starting and stopping TRex is strange. I wonder 
if it could be something with the memory allocation on the initial startup?

The cache increase leading to higher performance makes more sense. I wonder if 
there is any correlation between number of flows and the cache size? If that is 
the case then it would make sense to use the value for number of flows as the 
cache size too (instead of having the additional input argument).

BR,
Michael
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#23425): 
https://lists.opnfv.org/g/opnfv-tech-discuss/message/23425
Mute This Topic: https://lists.opnfv.org/mt/32688323/21656
Mute #nfvbench: https://lists.opnfv.org/mk?hashtag=nfvbench&subid=2783016
Group Owner: opnfv-tech-discuss+ow...@lists.opnfv.org
Unsubscribe: https://lists.opnfv.org/g/opnfv-tech-discuss/unsub  
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[opnfv-tech-discuss] #nfvbench - Looking for line rate performances using NFVbench - CONTINUED

[Pierrick Louin via Lists.Opnfv.Org]

Hi Alec,
Thanks for your response

Here are my answers to your questions/remarks.
I didn’t insert them inline but extracted the Q & A to keep a clear reading

Good bye,
Pierrick

[alec]
The reference NIC that we use is Intel X710, XL710 and XXV710 (10G, 25G and 
40G) and we have not seen any particular issues with rerunning benchmarks over 
and over again using the same TRex instance.
Intel X520 is clearly not a preferred NIC because it misses lots of features 
only available in newer generation NIC (offload).
I may be able to get a hold on a Mellanox 25G NIC soon and this report is 
definitely a good step to optimize on that NIC.

[pierrick]
OK, the X520 NIC is not our reference card neither... but I had no X710 card 
installed at the moment.
I made some reference tests to look at possible software common mode issues 
(i.e. maybe NIC independent).

As regards the Mellanox Connect 5 card,our present challenge was rather to get 
a high rate generator using only one dual-port NIC, if possible.
Actually we couldn't succeed in reaching the same performances with a XXV710, 
even using the T-Rex sever alone.
(With 64 bytes packets, bi-directional: Connect 5 <=> 2 x 25 GBits/s | XXV710 
<=> ~ 2 x 12 Gbit/s with T-Rex (~ 2 x 9.5 GBits/s with NFVbench 3.5.0).
   *
*   *
[alec]
Reboot of the server that runs nfvbench?

[pierrick]
Yes! this is the only way I found to come back to the "bad" initial condition 
<=> when NFVbench shows poorer performances.
   *
*   *
[alec]
Are you saying that once you launch and restart TRex, all future NFVbench runs 
(with the second instance of Trex) work better? Please explain this in detail 
(eg describe the exact steps in sequence that you performed).
Eg.
Reboot server
Start nfvbench for first time (will launch TRex for the first time) = poor 
performance (quantify)
Restart TRex
Run same benchmark = good performance (quantify)
[alec]
Can you provide some numbers with/without cache?
I have not tested the cache size option for STLScVmRaw but this is definitely 
something worth trying on Intel NIC X710 family.

[pierrick]
Here are reported our test cases and results using a dual-port NIC Mellanox 
Connect 5 - with or without a prior unique T-Rex server run/stop sequence


1)  - Fresh reboot

- Perform four series of NFVbench tests


2)  -  Fresh reboot

- Start a T-Rex server:  standalone launch,  same release as embedded in the 
container (v2.59) but in the host
/bin/bash ./t-rex-64 --no-scapy-server --iom 0 --cfg /etc/LLL.yaml -i -c 6
./_t-rex-64 --no-scapy-server --iom 0 --cfg /etc/LLL.yaml -i -c 6 --mlx4-so 
--mlx5-so

- Stop the T-Rex server

- Perform four series of NFVbench tests

NFVbench tests are the following
 x.1 Cmdline: nfvbench -c LLL.cfg --rate=100% --duration=30 --interval=1 
--flow-count=1 -fs=64 -scc=1 --cores=6 --extra-stats --cache-size=0
 x.2 Cmdline: nfvbench -c LLL.cfg --rate=100% --duration=30 --interval=1 
--flow-count=1 -fs=64 -scc=1 --cores=6 --extra-stats --cache-size=1
 x.3 Cmdline: nfvbench -c LLL.cfg --rate=100% --duration=30 --interval=1 
--flow-count=1 -fs=64 -scc=1 --cores=6 --cache-size=0
 x.4 Cmdline: nfvbench -c LLL.cfg --rate=100% --duration=30 --interval=1 
--flow-count=1 -fs=64 -scc=1 --cores=6 --cache-size=1

cases 1.1 and 2.1 correspond to the default NFVbench behaviour v3.5.0

+--+---++---++-+---++-+-+
| case | cache | flow stats |  Requested TX Rate (bps)  |  Actual TX Rate (bps) 
 |  RX Rate (bps)  |  Requested TX Rate (pps)  |  Actual TX Rate (pps)  |  RX 
Rate (pps)  |   variability   |
+==+===++===++=+===++=+=+
|  1.1 |   0   |all |   50. Gbps|  11.4679 Gbps 
 |  11.4679 Gbps   |  74,404,760 pps   | 17,065,302 pps | 
17,065,302 pps  | +/- 0.5 Gbits/s |
+--+---++---++-+---++-+-+
|  1.2 | 1 |all |   50. Gbps|  10.3619 Gbps 
 |  10.3619 Gbps   |  74,404,760 pps   | 15,419,544 pps | 
15,419,544 pps  | +/- 0.5 Gbits/s |
+--+---++---++-+---++-+-+
|  1.3 |   0   |  latency   |   50. Gbps|  24.5055 Gbps 
 |  24.5055 Gbps   |  74,404,760 pps   | 36,466,471 pps | 
36,466,471 pps  | +/-0.01 Gbits/s |
+--+---++---++-+-