Re: [opnfv-tech-discuss] New project proposal: NFVbench

2017-04-17 Thread Frank Brockners (fbrockne) 
Hi Trevor,

fully agreed. Let's discuss this in the proposal review during the technical 
meeting on Thursday.
Are you coming to the hackfest next week?

Thanks, Frank

From: Cooper, Trevor [mailto:trevor.coo...@intel.com]
Sent: Montag, 17. April 2017 08:22
To: Frank Brockners (fbrockne) ; TSC OPNFV 
; TECH-DISCUSS OPNFV 

Cc: Carsten Rossenhoevel ; Alec Hothan (ahothan) 

Subject: RE: New project proposal: NFVbench

Hi Frank

I think that your proposal aligns with some of the stress testing concepts that 
we have been developing in the Testing Working Group. Re. While VSPERF does not 
utilize the full stack (with Open Stack), the test cases can provide a baseline 
for any NFVI data-plane suite ... i.e. Throughput Tests, Packet and Frame Delay 
Tests, Stream Performance Tests, Request/Response Performance Tests, Packet 
Delay Tests, Scalability Tests, Control Path and Datapath Coupling Tests, CPU 
and Memory Consumption Tests, Time to Establish Flows Tests, Noisy Neighbor 
Tests. We should try to maintain consistency of methods/metrics form component 
level tests to system level where possible. It will be good to discuss this 
further.

/Trevor

From: 
opnfv-tech-discuss-boun...@lists.opnfv.org
 [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Frank 
Brockners (fbrockne)
Sent: Wednesday, April 12, 2017 9:40 AM
To: TSC OPNFV >; 
TECH-DISCUSS OPNFV 
>
Cc: Carsten Rossenhoevel >; Alec Hothan 
(ahothan) >
Subject: [opnfv-tech-discuss] New project proposal: NFVbench

Hi OPNFV,

over the past few weeks we've distilled a proposals to create a toolkit to 
allow for black-box performance testing of NFVI with a network focus:
NFVbench: https://wiki.opnfv.org/display/nfvbench/NFVbench+Project+Proposal

The NFVbench project is to develop a toolkit that allows developers, 
integrators, testers and customers to measure and assess the L2/L3 forwarding 
performance of an NFV-infrastructure solution stack (i.e. OPNFV scenario) using 
a black-box approach.

We're hoping for a discussion in the technical community meeting on April/20, 
and are also asking for an official TSC review post the technical community 
review on May/2, so that NFVbench can participate in Euphrates. Consequently, 
NFVbench asks for tentative inclusion into Euphrates.

Your thoughts and ideas are greatly appreciated.

Thanks much, Frank, Carsten, Alec


___
opnfv-tech-discuss mailing list
opnfv-tech-discuss@lists.opnfv.org
https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss

Re: [opnfv-tech-discuss] New project proposal: NFVbench

2017-04-17 Thread Cooper, Trevor 
Hi Frank

I think that your proposal aligns with some of the stress testing concepts that 
we have been developing in the Testing Working Group. Re. While VSPERF does not 
utilize the full stack (with Open Stack), the test cases can provide a baseline 
for any NFVI data-plane suite ... i.e. Throughput Tests, Packet and Frame Delay 
Tests, Stream Performance Tests, Request/Response Performance Tests, Packet 
Delay Tests, Scalability Tests, Control Path and Datapath Coupling Tests, CPU 
and Memory Consumption Tests, Time to Establish Flows Tests, Noisy Neighbor 
Tests. We should try to maintain consistency of methods/metrics form component 
level tests to system level where possible. It will be good to discuss this 
further.

/Trevor

From: opnfv-tech-discuss-boun...@lists.opnfv.org 
[mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Frank 
Brockners (fbrockne)
Sent: Wednesday, April 12, 2017 9:40 AM
To: TSC OPNFV ; TECH-DISCUSS OPNFV 

Cc: Carsten Rossenhoevel ; Alec Hothan (ahothan) 

Subject: [opnfv-tech-discuss] New project proposal: NFVbench

Hi OPNFV,

over the past few weeks we've distilled a proposals to create a toolkit to 
allow for black-box performance testing of NFVI with a network focus:
NFVbench: https://wiki.opnfv.org/display/nfvbench/NFVbench+Project+Proposal

The NFVbench project is to develop a toolkit that allows developers, 
integrators, testers and customers to measure and assess the L2/L3 forwarding 
performance of an NFV-infrastructure solution stack (i.e. OPNFV scenario) using 
a black-box approach.

We're hoping for a discussion in the technical community meeting on April/20, 
and are also asking for an official TSC review post the technical community 
review on May/2, so that NFVbench can participate in Euphrates. Consequently, 
NFVbench asks for tentative inclusion into Euphrates.

Your thoughts and ideas are greatly appreciated.

Thanks much, Frank, Carsten, Alec


___
opnfv-tech-discuss mailing list
opnfv-tech-discuss@lists.opnfv.org
https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss

Re: [opnfv-tech-discuss] New project proposal: NFVbench

2017-04-14 Thread Alec Hothan (ahothan) 
Hi Yujun,

Inline…


From: "Yujun Zhang (ZTE)" <zhangyujun+...@gmail.com>
Date: Wednesday, April 12, 2017 at 5:56 PM
To: "Frank Brockners (fbrockne)" <fbroc...@cisco.com>, TSC OPNFV 
<opnfv-...@lists.opnfv.org>, TECH-DISCUSS OPNFV 
<opnfv-tech-discuss@lists.opnfv.org>, test-wg <test...@lists.opnfv.org>
Cc: Carsten Rossenhoevel <cr...@eantc.de>, "Alec Hothan (ahothan)" 
<ahot...@cisco.com>
Subject: Re: [opnfv-tech-discuss] New project proposal: NFVbench

Some question after reviewing the proposal.
· Technically, is data plane / L2/L3 forwarding the only performance 
indicator for NFV?
Generally speaking, L2 forwarding is the main performance indicator for an NFVi 
data plane: measure how many frame/sec can the data plane handle under a well 
defined set of constraints. The set of constraints is actually a very important 
part of the picture because it can cloud the picture if not defined properly. 
What NFVbench brings is an automation of the well defined set of constraints so 
that these benchmarks can be repeated by more than 1 party and so that 2 
different implementations can be compared more easily.
The description mentions L2/L3 forwarding performance, the L3 part is simply 
because all frames are fully formed IPv4 L3 frames (even if the L3 header is 
not usually used by the NFVi data plane) and as such - this will also allow 
simple L3 forwarding benchmarking of any service chain. There is no plan to 
cover full blown VNF performance benchmarking in this project.
To answer your question to the letter, L2/L3 forwarding performance is not the 
only performance indicator for NFV (but is an important one).
· Will NFVbench project expand to other performance metrics besides the 
main focus?
The simple answer is no. Many open source projects suffer from 
over-diversification or over-scoping (want to do too many things). Measuring 
the NFVi data plane properly is already a pretty difficult task that requires 
experience, focus and discipline and this already represents a lot more work to 
cover than we have resources.
· In what way will NFVbench plan to complement and leverage related 
projects?
I will reply to the positioning with other OPNFV projects in a separate email.
· What will be the main components of the toolkit? A test runner? Test 
cases? Drivers?
The toolkit will be a compact specialized test runner that can run standalone 
but can also be easily integrated into a more generic test harness (such as 
Jenkins, yardstic or qtip) using well defined APIs/interfaces. Regarding test 
cases, we have decided to follow a different approach with the use of pre-built 
packet paths. Many benchmarking tools provide the ability for the user to fully 
customize the benchmark parameters (as in writing python code). With NFVbench 
we want to limit that ability to developers of the tool itself for the 
following reasons:

· The crafting of network benchmarking conditions is best done by 
experts in networking, these are people who are the best positioned to know 
what makes sense to test and in what conditions

· having too many people or worst - users of the tool - modify the 
conditions of the benchmark will have a huge negative impact on the usability 
of the results generated (in that – it will be a lot more difficult to compare 
them since the conditions have been modified)

· and finally, the vast majority of targeted users of NFVbench will be 
non dataplane experts that have no intent to modify the code or to add more 
variations to the benchmarks than necessary
So, NFVbench will define what the benchmark is and how it is conducted, it will 
be open source so everyone can discuss publicly on benchmarking conditions and 
see how the benchmark is conducted and automated. One important note is that 
there is no such a thing as a perfect benchmark tool, but it is possible to get 
one that is good enough for the community.
Today, it is impossible to compare 2 full stack data plane benchmarks because 
the conditions they are conducted are generally very different and often too 
subtle for many people to make sense of. These benchmarks are also often very 
difficult to reproduce (not mentioning by third party).This is what NFVbench is 
planning to address.
Thanks
   Alec


The following text are quoted for quick reference

The NFVbench project develops a toolkit that allows developers, integrators, 
testers and customers to measure and assess the L2/L3 forwarding performance of 
an NFV-infrastructure solution stack (i.e. OPNFV scenario) using a black-box 
approach.
...

The main focus of NFVbench is the NFVI full stack data plane benchmarking using 
realistic production deployment conditions.

NFVbench does not focus on the following areas and will align with, complement 
and leverage projects that already cover properly these areas:
...

On Thu, Apr 13, 2017 at 12:40 AM Frank Brockne

Re: [opnfv-tech-discuss] New project proposal: NFVbench

2017-04-12 Thread Yujun Zhang (ZTE) 
Some question after reviewing the proposal.

   - Technically, is data plane / L2/L3 forwarding the *only* performance
   indicator for NFV?
   - Will NFVbench project expand to other performance metrics besides the
   main focus?
   - In what way will NFVbench plan to complement and leverage related
   projects?
   - What will be the main components of the toolkit? A test runner? Test
   cases? Drivers?

The following text are quoted for quick reference

The NFVbench project develops a *toolkit* that allows developers,
integrators, testers and customers to measure and assess the *L2/L3
forwarding performance* of an NFV-infrastructure solution stack (i.e. OPNFV
scenario) using a black-box approach.
...

The main focus of NFVbench is the NFVI full stack *data plane benchmarking*
using realistic production deployment conditions.

NFVbench does *not* focus on the following areas and will align with,
*complement
and leverage* projects that already cover properly these areas:
...

On Thu, Apr 13, 2017 at 12:40 AM Frank Brockners (fbrockne) <
fbroc...@cisco.com> wrote:

> Hi OPNFV,
>
>
>
> over the past few weeks we’ve distilled a proposals to create a toolkit to
> allow for black-box performance testing of NFVI with a network focus:
> NFVbench:
> https://wiki.opnfv.org/display/nfvbench/NFVbench+Project+Proposal
>
>
>
> The NFVbench project is to develop a toolkit that allows developers,
> integrators, testers and customers to measure and assess the L2/L3
> forwarding performance of an NFV-infrastructure solution stack (i.e. OPNFV
> scenario) using a black-box approach.
>
>
>
> We’re hoping for a discussion in the technical community meeting on
> April/20, and are also asking for an official TSC review post the technical
> community review on May/2, so that NFVbench can participate in Euphrates.
> Consequently, NFVbench asks for tentative inclusion into Euphrates.
>
>
>
> Your thoughts and ideas are greatly appreciated.
>
>
>
> Thanks much, Frank, Carsten, Alec
>
>
>
>
> ___
> opnfv-tech-discuss mailing list
> opnfv-tech-discuss@lists.opnfv.org
> https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
>
-- 
Yujun Zhang
___
opnfv-tech-discuss mailing list
opnfv-tech-discuss@lists.opnfv.org
https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss