Thanks for the clarification.
The first two questions are mainly coming from the fact, that people
usually will *guess* the scope from naming. Maybe choosing a cool name, not
so generic as "NFV bench" will cause less confusion on the project scope.
I would suggest something like "{project code name} - NFVi data plane
benchmarking". This is the naming convention for most OPNFV projects, like
Bamboo, Yardstick, QTIP and etc... My two cents.
About the toolkit, I couldn't agree more on the statement "The crafting of
network benchmarking conditions is best done by experts in networking,
these are people who are the best positioned to know what makes sense to
test and in what conditions".
One of the primary goal of QTIP would be provide *common* benchmarking
components and *post-process* services to domain experts. I'm eagering to
to get some inputs from "NFVBench" about what is the requirements and how
we could work together to provide the community a better benchmarking :-)
We may discuss it in a separate thread.
On Sat, Apr 15, 2017 at 1:13 AM Alec Hothan (ahothan) <ahot...@cisco.com>
wrote:
> Hi Yujun,
>
>
>
> Inline…
>
>
>
>
>
> *From: *"Yujun Zhang (ZTE)" <zhangyujun+...@gmail.com>
> *Date: *Wednesday, April 12, 2017 at 5:56 PM
> *To: *"Frank Brockners (fbrockne)" <fbroc...@cisco.com>, TSC OPNFV <
> opnfv-...@lists.opnfv.org>, TECH-DISCUSS OPNFV <
> opnfv-tech-discuss@lists.opnfv.org>, test-wg <test...@lists.opnfv.org>
> *Cc: *Carsten Rossenhoevel <cr...@eantc.de>, "Alec Hothan (ahothan)" <
> ahot...@cisco.com>
> *Subject: *Re: [opnfv-tech-discuss] New project proposal: NFVbench
>
>
>
> Some question after reviewing the proposal.
>
> · Technically, is data plane / L2/L3 forwarding the * only*
> performance indicator for NFV?
>
> Generally speaking, L2 forwarding is *the* main performance indicator for
> an NFVi data plane: measure how many frame/sec can the data plane handle
> under a well defined set of constraints. The set of constraints is actually
> a very important part of the picture because it can cloud the picture if
> not defined properly. What NFVbench brings is an automation of the well
> defined set of constraints so that these benchmarks can be repeated by more
> than 1 party and so that 2 different implementations can be compared more
> easily.
>
> The description mentions L2/L3 forwarding performance, the L3 part is
> simply because all frames are fully formed IPv4 L3 frames (even if the L3
> header is not usually used by the NFVi data plane) and as such - this will
> also allow simple L3 forwarding benchmarking of any service chain. There is
> no plan to cover full blown VNF performance benchmarking in this project.
>
> To answer your question to the letter, L2/L3 forwarding performance is
> *not* the only performance indicator for NFV (but is an important one).
>
> · Will NFVbench project expand to other performance metrics
> besides the main focus?
>
> The simple answer is no. Many open source projects suffer from
> over-diversification or over-scoping (want to do too many things).
> Measuring the NFVi data plane properly is already a pretty difficult task
> that requires experience, focus and discipline and this already represents
> a lot more work to cover than we have resources.
>
> · In what way will NFVbench plan to complement and leverage
> related projects?
>
> I will reply to the positioning with other OPNFV projects in a separate
> email.
>
> · What will be the main components of the toolkit? A test runner?
> Test cases? Drivers?
>
> The toolkit will be a compact specialized test runner that can run
> standalone but can also be easily integrated into a more generic test
> harness (such as Jenkins, yardstic or qtip) using well defined
> APIs/interfaces. Regarding test cases, we have decided to follow a
> different approach with the use of pre-built packet paths. Many
> benchmarking tools provide the ability for the user to fully customize the
> benchmark parameters (as in writing python code). With NFVbench we want to
> limit that ability to developers of the tool itself for the following
> reasons:
>
> · The crafting of network benchmarking conditions is best done by
> experts in networking, these are people who are the best positioned to know
> what makes sense to test and in what conditions
>
> · having too many people or worst - users of the tool - modify
> the conditions of the benchmark will have a huge negative impact on the
> usability of the results generated (in that – it will be a lot more
> difficult to compare them since the conditions have been modified)
>
> · and finally, the vast majority of targeted users of NFVbench
> will be non dataplane experts that have no intent to modify the code or to
> add more variations to the benchmarks than necessary
>
> So, NFVbench will define what the benchmark is and how it is conducted, it
> will be open source so everyone can discuss publicly on benchmarking
> conditions and see how the benchmark is conducted and automated. One
> important note is that there is no such a thing as a perfect benchmark
> tool, but it is possible to get one that is good enough for the community.
>
> Today, it is impossible to compare 2 full stack data plane benchmarks
> because the conditions they are conducted are generally very different and
> often too subtle for many people to make sense of. These benchmarks are
> also often very difficult to reproduce (not mentioning by third party).This
> is what NFVbench is planning to address.
>
> Thanks
>
> Alec
>
>
>
>
>
> The following text are quoted for quick reference
>
>
>
> The NFVbench project develops a *toolkit* that allows developers,
> integrators, testers and customers to measure and assess the *L2/L3
> forwarding performance* of an NFV-infrastructure solution stack (i.e.
> OPNFV scenario) using a black-box approach.
>
> ...
>
> The main focus of NFVbench is the NFVI full stack *data plane
> benchmarking* using realistic production deployment conditions.
>
> NFVbench does *not* focus on the following areas and will align with,
> *complement
> and leverage* projects that already cover properly these areas:
>
> ...
>
>
>
> On Thu, Apr 13, 2017 at 12:40 AM Frank Brockners (fbrockne) <
> fbroc...@cisco.com> wrote:
>
> Hi OPNFV,
>
>
>
> over the past few weeks we’ve distilled a proposals to create a toolkit to
> allow for black-box performance testing of NFVI with a network focus:
> NFVbench:
> https://wiki.opnfv.org/display/nfvbench/NFVbench+Project+Proposal
>
>
>
> The NFVbench project is to develop a toolkit that allows developers,
> integrators, testers and customers to measure and assess the L2/L3
> forwarding performance of an NFV-infrastructure solution stack (i.e. OPNFV
> scenario) using a black-box approach.
>
>
>
> We’re hoping for a discussion in the technical community meeting on
> April/20, and are also asking for an official TSC review post the technical
> community review on May/2, so that NFVbench can participate in Euphrates.
> Consequently, NFVbench asks for tentative inclusion into Euphrates.
>
>
>
> Your thoughts and ideas are greatly appreciated.
>
>
>
> Thanks much, Frank, Carsten, Alec
>
>
>
>
>
> _______________________________________________
> opnfv-tech-discuss mailing list
> opnfv-tech-discuss@lists.opnfv.org
> https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
>
> --
>
> Yujun Zhang
>
--
Yujun Zhang
_______________________________________________
opnfv-tech-discuss mailing list
opnfv-tech-discuss@lists.opnfv.org
https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
