I wish this discussion was brought up on the tech-discuss mailing list ;-)
I had exactly the same questions than Frank at the very bottom and
asked similar questions to David this morning on the release packaging.
Heather is also right on target to feel like tools should be
independent of the platform release! This is clearly one of the
benefits of having an independent versioning for projects vs release
which I have been trying to push with the release tagging discussion.
For me there is absolutely no question that testing tools developed in
OPNFV will be used outside of OPNFV release cycle and potentially on
NFVi platforms that are not even in OPNFV labs and this clearly
mandates an independent versioning for each tool project.
Trying to version tools in lock step to OPNFV releases was reasonable
for the early releases but we saw first hand how it became more and
more difficult to keep this going as we get more tools/projects and
more dependencies between projects – and now more users of these
projects. As a matter of fact, functest is the first to break ranks
with the way they are planning to manage their version tags for
euphrates. I feel like functest will not even really use the official
euphrates image (the one tagged with “5.0.0” or now “opnfv-5.0.0”)
because it is just too static/slow-paced. Instead functest will use a
“stable” or “euphrates” tagged image (from discussion with Cedric and
Morgan). Functest is used outside of OPNFV labs so clearly they need
it versioned properly outside of the OPNFV release cycle.
I’m in the same situation with NFVbench, which is already used outside
of OPNFV labs.
Having said that, the sky is not falling on us because these are
actually good problems to have and shows industry adoption of OPNFV
projects. But we need to adjust the way projects are versioned, from a
best effort ad-hoc basis to a more focused/streamlined one.
More inline on NFVbench…
*From: *"morgan.richo...@orange.com" <morgan.richo...@orange.com>
*Date: *Tuesday, October 10, 2017 at 9:25 AM
*To: *"Frank Brockners (fbrockne)" <fbroc...@cisco.com>, Sofia Wallin
<sofia.wal...@ericsson.com>
*Cc: *Fatih Degirmenci <fatih.degirme...@ericsson.com>, Jose Lausuch
<jalaus...@suse.com>, "Alec Hothan (ahothan)" <ahot...@cisco.com>,
'Brandon Wick' <bw...@linuxfoundation.org>
*Subject: *Re: Releasing OPNFV tools in Euphrates
design is done by LF design team
they create icons for the projects and create the global image
the testing catalog page provides interactivity on the image to
display the different test cases
it is based on the cases descriptions created in the test databases
http://testresults.opnfv.org/test/api/v1/projects/yardstick/cases
http://testresults.opnfv.org/test/api/v1/projects/vsperf/cases
changing the image => change the processing for the roll over,
onclick, ...
in a first step, regarding the time frame, we could
1) change the figure in the doc (static)
=> need LF update @Brandon possible to get visual for NFVBench and
include it on the main picture?
as far as I understand it is not connected to CI, Test API, ..so it
should be somewhere in the Performance Testing box but without
connection with Test API or Yardstick
[Alec] I did read emails sent by Morgan regarding that testing landing
page but was not sure how to position NFVbench.
NFVbench also covers the operational side because it is designed from
the get go to run in production. Today there is no solution for doing
data plane performance measurement and monitoring on a production
cloud because it is just too costly and too complex to do with
traditional benchmarking methods (HW traffic generators in every cloud
is just not reasonable). NFVbench fills that gap for practically no
cost (you just need a NIC card - a tiny fraction of the cost of a pod).
So yes it can be used as a testing tool (to test and measure data
plane performance) but is also and foremost a tool for production. To
make it simpler in euphrates we can just put it in the testing group
for now. To make it even simpler, perhaps we should just add
“NFVbench” next to vsperf in the diagram since they both tackle the
data plane performance? No need to add a new logo, we can worry about
a better arrangement for Fraser
2) reference NFVBench in the testing wiki page
https://wiki.opnfv.org/display/testing
<https://wiki.opnfv.org/display/testing>
[Alec] see suggestion above.
3) add a chapter on NFVbench in testing ecosystem documentation =>
patch
https://git.opnfv.org/opnfvdocs/tree/docs/testing/ecosystem/overview.rst
-- remove CPerf (I planned ot do it)
-- add NFVBench section in the table after Functest (alphabetical
order)
[Alec] I can take care of this, will send out a gerrit review.
-- reference your documentation (NFVBench User guide and NFVBench
Configuration guide) if any under
https://git.opnfv.org/opnfvdocs/tree/docs/testing/testing-user.rst
[Alec] there is already a gerrit review for this (waiting for Sofia’s
approval).
https://gerrit.opnfv.org/gerrit/#/c/43521/
4) keep current figure for testing catalog as NFVBench did not declare
any case in the database and is not running in CI as far as I understand
[Alec] fine with that, CI/XCI/CD for NFVbench requires some
standardization in pod data plane wiring and can come in Fraser.
As of now, I am testing NFVbench on my own openstack pods to make sure
it is working.
Thanks,
Alec
On 10/10/2017 18:04, Frank Brockners (fbrockne) wrote:
Hi Morgan,
NFVbench for now is stand-alone. Intention is to offer a “one
button to hit” full stack qualification tool. Cc’ing Alec.
So I’d agree that we should add it to the testing-tools world map
picture. Do you have the source for that picture?
Thanks, Frank
*From:*morgan.richo...@orange.com
<mailto:morgan.richo...@orange.com>
[mailto:morgan.richo...@orange.com]
*Sent:* Dienstag, 10. Oktober 2017 17:58
*To:* Frank Brockners (fbrockne) <fbroc...@cisco.com>
<mailto:fbroc...@cisco.com>; Sofia Wallin
<sofia.wal...@ericsson.com> <mailto:sofia.wal...@ericsson.com>
*Cc:* Fatih Degirmenci <fatih.degirme...@ericsson.com>
<mailto:fatih.degirme...@ericsson.com>; Jose Lausuch
<jalaus...@suse.com> <mailto:jalaus...@suse.com>
*Subject:* Re: Releasing OPNFV tools in Euphrates
+1 for operations
BTW Frank, is NFV bench integrated in yardstick or independent
if it is independent, it would have made sense to include it in
the Testing overview picture...
http://docs.opnfv.org/en/latest/testing/ecosystem/overview.html#the-opnfv-testing-ecosystem
we could remove Cperf and add NFBench instead, but it means that
the picture should also be updated and the testing catalog as well
http://testresults.opnfv.org/testing/#!/select/visual
<http://testresults.opnfv.org/testing/#%21/select/visual>
I send the question several times on the mailing list/during
testing meeting if any new project would like to be integrated but
did not get any feedback
Did I miss something?
/Morgan
On 10/10/2017 17:39, Frank Brockners (fbrockne) wrote:
Hi Sofia,
good idea – roping in Faith, Morgan, Jose.
Folks, any thoughts on how to evolve the current docs
structure. Key need is to evolve the structure so that we make
a home for e.g. the Calipso documentation is which an
operational tool.
Thanks, Frank
*From:*Sofia Wallin [mailto:sofia.wal...@ericsson.com]
*Sent:* Dienstag, 10. Oktober 2017 17:37
*To:* Frank Brockners (fbrockne) <fbroc...@cisco.com>
<mailto:fbroc...@cisco.com>
*Subject:* Re: Releasing OPNFV tools in Euphrates
Hi Frank,
Since I know that both the Test WG and Infra WG have done and
are working on structuring and moving their documentation to
docs.opnfv.org I think I would be fair to invite someone from
each side.
We have definitely come to the point where we need to review
the bigger picture but the urgent question right now is, how
much can we do for Euphrates.
CI related documentation in added under ‘Infrastructure’. And
I don’t know if it makes the most sense to add/change the Test
Frameworks to OPNFV tools or add some kind of Operations
section under Infrastructure. Or maybe Operations would be a
new, stand-alone section.
Maybe we can have a discussion with Fatih and Morgan?
//Sofia
*From: *Frank Brockners <fbroc...@cisco.com
<mailto:fbroc...@cisco.com>>
*Date: *Tuesday, 10 October 2017 at 14:22
*To: *Sofia Wallin <sofia.wal...@ericsson.com
<mailto:sofia.wal...@ericsson.com>>
*Subject: *RE: Releasing OPNFV tools in Euphrates
Hi Sofia,
thanks. We could include NFVbench under “test tools” indeed.
Calispo is an operations support tool. It dynamically
discovers and shows you the network topology of the system
across all different layers (see e.g. calipso.io). Equally
well, we need a home for XCI.
How about we rename the “Test Frameworks” / “Testing user
guides” page to “OPNFV Tools” – that way we can have all the
different tools – for testing (functional, performance,
compliance – i.e. we’d include Dovetail’s tools), operations,
CI in one big bucket. That might be the lowest hanging fruit
for Euphrates. We can create a more granular structure in the
future.
Thoughts?
Thanks, Frank
*From:*Sofia Wallin [mailto:sofia.wal...@ericsson.com]
*Sent:* Dienstag, 10. Oktober 2017 14:02
*To:* Frank Brockners (fbrockne) <fbroc...@cisco.com
<mailto:fbroc...@cisco.com>>
*Subject:* Re: Releasing OPNFV tools in Euphrates
Hi Frank,
Ok, we will of course make necessary changes but just to sort
a few things out.
Currently we have ‘Release’, ‘Test Framework’, ‘Development’
and ‘Infrastructure’ on our landing page,
http://docs.opnfv.org/en/latest/index.html
And my understand is that NFVBench is a test tool? Which means
that it should be included under ‘test framework’?
And for the Calipso project, who is the targeted user or
audience for this tool?
Thanks,
Sofia
*From: *Frank Brockners <fbroc...@cisco.com
<mailto:fbroc...@cisco.com>>
*Date: *Tuesday, 10 October 2017 at 13:26
*To: *Sofia Wallin <sofia.wal...@ericsson.com
<mailto:sofia.wal...@ericsson.com>>
*Subject: *FW: Releasing OPNFV tools in Euphrates
Hi Sofia,
that is the latest, that I saw. I was assuming that Doc’s
would reflect the structure for the new releases page. IMHO we
could name the “other initiatives page” that Brandon mentions
below “Tools”. In that we could refer to (at least)
·Calipso: https://git.opnfv.org/calipso/tree/docs
·NFVbench: https://git.opnfv.org/nfvbench/tree/docs
·XCI: https://git.opnfv.org/releng-xci/tree/docs
Could this be a possible starting point for you to implement
things? Once we had a draft, we could ask for wider review.
Thanks, Frank
*From:*Brandon Wick [mailto:bw...@linuxfoundation.org]
*Sent:* Mittwoch, 4. Oktober 2017 03:47
*To:* Heather Kirksey <hkirk...@linuxfoundation.org
<mailto:hkirk...@linuxfoundation.org>>
*Cc:* Fatih Degirmenci <fatih.degirme...@ericsson.com
<mailto:fatih.degirme...@ericsson.com>>; Tim Irnich
<tim.irn...@ericsson.com <mailto:tim.irn...@ericsson.com>>;
Sofia Wallin <sofia.wal...@ericsson.com
<mailto:sofia.wal...@ericsson.com>>; Frank Brockners
(fbrockne) <fbroc...@cisco.com <mailto:fbroc...@cisco.com>>;
Raymond Paik <rp...@linuxfoundation.org
<mailto:rp...@linuxfoundation.org>>; David McBride
<dmcbr...@linuxfoundation.org
<mailto:dmcbr...@linuxfoundation.org>>
*Subject:* Re: Releasing OPNFV tools in Euphrates
Thanks all,
Appreciate the discussion as our team has been wondering how
we can better package up and promote these important elements
of OPNFV.
Specifically in regard to the software downloads page for
Euphrates (https://www.opnfv.org/software/downloads), we're
asking the web team to come up with a mock up of a new version
that does the following:
·Highlights the link to Release Documentation more prominently
·Changes format for "Installers" Header (it looks too much
like a button)
·Adds a Column for Daisy
·Adds a section for OPNFV Testing where we link to the Test
Project User Guides
·Add a section where we can feature another initiative (like
XCI, Calipso, etc)
I'll share this with y'all when I'm back on 10/16 and we'll
rev from there.
Best,
Brandon Wick
OPNFV Head of Marketing, The Linux Foundation
Mobile: +1.917.282.0960 Skype: wick.brandon
Email / Google Talk: bw...@linuxfoundation.org
<mailto:bw...@linuxfoundation.org>
On Thu, Sep 28, 2017 at 10:30 AM, Heather Kirksey
<hkirk...@linuxfoundation.org
<mailto:hkirk...@linuxfoundation.org>> wrote:
Frank,
It's as if you were reading my mind (and of course
expanding on it). I've been having a nascent idea that
given that we're getting quite a nice set of test tools
over all w/ all the performance and benchmarking ones +
Yardstick and Functest, that we might want to start
publishing all of them in a more obvious and consumable
fashion (and perhaps even decouple their release from the
"platform" releases). I think that set of test tools is in
many ways our real "product" and that we might want to
highlight that more.
Some of this is probably more forward-looking that
Euphrates specifically (since that is so close) but I'd
really like to see our entire suite of testing tools
packaged and documented in a way that makes it very
obvious how to use them in a way that's separate from the
scenario/installer builds. I am also thinking we might
want a deep dive white paper on testing to highlight these
capabilities.
In terms of what we need to do specifically (perhaps of a
minimum viable variety), let's work with Brandon and David
and Ray to make sure we've got what we need to arm the web
team when they're generating the artifacts page. It seems
as though we've got high level needs identified.
Heather
On Thu, Sep 28, 2017 at 6:22 AM, Fatih Degirmenci
<fatih.degirme...@ericsson.com
<mailto:fatih.degirme...@ericsson.com>> wrote:
Hi,
I agree to Frank's points below.
We only display installers on release page and it is
important to display others in a similar structure as
the docs.opnfv.org <http://docs.opnfv.org> on release
page; Testing, Operations, Infrastructure perhaps.
We will have docs.opnfv.org <http://docs.opnfv.org>
obviously but first thing potential users see is the
release page and it is important to highlight what
else we are doing so users can directly go to
documentation or whatever released as part of the release.
/Fatih
*From: *Sofia Wallin <sofia.wal...@ericsson.com
<mailto:sofia.wal...@ericsson.com>>
*Date: *Thursday, 28 September 2017 at 15:13
*To: *Frank Brockners <fbroc...@cisco.com
<mailto:fbroc...@cisco.com>>, Raymond Paik
<rp...@linuxfoundation.org
<mailto:rp...@linuxfoundation.org>>, Fatih Degirmenci
<fatih.degirme...@ericsson.com
<mailto:fatih.degirme...@ericsson.com>>, David McBride
<dmcbr...@linuxfoundation.org
<mailto:dmcbr...@linuxfoundation.org>>
*Cc: *Heather Kirksey <hkirk...@linuxfoundation.org
<mailto:hkirk...@linuxfoundation.org>>, Brandon Wick
<bw...@linuxfoundation.org
<mailto:bw...@linuxfoundation.org>>
*Subject: *Re: Releasing OPNFV tools in Euphrates
Hi everyone,
From a documentation point of view, we have already
prepared an “Infrastructure” section where XCI related
stuff will be published.
Work is ongoing to move material from the wiki to
docs.opnfv.org <http://docs.opnfv.org>.
I’m not fully up to date with what the Calipso project
does, but if there is need to adjust the structure on
the landing page for docs I’m happy to have a discussion.
//Sofia
*From: *Frank Brockners <fbroc...@cisco.com
<mailto:fbroc...@cisco.com>>
*Date: *Thursday, 28 September 2017 at 11:47
*To: *Raymond Paik <rp...@linuxfoundation.org
<mailto:rp...@linuxfoundation.org>>, Fatih Degirmenci
<fatih.degirme...@ericsson.com
<mailto:fatih.degirme...@ericsson.com>>,
"dmcbr...@linuxfoundation.org
<mailto:dmcbr...@linuxfoundation.org>"
<dmcbr...@linuxfoundation.org
<mailto:dmcbr...@linuxfoundation.org>>
*Cc: *Sofia Wallin <sofia.wal...@ericsson.com
<mailto:sofia.wal...@ericsson.com>>, Heather Kirksey
<hkirk...@linuxfoundation.org
<mailto:hkirk...@linuxfoundation.org>>, Brandon Wick
<bw...@linuxfoundation.org
<mailto:bw...@linuxfoundation.org>>
*Subject: *Releasing OPNFV tools in Euphrates
Hi folks,
just had a quick discussion over IRC with Fatih on the
question of how we release artifacts in Euphrates
which are not packaged with an installer, examples are
·Testing tools (like Functest, Yardstick, NFVbench)
·Operational tools (like Calipso)
·XCI tooling
Right now the download page only shows the different
installers. If someone would want to run e.g. FuncTest
as a standalone tool (which several customers I know
of do), you’ll need to do a bit of digging. With
Euphrates, things get even more diversified from a
tooling perspective (see above), hence we should
evolve how we present **all** our release artifacts to
the world, as opposed to limit the view to
installers/scenarios.
The main impact would be on the software downloads
page https://www.opnfv.org/software/downloads - and
documentation http://docs.opnfv.org/
<http://docs.opnfv.org/> (cc’ing Sophia as well). IMHO
we’d need dedicated sections on both, the releases
page as well as the documentation page for the
categories listed above (for testing we obviously
already have a dedicated documentation section).
Thoughts?
Thanks, Frank
--
*Heather Kirksey*
Director, OPNFV
Mobile: +1.512.917.7938 <tel:%28512%29%20917-7938>
Email/Google Talk: hkirk...@linuxfoundation.org
<mailto:hkirk...@linuxfoundation.org>
Skype: HeatherReneeKirksey
IRC: HKirksey
FV_RGB.png
--
Morgan Richomme
Orange/ IMT/ OLN/ CNC/ NCA/ SINA
Network architect for innovative services
Future of the Network community member
Open source Orange community manager
tel. +33 (0) 296 072 106
mob. +33 (0) 637 753 326
morgan.richo...@orange.com <mailto:morgan.richo...@orange.com>
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu
ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and
delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
--
Morgan Richomme
Orange/ IMT/ OLN/ CNC/ NCA/ SINA
Network architect for innovative services
Future of the Network community member
Open source Orange community manager
tel. +33 (0) 296 072 106
mob. +33 (0) 637 753 326
morgan.richo...@orange.com <mailto:morgan.richo...@orange.com>
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
