Hi all: Thanks for the deep suggestion and discussion.
The Dovetail and C&C have discussed many times about some of the questions. I will add the related slides and works on the I created a wiki page for these questions discussions. It will help others who missed the discussions. If you have any suggestion and ideas about the dovetail certification, welcome to add on the wikipage. https://wiki.opnfv.org/display/dovetail/high+level+topics I add two more topics on the wiki page: 1) scripts managements Since the dovetail will deliverable the test cases and scripts. The scripts are used to validate the testcases and may be related to the test projects. 2) how to choose the testcases(test cases criteria) This one may be added in the the test cases discussion wiki page. for more details on test cases, please refer to : https://wiki.opnfv.org/pages/viewpage.action?pageId=6827269 I will add the related slides and works on the wiki. Best Regards hongbo -----Original Message----- From: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Dave Neary Sent: 2016年8月9日 3:17 To: SULLIVAN, BRYAN L; Prakash Ramchandran; opnfv-tech-discuss@lists.opnfv.org Subject: Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes Hi Bryan, On 08/08/2016 10:45 AM, SULLIVAN, BRYAN L wrote: > I missed the last Dovetail meeting but from what I heard about some of > the discussion, I’d like to seek clarification on some things that > might have been expressed in the meeting, e.g. that implementations > which will go thru C&C may not be expected to be compatible with > existing OPNFV test suites, or at least that not all of the OPNFV test > suites, e.g. > FuncTest and Yardstick, would be expected to be tested on an certified > implementations. > > First I’d like to verify that such opinions were expressed (e.g. per > Bin’s comment that as a result “Dovetail testing and OPNFV tests are > different”), and have them further explained if possible. Yes, I expressed the view that Functest and Yardstick, as testing projects, were designed to stretch the platform - that we will periodically add failing functest tests to validate that the tests pass after we make a change to the upstream projects under test. They will also include tests which are targeting specific scenarios, and are known to fail (and thus will not be run) on other scenarios. In that sense, the Dovetail test suite should be a subset of FuncTest and Yardstick which pass on multiple scenarios and installers, and can be run unchanged on stacks with (for example) a proprietary SDN controller. > Second , given that the notes do capture the discussion and concerns > correctly, here are some thoughts about that: > > 1) The C&C committee is responsible for setting the “what is > expected” out of a certification, within some flexibility within > Doevtail as to what/how/when that can be delivered. > > 2) Overall, it’s expected that any implementation is compatible > with the OPNFV test suites as test frameworks. The degree of > compatibility with specific test may be limited e.g. if the target > hardware/software function focused on by the tests is not supported by > the implementation (e.g. an implementation that supports only a > specific SDNC), but the significance of such N/As needs to be > carefully considered by the C&C committee or Dovetail. This is the context for the "subset of..." I mentioned above. Chris started a page in the wiki a couple of weeks ago to describe the criteria for a test case in the Dovetail test framework, I and others added to it: https://wiki.opnfv.org/pages/viewpage.action?pageId=6827269 Thanks, Dave. > Overall we need to ensure that any aspect of certification can work > equally (same end result) on an OPNFV-based implementation (meaning a > collection of the core components as released in some OPNFV release, > or even with slight variations e.g. different versions of the > components), or another implementation. We should not leave tests out > of the Dovetail scope just because they will “not work” on some > implementations. There may be good reasons for them not to work (the > N/As), but if those reasons are simply based upon the test design, the > platform vendor should provide a compatible version of the tests based > upon the OPNFV tests, so that we can still certify the platform > functionally. Examples of this may be: > > - FuncTest > > o vIMS (Clearwater IMS) is based upon Orange’s implementation of the > Cloudify blueprint, using Cloudify as a VNFM. In the process, the > Cloudify Manager is installed as a VM under OpenStack, and then > executes the vIMS blueprint. I see no reason that this should not work > essentially the same in any other environment. AFAIK, the only > possible differences, which would need to be addressed by adding > options to the existing FuncTest code for this, are that e.g. a > different approach to kicking off the FuncTest framework is needed due > to differences in Jumphost OS or configuration. But OPNFV should not > be responsible for accommodating platform implementations that vary in > this way; the vendor should step up and implement the support so their > product can be validated. > > o The rest of FuncTest is pretty generic and I see no reason why it > should not be supportable. > > - Yardstick > > o As with FuncTest, the framework under which Yardstick operates may > need some tweaks for compatibility with the vendor implementation. > These tweaks need to be contributed to OPNFV by the vendor. > > o The C&C program may not initially include performance benchmarking, > but any implementation should have demonstrated compatibility with the > Yardstick test suite. > > - Other tests that we develop for Dovetail may go beyond > FuncTest and Yardstick, to focus on more complex use cases or specific > technical capabilities. In principle I would expect that these would > migrate into FuncTest and Yardstick however, over time, because if > they are important they need to be part of the base test system. These > might include tests for reference VNFs that we collect and run as > blueprints under various VNFMs, e.g. thru the Models project. In those > cases, if a vendor does not support one of the VNFMs for some reason > (as with vIMS/Cloudily), then they need to contribute the support > using their VNFM to OPNFV. > > - The rest of the Dovetail tests will be based upon existing > upstream test suites including certification suites such as RefStack. > We need to be proactively reaching out to these upstream teams, e.g. > per https://etherpad.openstack.org/p/interop-challenge-meeting-2016-08-03. > > > > > > Thanks, > > Bryan Sullivan | AT&T > > > > *From:* opnfv-tech-discuss-boun...@lists.opnfv.org > [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] *On Behalf Of > *Prakash Ramchandran > *Sent:* Friday, August 05, 2016 8:08 AM > *To:* opnfv-tech-discuss@lists.opnfv.org > *Subject:* [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting > notes > > > > Here is today's OPNFV Dovetail meeting notes based on Gotomeeting and > #opnfv-dovetail channels ... > > Agenda: > > 1)start point(L3VPN, SFC and IPV6) > > 2)test cases structures > > 3) additional basic testcases( for SDN controller and NFVI…) > > https://wiki.opnfv.org/pages/viewpage.action?pageId=6827269 > > 4)other issues > > > > For more details, please refer to: > https://wiki.opnfv.org/display/dovetail/Dovetail+Home > > > > Summary Notes of discussions: > > > > Chris defines what should be content of Dovetail > > should focus on requirements and not project and release anagement > > info test will be on SUT that includes OPNFV VIM + NFVi as shown in > link > > The details to follow the test plan > > Bin says we should first verify Functest and Yardstick before we start > on Dovetail > > Dave & Chris say they should be standlone and as a subset may be > needed > > Purpose of the Dovetail here is to show what is needed for OPNF view > of NFV compliance > > Bin sees a potential issue here that Dovtail testing and OPNFV tests > are different > > CORD example may be able to be claim OPNFV compatibility through > Dovetail but not from OPNFV Platform testing in Projects and Releases > > Bin says Bryan wants CORD to run over OPNFV platform and Chris says > its different as Release testing is diffrent from Dovetail testing > > Same holds for OPEN-O and OpenBaton > > Hongbo and Chris want to start form IPv6 overlay testing and Bin > suggested we reuse most of what we have from Service VM IPv6 testing > > Mathew stated he has started on it and can help > > Testsuya Nakamura chimed he can help in adding some specifc IPv6 > related to it for test case structure > > Tetsuya says we need a clear definition for SUT IPv6, SFC or L3VPN > before we start test plans for IPv6 > > What we are starting with can be IPv6 to establish test plan, test > design and test case documentaion templates > > Prioritizing the test cases can be taken at next meeting and define a > link to the same > > #action Hongbo to establish a link and bring use cases to bring to > table for disucssing priority > > further discussions over use case can be over email > > > > 07:51] <collabot> > Minutes: > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.html > > [07:51] <collabot> Minutes (text): > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnf > v-dovetail.2016-08-05-14.05.txt > > [07:51] <collabot> > Log: > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.log.html > > > > > > > > Note People who attended the Gotomeeting are not listed here in link, > please add them when Hongbo posts the summary to Dovetail wiki. > > The include > > Chris Price > > Dave Neary > > Bi Hu > > Tetsuya Nakmura > > > > > > > > *Prakash Ramchandran* > > logo_huawei* R&D USA* > > *FutureWei Technologies, Inc* > > Email:prakash.ramchand...@huawei.com <mailto:s.c...@huawei.com> > > Work: +1 (408) 330-5489 > > Mobile:+1 (408) 406-5810 > > 2330 Central Expy, Santa Clara, CA 95050, USA > > > > / / > > / / > > > > > > > > _______________________________________________ > opnfv-tech-discuss mailing list > opnfv-tech-discuss@lists.opnfv.org > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > -- Dave Neary - NFV/SDN Community Strategy Open Source and Standards, Red Hat - http://community.redhat.com Ph: +1-978-399-2182 / Cell: +1-978-799-3338 _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
