Hi Bryan, On 08/08/2016 10:45 AM, SULLIVAN, BRYAN L wrote: > I missed the last Dovetail meeting but from what I heard about some of > the discussion, I’d like to seek clarification on some things that might > have been expressed in the meeting, e.g. that implementations which will > go thru C&C may not be expected to be compatible with existing OPNFV > test suites, or at least that not all of the OPNFV test suites, e.g. > FuncTest and Yardstick, would be expected to be tested on an certified > implementations. > > First I’d like to verify that such opinions were expressed (e.g. per > Bin’s comment that as a result “Dovetail testing and OPNFV tests are > different”), and have them further explained if possible.
Yes, I expressed the view that Functest and Yardstick, as testing projects, were designed to stretch the platform - that we will periodically add failing functest tests to validate that the tests pass after we make a change to the upstream projects under test. They will also include tests which are targeting specific scenarios, and are known to fail (and thus will not be run) on other scenarios. In that sense, the Dovetail test suite should be a subset of FuncTest and Yardstick which pass on multiple scenarios and installers, and can be run unchanged on stacks with (for example) a proprietary SDN controller. > Second , given that the notes do capture the discussion and concerns > correctly, here are some thoughts about that: > > 1) The C&C committee is responsible for setting the “what is > expected” out of a certification, within some flexibility within > Doevtail as to what/how/when that can be delivered. > > 2) Overall, it’s expected that any implementation is compatible > with the OPNFV test suites as test frameworks. The degree of > compatibility with specific test may be limited e.g. if the target > hardware/software function focused on by the tests is not supported by > the implementation (e.g. an implementation that supports only a specific > SDNC), but the significance of such N/As needs to be carefully > considered by the C&C committee or Dovetail. This is the context for the "subset of..." I mentioned above. Chris started a page in the wiki a couple of weeks ago to describe the criteria for a test case in the Dovetail test framework, I and others added to it: https://wiki.opnfv.org/pages/viewpage.action?pageId=6827269 Thanks, Dave. > Overall we need to ensure that any aspect of certification can work > equally (same end result) on an OPNFV-based implementation (meaning a > collection of the core components as released in some OPNFV release, or > even with slight variations e.g. different versions of the components), > or another implementation. We should not leave tests out of the Dovetail > scope just because they will “not work” on some implementations. There > may be good reasons for them not to work (the N/As), but if those > reasons are simply based upon the test design, the platform vendor > should provide a compatible version of the tests based upon the OPNFV > tests, so that we can still certify the platform functionally. Examples > of this may be: > > - FuncTest > > o vIMS (Clearwater IMS) is based upon Orange’s implementation of the > Cloudify blueprint, using Cloudify as a VNFM. In the process, the > Cloudify Manager is installed as a VM under OpenStack, and then executes > the vIMS blueprint. I see no reason that this should not work > essentially the same in any other environment. AFAIK, the only possible > differences, which would need to be addressed by adding options to the > existing FuncTest code for this, are that e.g. a different approach to > kicking off the FuncTest framework is needed due to differences in > Jumphost OS or configuration. But OPNFV should not be responsible for > accommodating platform implementations that vary in this way; the vendor > should step up and implement the support so their product can be validated. > > o The rest of FuncTest is pretty generic and I see no reason why it > should not be supportable. > > - Yardstick > > o As with FuncTest, the framework under which Yardstick operates may > need some tweaks for compatibility with the vendor implementation. These > tweaks need to be contributed to OPNFV by the vendor. > > o The C&C program may not initially include performance benchmarking, > but any implementation should have demonstrated compatibility with the > Yardstick test suite. > > - Other tests that we develop for Dovetail may go beyond > FuncTest and Yardstick, to focus on more complex use cases or specific > technical capabilities. In principle I would expect that these would > migrate into FuncTest and Yardstick however, over time, because if they > are important they need to be part of the base test system. These might > include tests for reference VNFs that we collect and run as blueprints > under various VNFMs, e.g. thru the Models project. In those cases, if a > vendor does not support one of the VNFMs for some reason (as with > vIMS/Cloudily), then they need to contribute the support using their > VNFM to OPNFV. > > - The rest of the Dovetail tests will be based upon existing > upstream test suites including certification suites such as RefStack. We > need to be proactively reaching out to these upstream teams, e.g. per > https://etherpad.openstack.org/p/interop-challenge-meeting-2016-08-03. > > > > > > Thanks, > > Bryan Sullivan | AT&T > > > > *From:* opnfv-tech-discuss-boun...@lists.opnfv.org > [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] *On Behalf Of > *Prakash Ramchandran > *Sent:* Friday, August 05, 2016 8:08 AM > *To:* opnfv-tech-discuss@lists.opnfv.org > *Subject:* [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes > > > > Here is today's OPNFV Dovetail meeting notes based on Gotomeeting and > #opnfv-dovetail channels ... > > Agenda: > > 1)start point(L3VPN, SFC and IPV6) > > 2)test cases structures > > 3) additional basic testcases( for SDN controller and NFVI…) > > https://wiki.opnfv.org/pages/viewpage.action?pageId=6827269 > > 4)other issues > > > > For more details, please refer to: > https://wiki.opnfv.org/display/dovetail/Dovetail+Home > > > > Summary Notes of discussions: > > > > Chris defines what should be content of Dovetail > > should focus on requirements and not project and release anagement > > info test will be on SUT that includes OPNFV VIM + NFVi as shown in link > > The details to follow the test plan > > Bin says we should first verify Functest and Yardstick before we start > on Dovetail > > Dave & Chris say they should be standlone and as a subset may be needed > > Purpose of the Dovetail here is to show what is needed for OPNF view of > NFV compliance > > Bin sees a potential issue here that Dovtail testing and OPNFV tests are > different > > CORD example may be able to be claim OPNFV compatibility through > Dovetail but not from OPNFV Platform testing in Projects and Releases > > Bin says Bryan wants CORD to run over OPNFV platform and Chris says > its different as Release testing is diffrent from Dovetail testing > > Same holds for OPEN-O and OpenBaton > > Hongbo and Chris want to start form IPv6 overlay testing and Bin > suggested we reuse most of what we have from Service VM IPv6 testing > > Mathew stated he has started on it and can help > > Testsuya Nakamura chimed he can help in adding some specifc IPv6 related > to it for test case structure > > Tetsuya says we need a clear definition for SUT IPv6, SFC or L3VPN > before we start test plans for IPv6 > > What we are starting with can be IPv6 to establish test plan, test > design and test case documentaion templates > > Prioritizing the test cases can be taken at next meeting and define a > link to the same > > #action Hongbo to establish a link and bring use cases to bring to table > for disucssing priority > > further discussions over use case can be over email > > > > 07:51] <collabot> > Minutes: > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.html > > [07:51] <collabot> Minutes (text): > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.txt > > [07:51] <collabot> > Log: > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.log.html > > > > > > > > Note People who attended the Gotomeeting are not listed here in link, > please add them when Hongbo posts the summary to Dovetail wiki. > > The include > > Chris Price > > Dave Neary > > Bi Hu > > Tetsuya Nakmura > > > > > > > > *Prakash Ramchandran* > > logo_huawei* R&D USA* > > *FutureWei Technologies, Inc* > > Email:prakash.ramchand...@huawei.com <mailto:s.c...@huawei.com> > > Work: +1 (408) 330-5489 > > Mobile:+1 (408) 406-5810 > > 2330 Central Expy, Santa Clara, CA 95050, USA > > > > / / > > / / > > > > > > > > _______________________________________________ > opnfv-tech-discuss mailing list > opnfv-tech-discuss@lists.opnfv.org > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > -- Dave Neary - NFV/SDN Community Strategy Open Source and Standards, Red Hat - http://community.redhat.com Ph: +1-978-399-2182 / Cell: +1-978-799-3338 _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
