Chris, I either don't understand or don't agree to the statement that the Functest isn't designed to be generic, at least as a framework. Certainly where we leverage upstream test suites e.g. Tempest/rally they are contextual, but so far we have no other cloud VIM in OPNFV except for OpenStack. Some tests might be specific to some SDNCs, but again thats just because we are using the upstream tests. Functest as a framework though *is* IMO a generic framework, and we should leverage it thru Dovetail for testing against expectations for the system components under test.
We do not yet AFAICT a concept of a high-level intent framework through which we could drive any implementation/VIM, and I don't expect any such framework soon. Certainly it's one of the goals of the broader communities e.g. ONF and the Multi-SDO Information Modeling activity, but will take a while to materialize. So I don't see how we can expect to use a truly generic test suite for any comprehensive purpose, anytime soon. On Aug 8, 2016, at 7:58 AM, Christopher Price <chrispric...@gmail.com<mailto:chrispric...@gmail.com>> wrote: Hi Bryan, There was a fair amount of discussion around what Dovetail is testing. One item I was trying to articulate is that the purpose of the Dovetail suite is to validate the compliance of a platform to the behaviors and interfaces targeted for compliance validation in OPNFV. This differs from the purpose of Functest and Yardstick which is to validate as thoroughly as possible the design of the OPNFV platform. I have one small concern with your comment: ”The rest of FuncTest is pretty generic“ Functest is anything but generic as far as a platform for generic platform feature validation is concerned, functest test cases derive mostly from the upstream community and do not, by design, provide neutral generic test cases for a platform. This is not a bad thing, it simply does not lend itself to creating test cases that should be able to be run over any platform composition. Our Dovetail suite needs to work even when someone swaps out OpenStack for another VIM, or a completely different SDN controller for instance. / Chris From: <opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>> on behalf of "SULLIVAN, BRYAN L" <bs3...@att.com<mailto:bs3...@att.com>> Date: Monday 8 August 2016 at 16:45 To: Prakash Ramchandran <prakash.ramchand...@huawei.com<mailto:prakash.ramchand...@huawei.com>>, TECH-DISCUSS OPNFV <opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>> Subject: Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes Hi all, I missed the last Dovetail meeting but from what I heard about some of the discussion, I’d like to seek clarification on some things that might have been expressed in the meeting, e.g. that implementations which will go thru C&C may not be expected to be compatible with existing OPNFV test suites, or at least that not all of the OPNFV test suites, e.g. FuncTest and Yardstick, would be expected to be tested on an certified implementations. First I’d like to verify that such opinions were expressed (e.g. per Bin’s comment that as a result “Dovetail testing and OPNFV tests are different”), and have them further explained if possible. Second, given that the notes do capture the discussion and concerns correctly, here are some thoughts about that: 1) The C&C committee is responsible for setting the “what is expected” out of a certification, within some flexibility within Doevtail as to what/how/when that can be delivered. 2) Overall, it’s expected that any implementation is compatible with the OPNFV test suites as test frameworks. The degree of compatibility with specific test may be limited e.g. if the target hardware/software function focused on by the tests is not supported by the implementation (e.g. an implementation that supports only a specific SDNC), but the significance of such N/As needs to be carefully considered by the C&C committee or Dovetail. Overall we need to ensure that any aspect of certification can work equally (same end result) on an OPNFV-based implementation (meaning a collection of the core components as released in some OPNFV release, or even with slight variations e.g. different versions of the components), or another implementation. We should not leave tests out of the Dovetail scope just because they will “not work” on some implementations. There may be good reasons for them not to work (the N/As), but if those reasons are simply based upon the test design, the platform vendor should provide a compatible version of the tests based upon the OPNFV tests, so that we can still certify the platform functionally. Examples of this may be: - FuncTest o vIMS (Clearwater IMS) is based upon Orange’s implementation of the Cloudify blueprint, using Cloudify as a VNFM. In the process, the Cloudify Manager is installed as a VM under OpenStack, and then executes the vIMS blueprint. I see no reason that this should not work essentially the same in any other environment. AFAIK, the only possible differences, which would need to be addressed by adding options to the existing FuncTest code for this, are that e.g. a different approach to kicking off the FuncTest framework is needed due to differences in Jumphost OS or configuration. But OPNFV should not be responsible for accommodating platform implementations that vary in this way; the vendor should step up and implement the support so their product can be validated. o The rest of FuncTest is pretty generic and I see no reason why it should not be supportable. - Yardstick o As with FuncTest, the framework under which Yardstick operates may need some tweaks for compatibility with the vendor implementation. These tweaks need to be contributed to OPNFV by the vendor. o The C&C program may not initially include performance benchmarking, but any implementation should have demonstrated compatibility with the Yardstick test suite. - Other tests that we develop for Dovetail may go beyond FuncTest and Yardstick, to focus on more complex use cases or specific technical capabilities. In principle I would expect that these would migrate into FuncTest and Yardstick however, over time, because if they are important they need to be part of the base test system. These might include tests for reference VNFs that we collect and run as blueprints under various VNFMs, e.g. thru the Models project. In those cases, if a vendor does not support one of the VNFMs for some reason (as with vIMS/Cloudily), then they need to contribute the support using their VNFM to OPNFV. - The rest of the Dovetail tests will be based upon existing upstream test suites including certification suites such as RefStack. We need to be proactively reaching out to these upstream teams, e.g. per https://etherpad.openstack.org/p/interop-challenge-meeting-2016-08-03. Thanks, Bryan Sullivan | AT&T From: opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Prakash Ramchandran Sent: Friday, August 05, 2016 8:08 AM To: opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org> Subject: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes Here is today's OPNFV Dovetail meeting notes based on Gotomeeting and #opnfv-dovetail channels ... Agenda: 1)start point(L3VPN, SFC and IPV6) 2)test cases structures 3) additional basic testcases( for SDN controller and NFVI…) https://wiki.opnfv.org/pages/viewpage.action?pageId=6827269 4)other issues For more details, please refer to: https://wiki.opnfv.org/display/dovetail/Dovetail+Home Summary Notes of discussions: Chris defines what should be content of Dovetail should focus on requirements and not project and release anagement info test will be on SUT that includes OPNFV VIM + NFVi as shown in link The details to follow the test plan Bin says we should first verify Functest and Yardstick before we start on Dovetail Dave & Chris say they should be standlone and as a subset may be needed Purpose of the Dovetail here is to show what is needed for OPNF view of NFV compliance Bin sees a potential issue here that Dovtail testing and OPNFV tests are different CORD example may be able to be claim OPNFV compatibility through Dovetail but not from OPNFV Platform testing in Projects and Releases Bin says Bryan wants CORD to run over OPNFV platform and Chris says its different as Release testing is diffrent from Dovetail testing Same holds for OPEN-O and OpenBaton Hongbo and Chris want to start form IPv6 overlay testing and Bin suggested we reuse most of what we have from Service VM IPv6 testing Mathew stated he has started on it and can help Testsuya Nakamura chimed he can help in adding some specifc IPv6 related to it for test case structure Tetsuya says we need a clear definition for SUT IPv6, SFC or L3VPN before we start test plans for IPv6 What we are starting with can be IPv6 to establish test plan, test design and test case documentaion templates Prioritizing the test cases can be taken at next meeting and define a link to the same #action Hongbo to establish a link and bring use cases to bring to table for disucssing priority further discussions over use case can be over email 07:51] <collabot> Minutes: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.html [07:51] <collabot> Minutes (text): http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.txt [07:51] <collabot> Log: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.log.html Note People who attended the Gotomeeting are not listed here in link, please add them when Hongbo posts the summary to Dovetail wiki. The include Chris Price Dave Neary Bi Hu Tetsuya Nakmura Prakash Ramchandran <image001.png> R&D USA FutureWei Technologies, Inc Email: prakash.ramchand...@huawei.com<mailto:s.c...@huawei.com> Work: +1 (408) 330-5489 Mobile: +1 (408) 406-5810 2330 Central Expy, Santa Clara, CA 95050, USA _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org> https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss <image001.png>
_______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
