Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes
+1. Let me give an example of IPv6, which Dovetail is using it as a template of developing test cases: Step 1: understand IPv6 feature and its gaps with Mitaka and Boron * Reference: http://artifacts.opnfv.org/ipv6/docs/userguide/index.html Step 2: understand current test case coverage in FuncTest and Yardstick * Reference 1: http://artifacts.opnfv.org/ipv6/docs/installationprocedure/index.html#testing-methodology * Reference 2: https://git.opnfv.org/cgit/yardstick/tree/tests/opnfv/test_cases/opnfv_yardstick_tc027.yaml Step 3: leverage as fully as possible of above test cases * In case of IPv6, I don't see a reason why current FuncTest and Yardstick cannot be reused * Some changes in Yardstick test code may be needed to replace the hard-coded IP address, ext-net name etc. with general parameters Step 4: identify the gaps of IPv6 use case in Dovetail with the IPv6 project's test case coverage in FuncTest and Yardstick Step 5: If needed, develop additional IPv6 test case in Dovetail *only* to cover the gaps identified in Step 4. This way, we leverage platform testing through FuncTest and Yardstick to ensure the platform compliance, while add additional value in Dovetail for C purpose. Thanks Bin -Original Message- From: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of SULLIVAN, BRYAN L Sent: Wednesday, August 10, 2016 1:43 PM To: Dave Neary <dne...@redhat.com> Cc: opnfv-tech-discuss@lists.opnfv.org Subject: Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes *** Security Advisory: This Message Originated Outside of AT ***. Reference http://cso.att.com/EmailSecurity/IDSP.html for more information. I would say rather that the Dovetail system needs to be aware of what system and components are under test, and select as fully as possible the compatible tests from the Functest and Yardstick suites. We should not narrow down the scope to just what will run across all implementations, even proprietary (e.g. those implementations not supporting the open source project VIMs with their published interfaces). There would be little in common, and we would be left trying to certify that something meets the "spirit" of an NFV/SDN system with any specific interoperability expectations. I don't think that is what the C program is about. On Aug 8, 2016, at 12:16 PM, Dave Neary <dne...@redhat.com> wrote: Hi Bryan, > On 08/08/2016 10:45 AM, SULLIVAN, BRYAN L wrote: > I missed the last Dovetail meeting but from what I heard about some of > the discussion, I�d like to seek clarification on some things that > might have been expressed in the meeting, e.g. that implementations > which will go thru C may not be expected to be compatible with > existing OPNFV test suites, or at least that not all of the OPNFV test > suites, e.g. > FuncTest and Yardstick, would be expected to be tested on an certified > implementations. > > First I�d like to verify that such opinions were expressed (e.g. per > Bin�s comment that as a result �Dovetail testing and OPNFV tests > are different�), and have them further explained if possible. Yes, I expressed the view that Functest and Yardstick, as testing projects, were designed to stretch the platform - that we will periodically add failing functest tests to validate that the tests pass after we make a change to the upstream projects under test. They will also include tests which are targeting specific scenarios, and are known to fail (and thus will not be run) on other scenarios. In that sense, the Dovetail test suite should be a subset of FuncTest and Yardstick which pass on multiple scenarios and installers, and can be run unchanged on stacks with (for example) a proprietary SDN controller. > Second , given that the notes do capture the discussion and concerns > correctly, here are some thoughts about that: > > 1) The C committee is responsible for setting the �what is > expected� out of a certification, within some flexibility within > Doevtail as to what/how/when that can be delivered. > > 2) Overall, it�s expected that any implementation is compatible > with the OPNFV test suites as test frameworks. The degree of > compatibility with specific test may be limited e.g. if the target > hardware/software function focused on by the tests is not supported by > the implementation (e.g. an implementation that supports only a > specific SDNC), but the significance of such N/As needs to be > carefully considered by the C committee or Dovetail. This is the context for the "subset of..." I mentioned above. Chris started a page in the wiki a couple of weeks ago to describe the criteria for a test case in the Dovetail test framework, I and others added to it: https://wiki.opnfv.org/pages/viewpage.action?pageId=6
Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes
Chris, I either don't understand or don't agree to the statement that the Functest isn't designed to be generic, at least as a framework. Certainly where we leverage upstream test suites e.g. Tempest/rally they are contextual, but so far we have no other cloud VIM in OPNFV except for OpenStack. Some tests might be specific to some SDNCs, but again thats just because we are using the upstream tests. Functest as a framework though *is* IMO a generic framework, and we should leverage it thru Dovetail for testing against expectations for the system components under test. We do not yet AFAICT a concept of a high-level intent framework through which we could drive any implementation/VIM, and I don't expect any such framework soon. Certainly it's one of the goals of the broader communities e.g. ONF and the Multi-SDO Information Modeling activity, but will take a while to materialize. So I don't see how we can expect to use a truly generic test suite for any comprehensive purpose, anytime soon. On Aug 8, 2016, at 7:58 AM, Christopher Price <chrispric...@gmail.com<mailto:chrispric...@gmail.com>> wrote: Hi Bryan, There was a fair amount of discussion around what Dovetail is testing. One item I was trying to articulate is that the purpose of the Dovetail suite is to validate the compliance of a platform to the behaviors and interfaces targeted for compliance validation in OPNFV. This differs from the purpose of Functest and Yardstick which is to validate as thoroughly as possible the design of the OPNFV platform. I have one small concern with your comment: ”The rest of FuncTest is pretty generic“ Functest is anything but generic as far as a platform for generic platform feature validation is concerned, functest test cases derive mostly from the upstream community and do not, by design, provide neutral generic test cases for a platform. This is not a bad thing, it simply does not lend itself to creating test cases that should be able to be run over any platform composition. Our Dovetail suite needs to work even when someone swaps out OpenStack for another VIM, or a completely different SDN controller for instance. / Chris From: <opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>> on behalf of "SULLIVAN, BRYAN L" <bs3...@att.com<mailto:bs3...@att.com>> Date: Monday 8 August 2016 at 16:45 To: Prakash Ramchandran <prakash.ramchand...@huawei.com<mailto:prakash.ramchand...@huawei.com>>, TECH-DISCUSS OPNFV <opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>> Subject: Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes Hi all, I missed the last Dovetail meeting but from what I heard about some of the discussion, I’d like to seek clarification on some things that might have been expressed in the meeting, e.g. that implementations which will go thru C may not be expected to be compatible with existing OPNFV test suites, or at least that not all of the OPNFV test suites, e.g. FuncTest and Yardstick, would be expected to be tested on an certified implementations. First I’d like to verify that such opinions were expressed (e.g. per Bin’s comment that as a result “Dovetail testing and OPNFV tests are different”), and have them further explained if possible. Second, given that the notes do capture the discussion and concerns correctly, here are some thoughts about that: 1) The C committee is responsible for setting the “what is expected” out of a certification, within some flexibility within Doevtail as to what/how/when that can be delivered. 2) Overall, it’s expected that any implementation is compatible with the OPNFV test suites as test frameworks. The degree of compatibility with specific test may be limited e.g. if the target hardware/software function focused on by the tests is not supported by the implementation (e.g. an implementation that supports only a specific SDNC), but the significance of such N/As needs to be carefully considered by the C committee or Dovetail. Overall we need to ensure that any aspect of certification can work equally (same end result) on an OPNFV-based implementation (meaning a collection of the core components as released in some OPNFV release, or even with slight variations e.g. different versions of the components), or another implementation. We should not leave tests out of the Dovetail scope just because they will “not work” on some implementations. There may be good reasons for them not to work (the N/As), but if those reasons are simply based upon the test design, the platform vendor should provide a compatible version of the tests based upon the OPNFV tests, so that we can still certify the platform functionally. Examples of this may be: - FuncTest ovIMS (Clearwater IMS) is based upon Orange’s implementation of the Cloudify blueprint, using
Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes
e. > > - Yardstick > > o As with FuncTest, the framework under which Yardstick operates may > need some tweaks for compatibility with the vendor implementation. These > tweaks need to be contributed to OPNFV by the vendor. > > o The C program may not initially include performance benchmarking, > but any implementation should have demonstrated compatibility with the > Yardstick test suite. > > - Other tests that we develop for Dovetail may go beyond > FuncTest and Yardstick, to focus on more complex use cases or specific > technical capabilities. In principle I would expect that these would > migrate into FuncTest and Yardstick however, over time, because if they > are important they need to be part of the base test system. These might > include tests for reference VNFs that we collect and run as blueprints > under various VNFMs, e.g. thru the Models project. In those cases, if a > vendor does not support one of the VNFMs for some reason (as with > vIMS/Cloudily), then they need to contribute the support using their > VNFM to OPNFV. > > - The rest of the Dovetail tests will be based upon existing > upstream test suites including certification suites such as RefStack. We > need to be proactively reaching out to these upstream teams, e.g. per > https://etherpad.openstack.org/p/interop-challenge-meeting-2016-08-03. > > > > > > Thanks, > > Bryan Sullivan | AT > > > > *From:* opnfv-tech-discuss-boun...@lists.opnfv.org > [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] *On Behalf Of > *Prakash Ramchandran > *Sent:* Friday, August 05, 2016 8:08 AM > *To:* opnfv-tech-discuss@lists.opnfv.org > *Subject:* [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes > > > > Here is today's OPNFV Dovetail meeting notes based on Gotomeeting and > #opnfv-dovetail channels ... > > Agenda: > > 1)start point(L3VPN, SFC and IPV6) > > 2)test cases structures > > 3) additional basic testcases( for SDN controller and NFVI�) > >https://wiki.opnfv.org/pages/viewpage.action?pageId=6827269 > > 4)other issues > > > > For more details, please refer to: > https://wiki.opnfv.org/display/dovetail/Dovetail+Home > > > > Summary Notes of discussions: > > > > Chris defines what should be content of Dovetail > > should focus on requirements and not project and release anagement > > info test will be on SUT that includes OPNFV VIM + NFVi as shown in link > > The details to follow the test plan > > Bin says we should first verify Functest and Yardstick before we start > on Dovetail > > Dave & Chris say they should be standlone and as a subset may be needed > > Purpose of the Dovetail here is to show what is needed for OPNF view of > NFV compliance > > Bin sees a potential issue here that Dovtail testing and OPNFV tests are > different > > CORD example may be able to be claim OPNFV compatibility through > Dovetail but not from OPNFV Platform testing in Projects and Releases > > Bin says Bryan wants CORD to run over OPNFV platform and Chris says > its different as Release testing is diffrent from Dovetail testing > > Same holds for OPEN-O and OpenBaton > > Hongbo and Chris want to start form IPv6 overlay testing and Bin > suggested we reuse most of what we have from Service VM IPv6 testing > > Mathew stated he has started on it and can help > > Testsuya Nakamura chimed he can help in adding some specifc IPv6 related > to it for test case structure > > Tetsuya says we need a clear definition for SUT IPv6, SFC or L3VPN > before we start test plans for IPv6 > > What we are starting with can be IPv6 to establish test plan, test > design and test case documentaion templates > > Prioritizing the test cases can be taken at next meeting and define a > link to the same > > #action Hongbo to establish a link and bring use cases to bring to table > for disucssing priority > > further discussions over use case can be over email > > > > 07:51] > Minutes: > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.html > > [07:51] Minutes (text): > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.txt > > [07:51] > Log: > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.log.html > > > > > > > > Note People who attended the Gotomeeting are not listed here in link, > please add them when Hongbo posts the summary to Dovetail wiki. > > The include > > Chris Price >
Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes
Hi all: Thanks for the deep suggestion and discussion. The Dovetail and C have discussed many times about some of the questions. I will add the related slides and works on the I created a wiki page for these questions discussions. It will help others who missed the discussions. If you have any suggestion and ideas about the dovetail certification, welcome to add on the wikipage. https://wiki.opnfv.org/display/dovetail/high+level+topics I add two more topics on the wiki page: 1) scripts managements Since the dovetail will deliverable the test cases and scripts. The scripts are used to validate the testcases and may be related to the test projects. 2) how to choose the testcases(test cases criteria) This one may be added in the the test cases discussion wiki page. for more details on test cases, please refer to : https://wiki.opnfv.org/pages/viewpage.action?pageId=6827269 I will add the related slides and works on the wiki. Best Regards hongbo -Original Message- From: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Dave Neary Sent: 2016年8月9日 3:17 To: SULLIVAN, BRYAN L; Prakash Ramchandran; opnfv-tech-discuss@lists.opnfv.org Subject: Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes Hi Bryan, On 08/08/2016 10:45 AM, SULLIVAN, BRYAN L wrote: > I missed the last Dovetail meeting but from what I heard about some of > the discussion, I’d like to seek clarification on some things that > might have been expressed in the meeting, e.g. that implementations > which will go thru C may not be expected to be compatible with > existing OPNFV test suites, or at least that not all of the OPNFV test > suites, e.g. > FuncTest and Yardstick, would be expected to be tested on an certified > implementations. > > First I’d like to verify that such opinions were expressed (e.g. per > Bin’s comment that as a result “Dovetail testing and OPNFV tests are > different”), and have them further explained if possible. Yes, I expressed the view that Functest and Yardstick, as testing projects, were designed to stretch the platform - that we will periodically add failing functest tests to validate that the tests pass after we make a change to the upstream projects under test. They will also include tests which are targeting specific scenarios, and are known to fail (and thus will not be run) on other scenarios. In that sense, the Dovetail test suite should be a subset of FuncTest and Yardstick which pass on multiple scenarios and installers, and can be run unchanged on stacks with (for example) a proprietary SDN controller. > Second , given that the notes do capture the discussion and concerns > correctly, here are some thoughts about that: > > 1) The C committee is responsible for setting the “what is > expected” out of a certification, within some flexibility within > Doevtail as to what/how/when that can be delivered. > > 2) Overall, it’s expected that any implementation is compatible > with the OPNFV test suites as test frameworks. The degree of > compatibility with specific test may be limited e.g. if the target > hardware/software function focused on by the tests is not supported by > the implementation (e.g. an implementation that supports only a > specific SDNC), but the significance of such N/As needs to be > carefully considered by the C committee or Dovetail. This is the context for the "subset of..." I mentioned above. Chris started a page in the wiki a couple of weeks ago to describe the criteria for a test case in the Dovetail test framework, I and others added to it: https://wiki.opnfv.org/pages/viewpage.action?pageId=6827269 Thanks, Dave. > Overall we need to ensure that any aspect of certification can work > equally (same end result) on an OPNFV-based implementation (meaning a > collection of the core components as released in some OPNFV release, > or even with slight variations e.g. different versions of the > components), or another implementation. We should not leave tests out > of the Dovetail scope just because they will “not work” on some > implementations. There may be good reasons for them not to work (the > N/As), but if those reasons are simply based upon the test design, the > platform vendor should provide a compatible version of the tests based > upon the OPNFV tests, so that we can still certify the platform > functionally. Examples of this may be: > > - FuncTest > > o vIMS (Clearwater IMS) is based upon Orange’s implementation of the > Cloudify blueprint, using Cloudify as a VNFM. In the process, the > Cloudify Manager is installed as a VM under OpenStack, and then > executes the vIMS blueprint. I see no reason that this should not work > essentially the same in any other environment. AF
Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes
ant they need to be part of the base test system. These might > include tests for reference VNFs that we collect and run as blueprints > under various VNFMs, e.g. thru the Models project. In those cases, if a > vendor does not support one of the VNFMs for some reason (as with > vIMS/Cloudily), then they need to contribute the support using their > VNFM to OPNFV. > > - The rest of the Dovetail tests will be based upon existing > upstream test suites including certification suites such as RefStack. We > need to be proactively reaching out to these upstream teams, e.g. per > https://etherpad.openstack.org/p/interop-challenge-meeting-2016-08-03. > > > > > > Thanks, > > Bryan Sullivan | AT > > > > *From:* opnfv-tech-discuss-boun...@lists.opnfv.org > [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] *On Behalf Of > *Prakash Ramchandran > *Sent:* Friday, August 05, 2016 8:08 AM > *To:* opnfv-tech-discuss@lists.opnfv.org > *Subject:* [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes > > > > Here is today's OPNFV Dovetail meeting notes based on Gotomeeting and > #opnfv-dovetail channels ... > > Agenda: > > 1)start point(L3VPN, SFC and IPV6) > > 2)test cases structures > > 3) additional basic testcases( for SDN controller and NFVI…) > > https://wiki.opnfv.org/pages/viewpage.action?pageId=6827269 > > 4)other issues > > > > For more details, please refer to: > https://wiki.opnfv.org/display/dovetail/Dovetail+Home > > > > Summary Notes of discussions: > > > > Chris defines what should be content of Dovetail > > should focus on requirements and not project and release anagement > > info test will be on SUT that includes OPNFV VIM + NFVi as shown in link > > The details to follow the test plan > > Bin says we should first verify Functest and Yardstick before we start > on Dovetail > > Dave & Chris say they should be standlone and as a subset may be needed > > Purpose of the Dovetail here is to show what is needed for OPNF view of > NFV compliance > > Bin sees a potential issue here that Dovtail testing and OPNFV tests are > different > > CORD example may be able to be claim OPNFV compatibility through > Dovetail but not from OPNFV Platform testing in Projects and Releases > > Bin says Bryan wants CORD to run over OPNFV platform and Chris says > its different as Release testing is diffrent from Dovetail testing > > Same holds for OPEN-O and OpenBaton > > Hongbo and Chris want to start form IPv6 overlay testing and Bin > suggested we reuse most of what we have from Service VM IPv6 testing > > Mathew stated he has started on it and can help > > Testsuya Nakamura chimed he can help in adding some specifc IPv6 related > to it for test case structure > > Tetsuya says we need a clear definition for SUT IPv6, SFC or L3VPN > before we start test plans for IPv6 > > What we are starting with can be IPv6 to establish test plan, test > design and test case documentaion templates > > Prioritizing the test cases can be taken at next meeting and define a > link to the same > > #action Hongbo to establish a link and bring use cases to bring to table > for disucssing priority > > further discussions over use case can be over email > > > > 07:51] > Minutes: > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.html > > [07:51] Minutes (text): > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.txt > > [07:51] > Log: > http://ircbot.wl.linuxfoundation.org/meetings/opnfv-dovetail/2016/opnfv-dovetail.2016-08-05-14.05.log.html > > > > > > > > Note People who attended the Gotomeeting are not listed here in link, > please add them when Hongbo posts the summary to Dovetail wiki. > > The include > > Chris Price > > Dave Neary > > Bi Hu > > Tetsuya Nakmura > > > > > > > > *Prakash Ramchandran* > > logo_huawei* R USA* > > *FutureWei Technologies, Inc* > > Email:prakash.ramchand...@huawei.com <mailto:s.c...@huawei.com> > > Work: +1 (408) 330-5489 > > Mobile:+1 (408) 406-5810 > > 2330 Central Expy, Santa Clara, CA 95050, USA > > > > / / > > / / > > > > > > > > ___ > opnfv-tech-discuss mailing list > opnfv-tech-discuss@lists.opnfv.org > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > -- Dave Neary - NFV/SDN Community Strategy Open Source and Standards, Red Hat - http://community.redhat.com Ph: +1-978-399-2182 / Cell: +1-978-799-3338 ___ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes
Hi Bryan, There was a fair amount of discussion around what Dovetail is testing. One item I was trying to articulate is that the purpose of the Dovetail suite is to validate the compliance of a platform to the behaviors and interfaces targeted for compliance validation in OPNFV. This differs from the purpose of Functest and Yardstick which is to validate as thoroughly as possible the design of the OPNFV platform. I have one small concern with your comment: ”The rest of FuncTest is pretty generic“ Functest is anything but generic as far as a platform for generic platform feature validation is concerned, functest test cases derive mostly from the upstream community and do not, by design, provide neutral generic test cases for a platform. This is not a bad thing, it simply does not lend itself to creating test cases that should be able to be run over any platform composition. Our Dovetail suite needs to work even when someone swaps out OpenStack for another VIM, or a completely different SDN controller for instance. / Chris From: <opnfv-tech-discuss-boun...@lists.opnfv.org> on behalf of "SULLIVAN, BRYAN L" <bs3...@att.com> Date: Monday 8 August 2016 at 16:45 To: Prakash Ramchandran <prakash.ramchand...@huawei.com>, TECH-DISCUSS OPNFV <opnfv-tech-discuss@lists.opnfv.org> Subject: Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes Hi all, I missed the last Dovetail meeting but from what I heard about some of the discussion, I’d like to seek clarification on some things that might have been expressed in the meeting, e.g. that implementations which will go thru C may not be expected to be compatible with existing OPNFV test suites, or at least that not all of the OPNFV test suites, e.g. FuncTest and Yardstick, would be expected to be tested on an certified implementations. First I’d like to verify that such opinions were expressed (e.g. per Bin’s comment that as a result “Dovetail testing and OPNFV tests are different”), and have them further explained if possible. Second, given that the notes do capture the discussion and concerns correctly, here are some thoughts about that: 1) The C committee is responsible for setting the “what is expected” out of a certification, within some flexibility within Doevtail as to what/how/when that can be delivered. 2) Overall, it’s expected that any implementation is compatible with the OPNFV test suites as test frameworks. The degree of compatibility with specific test may be limited e.g. if the target hardware/software function focused on by the tests is not supported by the implementation (e.g. an implementation that supports only a specific SDNC), but the significance of such N/As needs to be carefully considered by the C committee or Dovetail. Overall we need to ensure that any aspect of certification can work equally (same end result) on an OPNFV-based implementation (meaning a collection of the core components as released in some OPNFV release, or even with slight variations e.g. different versions of the components), or another implementation. We should not leave tests out of the Dovetail scope just because they will “not work” on some implementations. There may be good reasons for them not to work (the N/As), but if those reasons are simply based upon the test design, the platform vendor should provide a compatible version of the tests based upon the OPNFV tests, so that we can still certify the platform functionally. Examples of this may be: - FuncTest ovIMS (Clearwater IMS) is based upon Orange’s implementation of the Cloudify blueprint, using Cloudify as a VNFM. In the process, the Cloudify Manager is installed as a VM under OpenStack, and then executes the vIMS blueprint. I see no reason that this should not work essentially the same in any other environment. AFAIK, the only possible differences, which would need to be addressed by adding options to the existing FuncTest code for this, are that e.g. a different approach to kicking off the FuncTest framework is needed due to differences in Jumphost OS or configuration. But OPNFV should not be responsible for accommodating platform implementations that vary in this way; the vendor should step up and implement the support so their product can be validated. oThe rest of FuncTest is pretty generic and I see no reason why it should not be supportable. - Yardstick oAs with FuncTest, the framework under which Yardstick operates may need some tweaks for compatibility with the vendor implementation. These tweaks need to be contributed to OPNFV by the vendor. oThe C program may not initially include performance benchmarking, but any implementation should have demonstrated compatibility with the Yardstick test suite. - Other tests that we develop for Dovetail may go beyond FuncTest and Yardstick, to focus on more compl
Re: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes
Hi all, I missed the last Dovetail meeting but from what I heard about some of the discussion, I'd like to seek clarification on some things that might have been expressed in the meeting, e.g. that implementations which will go thru C may not be expected to be compatible with existing OPNFV test suites, or at least that not all of the OPNFV test suites, e.g. FuncTest and Yardstick, would be expected to be tested on an certified implementations. First I'd like to verify that such opinions were expressed (e.g. per Bin's comment that as a result "Dovetail testing and OPNFV tests are different"), and have them further explained if possible. Second, given that the notes do capture the discussion and concerns correctly, here are some thoughts about that: 1) The C committee is responsible for setting the "what is expected" out of a certification, within some flexibility within Doevtail as to what/how/when that can be delivered. 2) Overall, it's expected that any implementation is compatible with the OPNFV test suites as test frameworks. The degree of compatibility with specific test may be limited e.g. if the target hardware/software function focused on by the tests is not supported by the implementation (e.g. an implementation that supports only a specific SDNC), but the significance of such N/As needs to be carefully considered by the C committee or Dovetail. Overall we need to ensure that any aspect of certification can work equally (same end result) on an OPNFV-based implementation (meaning a collection of the core components as released in some OPNFV release, or even with slight variations e.g. different versions of the components), or another implementation. We should not leave tests out of the Dovetail scope just because they will "not work" on some implementations. There may be good reasons for them not to work (the N/As), but if those reasons are simply based upon the test design, the platform vendor should provide a compatible version of the tests based upon the OPNFV tests, so that we can still certify the platform functionally. Examples of this may be: - FuncTest o vIMS (Clearwater IMS) is based upon Orange's implementation of the Cloudify blueprint, using Cloudify as a VNFM. In the process, the Cloudify Manager is installed as a VM under OpenStack, and then executes the vIMS blueprint. I see no reason that this should not work essentially the same in any other environment. AFAIK, the only possible differences, which would need to be addressed by adding options to the existing FuncTest code for this, are that e.g. a different approach to kicking off the FuncTest framework is needed due to differences in Jumphost OS or configuration. But OPNFV should not be responsible for accommodating platform implementations that vary in this way; the vendor should step up and implement the support so their product can be validated. o The rest of FuncTest is pretty generic and I see no reason why it should not be supportable. - Yardstick o As with FuncTest, the framework under which Yardstick operates may need some tweaks for compatibility with the vendor implementation. These tweaks need to be contributed to OPNFV by the vendor. o The C program may not initially include performance benchmarking, but any implementation should have demonstrated compatibility with the Yardstick test suite. - Other tests that we develop for Dovetail may go beyond FuncTest and Yardstick, to focus on more complex use cases or specific technical capabilities. In principle I would expect that these would migrate into FuncTest and Yardstick however, over time, because if they are important they need to be part of the base test system. These might include tests for reference VNFs that we collect and run as blueprints under various VNFMs, e.g. thru the Models project. In those cases, if a vendor does not support one of the VNFMs for some reason (as with vIMS/Cloudily), then they need to contribute the support using their VNFM to OPNFV. - The rest of the Dovetail tests will be based upon existing upstream test suites including certification suites such as RefStack. We need to be proactively reaching out to these upstream teams, e.g. per https://etherpad.openstack.org/p/interop-challenge-meeting-2016-08-03. Thanks, Bryan Sullivan | AT From: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Prakash Ramchandran Sent: Friday, August 05, 2016 8:08 AM To: opnfv-tech-discuss@lists.opnfv.org Subject: [opnfv-tech-discuss] [Dovetail] Aug 5 Dovetail meeting notes Here is today's OPNFV Dovetail meeting notes based on Gotomeeting and #opnfv-dovetail channels ... Agenda: 1)start point(L3VPN, SFC and IPV6) 2)test cases structures 3) additional basic testcases( for SDN controller and NFVI...) https://wiki.opnfv.org/pages/viewpage.action
