Hi Damjan, Moving the files has tradeoffs. The relative value depends on the user's workflow. Most tests have their own launcher at the bottom of the test_<foo> file and at some point were able to run outside of the Makefile/run_tests. Relocating the files essentially breaks that, but if it is not used by anyone, that could be ok. We just need to clean up the code to remove those remnants.
If we were to clean up and package framework.py and related files as a python module, then the tests could be run from anywhere in the file system. If papi and the test framework were separated out of the repo, building tests and applications would become 50x easier. Why? Because python would handle the files and their locations/namespaces within the python venv. It would look something like: $ cd <wherever_test_foo_lives> $ python3 -m pip install . installing <test_dependencies> installing <framework> installing vpp_papi $ python3 ./test_<foo> None of these are insurmountable issues. We have to agree, and we have to agree to not punt on the associated hygiene. I agree with Florin, I don't have a strong opinion as to where the files reside. Unless, we can agree to move the tests to a separate repository, then I'd prefer to have them all in a single directory. Why? Because it would be trivial to run the stable release's tests against vpp master and know what code I need to update across versions. While we are on the subject of tests, I would like to socialize a few ideas for improvement. 1. Can we stop building packets in the test files? Yes, I know we need packets to test against, but can we just have the test file pass an associated pcap file, since that is what we send to the pg to run the test. A side effect of this is that the need for scapy in the tests is reduced. There is a video somewhere where all the different methods to sum up all the numbers from 1..1_000_000 are compared in python and the final statement of the video was to just use n(n + 1) / 2. 2. Can we make the testing packets more useful? We typically take a single packet and replicate it a number of times to ensure that the packet takes each path in the node. Additionally, the packet value choices aren't always useful. When the source port == the dest port, for example, typos in the code under test can be missed. Even better would be to increment ephemeral port values. 3. I would like to see vpp add a stride width value (1,2,4,8) to a node's registration info that was accessible to the test via an api, even if the api were only in the unit test plugin. This way, the tests can calculate the optimal packet stream characteristics, or at the very least warn that a given test doesn't sufficiently test the component. Paul On Fri, Mar 26, 2021 at 3:15 PM hemant via lists.fd.io <hemant= mnkcg....@lists.fd.io> wrote: > Damjan, > > Thanks for clarifying - I agree with you on vnet tests. > > Hemant > > -----Original Message----- > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Damjan > Marion via lists.fd.io > Sent: Friday, March 26, 2021 12:38 PM > To: Dave Wallace <dwallac...@gmail.com> > Cc: vpp-dev <vpp-dev@lists.fd.io> > Subject: Re: [vpp-dev] keeping tests outside of src/ > > > > > On 25.03.2021., at 21:14, Dave Wallace <dwallac...@gmail.com> wrote: > > > > Hi Damjan, > > > > This initiative originated with the wider adoption of plugin development > at the request of Dave Barach to allow the development of plugins outside > the VPP repo. After completing the job for plugins, there were several > requests to extend that to all of the features. Presumably this was > coupled with the desire to migrate feature source code from vnet into the > plugin arena, but I don't recall all of the details of the discussion. > > OK, there are 2 different things. One is testing of out-of-tree plugins, > another one is VPP tree layout. > I fully agree that we need to support testing of out-of-tree plugins. Than > can be fixed as simple as ‘make test > TEST_DIR=/path/to/out-of-tree-plugin/test’. > > I don’t see how those two things relate. > > > > > Unfortunately, this effort stalled across several releases due to lack > of cycles and I'm just now in the process of completing the job. > > > > I'm perfectly ok accepting a -2 for test code that maintainers prefer to > leave in .../vpp/test, but I don't see the original requirement to > co-locate plugin source & test code going away. So the majority of the > feature source & test code will remain structured that way and the end > result will be inconsistent at best. > > I’m not trying ot say we should -2, and I know that you submitted those > patches believing that this is the right thing to do. I am just under > impression that we are all not on the same page what is right thing to do. > > > > > > Personally, I think that it makes sense to continue to move features > source, test code, and documentation to be co-located in a modular and > consistent sub-tree structure. I also see value in migrating features out > of vnet into the plugin sub-tree. > > I disagree here, I believe it should stay separate. But this is just my > opinion, I’m fine to be minority here, i just would like to know that we > are all on the same page and that whatever we decide we decided with good > understanding of implications. > > Implications may be: > - licensing implications like the current saga with scapy > - deciding if CMake should install test infra as part of vpp-dev packaging > - dealing with tests which cover multiple source code components or infra > > Mechanical move of file is the easiest part. As currently src/ is > currently one entity handled by CMake, throwing tests in without test infra > being part of src/ looks to me very broken. > > > > > > For what its worth, the changes to test/Makefile gather all of the > source as soft links into the build tree > (.../vpp/build-root/build-test/src), but I understand that is not the same > your original plan. > > > > Thanks, > > -daw- > > > > On 3/25/2021 3:16 PM, Damjan Marion via lists.fd.io wrote: > >> Hi, > >> > >> It may be that it is not discussed or i was just ignorant, but I > noticed that there is ongoing activity to scatter tests all across the src/. > >> When I started "make test" long long time ago i intentionally put it > >> to separate tree following the pattern from other projects and to be > honest it makes me more sense that all tests are contained in the separate > tree. > >> > >> Are we sure that this test file scatter activity is right thing to do? > >> Anyone aware of any other project doing the same? > >> > >> — > >> Damjan > >> > >> > >> > >> > >> > >> > >> > > > > > > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19050): https://lists.fd.io/g/vpp-dev/message/19050 Mute This Topic: https://lists.fd.io/mt/81611239/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
