I forgot to mention two things. > When you add Gerrit comment "perftest"
We can make the job to run on every uploaded patch (if not Draft) when it is deemed reliable enough. > three main areas for improvements: Longer term goal is to add some logic to select testcases based on directories edited. The hard part is creating a mapping from directories to test tags. Vratko. From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco) via Lists.Fd.Io Sent: Friday, 2018-September-07 15:04 To: vpp-dev@lists.fd.io Cc: vpp-dev@lists.fd.io Subject: [vpp-dev] new per-patch job Hello VPP devs. There is a new way of testing performance impact of a VPP Change before merging it. When you add Gerrit comment "perftest" a job will start (different from the previously available one). It will compile both the code in the Change and in its parent, run few tests against both builds, and vote -1 if a regression is detected. The job takes around one hour to finish (similar to virl job), mostly spent by compiling. The job compares results of 4 tests by default: ip4base, ip6base, l2bdbasemaclrn and l2xcbase. The job supports tag selection (the same way as the previous job). The job is using 2-node skylake testbeds, so some tests are not available (e.g. the ones related to encapsulation). The regressions are detected using the same algorithm as in trending graphs, but the result is not 100% reliable. Sometimes, even running the same build twice shows statistically significant difference in performance. Going forward, there are three main areas for improvements: 1. Add more tests (to the set executed by default). This would find more kinds of performance regressions. 2. Make the job run faster. Some kind of artifact caching could be implemented, possibly using packagecloud. 3. Make the vote more reliable. Either by identifying and fixing the current causes of performance differences, or at least by working around them (somehow). So please, give the new job a try, and tell us which area of improvement to focus on. Vratko (and CSIT team).
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10429): https://lists.fd.io/g/vpp-dev/message/10429 Mute This Topic: https://lists.fd.io/mt/25306496/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
