A couple of thoughts that came to my mind after I wrote this. See below. On Thursday, December 5, 2019 at 6:22:50 PM UTC-5, Waldek Kozaczuk wrote: > > Once I commit the recent patch from Mason Davis, we will have a simple > CI/CD pipeline creating a fresh docker image and testing OSv build. This a > very first and fundamental step but there are still many that need to > follow to better test OSv. > > So before we identify other steps we should first agree on what the end > goal should be and how it should work at a high level. Besides the obvious > goal of automating the build and test process, I would also want to make it > easy for anyone to reproduce each build and more importantly each test run > (especially when it fails) WITHOUT having to re-build most of the > artifacts. This becomes important once the pipeline runs not only the unit > tests but many other test apps (see the apps tested by this script - > https://github.com/cloudius-systems/osv/blob/master/scripts/tests/compose_and_test_selected_apps.sh). > > Building each test app from scratch every time is expensive and unnecessary > with each OSv commit especially if we want to test the exact same app with > new OSv kernel. So at a high level every time we build OSv artifacts [*step > 1, "build" phase*], ideally, we should publish its artifacts somewhere > (?) and then run the tests [*step 2, "run" phase]* including both unit > tests and test apps using the artifacts produced in step 1 and publish the > results. This separation allows us to run the tests (which should be much > faster and more repeatable) anywhere we want and as many times as we want. > Another key tool that would glue both phases should be *capstan* ( > https://github.com/cloudius-systems/capstan#capstan) which would be used > to create so-called *MPM packages* in the *build* phase and used to > rapidly compose test images in the *run* phase. Which is what I roughly > put in > https://github.com/cloudius-systems/osv/issues/1044#issuecomment-556547062 > . > > Also running the tests on Travis most likely will use test host without > KVM acceleration which will run very sloooow (some unit tests take 10 > minutes or more vs seconds without KVM). So another motivation is that > hopefully, ScyllaDB will continue to let us run OSv tests which should take > much less time and require much less intervention on their side (no need to > mess with the building part where manual intervention is sometimes > required). Alternatively is there a place where we can run for free with > KVM acceleration on? This would require bare-metal machine or AWS i3 metal > instance or other cloud environments with nested acceleration available (I > think that GCE might support it). > > Let me know what you think about. > > So if we agree with my proposed objective and high-level design, then here > is the suggested list of steps we should execute: > > 1. Identify where OSv artifacts should be published. > - I suggest pushing to github OSv releases assets that we use to > publish releases artifacts. The problem is that it has a flat structure > plus which release would those artifacts belong to? Maybe we could have > fake 'latest' release where we would push artifacts prefixed or > postfixed > with the commit ID. I think this discussion on github community forum > might > describe what we want - > > https://github.community/t5/How-to-use-Git-and-GitHub/Nightly-Builds-Latest-Branch-Releases/m-p/5000/highlight/true#M1621 > 2. Create generic publish script > - if we publish to github it would use github REST api. The script > should be generic and let you publish any file to a specified release > tag, > It would probably use > https://developer.github.com/v3/repos/releases/#upload-a-release-asset > 3. Identify which OSv artifacts should be built with every commit and > where published. > - I suggest the following artifacts from > > https://github.com/cloudius-systems/osv/blob/master/scripts/build-capstan-mpm-packages > (for > starters): > > I mentioned "for starters" but in reality, we would indeed build very few artifacts - kernel and any modules (./modules directory, for example tests, run-java, httpserver-api, etc) that depend of source files in OSv tree and might change by a commit.
> > - build_osv_image_loader_and_bootstrap_package > - build_unit_tests_package > - build_httpserver_api_package > - build_http_server_tests_package > 1. Create another docker file that would create "run" image. > - For starters, it would only need QEMU, capstan and shallow clone > of OSv repo just to get scripts subdirectory. > - It would use capstan to compose test images out of packages > published in the "build" phase and run them (see > > https://github.com/cloudius-systems/osv/blob/master/scripts/tests/compose_and_test_selected_apps.sh > and > https://github.com/cloudius-systems/osv/wiki/Automated-Testing-Framework > ) > - for starters, after test image is composed we could just run > scripts/test.py to only run unit tests; later we will expand > - Eventually, it should publish test results somewhere (as a file > to the same github "latest" assets?) > - It should be also possible to do all that without docker > > In theory, we might use the same "build" docker image but given how "heavy" it is (currently in my estimates at ~2GB, eventually after some cleanup it should be smaller, but never smaller than 1GB), pulling it every time just to run the tests would be overkill if we run the tests somewhere else. By my estimates, the "run" image should be much lighter at ~100MB. > I may have missed something but I think this is how I see CI/CD pipeline > work. Feel free to criticize it and suggest improvements to do it in a > better way. > > Regards, > Waldek > > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/83a30e8f-5208-4fc6-a217-5631145ba08a%40googlegroups.com.
