Between May 5th and May 11th we ran 4 interviews with engineers who have experience using Jenkins as their CI solution. They complemented an additional conversation that happened back in March. None of the people interviewed has experience with Toaster, although at least two of them saw the demo of Toaster at Intel's Yocto Summit in April.
The contents and conclusions of those 5 conversations are summarised below. I will start by the conclusions: if you want to understand those a bit better, you can read the interviews summary provided after them. CONCLUSIONS * Toaster could provide a way to handle the 'artifacts' problem: it could become the build artifact access point instead of a custom application or Jenkins itself. For this to work well, we would need to provide an easy way to specify which artifacts should be collected, and where they should be stored, in a way that avoids the need for plugins or custom scripts, and that includes the option of selecting sstate objects as artifacts to be kept. We would also need to improve how Toaster presents rootfs files and other artifacts for download, which is not great at the moment. * The fact that people recommend keeping Jenkins 'configuration agnostic' as much as possible means that generating configuration in Toaster and passing it onto Jenkins is not a crazy idea. This is effectively what the python and shell scripts mentioned to us seem to be doing. Instead of writing a script, you would generate configuration instructions using Toaster. * Toaster could be particularly useful in contexts where developers must run builds in remote infrastructure. An existing Toaster project with a custom layer added could make the flow as easy as: push my changes to the layer, log in to Toaster, press 'run again' on a previous build for the desired image. * Toaster will provide the added incentive of collecting and displaying extensive information about the builds ran through Jenkins. None of the interviewees were familiar with that Toaster feature, but we have anecdotal evidence from previous interviews that suggests it is useful and valued by Yocto Project users. INTERVIEWS SUMMARY 1. Storing build artifacts Everybody must somehow solve the issue of collecting and saving build artifacts generated by Jenkins. Here is a good explanation of the problem taken from https://wiki.jenkins-ci.org/display/JENKINS/ArtifactDeployer+Plugin "By default, Jenkins archives artifacts generated by the build. These artifacts are stored in the JENKINS_HOME directory with all other elements such as job configuration files. There is no separation between infrastructure elements, project elements and outputs. It is often considered to be a bad practice and it doesn't help to manage JENKINS_HOME directory from an administrator point of view (backup issues, etc). And for now, we can't change the 'Archived artifacts' location." Or, as someone put it to us, "Jenkins by itself does not create a share drive where you can go and grab stuff." People handle the artifacts problem in different ways: * 2 of the people we spoke to tell Jenkins to upload artifacts to a custom tool, which provides its own web interface that users can access and download the artifacts from. One person repurposed the error reporting tool for that. * Another person saves the artifacts to the master Jenkins server with the help of a Jenkins plugin. Developers access the Jenkins UI and download artifacts from it (Jenkins can be tied to a LDAP server, so authentication becomes straightforward). This is a bit more convoluted than it sounds, however. Here is the explanation of the process: "Our shell scripts will make a directory called 'artifacts' at the top level of the Jenkins workspace. Then we move all the different files that BitBake generates and that we care about into that 'artifacts' directory. Then, at the Jenkins job level, we have the artifact archiver plugin set up so that everything under the artifacts directory gets stored with the job as artifacts, and that goes into the Jenkins master server and the lifetime of those is tied to the lifetime of that build in Jenkins, and there is a lot of policy you can set around that. You can say keep the last 10, or keep 2 weeks, and so on. But if you delete the job through the Jenkins interface the artifacts also get deleted." Artifact collection might involve more than rootfs files: people also need to collect logs and in one case, sstate files as well, which are uploaded to an http server so that they can be used by anybody running builds. Using plugins, however, comes with a maintenance overhead: there can be compatibility issues between versions of Jenkins, and they must be tested to make sure they work together and with Jenkins when upgrading to a new version. 2. Keeping Jenkins simple 3 out of the 5 people we spoke to recommended keeping the configuration in Jenkins itself as simple as possible. One of them warned against the huge number of plugins, and the ability to create very complex Jenkins projects (he showed us how you can nest Jenkins projects within other Jenkins projects). Another recommended including the build configuration required in the Git repo Jenkins is building (in this case, the build scripts were stored in a git submodule in a main git "super module"), which makes the maintenance of Jenkins jobs much easier (Jenkins itself is not great at facilitating configuration management for a large number of jobs). Most people used some kind of script to provide to Jenkins the required build configuration: one used Python scripts for a relatively simple Android build; people doing Yocto Project builds used shell scripts. To pass configuration from Toaster to Jenkins people recommended: * The Jenkins REST API * The Jenkins CLI * Use some kind of plugin architecture in Toaster, so that people can write support for their own CI solutions * Export some kind of configuration file that Jenkins can read and apply * Somehow expose the Jenkins UI configuration options within Toaster 3. Software developers run builds People brought up the fact that, in certain contexts, software developers must build images reproducing the configuration used by their distro team (for example, when the hardware they work with doesn't have a writable file system). In such cases, building locally becomes difficult due to the build size, hence giving them access to remote build servers via Jenkins. Making the building workflow as easy as possible for their developers was an important consideration, and 2 of the people we spoke to used Jenkins to facilitate the build process for developers: for example, developers would push their changes to a Git repository, and then would access Jenkins and tell it to build that repository. The nice web UI and the fact that Jenkins provides visibility on the state of the builds make developers' lives easier. 4. Should we integrate Toaster with Jenkins? When asked what they thought about us integrating Toaster with Jenkins, most people didn't see the point. Some thought that if they wanted to interact remotely with Jenkins, they could just use curl. Others that you can just use a script to do anything you want. Only one person thought they should be integrated, mainly because of the configuration ease provided by Toaster. In his own words: "configuring your builds with Toaster looks much easier than doing it in command line". If you have any questions about the above, please let me know. Cheers Belén -- _______________________________________________ toaster mailing list [email protected] https://lists.yoctoproject.org/listinfo/toaster
