Hi folks, getting back to the subject if anyone is interested in giving some feedback. In https://issues.apache.org/jira/browse/BIGTOP-4058 there are some examples of Jenkins pipelines that I have created to see if we can get rid of the Matrix reloaded plugin, but before proceeding I'd need to know if it is something that people would use/like or not :) The main use case for Matrix reloaded seems to be (IIUC) having a way to selectively trigger builds and smoke tests while doing a release (to bypass transient failures to specific components that would require to trigger all the builds again). In theory we could do the same with pipelines, but they are a little less flexible for the moment (at least from a UI perspective). Getting rid of the Matrix plugin would avoid some old/unsecure code and it would align us with more recent Jenkins standards.
Lemme know! Thanks in advance, Luca On Mon, Jan 15, 2024 at 12:06 PM Luca Toscano <[email protected]> wrote: > > As promised: https://issues.apache.org/jira/browse/BIGTOP-4058 > > I started adding some jobs to show the idea, feedback welcome :) > > Luca > > On Thu, Jan 11, 2024 at 3:56 PM Luca Toscano <[email protected]> wrote: > > > > Nevermind I fixed the apt issue, but the current problem is that the > > COMPONENTS of the smoke test cannot be grouped together. IIUC we want > > to be able to kick off a run with COMPONENTS set to all or a subset of > > daemons (yarn/hdfs/etc..), that will bootstrap the hadoop cluster via > > Docker. I'll open a task and work on it, hopefully there is an easy > > solution :) > > > > On Thu, Jan 11, 2024 at 12:11 PM Luca Toscano <[email protected]> > > wrote: > > > > > > I have also created: https://ci.bigtop.apache.org/job/pipeline-smoke-test/ > > > > > > It currently doesn't work since there seems to be an error when apt > > > runs inside the Docker container, but it is a generic script to give > > > you the idea. Release version and branch have a separate parameter, at > > > the moment we have 3.2.0 and 3.3.0, so a single script could in theory > > > be used multiple times with minimal changes (IIUC at the moment for > > > every release we create new smoke tests etc..).\ > > > > > > To test it: > > > > > > 1) Run build > > > 2) Open the build run's "Console output" > > > 3) Hit "Input Requested" and you'll see a series of dropdown options. > > > Once you select the right ones, the build should start. > > > > > > If you want to see an example: > > > https://ci.bigtop.apache.org/job/pipeline-smoke-test/5/console > > > > > > This is just an idea to gather feedback, it can surely be improved, > > > but it is a starting point :) > > > > > > Thanks! > > > > > > Luca > > > > > > On Wed, Jan 10, 2024 at 9:44 PM Luca Toscano <[email protected]> > > > wrote: > > > > > > > > Hi folks, > > > > > > > > I created https://ci.bigtop.apache.org/job/test-elukey as an example > > > > of how Jenkins pipelines can be used to replace the Matrix Reloaded > > > > plugin. The prototype is limited for the moment to: > > > > - It uses scripted pipelines, not declarative pipelines, since the > > > > former is more flexible in my opinion. > > > > - It mimics the Smoke Tests for Debian OSes, used in our release > > > > process IIUC (like > > > > https://ci.bigtop.apache.org/view/3.2.0-smoke-tests/job/Bigtop-3.2.0-debian-10-smoke-tests/1/) > > > > - It only echoes values, no build is done. > > > > - It allows dynamic parameters (namely to restrict OS/ARCH/COMPONENT > > > > at runtime), but the inputs need to be added in the build's console > > > > output rather than having specific "Build with params" in Jenkins UI. > > > > This is probably something that we can circumvent, but the current > > > > solution avoids to copy/paste values multiple times etc.. > > > > - Compared to Matrix reloaded, the flexibility of selecting dynamic > > > > parameters is less, but it should be sufficient for the release use > > > > case. > > > > > > > > We'd need to deprecate Matrix Reloaded > > > > (https://plugins.jenkins.io/matrix-reloaded/) since the plugin was > > > > abandoned since a long time ago, and no more fixes are provided from > > > > upstream. > > > > > > > > Do you think that this process is viable? If so we could use pipelines > > > > as experimental step during the next release, to see if they can be > > > > adopted. Let me know your thoughts and if I am missing something big > > > > (probably happening, apologies in advance). > > > > > > > > Thanks! > > > > > > > > Luca > > > > > > > > On Sun, Jan 7, 2024 at 3:38 PM Luca Toscano <[email protected]> > > > > wrote: > > > > > > > > > > Hi folks, > > > > > > > > > > just letting you know that I have installed the Pipelines plugin in > > > > > the Jenkins' master, to test if we can define our jobs in there and > > > > > avoid the Matrix reloaded plugin. It would be also very nice to have > > > > > all pipelines checked out in the BigTop's repo eventually. > > > > > > > > > > I'll open a jira to track all the work after some quick tests :) > > > > > > > > > > Let me know if you see anything weird, I've just restarted Jenkins to > > > > > pick up the new plugin. > > > > > > > > > > Luca
