As promised: https://issues.apache.org/jira/browse/BIGTOP-4058
I started adding some jobs to show the idea, feedback welcome :) Luca On Thu, Jan 11, 2024 at 3:56 PM Luca Toscano <[email protected]> wrote: > > Nevermind I fixed the apt issue, but the current problem is that the > COMPONENTS of the smoke test cannot be grouped together. IIUC we want > to be able to kick off a run with COMPONENTS set to all or a subset of > daemons (yarn/hdfs/etc..), that will bootstrap the hadoop cluster via > Docker. I'll open a task and work on it, hopefully there is an easy > solution :) > > On Thu, Jan 11, 2024 at 12:11 PM Luca Toscano <[email protected]> wrote: > > > > I have also created: https://ci.bigtop.apache.org/job/pipeline-smoke-test/ > > > > It currently doesn't work since there seems to be an error when apt > > runs inside the Docker container, but it is a generic script to give > > you the idea. Release version and branch have a separate parameter, at > > the moment we have 3.2.0 and 3.3.0, so a single script could in theory > > be used multiple times with minimal changes (IIUC at the moment for > > every release we create new smoke tests etc..).\ > > > > To test it: > > > > 1) Run build > > 2) Open the build run's "Console output" > > 3) Hit "Input Requested" and you'll see a series of dropdown options. > > Once you select the right ones, the build should start. > > > > If you want to see an example: > > https://ci.bigtop.apache.org/job/pipeline-smoke-test/5/console > > > > This is just an idea to gather feedback, it can surely be improved, > > but it is a starting point :) > > > > Thanks! > > > > Luca > > > > On Wed, Jan 10, 2024 at 9:44 PM Luca Toscano <[email protected]> wrote: > > > > > > Hi folks, > > > > > > I created https://ci.bigtop.apache.org/job/test-elukey as an example > > > of how Jenkins pipelines can be used to replace the Matrix Reloaded > > > plugin. The prototype is limited for the moment to: > > > - It uses scripted pipelines, not declarative pipelines, since the > > > former is more flexible in my opinion. > > > - It mimics the Smoke Tests for Debian OSes, used in our release > > > process IIUC (like > > > https://ci.bigtop.apache.org/view/3.2.0-smoke-tests/job/Bigtop-3.2.0-debian-10-smoke-tests/1/) > > > - It only echoes values, no build is done. > > > - It allows dynamic parameters (namely to restrict OS/ARCH/COMPONENT > > > at runtime), but the inputs need to be added in the build's console > > > output rather than having specific "Build with params" in Jenkins UI. > > > This is probably something that we can circumvent, but the current > > > solution avoids to copy/paste values multiple times etc.. > > > - Compared to Matrix reloaded, the flexibility of selecting dynamic > > > parameters is less, but it should be sufficient for the release use > > > case. > > > > > > We'd need to deprecate Matrix Reloaded > > > (https://plugins.jenkins.io/matrix-reloaded/) since the plugin was > > > abandoned since a long time ago, and no more fixes are provided from > > > upstream. > > > > > > Do you think that this process is viable? If so we could use pipelines > > > as experimental step during the next release, to see if they can be > > > adopted. Let me know your thoughts and if I am missing something big > > > (probably happening, apologies in advance). > > > > > > Thanks! > > > > > > Luca > > > > > > On Sun, Jan 7, 2024 at 3:38 PM Luca Toscano <[email protected]> > > > wrote: > > > > > > > > Hi folks, > > > > > > > > just letting you know that I have installed the Pipelines plugin in > > > > the Jenkins' master, to test if we can define our jobs in there and > > > > avoid the Matrix reloaded plugin. It would be also very nice to have > > > > all pipelines checked out in the BigTop's repo eventually. > > > > > > > > I'll open a jira to track all the work after some quick tests :) > > > > > > > > Let me know if you see anything weird, I've just restarted Jenkins to > > > > pick up the new plugin. > > > > > > > > Luca
