I need to build a parallel integration testing. I start from same source (same git repository same tag). I have several PCs (each one with agent running NOT as a service). Each PC must compile and launch a .exe with a command line parameter (that I take from environment) different for each test. So I supposed to use a pipeline: I start it via API many times, then the pipeline on each PC compile and launch the .exe , it connects to server, dowloads test, runs test and stop. In this way it is very simple: many tests are launched, they fills agents, I can scale horizontally. So please tell me the right approach to do this. Thanks, Mario
Il giorno martedì 1 marzo 2022 alle 16:26:54 UTC+1 Chad Wilson ha scritto: > Hi Mario > > That GitHub ticket is specifically about the case where you want to be > building in parallel based on different *material versions* i.e *inputs* > to the pipeline. (e.g developer pushes to a git material 3x, say 30s apart > - can all 3 builds start in parallel?) > > If you are *not* trying to build based on different material versions in > parallel and instead want to vary other aspects (env vars, environments, > parameters etc) for the *same material versions*, even without running > the same *pipeline *in parallel as in that ticket, there are different > ways to approach parallel runs in GoCD. > > - separate *jobs* within the same stage of a single pipeline that > override env vars differently at job scope, but execute the same tasks in > each job > - separate *pipelines* entirely, that have the same material > inputs/triggers, but have different configuration for the stage/job/tasks > > The best way to model it depends on what is inside your existing pipeline > (multiple stages? multiple jobs within stages?) and also what you want to > do as a result of completion: > > - trigger a downstream pipeline only once all 'n' parallel runs > complete? - jobs might work. > - trigger another pipeline/stage as soon as any single one of the > parallel jobs complete and then trigger again for each parallel run? - > pipelines might be a better approach. > > You'd probably need to share more context on how you want things to behave > *after* your parallel runs and the content of your existing pipeline to > suggest the best approach. > > There is some config overhead/duplication, but you can generally mitigate > duplication using either > > - pipeline templates > - *or* one of the config repo plugins to drive pipeline config from > source control > - https://github.com/tomzo/gocd-yaml-config-plugin (bundled) > - using YAML anchors for reusable stage/job/tasks > - https://github.com/gocd-contrib/gocd-groovy-dsl-config-plugin > - using scripting for reusable stage/job/tasks > > -Chad > > On Tue, Mar 1, 2022 at 10:43 PM Mario Giammarco <[email protected]> > wrote: > >> Sorry I need to change enviroment variables not parameters. >> >> Il giorno martedì 1 marzo 2022 alle 15:38:09 UTC+1 Mario Giammarco ha >> scritto: >> >>> Hi, >>> I resurrect this thread. >>> I have one pipeline and several agents. I need to run the pipeline >>> multiple times in parallel with different parameters. >>> It seems to me a common and easy question. >>> Reading this thread, bug description and other threads it seems it is >>> not possible. >>> I am really surprised about it. >>> Is it really not possible? >>> Is there a workaround? >>> Thanks, >>> Mario >>> >>> Il giorno lunedì 3 ottobre 2016 alle 08:00:08 UTC+2 Zabil C M ha scritto: >>> >>>> There's an issue logged for this here >>>> https://github.com/gocd/gocd/issues/2329 >>>> >>>> In short, you can't do this at the moment but we are willing to help >>>> out with and merge this feature if someone picks it up. >>>> >>>> On Wed, Sep 28, 2016 at 3:48 PM, <[email protected]> wrote: >>>> >>>>> One of the key features that made us decide to migrate to gocd several >>>>> months ago was the idea that pipelines are able to run in parallel and >>>>> that >>>>> more than one instance of a pipeline can be started. >>>>> Only now are we starting to realise that actually the same pipeline >>>>> cannot be executed multiple times in parallel since each stage will >>>>> always >>>>> run sequentially even if it's the same stage in different pipeline >>>>> instances. I'm stating this only based on this post >>>>> <https://groups.google.com/d/msg/go-cd/E0goLJITLy8/8UN7KrjCCAAJ> which >>>>> is the only mention of this issue that I've been able to find. >>>>> >>>>> So first off I'd like to verify if this is correct. Is it not possible >>>>> to run several instances of the same pipeline concurrently (in parallel >>>>> )without one instance being constrained by another (i.e. the second >>>>> instance of the same pipeline is able to complete even if a previously >>>>> initiated instance has not done so)? >>>>> >>>>> If this is the case, I'd really appreciate any help/ideas on >>>>> overcoming this limitation in some way. >>>>> The pipeline I'm working on has only one stage which simply runs a >>>>> docker and then deletes the container and image. The docker does some >>>>> work >>>>> on our machine learning models, there is no problem to run several >>>>> containers of this docker at the same time - and that's exactly what I'd >>>>> like to do - i.e. run another docker each time that the pipeline is >>>>> triggered (we're using the api to trigger it). >>>>> I'd like to see the output created by each docker and, of course, see >>>>> whether each pipeline has finished successfully or failed. The order of >>>>> execution and even the material version is not relevant, each instance of >>>>> the pipeline/docker has it's job to do. >>>>> >>>>> Finally, I'd like to ask if you guys feel that a feature that enables >>>>> the same stage to run concurrently in different pipeline instances is >>>>> feasible. And that a request for this is something which has a chance of >>>>> being accepted (I will, of course, contribute anything I can within my >>>>> technical skills) . >>>>> I'm sure that the are many use cases that will benefit from such a >>>>> feature. >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "go-cd" group. >>>>> >>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>> >>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/b9f673a6-01ba-4950-9d0e-9cafc2151014n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/go-cd/b9f673a6-01ba-4950-9d0e-9cafc2151014n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/b664a66e-c079-4090-abd2-757e62ce6b86n%40googlegroups.com.
