It sounds like having multiple jobs with the same tasks in a single stage and single pipeline might do what you need. This is what is done within GoCD itself to achieve parallel integration testing.
If you don't want to have multiple job definitions and manually vary the environment variables between jobs you can consider whether using the job option for "Run multiple instances" is suitable for your case. When you use this option and a stage is triggered, it will start the same job definition in parallel 'n' times with GO_JOB_RUN_COUNT=n and varying GO_JOB_RUN_INDEX=1..n. You can then change your scripting to partition the tests deterministically into buckets depending on the # of jobs in the _COUNT and select partition based on the index. Bit more advanced, but worth considering if you are just trying to split a chunk of work over 'n' agents. -Chad On Wed, Mar 2, 2022 at 6:39 AM Mario Giammarco <[email protected]> wrote: > I need to build a parallel integration testing. > I start from same source (same git repository same tag). > I have several PCs (each one with agent running NOT as a service). Each PC > must compile and launch a .exe with a command line parameter (that I take > from environment) different for each test. > So I supposed to use a pipeline: I start it via API many times, then the > pipeline on each PC compile and launch the .exe , it connects to server, > dowloads test, runs test and stop. > In this way it is very simple: many tests are launched, they fills agents, > I can scale horizontally. > So please tell me the right approach to do this. > Thanks, > Mario > > Il giorno martedì 1 marzo 2022 alle 16:26:54 UTC+1 Chad Wilson ha scritto: > >> Hi Mario >> >> That GitHub ticket is specifically about the case where you want to be >> building in parallel based on different *material versions* i.e *inputs* >> to the pipeline. (e.g developer pushes to a git material 3x, say 30s apart >> - can all 3 builds start in parallel?) >> >> If you are *not* trying to build based on different material versions in >> parallel and instead want to vary other aspects (env vars, environments, >> parameters etc) for the *same material versions*, even without running >> the same *pipeline *in parallel as in that ticket, there are different >> ways to approach parallel runs in GoCD. >> >> - separate *jobs* within the same stage of a single pipeline that >> override env vars differently at job scope, but execute the same tasks in >> each job >> - separate *pipelines* entirely, that have the same material >> inputs/triggers, but have different configuration for the stage/job/tasks >> >> The best way to model it depends on what is inside your existing pipeline >> (multiple stages? multiple jobs within stages?) and also what you want to >> do as a result of completion: >> >> - trigger a downstream pipeline only once all 'n' parallel runs >> complete? - jobs might work. >> - trigger another pipeline/stage as soon as any single one of the >> parallel jobs complete and then trigger again for each parallel run? - >> pipelines might be a better approach. >> >> You'd probably need to share more context on how you want things to >> behave *after* your parallel runs and the content of your existing >> pipeline to suggest the best approach. >> >> There is some config overhead/duplication, but you can generally mitigate >> duplication using either >> >> - pipeline templates >> - *or* one of the config repo plugins to drive pipeline config from >> source control >> - https://github.com/tomzo/gocd-yaml-config-plugin (bundled) >> - using YAML anchors for reusable stage/job/tasks >> - https://github.com/gocd-contrib/gocd-groovy-dsl-config-plugin >> - using scripting for reusable stage/job/tasks >> >> -Chad >> >> On Tue, Mar 1, 2022 at 10:43 PM Mario Giammarco <[email protected]> >> wrote: >> >>> Sorry I need to change enviroment variables not parameters. >>> >>> Il giorno martedì 1 marzo 2022 alle 15:38:09 UTC+1 Mario Giammarco ha >>> scritto: >>> >>>> Hi, >>>> I resurrect this thread. >>>> I have one pipeline and several agents. I need to run the pipeline >>>> multiple times in parallel with different parameters. >>>> It seems to me a common and easy question. >>>> Reading this thread, bug description and other threads it seems it is >>>> not possible. >>>> I am really surprised about it. >>>> Is it really not possible? >>>> Is there a workaround? >>>> Thanks, >>>> Mario >>>> >>>> Il giorno lunedì 3 ottobre 2016 alle 08:00:08 UTC+2 Zabil C M ha >>>> scritto: >>>> >>>>> There's an issue logged for this here >>>>> https://github.com/gocd/gocd/issues/2329 >>>>> >>>>> In short, you can't do this at the moment but we are willing to help >>>>> out with and merge this feature if someone picks it up. >>>>> >>>>> On Wed, Sep 28, 2016 at 3:48 PM, <[email protected]> wrote: >>>>> >>>>>> One of the key features that made us decide to migrate to gocd >>>>>> several months ago was the idea that pipelines are able to run in >>>>>> parallel >>>>>> and that more than one instance of a pipeline can be started. >>>>>> Only now are we starting to realise that actually the same pipeline >>>>>> cannot be executed multiple times in parallel since each stage will >>>>>> always >>>>>> run sequentially even if it's the same stage in different pipeline >>>>>> instances. I'm stating this only based on this post >>>>>> <https://groups.google.com/d/msg/go-cd/E0goLJITLy8/8UN7KrjCCAAJ> which >>>>>> is the only mention of this issue that I've been able to find. >>>>>> >>>>>> So first off I'd like to verify if this is correct. Is it not >>>>>> possible to run several instances of the same pipeline concurrently (in >>>>>> parallel )without one instance being constrained by another (i.e. the >>>>>> second instance of the same pipeline is able to complete even if a >>>>>> previously initiated instance has not done so)? >>>>>> >>>>>> If this is the case, I'd really appreciate any help/ideas on >>>>>> overcoming this limitation in some way. >>>>>> The pipeline I'm working on has only one stage which simply runs a >>>>>> docker and then deletes the container and image. The docker does some >>>>>> work >>>>>> on our machine learning models, there is no problem to run several >>>>>> containers of this docker at the same time - and that's exactly what I'd >>>>>> like to do - i.e. run another docker each time that the pipeline is >>>>>> triggered (we're using the api to trigger it). >>>>>> I'd like to see the output created by each docker and, of course, see >>>>>> whether each pipeline has finished successfully or failed. The order of >>>>>> execution and even the material version is not relevant, each instance of >>>>>> the pipeline/docker has it's job to do. >>>>>> >>>>>> Finally, I'd like to ask if you guys feel that a feature that enables >>>>>> the same stage to run concurrently in different pipeline instances is >>>>>> feasible. And that a request for this is something which has a chance of >>>>>> being accepted (I will, of course, contribute anything I can within my >>>>>> technical skills) . >>>>>> I'm sure that the are many use cases that will benefit from such a >>>>>> feature. >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "go-cd" group. >>>>>> >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to [email protected]. >>>>> >>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/b9f673a6-01ba-4950-9d0e-9cafc2151014n%40googlegroups.com >>> <https://groups.google.com/d/msgid/go-cd/b9f673a6-01ba-4950-9d0e-9cafc2151014n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/b664a66e-c079-4090-abd2-757e62ce6b86n%40googlegroups.com > <https://groups.google.com/d/msgid/go-cd/b664a66e-c079-4090-abd2-757e62ce6b86n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH9w7TaKNC9t0mr8eN5OHTyD_fPMhKL%2BbGZnJLw2183bCQ%40mail.gmail.com.
