I have read your suggestions but I think my case is different. Pipelines of the server side for testing are run by Azure Devops. When a backend est pipeline is ready (has launched a server) it calls via api the pipeline of gocd passing in an environment variable the ip address of the server to test. Now the pipeline of gocd should "find" a free agent and compile and run test client that will call the test server by ip. If there are no free agents we wait for.
Il giorno mercoledì 2 marzo 2022 alle 11:53:05 UTC+1 Chad Wilson ha scritto: > It sounds like having multiple jobs with the same tasks in a single stage > and single pipeline might do what you need. This is what is done within > GoCD itself to achieve parallel integration testing. > > If you don't want to have multiple job definitions and manually vary the > environment variables between jobs you can consider whether using the job > option for "Run multiple instances" is suitable for your case. > > When you use this option and a stage is triggered, it will start the same > job definition in parallel 'n' times with GO_JOB_RUN_COUNT=n and varying > GO_JOB_RUN_INDEX=1..n. You can then change your scripting to partition the > tests deterministically into buckets depending on the # of jobs in the > _COUNT and select partition based on the index. Bit more advanced, but > worth considering if you are just trying to split a chunk of work over 'n' > agents. > > -Chad > > On Wed, Mar 2, 2022 at 6:39 AM Mario Giammarco <[email protected]> wrote: > >> I need to build a parallel integration testing. >> I start from same source (same git repository same tag). >> I have several PCs (each one with agent running NOT as a service). Each >> PC must compile and launch a .exe with a command line parameter (that I >> take from environment) different for each test. >> So I supposed to use a pipeline: I start it via API many times, then the >> pipeline on each PC compile and launch the .exe , it connects to server, >> dowloads test, runs test and stop. >> In this way it is very simple: many tests are launched, they fills >> agents, I can scale horizontally. >> So please tell me the right approach to do this. >> Thanks, >> Mario >> >> Il giorno martedì 1 marzo 2022 alle 16:26:54 UTC+1 Chad Wilson ha scritto: >> >>> Hi Mario >>> >>> That GitHub ticket is specifically about the case where you want to be >>> building in parallel based on different *material versions* i.e *inputs* >>> to the pipeline. (e.g developer pushes to a git material 3x, say 30s apart >>> - can all 3 builds start in parallel?) >>> >>> If you are *not* trying to build based on different material versions >>> in parallel and instead want to vary other aspects (env vars, environments, >>> parameters etc) for the *same material versions*, even without running >>> the same *pipeline *in parallel as in that ticket, there are different >>> ways to approach parallel runs in GoCD. >>> >>> - separate *jobs* within the same stage of a single pipeline that >>> override env vars differently at job scope, but execute the same tasks >>> in >>> each job >>> - separate *pipelines* entirely, that have the same material >>> inputs/triggers, but have different configuration for the stage/job/tasks >>> >>> The best way to model it depends on what is inside your existing >>> pipeline (multiple stages? multiple jobs within stages?) and also what you >>> want to do as a result of completion: >>> >>> - trigger a downstream pipeline only once all 'n' parallel runs >>> complete? - jobs might work. >>> - trigger another pipeline/stage as soon as any single one of the >>> parallel jobs complete and then trigger again for each parallel run? - >>> pipelines might be a better approach. >>> >>> You'd probably need to share more context on how you want things to >>> behave *after* your parallel runs and the content of your existing >>> pipeline to suggest the best approach. >>> >>> There is some config overhead/duplication, but you can generally >>> mitigate duplication using either >>> >>> - pipeline templates >>> - *or* one of the config repo plugins to drive pipeline config from >>> source control >>> - https://github.com/tomzo/gocd-yaml-config-plugin (bundled) >>> - using YAML anchors for reusable stage/job/tasks >>> - https://github.com/gocd-contrib/gocd-groovy-dsl-config-plugin >>> - using scripting for reusable stage/job/tasks >>> >>> -Chad >>> >>> On Tue, Mar 1, 2022 at 10:43 PM Mario Giammarco <[email protected]> >>> wrote: >>> >>>> Sorry I need to change enviroment variables not parameters. >>>> >>>> Il giorno martedì 1 marzo 2022 alle 15:38:09 UTC+1 Mario Giammarco ha >>>> scritto: >>>> >>>>> Hi, >>>>> I resurrect this thread. >>>>> I have one pipeline and several agents. I need to run the pipeline >>>>> multiple times in parallel with different parameters. >>>>> It seems to me a common and easy question. >>>>> Reading this thread, bug description and other threads it seems it is >>>>> not possible. >>>>> I am really surprised about it. >>>>> Is it really not possible? >>>>> Is there a workaround? >>>>> Thanks, >>>>> Mario >>>>> >>>>> Il giorno lunedì 3 ottobre 2016 alle 08:00:08 UTC+2 Zabil C M ha >>>>> scritto: >>>>> >>>>>> There's an issue logged for this here >>>>>> https://github.com/gocd/gocd/issues/2329 >>>>>> >>>>>> In short, you can't do this at the moment but we are willing to help >>>>>> out with and merge this feature if someone picks it up. >>>>>> >>>>>> On Wed, Sep 28, 2016 at 3:48 PM, <[email protected]> wrote: >>>>>> >>>>>>> One of the key features that made us decide to migrate to gocd >>>>>>> several months ago was the idea that pipelines are able to run in >>>>>>> parallel >>>>>>> and that more than one instance of a pipeline can be started. >>>>>>> Only now are we starting to realise that actually the same pipeline >>>>>>> cannot be executed multiple times in parallel since each stage will >>>>>>> always >>>>>>> run sequentially even if it's the same stage in different pipeline >>>>>>> instances. I'm stating this only based on this post >>>>>>> <https://groups.google.com/d/msg/go-cd/E0goLJITLy8/8UN7KrjCCAAJ> which >>>>>>> is the only mention of this issue that I've been able to find. >>>>>>> >>>>>>> So first off I'd like to verify if this is correct. Is it not >>>>>>> possible to run several instances of the same pipeline concurrently (in >>>>>>> parallel )without one instance being constrained by another (i.e. the >>>>>>> second instance of the same pipeline is able to complete even if a >>>>>>> previously initiated instance has not done so)? >>>>>>> >>>>>>> If this is the case, I'd really appreciate any help/ideas on >>>>>>> overcoming this limitation in some way. >>>>>>> The pipeline I'm working on has only one stage which simply runs a >>>>>>> docker and then deletes the container and image. The docker does some >>>>>>> work >>>>>>> on our machine learning models, there is no problem to run several >>>>>>> containers of this docker at the same time - and that's exactly what >>>>>>> I'd >>>>>>> like to do - i.e. run another docker each time that the pipeline is >>>>>>> triggered (we're using the api to trigger it). >>>>>>> I'd like to see the output created by each docker and, of course, >>>>>>> see whether each pipeline has finished successfully or failed. The >>>>>>> order of >>>>>>> execution and even the material version is not relevant, each instance >>>>>>> of >>>>>>> the pipeline/docker has it's job to do. >>>>>>> >>>>>>> Finally, I'd like to ask if you guys feel that a feature that >>>>>>> enables the same stage to run concurrently in different pipeline >>>>>>> instances >>>>>>> is feasible. And that a request for this is something which has a >>>>>>> chance >>>>>>> of being accepted (I will, of course, contribute anything I can within >>>>>>> my >>>>>>> technical skills) . >>>>>>> I'm sure that the are many use cases that will benefit from such a >>>>>>> feature. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "go-cd" group. >>>>>>> >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>> >>>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "go-cd" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/b9f673a6-01ba-4950-9d0e-9cafc2151014n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/go-cd/b9f673a6-01ba-4950-9d0e-9cafc2151014n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/b664a66e-c079-4090-abd2-757e62ce6b86n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/go-cd/b664a66e-c079-4090-abd2-757e62ce6b86n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/060aea3c-1db9-479e-8106-47bce068763an%40googlegroups.com.
