With that description, if you want to use *environments <https://docs.gocd.org/current/introduction/concepts_in_go.html#environment>* rather than resources <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> (and assuming you don't use environments for any other purpose), I would
*1) Create 2 environments "fast" and "any"* *2) Map agents to environments* agents on GROUPA = machines that have less beefy hardware *- declare environments "any" when registered* agents on GROUPB = more expensive machines *- declare both environments "fast" and "any" when registered* *3) When configuring your pipelines* 1. have a couple of pipelines only run on the more expensive machines *<-- add these pipelines to "fast" environment* 2. have all other pipelines run in either group (next available agent) *<-- add these pipelines to "any" environment* This should give you roughly the semantics you say you want, but note it won't *prioritise* the GROUPB agents for use by the "couple of pipelines only run on the more expensive machines", it will just ensure they never run on the slower machines/agents. Something equivalent could also be done with resources <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources>. There is no way to "try another agent" from inside the actual job's tasks. In this sense, the contents of tasks/scripts aren't relevant to scheduling. The GoCD resources and environments have to be known at schedule time. When you use pipeline parameters, they are realised at configuration time as when you create a pipeline from a template, it will force you to set the parameter values. To clarify, when you talked earlier about "a resource requirement" are you *actually* referring to GoCD's concept of resources, or were you talking in a generic sense? The answers are assuming you are talking about GoCD resources <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> but now I am more confused by your shell script. *If you want to use resources* (rather than environments) to affect scheduling, while still avoiding duplication of your templates, we are suggesting you use a parameter like *this*, not put it into some task content. You are setting the parameterized value into the field that determines the job's scheduling, not something that happens at execution time like a task. But again, if your goal is to control scheduling at pipeline level, for all jobs in a pipeline, you don't need to use resources, and can just use environments as in my earlier example above. [image: image.png] -Chad On Sun, Jul 23, 2023 at 8:21 PM Joshua Franta <jos...@pracplay.com> wrote: > appreciate so many responses. i think we're a little apart so i'll take > the suggestion to give our example: > > GROUPA = machines that have less beefy hardware > GROUPB = more expensive machines > > we'd like to: > > 1. have a couple of pipelines only run on the more expensive machines > 2. have all other pipelines run in either group (next available agent) > > perhaps this was not clear from my previous explanations, but a couple of > people have suggested pipeline parameters. > > EXAMPLE > > a pipeline parameter is only going to be available to the job after the > job has already been assigned an agent, right? > > so if i have a pipeline called 'Priority' w/a parameter called "group-id" > and the pipeline has a 'Job' that is a shell script: > > ---- > ##!/bin/sh > > agent_resource="$GO_AGENT_RESOURCE_VARIABLE" > > if ! echo "#{group-id}" |grep -q "$agentr_esource"; then > > echo "agent can't run #{group-id} pipelines" > > ## won't this will make my pipeline fail when I want it to simply try > another agent? > exit 1 > fi > > ---- > > or perhaps people saying this know of some environment variable that where > we can request another agent? > > obviously pipeline parameters themseles don't do anything, so i'm confused > how i can affect assignment in a job that requires an agent before it runs. > this 2nd part is what i don't get above > > appreciate any clarifications or suggestions thx > > > On Sat, Jul 22, 2023 at 9:58 AM Chad Wilson <ch...@thoughtworks.com> > wrote: > >> >> On Sat, Jul 22, 2023 at 8:21 PM Joshua Franta <jos...@pracplay.com> >> wrote: >> >>> re: using environments rather than resources... environments can't be >>> defined at the pipeline level either though? >>> >> >> A pipeline is *assigned* to 0 or 1 environments (via the Admin > >> Environments UI if not using pipelines-as-code) - thus it's at the pipeline >> level by definition. It defines a scheduling requirement for all jobs in >> that pipeline. Which seems what you asked for with "able to communicate a >> resource requirement at the pipeline level" right? >> >> >> or i guess it's more correct to say that using environments is a bit of a >>> side-car feature, in that we use interact w/environments through a >>> different prisim/ui/config (no biggie) but also seems it's mutually >>> exclusive to maximizing overall usage of agents. for us if a given host >>> can execute something (a pipeline, a job) it should. and if it can't, it >>> shouldn't. >>> trying to force a hard divider can be useful for prod/qa staging, but it >>> seems to limit just being able to have pipelines declare their needs. >>> maybe i'm missing what you're saying but i don't think environments are >>> functionally equivalent to resources? >>> >> >> I didn't imply they were functionally equivalent, but I did try to imply >> they were a different mechanism of defining a requirement on a job's >> scheduling, at the pipeline level. If a pipeline is assigned to an >> "environment", its jobs must be scheduled on agents that also declare they >> support that "environment". Similarly if a pipeline job declares a resource >> requirement, the agent must also have that resource declared for it to be >> assigned. This is a very similar, but different level of configuration of a >> scheduling requirement, no? >> https://docs.gocd.org/current/configuration/managing_environments.html >> >> Anyway, perhaps I don't understand what you are trying to achieve. If you >> are currently trying to "prioritise" pipelines by using resources you can >> also "prioritise" pipelines by having pools of agents, say, dedicated to an >> environment you call "high-priority". As I said, "Don't need to get hung up >> on the name [environment]". >> >> >> >>> we use template parameters extensively already. >>> eg we even templatize further inside our own jobs by re-using scripts >>> that interact with template parameters on most commonly used templates (eg >>> our most popular template has maybe 10-15 pipelines). >>> however this is more of a job specific thing since it's at the job level. >>> if you're saying we could change every pipeline to read this at a >>> pipeline level is a non trivial change to every job. >>> >> >> You said you had many templates that varied only by the "resources" field >> for jobs. If that is the stated problem then parameters are a possible >> solution to remove duplication, no? >> >> >>> that's ok but i guess my overall question tho would be that if a given >>> job decided it couldn't execute the pipeline parameters... it has no way to >>> pass the job to another agent? >>> >> >> That's the same problem you have currently if the resource is typoed or >> wrong inside the template, no? If the resource requirement has no available >> agents, then it can't be scheduled. >> >> >>> in such an example it would just fail the job, no? again maybe i'm not >>> following but this seems to not allow the business/value level to declare >>> minimum needs >>> (environments seem like they are more about maximimal requirements, but >>> i'm no expert) >>> >> >> I'm not following what you're trying to say here, sorry. >> >> Perhaps this would be easier if you gave a specific example of how you >> achieve "have some pipelines that are given higher preferences for >> agent/build resources" currently, rather than talking in abstract terms? >> >> -Chad >> >> >> On Sat, Jul 22, 2023 at 6:56 AM Chad Wilson <ch...@thoughtworks.com> >>> wrote: >>> >>>> Have you tried to use "environments" (or a mix of environments and >>>> resources) to achieve what you are trying to? >>>> >>>> When scheduling jobs it's the combination of the resource and the >>>> environment that are matched to an agent, but the relevant environment is >>>> declared at the pipeline level like you refer to. Don't need to get hung up >>>> on the name so much. Yes, you can have "environment variables" attached to >>>> an environment and propagate those to all pipelines within it, but you >>>> don't have to use them like that. >>>> >>>> Alternatively, to make the templates less duplicated and allow the >>>> resource to flow from the pipeline *using* the template, you could try >>>> using template parameters >>>> <https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html> >>>> in the resources field? e.g #{job-resoure-requirement}? If there are only a >>>> small number of different resources used across the stages/jobs, you could >>>> use the parameters to "model" this I imagine. >>>> >>>> -Chad >>>> >>>> On Sat, Jul 22, 2023 at 6:54 PM Josh <jos...@pracplay.com> wrote: >>>> >>>>> QUESTION: >>>>> >>>>> Shouldn't we also be able to communicate a resource requirement at the >>>>> pipeline level, and not just inside a single job? >>>>> >>>>> I get that it definately needs to be at the job level since that's the >>>>> smallest unit of work and some machines can't execute certain tasks. >>>>> But at the value-stream/pipeline/business level, you also want to be >>>>> able to have some pipelines compiling on preferred resources, no? >>>>> >>>>> >>>>> is there a better way to accomplish this? >>>>> or perhaps this already is possible and i'm missing it. >>>>> i looked closely at the config since sometimes you can do something >>>>> simple that is not possible inside the UI, but I'm not seeing it. >>>>> >>>>> To restate use case: We have some pipelines that are given higher >>>>> preferences for agent/build resources. Wanting to do a lot more of this, >>>>> but it's tricky because resources can only be defined at the job level (in >>>>> the UI). Also we use a lot of templates, so having resources at job >>>>> level means we end up having lots of alsomost identical templates that >>>>> only >>>>> vary by the resources used (which somewhat defeats the point of the >>>>> templates and the value of gocd in this respect). >>>>> >>>>> hoping there is a config hack or maybe i'm missinig something. >>>>> also if this could be done in a plugin, any color there would be >>>>> helpful (and i would make sure it's open sourced if need be). >>>>> >>>>> thx >>>>> >>>>> ps i keep using other ci/cd products and gocd is still one of the all >>>>> around bests. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "go-cd" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to go-cd+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "go-cd" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> go-cd+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to go-cd+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "go-cd" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> go-cd+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to go-cd+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH8XmX4y11RT2PVXeUxQd8hvK0UBgbcx8ja-MWuOVqpfag%40mail.gmail.com.
