Hi Josh, I think there may be some confusion here regarding GoCD terminology and common concepts.
> i think the main source of confusion is that I thought parameters could only be referred to in scripts! > I didn't know you could refer to them inside of other configuration properties! To the best of my knowledge, Parameters (GoCD concept) cannot be referenced in scripts. You can call a script that uses parameters (scripting concept), but as far as I know, GoCD Parameters are not persisted in the Agent's runtime environment unless they are somehow passed in via the Task definition. Are you sure you aren't thinking of Environment Variables (GoCD concept)? Environment Variables can be defined in a few different places in GoCD. As the name suggests, these values are persisted in the Agent's runtime environment when a Task is executed. > I still have a question about how this works in examples using templates. > If we didn't define the pipeline parameter by default, how would gocd interpret what I'm guessing would be a blank resource? If a Template references a Parameter then every Pipeline that uses that Template _must_ define that Parameter. Depending on how the Parameter is used in the Template, leaving the Parameter Value blank may be valid. In the case of using Parameters to define Resources, my testing shows that each Parameter must define a single, valid, Resource. That is, if you want to specify multiple Parameterized Resources, you must use multiple Parameters. You cannot, for example, provide a Parameter Value of "foo, bar" to make your Pipeline's Job depend on the "foo" and "bar" Resources. GoCD rejects the configuration as invalid if you try to save it. Similarly, GoCD rejects the configuration as invalid if a Parameter is used in the Resource field and you try to leave its Value blank. Regarding your specific use case, you can solve it using either Environments or Resources. The right solution depends on your requirements and how you want to reason about your environment. The way I understand it, in the context of this discussion you have 2 groups of Agents (Agents1 and Agents2) and 2 groups of Pipelines (PipelinesA and PipelinesB). The Pipelines in PipelinesA can run on any Agent, but the Pipelines in PipelinesB must run on the Agents in Agents2. We will ignore the fact that Pipelines can contain multiple Stages and multiple Jobs and assume either that all of the Pipelines contain a single Stage with a single Job, or that the scheduling requirements are the same for all Jobs in a given Pipeline. You have also talked about Pipeline priority. Based on this, I assume your requirements are one of the following: 1. Agents should be used to the full extent possible; the workload in PipelinesB is heavier so those Pipelines must not run on Agents1, or 2. Pipelines in PipelinesB have a higher priority than those in PipelinesA; Agents in Agents2 should take Jobs from PipelinesA only if there are no pending Jobs for PipelinesB. GoCD supports the first scenario. You can achieve this by assigning 2 Resources/Environments. Pipelines in PipelinesA get 1 Resource/Environment; Pipelines in PipelinesB get the other. Agents in Agents1 get the PipelinesA Resource/Environment; Agents in Agents2 get both. GoCD does not support the concept of priority, so scenario 2 is not supported. The best you could accomplish would be to map each group of Pipelines to a single group of Agents. Hope this helps. If I'm way off base it might help me better understand your situation if you would provide snippets of your actual configuration. Cheers, Jason On Monday, 24 July 2023 at 11:43:58 UTC-4 Chad Wilson wrote: > On Mon, Jul 24, 2023 at 8:44 PM Joshua Franta <jos...@pracplay.com> wrote: > >> >> chad thanks for your answer. >> >> i think the main source of confusion is that I thought parameters could >> only be referred to in scripts! >> I didn't know you could refer to them inside of other configuration >> properties! >> Is this documented? Regardless that's super useful, there's probably >> some other things that can be cleaned up knowing that. >> > > > https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html#rules-around-usage-of-parameters > > > >> I tried this on a pipeline w/out any template and it worked as >> described. Just put the parameter reference in resource- UI accepts as >> long as parameter exists and works. >> >> I still have a question about how this works in examples using templates. >> If we didn't define the pipeline parameter by default, how would gocd >> interpret what I'm guessing would be a blank resource? >> >> eg we have >> >> 1. a pipeline template called FAST_OR_SLOW_PIPE >> 2. every pipeline implementing this template defines a parameter >> called PIPE_RESOURCE_PARAM >> >> What happens if somebody only defines PIPE_RESOURCE_PARAM when the >> pipeline is FAST? >> If it's left as empty for ANY-aka-SLOW resources, will gocd intepret this >> as a blank resource requirement and fail? >> Or will it ignore blank resources? >> > > I'm not sure - perhaps just try it empirically? It could either fail or > see it as blank i.e "no resource requirement" - I don't think there's a > strong case for either behaviour being more correct. > > -Chad > > >> On Sun, Jul 23, 2023 at 10:04 AM Chad Wilson <ch...@thoughtworks.com> >> wrote: >> >>> With that description, if you want to use *environments >>> <https://docs.gocd.org/current/introduction/concepts_in_go.html#environment>* >>> >>> rather than resources >>> <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> >>> (and assuming you don't use environments for any other purpose), I would >>> >>> *1) Create 2 environments "fast" and "any"* >>> >>> *2) Map agents to environments* >>> agents on GROUPA = machines that have less beefy hardware >>> *- declare environments "any" when registered* >>> >>> agents on GROUPB = more expensive machines >>> *- declare both environments "fast" and "any" when registered* >>> >>> *3) When configuring your pipelines* >>> >>> 1. have a couple of pipelines only run on the more expensive >>> machines *<-- add these pipelines to "fast" environment* >>> 2. have all other pipelines run in either group (next available >>> agent) *<-- add these pipelines to "any" environment* >>> >>> This should give you roughly the semantics you say you want, but note it >>> won't *prioritise* the GROUPB agents for use by the "couple of >>> pipelines only run on the more expensive machines", it will just ensure >>> they never run on the slower machines/agents. Something equivalent could >>> also be done with resources >>> <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> >>> . >>> >>> There is no way to "try another agent" from inside the actual job's >>> tasks. In this sense, the contents of tasks/scripts aren't relevant to >>> scheduling. The GoCD resources and environments have to be known at >>> schedule time. When you use pipeline parameters, they are realised at >>> configuration time as when you create a pipeline from a template, it will >>> force you to set the parameter values. >>> >>> To clarify, when you talked earlier about "a resource requirement" are >>> you *actually* referring to GoCD's concept of resources, or were you >>> talking in a generic sense? The answers are assuming you are talking about >>> GoCD resources >>> <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> >>> but now I am more confused by your shell script. *If you want to use >>> resources* (rather than environments) to affect scheduling, while still >>> avoiding duplication of your templates, we are suggesting you use a >>> parameter like *this*, not put it into some task content. You are >>> setting the parameterized value into the field that determines the job's >>> scheduling, not something that happens at execution time like a task. But >>> again, if your goal is to control scheduling at pipeline level, for all >>> jobs in a pipeline, you don't need to use resources, and can just use >>> environments as in my earlier example above. >>> >>> [image: image.png] >>> >>> -Chad >>> >>> >>> On Sun, Jul 23, 2023 at 8:21 PM Joshua Franta <jos...@pracplay.com> >>> wrote: >>> >>>> appreciate so many responses. i think we're a little apart so i'll >>>> take the suggestion to give our example: >>>> >>>> GROUPA = machines that have less beefy hardware >>>> GROUPB = more expensive machines >>>> >>>> we'd like to: >>>> >>>> 1. have a couple of pipelines only run on the more expensive >>>> machines >>>> 2. have all other pipelines run in either group (next available >>>> agent) >>>> >>>> perhaps this was not clear from my previous explanations, but a couple >>>> of people have suggested pipeline parameters. >>>> >>>> EXAMPLE >>>> >>>> a pipeline parameter is only going to be available to the job after the >>>> job has already been assigned an agent, right? >>>> >>>> so if i have a pipeline called 'Priority' w/a parameter called >>>> "group-id" and the pipeline has a 'Job' that is a shell script: >>>> >>>> ---- >>>> ##!/bin/sh >>>> >>>> agent_resource="$GO_AGENT_RESOURCE_VARIABLE" >>>> >>>> if ! echo "#{group-id}" |grep -q "$agentr_esource"; then >>>> >>>> echo "agent can't run #{group-id} pipelines" >>>> >>>> ## won't this will make my pipeline fail when I want it to simply >>>> try another agent? >>>> exit 1 >>>> fi >>>> >>>> ---- >>>> >>>> or perhaps people saying this know of some environment variable that >>>> where we can request another agent? >>>> >>>> obviously pipeline parameters themseles don't do anything, so i'm >>>> confused how i can affect assignment in a job that requires an agent >>>> before >>>> it runs. >>>> this 2nd part is what i don't get above >>>> >>>> appreciate any clarifications or suggestions thx >>>> >>>> >>>> On Sat, Jul 22, 2023 at 9:58 AM Chad Wilson <ch...@thoughtworks.com> >>>> wrote: >>>> >>>>> >>>>> On Sat, Jul 22, 2023 at 8:21 PM Joshua Franta <jos...@pracplay.com> >>>>> wrote: >>>>> >>>>>> re: using environments rather than resources... environments can't be >>>>>> defined at the pipeline level either though? >>>>>> >>>>> >>>>> A pipeline is *assigned* to 0 or 1 environments (via the Admin > >>>>> Environments UI if not using pipelines-as-code) - thus it's at the >>>>> pipeline >>>>> level by definition. It defines a scheduling requirement for all jobs in >>>>> that pipeline. Which seems what you asked for with "able to communicate a >>>>> resource requirement at the pipeline level" right? >>>>> >>>>> >>>>> or i guess it's more correct to say that using environments is a bit >>>>>> of a side-car feature, in that we use interact w/environments through a >>>>>> different prisim/ui/config (no biggie) but also seems it's mutually >>>>>> exclusive to maximizing overall usage of agents. for us if a given >>>>>> host >>>>>> can execute something (a pipeline, a job) it should. and if it can't, >>>>>> it >>>>>> shouldn't. >>>>>> trying to force a hard divider can be useful for prod/qa staging, but >>>>>> it seems to limit just being able to have pipelines declare their needs. >>>>>> maybe i'm missing what you're saying but i don't think environments >>>>>> are functionally equivalent to resources? >>>>>> >>>>> >>>>> I didn't imply they were functionally equivalent, but I did try to >>>>> imply they were a different mechanism of defining a requirement on a >>>>> job's >>>>> scheduling, at the pipeline level. If a pipeline is assigned to an >>>>> "environment", its jobs must be scheduled on agents that also declare >>>>> they >>>>> support that "environment". Similarly if a pipeline job declares a >>>>> resource >>>>> requirement, the agent must also have that resource declared for it to be >>>>> assigned. This is a very similar, but different level of configuration of >>>>> a >>>>> scheduling requirement, no? >>>>> https://docs.gocd.org/current/configuration/managing_environments.html >>>>> >>>>> Anyway, perhaps I don't understand what you are trying to achieve. If >>>>> you are currently trying to "prioritise" pipelines by using resources you >>>>> can also "prioritise" pipelines by having pools of agents, say, dedicated >>>>> to an environment you call "high-priority". As I said, "Don't need to get >>>>> hung up on the name [environment]". >>>>> >>>>> >>>>> >>>>>> we use template parameters extensively already. >>>>>> eg we even templatize further inside our own jobs by re-using scripts >>>>>> that interact with template parameters on most commonly used templates >>>>>> (eg >>>>>> our most popular template has maybe 10-15 pipelines). >>>>>> however this is more of a job specific thing since it's at the job >>>>>> level. >>>>>> if you're saying we could change every pipeline to read this at a >>>>>> pipeline level is a non trivial change to every job. >>>>>> >>>>> >>>>> You said you had many templates that varied only by the "resources" >>>>> field for jobs. If that is the stated problem then parameters are a >>>>> possible solution to remove duplication, no? >>>>> >>>>> >>>>>> that's ok but i guess my overall question tho would be that if a >>>>>> given job decided it couldn't execute the pipeline parameters... it has >>>>>> no >>>>>> way to pass the job to another agent? >>>>>> >>>>> >>>>> That's the same problem you have currently if the resource is typoed >>>>> or wrong inside the template, no? If the resource requirement has no >>>>> available agents, then it can't be scheduled. >>>>> >>>>> >>>>>> in such an example it would just fail the job, no? again maybe i'm >>>>>> not following but this seems to not allow the business/value level to >>>>>> declare minimum needs >>>>>> (environments seem like they are more about maximimal requirements, >>>>>> but i'm no expert) >>>>>> >>>>> >>>>> I'm not following what you're trying to say here, sorry. >>>>> >>>>> Perhaps this would be easier if you gave a specific example of how you >>>>> achieve "have some pipelines that are given higher preferences for >>>>> agent/build resources" currently, rather than talking in abstract terms? >>>>> >>>>> -Chad >>>>> >>>>> >>>>> On Sat, Jul 22, 2023 at 6:56 AM Chad Wilson <ch...@thoughtworks.com> >>>>>> wrote: >>>>>> >>>>>>> Have you tried to use "environments" (or a mix of environments and >>>>>>> resources) to achieve what you are trying to? >>>>>>> >>>>>>> When scheduling jobs it's the combination of the resource and the >>>>>>> environment that are matched to an agent, but the relevant environment >>>>>>> is >>>>>>> declared at the pipeline level like you refer to. Don't need to get >>>>>>> hung up >>>>>>> on the name so much. Yes, you can have "environment variables" attached >>>>>>> to >>>>>>> an environment and propagate those to all pipelines within it, but you >>>>>>> don't have to use them like that. >>>>>>> >>>>>>> Alternatively, to make the templates less duplicated and allow the >>>>>>> resource to flow from the pipeline *using* the template, you could >>>>>>> try using template parameters >>>>>>> <https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html> >>>>>>> >>>>>>> in the resources field? e.g #{job-resoure-requirement}? If there are >>>>>>> only a >>>>>>> small number of different resources used across the stages/jobs, you >>>>>>> could >>>>>>> use the parameters to "model" this I imagine. >>>>>>> >>>>>>> -Chad >>>>>>> >>>>>>> On Sat, Jul 22, 2023 at 6:54 PM Josh <jos...@pracplay.com> wrote: >>>>>>> >>>>>>>> QUESTION: >>>>>>>> >>>>>>>> Shouldn't we also be able to communicate a resource requirement at >>>>>>>> the pipeline level, and not just inside a single job? >>>>>>>> >>>>>>>> I get that it definately needs to be at the job level since that's >>>>>>>> the smallest unit of work and some machines can't execute certain >>>>>>>> tasks. >>>>>>>> But at the value-stream/pipeline/business level, you also want to >>>>>>>> be able to have some pipelines compiling on preferred resources, no? >>>>>>>> >>>>>>>> >>>>>>>> is there a better way to accomplish this? >>>>>>>> or perhaps this already is possible and i'm missing it. >>>>>>>> i looked closely at the config since sometimes you can do something >>>>>>>> simple that is not possible inside the UI, but I'm not seeing it. >>>>>>>> >>>>>>>> To restate use case: We have some pipelines that are given higher >>>>>>>> preferences for agent/build resources. Wanting to do a lot more of >>>>>>>> this, >>>>>>>> but it's tricky because resources can only be defined at the job level >>>>>>>> (in >>>>>>>> the UI). Also we use a lot of templates, so having resources at >>>>>>>> job >>>>>>>> level means we end up having lots of alsomost identical templates that >>>>>>>> only >>>>>>>> vary by the resources used (which somewhat defeats the point of the >>>>>>>> templates and the value of gocd in this respect). >>>>>>>> >>>>>>>> hoping there is a config hack or maybe i'm missinig something. >>>>>>>> also if this could be done in a plugin, any color there would be >>>>>>>> helpful (and i would make sure it's open sourced if need be). >>>>>>>> >>>>>>>> thx >>>>>>>> >>>>>>>> ps i keep using other ci/cd products and gocd is still one of the >>>>>>>> all around bests. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "go-cd" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to go-cd+un...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "go-cd" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> go-cd+un...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "go-cd" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to go-cd+un...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "go-cd" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> go-cd+un...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com >>>>> >>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "go-cd" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to go-cd+un...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com >>>> >>>> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "go-cd" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> go-cd+un...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8XmX4y11RT2PVXeUxQd8hvK0UBgbcx8ja-MWuOVqpfag%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8XmX4y11RT2PVXeUxQd8hvK0UBgbcx8ja-MWuOVqpfag%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to go-cd+un...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/CABr%2BOtoymQdOg0poCtRYDRfxgX4UKfukvtvvpCOZicadaPKWyA%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtoymQdOg0poCtRYDRfxgX4UKfukvtvvpCOZicadaPKWyA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/43cb7edd-5e0d-4b06-9dac-024e40509d43n%40googlegroups.com.
