On Mon, Jul 24, 2023 at 8:44 PM Joshua Franta <jos...@pracplay.com> wrote:
> > chad thanks for your answer. > > i think the main source of confusion is that I thought parameters could > only be referred to in scripts! > I didn't know you could refer to them inside of other configuration > properties! > Is this documented? Regardless that's super useful, there's probably > some other things that can be cleaned up knowing that. > https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html#rules-around-usage-of-parameters > I tried this on a pipeline w/out any template and it worked as described. > Just put the parameter reference in resource- UI accepts as long as > parameter exists and works. > > I still have a question about how this works in examples using templates. > If we didn't define the pipeline parameter by default, how would gocd > interpret what I'm guessing would be a blank resource? > > eg we have > > 1. a pipeline template called FAST_OR_SLOW_PIPE > 2. every pipeline implementing this template defines a parameter > called PIPE_RESOURCE_PARAM > > What happens if somebody only defines PIPE_RESOURCE_PARAM when the > pipeline is FAST? > If it's left as empty for ANY-aka-SLOW resources, will gocd intepret this > as a blank resource requirement and fail? > Or will it ignore blank resources? > I'm not sure - perhaps just try it empirically? It could either fail or see it as blank i.e "no resource requirement" - I don't think there's a strong case for either behaviour being more correct. -Chad > On Sun, Jul 23, 2023 at 10:04 AM Chad Wilson <ch...@thoughtworks.com> > wrote: > >> With that description, if you want to use *environments >> <https://docs.gocd.org/current/introduction/concepts_in_go.html#environment>* >> rather than resources >> <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> >> (and assuming you don't use environments for any other purpose), I would >> >> *1) Create 2 environments "fast" and "any"* >> >> *2) Map agents to environments* >> agents on GROUPA = machines that have less beefy hardware >> *- declare environments "any" when registered* >> >> agents on GROUPB = more expensive machines >> *- declare both environments "fast" and "any" when registered* >> >> *3) When configuring your pipelines* >> >> 1. have a couple of pipelines only run on the more expensive machines *<-- >> add these pipelines to "fast" environment* >> 2. have all other pipelines run in either group (next available >> agent) *<-- add these pipelines to "any" environment* >> >> This should give you roughly the semantics you say you want, but note it >> won't *prioritise* the GROUPB agents for use by the "couple of pipelines >> only run on the more expensive machines", it will just ensure they never >> run on the slower machines/agents. Something equivalent could also be done >> with resources >> <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> >> . >> >> There is no way to "try another agent" from inside the actual job's >> tasks. In this sense, the contents of tasks/scripts aren't relevant to >> scheduling. The GoCD resources and environments have to be known at >> schedule time. When you use pipeline parameters, they are realised at >> configuration time as when you create a pipeline from a template, it will >> force you to set the parameter values. >> >> To clarify, when you talked earlier about "a resource requirement" are >> you *actually* referring to GoCD's concept of resources, or were you >> talking in a generic sense? The answers are assuming you are talking about >> GoCD resources >> <https://docs.gocd.org/current/introduction/concepts_in_go.html#resources> >> but now I am more confused by your shell script. *If you want to use >> resources* (rather than environments) to affect scheduling, while still >> avoiding duplication of your templates, we are suggesting you use a >> parameter like *this*, not put it into some task content. You are >> setting the parameterized value into the field that determines the job's >> scheduling, not something that happens at execution time like a task. But >> again, if your goal is to control scheduling at pipeline level, for all >> jobs in a pipeline, you don't need to use resources, and can just use >> environments as in my earlier example above. >> >> [image: image.png] >> >> -Chad >> >> >> On Sun, Jul 23, 2023 at 8:21 PM Joshua Franta <jos...@pracplay.com> >> wrote: >> >>> appreciate so many responses. i think we're a little apart so i'll take >>> the suggestion to give our example: >>> >>> GROUPA = machines that have less beefy hardware >>> GROUPB = more expensive machines >>> >>> we'd like to: >>> >>> 1. have a couple of pipelines only run on the more expensive >>> machines >>> 2. have all other pipelines run in either group (next available >>> agent) >>> >>> perhaps this was not clear from my previous explanations, but a couple >>> of people have suggested pipeline parameters. >>> >>> EXAMPLE >>> >>> a pipeline parameter is only going to be available to the job after the >>> job has already been assigned an agent, right? >>> >>> so if i have a pipeline called 'Priority' w/a parameter called >>> "group-id" and the pipeline has a 'Job' that is a shell script: >>> >>> ---- >>> ##!/bin/sh >>> >>> agent_resource="$GO_AGENT_RESOURCE_VARIABLE" >>> >>> if ! echo "#{group-id}" |grep -q "$agentr_esource"; then >>> >>> echo "agent can't run #{group-id} pipelines" >>> >>> ## won't this will make my pipeline fail when I want it to simply try >>> another agent? >>> exit 1 >>> fi >>> >>> ---- >>> >>> or perhaps people saying this know of some environment variable that >>> where we can request another agent? >>> >>> obviously pipeline parameters themseles don't do anything, so i'm >>> confused how i can affect assignment in a job that requires an agent before >>> it runs. >>> this 2nd part is what i don't get above >>> >>> appreciate any clarifications or suggestions thx >>> >>> >>> On Sat, Jul 22, 2023 at 9:58 AM Chad Wilson <ch...@thoughtworks.com> >>> wrote: >>> >>>> >>>> On Sat, Jul 22, 2023 at 8:21 PM Joshua Franta <jos...@pracplay.com> >>>> wrote: >>>> >>>>> re: using environments rather than resources... environments can't be >>>>> defined at the pipeline level either though? >>>>> >>>> >>>> A pipeline is *assigned* to 0 or 1 environments (via the Admin > >>>> Environments UI if not using pipelines-as-code) - thus it's at the pipeline >>>> level by definition. It defines a scheduling requirement for all jobs in >>>> that pipeline. Which seems what you asked for with "able to communicate a >>>> resource requirement at the pipeline level" right? >>>> >>>> >>>> or i guess it's more correct to say that using environments is a bit of >>>>> a side-car feature, in that we use interact w/environments through a >>>>> different prisim/ui/config (no biggie) but also seems it's mutually >>>>> exclusive to maximizing overall usage of agents. for us if a given host >>>>> can execute something (a pipeline, a job) it should. and if it can't, it >>>>> shouldn't. >>>>> trying to force a hard divider can be useful for prod/qa staging, but >>>>> it seems to limit just being able to have pipelines declare their needs. >>>>> maybe i'm missing what you're saying but i don't think environments >>>>> are functionally equivalent to resources? >>>>> >>>> >>>> I didn't imply they were functionally equivalent, but I did try to >>>> imply they were a different mechanism of defining a requirement on a job's >>>> scheduling, at the pipeline level. If a pipeline is assigned to an >>>> "environment", its jobs must be scheduled on agents that also declare they >>>> support that "environment". Similarly if a pipeline job declares a resource >>>> requirement, the agent must also have that resource declared for it to be >>>> assigned. This is a very similar, but different level of configuration of a >>>> scheduling requirement, no? >>>> https://docs.gocd.org/current/configuration/managing_environments.html >>>> >>>> Anyway, perhaps I don't understand what you are trying to achieve. If >>>> you are currently trying to "prioritise" pipelines by using resources you >>>> can also "prioritise" pipelines by having pools of agents, say, dedicated >>>> to an environment you call "high-priority". As I said, "Don't need to get >>>> hung up on the name [environment]". >>>> >>>> >>>> >>>>> we use template parameters extensively already. >>>>> eg we even templatize further inside our own jobs by re-using scripts >>>>> that interact with template parameters on most commonly used templates (eg >>>>> our most popular template has maybe 10-15 pipelines). >>>>> however this is more of a job specific thing since it's at the job >>>>> level. >>>>> if you're saying we could change every pipeline to read this at a >>>>> pipeline level is a non trivial change to every job. >>>>> >>>> >>>> You said you had many templates that varied only by the "resources" >>>> field for jobs. If that is the stated problem then parameters are a >>>> possible solution to remove duplication, no? >>>> >>>> >>>>> that's ok but i guess my overall question tho would be that if a given >>>>> job decided it couldn't execute the pipeline parameters... it has no way >>>>> to >>>>> pass the job to another agent? >>>>> >>>> >>>> That's the same problem you have currently if the resource is typoed or >>>> wrong inside the template, no? If the resource requirement has no available >>>> agents, then it can't be scheduled. >>>> >>>> >>>>> in such an example it would just fail the job, no? again maybe i'm >>>>> not following but this seems to not allow the business/value level to >>>>> declare minimum needs >>>>> (environments seem like they are more about maximimal requirements, >>>>> but i'm no expert) >>>>> >>>> >>>> I'm not following what you're trying to say here, sorry. >>>> >>>> Perhaps this would be easier if you gave a specific example of how you >>>> achieve "have some pipelines that are given higher preferences for >>>> agent/build resources" currently, rather than talking in abstract terms? >>>> >>>> -Chad >>>> >>>> >>>> On Sat, Jul 22, 2023 at 6:56 AM Chad Wilson <ch...@thoughtworks.com> >>>>> wrote: >>>>> >>>>>> Have you tried to use "environments" (or a mix of environments and >>>>>> resources) to achieve what you are trying to? >>>>>> >>>>>> When scheduling jobs it's the combination of the resource and the >>>>>> environment that are matched to an agent, but the relevant environment is >>>>>> declared at the pipeline level like you refer to. Don't need to get hung >>>>>> up >>>>>> on the name so much. Yes, you can have "environment variables" attached >>>>>> to >>>>>> an environment and propagate those to all pipelines within it, but you >>>>>> don't have to use them like that. >>>>>> >>>>>> Alternatively, to make the templates less duplicated and allow the >>>>>> resource to flow from the pipeline *using* the template, you could >>>>>> try using template parameters >>>>>> <https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html> >>>>>> in the resources field? e.g #{job-resoure-requirement}? If there are >>>>>> only a >>>>>> small number of different resources used across the stages/jobs, you >>>>>> could >>>>>> use the parameters to "model" this I imagine. >>>>>> >>>>>> -Chad >>>>>> >>>>>> On Sat, Jul 22, 2023 at 6:54 PM Josh <jos...@pracplay.com> wrote: >>>>>> >>>>>>> QUESTION: >>>>>>> >>>>>>> Shouldn't we also be able to communicate a resource requirement at >>>>>>> the pipeline level, and not just inside a single job? >>>>>>> >>>>>>> I get that it definately needs to be at the job level since that's >>>>>>> the smallest unit of work and some machines can't execute certain tasks. >>>>>>> But at the value-stream/pipeline/business level, you also want to be >>>>>>> able to have some pipelines compiling on preferred resources, no? >>>>>>> >>>>>>> >>>>>>> is there a better way to accomplish this? >>>>>>> or perhaps this already is possible and i'm missing it. >>>>>>> i looked closely at the config since sometimes you can do something >>>>>>> simple that is not possible inside the UI, but I'm not seeing it. >>>>>>> >>>>>>> To restate use case: We have some pipelines that are given higher >>>>>>> preferences for agent/build resources. Wanting to do a lot more of >>>>>>> this, >>>>>>> but it's tricky because resources can only be defined at the job level >>>>>>> (in >>>>>>> the UI). Also we use a lot of templates, so having resources at job >>>>>>> level means we end up having lots of alsomost identical templates that >>>>>>> only >>>>>>> vary by the resources used (which somewhat defeats the point of the >>>>>>> templates and the value of gocd in this respect). >>>>>>> >>>>>>> hoping there is a config hack or maybe i'm missinig something. >>>>>>> also if this could be done in a plugin, any color there would be >>>>>>> helpful (and i would make sure it's open sourced if need be). >>>>>>> >>>>>>> thx >>>>>>> >>>>>>> ps i keep using other ci/cd products and gocd is still one of the >>>>>>> all around bests. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "go-cd" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to go-cd+unsubscr...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "go-cd" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> go-cd+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "go-cd" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to go-cd+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "go-cd" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> go-cd+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to go-cd+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com >>> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "go-cd" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> go-cd+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/CAA1RwH8XmX4y11RT2PVXeUxQd8hvK0UBgbcx8ja-MWuOVqpfag%40mail.gmail.com >> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8XmX4y11RT2PVXeUxQd8hvK0UBgbcx8ja-MWuOVqpfag%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to go-cd+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CABr%2BOtoymQdOg0poCtRYDRfxgX4UKfukvtvvpCOZicadaPKWyA%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CABr%2BOtoymQdOg0poCtRYDRfxgX4UKfukvtvvpCOZicadaPKWyA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH-mgC%2Bg0cRa_3rnWak49h4WEeLYck8GimQX2dXG5dxfcQ%40mail.gmail.com.
