You would have agents on the various machines, tag the agents with specific
resource tags, and tag specific jobs with those resource tags.
— Sriram
On Sun, 23 Jul 2023 at 8:21 PM, Joshua Franta <jos...@pracplay.com> wrote:
> appreciate so many responses. i think we're a little apart so i'll take
> the suggestion to give our example:
>
> GROUPA = machines that have less beefy hardware
> GROUPB = more expensive machines
>
> we'd like to:
>
> 1. have a couple of pipelines only run on the more expensive machines
> 2. have all other pipelines run in either group (next available agent)
>
> perhaps this was not clear from my previous explanations, but a couple of
> people have suggested pipeline parameters.
>
> EXAMPLE
>
> a pipeline parameter is only going to be available to the job after the
> job has already been assigned an agent, right?
>
> so if i have a pipeline called 'Priority' w/a parameter called "group-id"
> and the pipeline has a 'Job' that is a shell script:
>
> ----
> ##!/bin/sh
>
> agent_resource="$GO_AGENT_RESOURCE_VARIABLE"
>
> if ! echo "#{group-id}" |grep -q "$agentr_esource"; then
>
> echo "agent can't run #{group-id} pipelines"
>
> ## won't this will make my pipeline fail when I want it to simply try
> another agent?
> exit 1
> fi
>
> ----
>
> or perhaps people saying this know of some environment variable that where
> we can request another agent?
>
> obviously pipeline parameters themseles don't do anything, so i'm confused
> how i can affect assignment in a job that requires an agent before it runs.
> this 2nd part is what i don't get above
>
> appreciate any clarifications or suggestions thx
>
>
> On Sat, Jul 22, 2023 at 9:58 AM Chad Wilson <ch...@thoughtworks.com>
> wrote:
>
>>
>> On Sat, Jul 22, 2023 at 8:21 PM Joshua Franta <jos...@pracplay.com>
>> wrote:
>>
>>> re: using environments rather than resources... environments can't be
>>> defined at the pipeline level either though?
>>>
>>
>> A pipeline is *assigned* to 0 or 1 environments (via the Admin >
>> Environments UI if not using pipelines-as-code) - thus it's at the pipeline
>> level by definition. It defines a scheduling requirement for all jobs in
>> that pipeline. Which seems what you asked for with "able to communicate a
>> resource requirement at the pipeline level" right?
>>
>>
>> or i guess it's more correct to say that using environments is a bit of a
>>> side-car feature, in that we use interact w/environments through a
>>> different prisim/ui/config (no biggie) but also seems it's mutually
>>> exclusive to maximizing overall usage of agents. for us if a given host
>>> can execute something (a pipeline, a job) it should. and if it can't, it
>>> shouldn't.
>>> trying to force a hard divider can be useful for prod/qa staging, but it
>>> seems to limit just being able to have pipelines declare their needs.
>>> maybe i'm missing what you're saying but i don't think environments are
>>> functionally equivalent to resources?
>>>
>>
>> I didn't imply they were functionally equivalent, but I did try to imply
>> they were a different mechanism of defining a requirement on a job's
>> scheduling, at the pipeline level. If a pipeline is assigned to an
>> "environment", its jobs must be scheduled on agents that also declare they
>> support that "environment". Similarly if a pipeline job declares a resource
>> requirement, the agent must also have that resource declared for it to be
>> assigned. This is a very similar, but different level of configuration of a
>> scheduling requirement, no?
>> https://docs.gocd.org/current/configuration/managing_environments.html
>>
>> Anyway, perhaps I don't understand what you are trying to achieve. If you
>> are currently trying to "prioritise" pipelines by using resources you can
>> also "prioritise" pipelines by having pools of agents, say, dedicated to an
>> environment you call "high-priority". As I said, "Don't need to get hung up
>> on the name [environment]".
>>
>>
>>
>>> we use template parameters extensively already.
>>> eg we even templatize further inside our own jobs by re-using scripts
>>> that interact with template parameters on most commonly used templates (eg
>>> our most popular template has maybe 10-15 pipelines).
>>> however this is more of a job specific thing since it's at the job level.
>>> if you're saying we could change every pipeline to read this at a
>>> pipeline level is a non trivial change to every job.
>>>
>>
>> You said you had many templates that varied only by the "resources" field
>> for jobs. If that is the stated problem then parameters are a possible
>> solution to remove duplication, no?
>>
>>
>>> that's ok but i guess my overall question tho would be that if a given
>>> job decided it couldn't execute the pipeline parameters... it has no way to
>>> pass the job to another agent?
>>>
>>
>> That's the same problem you have currently if the resource is typoed or
>> wrong inside the template, no? If the resource requirement has no available
>> agents, then it can't be scheduled.
>>
>>
>>> in such an example it would just fail the job, no? again maybe i'm not
>>> following but this seems to not allow the business/value level to declare
>>> minimum needs
>>> (environments seem like they are more about maximimal requirements, but
>>> i'm no expert)
>>>
>>
>> I'm not following what you're trying to say here, sorry.
>>
>> Perhaps this would be easier if you gave a specific example of how you
>> achieve "have some pipelines that are given higher preferences for
>> agent/build resources" currently, rather than talking in abstract terms?
>>
>> -Chad
>>
>>
>> On Sat, Jul 22, 2023 at 6:56 AM Chad Wilson <ch...@thoughtworks.com>
>>> wrote:
>>>
>>>> Have you tried to use "environments" (or a mix of environments and
>>>> resources) to achieve what you are trying to?
>>>>
>>>> When scheduling jobs it's the combination of the resource and the
>>>> environment that are matched to an agent, but the relevant environment is
>>>> declared at the pipeline level like you refer to. Don't need to get hung up
>>>> on the name so much. Yes, you can have "environment variables" attached to
>>>> an environment and propagate those to all pipelines within it, but you
>>>> don't have to use them like that.
>>>>
>>>> Alternatively, to make the templates less duplicated and allow the
>>>> resource to flow from the pipeline *using* the template, you could try
>>>> using template parameters
>>>> <https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html>
>>>> in the resources field? e.g #{job-resoure-requirement}? If there are only a
>>>> small number of different resources used across the stages/jobs, you could
>>>> use the parameters to "model" this I imagine.
>>>>
>>>> -Chad
>>>>
>>>> On Sat, Jul 22, 2023 at 6:54 PM Josh <jos...@pracplay.com> wrote:
>>>>
>>>>> QUESTION:
>>>>>
>>>>> Shouldn't we also be able to communicate a resource requirement at the
>>>>> pipeline level, and not just inside a single job?
>>>>>
>>>>> I get that it definately needs to be at the job level since that's the
>>>>> smallest unit of work and some machines can't execute certain tasks.
>>>>> But at the value-stream/pipeline/business level, you also want to be
>>>>> able to have some pipelines compiling on preferred resources, no?
>>>>>
>>>>>
>>>>> is there a better way to accomplish this?
>>>>> or perhaps this already is possible and i'm missing it.
>>>>> i looked closely at the config since sometimes you can do something
>>>>> simple that is not possible inside the UI, but I'm not seeing it.
>>>>>
>>>>> To restate use case: We have some pipelines that are given higher
>>>>> preferences for agent/build resources. Wanting to do a lot more of this,
>>>>> but it's tricky because resources can only be defined at the job level (in
>>>>> the UI). Also we use a lot of templates, so having resources at job
>>>>> level means we end up having lots of alsomost identical templates that
>>>>> only
>>>>> vary by the resources used (which somewhat defeats the point of the
>>>>> templates and the value of gocd in this respect).
>>>>>
>>>>> hoping there is a config hack or maybe i'm missinig something.
>>>>> also if this could be done in a plugin, any color there would be
>>>>> helpful (and i would make sure it's open sourced if need be).
>>>>>
>>>>> thx
>>>>>
>>>>> ps i keep using other ci/cd products and gocd is still one of the all
>>>>> around bests.
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "go-cd" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to go-cd+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "go-cd" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> go-cd+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to go-cd+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "go-cd" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> go-cd+unsubscr...@googlegroups.com.
>>
> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com
>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com
> <https://groups.google.com/d/msgid/go-cd/CABr%2BOtr%3D-R3fKE0devgPVKgNbHGVnsn5E5OwxuP%3DMt1No_RNdg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/go-cd/CANiY96Yn2LZt4bZYJAVb80dTq%3DwgV4CHbSbBnGtxVtu2wJVo-g%40mail.gmail.com.
