On Sat, Jul 22, 2023 at 8:21 PM Joshua Franta <jos...@pracplay.com> wrote:
> re: using environments rather than resources... environments can't be > defined at the pipeline level either though? > A pipeline is *assigned* to 0 or 1 environments (via the Admin > Environments UI if not using pipelines-as-code) - thus it's at the pipeline level by definition. It defines a scheduling requirement for all jobs in that pipeline. Which seems what you asked for with "able to communicate a resource requirement at the pipeline level" right? or i guess it's more correct to say that using environments is a bit of a > side-car feature, in that we use interact w/environments through a > different prisim/ui/config (no biggie) but also seems it's mutually > exclusive to maximizing overall usage of agents. for us if a given host > can execute something (a pipeline, a job) it should. and if it can't, it > shouldn't. > trying to force a hard divider can be useful for prod/qa staging, but it > seems to limit just being able to have pipelines declare their needs. > maybe i'm missing what you're saying but i don't think environments are > functionally equivalent to resources? > I didn't imply they were functionally equivalent, but I did try to imply they were a different mechanism of defining a requirement on a job's scheduling, at the pipeline level. If a pipeline is assigned to an "environment", its jobs must be scheduled on agents that also declare they support that "environment". Similarly if a pipeline job declares a resource requirement, the agent must also have that resource declared for it to be assigned. This is a very similar, but different level of configuration of a scheduling requirement, no? https://docs.gocd.org/current/configuration/managing_environments.html Anyway, perhaps I don't understand what you are trying to achieve. If you are currently trying to "prioritise" pipelines by using resources you can also "prioritise" pipelines by having pools of agents, say, dedicated to an environment you call "high-priority". As I said, "Don't need to get hung up on the name [environment]". > we use template parameters extensively already. > eg we even templatize further inside our own jobs by re-using scripts that > interact with template parameters on most commonly used templates (eg our > most popular template has maybe 10-15 pipelines). > however this is more of a job specific thing since it's at the job level. > if you're saying we could change every pipeline to read this at a pipeline > level is a non trivial change to every job. > You said you had many templates that varied only by the "resources" field for jobs. If that is the stated problem then parameters are a possible solution to remove duplication, no? > that's ok but i guess my overall question tho would be that if a given job > decided it couldn't execute the pipeline parameters... it has no way to > pass the job to another agent? > That's the same problem you have currently if the resource is typoed or wrong inside the template, no? If the resource requirement has no available agents, then it can't be scheduled. > in such an example it would just fail the job, no? again maybe i'm not > following but this seems to not allow the business/value level to declare > minimum needs > (environments seem like they are more about maximimal requirements, but > i'm no expert) > I'm not following what you're trying to say here, sorry. Perhaps this would be easier if you gave a specific example of how you achieve "have some pipelines that are given higher preferences for agent/build resources" currently, rather than talking in abstract terms? -Chad On Sat, Jul 22, 2023 at 6:56 AM Chad Wilson <ch...@thoughtworks.com> wrote: > >> Have you tried to use "environments" (or a mix of environments and >> resources) to achieve what you are trying to? >> >> When scheduling jobs it's the combination of the resource and the >> environment that are matched to an agent, but the relevant environment is >> declared at the pipeline level like you refer to. Don't need to get hung up >> on the name so much. Yes, you can have "environment variables" attached to >> an environment and propagate those to all pipelines within it, but you >> don't have to use them like that. >> >> Alternatively, to make the templates less duplicated and allow the >> resource to flow from the pipeline *using* the template, you could try >> using template parameters >> <https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html> >> in the resources field? e.g #{job-resoure-requirement}? If there are only a >> small number of different resources used across the stages/jobs, you could >> use the parameters to "model" this I imagine. >> >> -Chad >> >> On Sat, Jul 22, 2023 at 6:54 PM Josh <jos...@pracplay.com> wrote: >> >>> QUESTION: >>> >>> Shouldn't we also be able to communicate a resource requirement at the >>> pipeline level, and not just inside a single job? >>> >>> I get that it definately needs to be at the job level since that's the >>> smallest unit of work and some machines can't execute certain tasks. >>> But at the value-stream/pipeline/business level, you also want to be >>> able to have some pipelines compiling on preferred resources, no? >>> >>> >>> is there a better way to accomplish this? >>> or perhaps this already is possible and i'm missing it. >>> i looked closely at the config since sometimes you can do something >>> simple that is not possible inside the UI, but I'm not seeing it. >>> >>> To restate use case: We have some pipelines that are given higher >>> preferences for agent/build resources. Wanting to do a lot more of this, >>> but it's tricky because resources can only be defined at the job level (in >>> the UI). Also we use a lot of templates, so having resources at job >>> level means we end up having lots of alsomost identical templates that only >>> vary by the resources used (which somewhat defeats the point of the >>> templates and the value of gocd in this respect). >>> >>> hoping there is a config hack or maybe i'm missinig something. >>> also if this could be done in a plugin, any color there would be helpful >>> (and i would make sure it's open sourced if need be). >>> >>> thx >>> >>> ps i keep using other ci/cd products and gocd is still one of the all >>> around bests. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to go-cd+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com >>> <https://groups.google.com/d/msgid/go-cd/a9a4ba2c-b1c9-4202-9408-3e2566929b59n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "go-cd" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/go-cd/_j5JGmoA2kI/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> go-cd+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com >> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8zGo6mu0ss0jCCyw0D7Hw4JOwEwfcfNu20yqo0aRRdWw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to go-cd+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CABr%2BOtrG%2B0X3y4B6AWnN7N0K-OSpcKb4KdG-LbS8fCnMOR8Zdw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH8d2amyXbcEhe8hOnbm6BmS-moT9u7Oo%3D4zzU%2BoXfnekQ%40mail.gmail.com.
