Thanks Chad and M, I would look into some of the options you have provided. Good to know that I am not missing something out of the box and this needs some creative thinking.
All pipelines under consideration differ just in the name and the material branch names. Everything else is exactly the same. We use this pretty much only through the UI, not using API or pipeline as code as yet. One option we are also considering is to templatise the cruise-config.xml and generate some relevant sections and re-upload when such need arises but I think the options suggested by you are better things to consider. I would keep the group posted as we solve this. Warm regards, Pankaj On Thursday, March 10, 2022 at 9:19:51 PM UTC+5:30 [email protected] wrote: > I've used the JSON plugin and a jsonnet (https://jsonnet.org/) library to > achieve this before. > > It worked pretty well, but did require some discipline on behalf of the > programmers, i.e. to provide make targets to build, push to the container > repo and deploy to the test cluster, which had uniform names etc. > > M > > On Thursday, March 10, 2022 at 2:37:51 PM UTC Chad Wilson wrote: > >> Hi Pankaj >> >> How do you maintain the pipeline config now? Via the UI? Via API? Via one >> of the pipelines-as-code/confg repo plugins? >> >> In any case, I would say it depends a bit on >> >> - what the content of your pipelines are >> - how much variation there is between these pipelines >> - when you say developers "maintain" their pipelines, does that >> include changing its config and definition? Or just making sure it's >> green? >> - when folks need to change the pipelines, will it be acceptable (or >> desirable) for them to be forced to do it via source control? >> >> ...but in terms of general approaches to reduce duplication and factor >> out commonality in pipelines, there are a few approaches, which have >> upsides and downsides. Perhaps you can take a look at them and ask some >> more specific questions based on your needs? >> >> >> - *Pipeline templates* >> https://docs.gocd.org/current/configuration/pipeline_templates.html - >> maintained via UI|API IIRC, but not really customisable by those >> pipelines >> that use the template, except by parameters/env/materials >> - One of the* Pipelines as Code* plugins (aka Config Repository >> plugins) >> https://docs.gocd.org/current/advanced_usage/pipelines_as_code.html >> - At least with Pipelines-as-Code you can find-and-replace >> en-masse, but... >> - With the YAML plugin you can use YAML anchors to factor out >> commonality in a limited way. >> - With either the YAML or JSON plugins you could conceptually >> template the YAML/JSON and generate it using a tool as well, although >> I >> have never done this and have no opinion on whether it's a good idea >> - With the Groovy plugin you can write Groovy script to express >> your pipelines using a DSL, which could potentially allow you to model >> the >> differences between them, dependencies etc. >> - This may or may not work for depending on how you plan to >> supporting "self-service" and developers making changes to their own >> pipelines, unless you work with some kind of PR approval model into a >> Git >> repository you centrally maintain >> - Write some tooling against the *GoCD API * >> https://api.gocd.org/current/#pipeline-config >> - There were some client libraries to support this such as gomatic >> <https://github.com/gocd-contrib/gomatic>, but I haven't >> personally used those, and not sure what state they are in. >> >> -Chad >> >> >> >> On Thu, Mar 10, 2022 at 8:12 PM '[email protected]' via go-cd < >> [email protected]> wrote: >> >>> Hi community, >>> >>> Every developer in our organization gets a pipeline of his own to do the >>> full build and test process. Very frequently we are making changes that >>> have to be replicated across all pipelines, for example adding a new >>> material, adding an additional step, minor changes in the command syntax of >>> one of the steps in pipeline. Since each developer is supposed to maintain >>> their own pipeline, they realize it only when the pipeline breaks because >>> of this change and we are in a continuous maintenance loop. >>> >>> Our need is to easily replicate these changes across all pipelines. Most >>> configuration is same except the name of the pipeline is different. Whats >>> the community recommendation for this. How else to manage this situation? >>> >>> Warm regards, >>> Pankaj >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/f66ff769-da55-4383-9d55-7d5c201153fdn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/go-cd/f66ff769-da55-4383-9d55-7d5c201153fdn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/c430da12-da98-4991-bd6f-cc6a9c963d0dn%40googlegroups.com.
