Hi Josh,
Thank you for taking the time to share your experience working with GoCD
and the solution that ultimately worked for you.
I, for one, appreciate seeing these loops get closed.
Cheers,
Jason
On Thursday, January 30, 2025 at 5:41:44 a.m. UTC-5 Josh wrote:
>
> TL;DR follow-up post: I wasn't missing anything about artifacts, there's
> no way to process multiple pipelines to the same 'fetch' atm.
>
> Of all the solutions I proposed (as well as other ideas from chad/jason,
> thx again ;).. the simplest was just to integrate our existing pkg
> mechanism tighter and perform this function.
> As mentioned our last pipeline template stage was already packaging
> external artifacts that were more easily attainable (the next best option
> would've been to fetch artifacts directly from gocd, but this would have
> required re-creating things in our existing packaging tool).
>
> Our existing pkg solution (cdpk) is not open source, but it's a minified
> run-anywhere clone of alpine apk, with less features and no binary lock-in
> (it's posix, so you can run on many os-s w/out modification).
>
>
> We modified the build stage to check a pre-existing template parameter-
> lets call it 'upstream-pipelines'- that simply uses cdpk to do it's fetch
> business, extract locally.
> Then put these files into an pipelines/<PIPELINE>/upstream/ folder.
> Then modify the build process to be able to find it's dependencies in this
> location.
>
> This required only modifying and testing one template stage's script, it
> took less than an hour to test.
> About another hour or two to deploy to all pipelines (as they needed their
> builds modified slightly).
>
> Overall it just worked.
> In the first week the failure rate of the test pipelines about halved.
>
> As I mentioned this solved a rarely occuring problem, but as our
> deployment frequency increased it seemed like it was occuring more.
> Plus now we have access to more of our packaging tooling directly in gocd,
> which almost certainly will come in handy.
>
> Thanks for all the assists everyone!
>
> ps always give thanks to chad for his gocd leadership
>
>
> On Monday, December 16, 2024 at 11:08:18 PM UTC-6 Josh wrote:
>
>>
>> we've used gocd for many years and it's a great product.
>>
>> been having an occasional issue that is increasing as we increase
>> deployment frequency.
>> sometimes pipelines briefly get "stuck" on old dlls, meaning that
>> sometimes a downstream pipeline will fail to run because it's been packaged
>> with older dlls. it's only between upstream and downstream pipelines, never
>> in the same pipeline. this occurs infrequently infrequently but perhaps as
>> much as 1 out of every 5 builds.
>>
>> the workaround fix is to run a script on all the agents that periodically
>> refreshes and rebuilds all the pipelines manually. not sure why this works
>> but it always does.
>>
>> haven't been able to figure out the cause, i'm wondering if it's a
>> misunderstanding about artifacts, or otherwise misconfigured artifacts?
>>
>> here's the situation:
>>
>> - we have a pipeline template that runs 8 or 9 pipelines
>> - the template (and thus every pipeline) has 4 stages: prep, build,
>> test and package
>> - prep stage: doesn't do much, mostly just analysis
>> - build stage: pulls code from the repo and builds it, builds
>> artifacts from all binaries built and puts them in gocd at:
>> #{project-name}/build
>> - test stage, fetches those artifacts stored in
>> #{project-name}/build, puts them in local build directory and then runs
>> tests, saves test artifact (not used in build/pkging)
>> - package stage: fetches artifacts stored at stored in
>> #{project-name}/build, puts them in local build directory and packages
>> them
>> up
>>
>> as i say, most of the time it works great but occasionally a mismatch
>> between a previously built upstream pipeline (older version) gets mixed in
>> with a newer pipeline build, and while it compiles when you run something
>> with the mismatched versions it generates a runtime exception.
>>
>> as i'm describing this, i believe the cause might be that since we have
>> multiple agents, a given agent might not always be scheduled to build every
>> pipeline stage.
>>
>> so eg if project2 is downstream from project1:
>>
>> agentA builds project1.verX
>> agentB builds project2.verX
>>
>> [project 2 changes]
>>
>> agentA builds project2.verY
>> agentA still has project1.verX binaries locally, so these get built
>> against project2.verY
>>
>> then when the binaries get packaged up, you get the version mismatch.
>>
>> it seems like what maybe should occur is that we should have pipelines
>> also fetch artifacts from all their upstream dependencies (vs just fetching
>> from their upstream stages, as i described above).
>>
>> however I'm not certain how to do this with pipeline templates, since we
>> could have multiple upstream pipelines to fetch from?
>>
>> so i wanted to add an arbitrary # of 'fetch artifact' tasks to a build
>> stage's pipeline, and then put all it's upstream pipelines as parameters...
>> how can i make the pipeline properly fetch all of:
>>
>> - zero upstream pipelines
>> - one upstream pipeline
>> - multiple upstream pipelines
>>
>> ?
>>
>> Hopefully this makes sense.
>>
>> My Idea:
>>
>> - Is there a way i can somehow create a 'upstream-pipeline-list'
>> parameter, have each pipeline list their upstreams in CSV fasion, and
>> then
>> have gocd fetch EACH of these upstream pipeline builds prior to actually
>> building the stage?
>>
>> To me putting #{upstream-pipeline-list} in a single 'fetch artifact' task
>> doesn't seem right, since the context of the task seems to only take one
>> source location, not multiple.
>>
>> But I misunderstood this before regarding resources, so I figured it was
>> worth asking.
>>
>> Or maybe there's some other even more obvious thing I"m missing (outside
>> of a monorepo, we can't use a monorepo here at least not presently). What
>> is the 'GOCD WAY' to handle this properly?
>>
>> appreciate any assistance
>>
>> -j
>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/go-cd/e82f6c56-06de-4b95-8abd-77beb668686bn%40googlegroups.com.
