Great to hear. Really appreciate the feedback! On Thu, 23 May 2024, 15:39 'Hans Dampf' via go-cd, <go-cd@googlegroups.com> wrote:
> So we have been running the new version for a couple of days now, and I > can't see any further problems with it. The performance is great with >1000 > Pipelines now. > It is a really great improvement. > > Wolfgang Achinger schrieb am Montag, 13. Mai 2024 um 10:30:42 UTC+2: > >> I'm currently in the process of upgrading the server and agents of all >> our gocd setups. We will monitor it the next few days and I will come back >> if we notice anything >> >> Am Mo., 13. Mai 2024 um 10:24 Uhr schrieb Chad Wilson < >> ch...@thoughtworks.com>: >> >>> Great to hear - back to how it was *supposed* to behave! I hope it >>> hasn't caused any other regressions 🙏 >>> >>> Now that this is cleaned up, let me know if it exposes any other >>> unexpected weird slowness you can't get to the bottom of and I'll see if I >>> can chip away at the other various niggles. >>> >>> -Chad >>> >>> On Mon, May 13, 2024 at 4:10 PM 'Wolfgang Achinger' via go-cd < >>> go...@googlegroups.com> wrote: >>> >>>> Dude the fix is amazing !!!!! >>>> >>>> Am Mo., 13. Mai 2024 um 04:05 Uhr schrieb Chad Wilson < >>>> ch...@thoughtworks.com>: >>>> >>>>> Apologies for the slow release of this (been rather busy personally) >>>>> but 24.1.0 is out with what I think should be a fix for this issue. >>>>> >>>>> If you have any feedback it'd be appreciated. >>>>> >>>>> -Chad >>>>> >>>>> >>>>> On Mon, 19 Feb 2024, 15:15 'Wolfgang Achinger' via go-cd, < >>>>> go...@googlegroups.com> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> this is actually incredible news for us. We will look forward to the >>>>>> release with the fix. >>>>>> Thanks for the support. >>>>>> >>>>>> The workarounds seem not to be viable for us, since we use the >>>>>> environment for a lot of global variables and customizations. >>>>>> >>>>>> Regards >>>>>> >>>>>> Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson < >>>>>> ch...@thoughtworks.com>: >>>>>> >>>>>>> Hiya folks >>>>>>> >>>>>>> I've been able to replicate this problem and should be able to fix >>>>>>> it for a subsequent release - thanks for the help debugging. >>>>>>> >>>>>>> The problem appears to arrive when there is a large number of >>>>>>> pipelines* mapped to an environment;* and also a large number of >>>>>>> agents for that environment. The logic for calculating the API response >>>>>>> agents > environments is accidentally very, very inefficient (I think >>>>>>> it's >>>>>>> O(n^2 x m^2) or something crazy. I replicated something similar to what >>>>>>> you >>>>>>> describe with 5,000 pipelines and 60 or so agents, all mapped into the >>>>>>> same, single, logical environment. >>>>>>> >>>>>>> [image: image.png] >>>>>>> >>>>>>> >>>>>>> In your case if you have all say 1,690 pipelines mapped to a single >>>>>>> environment (from your stats below), and all of your 95 agents are in >>>>>>> the >>>>>>> same environment, you'd definitely trigger this issue. I can't tell >>>>>>> exactly >>>>>>> from what you have shared how the pipelines and agents are mapped to >>>>>>> environments, so this is a guess - can you confirm how many agents and >>>>>>> pipelines are mapped to the environment below? >>>>>>> >>>>>>> "Number of pipelines": 1690, >>>>>>> "Number of environments": 1, >>>>>>> "Number of agents": 95, >>>>>>> >>>>>>> >>>>>>> If it's the same problem, you will probably find that *untagging >>>>>>> the agents from the environment *also has a similar speed-up effect >>>>>>> to deleting all of the agents (although then the pipelines requiring >>>>>>> that >>>>>>> environment won't schedule either, obviously). >>>>>>> >>>>>>> Another workaround in the meantime, *if you don't rely on the >>>>>>> environment* >>>>>>> >>>>>>> - to define environment variables/secure environment variables >>>>>>> that apply across all pipelines/jobs >>>>>>> - to affect whether jobs are scheduled to special agents >>>>>>> >>>>>>> ... may be to untag all pipelines and agents from the environment >>>>>>> you use and just use the default/empty environment. >>>>>>> >>>>>>> -Chad >>>>>>> >>>>>>> On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd < >>>>>>> go...@googlegroups.com> wrote: >>>>>>> >>>>>>>> > By 'resources' I am referring to the GoCD functionality where you >>>>>>>> can tag agents with resources that they offer, which are then matched >>>>>>>> to >>>>>>>> pipeline jobs that say they *require* those resources to run as >>>>>>>> part of agent assignment. >>>>>>>> 10 Agents have 5 resources attached >>>>>>>> 85 have 1 resource attached >>>>>>>> >>>>>>>> We use the resources to different special agents. They do the same >>>>>>>> as the rest, but they are placed in dedicated networks. >>>>>>>> >>>>>>>> > To make sure I understand you, are you saying that the problem >>>>>>>> has been here for the last year, perhaps gradually getting worse a >>>>>>>> story >>>>>>>> add more agents or pipelines - but not an issue suddenly created after >>>>>>>> a >>>>>>>> particular upgrade or change? >>>>>>>> That's correct. It's more an over-time issue than a sudden issue. >>>>>>>> >>>>>>>> I sent the additional information out, but not directly, they come >>>>>>>> from a different mail address over a secure transfer method. >>>>>>>> >>>>>>>> Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson < >>>>>>>> ch...@thoughtworks.com>: >>>>>>>> >>>>>>>>> Cool, thanks! Just trying to gather enough information to see if I >>>>>>>>> can replicate or find the issue in a dedicated chunk of time this >>>>>>>>> weekend. >>>>>>>>> >>>>>>>>> You can email it to me, and/or encrypt with my GPG key if you'd >>>>>>>>> like ( >>>>>>>>> https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc >>>>>>>>> ) >>>>>>>>> >>>>>>>>> By 'resources' I am referring to the GoCD functionality where you >>>>>>>>> can tag agents with resources that they offer, which are then matched >>>>>>>>> to >>>>>>>>> pipeline jobs that say they *require* those resources to run as >>>>>>>>> part of agent assignment. >>>>>>>>> >>>>>>>>> > No we use this setup no for about a year, patch the system on a >>>>>>>>> regular basis including the latest gocd stable version. >>>>>>>>> >>>>>>>>> To make sure I understand you, are you saying that the problem has >>>>>>>>> been here for the last year, perhaps gradually getting worse a story >>>>>>>>> add >>>>>>>>> more agents or pipelines - but not an issue suddenly created after a >>>>>>>>> particular upgrade or change? >>>>>>>>> >>>>>>>>> -Chad >>>>>>>>> >>>>>>>>> On Fri, 16 Feb 2024, 00:29 'Wolfgang Achinger' via go-cd, < >>>>>>>>> go...@googlegroups.com> wrote: >>>>>>>>> >>>>>>>>>> > And how many resources are defined across the agents? >>>>>>>>>> What exactly do you mean here? System resources? XMS XMX Values >>>>>>>>>> of java ? >>>>>>>>>> >>>>>>>>>> - Is this a problem that has always been there, or something that >>>>>>>>>> has changed with a GoCD version or other change in environment? >>>>>>>>>> No we use this setup no for about a year, patch the system on a >>>>>>>>>> regular basis including the latest gocd stable version. >>>>>>>>>> >>>>>>>>>> - Is it faster when the server is restarted, and gets slower over >>>>>>>>>> time (or the same after a restart)? >>>>>>>>>> No a restart does not affect the speed at all. It stays constant >>>>>>>>>> >>>>>>>>>> - Why do you feel it is the # of jobs/stages the agents have >>>>>>>>>> processed that is a key factor, rather than simply the # of agents >>>>>>>>>> or some >>>>>>>>>> other agent configuration factor? >>>>>>>>>> I don't know it was more a wild guess. After later testing, i >>>>>>>>>> don't think this anymore. I cleaned up some tables and reduced the >>>>>>>>>> agent >>>>>>>>>> history visible in the GUI, but this did not affect the speed (Well, >>>>>>>>>> it >>>>>>>>>> increased the speed of the listing of the agent history itself but >>>>>>>>>> not the >>>>>>>>>> loading time of the popups). >>>>>>>>>> >>>>>>>>>> If it is ok i will send the support output directly your our >>>>>>>>>> mailadress so it will not get shared in the thread. >>>>>>>>>> >>>>>>>>>> Am Do., 15. Feb. 2024 um 15:50 Uhr schrieb Chad Wilson < >>>>>>>>>> ch...@thoughtworks.com>: >>>>>>>>>> >>>>>>>>>>> And how many resources are defined across the agents? >>>>>>>>>>> >>>>>>>>>>> Can you please answer the earlier questions I asked as well? >>>>>>>>>>> It's rather difficult to efficiently help if you don't respond to >>>>>>>>>>> the >>>>>>>>>>> questions that characterise the problem from a maintainer >>>>>>>>>>> perspective. :-) >>>>>>>>>>> >>>>>>>>>>> - Is this a problem that has always been there, or something >>>>>>>>>>> that has changed with a GoCD version or other change in environment? >>>>>>>>>>> - Is it faster when the server is restarted, and gets slower >>>>>>>>>>> over time (or the same after a restart)? >>>>>>>>>>> - Why do you feel it is the # of jobs/stages the agents have >>>>>>>>>>> processed that is a key factor, rather than simply the # of agents >>>>>>>>>>> or some >>>>>>>>>>> other agent configuration factor? >>>>>>>>>>> >>>>>>>>>>> Additionally, can you share a redacted output from /go/api/support >>>>>>>>>>> ? You can enter the URL in the browser when logged in as an admin. >>>>>>>>>>> Be >>>>>>>>>>> careful of the "Runtime Information" and "System Health Information" >>>>>>>>>>> sections when sharing. These are the two main places which might >>>>>>>>>>> leak >>>>>>>>>>> unintentional information from your setup. Redact the individual >>>>>>>>>>> values >>>>>>>>>>> which feel sensitive to you. >>>>>>>>>>> >>>>>>>>>>> -Chad >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Feb 15, 2024 at 10:17 PM 'Wolfgang Achinger' via go-cd < >>>>>>>>>>> go...@googlegroups.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> 1 environment >>>>>>>>>>>> 164 materials >>>>>>>>>>>> 0 elastic agents >>>>>>>>>>>> 2 config repos >>>>>>>>>>>> 0 artifact stores >>>>>>>>>>>> 0 pluggable scms >>>>>>>>>>>> >>>>>>>>>>>> Am Do., 15. Feb. 2024 um 15:01 Uhr schrieb Chad Wilson < >>>>>>>>>>>> ch...@thoughtworks.com>: >>>>>>>>>>>> >>>>>>>>>>>>> How many distinct environments and resources do you have >>>>>>>>>>>>> across these 1200 pipelines, roughly? >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Feb 15, 2024 at 5:38 PM 'Wolfgang Achinger' via go-cd < >>>>>>>>>>>>> go...@googlegroups.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Additional information >>>>>>>>>>>>>> Since the pipelines are configured via ~150 yaml files. >>>>>>>>>>>>>> I tested it now with one big, merged config file with all >>>>>>>>>>>>>> pipelines >>>>>>>>>>>>>> But this did not change anything, >>>>>>>>>>>>>> performance slow. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>> Google Groups "go-cd" group. >>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>>> it, send an email to go-cd+un...@googlegroups.com. >>>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com >>>>>>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>>> . >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to a >>>>>>>>>>>>> topic in the Google Groups "go-cd" group. >>>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe >>>>>>>>>>>>> . >>>>>>>>>>>>> To unsubscribe from this group and all its topics, send an >>>>>>>>>>>>> email to go-cd+un...@googlegroups.com. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com >>>>>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>> . >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "go-cd" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to go-cd+un...@googlegroups.com. >>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLBiz4s%3DqpKr1Yrgw1TSyfRePkcdGXPesfrmAiu2e9aN6g%40mail.gmail.com >>>>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLBiz4s%3DqpKr1Yrgw1TSyfRePkcdGXPesfrmAiu2e9aN6g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>> . >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>> in the Google Groups "go-cd" group. >>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>>> to go-cd+un...@googlegroups.com. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH-jix9BM1k24yCedriGXryO7zRiMtEXaOxOmqZ-PamU3g%40mail.gmail.com >>>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH-jix9BM1k24yCedriGXryO7zRiMtEXaOxOmqZ-PamU3g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "go-cd" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to go-cd+un...@googlegroups.com. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLDdSv8pY0w6ZzJF8Uv_gSx_Y_APdLjF3JHXGcxAE-rhrQ%40mail.gmail.com >>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLDdSv8pY0w6ZzJF8Uv_gSx_Y_APdLjF3JHXGcxAE-rhrQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>> the Google Groups "go-cd" group. >>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>> to go-cd+un...@googlegroups.com. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH9HfWexWAHGbTLQvuvZdTYRonH-bJ_OpcSkrvd3RSCirQ%40mail.gmail.com >>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH9HfWexWAHGbTLQvuvZdTYRonH-bJ_OpcSkrvd3RSCirQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "go-cd" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to go-cd+un...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLAbXaud2BiWWYYNXK5AeVf8JHp7Sw6Hioe3OkPzm8Ga%2Bw%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLAbXaud2BiWWYYNXK5AeVf8JHp7Sw6Hioe3OkPzm8Ga%2Bw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "go-cd" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> go-cd+un...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH9yFok2t7ZVnmdvLfsPG-v1JD%3DfUm2_GwJ%2B3j%3Dg5WLDBw%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH9yFok2t7ZVnmdvLfsPG-v1JD%3DfUm2_GwJ%2B3j%3Dg5WLDBw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "go-cd" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to go-cd+un...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLAJ7MoasJowhsq24DwNG9rOyiYTkaAjqnSnoQm8MUsK-A%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLAJ7MoasJowhsq24DwNG9rOyiYTkaAjqnSnoQm8MUsK-A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "go-cd" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> go-cd+un...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8rqwVfS4oH9s7XMNHe6o1fW_FUtEPcL%3D-uZaNKFS6reA%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8rqwVfS4oH9s7XMNHe6o1fW_FUtEPcL%3D-uZaNKFS6reA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "go-cd" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to go-cd+un...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/CANhjCLDd28UMfH1OJYnB%2BXR8bf9SHfcgST8aLqyBzTS_m7KcbA%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLDd28UMfH1OJYnB%2BXR8bf9SHfcgST8aLqyBzTS_m7KcbA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "go-cd" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> go-cd+un...@googlegroups.com. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CAA1RwH_85-CcLUWMLhEzfmJqkPMP6HLn%3D_j6mxg7%3DMQZnpXxzw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH_85-CcLUWMLhEzfmJqkPMP6HLn%3D_j6mxg7%3DMQZnpXxzw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to go-cd+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/de7512e8-aec8-4802-87bc-b42fc527f453n%40googlegroups.com > <https://groups.google.com/d/msgid/go-cd/de7512e8-aec8-4802-87bc-b42fc527f453n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH_3_3-7VfFtmxaLUnY1%2BHZVgAb3LrAwZk2QjeATLnKhEA%40mail.gmail.com.
