Great to hear - back to how it was *supposed* to behave! I hope it hasn't caused any other regressions 🙏
Now that this is cleaned up, let me know if it exposes any other unexpected weird slowness you can't get to the bottom of and I'll see if I can chip away at the other various niggles. -Chad On Mon, May 13, 2024 at 4:10 PM 'Wolfgang Achinger' via go-cd < go-cd@googlegroups.com> wrote: > Dude the fix is amazing !!!!! > > Am Mo., 13. Mai 2024 um 04:05 Uhr schrieb Chad Wilson < > ch...@thoughtworks.com>: > >> Apologies for the slow release of this (been rather busy personally) but >> 24.1.0 is out with what I think should be a fix for this issue. >> >> If you have any feedback it'd be appreciated. >> >> -Chad >> >> >> On Mon, 19 Feb 2024, 15:15 'Wolfgang Achinger' via go-cd, < >> go-cd@googlegroups.com> wrote: >> >>> Hello, >>> >>> this is actually incredible news for us. We will look forward to the >>> release with the fix. >>> Thanks for the support. >>> >>> The workarounds seem not to be viable for us, since we use the >>> environment for a lot of global variables and customizations. >>> >>> Regards >>> >>> Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson < >>> ch...@thoughtworks.com>: >>> >>>> Hiya folks >>>> >>>> I've been able to replicate this problem and should be able to fix it >>>> for a subsequent release - thanks for the help debugging. >>>> >>>> The problem appears to arrive when there is a large number of pipelines* >>>> mapped to an environment;* and also a large number of agents for that >>>> environment. The logic for calculating the API response agents > >>>> environments is accidentally very, very inefficient (I think it's O(n^2 x >>>> m^2) or something crazy. I replicated something similar to what you >>>> describe with 5,000 pipelines and 60 or so agents, all mapped into the >>>> same, single, logical environment. >>>> >>>> [image: image.png] >>>> >>>> >>>> In your case if you have all say 1,690 pipelines mapped to a single >>>> environment (from your stats below), and all of your 95 agents are in the >>>> same environment, you'd definitely trigger this issue. I can't tell exactly >>>> from what you have shared how the pipelines and agents are mapped to >>>> environments, so this is a guess - can you confirm how many agents and >>>> pipelines are mapped to the environment below? >>>> >>>> "Number of pipelines": 1690, >>>> "Number of environments": 1, >>>> "Number of agents": 95, >>>> >>>> >>>> If it's the same problem, you will probably find that *untagging the >>>> agents from the environment *also has a similar speed-up effect to >>>> deleting all of the agents (although then the pipelines requiring that >>>> environment won't schedule either, obviously). >>>> >>>> Another workaround in the meantime, *if you don't rely on the >>>> environment* >>>> >>>> - to define environment variables/secure environment variables that >>>> apply across all pipelines/jobs >>>> - to affect whether jobs are scheduled to special agents >>>> >>>> ... may be to untag all pipelines and agents from the environment you >>>> use and just use the default/empty environment. >>>> >>>> -Chad >>>> >>>> On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd < >>>> go-cd@googlegroups.com> wrote: >>>> >>>>> > By 'resources' I am referring to the GoCD functionality where you >>>>> can tag agents with resources that they offer, which are then matched to >>>>> pipeline jobs that say they *require* those resources to run as part >>>>> of agent assignment. >>>>> 10 Agents have 5 resources attached >>>>> 85 have 1 resource attached >>>>> >>>>> We use the resources to different special agents. They do the same as >>>>> the rest, but they are placed in dedicated networks. >>>>> >>>>> > To make sure I understand you, are you saying that the problem has >>>>> been here for the last year, perhaps gradually getting worse a story add >>>>> more agents or pipelines - but not an issue suddenly created after a >>>>> particular upgrade or change? >>>>> That's correct. It's more an over-time issue than a sudden issue. >>>>> >>>>> I sent the additional information out, but not directly, they come >>>>> from a different mail address over a secure transfer method. >>>>> >>>>> Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson < >>>>> ch...@thoughtworks.com>: >>>>> >>>>>> Cool, thanks! Just trying to gather enough information to see if I >>>>>> can replicate or find the issue in a dedicated chunk of time this >>>>>> weekend. >>>>>> >>>>>> You can email it to me, and/or encrypt with my GPG key if you'd like ( >>>>>> https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc >>>>>> ) >>>>>> >>>>>> By 'resources' I am referring to the GoCD functionality where you can >>>>>> tag agents with resources that they offer, which are then matched to >>>>>> pipeline jobs that say they *require* those resources to run as part >>>>>> of agent assignment. >>>>>> >>>>>> > No we use this setup no for about a year, patch the system on a >>>>>> regular basis including the latest gocd stable version. >>>>>> >>>>>> To make sure I understand you, are you saying that the problem has >>>>>> been here for the last year, perhaps gradually getting worse a story add >>>>>> more agents or pipelines - but not an issue suddenly created after a >>>>>> particular upgrade or change? >>>>>> >>>>>> -Chad >>>>>> >>>>>> On Fri, 16 Feb 2024, 00:29 'Wolfgang Achinger' via go-cd, < >>>>>> go-cd@googlegroups.com> wrote: >>>>>> >>>>>>> > And how many resources are defined across the agents? >>>>>>> What exactly do you mean here? System resources? XMS XMX Values of >>>>>>> java ? >>>>>>> >>>>>>> - Is this a problem that has always been there, or something that >>>>>>> has changed with a GoCD version or other change in environment? >>>>>>> No we use this setup no for about a year, patch the system on a >>>>>>> regular basis including the latest gocd stable version. >>>>>>> >>>>>>> - Is it faster when the server is restarted, and gets slower over >>>>>>> time (or the same after a restart)? >>>>>>> No a restart does not affect the speed at all. It stays constant >>>>>>> >>>>>>> - Why do you feel it is the # of jobs/stages the agents have >>>>>>> processed that is a key factor, rather than simply the # of agents or >>>>>>> some >>>>>>> other agent configuration factor? >>>>>>> I don't know it was more a wild guess. After later testing, i don't >>>>>>> think this anymore. I cleaned up some tables and reduced the agent >>>>>>> history >>>>>>> visible in the GUI, but this did not affect the speed (Well, it >>>>>>> increased >>>>>>> the speed of the listing of the agent history itself but not the loading >>>>>>> time of the popups). >>>>>>> >>>>>>> If it is ok i will send the support output directly your our >>>>>>> mailadress so it will not get shared in the thread. >>>>>>> >>>>>>> Am Do., 15. Feb. 2024 um 15:50 Uhr schrieb Chad Wilson < >>>>>>> ch...@thoughtworks.com>: >>>>>>> >>>>>>>> And how many resources are defined across the agents? >>>>>>>> >>>>>>>> Can you please answer the earlier questions I asked as well? It's >>>>>>>> rather difficult to efficiently help if you don't respond to the >>>>>>>> questions >>>>>>>> that characterise the problem from a maintainer perspective. :-) >>>>>>>> >>>>>>>> - Is this a problem that has always been there, or something that >>>>>>>> has changed with a GoCD version or other change in environment? >>>>>>>> - Is it faster when the server is restarted, and gets slower over >>>>>>>> time (or the same after a restart)? >>>>>>>> - Why do you feel it is the # of jobs/stages the agents have >>>>>>>> processed that is a key factor, rather than simply the # of agents or >>>>>>>> some >>>>>>>> other agent configuration factor? >>>>>>>> >>>>>>>> Additionally, can you share a redacted output from /go/api/support >>>>>>>> ? You can enter the URL in the browser when logged in as an admin. Be >>>>>>>> careful of the "Runtime Information" and "System Health Information" >>>>>>>> sections when sharing. These are the two main places which might leak >>>>>>>> unintentional information from your setup. Redact the individual values >>>>>>>> which feel sensitive to you. >>>>>>>> >>>>>>>> -Chad >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Feb 15, 2024 at 10:17 PM 'Wolfgang Achinger' via go-cd < >>>>>>>> go-cd@googlegroups.com> wrote: >>>>>>>> >>>>>>>>> 1 environment >>>>>>>>> 164 materials >>>>>>>>> 0 elastic agents >>>>>>>>> 2 config repos >>>>>>>>> 0 artifact stores >>>>>>>>> 0 pluggable scms >>>>>>>>> >>>>>>>>> Am Do., 15. Feb. 2024 um 15:01 Uhr schrieb Chad Wilson < >>>>>>>>> ch...@thoughtworks.com>: >>>>>>>>> >>>>>>>>>> How many distinct environments and resources do you have across >>>>>>>>>> these 1200 pipelines, roughly? >>>>>>>>>> >>>>>>>>>> On Thu, Feb 15, 2024 at 5:38 PM 'Wolfgang Achinger' via go-cd < >>>>>>>>>> go-cd@googlegroups.com> wrote: >>>>>>>>>> >>>>>>>>>>> Additional information >>>>>>>>>>> Since the pipelines are configured via ~150 yaml files. >>>>>>>>>>> I tested it now with one big, merged config file with all >>>>>>>>>>> pipelines >>>>>>>>>>> But this did not change anything, >>>>>>>>>>> performance slow. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "go-cd" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to go-cd+unsubscr...@googlegroups.com. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com >>>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>> in the Google Groups "go-cd" group. >>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>> to go-cd+unsubscr...@googlegroups.com. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com >>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "go-cd" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to go-cd+unsubscr...@googlegroups.com. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLBiz4s%3DqpKr1Yrgw1TSyfRePkcdGXPesfrmAiu2e9aN6g%40mail.gmail.com >>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLBiz4s%3DqpKr1Yrgw1TSyfRePkcdGXPesfrmAiu2e9aN6g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "go-cd" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> go-cd+unsubscr...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH-jix9BM1k24yCedriGXryO7zRiMtEXaOxOmqZ-PamU3g%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH-jix9BM1k24yCedriGXryO7zRiMtEXaOxOmqZ-PamU3g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "go-cd" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to go-cd+unsubscr...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLDdSv8pY0w6ZzJF8Uv_gSx_Y_APdLjF3JHXGcxAE-rhrQ%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLDdSv8pY0w6ZzJF8Uv_gSx_Y_APdLjF3JHXGcxAE-rhrQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "go-cd" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> go-cd+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH9HfWexWAHGbTLQvuvZdTYRonH-bJ_OpcSkrvd3RSCirQ%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH9HfWexWAHGbTLQvuvZdTYRonH-bJ_OpcSkrvd3RSCirQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "go-cd" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to go-cd+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLAbXaud2BiWWYYNXK5AeVf8JHp7Sw6Hioe3OkPzm8Ga%2Bw%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLAbXaud2BiWWYYNXK5AeVf8JHp7Sw6Hioe3OkPzm8Ga%2Bw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "go-cd" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> go-cd+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH9yFok2t7ZVnmdvLfsPG-v1JD%3DfUm2_GwJ%2B3j%3Dg5WLDBw%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH9yFok2t7ZVnmdvLfsPG-v1JD%3DfUm2_GwJ%2B3j%3Dg5WLDBw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to go-cd+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CANhjCLAJ7MoasJowhsq24DwNG9rOyiYTkaAjqnSnoQm8MUsK-A%40mail.gmail.com >>> <https://groups.google.com/d/msgid/go-cd/CANhjCLAJ7MoasJowhsq24DwNG9rOyiYTkaAjqnSnoQm8MUsK-A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "go-cd" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> go-cd+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/CAA1RwH8rqwVfS4oH9s7XMNHe6o1fW_FUtEPcL%3D-uZaNKFS6reA%40mail.gmail.com >> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8rqwVfS4oH9s7XMNHe6o1fW_FUtEPcL%3D-uZaNKFS6reA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to go-cd+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CANhjCLDd28UMfH1OJYnB%2BXR8bf9SHfcgST8aLqyBzTS_m7KcbA%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CANhjCLDd28UMfH1OJYnB%2BXR8bf9SHfcgST8aLqyBzTS_m7KcbA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH_85-CcLUWMLhEzfmJqkPMP6HLn%3D_j6mxg7%3DMQZnpXxzw%40mail.gmail.com.
