Hello, this is actually incredible news for us. We will look forward to the release with the fix. Thanks for the support.
The workarounds seem not to be viable for us, since we use the environment for a lot of global variables and customizations. Regards Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson < ch...@thoughtworks.com>: > Hiya folks > > I've been able to replicate this problem and should be able to fix it for > a subsequent release - thanks for the help debugging. > > The problem appears to arrive when there is a large number of pipelines* > mapped to an environment;* and also a large number of agents for that > environment. The logic for calculating the API response agents > > environments is accidentally very, very inefficient (I think it's O(n^2 x > m^2) or something crazy. I replicated something similar to what you > describe with 5,000 pipelines and 60 or so agents, all mapped into the > same, single, logical environment. > > [image: image.png] > > > In your case if you have all say 1,690 pipelines mapped to a single > environment (from your stats below), and all of your 95 agents are in the > same environment, you'd definitely trigger this issue. I can't tell exactly > from what you have shared how the pipelines and agents are mapped to > environments, so this is a guess - can you confirm how many agents and > pipelines are mapped to the environment below? > > "Number of pipelines": 1690, > "Number of environments": 1, > "Number of agents": 95, > > > If it's the same problem, you will probably find that *untagging the > agents from the environment *also has a similar speed-up effect to > deleting all of the agents (although then the pipelines requiring that > environment won't schedule either, obviously). > > Another workaround in the meantime, *if you don't rely on the environment* > > - to define environment variables/secure environment variables that > apply across all pipelines/jobs > - to affect whether jobs are scheduled to special agents > > ... may be to untag all pipelines and agents from the environment you use > and just use the default/empty environment. > > -Chad > > On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd < > go-cd@googlegroups.com> wrote: > >> > By 'resources' I am referring to the GoCD functionality where you can >> tag agents with resources that they offer, which are then matched to >> pipeline jobs that say they *require* those resources to run as part of >> agent assignment. >> 10 Agents have 5 resources attached >> 85 have 1 resource attached >> >> We use the resources to different special agents. They do the same as the >> rest, but they are placed in dedicated networks. >> >> > To make sure I understand you, are you saying that the problem has been >> here for the last year, perhaps gradually getting worse a story add more >> agents or pipelines - but not an issue suddenly created after a particular >> upgrade or change? >> That's correct. It's more an over-time issue than a sudden issue. >> >> I sent the additional information out, but not directly, they come from a >> different mail address over a secure transfer method. >> >> Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson < >> ch...@thoughtworks.com>: >> >>> Cool, thanks! Just trying to gather enough information to see if I can >>> replicate or find the issue in a dedicated chunk of time this weekend. >>> >>> You can email it to me, and/or encrypt with my GPG key if you'd like ( >>> https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc) >>> >>> By 'resources' I am referring to the GoCD functionality where you can >>> tag agents with resources that they offer, which are then matched to >>> pipeline jobs that say they *require* those resources to run as part of >>> agent assignment. >>> >>> > No we use this setup no for about a year, patch the system on a >>> regular basis including the latest gocd stable version. >>> >>> To make sure I understand you, are you saying that the problem has been >>> here for the last year, perhaps gradually getting worse a story add more >>> agents or pipelines - but not an issue suddenly created after a particular >>> upgrade or change? >>> >>> -Chad >>> >>> On Fri, 16 Feb 2024, 00:29 'Wolfgang Achinger' via go-cd, < >>> go-cd@googlegroups.com> wrote: >>> >>>> > And how many resources are defined across the agents? >>>> What exactly do you mean here? System resources? XMS XMX Values of java >>>> ? >>>> >>>> - Is this a problem that has always been there, or something that has >>>> changed with a GoCD version or other change in environment? >>>> No we use this setup no for about a year, patch the system on a regular >>>> basis including the latest gocd stable version. >>>> >>>> - Is it faster when the server is restarted, and gets slower over time >>>> (or the same after a restart)? >>>> No a restart does not affect the speed at all. It stays constant >>>> >>>> - Why do you feel it is the # of jobs/stages the agents have processed >>>> that is a key factor, rather than simply the # of agents or some other >>>> agent configuration factor? >>>> I don't know it was more a wild guess. After later testing, i don't >>>> think this anymore. I cleaned up some tables and reduced the agent history >>>> visible in the GUI, but this did not affect the speed (Well, it increased >>>> the speed of the listing of the agent history itself but not the loading >>>> time of the popups). >>>> >>>> If it is ok i will send the support output directly your our >>>> mailadress so it will not get shared in the thread. >>>> >>>> Am Do., 15. Feb. 2024 um 15:50 Uhr schrieb Chad Wilson < >>>> ch...@thoughtworks.com>: >>>> >>>>> And how many resources are defined across the agents? >>>>> >>>>> Can you please answer the earlier questions I asked as well? It's >>>>> rather difficult to efficiently help if you don't respond to the questions >>>>> that characterise the problem from a maintainer perspective. :-) >>>>> >>>>> - Is this a problem that has always been there, or something that has >>>>> changed with a GoCD version or other change in environment? >>>>> - Is it faster when the server is restarted, and gets slower over time >>>>> (or the same after a restart)? >>>>> - Why do you feel it is the # of jobs/stages the agents have processed >>>>> that is a key factor, rather than simply the # of agents or some other >>>>> agent configuration factor? >>>>> >>>>> Additionally, can you share a redacted output from /go/api/support ? >>>>> You can enter the URL in the browser when logged in as an admin. Be >>>>> careful >>>>> of the "Runtime Information" and "System Health Information" sections when >>>>> sharing. These are the two main places which might leak unintentional >>>>> information from your setup. Redact the individual values which feel >>>>> sensitive to you. >>>>> >>>>> -Chad >>>>> >>>>> >>>>> On Thu, Feb 15, 2024 at 10:17 PM 'Wolfgang Achinger' via go-cd < >>>>> go-cd@googlegroups.com> wrote: >>>>> >>>>>> 1 environment >>>>>> 164 materials >>>>>> 0 elastic agents >>>>>> 2 config repos >>>>>> 0 artifact stores >>>>>> 0 pluggable scms >>>>>> >>>>>> Am Do., 15. Feb. 2024 um 15:01 Uhr schrieb Chad Wilson < >>>>>> ch...@thoughtworks.com>: >>>>>> >>>>>>> How many distinct environments and resources do you have across >>>>>>> these 1200 pipelines, roughly? >>>>>>> >>>>>>> On Thu, Feb 15, 2024 at 5:38 PM 'Wolfgang Achinger' via go-cd < >>>>>>> go-cd@googlegroups.com> wrote: >>>>>>> >>>>>>>> Additional information >>>>>>>> Since the pipelines are configured via ~150 yaml files. >>>>>>>> I tested it now with one big, merged config file with all pipelines >>>>>>>> But this did not change anything, >>>>>>>> performance slow. >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "go-cd" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to go-cd+unsubscr...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "go-cd" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> go-cd+unsubscr...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "go-cd" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to go-cd+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLBiz4s%3DqpKr1Yrgw1TSyfRePkcdGXPesfrmAiu2e9aN6g%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLBiz4s%3DqpKr1Yrgw1TSyfRePkcdGXPesfrmAiu2e9aN6g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "go-cd" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> go-cd+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH-jix9BM1k24yCedriGXryO7zRiMtEXaOxOmqZ-PamU3g%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH-jix9BM1k24yCedriGXryO7zRiMtEXaOxOmqZ-PamU3g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "go-cd" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to go-cd+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/CANhjCLDdSv8pY0w6ZzJF8Uv_gSx_Y_APdLjF3JHXGcxAE-rhrQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLDdSv8pY0w6ZzJF8Uv_gSx_Y_APdLjF3JHXGcxAE-rhrQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "go-cd" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> go-cd+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CAA1RwH9HfWexWAHGbTLQvuvZdTYRonH-bJ_OpcSkrvd3RSCirQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH9HfWexWAHGbTLQvuvZdTYRonH-bJ_OpcSkrvd3RSCirQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to go-cd+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/CANhjCLAbXaud2BiWWYYNXK5AeVf8JHp7Sw6Hioe3OkPzm8Ga%2Bw%40mail.gmail.com >> <https://groups.google.com/d/msgid/go-cd/CANhjCLAbXaud2BiWWYYNXK5AeVf8JHp7Sw6Hioe3OkPzm8Ga%2Bw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "go-cd" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > go-cd+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CAA1RwH9yFok2t7ZVnmdvLfsPG-v1JD%3DfUm2_GwJ%2B3j%3Dg5WLDBw%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CAA1RwH9yFok2t7ZVnmdvLfsPG-v1JD%3DfUm2_GwJ%2B3j%3Dg5WLDBw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CANhjCLAJ7MoasJowhsq24DwNG9rOyiYTkaAjqnSnoQm8MUsK-A%40mail.gmail.com.
