> can you confirm how many agents and pipelines are mapped to the environment below? Yes that is true
Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson < ch...@thoughtworks.com>: > Hiya folks > > I've been able to replicate this problem and should be able to fix it for > a subsequent release - thanks for the help debugging. > > The problem appears to arrive when there is a large number of pipelines* > mapped to an environment;* and also a large number of agents for that > environment. The logic for calculating the API response agents > > environments is accidentally very, very inefficient (I think it's O(n^2 x > m^2) or something crazy. I replicated something similar to what you > describe with 5,000 pipelines and 60 or so agents, all mapped into the > same, single, logical environment. > > [image: image.png] > > > In your case if you have all say 1,690 pipelines mapped to a single > environment (from your stats below), and all of your 95 agents are in the > same environment, you'd definitely trigger this issue. I can't tell exactly > from what you have shared how the pipelines and agents are mapped to > environments, so this is a guess - can you confirm how many agents and > pipelines are mapped to the environment below? > > "Number of pipelines": 1690, > "Number of environments": 1, > "Number of agents": 95, > > > If it's the same problem, you will probably find that *untagging the > agents from the environment *also has a similar speed-up effect to > deleting all of the agents (although then the pipelines requiring that > environment won't schedule either, obviously). > > Another workaround in the meantime, *if you don't rely on the environment* > > - to define environment variables/secure environment variables that > apply across all pipelines/jobs > - to affect whether jobs are scheduled to special agents > > ... may be to untag all pipelines and agents from the environment you use > and just use the default/empty environment. > > -Chad > > On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd < > go-cd@googlegroups.com> wrote: > >> > By 'resources' I am referring to the GoCD functionality where you can >> tag agents with resources that they offer, which are then matched to >> pipeline jobs that say they *require* those resources to run as part of >> agent assignment. >> 10 Agents have 5 resources attached >> 85 have 1 resource attached >> >> We use the resources to different special agents. They do the same as the >> rest, but they are placed in dedicated networks. >> >> > To make sure I understand you, are you saying that the problem has been >> here for the last year, perhaps gradually getting worse a story add more >> agents or pipelines - but not an issue suddenly created after a particular >> upgrade or change? >> That's correct. It's more an over-time issue than a sudden issue. >> >> I sent the additional information out, but not directly, they come from a >> different mail address over a secure transfer method. >> >> Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson < >> ch...@thoughtworks.com>: >> >>> Cool, thanks! Just trying to gather enough information to see if I can >>> replicate or find the issue in a dedicated chunk of time this weekend. >>> >>> You can email it to me, and/or encrypt with my GPG key if you'd like ( >>> https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc) >>> >>> By 'resources' I am referring to the GoCD functionality where you can >>> tag agents with resources that they offer, which are then matched to >>> pipeline jobs that say they *require* those resources to run as part of >>> agent assignment. >>> >>> > No we use this setup no for about a year, patch the system on a >>> regular basis including the latest gocd stable version. >>> >>> To make sure I understand you, are you saying that the problem has been >>> here for the last year, perhaps gradually getting worse a story add more >>> agents or pipelines - but not an issue suddenly created after a particular >>> upgrade or change? >>> >>> -Chad >>> >>> On Fri, 16 Feb 2024, 00:29 'Wolfgang Achinger' via go-cd, < >>> go-cd@googlegroups.com> wrote: >>> >>>> > And how many resources are defined across the agents? >>>> What exactly do you mean here? System resources? XMS XMX Values of java >>>> ? >>>> >>>> - Is this a problem that has always been there, or something that has >>>> changed with a GoCD version or other change in environment? >>>> No we use this setup no for about a year, patch the system on a regular >>>> basis including the latest gocd stable version. >>>> >>>> - Is it faster when the server is restarted, and gets slower over time >>>> (or the same after a restart)? >>>> No a restart does not affect the speed at all. It stays constant >>>> >>>> - Why do you feel it is the # of jobs/stages the agents have processed >>>> that is a key factor, rather than simply the # of agents or some other >>>> agent configuration factor? >>>> I don't know it was more a wild guess. After later testing, i don't >>>> think this anymore. I cleaned up some tables and reduced the agent history >>>> visible in the GUI, but this did not affect the speed (Well, it increased >>>> the speed of the listing of the agent history itself but not the loading >>>> time of the popups). >>>> >>>> If it is ok i will send the support output directly your our >>>> mailadress so it will not get shared in the thread. >>>> >>>> Am Do., 15. Feb. 2024 um 15:50 Uhr schrieb Chad Wilson < >>>> ch...@thoughtworks.com>: >>>> >>>>> And how many resources are defined across the agents? >>>>> >>>>> Can you please answer the earlier questions I asked as well? It's >>>>> rather difficult to efficiently help if you don't respond to the questions >>>>> that characterise the problem from a maintainer perspective. :-) >>>>> >>>>> - Is this a problem that has always been there, or something that has >>>>> changed with a GoCD version or other change in environment? >>>>> - Is it faster when the server is restarted, and gets slower over time >>>>> (or the same after a restart)? >>>>> - Why do you feel it is the # of jobs/stages the agents have processed >>>>> that is a key factor, rather than simply the # of agents or some other >>>>> agent configuration factor? >>>>> >>>>> Additionally, can you share a redacted output from /go/api/support ? >>>>> You can enter the URL in the browser when logged in as an admin. Be >>>>> careful >>>>> of the "Runtime Information" and "System Health Information" sections when >>>>> sharing. These are the two main places which might leak unintentional >>>>> information from your setup. Redact the individual values which feel >>>>> sensitive to you. >>>>> >>>>> -Chad >>>>> >>>>> >>>>> On Thu, Feb 15, 2024 at 10:17 PM 'Wolfgang Achinger' via go-cd < >>>>> go-cd@googlegroups.com> wrote: >>>>> >>>>>> 1 environment >>>>>> 164 materials >>>>>> 0 elastic agents >>>>>> 2 config repos >>>>>> 0 artifact stores >>>>>> 0 pluggable scms >>>>>> >>>>>> Am Do., 15. Feb. 2024 um 15:01 Uhr schrieb Chad Wilson < >>>>>> ch...@thoughtworks.com>: >>>>>> >>>>>>> How many distinct environments and resources do you have across >>>>>>> these 1200 pipelines, roughly? >>>>>>> >>>>>>> On Thu, Feb 15, 2024 at 5:38 PM 'Wolfgang Achinger' via go-cd < >>>>>>> go-cd@googlegroups.com> wrote: >>>>>>> >>>>>>>> Additional information >>>>>>>> Since the pipelines are configured via ~150 yaml files. >>>>>>>> I tested it now with one big, merged config file with all pipelines >>>>>>>> But this did not change anything, >>>>>>>> performance slow. >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "go-cd" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to go-cd+unsubscr...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "go-cd" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> go-cd+unsubscr...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "go-cd" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to go-cd+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/go-cd/CANhjCLBiz4s%3DqpKr1Yrgw1TSyfRePkcdGXPesfrmAiu2e9aN6g%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLBiz4s%3DqpKr1Yrgw1TSyfRePkcdGXPesfrmAiu2e9aN6g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "go-cd" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> go-cd+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH-jix9BM1k24yCedriGXryO7zRiMtEXaOxOmqZ-PamU3g%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH-jix9BM1k24yCedriGXryO7zRiMtEXaOxOmqZ-PamU3g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "go-cd" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to go-cd+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/CANhjCLDdSv8pY0w6ZzJF8Uv_gSx_Y_APdLjF3JHXGcxAE-rhrQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/go-cd/CANhjCLDdSv8pY0w6ZzJF8Uv_gSx_Y_APdLjF3JHXGcxAE-rhrQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "go-cd" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> go-cd+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CAA1RwH9HfWexWAHGbTLQvuvZdTYRonH-bJ_OpcSkrvd3RSCirQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH9HfWexWAHGbTLQvuvZdTYRonH-bJ_OpcSkrvd3RSCirQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to go-cd+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/CANhjCLAbXaud2BiWWYYNXK5AeVf8JHp7Sw6Hioe3OkPzm8Ga%2Bw%40mail.gmail.com >> <https://groups.google.com/d/msgid/go-cd/CANhjCLAbXaud2BiWWYYNXK5AeVf8JHp7Sw6Hioe3OkPzm8Ga%2Bw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "go-cd" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > go-cd+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CAA1RwH9yFok2t7ZVnmdvLfsPG-v1JD%3DfUm2_GwJ%2B3j%3Dg5WLDBw%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CAA1RwH9yFok2t7ZVnmdvLfsPG-v1JD%3DfUm2_GwJ%2B3j%3Dg5WLDBw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CANhjCLAfkp645E6hxmkUzUF_oBMFFjV2qVD-ee5eWCwXrDODTw%40mail.gmail.com.
