Re: [go-cd] Performance of popups in the gui

[Chad Wilson]

Great to hear. Really appreciate the feedback!

On Thu, 23 May 2024, 15:39 'Hans Dampf' via go-cd, 
wrote:

> So we have been running the new version for a couple of days now, and I
> can't see any further problems with it. The performance is great with >1000
> Pipelines now.
> It is a really great improvement.
>
> Wolfgang Achinger schrieb am Montag, 13. Mai 2024 um 10:30:42 UTC+2:
>
>> I'm currently in the process of upgrading the server and agents of all
>> our gocd setups. We will monitor it the next few days and I will come back
>> if we notice anything
>>
>> Am Mo., 13. Mai 2024 um 10:24 Uhr schrieb Chad Wilson <
>> ch...@thoughtworks.com>:
>>
>>> Great to hear - back to how it was *supposed* to behave! I hope it
>>> hasn't caused any other regressions 
>>>
>>> Now that this is cleaned up, let me know if it exposes any other
>>> unexpected weird slowness you can't get to the bottom of and I'll see if I
>>> can chip away at the other various niggles.
>>>
>>> -Chad
>>>
>>> On Mon, May 13, 2024 at 4:10 PM 'Wolfgang Achinger' via go-cd <
>>> go...@googlegroups.com> wrote:
>>>
 Dude the fix is amazing !

 Am Mo., 13. Mai 2024 um 04:05 Uhr schrieb Chad Wilson <
 ch...@thoughtworks.com>:

> Apologies for the slow release of this (been rather busy personally)
> but 24.1.0 is out with what I think should be a fix for this issue.
>
> If you have any feedback it'd be appreciated.
>
> -Chad
>
>
> On Mon, 19 Feb 2024, 15:15 'Wolfgang Achinger' via go-cd, <
> go...@googlegroups.com> wrote:
>
>> Hello,
>>
>> this is actually incredible news for us. We will look forward to the
>> release with the fix.
>> Thanks for the support.
>>
>> The workarounds seem not to be viable for us, since we use the
>> environment for a lot of global variables and customizations.
>>
>> Regards
>>
>> Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson <
>> ch...@thoughtworks.com>:
>>
>>> Hiya folks
>>>
>>> I've been able to replicate this problem and should be able to fix
>>> it for a subsequent release - thanks for the help debugging.
>>>
>>> The problem appears to arrive when there is a large number of
>>> pipelines* mapped to an environment;* and also a large number of
>>> agents for that environment. The logic for calculating the API response
>>> agents > environments is accidentally very, very inefficient (I think 
>>> it's
>>> O(n^2 x m^2) or something crazy. I replicated something similar to what 
>>> you
>>> describe with 5,000 pipelines and 60 or so agents, all mapped into the
>>> same, single, logical environment.
>>>
>>> [image: image.png]
>>>
>>>
>>> In your case if you have all say 1,690 pipelines mapped to a single
>>> environment (from your stats below), and all of your 95 agents are in 
>>> the
>>> same environment, you'd definitely trigger this issue. I can't tell 
>>> exactly
>>> from what you have shared how the pipelines and agents are mapped to
>>> environments, so this is a guess - can you confirm how many agents and
>>> pipelines are mapped to the environment below?
>>>
>>> "Number of pipelines": 1690,
>>> "Number of environments": 1,
>>> "Number of agents": 95,
>>>
>>>
>>> If it's the same problem, you will probably find that *untagging
>>> the agents from the environment *also has a similar speed-up effect
>>> to deleting all of the agents (although then the pipelines requiring 
>>> that
>>> environment won't schedule either, obviously).
>>>
>>> Another workaround in the meantime, *if you don't rely on the
>>> environment*
>>>
>>>- to define environment variables/secure environment variables
>>>that apply across all pipelines/jobs
>>>- to affect whether jobs are scheduled to special agents
>>>
>>> ... may be to untag all pipelines and agents from the environment
>>> you use and just use the default/empty environment.
>>>
>>> -Chad
>>>
>>> On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd <
>>> go...@googlegroups.com> wrote:
>>>
 > By 'resources' I am referring to the GoCD functionality where you
 can tag agents with resources that they offer, which are then matched 
 to
 pipeline jobs that say they *require* those resources to run as
 part of agent assignment.
 10 Agents have 5 resources attached
 85 have 1 resource attached

 We use the resources to different special agents. They do the same
 as the rest, but they are placed in dedicated networks.

 > To make sure I understand you, are you saying that the problem
 has been here for the last year, perhaps gradually getting worse a 
 story
 add more agents or pipelines - but not an issue 

Re: [go-cd] Performance of popups in the gui

['Hans Dampf' via go-cd]

So we have been running the new version for a couple of days now, and I 
can't see any further problems with it. The performance is great with >1000 
Pipelines now.
It is a really great improvement. 

Wolfgang Achinger schrieb am Montag, 13. Mai 2024 um 10:30:42 UTC+2:

> I'm currently in the process of upgrading the server and agents of all our 
> gocd setups. We will monitor it the next few days and I will come back if 
> we notice anything 
>
> Am Mo., 13. Mai 2024 um 10:24 Uhr schrieb Chad Wilson <
> ch...@thoughtworks.com>:
>
>> Great to hear - back to how it was *supposed* to behave! I hope it 
>> hasn't caused any other regressions 
>>
>> Now that this is cleaned up, let me know if it exposes any other 
>> unexpected weird slowness you can't get to the bottom of and I'll see if I 
>> can chip away at the other various niggles.
>>
>> -Chad
>>
>> On Mon, May 13, 2024 at 4:10 PM 'Wolfgang Achinger' via go-cd <
>> go...@googlegroups.com> wrote:
>>
>>> Dude the fix is amazing !
>>>
>>> Am Mo., 13. Mai 2024 um 04:05 Uhr schrieb Chad Wilson <
>>> ch...@thoughtworks.com>:
>>>
 Apologies for the slow release of this (been rather busy personally) 
 but 24.1.0 is out with what I think should be a fix for this issue.

 If you have any feedback it'd be appreciated.

 -Chad


 On Mon, 19 Feb 2024, 15:15 'Wolfgang Achinger' via go-cd, <
 go...@googlegroups.com> wrote:

> Hello,
>
> this is actually incredible news for us. We will look forward to the 
> release with the fix. 
> Thanks for the support.
>
> The workarounds seem not to be viable for us, since we use the 
> environment for a lot of global variables and customizations.
>
> Regards
>
> Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson <
> ch...@thoughtworks.com>:
>
>> Hiya folks
>>
>> I've been able to replicate this problem and should be able to fix it 
>> for a subsequent release - thanks for the help debugging.
>>
>> The problem appears to arrive when there is a large number of 
>> pipelines* mapped to an environment;* and also a large number of 
>> agents for that environment. The logic for calculating the API response 
>> agents > environments is accidentally very, very inefficient (I think 
>> it's 
>> O(n^2 x m^2) or something crazy. I replicated something similar to what 
>> you 
>> describe with 5,000 pipelines and 60 or so agents, all mapped into the 
>> same, single, logical environment.
>>
>> [image: image.png]
>>
>>
>> In your case if you have all say 1,690 pipelines mapped to a single 
>> environment (from your stats below), and all of your 95 agents are in 
>> the 
>> same environment, you'd definitely trigger this issue. I can't tell 
>> exactly 
>> from what you have shared how the pipelines and agents are mapped to 
>> environments, so this is a guess - can you confirm how many agents and 
>> pipelines are mapped to the environment below?
>>
>> "Number of pipelines": 1690,
>> "Number of environments": 1,
>> "Number of agents": 95,
>>
>>
>> If it's the same problem, you will probably find that *untagging the 
>> agents from the environment *also has a similar speed-up effect to 
>> deleting all of the agents (although then the pipelines requiring that 
>> environment won't schedule either, obviously).
>>
>> Another workaround in the meantime, *if you don't rely on the 
>> environment* 
>>
>>- to define environment variables/secure environment variables 
>>that apply across all pipelines/jobs
>>- to affect whether jobs are scheduled to special agents
>>
>> ... may be to untag all pipelines and agents from the environment you 
>> use and just use the default/empty environment.
>>
>> -Chad
>>
>> On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd <
>> go...@googlegroups.com> wrote:
>>
>>> > By 'resources' I am referring to the GoCD functionality where you 
>>> can tag agents with resources that they offer, which are then matched 
>>> to 
>>> pipeline jobs that say they *require* those resources to run as 
>>> part of agent assignment.
>>> 10 Agents have 5 resources attached 
>>> 85 have 1 resource attached
>>>
>>> We use the resources to different special agents. They do the same 
>>> as the rest, but they are placed in dedicated networks. 
>>>
>>> > To make sure I understand you, are you saying that the problem has 
>>> been here for the last year, perhaps gradually getting worse a story 
>>> add 
>>> more agents or pipelines - but not an issue suddenly created after a 
>>> particular upgrade or change?
>>> That's correct. It's more an over-time issue than a sudden issue. 
>>>
>>> I sent the additional information out, but not 

Re: [go-cd] Performance of popups in the gui

['Wolfgang Achinger' via go-cd]

I'm currently in the process of upgrading the server and agents of all our
gocd setups. We will monitor it the next few days and I will come back if
we notice anything

Am Mo., 13. Mai 2024 um 10:24 Uhr schrieb Chad Wilson <
ch...@thoughtworks.com>:

> Great to hear - back to how it was *supposed* to behave! I hope it hasn't
> caused any other regressions 
>
> Now that this is cleaned up, let me know if it exposes any other
> unexpected weird slowness you can't get to the bottom of and I'll see if I
> can chip away at the other various niggles.
>
> -Chad
>
> On Mon, May 13, 2024 at 4:10 PM 'Wolfgang Achinger' via go-cd <
> go-cd@googlegroups.com> wrote:
>
>> Dude the fix is amazing !
>>
>> Am Mo., 13. Mai 2024 um 04:05 Uhr schrieb Chad Wilson <
>> ch...@thoughtworks.com>:
>>
>>> Apologies for the slow release of this (been rather busy personally) but
>>> 24.1.0 is out with what I think should be a fix for this issue.
>>>
>>> If you have any feedback it'd be appreciated.
>>>
>>> -Chad
>>>
>>>
>>> On Mon, 19 Feb 2024, 15:15 'Wolfgang Achinger' via go-cd, <
>>> go-cd@googlegroups.com> wrote:
>>>
 Hello,

 this is actually incredible news for us. We will look forward to the
 release with the fix.
 Thanks for the support.

 The workarounds seem not to be viable for us, since we use the
 environment for a lot of global variables and customizations.

 Regards

 Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson <
 ch...@thoughtworks.com>:

> Hiya folks
>
> I've been able to replicate this problem and should be able to fix it
> for a subsequent release - thanks for the help debugging.
>
> The problem appears to arrive when there is a large number of pipelines*
> mapped to an environment;* and also a large number of agents for that
> environment. The logic for calculating the API response agents >
> environments is accidentally very, very inefficient (I think it's O(n^2 x
> m^2) or something crazy. I replicated something similar to what you
> describe with 5,000 pipelines and 60 or so agents, all mapped into the
> same, single, logical environment.
>
> [image: image.png]
>
>
> In your case if you have all say 1,690 pipelines mapped to a single
> environment (from your stats below), and all of your 95 agents are in the
> same environment, you'd definitely trigger this issue. I can't tell 
> exactly
> from what you have shared how the pipelines and agents are mapped to
> environments, so this is a guess - can you confirm how many agents and
> pipelines are mapped to the environment below?
>
> "Number of pipelines": 1690,
> "Number of environments": 1,
> "Number of agents": 95,
>
>
> If it's the same problem, you will probably find that *untagging the
> agents from the environment *also has a similar speed-up effect to
> deleting all of the agents (although then the pipelines requiring that
> environment won't schedule either, obviously).
>
> Another workaround in the meantime, *if you don't rely on the
> environment*
>
>- to define environment variables/secure environment variables
>that apply across all pipelines/jobs
>- to affect whether jobs are scheduled to special agents
>
> ... may be to untag all pipelines and agents from the environment you
> use and just use the default/empty environment.
>
> -Chad
>
> On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd <
> go-cd@googlegroups.com> wrote:
>
>> > By 'resources' I am referring to the GoCD functionality where you
>> can tag agents with resources that they offer, which are then matched to
>> pipeline jobs that say they *require* those resources to run as part
>> of agent assignment.
>> 10 Agents have 5 resources attached
>> 85 have 1 resource attached
>>
>> We use the resources to different special agents. They do the same as
>> the rest, but they are placed in dedicated networks.
>>
>> > To make sure I understand you, are you saying that the problem has
>> been here for the last year, perhaps gradually getting worse a story add
>> more agents or pipelines - but not an issue suddenly created after a
>> particular upgrade or change?
>> That's correct. It's more an over-time issue than a sudden issue.
>>
>> I sent the additional information out, but not directly, they come
>> from a different mail address over a secure transfer method.
>>
>> Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson <
>> ch...@thoughtworks.com>:
>>
>>> Cool, thanks! Just trying to gather enough information to see if I
>>> can replicate or find the issue in a dedicated chunk of time this 
>>> weekend.
>>>
>>> You can email it to me, and/or encrypt with my GPG key if you'd like
>>> (
>>> 

Re: [go-cd] Performance of popups in the gui

[Chad Wilson]

Great to hear - back to how it was *supposed* to behave! I hope it hasn't
caused any other regressions 

Now that this is cleaned up, let me know if it exposes any other unexpected
weird slowness you can't get to the bottom of and I'll see if I can chip
away at the other various niggles.

-Chad

On Mon, May 13, 2024 at 4:10 PM 'Wolfgang Achinger' via go-cd <
go-cd@googlegroups.com> wrote:

> Dude the fix is amazing !
>
> Am Mo., 13. Mai 2024 um 04:05 Uhr schrieb Chad Wilson <
> ch...@thoughtworks.com>:
>
>> Apologies for the slow release of this (been rather busy personally) but
>> 24.1.0 is out with what I think should be a fix for this issue.
>>
>> If you have any feedback it'd be appreciated.
>>
>> -Chad
>>
>>
>> On Mon, 19 Feb 2024, 15:15 'Wolfgang Achinger' via go-cd, <
>> go-cd@googlegroups.com> wrote:
>>
>>> Hello,
>>>
>>> this is actually incredible news for us. We will look forward to the
>>> release with the fix.
>>> Thanks for the support.
>>>
>>> The workarounds seem not to be viable for us, since we use the
>>> environment for a lot of global variables and customizations.
>>>
>>> Regards
>>>
>>> Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson <
>>> ch...@thoughtworks.com>:
>>>
 Hiya folks

 I've been able to replicate this problem and should be able to fix it
 for a subsequent release - thanks for the help debugging.

 The problem appears to arrive when there is a large number of pipelines*
 mapped to an environment;* and also a large number of agents for that
 environment. The logic for calculating the API response agents >
 environments is accidentally very, very inefficient (I think it's O(n^2 x
 m^2) or something crazy. I replicated something similar to what you
 describe with 5,000 pipelines and 60 or so agents, all mapped into the
 same, single, logical environment.

 [image: image.png]


 In your case if you have all say 1,690 pipelines mapped to a single
 environment (from your stats below), and all of your 95 agents are in the
 same environment, you'd definitely trigger this issue. I can't tell exactly
 from what you have shared how the pipelines and agents are mapped to
 environments, so this is a guess - can you confirm how many agents and
 pipelines are mapped to the environment below?

 "Number of pipelines": 1690,
 "Number of environments": 1,
 "Number of agents": 95,


 If it's the same problem, you will probably find that *untagging the
 agents from the environment *also has a similar speed-up effect to
 deleting all of the agents (although then the pipelines requiring that
 environment won't schedule either, obviously).

 Another workaround in the meantime, *if you don't rely on the
 environment*

- to define environment variables/secure environment variables that
apply across all pipelines/jobs
- to affect whether jobs are scheduled to special agents

 ... may be to untag all pipelines and agents from the environment you
 use and just use the default/empty environment.

 -Chad

 On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd <
 go-cd@googlegroups.com> wrote:

> > By 'resources' I am referring to the GoCD functionality where you
> can tag agents with resources that they offer, which are then matched to
> pipeline jobs that say they *require* those resources to run as part
> of agent assignment.
> 10 Agents have 5 resources attached
> 85 have 1 resource attached
>
> We use the resources to different special agents. They do the same as
> the rest, but they are placed in dedicated networks.
>
> > To make sure I understand you, are you saying that the problem has
> been here for the last year, perhaps gradually getting worse a story add
> more agents or pipelines - but not an issue suddenly created after a
> particular upgrade or change?
> That's correct. It's more an over-time issue than a sudden issue.
>
> I sent the additional information out, but not directly, they come
> from a different mail address over a secure transfer method.
>
> Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson <
> ch...@thoughtworks.com>:
>
>> Cool, thanks! Just trying to gather enough information to see if I
>> can replicate or find the issue in a dedicated chunk of time this 
>> weekend.
>>
>> You can email it to me, and/or encrypt with my GPG key if you'd like (
>> https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc
>> )
>>
>> By 'resources' I am referring to the GoCD functionality where you can
>> tag agents with resources that they offer, which are then matched to
>> pipeline jobs that say they *require* those resources to run as part
>> of agent assignment.
>>
>> > No we use this setup no for about a 

Re: [go-cd] Performance of popups in the gui

['Wolfgang Achinger' via go-cd]

Dude the fix is amazing !

Am Mo., 13. Mai 2024 um 04:05 Uhr schrieb Chad Wilson <
ch...@thoughtworks.com>:

> Apologies for the slow release of this (been rather busy personally) but
> 24.1.0 is out with what I think should be a fix for this issue.
>
> If you have any feedback it'd be appreciated.
>
> -Chad
>
>
> On Mon, 19 Feb 2024, 15:15 'Wolfgang Achinger' via go-cd, <
> go-cd@googlegroups.com> wrote:
>
>> Hello,
>>
>> this is actually incredible news for us. We will look forward to the
>> release with the fix.
>> Thanks for the support.
>>
>> The workarounds seem not to be viable for us, since we use the
>> environment for a lot of global variables and customizations.
>>
>> Regards
>>
>> Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson <
>> ch...@thoughtworks.com>:
>>
>>> Hiya folks
>>>
>>> I've been able to replicate this problem and should be able to fix it
>>> for a subsequent release - thanks for the help debugging.
>>>
>>> The problem appears to arrive when there is a large number of pipelines*
>>> mapped to an environment;* and also a large number of agents for that
>>> environment. The logic for calculating the API response agents >
>>> environments is accidentally very, very inefficient (I think it's O(n^2 x
>>> m^2) or something crazy. I replicated something similar to what you
>>> describe with 5,000 pipelines and 60 or so agents, all mapped into the
>>> same, single, logical environment.
>>>
>>> [image: image.png]
>>>
>>>
>>> In your case if you have all say 1,690 pipelines mapped to a single
>>> environment (from your stats below), and all of your 95 agents are in the
>>> same environment, you'd definitely trigger this issue. I can't tell exactly
>>> from what you have shared how the pipelines and agents are mapped to
>>> environments, so this is a guess - can you confirm how many agents and
>>> pipelines are mapped to the environment below?
>>>
>>> "Number of pipelines": 1690,
>>> "Number of environments": 1,
>>> "Number of agents": 95,
>>>
>>>
>>> If it's the same problem, you will probably find that *untagging the
>>> agents from the environment *also has a similar speed-up effect to
>>> deleting all of the agents (although then the pipelines requiring that
>>> environment won't schedule either, obviously).
>>>
>>> Another workaround in the meantime, *if you don't rely on the
>>> environment*
>>>
>>>- to define environment variables/secure environment variables that
>>>apply across all pipelines/jobs
>>>- to affect whether jobs are scheduled to special agents
>>>
>>> ... may be to untag all pipelines and agents from the environment you
>>> use and just use the default/empty environment.
>>>
>>> -Chad
>>>
>>> On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd <
>>> go-cd@googlegroups.com> wrote:
>>>
 > By 'resources' I am referring to the GoCD functionality where you can
 tag agents with resources that they offer, which are then matched to
 pipeline jobs that say they *require* those resources to run as part
 of agent assignment.
 10 Agents have 5 resources attached
 85 have 1 resource attached

 We use the resources to different special agents. They do the same as
 the rest, but they are placed in dedicated networks.

 > To make sure I understand you, are you saying that the problem has
 been here for the last year, perhaps gradually getting worse a story add
 more agents or pipelines - but not an issue suddenly created after a
 particular upgrade or change?
 That's correct. It's more an over-time issue than a sudden issue.

 I sent the additional information out, but not directly, they come from
 a different mail address over a secure transfer method.

 Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson <
 ch...@thoughtworks.com>:

> Cool, thanks! Just trying to gather enough information to see if I can
> replicate or find the issue in a dedicated chunk of time this weekend.
>
> You can email it to me, and/or encrypt with my GPG key if you'd like (
> https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc
> )
>
> By 'resources' I am referring to the GoCD functionality where you can
> tag agents with resources that they offer, which are then matched to
> pipeline jobs that say they *require* those resources to run as part
> of agent assignment.
>
> > No we use this setup no for about a year, patch the system on a
> regular basis including the latest gocd stable version.
>
> To make sure I understand you, are you saying that the problem has
> been here for the last year, perhaps gradually getting worse a story add
> more agents or pipelines - but not an issue suddenly created after a
> particular upgrade or change?
>
> -Chad
>
> On Fri, 16 Feb 2024, 00:29 'Wolfgang Achinger' via go-cd, <
> go-cd@googlegroups.com> wrote:
>
>> > And how many 

Re: [go-cd] Performance of popups in the gui

[Chad Wilson]

Apologies for the slow release of this (been rather busy personally) but
24.1.0 is out with what I think should be a fix for this issue.

If you have any feedback it'd be appreciated.

-Chad


On Mon, 19 Feb 2024, 15:15 'Wolfgang Achinger' via go-cd, <
go-cd@googlegroups.com> wrote:

> Hello,
>
> this is actually incredible news for us. We will look forward to the
> release with the fix.
> Thanks for the support.
>
> The workarounds seem not to be viable for us, since we use the environment
> for a lot of global variables and customizations.
>
> Regards
>
> Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson <
> ch...@thoughtworks.com>:
>
>> Hiya folks
>>
>> I've been able to replicate this problem and should be able to fix it for
>> a subsequent release - thanks for the help debugging.
>>
>> The problem appears to arrive when there is a large number of pipelines*
>> mapped to an environment;* and also a large number of agents for that
>> environment. The logic for calculating the API response agents >
>> environments is accidentally very, very inefficient (I think it's O(n^2 x
>> m^2) or something crazy. I replicated something similar to what you
>> describe with 5,000 pipelines and 60 or so agents, all mapped into the
>> same, single, logical environment.
>>
>> [image: image.png]
>>
>>
>> In your case if you have all say 1,690 pipelines mapped to a single
>> environment (from your stats below), and all of your 95 agents are in the
>> same environment, you'd definitely trigger this issue. I can't tell exactly
>> from what you have shared how the pipelines and agents are mapped to
>> environments, so this is a guess - can you confirm how many agents and
>> pipelines are mapped to the environment below?
>>
>> "Number of pipelines": 1690,
>> "Number of environments": 1,
>> "Number of agents": 95,
>>
>>
>> If it's the same problem, you will probably find that *untagging the
>> agents from the environment *also has a similar speed-up effect to
>> deleting all of the agents (although then the pipelines requiring that
>> environment won't schedule either, obviously).
>>
>> Another workaround in the meantime, *if you don't rely on the
>> environment*
>>
>>- to define environment variables/secure environment variables that
>>apply across all pipelines/jobs
>>- to affect whether jobs are scheduled to special agents
>>
>> ... may be to untag all pipelines and agents from the environment you use
>> and just use the default/empty environment.
>>
>> -Chad
>>
>> On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd <
>> go-cd@googlegroups.com> wrote:
>>
>>> > By 'resources' I am referring to the GoCD functionality where you can
>>> tag agents with resources that they offer, which are then matched to
>>> pipeline jobs that say they *require* those resources to run as part of
>>> agent assignment.
>>> 10 Agents have 5 resources attached
>>> 85 have 1 resource attached
>>>
>>> We use the resources to different special agents. They do the same as
>>> the rest, but they are placed in dedicated networks.
>>>
>>> > To make sure I understand you, are you saying that the problem has
>>> been here for the last year, perhaps gradually getting worse a story add
>>> more agents or pipelines - but not an issue suddenly created after a
>>> particular upgrade or change?
>>> That's correct. It's more an over-time issue than a sudden issue.
>>>
>>> I sent the additional information out, but not directly, they come from
>>> a different mail address over a secure transfer method.
>>>
>>> Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson <
>>> ch...@thoughtworks.com>:
>>>
 Cool, thanks! Just trying to gather enough information to see if I can
 replicate or find the issue in a dedicated chunk of time this weekend.

 You can email it to me, and/or encrypt with my GPG key if you'd like (
 https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc
 )

 By 'resources' I am referring to the GoCD functionality where you can
 tag agents with resources that they offer, which are then matched to
 pipeline jobs that say they *require* those resources to run as part
 of agent assignment.

 > No we use this setup no for about a year, patch the system on a
 regular basis including the latest gocd stable version.

 To make sure I understand you, are you saying that the problem has been
 here for the last year, perhaps gradually getting worse a story add more
 agents or pipelines - but not an issue suddenly created after a particular
 upgrade or change?

 -Chad

 On Fri, 16 Feb 2024, 00:29 'Wolfgang Achinger' via go-cd, <
 go-cd@googlegroups.com> wrote:

> > And how many resources are defined across the agents?
> What exactly do you mean here? System resources? XMS XMX Values of
> java ?
>
> - Is this a problem that has always been there, or something that has
> changed with a GoCD version or 

Re: [go-cd] Performance of popups in the gui

['Wolfgang Achinger' via go-cd]

> can you confirm how many agents and pipelines are mapped to the
environment below?
Yes that is true

Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson <
ch...@thoughtworks.com>:

> Hiya folks
>
> I've been able to replicate this problem and should be able to fix it for
> a subsequent release - thanks for the help debugging.
>
> The problem appears to arrive when there is a large number of pipelines*
> mapped to an environment;* and also a large number of agents for that
> environment. The logic for calculating the API response agents >
> environments is accidentally very, very inefficient (I think it's O(n^2 x
> m^2) or something crazy. I replicated something similar to what you
> describe with 5,000 pipelines and 60 or so agents, all mapped into the
> same, single, logical environment.
>
> [image: image.png]
>
>
> In your case if you have all say 1,690 pipelines mapped to a single
> environment (from your stats below), and all of your 95 agents are in the
> same environment, you'd definitely trigger this issue. I can't tell exactly
> from what you have shared how the pipelines and agents are mapped to
> environments, so this is a guess - can you confirm how many agents and
> pipelines are mapped to the environment below?
>
> "Number of pipelines": 1690,
> "Number of environments": 1,
> "Number of agents": 95,
>
>
> If it's the same problem, you will probably find that *untagging the
> agents from the environment *also has a similar speed-up effect to
> deleting all of the agents (although then the pipelines requiring that
> environment won't schedule either, obviously).
>
> Another workaround in the meantime, *if you don't rely on the environment*
>
>- to define environment variables/secure environment variables that
>apply across all pipelines/jobs
>- to affect whether jobs are scheduled to special agents
>
> ... may be to untag all pipelines and agents from the environment you use
> and just use the default/empty environment.
>
> -Chad
>
> On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd <
> go-cd@googlegroups.com> wrote:
>
>> > By 'resources' I am referring to the GoCD functionality where you can
>> tag agents with resources that they offer, which are then matched to
>> pipeline jobs that say they *require* those resources to run as part of
>> agent assignment.
>> 10 Agents have 5 resources attached
>> 85 have 1 resource attached
>>
>> We use the resources to different special agents. They do the same as the
>> rest, but they are placed in dedicated networks.
>>
>> > To make sure I understand you, are you saying that the problem has been
>> here for the last year, perhaps gradually getting worse a story add more
>> agents or pipelines - but not an issue suddenly created after a particular
>> upgrade or change?
>> That's correct. It's more an over-time issue than a sudden issue.
>>
>> I sent the additional information out, but not directly, they come from a
>> different mail address over a secure transfer method.
>>
>> Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson <
>> ch...@thoughtworks.com>:
>>
>>> Cool, thanks! Just trying to gather enough information to see if I can
>>> replicate or find the issue in a dedicated chunk of time this weekend.
>>>
>>> You can email it to me, and/or encrypt with my GPG key if you'd like (
>>> https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc)
>>>
>>> By 'resources' I am referring to the GoCD functionality where you can
>>> tag agents with resources that they offer, which are then matched to
>>> pipeline jobs that say they *require* those resources to run as part of
>>> agent assignment.
>>>
>>> > No we use this setup no for about a year, patch the system on a
>>> regular basis including the latest gocd stable version.
>>>
>>> To make sure I understand you, are you saying that the problem has been
>>> here for the last year, perhaps gradually getting worse a story add more
>>> agents or pipelines - but not an issue suddenly created after a particular
>>> upgrade or change?
>>>
>>> -Chad
>>>
>>> On Fri, 16 Feb 2024, 00:29 'Wolfgang Achinger' via go-cd, <
>>> go-cd@googlegroups.com> wrote:
>>>
 > And how many resources are defined across the agents?
 What exactly do you mean here? System resources? XMS XMX Values of java
 ?

 - Is this a problem that has always been there, or something that has
 changed with a GoCD version or other change in environment?
 No we use this setup no for about a year, patch the system on a regular
 basis including the latest gocd stable version.

 - Is it faster when the server is restarted, and gets slower over time
 (or the same after a restart)?
 No a restart does not affect the speed at all. It stays constant

 - Why do you feel it is the # of jobs/stages the agents have processed
 that is a key factor, rather than simply the # of agents or some other
 agent configuration factor?
 I don't know it was more a wild 

Re: [go-cd] Performance of popups in the gui

['Wolfgang Achinger' via go-cd]

Hello,

this is actually incredible news for us. We will look forward to the
release with the fix.
Thanks for the support.

The workarounds seem not to be viable for us, since we use the environment
for a lot of global variables and customizations.

Regards

Am Sa., 17. Feb. 2024 um 18:08 Uhr schrieb Chad Wilson <
ch...@thoughtworks.com>:

> Hiya folks
>
> I've been able to replicate this problem and should be able to fix it for
> a subsequent release - thanks for the help debugging.
>
> The problem appears to arrive when there is a large number of pipelines*
> mapped to an environment;* and also a large number of agents for that
> environment. The logic for calculating the API response agents >
> environments is accidentally very, very inefficient (I think it's O(n^2 x
> m^2) or something crazy. I replicated something similar to what you
> describe with 5,000 pipelines and 60 or so agents, all mapped into the
> same, single, logical environment.
>
> [image: image.png]
>
>
> In your case if you have all say 1,690 pipelines mapped to a single
> environment (from your stats below), and all of your 95 agents are in the
> same environment, you'd definitely trigger this issue. I can't tell exactly
> from what you have shared how the pipelines and agents are mapped to
> environments, so this is a guess - can you confirm how many agents and
> pipelines are mapped to the environment below?
>
> "Number of pipelines": 1690,
> "Number of environments": 1,
> "Number of agents": 95,
>
>
> If it's the same problem, you will probably find that *untagging the
> agents from the environment *also has a similar speed-up effect to
> deleting all of the agents (although then the pipelines requiring that
> environment won't schedule either, obviously).
>
> Another workaround in the meantime, *if you don't rely on the environment*
>
>- to define environment variables/secure environment variables that
>apply across all pipelines/jobs
>- to affect whether jobs are scheduled to special agents
>
> ... may be to untag all pipelines and agents from the environment you use
> and just use the default/empty environment.
>
> -Chad
>
> On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd <
> go-cd@googlegroups.com> wrote:
>
>> > By 'resources' I am referring to the GoCD functionality where you can
>> tag agents with resources that they offer, which are then matched to
>> pipeline jobs that say they *require* those resources to run as part of
>> agent assignment.
>> 10 Agents have 5 resources attached
>> 85 have 1 resource attached
>>
>> We use the resources to different special agents. They do the same as the
>> rest, but they are placed in dedicated networks.
>>
>> > To make sure I understand you, are you saying that the problem has been
>> here for the last year, perhaps gradually getting worse a story add more
>> agents or pipelines - but not an issue suddenly created after a particular
>> upgrade or change?
>> That's correct. It's more an over-time issue than a sudden issue.
>>
>> I sent the additional information out, but not directly, they come from a
>> different mail address over a secure transfer method.
>>
>> Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson <
>> ch...@thoughtworks.com>:
>>
>>> Cool, thanks! Just trying to gather enough information to see if I can
>>> replicate or find the issue in a dedicated chunk of time this weekend.
>>>
>>> You can email it to me, and/or encrypt with my GPG key if you'd like (
>>> https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc)
>>>
>>> By 'resources' I am referring to the GoCD functionality where you can
>>> tag agents with resources that they offer, which are then matched to
>>> pipeline jobs that say they *require* those resources to run as part of
>>> agent assignment.
>>>
>>> > No we use this setup no for about a year, patch the system on a
>>> regular basis including the latest gocd stable version.
>>>
>>> To make sure I understand you, are you saying that the problem has been
>>> here for the last year, perhaps gradually getting worse a story add more
>>> agents or pipelines - but not an issue suddenly created after a particular
>>> upgrade or change?
>>>
>>> -Chad
>>>
>>> On Fri, 16 Feb 2024, 00:29 'Wolfgang Achinger' via go-cd, <
>>> go-cd@googlegroups.com> wrote:
>>>
 > And how many resources are defined across the agents?
 What exactly do you mean here? System resources? XMS XMX Values of java
 ?

 - Is this a problem that has always been there, or something that has
 changed with a GoCD version or other change in environment?
 No we use this setup no for about a year, patch the system on a regular
 basis including the latest gocd stable version.

 - Is it faster when the server is restarted, and gets slower over time
 (or the same after a restart)?
 No a restart does not affect the speed at all. It stays constant

 - Why do you feel it is the # of jobs/stages the agents have 

Re: [go-cd] Performance of popups in the gui

[Chad Wilson]

Hiya folks

I've been able to replicate this problem and should be able to fix it for a
subsequent release - thanks for the help debugging.

The problem appears to arrive when there is a large number of pipelines*
mapped to an environment;* and also a large number of agents for that
environment. The logic for calculating the API response agents >
environments is accidentally very, very inefficient (I think it's O(n^2 x
m^2) or something crazy. I replicated something similar to what you
describe with 5,000 pipelines and 60 or so agents, all mapped into the
same, single, logical environment.

[image: image.png]


In your case if you have all say 1,690 pipelines mapped to a single
environment (from your stats below), and all of your 95 agents are in the
same environment, you'd definitely trigger this issue. I can't tell exactly
from what you have shared how the pipelines and agents are mapped to
environments, so this is a guess - can you confirm how many agents and
pipelines are mapped to the environment below?

"Number of pipelines": 1690,
"Number of environments": 1,
"Number of agents": 95,


If it's the same problem, you will probably find that *untagging the agents
from the environment *also has a similar speed-up effect to deleting all of
the agents (although then the pipelines requiring that environment won't
schedule either, obviously).

Another workaround in the meantime, *if you don't rely on the environment*

   - to define environment variables/secure environment variables that
   apply across all pipelines/jobs
   - to affect whether jobs are scheduled to special agents

... may be to untag all pipelines and agents from the environment you use
and just use the default/empty environment.

-Chad

On Fri, Feb 16, 2024 at 3:40 PM 'Wolfgang Achinger' via go-cd <
go-cd@googlegroups.com> wrote:

> > By 'resources' I am referring to the GoCD functionality where you can
> tag agents with resources that they offer, which are then matched to
> pipeline jobs that say they *require* those resources to run as part of
> agent assignment.
> 10 Agents have 5 resources attached
> 85 have 1 resource attached
>
> We use the resources to different special agents. They do the same as the
> rest, but they are placed in dedicated networks.
>
> > To make sure I understand you, are you saying that the problem has been
> here for the last year, perhaps gradually getting worse a story add more
> agents or pipelines - but not an issue suddenly created after a particular
> upgrade or change?
> That's correct. It's more an over-time issue than a sudden issue.
>
> I sent the additional information out, but not directly, they come from a
> different mail address over a secure transfer method.
>
> Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson <
> ch...@thoughtworks.com>:
>
>> Cool, thanks! Just trying to gather enough information to see if I can
>> replicate or find the issue in a dedicated chunk of time this weekend.
>>
>> You can email it to me, and/or encrypt with my GPG key if you'd like (
>> https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc)
>>
>> By 'resources' I am referring to the GoCD functionality where you can tag
>> agents with resources that they offer, which are then matched to pipeline
>> jobs that say they *require* those resources to run as part of agent
>> assignment.
>>
>> > No we use this setup no for about a year, patch the system on a regular
>> basis including the latest gocd stable version.
>>
>> To make sure I understand you, are you saying that the problem has been
>> here for the last year, perhaps gradually getting worse a story add more
>> agents or pipelines - but not an issue suddenly created after a particular
>> upgrade or change?
>>
>> -Chad
>>
>> On Fri, 16 Feb 2024, 00:29 'Wolfgang Achinger' via go-cd, <
>> go-cd@googlegroups.com> wrote:
>>
>>> > And how many resources are defined across the agents?
>>> What exactly do you mean here? System resources? XMS XMX Values of java ?
>>>
>>> - Is this a problem that has always been there, or something that has
>>> changed with a GoCD version or other change in environment?
>>> No we use this setup no for about a year, patch the system on a regular
>>> basis including the latest gocd stable version.
>>>
>>> - Is it faster when the server is restarted, and gets slower over time
>>> (or the same after a restart)?
>>> No a restart does not affect the speed at all. It stays constant
>>>
>>> - Why do you feel it is the # of jobs/stages the agents have processed
>>> that is a key factor, rather than simply the # of agents or some other
>>> agent configuration factor?
>>> I don't know it was more a wild guess. After later testing, i don't
>>> think this anymore. I cleaned up some tables and reduced the agent history
>>> visible in the GUI, but this did not affect the speed (Well, it increased
>>> the speed of the listing of the agent history itself but not the loading
>>> time of the popups).
>>>
>>> If it is ok i will send the 

Re: [go-cd] Performance of popups in the gui

['Wolfgang Achinger' via go-cd]

> By 'resources' I am referring to the GoCD functionality where you can tag
agents with resources that they offer, which are then matched to pipeline
jobs that say they *require* those resources to run as part of agent
assignment.
10 Agents have 5 resources attached
85 have 1 resource attached

We use the resources to different special agents. They do the same as the
rest, but they are placed in dedicated networks.

> To make sure I understand you, are you saying that the problem has been
here for the last year, perhaps gradually getting worse a story add more
agents or pipelines - but not an issue suddenly created after a particular
upgrade or change?
That's correct. It's more an over-time issue than a sudden issue.

I sent the additional information out, but not directly, they come from a
different mail address over a secure transfer method.

Am Do., 15. Feb. 2024 um 17:57 Uhr schrieb Chad Wilson <
ch...@thoughtworks.com>:

> Cool, thanks! Just trying to gather enough information to see if I can
> replicate or find the issue in a dedicated chunk of time this weekend.
>
> You can email it to me, and/or encrypt with my GPG key if you'd like (
> https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc)
>
> By 'resources' I am referring to the GoCD functionality where you can tag
> agents with resources that they offer, which are then matched to pipeline
> jobs that say they *require* those resources to run as part of agent
> assignment.
>
> > No we use this setup no for about a year, patch the system on a regular
> basis including the latest gocd stable version.
>
> To make sure I understand you, are you saying that the problem has been
> here for the last year, perhaps gradually getting worse a story add more
> agents or pipelines - but not an issue suddenly created after a particular
> upgrade or change?
>
> -Chad
>
> On Fri, 16 Feb 2024, 00:29 'Wolfgang Achinger' via go-cd, <
> go-cd@googlegroups.com> wrote:
>
>> > And how many resources are defined across the agents?
>> What exactly do you mean here? System resources? XMS XMX Values of java ?
>>
>> - Is this a problem that has always been there, or something that has
>> changed with a GoCD version or other change in environment?
>> No we use this setup no for about a year, patch the system on a regular
>> basis including the latest gocd stable version.
>>
>> - Is it faster when the server is restarted, and gets slower over time
>> (or the same after a restart)?
>> No a restart does not affect the speed at all. It stays constant
>>
>> - Why do you feel it is the # of jobs/stages the agents have processed
>> that is a key factor, rather than simply the # of agents or some other
>> agent configuration factor?
>> I don't know it was more a wild guess. After later testing, i don't think
>> this anymore. I cleaned up some tables and reduced the agent history
>> visible in the GUI, but this did not affect the speed (Well, it increased
>> the speed of the listing of the agent history itself but not the loading
>> time of the popups).
>>
>> If it is ok i will send the support output directly your our mailadress
>> so it will not get shared in the thread.
>>
>> Am Do., 15. Feb. 2024 um 15:50 Uhr schrieb Chad Wilson <
>> ch...@thoughtworks.com>:
>>
>>> And how many resources are defined across the agents?
>>>
>>> Can you please answer the earlier questions I asked as well? It's rather
>>> difficult to efficiently help if you don't respond to the questions that
>>> characterise the problem from a maintainer perspective. :-)
>>>
>>> - Is this a problem that has always been there, or something that has
>>> changed with a GoCD version or other change in environment?
>>> - Is it faster when the server is restarted, and gets slower over time
>>> (or the same after a restart)?
>>> - Why do you feel it is the # of jobs/stages the agents have processed
>>> that is a key factor, rather than simply the # of agents or some other
>>> agent configuration factor?
>>>
>>> Additionally, can you share a redacted output from /go/api/support ?
>>> You can enter the URL in the browser when logged in as an admin. Be careful
>>> of the "Runtime Information" and "System Health Information" sections when
>>> sharing. These are the two main places which might leak unintentional
>>> information from your setup. Redact the individual values which feel
>>> sensitive to you.
>>>
>>> -Chad
>>>
>>>
>>> On Thu, Feb 15, 2024 at 10:17 PM 'Wolfgang Achinger' via go-cd <
>>> go-cd@googlegroups.com> wrote:
>>>
 1 environment
 164 materials
 0 elastic agents
 2 config repos
 0 artifact stores
 0 pluggable scms

 Am Do., 15. Feb. 2024 um 15:01 Uhr schrieb Chad Wilson <
 ch...@thoughtworks.com>:

> How many distinct environments and resources do you have across these
> 1200 pipelines, roughly?
>
> On Thu, Feb 15, 2024 at 5:38 PM 'Wolfgang Achinger' via go-cd <
> go-cd@googlegroups.com> wrote:
>
>> Additional information

Re: [go-cd] Performance of popups in the gui

[Chad Wilson]

Cool, thanks! Just trying to gather enough information to see if I can
replicate or find the issue in a dedicated chunk of time this weekend.

You can email it to me, and/or encrypt with my GPG key if you'd like (
https://github.com/chadlwilson/chadlwilson/blob/main/gpg-public-key.asc)

By 'resources' I am referring to the GoCD functionality where you can tag
agents with resources that they offer, which are then matched to pipeline
jobs that say they *require* those resources to run as part of agent
assignment.

> No we use this setup no for about a year, patch the system on a regular
basis including the latest gocd stable version.

To make sure I understand you, are you saying that the problem has been
here for the last year, perhaps gradually getting worse a story add more
agents or pipelines - but not an issue suddenly created after a particular
upgrade or change?

-Chad

On Fri, 16 Feb 2024, 00:29 'Wolfgang Achinger' via go-cd, <
go-cd@googlegroups.com> wrote:

> > And how many resources are defined across the agents?
> What exactly do you mean here? System resources? XMS XMX Values of java ?
>
> - Is this a problem that has always been there, or something that has
> changed with a GoCD version or other change in environment?
> No we use this setup no for about a year, patch the system on a regular
> basis including the latest gocd stable version.
>
> - Is it faster when the server is restarted, and gets slower over time (or
> the same after a restart)?
> No a restart does not affect the speed at all. It stays constant
>
> - Why do you feel it is the # of jobs/stages the agents have processed
> that is a key factor, rather than simply the # of agents or some other
> agent configuration factor?
> I don't know it was more a wild guess. After later testing, i don't think
> this anymore. I cleaned up some tables and reduced the agent history
> visible in the GUI, but this did not affect the speed (Well, it increased
> the speed of the listing of the agent history itself but not the loading
> time of the popups).
>
> If it is ok i will send the support output directly your our mailadress
> so it will not get shared in the thread.
>
> Am Do., 15. Feb. 2024 um 15:50 Uhr schrieb Chad Wilson <
> ch...@thoughtworks.com>:
>
>> And how many resources are defined across the agents?
>>
>> Can you please answer the earlier questions I asked as well? It's rather
>> difficult to efficiently help if you don't respond to the questions that
>> characterise the problem from a maintainer perspective. :-)
>>
>> - Is this a problem that has always been there, or something that has
>> changed with a GoCD version or other change in environment?
>> - Is it faster when the server is restarted, and gets slower over time
>> (or the same after a restart)?
>> - Why do you feel it is the # of jobs/stages the agents have processed
>> that is a key factor, rather than simply the # of agents or some other
>> agent configuration factor?
>>
>> Additionally, can you share a redacted output from /go/api/support ? You
>> can enter the URL in the browser when logged in as an admin. Be careful of
>> the "Runtime Information" and "System Health Information" sections when
>> sharing. These are the two main places which might leak unintentional
>> information from your setup. Redact the individual values which feel
>> sensitive to you.
>>
>> -Chad
>>
>>
>> On Thu, Feb 15, 2024 at 10:17 PM 'Wolfgang Achinger' via go-cd <
>> go-cd@googlegroups.com> wrote:
>>
>>> 1 environment
>>> 164 materials
>>> 0 elastic agents
>>> 2 config repos
>>> 0 artifact stores
>>> 0 pluggable scms
>>>
>>> Am Do., 15. Feb. 2024 um 15:01 Uhr schrieb Chad Wilson <
>>> ch...@thoughtworks.com>:
>>>
 How many distinct environments and resources do you have across these
 1200 pipelines, roughly?

 On Thu, Feb 15, 2024 at 5:38 PM 'Wolfgang Achinger' via go-cd <
 go-cd@googlegroups.com> wrote:

> Additional information
> Since the pipelines are configured via ~150 yaml files.
> I tested it now with one big, merged config file with all pipelines
> But this did not change anything,
> performance slow.
>
> --
> You received this message because you are subscribed to the Google
> Groups "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com
> 
> .
>
 --
 You received this message because you are subscribed to a topic in the
 Google Groups "go-cd" group.
 To unsubscribe from this topic, visit
 https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe.
 To unsubscribe from this group and all 

Re: [go-cd] Performance of popups in the gui

['Wolfgang Achinger' via go-cd]

> And how many resources are defined across the agents?
What exactly do you mean here? System resources? XMS XMX Values of java ?

- Is this a problem that has always been there, or something that has
changed with a GoCD version or other change in environment?
No we use this setup no for about a year, patch the system on a regular
basis including the latest gocd stable version.

- Is it faster when the server is restarted, and gets slower over time (or
the same after a restart)?
No a restart does not affect the speed at all. It stays constant

- Why do you feel it is the # of jobs/stages the agents have processed that
is a key factor, rather than simply the # of agents or some other agent
configuration factor?
I don't know it was more a wild guess. After later testing, i don't think
this anymore. I cleaned up some tables and reduced the agent history
visible in the GUI, but this did not affect the speed (Well, it increased
the speed of the listing of the agent history itself but not the loading
time of the popups).

If it is ok i will send the support output directly your our mailadress  so
it will not get shared in the thread.

Am Do., 15. Feb. 2024 um 15:50 Uhr schrieb Chad Wilson <
ch...@thoughtworks.com>:

> And how many resources are defined across the agents?
>
> Can you please answer the earlier questions I asked as well? It's rather
> difficult to efficiently help if you don't respond to the questions that
> characterise the problem from a maintainer perspective. :-)
>
> - Is this a problem that has always been there, or something that has
> changed with a GoCD version or other change in environment?
> - Is it faster when the server is restarted, and gets slower over time (or
> the same after a restart)?
> - Why do you feel it is the # of jobs/stages the agents have processed
> that is a key factor, rather than simply the # of agents or some other
> agent configuration factor?
>
> Additionally, can you share a redacted output from /go/api/support ? You
> can enter the URL in the browser when logged in as an admin. Be careful of
> the "Runtime Information" and "System Health Information" sections when
> sharing. These are the two main places which might leak unintentional
> information from your setup. Redact the individual values which feel
> sensitive to you.
>
> -Chad
>
>
> On Thu, Feb 15, 2024 at 10:17 PM 'Wolfgang Achinger' via go-cd <
> go-cd@googlegroups.com> wrote:
>
>> 1 environment
>> 164 materials
>> 0 elastic agents
>> 2 config repos
>> 0 artifact stores
>> 0 pluggable scms
>>
>> Am Do., 15. Feb. 2024 um 15:01 Uhr schrieb Chad Wilson <
>> ch...@thoughtworks.com>:
>>
>>> How many distinct environments and resources do you have across these
>>> 1200 pipelines, roughly?
>>>
>>> On Thu, Feb 15, 2024 at 5:38 PM 'Wolfgang Achinger' via go-cd <
>>> go-cd@googlegroups.com> wrote:
>>>
 Additional information
 Since the pipelines are configured via ~150 yaml files.
 I tested it now with one big, merged config file with all pipelines
 But this did not change anything,
 performance slow.

 --
 You received this message because you are subscribed to the Google
 Groups "go-cd" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to go-cd+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com
 
 .

>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "go-cd" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> go-cd+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to go-cd+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/CANhjCLBiz4s%3DqpKr1Yrgw1TSyfRePkcdGXPesfrmAiu2e9aN6g%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "go-cd" group.
> To unsubscribe from this topic, visit
> 

Re: [go-cd] Performance of popups in the gui

[Chad Wilson]

And how many resources are defined across the agents?

Can you please answer the earlier questions I asked as well? It's rather
difficult to efficiently help if you don't respond to the questions that
characterise the problem from a maintainer perspective. :-)

- Is this a problem that has always been there, or something that has
changed with a GoCD version or other change in environment?
- Is it faster when the server is restarted, and gets slower over time (or
the same after a restart)?
- Why do you feel it is the # of jobs/stages the agents have processed that
is a key factor, rather than simply the # of agents or some other agent
configuration factor?

Additionally, can you share a redacted output from /go/api/support ? You
can enter the URL in the browser when logged in as an admin. Be careful of
the "Runtime Information" and "System Health Information" sections when
sharing. These are the two main places which might leak unintentional
information from your setup. Redact the individual values which feel
sensitive to you.

-Chad


On Thu, Feb 15, 2024 at 10:17 PM 'Wolfgang Achinger' via go-cd <
go-cd@googlegroups.com> wrote:

> 1 environment
> 164 materials
> 0 elastic agents
> 2 config repos
> 0 artifact stores
> 0 pluggable scms
>
> Am Do., 15. Feb. 2024 um 15:01 Uhr schrieb Chad Wilson <
> ch...@thoughtworks.com>:
>
>> How many distinct environments and resources do you have across these
>> 1200 pipelines, roughly?
>>
>> On Thu, Feb 15, 2024 at 5:38 PM 'Wolfgang Achinger' via go-cd <
>> go-cd@googlegroups.com> wrote:
>>
>>> Additional information
>>> Since the pipelines are configured via ~150 yaml files.
>>> I tested it now with one big, merged config file with all pipelines
>>> But this did not change anything,
>>> performance slow.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to go-cd+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "go-cd" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> go-cd+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CANhjCLBiz4s%3DqpKr1Yrgw1TSyfRePkcdGXPesfrmAiu2e9aN6g%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CAA1RwH-jix9BM1k24yCedriGXryO7zRiMtEXaOxOmqZ-PamU3g%40mail.gmail.com.

Re: [go-cd] Performance of popups in the gui

['Wolfgang Achinger' via go-cd]

1 environment
164 materials
0 elastic agents
2 config repos
0 artifact stores
0 pluggable scms

Am Do., 15. Feb. 2024 um 15:01 Uhr schrieb Chad Wilson <
ch...@thoughtworks.com>:

> How many distinct environments and resources do you have across these 1200
> pipelines, roughly?
>
> On Thu, Feb 15, 2024 at 5:38 PM 'Wolfgang Achinger' via go-cd <
> go-cd@googlegroups.com> wrote:
>
>> Additional information
>> Since the pipelines are configured via ~150 yaml files.
>> I tested it now with one big, merged config file with all pipelines
>> But this did not change anything,
>> performance slow.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to go-cd+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "go-cd" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CANhjCLBiz4s%3DqpKr1Yrgw1TSyfRePkcdGXPesfrmAiu2e9aN6g%40mail.gmail.com.

Re: [go-cd] Performance of popups in the gui

[Chad Wilson]

How many distinct environments and resources do you have across these 1200
pipelines, roughly?

On Thu, Feb 15, 2024 at 5:38 PM 'Wolfgang Achinger' via go-cd <
go-cd@googlegroups.com> wrote:

> Additional information
> Since the pipelines are configured via ~150 yaml files.
> I tested it now with one big, merged config file with all pipelines
> But this did not change anything,
> performance slow.
>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CAA1RwH8Vb0U5YPQNB4Qzf2d6kP8KiYRBsgXr1Jux3xEMEN_H5A%40mail.gmail.com.

Re: [go-cd] Performance of popups in the gui

['Wolfgang Achinger' via go-cd]

Additional information
Since the pipelines are configured via ~150 yaml files.
I tested it now with one big, merged config file with all pipelines
But this did not change anything,
performance slow.

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CANhjCLCY1Gsq8fef%2Bb0t8bHSfvgoZHdFHFK%2B1eWzBxYJFjqM3g%40mail.gmail.com.

Re: [go-cd] Performance of popups in the gui

['Wolfgang Achinger' via go-cd]

For comparison, that's a call of our oldest setup with 60 agents but a lot
less builds/pipeline
It is a lot faster.
[image: oldsetup.png]


Am Mo., 12. Feb. 2024 um 17:13 Uhr schrieb Chad Wilson <
ch...@thoughtworks.com>:

> Hi Hans
>
> Is this slowness specific to builds that are currently running? Or is the
> slowness (and speed up with deleted agents) the same even with
> builds/stages that are no longer running (even if the agents are)?
>
> Anything majorly different about this GoCD server setup compared to the
> others, e.g # of agents in the DB (deleted or not) or # of pipeline runs or
> configuration/version - that type of thing?
>
> To narrow things down a bit more, it's worth opening your browser Inspect
> and looking at the network tab when clicking the button and seeing which of
> the requests are slow:
>
> [image: image.png]
>
> On the assumption that it's the server that's slow rather than the UI,
> that will at least tell us which API is slow, and thus which code path or
> DB queries to look at contributing to the 5-6 seconds (or whether it is all
> of them).
>
> If it's much faster when the agents are deleted, my guess is that it could
> be to do with the logic to essentially allow the agents to be linked, but I
> am not sure intuitively which logic that might be.
>
> -Chad
>
> On Mon, Feb 12, 2024 at 11:19 PM 'Hans Dampf' via go-cd <
> go-cd@googlegroups.com> wrote:
>
>> Hello,
>>
>> we notice slow loads in the gui when you open the popup of a stage.  The
>> loading animation in the popup takes 5-6 seconds on every stage, and we try
>> to narrow down why.
>>
>> What we know so far is if we disable *and* delete all agents in the gui
>> the loading is instant. With the agents registered (imported db table
>> "agents"), the load time is back to several seconds.
>>
>> Gocd does the following sql statment in the database. 'update agents set
>> deleted=true where uuid in("alot uuids")'
>>
>> If I execute the statement directly in the database, it does not increase
>> the performance. You have to delete the agents via gui.
>>
>> The database backend is a postgres 14.
>> We have several other gocd setups on the same databaseserver without this
>> problem.
>>
>> The setup consists of 19 agent-servers with 5 agents each, overall 95
>> agents.
>> Every agent did process ~ 15000 stages so far.
>>
>> I haven+t figured out at the moment what exactly gocd does during the
>> creation of the popup. But it must have something to do with the agents
>> and/or the agent runtime history.
>>
>> Maybe someone can point me in the right direct where the bottleneck could
>> be.
>>
>> Regards
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to go-cd+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/502ce13f-1696-490c-aa91-6ca542f7e78cn%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "go-cd" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/go-cd/c1n1Aq7hG1k/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CAA1RwH-wCHVD9UJQ_ZPXMkXWKRi6vsLRR9tMMFdAZMiTk6hySQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CANhjCLBa4wK3tYe6FLvLTOY-CiU9jjixaVDwgBicyDADqR2wcQ%40mail.gmail.com.

Re: [go-cd] Performance of popups in the gui

[Chad Wilson]

Hi Hans

Is this slowness specific to builds that are currently running? Or is the
slowness (and speed up with deleted agents) the same even with
builds/stages that are no longer running (even if the agents are)?

Anything majorly different about this GoCD server setup compared to the
others, e.g # of agents in the DB (deleted or not) or # of pipeline runs or
configuration/version - that type of thing?

To narrow things down a bit more, it's worth opening your browser Inspect
and looking at the network tab when clicking the button and seeing which of
the requests are slow:

[image: image.png]

On the assumption that it's the server that's slow rather than the UI, that
will at least tell us which API is slow, and thus which code path or DB
queries to look at contributing to the 5-6 seconds (or whether it is all of
them).

If it's much faster when the agents are deleted, my guess is that it could
be to do with the logic to essentially allow the agents to be linked, but I
am not sure intuitively which logic that might be.

-Chad

On Mon, Feb 12, 2024 at 11:19 PM 'Hans Dampf' via go-cd <
go-cd@googlegroups.com> wrote:

> Hello,
>
> we notice slow loads in the gui when you open the popup of a stage.  The
> loading animation in the popup takes 5-6 seconds on every stage, and we try
> to narrow down why.
>
> What we know so far is if we disable *and* delete all agents in the gui
> the loading is instant. With the agents registered (imported db table
> "agents"), the load time is back to several seconds.
>
> Gocd does the following sql statment in the database. 'update agents set
> deleted=true where uuid in("alot uuids")'
>
> If I execute the statement directly in the database, it does not increase
> the performance. You have to delete the agents via gui.
>
> The database backend is a postgres 14.
> We have several other gocd setups on the same databaseserver without this
> problem.
>
> The setup consists of 19 agent-servers with 5 agents each, overall 95
> agents.
> Every agent did process ~ 15000 stages so far.
>
> I haven+t figured out at the moment what exactly gocd does during the
> creation of the popup. But it must have something to do with the agents
> and/or the agent runtime history.
>
> Maybe someone can point me in the right direct where the bottleneck could
> be.
>
> Regards
>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/502ce13f-1696-490c-aa91-6ca542f7e78cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CAA1RwH-wCHVD9UJQ_ZPXMkXWKRi6vsLRR9tMMFdAZMiTk6hySQ%40mail.gmail.com.

[go-cd] Performance of popups in the gui

['Hans Dampf' via go-cd]

Hello,

we notice slow loads in the gui when you open the popup of a stage.  The 
loading animation in the popup takes 5-6 seconds on every stage, and we try 
to narrow down why.

What we know so far is if we disable *and* delete all agents in the gui the 
loading is instant. With the agents registered (imported db table 
"agents"), the load time is back to several seconds. 

Gocd does the following sql statment in the database. 'update agents set 
deleted=true where uuid in("alot uuids")'

If I execute the statement directly in the database, it does not increase 
the performance. You have to delete the agents via gui. 

The database backend is a postgres 14.
We have several other gocd setups on the same databaseserver without this 
problem. 

The setup consists of 19 agent-servers with 5 agents each, overall 95 agents. 

Every agent did process ~ 15000 stages so far. 

I haven+t figured out at the moment what exactly gocd does during the 
creation of the popup. But it must have something to do with the agents 
and/or the agent runtime history. 

Maybe someone can point me in the right direct where the bottleneck could 
be. 

Regards

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/502ce13f-1696-490c-aa91-6ca542f7e78cn%40googlegroups.com.