Thank you Sriram. Please find my comments below. >Do the various jobs have an elastic profile ID set? Yes, I have two environments staging and prod, so we have separate profiles set for them.
Here is pretty much what each profile has: 1. ec2_ami 2. ec2_instance_profile 3. ec2_subnets 4. ec2_instance_type 5. ec2_key 6. ec2_user_data *echo "agent.auto.register.environments=staging,sandbox" | sudo tee -a /var/lib/go-agent/config/autoregister.properties > /dev/null* 7. ec2_sg >What is the error that you see due to the max count limit? ``` [go] Received request to create an instance for brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job at 2024-04-09 11:21:38 +00:00 [go] Successfully created new instance i-093b44f70992505cc in subnet-555bba0d [go] Received request to create an instance for brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job at 2024-04-09 11:23:38 +00:00 [go] The number of instances currently running is currently at the maximum permissible limit, "2". Not creating more instances for jobs: brxt-core-service-deploy-staging/86/prepare-for-deploy-stage/1/prepare-for-deploy-job, brxt-core-service-deploy-staging/86/deploy-stage/1/deploy-job, brxt-core-service-deploy-staging/86/verify-stage/1/verify-job, brxt-config-service-deploy-staging/18/deploy-stage/1/deploy-job, brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job. [go] Received request to create an instance for brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job at 2024-04-09 11:25:39 +00:00 [go] The number of instances currently running is currently at the maximum permissible limit, "2". Not creating more instances for jobs: brxt-core-service-deploy-staging/86/prepare-for-deploy-stage/1/prepare-for-deploy-job, brxt-core-service-deploy-staging/86/deploy-stage/1/deploy-job, brxt-core-service-deploy-staging/86/verify-stage/1/verify-job, brxt-config-service-deploy-staging/18/deploy-stage/1/deploy-job, brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job. [go] Received request to create an instance for brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job at 2024-04-09 11:27:39 +00:00 [go] The number of instances currently running is currently at the maximum permissible limit, "2". Not creating more instances for jobs: brxt-core-service-deploy-staging/86/prepare-for-deploy-stage/1/prepare-for-deploy-job, brxt-core-service-deploy-staging/86/deploy-stage/1/deploy-job, brxt-core-service-deploy-staging/86/verify-stage/1/verify-job, brxt-config-service-deploy-staging/18/deploy-stage/1/deploy-job, brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job. [go] Received request to create an instance for brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job at 2024-04-09 11:39:58 +00:00 [go] The number of instances currently running is currently at the maximum permissible limit, "2". Not creating more instances for jobs: brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job. [go] Received request to create an instance for brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job at 2024-04-09 11:41:56 +00:00 [go] Successfully created new instance i-0ca1b2dc4996c210b in subnet-555bba0d [go] Received request to create an instance for brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job at 2024-04-09 11:43:56 +00:00 [go] Successfully created new instance i-0bc0bf6e763b6ebf0 in subnet-555bba0d [go] Received request to create an instance for brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job at 2024-04-09 11:45:56 +00:00 [go] The number of instances currently running is currently at the maximum permissible limit, "2". Not creating more instances for jobs: brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job. [go] Received request to create an instance for brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job at 2024-04-09 11:47:56 +00:00 [go] The number of instances currently running is currently at the maximum permissible limit, "2". Not creating more instances for jobs: brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job. Go cancelled this job as it has not been assigned an agent for more than 10 minute(s)``` In here, that all happened as you see in the log, so we have two instances running but none of them got assigned to the job and then job failed eventually. >When you say "staging job", do you have a stage in a pipeline called "staging" with one job in it? Or do you have a stage in a pipeline with one job called "staging" and the other called "prod"? Attached is one of our pipelines, if you trigger the build job that in turn triggers the second and second triggers the third. Attached is the snippet from the dashboard. Please let me know if I have not covered any point or if you need more details on any specific thing, thank you. Regards Satya On Thu, Apr 25, 2024 at 3:21 PM Sriram Narayanan <sriram...@gmail.com> wrote: > > > On Thu, Apr 25, 2024 at 10:01 PM Satya Elipe <satya.el...@gmail.com> > wrote: > >> Hi All >> >> I'm encountering some issues with the way Elastic agents are launched, >> assigned, and terminated. Despite setting the maximum agent count to two, >> both agents launch sequentially, with only the first being assigned to the >> job. >> > > Do you want the job to run on both the agents? If so, then these > instructions will help you: > https://docs.gocd.org/current/advanced_usage/admin_spawn_multiple_jobs.html > > >> >> Here's where it gets tricky: when the staging job completes and triggers >> the production job, I expect one of the active agents to take over. >> Instead, the production job attempts to launch new agents, fails due to the >> max count limit, and runs without any agents, leading to failure. >> >> >> > Do the various jobs have an elastic profile ID set? > > What is the error that you see due to the max count limit? > > When you say "staging job", do you have a stage in a pipeline called > "staging" with one job in it? Or do you have a stage in a pipeline with one > job called "staging" and the other called "prod"? > > Could you share how your pipelines are composed? I'm especially asking > this since many new users tend to use GoCD after using other tools and > carry over some of the terminology but also the constraints. If you share > your pipeline structure and what you want to achieve, then we can design > something together. > > >> Additionally, some agent instances remain active for an extended period, >> requiring manual termination. This disrupts the workflow significantly. >> >> >> > On our cluster, we see the pods being activated upon need, then the > relevant job runs in the pod, and the pod is then deactivated. We are > sticking to the default of "10 pods" right now, and will be increasing the > limit after certain parallel-load reviews. > > Could you share your Cluster Profile and the Elastic Profile? Please take > care to obfuscate any org-specific information such as IP addresses, > hostnames, AWS ARNs, URLs, etc. > > >> Has anyone experienced similar issues, or anyone has any suggestions for >> a workaround? >> >> >> Thanks in advance ! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to go-cd+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/CADKEDRo_0yJjA0y31vOkXzgtVA_MOiSPQEc_uB3fE%3DfguO-wWQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/go-cd/CADKEDRo_0yJjA0y31vOkXzgtVA_MOiSPQEc_uB3fE%3DfguO-wWQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to go-cd+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CANiY96Yvww_OoK3tzZ%3DogcDjN82FGfydkOoFvxxpCDwHRzKV9A%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CANiY96Yvww_OoK3tzZ%3DogcDjN82FGfydkOoFvxxpCDwHRzKV9A%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CADKEDRpY%2ByC2YiN-nE2ZTY9DNSwYrAVmNaFZynDnAD0yXDP6kg%40mail.gmail.com.
