Re: [go-cd] Elastic agents plugin usage

[Chad Wilson]

It seems you are using the third party EC2 Elastic Agent Plugin
. This plugin
does not implement the plugin API correctly
 and
does not support environments correctly, which is presumably why you have
that user data hack.

It seems to me from a quick look at the code, this plugin only runs a
single job on an EC2 instance before terminating it so you shouldn't expect
re-use for multiple jobs. If you want to know more about how it is
designed, you are better to ask on their GitHub repo.

I don't know 100% why your agents aren't shutting down correctly, and you
probably need to look at the plugin logs (on both server and the agent
itself) to investigate.

However, since it looks like you have a ec2_user_data hack in place to get
some environment support with the plugin, you need to manually make sure
that the environments in the config agent.auto.register.
environments=staging,sandbox *exactly match the possible pipeline
environments for all possible jobs* you assign to this elastic agent
profile ID.

I also think having multiple environments registered here will possibly
cause chaos, because that is not how elastic agents manage environments
normally. They normally register only a single environment.

The problem will be that if *any single job* on your GoCD is assigned to
say, profile "*elastic_profile_staging*" with the autoregister config like
you have below, but then that job is configured for a pipeline inside a
GoCD environment called* "**other_env**" *an elastic agent will start but
then never get assigned the job. This is because it has registered only for
"*staging,sandbox*" via your hardcoded user_data, NOT "*other_env*".

This breaks the elastic agent plugin contract - GoCD thinks it has already
told the plugin to create an agent for "*other_env*", but it never does.
Now GoCD is confused as to what is happening with the agents. Thus the job
will likely never get assigned, and the plugin will never complete a job,
and it will never shut down the EC2 instance. Perhaps this is what is
happening to you? Might want to check if you have EC2 instances whose agent
logs don't show them doing any work or mismatched environments and elastic
profiles.

With a correctly behaving plugin GoCD tells the plugin a single environment
to register for (the one it needs a new agent to run a job on), and expects
the plugin to register for the environment it tells it to. This EC2 plugin
breaks that contract, which allows you to misconfigure things very easily
and create all sorts of problems. Personally I wouldn't use it if I am
using GoCD environments, but that's your decision to make.

-Chad

On Thu, Apr 25, 2024 at 10:48 PM Satya Elipe  wrote:

> Thank you Sriram.
> Please find my comments below.
>
> >Do the various jobs have an elastic profile ID set?
> Yes, I have two environments staging and prod, so we have separate
> profiles set for them.
>
> Here is pretty much what each profile has:
>
>1. ec2_ami
>2. ec2_instance_profile
>3. ec2_subnets
>4. ec2_instance_type
>5. ec2_key
>6. ec2_user_data
>*echo "agent.auto.register.environments=staging,sandbox" | sudo tee -a
>/var/lib/go-agent/config/autoregister.properties > /dev/null*
>7. ec2_sg
>
>
> >What is the error that you see due to the max count limit?
> ```
> [go] Received request to create an instance for
> brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
> at 2024-04-09 11:21:38 +00:00
> [go] Successfully created new instance i-093b44f70992505cc in
> subnet-555bba0d
> [go] Received request to create an instance for
> brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
> at 2024-04-09 11:23:38 +00:00
> [go] The number of instances currently running is currently at the maximum
> permissible limit, "2". Not creating more instances for jobs:
> brxt-core-service-deploy-staging/86/prepare-for-deploy-stage/1/prepare-for-deploy-job,
> brxt-core-service-deploy-staging/86/deploy-stage/1/deploy-job,
> brxt-core-service-deploy-staging/86/verify-stage/1/verify-job,
> brxt-config-service-deploy-staging/18/deploy-stage/1/deploy-job,
> brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job.
> [go] Received request to create an instance for
> brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
> at 2024-04-09 11:25:39 +00:00
> [go] The number of instances currently running is currently at the maximum
> permissible limit, "2". Not creating more instances for jobs:
> brxt-core-service-deploy-staging/86/prepare-for-deploy-stage/1/prepare-for-deploy-job,
> brxt-core-service-deploy-staging/86/deploy-stage/1/deploy-job,
> brxt-core-service-deploy-staging/86/verify-stage/1/verify-job,
> brxt-config-service-deploy-staging/18/deploy-stage/1/deploy-job,
> 

Re: [go-cd] Elastic agents plugin usage

[Satya Elipe]

Thank you Sriram.
Please find my comments below.

>Do the various jobs have an elastic profile ID set?
Yes, I have two environments staging and prod, so we have separate profiles
set for them.

Here is pretty much what each profile has:

   1. ec2_ami
   2. ec2_instance_profile
   3. ec2_subnets
   4. ec2_instance_type
   5. ec2_key
   6. ec2_user_data
   *echo "agent.auto.register.environments=staging,sandbox" | sudo tee -a
   /var/lib/go-agent/config/autoregister.properties > /dev/null*
   7. ec2_sg


>What is the error that you see due to the max count limit?
```
[go] Received request to create an instance for
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
at 2024-04-09 11:21:38 +00:00
[go] Successfully created new instance i-093b44f70992505cc in
subnet-555bba0d
[go] Received request to create an instance for
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
at 2024-04-09 11:23:38 +00:00
[go] The number of instances currently running is currently at the maximum
permissible limit, "2". Not creating more instances for jobs:
brxt-core-service-deploy-staging/86/prepare-for-deploy-stage/1/prepare-for-deploy-job,
brxt-core-service-deploy-staging/86/deploy-stage/1/deploy-job,
brxt-core-service-deploy-staging/86/verify-stage/1/verify-job,
brxt-config-service-deploy-staging/18/deploy-stage/1/deploy-job,
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job.
[go] Received request to create an instance for
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
at 2024-04-09 11:25:39 +00:00
[go] The number of instances currently running is currently at the maximum
permissible limit, "2". Not creating more instances for jobs:
brxt-core-service-deploy-staging/86/prepare-for-deploy-stage/1/prepare-for-deploy-job,
brxt-core-service-deploy-staging/86/deploy-stage/1/deploy-job,
brxt-core-service-deploy-staging/86/verify-stage/1/verify-job,
brxt-config-service-deploy-staging/18/deploy-stage/1/deploy-job,
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job.
[go] Received request to create an instance for
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
at 2024-04-09 11:27:39 +00:00
[go] The number of instances currently running is currently at the maximum
permissible limit, "2". Not creating more instances for jobs:
brxt-core-service-deploy-staging/86/prepare-for-deploy-stage/1/prepare-for-deploy-job,
brxt-core-service-deploy-staging/86/deploy-stage/1/deploy-job,
brxt-core-service-deploy-staging/86/verify-stage/1/verify-job,
brxt-config-service-deploy-staging/18/deploy-stage/1/deploy-job,
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job.
[go] Received request to create an instance for
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
at 2024-04-09 11:39:58 +00:00
[go] The number of instances currently running is currently at the maximum
permissible limit, "2". Not creating more instances for jobs:
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job.
[go] Received request to create an instance for
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
at 2024-04-09 11:41:56 +00:00
[go] Successfully created new instance i-0ca1b2dc4996c210b in
subnet-555bba0d
[go] Received request to create an instance for
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
at 2024-04-09 11:43:56 +00:00
[go] Successfully created new instance i-0bc0bf6e763b6ebf0 in
subnet-555bba0d
[go] Received request to create an instance for
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
at 2024-04-09 11:45:56 +00:00
[go] The number of instances currently running is currently at the maximum
permissible limit, "2". Not creating more instances for jobs:
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job.
[go] Received request to create an instance for
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job
at 2024-04-09 11:47:56 +00:00
[go] The number of instances currently running is currently at the maximum
permissible limit, "2". Not creating more instances for jobs:
brxt-config-service-deploy-production/19/prepare-deploy-stage/1/prepare-deploy-job.
Go cancelled this job as it has not been assigned an agent for more than 10
minute(s)```

In here, that all happened as you see in the log, so we have two instances
running but none of them got assigned to the job and then job failed
eventually.

>When you say "staging job", do you have a stage in a pipeline called
"staging" with one job in it? Or do you have a stage in a pipeline with one
job called "staging" and the other called "prod"?
Attached is one of our pipelines, if you trigger the build job that in
turn triggers the second and second triggers the third. Attached is 

Re: [go-cd] Elastic agents plugin usage

[Chad Wilson]

Can you be specific about the type of elastic agents you are creating and
the plugin you are using. Kubernetes? Docker? Something else? There are
many elastic agent plugins.


> Here's where it gets tricky: when the staging job completes and triggers
> the production job, I expect one of the active agents to take over.
> Instead, the production job attempts to launch new agents, fails due to the
> max count limit, and runs without any agents, leading to failure.
>

I believe elastic agents are generally launched sequentially - i.e a new
one won't be launched until there are no pending-launch ones - but this
depends on the specific elastic agent type.

If you are new to elastic agents, you'll want to be aware that in almost
all elastic agent plugin variants the elastic agents are
single-shot/single-job usage, and are not re-used. The specific type of
elastic agent and its plugin implementation defines how it handles such
things though, so need to know specifics to guess.

Look at the specific elastic agent plugin's log on the server to see what
it is doing. Perhaps your elastic agents are not shutting down
automatically for some reason due to a configuration issue or a problem
with the jobs you are running?

-Chad

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CAA1RwH9msgAM_RtZu%2BEYbzHoQge5wh62a1i-YV7ueE_KoAkxtQ%40mail.gmail.com.

Re: [go-cd] Mount .ssh from EFS to the container (from the image gocd/gocd-server:v22.3.0)

[Sriram Narayanan]

On Thu, Apr 25, 2024 at 10:16 PM Satya Elipe  wrote:

> Hi all
>
> Wonder, what's the way around to mount .ssh from EFS into the gocd base
> container (from the image gocd/gocd-server:v22.3.0).
>
>
> We have saved all our content into EFS under /godata and maps that into
> the container as /godata.
>
>
> We are using gocd/gocd-server:v22.3.0.
>
>
> It all runs good, mapping was fine too but just one thing that’s not
> happening is “.ssh” folder.
>
>
> I have .ssh with all required keys in EFS under /godata and /godata within
> the container also has .ssh but not /go-working-dir.
>
>
> Is that supported, am I mis-configuring it, or do we need to handle that
> outside of the base image ?
>

At a high level, the .ssh folder should be mounted into /home/go.
e.g. docker run -v /path/to/godata:/godata -v /path/to/home-dir:/home/go
gocd/gocd-server:v23.5.0
IMPORTANT: You must set the user ID of the files within .ssh to 1000. This
is the user ID of the gocd process within the container.

See:
https://github.com/gocd/docker-gocd-server?tab=readme-ov-file#mounting-volumes

Given that you are using Kubernetes, please see the Helm chart
documentation here
https://github.com/gocd/helm-chart/blob/master/gocd/README.md

It provides info on just about every configurable attribute for the GoCD
server and the agent.

Of particular importance for you are these two attributes:
server.persistence.subpath.homego
agent.persistence.subpath.homego

Please see that document and jot down your action plan since you will need
to provide the SSH keys to the server _and_ the agent containers.

IMPORTANT: You must set the user ID of the files within .ssh to 1000. This
is the user ID of the gocd process within the container.



>
> Many thanks in advance !
>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CADKEDRrQOX11i951ZPiUYeOdMqThbCoZG7_WAqgBJFg1BxqxfQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CANiY96aM47Ck0vc%3D1BnjnMd%2BT9eu4BKokLqLXMG0mNAezT2V_A%40mail.gmail.com.

Re: [go-cd] Elastic agents plugin usage

[Sriram Narayanan]

On Thu, Apr 25, 2024 at 10:01 PM Satya Elipe  wrote:

> Hi All
>
> I'm encountering some issues with the way Elastic agents are launched,
> assigned, and terminated. Despite setting the maximum agent count to two,
> both agents launch sequentially, with only the first being assigned to the
> job.
>

Do you want the job to run on both the agents? If so, then these
instructions will help you:
https://docs.gocd.org/current/advanced_usage/admin_spawn_multiple_jobs.html


>
> Here's where it gets tricky: when the staging job completes and triggers
> the production job, I expect one of the active agents to take over.
> Instead, the production job attempts to launch new agents, fails due to the
> max count limit, and runs without any agents, leading to failure.
>
>
>
Do the various jobs have an elastic profile ID set?

What is the error that you see due to the max count limit?

When you say "staging job", do you have a stage in a pipeline called
"staging" with one job in it? Or do you have a stage in a pipeline with one
job called "staging" and the other called "prod"?

Could you share how your pipelines are composed? I'm especially asking this
since many new users tend to use GoCD after using other tools and carry
over some of the terminology but also the constraints. If you share your
pipeline structure and what you want to achieve, then we can design
something together.


> Additionally, some agent instances remain active for an extended period,
> requiring manual termination. This disrupts the workflow significantly.
>
>
>
On our cluster, we see the pods being activated upon need, then the
relevant job runs in the pod, and the pod is then deactivated. We are
sticking to the default of "10 pods" right now, and will be increasing the
limit after certain parallel-load reviews.

Could you share your Cluster Profile and the Elastic Profile? Please take
care to obfuscate any org-specific information such as IP addresses,
hostnames, AWS ARNs, URLs, etc.


> Has anyone experienced similar issues, or anyone has any suggestions for a
> workaround?
>
>
> Thanks in advance !
>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CADKEDRo_0yJjA0y31vOkXzgtVA_MOiSPQEc_uB3fE%3DfguO-wWQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CANiY96Yvww_OoK3tzZ%3DogcDjN82FGfydkOoFvxxpCDwHRzKV9A%40mail.gmail.com.

[go-cd] Mount .ssh from EFS to the container (from the image gocd/gocd-server:v22.3.0)

[Satya Elipe]

Hi all

Wonder, what's the way around to mount .ssh from EFS into the gocd base
container (from the image gocd/gocd-server:v22.3.0).


We have saved all our content into EFS under /godata and maps that into the
container as /godata.


We are using gocd/gocd-server:v22.3.0.


It all runs good, mapping was fine too but just one thing that’s not
happening is “.ssh” folder.


I have .ssh with all required keys in EFS under /godata and /godata within
the container also has .ssh but not /go-working-dir.


Is that supported, am I mis-configuring it, or do we need to handle that
outside of the base image ?


Many thanks in advance !

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CADKEDRrQOX11i951ZPiUYeOdMqThbCoZG7_WAqgBJFg1BxqxfQ%40mail.gmail.com.

[go-cd] Elastic agents plugin usage

[Satya Elipe]

Hi All

I'm encountering some issues with the way Elastic agents are launched,
assigned, and terminated. Despite setting the maximum agent count to two,
both agents launch sequentially, with only the first being assigned to the
job.


Here's where it gets tricky: when the staging job completes and triggers
the production job, I expect one of the active agents to take over.
Instead, the production job attempts to launch new agents, fails due to the
max count limit, and runs without any agents, leading to failure.


Additionally, some agent instances remain active for an extended period,
requiring manual termination. This disrupts the workflow significantly.


Has anyone experienced similar issues, or anyone has any suggestions for a
workaround?


Thanks in advance !

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CADKEDRo_0yJjA0y31vOkXzgtVA_MOiSPQEc_uB3fE%3DfguO-wWQ%40mail.gmail.com.

Re: [go-cd] "/godata/db/h2db/cruise.mv.db" is locked

[Satya Elipe]

Sure, I will.

Thank you .

On Thu, Apr 25, 2024 at 2:56 PM Sriram Narayanan 
wrote:

>
>
> On Thu, Apr 25, 2024 at 9:53 PM Satya Elipe  wrote:
>
>> Thank you, Sriram.
>>
>>
>> I managed to launch the new container alongside the old one.
>>
>
> Good!
>
>
>>
>> As you use the Elastic Agents plugin, can I ask you a question please.
>>
>
> Could you post this as a separate email question? That will help
> improve discoverability. Meanwhile, I'll read your email a bit and then
> have a response ready as well.
>
>
>>
>> I'm encountering some issues with the way agents are launched, assigned,
>> and terminated. Despite setting the maximum agent count to two, both agents
>> launch sequentially, with only the first being assigned to the job.
>>
>>
>> Here's where it gets tricky: when the staging job completes and triggers
>> the production job, I expect one of the active agents to take over.
>> Instead, the production job attempts to launch new agents, fails due to the
>> max count limit, and runs without any agents, leading to failure.
>>
>>
>> Additionally, some agent instances remain active for an extended period,
>> requiring manual termination. This disrupts the workflow significantly.
>>
>>
>> Have you experienced similar issues, or do you have any suggestions for a
>> workaround?
>>
>>
>> Thanks in advance !
>>
>>
>>
>>
>> On Tue, Apr 23, 2024 at 4:13 PM Sriram Narayanan 
>> wrote:
>>
>>>
>>>
>>> On Tue, Apr 23, 2024 at 10:22 PM Satya Elipe 
>>> wrote:
>>>
 Hi All

 I'm dockerizing our static/standalone GoCD server.

 Hence copied the content from the static GoCD server to EFS and mounted
 that EFS content into the container (GoCD server container).

 And with that, container starts and runs but it can't fetch any data,
 and I see the below in the log:

 ```jvm 2| Caused by:
 org.h2.jdbc.JdbcSQLNonTransientConnectionException: Database may be already
 in use: null. Possible solutions: close all other connection(s); use the
 server mode [90020-200]

 jvm 2| Caused by: java.lang.IllegalStateException: The file is
 locked: nio:/godata/db/h2db/cruise.mv.db [1.4.200/7]```

 Looks like the static/old server acquired the lock and when the data
 from that server copied to EFS lock is still maintained and hence the new
 server has the db file locked and couldn't fetch any data.

 Wonder, how are the people handling this situation ?
 We need the old server running as well when the new server is being
 brought up within the docker container.

 Any inputs will be of great help.

>>>
>>> If this is an important GoCD instance, then I urge you to move off H2DB
>>> to PostgreSQL.
>>>
>>> We recently migrated a GoCD instance that has about 6 years of data in
>>> H2DB to use RDS and hosted GoCD itself as a Container. We are running on an
>>> EKS cluster and are making use of the Elastic Agent plugin.
>>>
>>> The H2DB that GoCD uses does not allow multiple processes to share the
>>> same database file. I recommend that you stop the original GoCD process
>>> (the jobs will keep running and will wait for the server to come back),
>>> rsync the DB contents to the EFS store to speed up the copy and remove the
>>> locks, and then start the new GoCD instance as a container with the DB
>>> mounted via EFS.
>>>
>>> If you could consider moving off H2DB and using postgres, then use the
>>> DB Migrator tool and do so. You process would then be:
>>> 1. Stop the older GoCD server
>>> 2. Use the DB Migrator tool to migrate from the H2DB file to a
>>> PostgreSQL instance.
>>> 3. Start the GoCD instance as a Container process with the configuration
>>> to point it to the PostgreSQL instance.
>>>
>>> You may want to triel this once and then perform the actual switch.
>>>
>>>
>>>

 Many thanks,
 Satya

 --
 You received this message because you are subscribed to the Google
 Groups "go-cd" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to go-cd+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/go-cd/CADKEDRoqGkWLv6%3DUG8CA-NAD6sRETHiE6bB5Q2crf2Ezbn-cMw%40mail.gmail.com
 
 .

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to go-cd+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/go-cd/CANiY96af882MjPPSVFowz9hNqakLNS7xF%2Bu7mLWb_2FXQ8fQrg%40mail.gmail.com
>>> 
>>> .
>>>
>> --
>> You 

Re: [go-cd] "/godata/db/h2db/cruise.mv.db" is locked

[Sriram Narayanan]

On Thu, Apr 25, 2024 at 9:53 PM Satya Elipe  wrote:

> Thank you, Sriram.
>
>
> I managed to launch the new container alongside the old one.
>

Good!


>
> As you use the Elastic Agents plugin, can I ask you a question please.
>

Could you post this as a separate email question? That will help
improve discoverability. Meanwhile, I'll read your email a bit and then
have a response ready as well.


>
> I'm encountering some issues with the way agents are launched, assigned,
> and terminated. Despite setting the maximum agent count to two, both agents
> launch sequentially, with only the first being assigned to the job.
>
>
> Here's where it gets tricky: when the staging job completes and triggers
> the production job, I expect one of the active agents to take over.
> Instead, the production job attempts to launch new agents, fails due to the
> max count limit, and runs without any agents, leading to failure.
>
>
> Additionally, some agent instances remain active for an extended period,
> requiring manual termination. This disrupts the workflow significantly.
>
>
> Have you experienced similar issues, or do you have any suggestions for a
> workaround?
>
>
> Thanks in advance !
>
>
>
>
> On Tue, Apr 23, 2024 at 4:13 PM Sriram Narayanan 
> wrote:
>
>>
>>
>> On Tue, Apr 23, 2024 at 10:22 PM Satya Elipe 
>> wrote:
>>
>>> Hi All
>>>
>>> I'm dockerizing our static/standalone GoCD server.
>>>
>>> Hence copied the content from the static GoCD server to EFS and mounted
>>> that EFS content into the container (GoCD server container).
>>>
>>> And with that, container starts and runs but it can't fetch any data,
>>> and I see the below in the log:
>>>
>>> ```jvm 2| Caused by:
>>> org.h2.jdbc.JdbcSQLNonTransientConnectionException: Database may be already
>>> in use: null. Possible solutions: close all other connection(s); use the
>>> server mode [90020-200]
>>>
>>> jvm 2| Caused by: java.lang.IllegalStateException: The file is
>>> locked: nio:/godata/db/h2db/cruise.mv.db [1.4.200/7]```
>>>
>>> Looks like the static/old server acquired the lock and when the data
>>> from that server copied to EFS lock is still maintained and hence the new
>>> server has the db file locked and couldn't fetch any data.
>>>
>>> Wonder, how are the people handling this situation ?
>>> We need the old server running as well when the new server is being
>>> brought up within the docker container.
>>>
>>> Any inputs will be of great help.
>>>
>>
>> If this is an important GoCD instance, then I urge you to move off H2DB
>> to PostgreSQL.
>>
>> We recently migrated a GoCD instance that has about 6 years of data in
>> H2DB to use RDS and hosted GoCD itself as a Container. We are running on an
>> EKS cluster and are making use of the Elastic Agent plugin.
>>
>> The H2DB that GoCD uses does not allow multiple processes to share the
>> same database file. I recommend that you stop the original GoCD process
>> (the jobs will keep running and will wait for the server to come back),
>> rsync the DB contents to the EFS store to speed up the copy and remove the
>> locks, and then start the new GoCD instance as a container with the DB
>> mounted via EFS.
>>
>> If you could consider moving off H2DB and using postgres, then use the DB
>> Migrator tool and do so. You process would then be:
>> 1. Stop the older GoCD server
>> 2. Use the DB Migrator tool to migrate from the H2DB file to a PostgreSQL
>> instance.
>> 3. Start the GoCD instance as a Container process with the configuration
>> to point it to the PostgreSQL instance.
>>
>> You may want to triel this once and then perform the actual switch.
>>
>>
>>
>>>
>>> Many thanks,
>>> Satya
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to go-cd+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/go-cd/CADKEDRoqGkWLv6%3DUG8CA-NAD6sRETHiE6bB5Q2crf2Ezbn-cMw%40mail.gmail.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to go-cd+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/CANiY96af882MjPPSVFowz9hNqakLNS7xF%2Bu7mLWb_2FXQ8fQrg%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion 

Re: [go-cd] "/godata/db/h2db/cruise.mv.db" is locked

[Satya Elipe]

Thank you, Sriram.


I managed to launch the new container alongside the old one.


As you use the Elastic Agents plugin, can I ask you a question please.


I'm encountering some issues with the way agents are launched, assigned,
and terminated. Despite setting the maximum agent count to two, both agents
launch sequentially, with only the first being assigned to the job.


Here's where it gets tricky: when the staging job completes and triggers
the production job, I expect one of the active agents to take over.
Instead, the production job attempts to launch new agents, fails due to the
max count limit, and runs without any agents, leading to failure.


Additionally, some agent instances remain active for an extended period,
requiring manual termination. This disrupts the workflow significantly.


Have you experienced similar issues, or do you have any suggestions for a
workaround?


Thanks in advance !




On Tue, Apr 23, 2024 at 4:13 PM Sriram Narayanan 
wrote:

>
>
> On Tue, Apr 23, 2024 at 10:22 PM Satya Elipe 
> wrote:
>
>> Hi All
>>
>> I'm dockerizing our static/standalone GoCD server.
>>
>> Hence copied the content from the static GoCD server to EFS and mounted
>> that EFS content into the container (GoCD server container).
>>
>> And with that, container starts and runs but it can't fetch any data, and
>> I see the below in the log:
>>
>> ```jvm 2| Caused by:
>> org.h2.jdbc.JdbcSQLNonTransientConnectionException: Database may be already
>> in use: null. Possible solutions: close all other connection(s); use the
>> server mode [90020-200]
>>
>> jvm 2| Caused by: java.lang.IllegalStateException: The file is
>> locked: nio:/godata/db/h2db/cruise.mv.db [1.4.200/7]```
>>
>> Looks like the static/old server acquired the lock and when the data from
>> that server copied to EFS lock is still maintained and hence the new server
>> has the db file locked and couldn't fetch any data.
>>
>> Wonder, how are the people handling this situation ?
>> We need the old server running as well when the new server is being
>> brought up within the docker container.
>>
>> Any inputs will be of great help.
>>
>
> If this is an important GoCD instance, then I urge you to move off H2DB to
> PostgreSQL.
>
> We recently migrated a GoCD instance that has about 6 years of data in
> H2DB to use RDS and hosted GoCD itself as a Container. We are running on an
> EKS cluster and are making use of the Elastic Agent plugin.
>
> The H2DB that GoCD uses does not allow multiple processes to share the
> same database file. I recommend that you stop the original GoCD process
> (the jobs will keep running and will wait for the server to come back),
> rsync the DB contents to the EFS store to speed up the copy and remove the
> locks, and then start the new GoCD instance as a container with the DB
> mounted via EFS.
>
> If you could consider moving off H2DB and using postgres, then use the DB
> Migrator tool and do so. You process would then be:
> 1. Stop the older GoCD server
> 2. Use the DB Migrator tool to migrate from the H2DB file to a PostgreSQL
> instance.
> 3. Start the GoCD instance as a Container process with the configuration
> to point it to the PostgreSQL instance.
>
> You may want to triel this once and then perform the actual switch.
>
>
>
>>
>> Many thanks,
>> Satya
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to go-cd+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/CADKEDRoqGkWLv6%3DUG8CA-NAD6sRETHiE6bB5Q2crf2Ezbn-cMw%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CANiY96af882MjPPSVFowz9hNqakLNS7xF%2Bu7mLWb_2FXQ8fQrg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CADKEDRpg_6YdfCGt7_sOp%3DkOyb6DVSf19mAuJdzeXbvgk76VAA%40mail.gmail.com.

Re: [go-cd] GOCD server Db issue

[Chad Wilson]

You may have killed GoCD while it was starting up? If you are sure that you
do not have multiple processes accessing the same database at the same time
and it is still stuck, you will have to try manually clearing the lock
.
Backup your database file before doing this.

Need to connect to the database and use this to clear the lock. You'll have
to do so using something like is done here

but *instead
of running an export/import* to recreate the DB you just want to

1) backup the cruise.mv.db file ()
2) find the h2-1.4.200.jar file
3) create clear-liquibase-lock.sql with UPDATE DATABASECHANGELOGLOCK SET
LOCKED=0
4) run something like this on the main DB file java -cp h2-1.4.200.jar
org.h2.tools.RunScript -url jdbc:h2:./cruise -user sa -script
clear-liquibase-lock.sql

-Chad


On Thu, Apr 25, 2024 at 7:22 PM Vijayakumaran A. <
vijayakumara...@praniontech.com> wrote:

> HI Team,
>
> When we restarting gocd server we have below error please advice us. We
> use h2db.
>
> 2024-04-25 16:22:33,896 INFO  [WrapperJarAppMain] Jetty9Server:199 -
> Configuring Jetty using /etc/go/jetty.xml
> 2024-04-25 16:22:33,958 WARN  [WrapperJarAppMain] Server:357 -
> ErrorPageMapper not supported for Server level Error Handling
> 2024-04-25 16:22:34,088 WARN  [WrapperJarAppMain] AbstractHandler:96 - No
> Server set for ResourceHandler@5904a259{STOPPED}
> 2024-04-25 16:22:40,659 WARN  [WrapperJarAppMain] ConnectionManager:117 -
> The file /etc/go/db.properties specified by `go.db.config` does not exist.
> 2024-04-25 16:22:41,319 INFO  [WrapperJarAppMain] DatabaseMigrator:40 -
> Upgrading database, this might take a while depending on the size of the
> database.
> 2024-04-25 16:22:41,320 INFO  [WrapperJarAppMain] DatabaseMigrator:49 -
> 
> 2024-04-25 16:22:41,320 INFO  [WrapperJarAppMain] DatabaseMigrator:49 -
> WARNING: Shutting down your server at this point will lead to a database
> corruption. Please wait until the database upgrade completes.
> 2024-04-25 16:22:41,320 INFO  [WrapperJarAppMain] DatabaseMigrator:49 -
> 
> 2024-04-25 16:27:42,107 ERROR [WrapperJarAppMain] DatabaseMigrator:65 -
> Unable to create database upgrade script for database. The problem was:
> Could not acquire change log lock.  Currently locked by 172.21.0.1
> (172.21.0.1) since 4/25/24, 12:19 PM
> liquibase.exception.LockException: Could not acquire change log lock.
> Currently locked by 172.21.0.1 (172.21.0.1) since 4/25/24, 12:19 PM
>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/a1b7a304-1559-441e-b002-b385ae93184en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CAA1RwH_UXfcQVRzEoksuCSCKvjPAvQ3%3D%2BmDm35ZLqHtwgK%2Bz_Q%40mail.gmail.com.