Re: [go-cd] GOCD restart issue

2024-05-07 Thread Chad Wilson 
Did someone recently change something in your compose volume mounts?

It's probably not a good idea to mount things directly into /go-working-dir
like the line "*-
./gocd/docker-elastic-agents-3.2.3-415.jar:/go-working-dir/plugins/external*
"

The container entrypoint expects to own and manage these locations with
symlinks, as Ram mentioned earlier. This is why the */go-working-dir* path
is not documented - it is internal and you shouldn't interfere with it or
couple your config to undocumented internals.

This mount being there at container start is probably going to prevent the
entrypoint from linking things to /godata correctly and you will probably
end up with a `plugins` directory which is not writable by GoCD and errors
like you had below.

Since you already have a /godata volume - put the plugin jars directly into
./gocd/data/plugins/external (on your host machine), and remove the
individual jar mount. Basically, please follow the documentation:
https://hub.docker.com/r/gocd/gocd-server

If removing that plugin mount still doesn't work, check the other mounts
and the permissions of those directories as seen by the container when it
starts by execing into it and exploring.

-Chad

On Tue, May 7, 2024 at 11:48 PM Vijayakumaran A. <
vijayakumara...@praniontech.com> wrote:

> We have started container using docker start servicename.
>
> Here is the composer file of gocd server part
>
> server:
> container_name: gocd-server
> image: gocd/gocd-server:v23.4.0
> restart: always
> ports:
>   - "8153:8153"
> volumes:
>   - ./gocd/data:/godata
>   - ./gocd/data/home:/home/go
>   - ./gocd/scripts/server:/docker-entrypoint.d
>   - ./gocd/scripts/shared:/shared
>   - ./gocd/passwd:/godata/config/passwd
>   -
> ./gocd/docker-elastic-agents-3.2.3-415.jar:/go-working-dir/plugins/external
> networks:
>   - gocd
>
>
>
> On Tuesday, May 7, 2024 at 9:05:30 PM UTC+5:30 Chad Wilson wrote:
>
>> Please share how you are starting the container and what mounts you are
>> using, especially to /godata.
>>
>> Your container may have permissions issues writing to /godata or issues
>> mounting the filesystem in writeable mode. Is it some kind of NFS volume?
>> Is it mounted by another container/host already and thus is being mounted
>> in read-only mode? is this a Helm/Kubernetes install, or just a regular
>> container on a server?
>>
>> You can exec into the container and check /godata and /go-working-dir are
>> writable by the `go` user.
>>
>> -Chad
>>
>> On Tue, May 7, 2024 at 11:21 PM Sriram Narayanan 
>> wrote:
>>
>>>
>>>
>>> On Tue, May 7, 2024 at 10:33 PM Vijayakumaran A. <
>>> vijayak...@praniontech.com> wrote:
>>>
 Team,we have just restarted the gocd container now gocd not started.

 Having below in docker logs please help
 [image: go1.PNG][image: go2.PNG]


>>> Looks like something is wrong with the container system.
>>> /go-working-dir/ is a runtime symlink directory created here
>>> https://github.com/gocd/docker-gocd-server/blob/master/docker-entrypoint.sh#L41
>>>
>>> If a directory is not getting created, then there is an underlying issue
>>> unrelated to GoCD and entirely related to the underlying container system.
>>> Please see if a container restart helps you continue. You should also check
>>> the container system's logs. To share something that we saw at our project
>>> two weeks ago, one of the EKS clusters was still running at version 1.23
>>> and we would often see the error "Out of storage space" (sometimes once a
>>> day). When I'd exec into the container, I found that we were unable to make
>>> directories but were able to create files. We upgrades to EKS 1.24 and that
>>> error has not recurred since then.
>>>
>>> I also see that you are using GoCD 23.4 and the H2DB. If this is a GoCD
>>> setup that you are using for your project, then I urge you to use the
>>> postgres database instead. See
>>> https://github.com/gocd/gocd-database-migrator
>>>
>>>
 --
 You received this message because you are subscribed to the Google
 Groups "go-cd" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to go-cd+un...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/go-cd/5df023fd-f859-4a6b-99e7-92e3063653e7n%40googlegroups.com
 
 .

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to go-cd+un...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/go-cd/CANiY96bSuG28hLLP_A%2Bij%3DKt0xO%3Digxb3zawjKXcj_bjnAWpLg%40mail.gmail.com
>>>

Re: [go-cd] GOCD restart issue

2024-05-07 Thread Vijayakumaran A. 
its is just regular container in linux server and used local disk for volume

On Tuesday, May 7, 2024 at 9:18:37 PM UTC+5:30 Vijayakumaran A. wrote:

> We have started container using docker start servicename.
>
> Here is the composer file of gocd server part 
>
> server:
> container_name: gocd-server
> image: gocd/gocd-server:v23.4.0
> restart: always
> ports:
>   - "8153:8153"
> volumes:
>   - ./gocd/data:/godata
>   - ./gocd/data/home:/home/go
>   - ./gocd/scripts/server:/docker-entrypoint.d
>   - ./gocd/scripts/shared:/shared
>   - ./gocd/passwd:/godata/config/passwd
>   - 
> ./gocd/docker-elastic-agents-3.2.3-415.jar:/go-working-dir/plugins/external
> networks:
>   - gocd
>
>
>
> On Tuesday, May 7, 2024 at 9:05:30 PM UTC+5:30 Chad Wilson wrote:
>
>> Please share how you are starting the container and what mounts you are 
>> using, especially to /godata.
>>
>> Your container may have permissions issues writing to /godata or issues 
>> mounting the filesystem in writeable mode. Is it some kind of NFS volume? 
>> Is it mounted by another container/host already and thus is being mounted 
>> in read-only mode? is this a Helm/Kubernetes install, or just a regular 
>> container on a server?
>>
>> You can exec into the container and check /godata and /go-working-dir are 
>> writable by the `go` user.
>>
>> -Chad
>>
>> On Tue, May 7, 2024 at 11:21 PM Sriram Narayanan  
>> wrote:
>>
>>>
>>>
>>> On Tue, May 7, 2024 at 10:33 PM Vijayakumaran A. <
>>> vijayak...@praniontech.com> wrote:
>>>
 Team,we have just restarted the gocd container now gocd not started.

 Having below in docker logs please help 
 [image: go1.PNG][image: go2.PNG]


>>> Looks like something is wrong with the container system. 
>>> /go-working-dir/ is a runtime symlink directory created here 
>>> https://github.com/gocd/docker-gocd-server/blob/master/docker-entrypoint.sh#L41
>>>
>>> If a directory is not getting created, then there is an underlying issue 
>>> unrelated to GoCD and entirely related to the underlying container system. 
>>> Please see if a container restart helps you continue. You should also check 
>>> the container system's logs. To share something that we saw at our project 
>>> two weeks ago, one of the EKS clusters was still running at version 1.23 
>>> and we would often see the error "Out of storage space" (sometimes once a 
>>> day). When I'd exec into the container, I found that we were unable to make 
>>> directories but were able to create files. We upgrades to EKS 1.24 and that 
>>> error has not recurred since then.
>>>
>>> I also see that you are using GoCD 23.4 and the H2DB. If this is a GoCD 
>>> setup that you are using for your project, then I urge you to use the 
>>> postgres database instead. See 
>>> https://github.com/gocd/gocd-database-migrator
>>>  
>>>
 -- 
 You received this message because you are subscribed to the Google 
 Groups "go-cd" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to go-cd+un...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/go-cd/5df023fd-f859-4a6b-99e7-92e3063653e7n%40googlegroups.com
  
 
 .

>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to go-cd+un...@googlegroups.com.
>>>
>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/go-cd/CANiY96bSuG28hLLP_A%2Bij%3DKt0xO%3Digxb3zawjKXcj_bjnAWpLg%40mail.gmail.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/2617898c-d51e-4d67-9465-8027874ff647n%40googlegroups.com.

Re: [go-cd] GOCD restart issue

2024-05-07 Thread Vijayakumaran A. 
We have started container using docker start servicename.

Here is the composer file of gocd server part 

server:
container_name: gocd-server
image: gocd/gocd-server:v23.4.0
restart: always
ports:
  - "8153:8153"
volumes:
  - ./gocd/data:/godata
  - ./gocd/data/home:/home/go
  - ./gocd/scripts/server:/docker-entrypoint.d
  - ./gocd/scripts/shared:/shared
  - ./gocd/passwd:/godata/config/passwd
  - 
./gocd/docker-elastic-agents-3.2.3-415.jar:/go-working-dir/plugins/external
networks:
  - gocd



On Tuesday, May 7, 2024 at 9:05:30 PM UTC+5:30 Chad Wilson wrote:

> Please share how you are starting the container and what mounts you are 
> using, especially to /godata.
>
> Your container may have permissions issues writing to /godata or issues 
> mounting the filesystem in writeable mode. Is it some kind of NFS volume? 
> Is it mounted by another container/host already and thus is being mounted 
> in read-only mode? is this a Helm/Kubernetes install, or just a regular 
> container on a server?
>
> You can exec into the container and check /godata and /go-working-dir are 
> writable by the `go` user.
>
> -Chad
>
> On Tue, May 7, 2024 at 11:21 PM Sriram Narayanan  
> wrote:
>
>>
>>
>> On Tue, May 7, 2024 at 10:33 PM Vijayakumaran A. <
>> vijayak...@praniontech.com> wrote:
>>
>>> Team,we have just restarted the gocd container now gocd not started.
>>>
>>> Having below in docker logs please help 
>>> [image: go1.PNG][image: go2.PNG]
>>>
>>>
>> Looks like something is wrong with the container system. /go-working-dir/ 
>> is a runtime symlink directory created here 
>> https://github.com/gocd/docker-gocd-server/blob/master/docker-entrypoint.sh#L41
>>
>> If a directory is not getting created, then there is an underlying issue 
>> unrelated to GoCD and entirely related to the underlying container system. 
>> Please see if a container restart helps you continue. You should also check 
>> the container system's logs. To share something that we saw at our project 
>> two weeks ago, one of the EKS clusters was still running at version 1.23 
>> and we would often see the error "Out of storage space" (sometimes once a 
>> day). When I'd exec into the container, I found that we were unable to make 
>> directories but were able to create files. We upgrades to EKS 1.24 and that 
>> error has not recurred since then.
>>
>> I also see that you are using GoCD 23.4 and the H2DB. If this is a GoCD 
>> setup that you are using for your project, then I urge you to use the 
>> postgres database instead. See 
>> https://github.com/gocd/gocd-database-migrator
>>  
>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to go-cd+un...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/go-cd/5df023fd-f859-4a6b-99e7-92e3063653e7n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go-cd+un...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/CANiY96bSuG28hLLP_A%2Bij%3DKt0xO%3Digxb3zawjKXcj_bjnAWpLg%40mail.gmail.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/d1136435-78d7-42b6-9b1c-5c2d2a49ded2n%40googlegroups.com.

Re: [go-cd] Mount .ssh from EFS to the container (from the image gocd/gocd-server:v22.3.0)

2024-05-07 Thread Chad Wilson 
To add onto the many options here, if you only need the SSH keys to be used
by Git for clones etc, you can entirely customise how GIT uses SSH using
the *GIT_SSH_COMMAND* env var
;
set at the container level.

GIT_SSH_COMMAND="ssh -i /path/to/your/private/key"

Then you can put the private key anywhere you like (including /godata - not
just the home dir) which the GoCD server/agent has access to (as long as it
has the right file permissions (400 or 600) and is readable by
`go`/UID=1000 user, as Ram notes).

-Chad

On Tue, May 7, 2024 at 11:11 PM Sriram Narayanan 
wrote:

>
>
> On Mon, May 6, 2024 at 3:30 AM Jason Smyth  wrote:
>
>> Hi Satya,
>>
>> A possible workaround to the limitation is updating the server image and
>> adding a symlink that points ~/.ssh/ to wherever you want to actually mount
>> the data.
>>
>> I have never experimented with using a symlink for the .ssh directory,
>> though, so this may not work.
>>
>
> I haven't tried this yet, but one would explore adding a custom shell
> script at the /docker-entrypoint.d/ mount point which could create such a
> symlink
>
> Nice tip, Jason.
>
>
>>
>> Hope this helps,
>> Jason
>>
>>
>> On Sunday 28 April 2024 at 12:12:16 UTC-4 Sriram Narayanan wrote:
>>
>>> On Sat, Apr 27, 2024 at 7:10 PM Satya Elipe  wrote:
>>>
 Thank you Sriram.

 So, ".ssh" folder mounting will be separate from the rest of the data
 (/godata, for plugins, pipelines, db etc)...so there would be two separate
 mount points into the container ?

 I'm using ECS at the moment and not kubernetes, so my task definition
 will have two mount points like below:

 ```

 "mountPoints": [
 {
 "sourceVolume": "efs_id:/godata",

 "containerPath": "/godata"
 },
 {
 "sourceVolume": "efs_id:/godata/.ssh",

 "containerPath": "/home/go/.ssh"
 }
 ],

 ```

 So mounting /godata and efs_id:/godata/.ssh from EFS into the
 container at /godata and /home/go/.ssh locations respectively (per
 above code) seems to work.

 In this case entry_point.sh from the base image is able to
 map/consider and execute them properly, hence the server is up and running
 and functioning properly.

 Is that the way it has to be, I think the github repo for gocd server
 says that I guess, but perhaps I feel that extra mount point just for .ssh
 is overkill and if .ssh can also be entertained by entry_point.sh from one
 single mount point /godata in my case, that would be great ?

 If I do not mount .ssh into /home/go/.ssh separately into the container
 - things seem to fail complaining that "key verification failed", I'm not
 sure whether I'm still missing something here.

>>>
>>> Hey, I had got caught by surprise earlier during the "elastic agent"
>>> discussions and had assumed that you must be using EKS. Sorry, my bias had
>>> clouded my judgement then. Thankfully Chad and you cleared that up.
>>>
>>> ssh by default checks ~/.ssh/ for the keys. Within the GoCD server and
>>> agent containers, this home (~) is the /home/go directory, and hence we
>>> mount the .ssh folder there. There are use cases where the keys are made
>>> available via a different network share and not mixed with configurations
>>> that regular GoCD admins would have access to, and hence being able to
>>> mount from a separate place to ~/.ssh is helpful. You could always place
>>> the .ssh directory along side other directories that would get to godata,
>>> while also explicitly specifying a mount to /home/go. At present, GoCD does
>>> not have a configuration option to point it to a private key at a path
>>> other than ~/ssh
>>>
>>> https://docs.gocd.org/current/faq/docker_container_ssh_keys.html
>>>
>>>

 Many thanks
 Satya

 On Thu, Apr 25, 2024 at 3:31 PM Sriram Narayanan 
 wrote:

>
>
> On Thu, Apr 25, 2024 at 10:16 PM Satya Elipe 
> wrote:
>
>> Hi all
>>
>> Wonder, what's the way around to mount .ssh from EFS into the gocd
>> base container (from the image gocd/gocd-server:v22.3.0).
>>
>>
>> We have saved all our content into EFS under /godata and maps that
>> into the container as /godata.
>>
>>
>> We are using gocd/gocd-server:v22.3.0.
>>
>>
>> It all runs good, mapping was fine too but just one thing that’s not
>> happening is “.ssh” folder.
>>
>>
>> I have .ssh with all required keys in EFS under /godata and /godata
>> within the container also has .ssh but not /go-working-dir.
>>
>>
>> Is that supported, am I mis-configuring it, or do we need to handle
>> that outside of the base image ?
>>
>

Re: [go-cd] Mount .ssh from EFS to the container (from the image gocd/gocd-server:v22.3.0)

2024-05-07 Thread Sriram Narayanan 
On Mon, May 6, 2024 at 3:30 AM Jason Smyth  wrote:

> Hi Satya,
>
> A possible workaround to the limitation is updating the server image and
> adding a symlink that points ~/.ssh/ to wherever you want to actually mount
> the data.
>
> I have never experimented with using a symlink for the .ssh directory,
> though, so this may not work.
>

I haven't tried this yet, but one would explore adding a custom shell
script at the /docker-entrypoint.d/ mount point which could create such a
symlink

Nice tip, Jason.


>
> Hope this helps,
> Jason
>
>
> On Sunday 28 April 2024 at 12:12:16 UTC-4 Sriram Narayanan wrote:
>
>> On Sat, Apr 27, 2024 at 7:10 PM Satya Elipe  wrote:
>>
>>> Thank you Sriram.
>>>
>>> So, ".ssh" folder mounting will be separate from the rest of the data
>>> (/godata, for plugins, pipelines, db etc)...so there would be two separate
>>> mount points into the container ?
>>>
>>> I'm using ECS at the moment and not kubernetes, so my task definition
>>> will have two mount points like below:
>>>
>>> ```
>>>
>>> "mountPoints": [
>>> {
>>> "sourceVolume": "efs_id:/godata",
>>>
>>> "containerPath": "/godata"
>>> },
>>> {
>>> "sourceVolume": "efs_id:/godata/.ssh",
>>>
>>> "containerPath": "/home/go/.ssh"
>>> }
>>> ],
>>>
>>> ```
>>>
>>> So mounting /godata and efs_id:/godata/.ssh from EFS into the container
>>> at /godata and /home/go/.ssh locations respectively (per above code) seems
>>> to work.
>>>
>>> In this case entry_point.sh from the base image is able to map/consider
>>> and execute them properly, hence the server is up and running and
>>> functioning properly.
>>>
>>> Is that the way it has to be, I think the github repo for gocd server
>>> says that I guess, but perhaps I feel that extra mount point just for .ssh
>>> is overkill and if .ssh can also be entertained by entry_point.sh from one
>>> single mount point /godata in my case, that would be great ?
>>>
>>> If I do not mount .ssh into /home/go/.ssh separately into the container
>>> - things seem to fail complaining that "key verification failed", I'm not
>>> sure whether I'm still missing something here.
>>>
>>
>> Hey, I had got caught by surprise earlier during the "elastic agent"
>> discussions and had assumed that you must be using EKS. Sorry, my bias had
>> clouded my judgement then. Thankfully Chad and you cleared that up.
>>
>> ssh by default checks ~/.ssh/ for the keys. Within the GoCD server and
>> agent containers, this home (~) is the /home/go directory, and hence we
>> mount the .ssh folder there. There are use cases where the keys are made
>> available via a different network share and not mixed with configurations
>> that regular GoCD admins would have access to, and hence being able to
>> mount from a separate place to ~/.ssh is helpful. You could always place
>> the .ssh directory along side other directories that would get to godata,
>> while also explicitly specifying a mount to /home/go. At present, GoCD does
>> not have a configuration option to point it to a private key at a path
>> other than ~/ssh
>>
>> https://docs.gocd.org/current/faq/docker_container_ssh_keys.html
>>
>>
>>>
>>> Many thanks
>>> Satya
>>>
>>> On Thu, Apr 25, 2024 at 3:31 PM Sriram Narayanan 
>>> wrote:
>>>


 On Thu, Apr 25, 2024 at 10:16 PM Satya Elipe 
 wrote:

> Hi all
>
> Wonder, what's the way around to mount .ssh from EFS into the gocd
> base container (from the image gocd/gocd-server:v22.3.0).
>
>
> We have saved all our content into EFS under /godata and maps that
> into the container as /godata.
>
>
> We are using gocd/gocd-server:v22.3.0.
>
>
> It all runs good, mapping was fine too but just one thing that’s not
> happening is “.ssh” folder.
>
>
> I have .ssh with all required keys in EFS under /godata and /godata
> within the container also has .ssh but not /go-working-dir.
>
>
> Is that supported, am I mis-configuring it, or do we need to handle
> that outside of the base image ?
>

 At a high level, the .ssh folder should be mounted into /home/go.
 e.g. docker run -v /path/to/godata:/godata -v /path/to/home-dir:/home/go
 gocd/gocd-server:v23.5.0
 IMPORTANT: You must set the user ID of the files within .ssh to 1000.
 This is the user ID of the gocd process within the container.

 See:
 https://github.com/gocd/docker-gocd-server?tab=readme-ov-file#mounting-volumes

 Given that you are using Kubernetes, please see the Helm chart
 documentation here
 https://github.com/gocd/helm-chart/blob/master/gocd/README.md

 It provides info on just about every configurable attribute for the
 GoCD server and the agent.

 Of particular importance for you are these two attributes: