[go-cd] Re: PATH not honoured

[Jason Smyth]

Hi Pablo,

Why do you feel that the PATH variable is not being honoured? You provide 
two examples of details that suggest that it _should_ be working, but 
assert that is not. What behaviour are you seeing that gives you that 
impression?

With that said, I'm not sure that overriding the PATH is the best way to 
accomplish your objective. I think using the Java-specific variables like 
JAVA_HOME would be a cleaner way to handle your example case of multiple 
Java versions.

I'm struggling to imagine a scenario where completely overriding the PATH 
isn't more trouble than its worth, though of course that doesn't mean that 
such a use case doesn't exist.

Hope this helps.

Cheers,
Jason Smyth


On Monday 8 July 2024 at 08:53:41 UTC-4 Pablo León wrote:

> Hi,
>
> I have configured a PATH environment variable in my pipeline,
> I see an "overriding environment variable 'PATH' with value 
> '/usr/lib/jvm/java-11-openjdk:/usr/bin'" line in the logs, I even can see 
> the new value of PATH in an "printenv" task.
>
> However, this value doesn't seems to be honoured as a search path!
>
> What is the recommend way to change the search path in go-cd?
>
> I know I can change this in the 
> /usr/share/go-agent/wrapper-config/wrapper-properties.conf 
> file, but what if I need a different paths for each job? (for example for 
> different java versions)
>
> Cheers!
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/dd726fae-4921-4550-869d-0f2169c366can%40googlegroups.com.

Re: [go-cd] Assign Administrator Permissions to Role Without Manually Modifying cruise-config.xml

[Jason Smyth]

Hi Chad, Aravind,

Thank you for the feedback.

The primary context of this question is creating our own internal 
documentation for how to configure GoCD authentication and authorization. 
Since this (theoretically) only needs to be done once on initial setup, 
it's probably not worth the headache of documenting how to do it via the 
API. For the sake of simplicity, the following workflow should suffice:

   1. Install GoCD.
   2. Install the LDAP auth plugin.
   3. Configure the LDAP connector.
   4. Log in.
   5. Configure the Admin role (and make sure you are in the group).
   6. Navigate to the Users Management page (and make sure you have been 
   assigned the Admin role).
   7. Click the toggle in the SYSTEM ADMIN column to make your user the 
   super-user.
   8. Navigate to the Config XML page.
   9. Locate the server/security/admins node (should be near the bottom of 
   the XML document).
   10. Replace "yourUserName" with 
   "AdminRoleName".

Now that I type it all out, it doesn't actually seem all that "simple", but 
still, good enough to get the job done. Since we are using LDAP 
authentication, we can manage any subsequent changes to super-user 
privileges via the appropriate AD group.

To correct a small mistake in Chad's list of delegable permissions, the 
actual list is: Environments, Config Repos, Cluster Profiles, and Elastic 
Agent Profiles. Pipeline Groups are, sadly, not included (hence my other 
email thread).

Thanks again,
Jason


On Sunday 2 June 2024 at 08:27:26 UTC-4 Aravind SV wrote:

> Hello!
>
> Yes, what Chad says makes sense to me. 
>
>
> I don't recall discussing a UI for the  tag though. There is an 
> API, if that helps you,  Jason: 
> https://api.gocd.org/current/#update-system-admins
>
> Cheers,
> Aravind
>
>
> On Sat, 1 Jun 2024 at 10:21, Chad Wilson  wrote:
>
>> Interesting, had not noticed that limitation (didn't know you could 
>> assign a role to super-admin at all!). Personally I don't know a UI-driven 
>> way.
>>
>> Looks like it was vaguely discussed as part of 
>> https://github.com/gocd/gocd/issues/3712 but I cannot see that 
>> possibility to map that within the Role Management page, nor a specific 
>> open issue/feature request for that.
>>
>> I believe there were a number of aspects of more granular auth for 
>> global entities <https://github.com/gocd/gocd/issues/7222> that wasn't 
>> necessarily completed, but I think this work was intended to reduce the 
>> need for super-admins in general. Keep in mind this work was mainly 
>> happening in H2 2019 and Thoughtworks announced closure of studios for end 
>> 2020 on Nov 18 2019. I believe the focus went to open sourcing pieces in H1 
>> 2020 so this possibly never got to its full vision :-). 
>>
>> Having said that, from your other post it appears you are on a very old 
>> GoCD version so I am not sure if what you are seeing is the same as what I 
>> am seeing now.
>>
>> In any case, you may wish to update to (or play with a trial of) a later 
>> version to see if a sufficient number of global entities can be directly 
>> delegated to roles such that the super-admin permissions are much less 
>> necessary than earlier, and perhaps less necessary to map to roles 
>> frequently. I believe it at least supports environments/cluster 
>> profiles/elastic profiles/pipeline groups.
>>
>> -Chad
>>
>> On Sat, Jun 1, 2024 at 4:22 AM Jason Smyth  wrote:
>>
>>> Hi everyone,
>>>
>>> We are looking to improve our GoCD permissions management by using more 
>>> role-based permissions.
>>>
>>>
>>> The role-based security documentation 
>>> <https://docs.gocd.org/19.8.0/configuration/dev_authorization.html#role-based-security>
>>>  states 
>>> that it is possible to add a role to the server's security node admin list 
>>> and that all users of the role will inherit admin permissions.
>>>
>>> We tested this and it seems to work as described, however I am unable to 
>>> find a mechanism for managing this within GoCD. I was only able to get it 
>>> working by manually editing the cruise-config.xml file.
>>>
>>> Am I missing something, or is this really the only way to manage 
>>> role-based administrative access to GoCD?
>>>
>>> Thanks in advance,
>>> Jason
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to

[go-cd] Managing Pipeline Permissions for Multiple Pipeline Groups

[Jason Smyth]

Hi everyone,

We are working on upgrading from GoCD 19.8.0 to the current version. One of 
the major changes we need to account for is the default permissions on 
pipeline groups.

In 19.8.0, pipelines are open by default, i.e., if there are no permissions 
explicitly defined for a pipeline group, all users can view and operate the 
pipelines it contains. In current versions, pipelines are secure by 
default; if there are no permissions explicitly defined for a pipeline 
group then only system administrators can view/operate them.

Our current model is this:

   - All pipelines are stored in a single config repo.
   - Pipeline groups are used to represent an individual application.
   - A pipeline group generally consists of a build pipeline and several 
   deployment pipelines.
   - Production pipelines are separated into their own pipeline group 
   because they already have some requirements around restricting their 
   operability.


This presents a couple of challenges:

   1. When moving from open-by-default to secure-by-default we will need to 
   explicitly specify the permissions for ~230 pipeline groups, all of which 
   have essentially the same permissions requirements.
   2. Post upgrade, we cannot restrict system administration privileges 
   because anyone who has access to create a new pipeline group via the config 
   repo will need sysadmin access to set the pipeline group permissions after 
   the pipelines are imported.

Does GoCD have any mechanism for grouping pipeline groups for the purpose 
of standardizing permissions across them? Alternately, is there a way that 
we can define permissions in the config repo instead of having to put them 
into cruise-config.xml post-import?

Any thoughts or suggestions are welcome.

Regards,
Jason

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/7a022c24-8d18-48dc-8909-2d6c5330e49bn%40googlegroups.com.

[go-cd] Assign Administrator Permissions to Role Without Manually Modifying cruise-config.xml

[Jason Smyth]

Hi everyone,

We are looking to improve our GoCD permissions management by using more 
role-based permissions.


The role-based security documentation 

 states 
that it is possible to add a role to the server's security node admin list 
and that all users of the role will inherit admin permissions.

We tested this and it seems to work as described, however I am unable to 
find a mechanism for managing this within GoCD. I was only able to get it 
working by manually editing the cruise-config.xml file.

Am I missing something, or is this really the only way to manage role-based 
administrative access to GoCD?

Thanks in advance,
Jason

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/15df8afb-fa71-4d37-aa5d-a1d01939cb5cn%40googlegroups.com.

RE: [go-cd] How to Configure Redundant LDAP Authorization?

[Jason Smyth]

Hi Chad,

Thank you for the feedback.

I ran some tests, and it seems that the LDAP Authorization plugin does not 
validate that the certificate name matches. At least, I was able to 
successfully connect using both hostname and IP address.

So, it seems our options are:


  1.  Use round-robin DNS or a TCP load balancer and accept the risk that a 
future update to the plugin may tighten security and break our implementation, 
or
  2.  Create duplicate LDAP Authorization connectors like we currently have for 
LDAP Authentication and accept that we would need duplicate role configurations 
as well.

Does that seem about right? Is there anything I am mssing?

Regards,
Jason Smyth
From: go-cd@googlegroups.com  On Behalf Of Chad Wilson
Sent: Friday, May 17, 2024 11:35 PM
To: go-cd@googlegroups.com
Subject: Re: [go-cd] How to Configure Redundant LDAP Authorization?

Minor correction, it's possible the LDAP authz plugin is validating the certs 
in a way that the authentication plugin does not, despite both being on old 
LDAP client API versions. Would need to dig deeper to validate.

https://github.com/gocd/gocd-ldap-authentication-plugin/blob/a0236ed52cc95646f30c72a20360893c548c/src/main/java/cd/go/apacheds/ConnectionConfiguration.java

Vs

https://github.com/gocd/gocd-ldap-authorization-plugin/blob/master/src/main/java/com/thoughtworks/gocd/authorization/ldap/apacheds/ConnectionConfiguration.java

-Chad
On Sat, 18 May 2024, 11:03 Chad Wilson, 
mailto:ch...@thoughtworks.com>> wrote:
I discovered recently that the plugins are on an ancient version of the Apache 
LDAP library that means they don't actually seem to validate the server certs 
fully by default (e.g on expiry), so may not validate the hostname either. But 
that's probably a bug, not a feature? Your call if you want to give it a go and 
rely on it for now.

If the default is made more secure in future I'd like to think there's be an 
opt-out. But yeah - those LDAP plugins need some love.

-Chad
On Sat, 18 May 2024, 02:29 Jason Smyth, 
mailto:jsm...@taqauto.com>> wrote:
Hi Sriram,

Thank you for the feedback.

Do you know how the plugin handles SSL negotiation? We considered DNS 
round-robin but ruled it a non-starter, under the assumption that LDAPS would 
require that the hostname and certificate name match.

Regards,
Jason Smyth

From: go-cd@googlegroups.com<mailto:go-cd@googlegroups.com> 
mailto:go-cd@googlegroups.com>> On Behalf Of Sriram 
Narayanan
Sent: Friday, May 17, 2024 12:46 PM
To: go-cd@googlegroups.com<mailto:go-cd@googlegroups.com>
Subject: Re: [go-cd] How to Configure Redundant LDAP Authorization?



On Fri, 17 May 2024 at 10:53 PM, Jason Smyth 
mailto:jsm...@taqauto.com>> wrote:
Hi everyone,

We are looking to move from the bundled LDAP authentication plugin to the LDAP 
authorization plugin<https://github.com/gocd/gocd-ldap-authorization-plugin>.

For redundancy, our current setup uses 2 LDAP connectors, each pointing to a 
different Active Directory domain controller in the same domain.  If we switch 
to the LDAP authorization plugin we can still create redundant authentication 
links, but does this mean we will need to create duplicate role configurations 
as well?

Is there any documentation we should be referencing in terms of the "right" way 
to set up a redundant connection to AD?


For connection redundancy, I’ve used TCP load balancers. For on premise setups, 
I’ve used DNS round robin to point to two different load balancer instances.



Any feedback is appreciated.

Regards,
Jason Smyth

--
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
go-cd+unsubscr...@googlegroups.com<mailto:go-cd+unsubscr...@googlegroups.com>.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/4b37a890-f442-4966-a053-0fb985f73e3cn%40googlegroups.com<https://groups.google.com/d/msgid/go-cd/4b37a890-f442-4966-a053-0fb985f73e3cn%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to a topic in the Google 
Groups "go-cd" group.
To unsubscribe from this topic, visit 
https://groups.google.com/d/topic/go-cd/eEHCCj-vOuo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to 
go-cd+unsubscr...@googlegroups.com<mailto:go-cd+unsubscr...@googlegroups.com>.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CANiY96ZECHfCOjUw5f-XS6kvsChV%2B8K%3Dry21%3DW%3DeOuFM011opw%40mail.gmail.com<https://groups.google.com/d/msgid/go-cd/CANiY96ZECHfCOjUw5f-XS6kvsChV%2B8K%3Dry21%3DW%3DeOuFM011opw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this gr

RE: [go-cd] How to Configure Redundant LDAP Authorization?

[Jason Smyth]

Hi Sriram,

Thank you for the feedback.

Do you know how the plugin handles SSL negotiation? We considered DNS 
round-robin but ruled it a non-starter, under the assumption that LDAPS would 
require that the hostname and certificate name match.

Regards,
Jason Smyth

From: go-cd@googlegroups.com  On Behalf Of Sriram 
Narayanan
Sent: Friday, May 17, 2024 12:46 PM
To: go-cd@googlegroups.com
Subject: Re: [go-cd] How to Configure Redundant LDAP Authorization?



On Fri, 17 May 2024 at 10:53 PM, Jason Smyth 
mailto:jsm...@taqauto.com>> wrote:
Hi everyone,

We are looking to move from the bundled LDAP authentication plugin to the LDAP 
authorization plugin<https://github.com/gocd/gocd-ldap-authorization-plugin>.

For redundancy, our current setup uses 2 LDAP connectors, each pointing to a 
different Active Directory domain controller in the same domain.  If we switch 
to the LDAP authorization plugin we can still create redundant authentication 
links, but does this mean we will need to create duplicate role configurations 
as well?

Is there any documentation we should be referencing in terms of the "right" way 
to set up a redundant connection to AD?


For connection redundancy, I’ve used TCP load balancers. For on premise setups, 
I’ve used DNS round robin to point to two different load balancer instances.



Any feedback is appreciated.

Regards,
Jason Smyth

--
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
go-cd+unsubscr...@googlegroups.com<mailto:go-cd+unsubscr...@googlegroups.com>.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/4b37a890-f442-4966-a053-0fb985f73e3cn%40googlegroups.com<https://groups.google.com/d/msgid/go-cd/4b37a890-f442-4966-a053-0fb985f73e3cn%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to a topic in the Google 
Groups "go-cd" group.
To unsubscribe from this topic, visit 
https://groups.google.com/d/topic/go-cd/eEHCCj-vOuo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to 
go-cd+unsubscr...@googlegroups.com<mailto:go-cd+unsubscr...@googlegroups.com>.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CANiY96ZECHfCOjUw5f-XS6kvsChV%2B8K%3Dry21%3DW%3DeOuFM011opw%40mail.gmail.com<https://groups.google.com/d/msgid/go-cd/CANiY96ZECHfCOjUw5f-XS6kvsChV%2B8K%3Dry21%3DW%3DeOuFM011opw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/DM6PR16MB36713451AFA69A524EC80AB6CFEE2%40DM6PR16MB3671.namprd16.prod.outlook.com.

[go-cd] How to Configure Redundant LDAP Authorization?

[Jason Smyth]

Hi everyone,

We are looking to move from the bundled LDAP authentication plugin to the LDAP 
authorization plugin 
<https://github.com/gocd/gocd-ldap-authorization-plugin>.

For redundancy, our current setup uses 2 LDAP connectors, each pointing to 
a different Active Directory domain controller in the same domain.  If we 
switch to the LDAP authorization plugin we can still create redundant 
authentication links, but does this mean we will need to create duplicate 
role configurations as well?

Is there any documentation we should be referencing in terms of the "right" 
way to set up a redundant connection to AD?

Any feedback is appreciated.

Regards,
Jason Smyth

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/4b37a890-f442-4966-a053-0fb985f73e3cn%40googlegroups.com.

[go-cd] Re: Cloning error

[Jason Smyth]

Hi Granton,

This does not seem to be a GoCD error.

I recommend trying to clone the repository from the agent's command line 
and troubleshooting any errors that Git reports outside of the GoCD 
processes.

Hope this helps,
Jason


On Wednesday 1 May 2024 at 05:02:08 UTC-4 Granton Waribe wrote:

> Hello,
> I have constructed a pipeline and whenever I run it, I get this error 
> message.
> [image: Screenshot from 2024-05-01 11-57-10.png]
> I have tried using shallow clone 
> I have globally increased git buffer size and
> switched to a different network.
> with all that done, I still get the same error.
>
> How do I solve this error?
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/de415aff-337a-4635-ba38-94df7f3e85d0n%40googlegroups.com.

Re: [go-cd] Mount .ssh from EFS to the container (from the image gocd/gocd-server:v22.3.0)

[Jason Smyth]

Hi Satya,

A possible workaround to the limitation is updating the server image and 
adding a symlink that points ~/.ssh/ to wherever you want to actually mount 
the data.

I have never experimented with using a symlink for the .ssh directory, 
though, so this may not work.

Hope this helps,
Jason


On Sunday 28 April 2024 at 12:12:16 UTC-4 Sriram Narayanan wrote:

> On Sat, Apr 27, 2024 at 7:10 PM Satya Elipe  wrote:
>
>> Thank you Sriram.
>>
>> So, ".ssh" folder mounting will be separate from the rest of the data 
>> (/godata, for plugins, pipelines, db etc)...so there would be two separate 
>> mount points into the container ? 
>>
>> I'm using ECS at the moment and not kubernetes, so my task definition 
>> will have two mount points like below:
>>
>> ```
>>
>> "mountPoints": [
>> {
>> "sourceVolume": "efs_id:/godata",
>>
>> "containerPath": "/godata"
>> },
>> {
>> "sourceVolume": "efs_id:/godata/.ssh",
>>
>> "containerPath": "/home/go/.ssh"
>> }
>> ],
>>
>> ```
>>
>> So mounting /godata and efs_id:/godata/.ssh from EFS into the container 
>> at /godata and /home/go/.ssh locations respectively (per above code) seems 
>> to work. 
>>
>> In this case entry_point.sh from the base image is able to map/consider 
>> and execute them properly, hence the server is up and running and 
>> functioning properly.
>>
>> Is that the way it has to be, I think the github repo for gocd server 
>> says that I guess, but perhaps I feel that extra mount point just for .ssh 
>> is overkill and if .ssh can also be entertained by entry_point.sh from one 
>> single mount point /godata in my case, that would be great ?
>>
>> If I do not mount .ssh into /home/go/.ssh separately into the container - 
>> things seem to fail complaining that "key verification failed", I'm not 
>> sure whether I'm still missing something here.
>>
>
> Hey, I had got caught by surprise earlier during the "elastic agent" 
> discussions and had assumed that you must be using EKS. Sorry, my bias had 
> clouded my judgement then. Thankfully Chad and you cleared that up.
>
> ssh by default checks ~/.ssh/ for the keys. Within the GoCD server and 
> agent containers, this home (~) is the /home/go directory, and hence we 
> mount the .ssh folder there. There are use cases where the keys are made 
> available via a different network share and not mixed with configurations 
> that regular GoCD admins would have access to, and hence being able to 
> mount from a separate place to ~/.ssh is helpful. You could always place 
> the .ssh directory along side other directories that would get to godata, 
> while also explicitly specifying a mount to /home/go. At present, GoCD does 
> not have a configuration option to point it to a private key at a path 
> other than ~/ssh
>
> https://docs.gocd.org/current/faq/docker_container_ssh_keys.html
>  
>
>>
>> Many thanks
>> Satya
>>
>> On Thu, Apr 25, 2024 at 3:31 PM Sriram Narayanan  
>> wrote:
>>
>>>
>>>
>>> On Thu, Apr 25, 2024 at 10:16 PM Satya Elipe  wrote:
>>>
 Hi all 

 Wonder, what's the way around to mount .ssh from EFS into the gocd base 
 container (from the image gocd/gocd-server:v22.3.0).


 We have saved all our content into EFS under /godata and maps that into 
 the container as /godata.


 We are using gocd/gocd-server:v22.3.0.


 It all runs good, mapping was fine too but just one thing that’s not 
 happening is “.ssh” folder.


 I have .ssh with all required keys in EFS under /godata and /godata 
 within the container also has .ssh but not /go-working-dir.


 Is that supported, am I mis-configuring it, or do we need to handle 
 that outside of the base image ?

>>>
>>> At a high level, the .ssh folder should be mounted into /home/go. 
>>> e.g. docker run -v /path/to/godata:/godata -v /path/to/home-dir:/home/go 
>>> gocd/gocd-server:v23.5.0
>>> IMPORTANT: You must set the user ID of the files within .ssh to 1000. 
>>> This is the user ID of the gocd process within the container.
>>>
>>> See: 
>>> https://github.com/gocd/docker-gocd-server?tab=readme-ov-file#mounting-volumes
>>>
>>> Given that you are using Kubernetes, please see the Helm chart 
>>> documentation here 
>>> https://github.com/gocd/helm-chart/blob/master/gocd/README.md
>>>
>>> It provides info on just about every configurable attribute for the GoCD 
>>> server and the agent.
>>>
>>> Of particular importance for you are these two attributes:
>>> server.persistence.subpath.homego
>>> agent.persistence.subpath.homego
>>>
>>> Please see that document and jot down your action plan since you will 
>>> need to provide the SSH keys to the server _and_ the agent containers.
>>>
>>> IMPORTANT: You must set the user ID of the files within .ssh to 1000. 
>>> This is the user ID of the gocd process wi

Re: [go-cd] issues loading cruise-config.xml on gocd version 23.4

[Jason Smyth]

Hi there,

The initial error/warning messages talk about 
*gocd-file-based-secrets-plugin.jar* in particular and, based on my 
reading, I believe Sriram is correct that they can be safely ignored.

I recommend focussing on this part of the log:

2023-11-10 17:24:21,478 ERROR [goPluginLocationMonitor] 
GoFileConfigDataSource:466 - Unable to load config file: 
/etc/go/cruise-config.xml Parameter 'TEST_JOB_R
ESOURCEJOB_RESOURCE' is not defined. All pipelines using this parameter 
directly or via a template must define it., Resource name '#{JOB_RESOURCE}' 
is not valid
. Valid names much match '^[-\w\s|.]*$'
com.thoughtworks.go.config.exceptions.GoConfigInvalidException: Parameter 
'TEST_JOB_RESOURCEJOB_RESOURCE' is not defined. All pipelines using this 
parameter dir
ectly or via a template must define it., Resource name '#{JOB_RESOURCE}' is 
not valid. Valid names much match '^[-\w\s|.]*$'
at 
com.thoughtworks.go.config.MagicalGoConfigXmlLoader.validateCruiseConfig(MagicalGoConfigXmlLoader.java:133)
at 
com.thoughtworks.go.config.MagicalGoConfigXmlLoader.preprocessAndValidate(MagicalGoConfigXmlLoader.java:122)
at 
com.thoughtworks.go.config.FullConfigSaveFlow.preprocessAndValidate(FullConfigSaveFlow.java:97)
at 
com.thoughtworks.go.config.FullConfigSaveNormalFlow.execute(FullConfigSaveNormalFlow.java:59)
at 
com.thoughtworks.go.config.GoFileConfigDataSource.forceLoad(GoFileConfigDataSource.java:427)
at 
com.thoughtworks.go.config.GoFileConfigDataSource.load(GoFileConfigDataSource.java:268)

This explicitly states that the configuration file is invalid, and provides 
2 distinct, but possibly related, reasons:

   1. Parameter 'TEST_JOB_RESOURCEJOB_RESOURCE' is not defined.
   - In my experience, this error is thrown under the following conditions:
 - There is a template that includes references to parameters.
 - There is a pipeline that is based on that template that does not 
 define one (or more) of the parameters that the template requires.
  - I suspect the reason this exception is being thrown post-upgrade is 
  explained by the second part of the error.
   2. Resource name '#{JOB_RESOURCE}' is not valid. Valid names much match 
   '^[-\w\s|.]*$'
   - I suspect this error means that somewhere in the configuration is a 
  resource that is defined via a parameter reference.
  - I also suspect that there has been a change in GoCD somewhere 
  between version 21.2 and 23.4 that invalidates this use-case.
  - If both of the above are true, then:
 - The change may have been intentional. If so, it is likely 
 documented in the release notes for the version where it was 
introduced.
 - The change may be an unavoidable side-effect of some other 
 intentional change. If so, whether or not it is documented in the 
release 
 notes probably depends on whether the side-effect was noticed during 
 testing.
 - The change may be an unintentional regression. If so, it is like 
 not documented in the release notes and we should probably have a 
GitHub 
 issue created for it so that it can be tracked and, hopefully, fixed.
  

> But we are not sure what would be the impact if we remove these 3 jars as 
they are bundled with gocd.

Based on their names, it seems reasonable to assume that the impact is as 
follows:

   - *gocd-file-based-secrets-plugin.jar*: Removing this will disable 
   GoCD's file-based secrets plugin. If you are using file-based secrets 
   management in your configuration, it will stop working.
   - *gocd-json-config-plugin.jar*: Removing this will disable GoCD's 
   ability to import pipeline definitions defined as JSON files in a source 
   code repository. If you are using the Configuration Repository feature and 
   one or more of your repos depend on plugin ID json.config.plugin, they will 
   stop working.
   - *gocd-yaml-config-plugin.jar*: Removing this will disable GoCD's 
   ability to import pipeline definitions defined as YAML files in a source 
   code repository. If you are using the Configuration Repository feature and 
   one or more of your repos depend on plugin ID yaml.config.plugin, they will 
   stop working.

The fact that you say the server loads fine if you remove these 3 plugins 
suggests to me that you are using at least one of them and the data that is 
invalidating the cruise-config.xml file is coming from that plugin's 
output. Given the error message at the bottom of the log trace, though, I 
do not believe the issue is with any of these plugins. Rather, the issue 
seems to stem from a change in GoCD itself.

Hope this helps,
Jason Smyth


On Friday, 10 November 2023 at 17:03:29 UTC-5 sjagan...@mdsol.com wrote:

> This error is invalidating cruise-config.xml
>
> On Friday, Novem

[go-cd] Re: upgrade gocd issue

[Jason Smyth]

Hi there,

To me this looks like a quoting issue.

>From the error message, I think the agent is trying to run a program called 
"git pull", rather than trying to run a program called "git" while passing 
"pull" as the first argument.

I don't know why behaviour would have changed from GoCD version 21.2 to 
23.3, but I recommend looking at your job definitions to see if they need 
to be restructured to properly pass the instructions to the agent.

Hope this helps,
Jason Smyth


On Thursday, 9 November 2023 at 22:00:38 UTC-5 sjagan...@mdsol.com wrote:

> Could any one please help with this - I am writing to report a challenge 
> we’ve encountered following our recent upgrade from GOCD version 21.2 to 
> version 23.3. During our testing phase, we observed that certain pipelines, 
> which execute on Windows agents, are failing with the following error when 
> run through "Custom Command"
>  
>  [go] Task: git pull
> Took: 0.179s
> Exited: 1
> '"git pull"' is not recognized as an internal or external command,
> operable program or batch file.
>
>
> This issue seems to be isolated to the Windows agents as the pipelines 
> running on Linux agents are performing without any issues.
> The same pipelines were functioning correctly on the previous version of 
> the GOCD server,
> which suggests a potential configuration or environment variable path 
> discrepancy in the new version.
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/829c4c54-5fc6-4a28-9cee-5d0887202536n%40googlegroups.com.

Re: [go-cd] About multiple go agent

[Jason Smyth]

Hi Vijay,

I don't think your use case is clear here.

Do you want 1 agent container running multiple instances of the agent 
process? Or multiple agents, each running in their own container?

The latter should be fairly straight forward by updating the Docker Compose 
file to spin up multiple agent containers.

The former may require building a custom image that runs multiple agents 
and then updating the Compose file to use that image.

Hope this helps,
Jason


On Friday, 8 September 2023 at 00:51:55 UTC-4 srir...@gmail.com wrote:

> On Fri, 8 Sep 2023 at 12:58 AM, Vijay A  wrote:
>
>> Hi Team,
>>
>> I am new here it is possible to have multiple go agent in same machine 
>> using docker compose file.we have setup go and agent using docker-compose.
>>
>
>
> See: 
>
> https://github.com/gocd/gocd/blob/master/installers/go-agent/release/README-running-multiple-agents.md#
>
> You will need to tweak your compose file accordingly.
>
> — Sriram
>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go-cd+un...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/79df35f1-b288-45ae-bd88-b4a37eb61508n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/812b3130-b4d3-4b69-ae85-1ade60de35c8n%40googlegroups.com.

[go-cd] Re: Gocd dashboard issue

[Jason Smyth]

Hi there,

Have you triggered a new instance of the pipeline since adding the new 
stage?

It's possible that the Dashboard is showing a representation of the 
pipeline as it existed when that instance was created.

Hope this helps,
Jason


On Wednesday, 2 August 2023 at 08:52:35 UTC-4 kowsik...@gmail.com wrote:

> I added a new stage for the existing pipeline via yaml.
> the new stage was not showing in the dashboard. but on the pipeline 
> settings page, I can see the newly added stage.
> I am not sure what the issue is in the dashboard.
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/01af41c2-75f7-4b0a-924e-5ef06acb4714n%40googlegroups.com.

RE: [go-cd] shouldn't required resources also be at the pipeline level?

[Jason Smyth]

. 
(unlike environment variables)

I believe this is in reference to the earlier script-based example you gave 
which is a little confusing.

Anyway, seems you have a way forward here for your core requirement.
On Tue, 25 Jul 2023, 11:39 Joshua Franta, 
mailto:jos...@pracplay.com>> wrote:
Jason, your knowledge here is off. Parameters can be used in scripts, see a 
previous email I this thread that shows how it works.

On Mon, Jul 24, 2023, 4:11 PM Jason Smyth 
mailto:jsm...@taqauto.com>> wrote:
Hi Josh,

I think there may be some confusion here regarding GoCD terminology and common 
concepts.

> i think the main source of confusion is that I thought parameters could only 
> be referred to in scripts!
> I didn't know you could refer to them inside of other configuration 
> properties!

To the best of my knowledge, Parameters (GoCD concept) cannot be referenced in 
scripts. You can call a script that uses parameters (scripting concept), but as 
far as I know, GoCD Parameters are not persisted in the Agent's runtime 
environment unless they are somehow passed in via the Task definition. Are you 
sure you aren't thinking of Environment Variables (GoCD concept)? Environment 
Variables can be defined in a few different places in GoCD. As the name 
suggests, these values are persisted in the Agent's runtime environment when a 
Task is executed.

> I still have a question about how this works in examples using templates.
> If we didn't define the pipeline parameter by default, how would gocd 
> interpret what I'm guessing would be a blank resource?

If a Template references a Parameter then every Pipeline that uses that 
Template _must_ define that Parameter. Depending on how the Parameter is used 
in the Template, leaving the Parameter Value blank may be valid.

In the case of using Parameters to define Resources, my testing shows that each 
Parameter must define a single, valid, Resource. That is, if you want to 
specify multiple Parameterized Resources, you must use multiple Parameters. You 
cannot, for example, provide a Parameter Value of "foo, bar" to make your 
Pipeline's Job depend on the "foo" and "bar" Resources. GoCD rejects the 
configuration as invalid if you try to save it. Similarly, GoCD rejects the 
configuration as invalid if a Parameter is used in the Resource field and you 
try to leave its Value blank.

Regarding your specific use case, you can solve it using either Environments or 
Resources. The right solution depends on your requirements and how you want to 
reason about your environment.

The way I understand it, in the context of this discussion you have 2 groups of 
Agents (Agents1 and Agents2) and 2 groups of Pipelines (PipelinesA and 
PipelinesB). The Pipelines in PipelinesA can run on any Agent, but the 
Pipelines in PipelinesB must run on the Agents in Agents2. We will ignore the 
fact that Pipelines can contain multiple Stages and multiple Jobs and assume 
either that all of the Pipelines contain a single Stage with a single Job, or 
that the scheduling requirements are the same for all Jobs in a given Pipeline. 
You have also talked about Pipeline priority.

Based on this, I assume your requirements are one of the following:

1. Agents should be used to the full extent possible; the workload in 
PipelinesB is heavier so those Pipelines must not run on Agents1, or
2. Pipelines in PipelinesB have a higher priority than those in PipelinesA; 
Agents in Agents2 should take Jobs from PipelinesA only if there are no pending 
Jobs for PipelinesB.

GoCD supports the first scenario. You can achieve this by assigning 2 
Resources/Environments. Pipelines in PipelinesA get 1 Resource/Environment; 
Pipelines in PipelinesB get the other. Agents in Agents1 get the PipelinesA 
Resource/Environment; Agents in Agents2 get both.

GoCD does not support the concept of priority, so scenario 2 is not supported. 
The best you could accomplish would be to map each group of Pipelines to a 
single group of Agents.

Hope this helps. If I'm way off base it might help me better understand your 
situation if you would provide snippets of your actual configuration.

Cheers,
Jason


On Monday, 24 July 2023 at 11:43:58 UTC-4 Chad Wilson wrote:
On Mon, Jul 24, 2023 at 8:44 PM Joshua Franta 
mailto:jos...@pracplay.com>> wrote:

chad thanks for your answer.

i think the main source of confusion is that I thought parameters could only be 
referred to in scripts!
I didn't know you could refer to them inside of other configuration properties!
Is this documented?   Regardless that's super useful, there's probably some 
other things that can be cleaned up knowing that.

https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html#rules-around-usage-of-parameters

I tried this on a pipeline w/out any template and it worked as described.   
Just put the parameter reference in resource

Re: [go-cd] shouldn't required resources also be at the pipeline level?

[Jason Smyth]

Hi Josh,

I think there may be some confusion here regarding GoCD terminology and 
common concepts.

> i think the main source of confusion is that I thought parameters could 
only be referred to in scripts!
> I didn't know you could refer to them inside of other configuration 
properties!

To the best of my knowledge, Parameters (GoCD concept) cannot be referenced 
in scripts. You can call a script that uses parameters (scripting concept), 
but as far as I know, GoCD Parameters are not persisted in the Agent's 
runtime environment unless they are somehow passed in via the Task 
definition. Are you sure you aren't thinking of Environment Variables (GoCD 
concept)? Environment Variables can be defined in a few different places in 
GoCD. As the name suggests, these values are persisted in the Agent's 
runtime environment when a Task is executed.

> I still have a question about how this works in examples using templates.
> If we didn't define the pipeline parameter by default, how would gocd 
interpret what I'm guessing would be a blank resource?

If a Template references a Parameter then every Pipeline that uses that 
Template _must_ define that Parameter. Depending on how the Parameter is 
used in the Template, leaving the Parameter Value blank may be valid.

In the case of using Parameters to define Resources, my testing shows that 
each Parameter must define a single, valid, Resource. That is, if you want 
to specify multiple Parameterized Resources, you must use multiple 
Parameters. You cannot, for example, provide a Parameter Value of "foo, 
bar" to make your Pipeline's Job depend on the "foo" and "bar" Resources. 
GoCD rejects the configuration as invalid if you try to save it. Similarly, 
GoCD rejects the configuration as invalid if a Parameter is used in the 
Resource field and you try to leave its Value blank.

Regarding your specific use case, you can solve it using either 
Environments or Resources. The right solution depends on your requirements 
and how you want to reason about your environment.

The way I understand it, in the context of this discussion you have 2 
groups of Agents (Agents1 and Agents2) and 2 groups of Pipelines 
(PipelinesA and PipelinesB). The Pipelines in PipelinesA can run on any 
Agent, but the Pipelines in PipelinesB must run on the Agents in Agents2. 
We will ignore the fact that Pipelines can contain multiple Stages and 
multiple Jobs and assume either that all of the Pipelines contain a single 
Stage with a single Job, or that the scheduling requirements are the same 
for all Jobs in a given Pipeline. You have also talked about Pipeline 
priority.

Based on this, I assume your requirements are one of the following:

1. Agents should be used to the full extent possible; the workload in 
PipelinesB is heavier so those Pipelines must not run on Agents1, or
2. Pipelines in PipelinesB have a higher priority than those in PipelinesA; 
Agents in Agents2 should take Jobs from PipelinesA only if there are no 
pending Jobs for PipelinesB.

GoCD supports the first scenario. You can achieve this by assigning 2 
Resources/Environments. Pipelines in PipelinesA get 1 Resource/Environment; 
Pipelines in PipelinesB get the other. Agents in Agents1 get the PipelinesA 
Resource/Environment; Agents in Agents2 get both.

GoCD does not support the concept of priority, so scenario 2 is not 
supported. The best you could accomplish would be to map each group of 
Pipelines to a single group of Agents.

Hope this helps. If I'm way off base it might help me better understand 
your situation if you would provide snippets of your actual configuration.

Cheers,
Jason


On Monday, 24 July 2023 at 11:43:58 UTC-4 Chad Wilson wrote:

> On Mon, Jul 24, 2023 at 8:44 PM Joshua Franta  wrote:
>
>>
>> chad thanks for your answer.
>>
>> i think the main source of confusion is that I thought parameters could 
>> only be referred to in scripts!
>> I didn't know you could refer to them inside of other configuration 
>> properties!
>> Is this documented?   Regardless that's super useful, there's probably 
>> some other things that can be cleaned up knowing that.
>>
>
>
> https://docs.gocd.org/current/configuration/admin_use_parameters_in_configuration.html#rules-around-usage-of-parameters
>
>  
>
>> I tried this on a pipeline w/out any template and it worked as 
>> described.   Just put the parameter reference in resource- UI accepts as 
>> long as parameter exists and works.
>>
>> I still have a question about how this works in examples using templates.
>> If we didn't define the pipeline parameter by default, how would gocd 
>> interpret what I'm guessing would be a blank resource?
>>
>> eg we have 
>>
>>1. a pipeline template called FAST_OR_SLOW_PIPE 
>>2. every pipeline implementing this template defines a parameter 
>>called  PIPE_RESOURCE_PARAM
>>
>> What happens if somebody only defines PIPE_RESOURCE_PARAM when the 
>> pipeline is FAST?
>> If it's left as empty for ANY-aka-SLOW resources, wi

[go-cd] Is it Possible to Export a De-Normalized Pipeline Based on Template?

[Jason Smyth]

Hi everyone,

We have a GoCD instance with about 1200 pipelines, most of which are 
defined directly in the GoCD configuration XML file. Of the pipelines that 
are defined in the XML config, most are based on templates.

We are trying to migrate most, if not all, pipelines into (a) configuration 
repo(s), with the thought being to abandon templates altogether. Our 
management of templates has been poor historically, leading to template 
sprawl. We have 439 templates, of which 122 are unused.

Exporting pipelines to JSON or YAML is perfectly straight-forward. However, 
for pipelines that are based on templates, the export retains the template 
reference. Is there a way to export the de-normalized pipeline definition 
so that we do not have to hand-craft the JSON/YAML structure for each 
template?

I came across this GitHub issue . 
I am inclined to agree with the author that the correct default behaviour 
for export is to maintain the template reference; that is the true 
representation of the pipeline's configuration. However, since we are 
looking to move away from using templates, it would be nice if there were 
an easy way for us to export the merge of the pipeline and its associated 
template.

I also came across this mail list conversation 
 but it, 
along with the links it contains, is mostly about support for templates 
defined in the config repos.

Does anyone have any ideas on how we can easily accomplish our goal?

Any thoughts or guidance welcome.

Regards,
Jason

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/e3cd8df8-7463-4069-b495-9a48d6617099n%40googlegroups.com.

[go-cd] Re: Why pipelines defined in configuration repository can not be cloned?

[Jason Smyth]

Hi there,

Maybe I am missing something, but is there some reason you can't copy the 
file that defines the pipeline(s) in your configuration repository?

Hope this helps,
Jason


On Monday, 24 April 2023 at 06:01:02 UTC-4 stygia...@gmail.com wrote:

> I originally wanted to clone multiple pipelines and modify their 
> meterial to different git branch. But then I find out that I can't clone or 
> extract template from pipelines that are defined in configuration 
> repository. I wonder if there is any other way to accomplish this when 
> using the "pipeline as code" feature. All I can think of is to encapsulate 
> the GoCD api.
> By the way, template may not suit me because I can not fetch artifacts 
> from another template
>
> Thanks in advance.
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/31db6aef-cde0-4975-b845-215b561643cdn%40googlegroups.com.

Re: [go-cd] GoCD Artifactory best pratices

[Jason Smyth]

Hi there,

I think the best way to do this in GoCD is using build artifacts, pipeline 
materials and the Fetch Artifact task.

The build pipeline should publish the artifact to Artifactory and generate 
a GoCD build artifact that references the Artifactory location. The QA 
pipeline should have the build pipeline as a material and should fetch the 
GoCD artifact using the Fetch Artifact task with a Pipeline value of 
"build". The Prod pipeline should have the QA pipeline as a material and 
should fetch the GoCD artifact using the Fetch Artifact task with a 
Pipeline value of "build/QA".

This ensures that it is impossible to deploy an artifact to production 
without first successfully deploying it to QA.

Hope this helps,
Jason


On Wednesday, 14 December 2022 at 13:39:56 UTC-5 ma...@thoughtworks.com 
wrote:

> You could upload it from every stage that passes, but I don't see a 
> benefit to doing that -- and as you said it's duplicative. What I'm 
> suggesting is that every downstream stage/pipeline can use the build stage 
> as a material so you can fetch artifacts from it.
>
> Something like:
>
> build-and-test (generates artifact) --> deploy QA --> validate QA --> 
> promote-to-prod-manual-gate --> deploy-prod
>
> Every stage that needs to use the built artifact can include 
> build-and-test as a material so that you can do a Fetch Artifact task. I 
> believe you had a concern/question that it meant stages downstream would 
> have access to an artifact that possibly didn't pass validations? I don't 
> think that's an issue because you would have more than 1 material. there's 
> an implicit dependency on the prior stage succeeding by default. For 
> instance, "deploy-prod" would only run if all the prior stages succeeded, I 
> believe, so it wouldn't actually deploy without going through all of your 
> validation stages (deploy-QA, validate-QA, promote-to-prod-manual-gate).
>
> So in other words, deploy-prod would need to depend on both 
> promote-to-prod-manual-gate and build-and-test.
>
> If you can't get it working, let me dust off my GoCD hat and look at it 
> tonight after working hours.
>
> Best,
>
> Marques
>
> On Wed, Dec 14, 2022 at 10:26 AM Anh Le  wrote:
>
>> 1. So you are suggesting there is no way to avoid temporarily duplicating 
>> the artifact between pipelines following my use case right? I'm still new 
>> to GoCD and haven't play much with plugins
>>
>> Not sure if I get your idea correctly, does that mean if I model my 
>> pipelines like this:
>> build pipeline - deploy to QA pipeline - mark passed QA Gate pipeline - 
>> deploy to Production pipeline
>> If I have a perfect artifact, I will need to Publishing artifacts and *Fetch 
>> Artifacts* for *each pipeline*
>>
>> Since storage is cheap, I'm perfectly fine with this approach. But having 
>> to upload the same file every time could sound not optimal
>>
>> 2. Is it possible to avoid that duplication?  (but it could be 
>> over-engineer, I'm not sure, haven't spent much time doing this after 
>> reading the Continuous Delivery book)
>>
>> I'm thinking is there any vendor/tool/plugins that does artifact 
>> management strategy and could be integrated well with GoCD OOTB: 
>> - An artifact (it is a file generated from the build step anyway) should 
>> have some states:
>> + Deployed to QA
>> + Passed QA Gate
>> Mark passed QA Gate pipeline and Deploy to Production 
>> pipeline should fetch artifacts based on that state from a central 
>> Artifactory
>> On Wednesday, 14 December 2022 at 22:54:46 UTC+7 ma...@thoughtworks.com 
>> wrote:
>>
>>> It’s been a few years since I really looked at GoCD, so my memory is 
>>> hazy on this.
>>>
>>> You should be able to store artifacts on the GoCD server itself by 
>>> configuring a build artifact path. Then downstream stages or pipelines can 
>>> grab them from gocd server directly. There’s an S3 plugin as well if you 
>>> wanted to persist them in AWS. Then you can upload to artifactory when 
>>> you’ve had an artifact pass certain stages for confidence.
>>>
>>> You should be able to pass artifacts through stages so that a downstream 
>>> stage can fetch an artifact that passed the previous stage. If a stage 
>>> depends on the previous stage passing, it cannot run until those conditions 
>>> are met, so they won’t run with artifacts that didn’t succeed in upstream 
>>> stages.
>>>
>>> Once it’s passed production, you can then publish it to artifactory such 
>>> that artifactory only has production quality packages.
>>>
>>> Does that help?
>>>
>>> -Marques
>>>
>>> On Wed, Dec 14, 2022 at 7:33 AM Anh Le  wrote:
>>>
 Hello everyone,

 I'm learning GoCD, I only know how to use the Files and Directories out 
 of the box as Artifactory.
 I have few question about artifacts management best practices or your 
 preferred approach
 Assumption:
  - Already has a build pipeline (to generate artifact)
  - Deploy to QA is manual deploy and depend on build pi

Re: [go-cd] Unable to Manage GoCD Server Process - go-server: unrecognized service

[Jason Smyth]

Thank you for the suggestions, Sriram, Ketan.

I checked both chkconfig and the init.d directory and neither recognized 
go-server either:

$ sudo chkconfig go-server --list
error reading information on service go-server: No such file or directory
$ sudo sh -c 'ls /etc/init.d/go-*'
ls: cannot access /etc/init.d/go-*: No such file or directory

I searched under /etc and found a few files related to GoCD:

$ sudo find /etc/ -name 'go-*'
/etc/default/go-server.pre-migration
/etc/default/go-server~
/etc/default/go-server.rpmnew
/etc/init/go-server.conf

I think the files in /etc/default are legacy files that are no longer used 
by GoCD, but the conf file in /etc/init looks legitimate. A bit more 
research and I confirmed that the go-server service on these servers is 
controlled via initctl (upstart) rather than by the service command:

$ initctl status go-server
go-server start/running, process 19196

Thank you again for the hints that got me moving in the right direction.

Regards,
Jason Smyth

On Sunday, 28 August 2022 at 05:16:11 UTC-4 ketanpad...@gmail.com wrote:

> Try /etc/init.d/go-{server,agent} instead.
>
> On Sun, 28 Aug, 2022, 05:52 Jason Smyth,  wrote:
>
>> Hello everyone,
>>
>> We are running GoCD 19.8.0 on CentOS 6.10 (Final).
>>
>> On both the production and test server we are apparently unable to manage 
>> the GoCD server process. The documentation 
>> <https://docs.gocd.org/current/installation/install/server/linux.html#managing-the-gocd-server-process>
>>  says 
>> that we should use service go-server to interact with the GoCD server, 
>> but this returns an "unrecognized service" error on both machines.
>>
>> GoCD is up and running so presumably the system knows both that it is 
>> supposed to start the service and how to do so; I just can't figure out how 
>> to stop it so that I can do maintenance. Presumably I can reboot the server 
>> to restart the service, but I want to stop the service, not restart it.
>>
>> Any ideas on where we can go from here are greatly appreciated.
>>
>> Regards,
>> Jason Smyth
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go-cd+un...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/ffbe8be2-4b54-4709-8772-5fd0009c6fd6n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/go-cd/ffbe8be2-4b54-4709-8772-5fd0009c6fd6n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/10ebf36a-e039-493a-a90a-1c469256688en%40googlegroups.com.

[go-cd] Unable to Manage GoCD Server Process - go-server: unrecognized service

[Jason Smyth]

Hello everyone,

We are running GoCD 19.8.0 on CentOS 6.10 (Final).

On both the production and test server we are apparently unable to manage 
the GoCD server process. The documentation 
<https://docs.gocd.org/current/installation/install/server/linux.html#managing-the-gocd-server-process>
 says 
that we should use service go-server to interact with the GoCD server, but 
this returns an "unrecognized service" error on both machines.

GoCD is up and running so presumably the system knows both that it is 
supposed to start the service and how to do so; I just can't figure out how 
to stop it so that I can do maintenance. Presumably I can reboot the server 
to restart the service, but I want to stop the service, not restart it.

Any ideas on where we can go from here are greatly appreciated.

Regards,
Jason Smyth

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/ffbe8be2-4b54-4709-8772-5fd0009c6fd6n%40googlegroups.com.

Re: [go-cd] VSM (Dependencies) triggering sequence issues

[Jason Smyth]

Hi there,

My understanding, which Ashwanth's description seems to support, is that 
GoCD's design is such that a Pipeline should be triggered if any of its 
materials have changed, i.e., regardless of the number of dependencies a 
Pipeline has the answer to "has there been a change" is "no" only if none 
of the materials have changed.

The GoCD documentation 
<https://docs.gocd.org/current/introduction/concepts_in_go.html#fan_in_out> 
discusses 
the concepts of fan-out and fan-in. Based on what I read there, I believe 
that fan-in only creates a *logical AND* condition if there is an upstream 
fan-out. This means that, if you want Pipeline C to depend on Pipelines A 
and B and only be triggered if both upstream Pipelines succeed, then 
Pipelines A and B must have a shared upstream dependency (e.g., Pipeline 
Z). Without the upstream fan-out, GoCD views Pipelines A and B as 2 
distinct sources and will trigger an instance of Pipeline C when either of 
its sources changes.

I believe GoCD's design is correct in this regard. The principles of 
Continuous Integration, Deployment and Delivery state that we should test 
every change, no matter how small. These principles are founded on the idea 
of obtaining fast feedback so that we can fix mistakes as quickly as 
possible. With these design principles in mind, I think triggering a 
Pipeline if any of its materials change is the correct behaviour.

>  I am looking for a way to have GoCD, recognize that although an upstream 
pipeline was successful in the past (> 1 week) it should not trigger a 
downstream pipeline if the production pipelines have not been triggered or 
successful.

I reviewed the screenshots you provided and I am going to assume that there 
are 8 Pipelines that are relevant to this conversation: 4 each (Build, 
Deploy, UAT, Prod) for 2 applications (Common, Audit). I have never tried 
to implement a fan-in scenario, but I think the following might work for 
you:

Configure GoCD such that each Audit Pipeline depends on the corresponding 
Common Pipeline. I.E., the Audit-Build Pipeline should depend on the Audit 
source code and the Common-Build Pipeline, the Audit-Deploy Pipeline should 
depend on the Audit-Build Pipeline and the Common-Deploy Pipeline, and so 
on. The theory is that this design creates the necessary upstream fan-outs 
required for GoCD to enforce the downstream *logical AND* on fan-in.

One caveat here is that a successful code change in Common will cause a 
deployment in Audit, regardless of whether or not Audit has been changed. 
This is because each Audit deployment can be thought of as a combination of 
(Audit Code + Common Code) and the CI/CD principles described above say 
that we should deploy (and test) a change to either component inside our 
combined bundle. If this caveat is not acceptable for your use case(s), 
then this design probably won't work for you but may give you a decent 
jumping off point.

Hope this helps,
Jason Smyth


On Thursday, 25 August 2022 at 23:58:23 UTC-4 ashwant...@gmail.com wrote:

> I'll probably share my side of the story from a similar experience, it was 
> so long ago so I don't remember the exact details but we were unable to get 
> pipelines with multiple upstream dependencies work as expected when we 
> wanted *logical AND* type conditions across them. The way we solved them 
> was to move to manual triggers and use the API to trigger the production 
> pipelines. For example, after UAT and Common, we would have a Pre-Prod 
> which would depend on both UAT and Common and use artifacts (version of 
> upstream labels) to check if they've changed since the last time they ran. 
> The idea is, both the versions should change simultaneously for the API 
> trigger to happen. We do this via a simple bash script that exposes a new 
> artifact only if both the labels have changed, else it just returns the 
> same existing artifact as the output which will be consumed in the next run 
> of the pipeline. If both the versions change at the same time, then we 
> would use the /schedule API of the Prod pipeline to trigger it.
>
> I'm not even sure if this is the right way, but this approach made 
> reasoning very easy for us and a better audit on who and how our prod 
> pipeline was triggered.
>
> Thanks,
>
> On Thu, 25 Aug 2022 at 22:55, Sifu Tian  wrote:
>
>> Hi Folks,
>>
>> We have a strange scenario when trying to leverage auto trigger (Trigger 
>> on completion of previous stage)
>>
>> Scenario 1:  We have Production deploy pipeline with 2 upstread 
>> dependences that need to be satisfied to trigger the pipeline (UAT Pipeline 
>> Success and Common Pipeline Success)
>> When UAT is triggered by changes and is successful, the pipeline is 
>> triggered even though Common has not been triggered in p

[go-cd] Re: powershell script hung while running

[Jason Smyth]

Hello Nayan,

Unlike Bash, PowerShell (PS) stores environment variables in a separate 
namespace from "regular" variables. The code sample you provided references 
regular variables, not environment variables.

The easiest way I know of to work with (existing) environment variables in 
PS is to use the "env" PS drive. This drive provides a shortcut for access 
environment variables. Some examples:

# Print the value of the "myVar" variable in the current scope:
Write-Output -InputObject $myVar

# Print the value of the "myVar" environment variable:
Write-Output -InputObject $env:myVar

# List all environment variables:
Get-ChildItem -Path env:

With that said, this isn't really a GoCD question. If you have further 
questions about getting PS to do the things you want it to do, I recommend 
posting your question(s) to a PowerShell-specific forum.

Regards,
Jason Smyth

On Tuesday, 23 August 2022 at 09:08:02 UTC-4 makwanan...@gmail.com wrote:

> I trying like this 
>
> [image: env_val.png][image: powershell.png]
>
> but non of the work for me to print the msg
> On Tuesday, August 23, 2022 at 6:27:18 PM UTC+5:30 
> jsm...@scimarketview.com wrote:
>
>> Hello Nayan,
>>
>> It's tough to say without seeing your code. There are multiple ways to 
>> access variables in PowerShell, depending on what you are trying to achieve.
>>
>> Write-Output (or its alias, echo) takes an object and outputs it to the 
>> PowerShell pipeline. By default, the pipeline's output is sent to the 
>> terminal, so one way to print the contents of an environment variable is to 
>> use the special "env" PowerShell drive:
>>
>> Write-Output -InputObject $env:COMPUTERNAME
>>
>> Hope this helps.
>>
>> Regards,
>> Jason Smyth
>>
>>
>> On Tuesday, 23 August 2022 at 08:20:09 UTC-4 makwanan...@gmail.com wrote:
>>
>>> thank you for replay Jason
>>> i resolve the issue I am truing to run powershell script like 
>>> .\filename.ps1 but its hugs now I am running as
>>> * powershell.exe --executionpolicy remotesigned -File file.ps1*
>>> But now the issue is I unable to access the environment variable in 
>>> power shell script I tried echo and Write-OutPut but non of them print 
>>> value of variable so please help me out fir the same 
>>>
>>> Thanks and regards
>>> Nayan Makwana
>>> On Monday, August 22, 2022 at 8:28:35 PM UTC+5:30 
>>> jsm...@scimarketview.com wrote:
>>>
>>>> Hello Nayan,
>>>>
>>>> Would you please clarify what you mean when you say "it hugs"? All of 
>>>> the following seem likely, and the way to resolve the issue will depend on 
>>>> what the actual situation is:
>>>>
>>>> 1. GoCD fails to schedule the job.
>>>> 2. The job is scheduled as expected but is never assigned to an agent.
>>>> 3. The job is assigned to an agent but the work seems to not get 
>>>> started.
>>>> 4. The agent does some of the work but then hangs without completing.
>>>>
>>>> Each of these scenarios are different, and each requires different 
>>>> troubleshooting steps.
>>>>
>>>> Do any of them apply to you? Or does "it hugs" mean something else in 
>>>> this context?
>>>>
>>>> Regards,
>>>> Jason Smyth
>>>>
>>>>
>>>> On Monday, 22 August 2022 at 08:30:13 UTC-4 makwanan...@gmail.com 
>>>> wrote:
>>>>
>>>>> Dear folks,
>>>>> I just created a PowerShell script that just prints a hello msg but it 
>>>>> hugs when I execute the job 
>>>>>
>>>>> how to rectify the issue I am using gocd 22.2.0 Linux server and 
>>>>> windows 10 gocd agent
>>>>>
>>>>> Thank You
>>>>> Nayan Makwana
>>>>
>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/9e68c255-6d02-46be-8d97-2c6acf20018en%40googlegroups.com.

[go-cd] Re: powershell script hung while running

[Jason Smyth]

Hello Nayan,

It's tough to say without seeing your code. There are multiple ways to 
access variables in PowerShell, depending on what you are trying to achieve.

Write-Output (or its alias, echo) takes an object and outputs it to the 
PowerShell pipeline. By default, the pipeline's output is sent to the 
terminal, so one way to print the contents of an environment variable is to 
use the special "env" PowerShell drive:

Write-Output -InputObject $env:COMPUTERNAME

Hope this helps.

Regards,
Jason Smyth


On Tuesday, 23 August 2022 at 08:20:09 UTC-4 makwanan...@gmail.com wrote:

> thank you for replay Jason
> i resolve the issue I am truing to run powershell script like 
> .\filename.ps1 but its hugs now I am running as
> * powershell.exe --executionpolicy remotesigned -File file.ps1*
> But now the issue is I unable to access the environment variable in power 
> shell script I tried echo and Write-OutPut but non of them print value of 
> variable so please help me out fir the same 
>
> Thanks and regards
> Nayan Makwana
> On Monday, August 22, 2022 at 8:28:35 PM UTC+5:30 jsm...@scimarketview.com 
> wrote:
>
>> Hello Nayan,
>>
>> Would you please clarify what you mean when you say "it hugs"? All of the 
>> following seem likely, and the way to resolve the issue will depend on what 
>> the actual situation is:
>>
>> 1. GoCD fails to schedule the job.
>> 2. The job is scheduled as expected but is never assigned to an agent.
>> 3. The job is assigned to an agent but the work seems to not get started.
>> 4. The agent does some of the work but then hangs without completing.
>>
>> Each of these scenarios are different, and each requires different 
>> troubleshooting steps.
>>
>> Do any of them apply to you? Or does "it hugs" mean something else in 
>> this context?
>>
>> Regards,
>> Jason Smyth
>>
>>
>> On Monday, 22 August 2022 at 08:30:13 UTC-4 makwanan...@gmail.com wrote:
>>
>>> Dear folks,
>>> I just created a PowerShell script that just prints a hello msg but it 
>>> hugs when I execute the job 
>>>
>>> how to rectify the issue I am using gocd 22.2.0 Linux server and windows 
>>> 10 gocd agent
>>>
>>> Thank You
>>> Nayan Makwana
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/7c97a0f7-28b3-428c-aa83-19c39a91853fn%40googlegroups.com.

[go-cd] Re: powershell script hung while running

[Jason Smyth]

Hello Nayan,

Would you please clarify what you mean when you say "it hugs"? All of the 
following seem likely, and the way to resolve the issue will depend on what 
the actual situation is:

1. GoCD fails to schedule the job.
2. The job is scheduled as expected but is never assigned to an agent.
3. The job is assigned to an agent but the work seems to not get started.
4. The agent does some of the work but then hangs without completing.

Each of these scenarios are different, and each requires different 
troubleshooting steps.

Do any of them apply to you? Or does "it hugs" mean something else in this 
context?

Regards,
Jason Smyth


On Monday, 22 August 2022 at 08:30:13 UTC-4 makwanan...@gmail.com wrote:

> Dear folks,
> I just created a PowerShell script that just prints a hello msg but it 
> hugs when I execute the job 
>
> how to rectify the issue I am using gocd 22.2.0 Linux server and windows 
> 10 gocd agent
>
> Thank You
> Nayan Makwana

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/3ee7ab6e-2656-4e2a-8c1d-41d8e782c17bn%40googlegroups.com.

[go-cd] Re: The best SUDO strategy for agents

[Jason Smyth]

Hi Paul,

Assuming you are using elastic agents then, unless the GoCD user already 
has the required privileges in the provided images then you will need to 
either use an init container to modify the image at launch time or build 
your own images.

Are you sure that providing sudo privileges is the way to go, though? Why 
not build custom images (possibly based on the GoCD-provided agent images) 
that already include the required packages instead of installing them at 
run time?

Cheers,
Jason Smyth


On Wednesday, 10 August 2022 at 13:05:25 UTC-4 meltin...@gmail.com wrote:

>
> Hi,
>
> I would like to install some packages on the agent to support my tasks 
> what is the best way to achieve this? The agents are running on a closed 
> K8S installation so I am not worry too much about security.
>
> Can this be done with the agent images available or would it be better to 
> create my own giving the go user sudo privs?
>
> apiVersion: v1
> kind: Pod
> metadata:
>   name: gocd-agent-{{ POD_POSTFIX }}
>   labels:
> app: siab-agent
> spec:
>   serviceAccountName: default
>   containers:
> - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
>   image: gocd/gocd-agent-ubuntu-20.04:v22.1.0
>   securityContext:
> privileged: true
>
> Cheers.
>
> Paul
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/2094d044-a334-4f16-bb91-2ae7bfca7c6cn%40googlegroups.com.

Re: [go-cd] Setting up a test server

[Jason Smyth]

Hi there,

If your GoCD setup only uses static agents then restoring the production 
instance to a test server should be perfectly safe. Since it is the agents 
that do the work and the agents will not be connected to the test GoCD 
instance, there would be no workers available to do anything that might 
interact with your production systems. Once the test system is online you 
should make any necessary changes to the configuration to ensure that if a 
pipeline _does_ run from the test instance then it will not interact with 
your production environment(s). Then you can bring one or more test agents 
online and connect them to the test server.

If you are using elastic agents, things might be more complicated. I have 
not experimented with elastic agents yet, but I believe that the way they 
work is that the GoCD server will call out to whichever orchestrator to 
request an agent be brought online whenever a job needs to run. If that is 
the case, you will need to ensure that the test GoCD server is unable to 
talk to the elastic agent controller that the production server uses.

With that said, I am curious about your use case. What is it that you want 
to test in GoCD that you believe requires using a copy of the production 
GoCD database?

Regards,
Jason Smyth


On Sunday, 31 July 2022 at 12:28:01 UTC-4 timi...@gmail.com wrote:

>
> Thanks for the answer Ashwanth.  I am wondering if there are any settings 
> in the production database that I would have to change in the restored 
> version of the database on the test server, seeing that it will be on a 
> separate server from the production GoCD server.  I wouldn't want any 
> production data touched on the test server, if that's possible?
>
> Thanks in advance!
> On Sunday, July 31, 2022 at 9:16:11 AM UTC-7 ashwant...@gmail.com wrote:
>
>> Maybe using GoCD's backup and restore may be?
>>
>> Ref - 
>> https://docs.gocd.org/current/advanced_usage/one_click_backup.html#restoring-gocd-using-backup
>>
>> Thanks,
>>
>>
>>
>> On Sun, 31 Jul 2022 at 21:27, Tim Stevens  wrote:
>>
>>> Hello, I would like to setup a test server with GoCD installed, but 
>>> using a copy of the production database.  Anybody know if there is a way to 
>>> export the data from the production database and import it into the test 
>>> database?
>>>
>>> Thanks in advance!
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to go-cd+un...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/go-cd/cf1d7e99-3e3f-4487-b669-c50faf522aacn%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/go-cd/cf1d7e99-3e3f-4487-b669-c50faf522aacn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>>
>> -- 
>>
>> Ashwanth Kumar / ashwanthkumar.in
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/7c7968cc-7e1a-4b77-9902-fcbacf1745bfn%40googlegroups.com.

Re: [go-cd] Re: How to lift an app out of it's sandbox into deployment?

[Jason Smyth]

Hi Chris,

Ketan raises a good point: with the current permissions, the go user has no 
access to /home/ubuntu, unless it is already a member of the ubuntu group.

The steps he provided should address the permissions issue, but after 
re-reading the initial problem statement, maybe "How to get an Agent 
process out of its sandbox?" is not the right question to ask. Maybe the 
question should be "How to grant the application running in its container 
permissions to write to the sandbox directory?"

Something like this in the deployment script might work:

cd app
chmod o+w ./path/to/log/dir # make sure container can write to log directory
make start

This could be considered a security risk since it makes the log directory 
writable by any user, but the sandbox itself is protected because 
/var/lib/agent restricts access to the go user and go group (mode 750). The 
container would bypass this limitation by directly mounting the writable 
directory to its local file system.

If this works, it provides the additional benefit that it does not 
introduce any new requirements to the Agent environment. This helps keep 
the process decoupled from the runtime environment.

Hope this helps,
Jason Smyth


On Wednesday, 29 June 2022 at 22:29:46 UTC-4 ketanpad...@gmail.com wrote:

> This will almost certainly not fly. /home/ubuntu will also need permission 
> change. Adding a world writeable permission 777 is a very bad idea. I would 
> recommend that you add the go user to the same group as the ubuntu user, 
> and make sure the deployment folder has write permission for the ubuntu 
> group.
>
> As ubuntu user:
>
> - rm -rf /home/ubuntu/app
> - mkdir /home/ubuntu/app
> - chmod g=u /home/ubuntu/app # group has same permission as user
>
> As root:
> - usermod -a -G ubuntu go # add go user to ubuntu
> - restart your go agent
>
> On Thu, 30 Jun, 2022, 01:56 Chris,  wrote:
>
>> I changed the user and group ownership of /home/ubuntu/app to go, 
>> although 777 should have sufficed for testing purposes.
>>
>> When I su to user go, I am able to view /home/ubuntu/app, and also touch 
>> foo.txt in /home/ubuntu/app . I
>>
>> go@app:/home/ubuntu$ cd app
>> go@app:/home/ubuntu/app$ touch foo.txt
>> go@test-app:/home/ubuntu/app$ ls -la
>> total 8
>> drwxr-xr-x 2 go go 4096 Jun 29 20:10 .
>> drwxr-x--- 7 ubuntu ubuntu 4096 Jun 29 20:09 ..
>> -rw-rw-r-- 1 go go0 Jun 29 20:10 foo.txt
>> go@app:/home/ubuntu/app$ rm foo.txt
>> go@app:/home/ubuntu/app$ exit
>>
>> Go agent is running as go:
>> go   634  0.0  0.2  16452  2464 ?Sl   Jun24   4:09 
>> /usr/share/go-agent/bin/.
>>
>> The go agent is still unable to copy a file into /home/ubuntu/app   , or 
>> even cd into /home/ubuntu/app  , with any level of permission (777, 
>> changing it to go, etc)
>>
>>
>> On Wednesday, June 29, 2022 at 3:04:25 PM UTC-4 jsm...@scimarketview.com 
>> wrote:
>>
>>> Hello Chris,
>>>
>>> This appears to be a permissions issue rather than anything specific to 
>>> GoCD.
>>>
>>> I recommend logging on to the Agent and manually running the script as 
>>> the GoCD user (via sudo or su). From there you should be able to 
>>> troubleshoot the permissions error.
>>>
>>> Once the script runs properly when run locally on the Agent, the 
>>> Pipeline should also run without issue.
>>>
>>> Hope this helps,
>>> Jason Smyth
>>>
>>>
>>> On Wednesday, 29 June 2022 at 12:21:49 UTC-4 Chris wrote:
>>>
>>>> Hello
>>>>
>>>> I face the following issue using gocd, with my gocd agent:
>>>>
>>>> My app is containerized [docker] and needs to log via bind mount. The 
>>>> app cannot log in the sandbox, as it does not have permissions to the log 
>>>> directory.
>>>>
>>>> I would like the app to be copied out from the sandbox, into 
>>>> /home/ubuntu/app , where it would have permissions to write to it's log 
>>>> directory.
>>>>
>>>> Nothing has worked out so far, including trying to force permissions to 
>>>> 777 to the directory, and I always get the following error message:
>>>>
>>>> cp: failed to access '/home/ubuntu/app': Permission denied
>>>> deployment/dev/deploy.sh: line 2: cd: /home/ubuntu/app: Permission 
>>>> denied
>>>>
>>>> Deploy script:
>>>> cp -r app/. /home/ubuntu/app
>>>> cd /home/ubuntu/app
>>>> make start
>>>>
>>>> Th

[go-cd] Re: How to lift an app out of it's sandbox into deployment?

[Jason Smyth]

Hello Chris,

This appears to be a permissions issue rather than anything specific to 
GoCD.

I recommend logging on to the Agent and manually running the script as the 
GoCD user (via sudo or su). From there you should be able to troubleshoot 
the permissions error.

Once the script runs properly when run locally on the Agent, the Pipeline 
should also run without issue.

Hope this helps,
Jason Smyth


On Wednesday, 29 June 2022 at 12:21:49 UTC-4 Chris wrote:

> Hello
>
> I face the following issue using gocd, with my gocd agent:
>
> My app is containerized [docker] and needs to log via bind mount. The app 
> cannot log in the sandbox, as it does not have permissions to the log 
> directory.
>
> I would like the app to be copied out from the sandbox, into 
> /home/ubuntu/app , where it would have permissions to write to it's log 
> directory.
>
> Nothing has worked out so far, including trying to force permissions to 
> 777 to the directory, and I always get the following error message:
>
> cp: failed to access '/home/ubuntu/app': Permission denied
> deployment/dev/deploy.sh: line 2: cd: /home/ubuntu/app: Permission denied
>
> Deploy script:
> cp -r app/. /home/ubuntu/app
> cd /home/ubuntu/app
> make start
>
> Thank you!
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/258d0702-6683-4143-9723-77e85d5e4d2en%40googlegroups.com.

[go-cd] Re: GoCD Agent how to copy artifact and run command on different folder (outside of agent working directory)

[Jason Smyth]

Hi there,

Have you tried using npm's --prefix parameter? I'm not very familiar with 
npm, but an internet search suggests this as an option for invoking npm 
from a different location.

Alternately you could write a script that invokes npm and does whatever 
else you need, check that into source control and pull it into the Agent's 
sandbox at runtime. This has the added advantage of having your deployment 
process be less dependent on GoCD.

Hope this helps,
Jason Smyth


On Friday, 24 June 2022 at 20:11:10 UTC-4 bahun...@gmail.com wrote:

> Sorry, I missing the attachment image in the previous mail.
>
> [image: Screen Shot 2022-06-25 at 06.58.35.png]
>
> On Saturday, June 25, 2022 at 7:00:19 AM UTC+7 Hung Nguyen wrote:
>
>> Hi,
>>
>> The problem is the project working dir is outside of GoCD Agent working 
>> dir, please check the attachment image,
>>
>> Thank you.
>> On Saturday, June 25, 2022 at 1:58:36 AM UTC+7 jsm...@scimarketview.com 
>> wrote:
>>
>>> Hi there,
>>>
>>> It seems to be this should be fairly easy to do by simply following the 
>>> documentation:
>>>
>>> https://docs.gocd.org/current/configuration/admin_add_task.html
>>>
>>> If that is not working for you, please let us know what you have tried 
>>> and what errors you are encountering.
>>>
>>> Cheers,
>>> Jason Smyth
>>>
>>>
>>> On Friday, 24 June 2022 at 00:31:08 UTC-4 bahun...@gmail.com wrote:
>>>
>>>> In my "deploy" pipeline, I have pulled the built artifact from upstream 
>>>> pipeline, but the problem is my working directory is different compared to 
>>>> the GoCD Agent working directory, so I can't deploy that artifact to my 
>>>> working dir. Additionally, I must run some command on my working dir after 
>>>> moving the artifact, e.g. `npm run production`.
>>>>
>>>>- My working dir: `/var/www/vhosts/project`
>>>>- GoCD Agent working dir: `/var/lib/go-agent`
>>>>
>>>> So, how can I copy the artifact to the outside of GoCD Agent and then 
>>>> run some additional commands?
>>>>
>>>> Thanks a lot.
>>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/60889e50-45c9-40b0-aad0-dab6e39009dfn%40googlegroups.com.

[go-cd] Re: GoCD Agent how to copy artifact and run command on different folder (outside of agent working directory)

[Jason Smyth]

Hi there,

It seems to be this should be fairly easy to do by simply following the 
documentation:

https://docs.gocd.org/current/configuration/admin_add_task.html

If that is not working for you, please let us know what you have tried and 
what errors you are encountering.

Cheers,
Jason Smyth


On Friday, 24 June 2022 at 00:31:08 UTC-4 bahun...@gmail.com wrote:

> In my "deploy" pipeline, I have pulled the built artifact from upstream 
> pipeline, but the problem is my working directory is different compared to 
> the GoCD Agent working directory, so I can't deploy that artifact to my 
> working dir. Additionally, I must run some command on my working dir after 
> moving the artifact, e.g. `npm run production`.
>
>- My working dir: `/var/www/vhosts/project`
>- GoCD Agent working dir: `/var/lib/go-agent`
>
> So, how can I copy the artifact to the outside of GoCD Agent and then run 
> some additional commands?
>
> Thanks a lot.
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/bf7f0968-e816-4a11-b53d-c592af04023an%40googlegroups.com.

[go-cd] Re: Go CD Server - add new pipeline

[Jason Smyth]

Hi Tim,

The URL tells GoCD where to find the Material. For example, the URL for a 
Git Material points to the location of the Git repository that the code is 
stored in.

Hope this helps,
Jason


On Tuesday, 26 April 2022 at 07:45:51 UTC-4 timi...@gmail.com wrote:

> Hello, I am new to Go CD and am trying to get through adding a new 
> pipeline.  The below screenshot shows that a Repository URL is required for 
> the type of material being used.  Problem is that I don't know what that 
> URL should be?  Any help would be appreciated!
>
> Tim
> [image: Go CD Server Question.png]
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/8e03c816-92e7-4bf7-8650-0f90ac49a596n%40googlegroups.com.

[go-cd] Re: Cannot match any resource under pipeline(deploying terraform through gocd)

[Jason Smyth]

Hi Shymala,

This error is typically caused by a disconnect between the location to 
which files are created on disk and the location from which GoCD expects to 
upload them as defined in the Artifacts section.

In this case it looks like the Terraform job is creating the output file 
outside of the Agent's working directory ("../../../terraformout.plan"), 
causing GoCD to be unable to find the file to upload.

Hope this helps,
Jason


On Monday, 18 April 2022 at 09:43:36 UTC-4 shyam...@gmail.com wrote:

> I have attached some more details.
>
>
>
> On Monday, April 18, 2022 at 2:23:40 PM UTC+1 Shyamala Ngd wrote:
>
>> Hi ,
>>
>> I am trying to deploy terraform through gocd on AWS.
>>
>> After successfully running terraform plan gocd is not able to upload the 
>> plan.
>>
>> I am getting the below error.
>>
>>
>> &1|12:48:29.593 &1|12:48:29.593 This plan was saved to: 
>> ../../../terraformout.plan &1|12:48:29.593 &1|12:48:29.593 To perform 
>> exactly these actions, run the following command to apply: &1|12:48:29.593 
>> terraform apply "../../../terraformout.plan" &1|12:48:29.594 
>> ?0|12:48:29.689 [go] Task status: passed (9054 ms) j0|12:48:29.699 [go] 
>> Current job status: passed ##|12:48:29.711 [go] Start to create properties 
>> Argos-PII-Production/17/plan-deployment/1/terraform-plan on 
>> ip-10-0-0-170.eu-west-1.compute.internal [/go] ar|12:48:29.711 [go] Start 
>> to upload Argos-PII-Production/17/plan-deployment/1/terraform-plan on 
>> ip-10-0-0-170.eu-west-1.compute.internal [/go] ae|12:48:29.712 [go] The 
>> rule [terraformout.plan] cannot match any resource under 
>> [pipelines/Argos-PII-Production] ae|12:48:29.712 [go] [go] Uploading 
>> finished. Failed to upload [terraformout.plan]. ex|12:48:29.712 [go] Job 
>> completed Argos-PII-Production/17/plan-deployment/1/terraform-plan on 
>> ip-10-0-0-170.eu-west-1.compute.internal [/go]
>>
>> I am new to gocd and not sure where is it going wrong.
>>
>> Can you please help.
>>
>> Thanks,
>> Shymala.
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/21a6097e-2f8a-400c-9e8d-c6d851e270cfn%40googlegroups.com.

[go-cd] Re: Get env variable print out

[Jason Smyth]

Hello,

As you have discovered, the command interpreter in GoCD is not a shell and 
does not perform environment variable substitution.

You can work around this limitation by explicitly invoking a shell (e.g. 
bash -c "echo ${GO_STAGE_COUNTER}") but the preferred method is to save the 
commands you want GoCD to execute into a script that is stored in your 
version control system and have GoCD run that.

Hope this helps,
Jason
 

On Tuesday, 5 April 2022 at 00:44:44 UTC-4 irwa...@gmail.com wrote:

> HI I'm new to GOCD and currently evaluate this product to replace existing 
> CICD pipeline . I try to do simple thing to build docker image and tag the 
> version image to running build count from GoCD but unfortunatelly no 
> success.
>
> Executing simple echo ${GO_STAGE_COUNTER} not resolve to any return value 
> instead return the original string format
>
> [image: gocd.png]
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/b95222ce-6811-41c8-bc8b-1cf31c315ae0n%40googlegroups.com.

Re: [go-cd] best practices for maintaining go pipelines

[Jason Smyth]

Hi Pankaj,

If you are sure that the only difference between the Pipelines is the name 
of the Pipeline and the name of the Git branch, you may want to look at 
Templates.

If each developer has their own Pipeline that is based on the same Template 
then if and when the process (steps taken during execution) need to change 
then you only have to update the Template and all associated Pipelines will 
be updated automatically.

This won't solve Material-related issues but, depending on how often the 
Materials change, it could be a good first step while you look into other 
options.

Hope this helps,
Jason


On Thursday, 10 March 2022 at 23:31:34 UTC-5 ketanpad...@gmail.com wrote:

> The groovy DSL plugin lets you do most of what jsonnet does without the 
> additional compile step. Some examples here 
> 
> .
>
> - Ketan
>
>
>
> On Thu, Mar 10, 2022 at 9:19 PM Michael Day  wrote:
>
>> I've used the JSON plugin and a jsonnet (https://jsonnet.org/) library 
>> to achieve this before.
>>
>> It worked pretty well, but did require some discipline on behalf of the 
>> programmers, i.e. to provide make targets to build, push to the container 
>> repo and deploy to the test cluster, which had uniform names etc.
>>
>> M
>>
>> On Thursday, March 10, 2022 at 2:37:51 PM UTC Chad Wilson wrote:
>>
>>> Hi Pankaj
>>>
>>> How do you maintain the pipeline config now? Via the UI? Via API? Via 
>>> one of the pipelines-as-code/confg repo plugins?
>>>
>>> In any case, I would say it depends a bit on
>>>
>>>- what the content of your pipelines are
>>>- how much variation there is between these pipelines
>>>- when you say developers "maintain" their pipelines, does that 
>>>include changing its config and definition? Or just making sure it's 
>>> green?
>>>- when folks need to change the pipelines, will it be acceptable (or 
>>>desirable) for them to be forced to do it via source control?
>>>
>>> ...but in terms of general approaches to reduce duplication and factor 
>>> out commonality in pipelines, there are a few approaches, which have 
>>> upsides and downsides. Perhaps you can take a look at them and ask some 
>>> more specific questions based on your needs?
>>>
>>>
>>>- *Pipeline templates* 
>>>https://docs.gocd.org/current/configuration/pipeline_templates.html 
>>>- maintained via UI|API IIRC, but not really customisable by those 
>>>pipelines that use the template, except by parameters/env/materials
>>>- One of the* Pipelines as Code* plugins (aka Config Repository 
>>>plugins) 
>>>https://docs.gocd.org/current/advanced_usage/pipelines_as_code.html
>>>   - At least with Pipelines-as-Code you can find-and-replace 
>>>   en-masse, but...
>>>   - With the YAML plugin you can use YAML anchors to factor out 
>>>   commonality in a limited way.
>>>   - With either the YAML or JSON plugins you could conceptually 
>>>   template the YAML/JSON and generate it using a tool as well, although 
>>> I 
>>>   have never done this and have no opinion on whether it's a good idea
>>>   - With the Groovy plugin you can write Groovy script to express 
>>>   your pipelines using a DSL, which could potentially allow you to 
>>> model the 
>>>   differences between them, dependencies etc.
>>>   - This may or may not work for depending on how you plan to 
>>>   supporting "self-service" and developers making changes to their own 
>>>   pipelines, unless you work with some kind of PR approval model into a 
>>> Git 
>>>   repository you centrally maintain
>>>- Write some tooling against the *GoCD API *
>>>https://api.gocd.org/current/#pipeline-config
>>>   - There were some client libraries to support this such as gomatic 
>>>   , but I haven't 
>>>   personally used those, and not sure what state they are in.
>>>   
>>> -Chad
>>>
>>>
>>>
>>> On Thu, Mar 10, 2022 at 8:12 PM 'pan...@mammoth.io' via go-cd <
>>> go...@googlegroups.com> wrote:
>>>
 Hi community,

 Every developer in our organization gets a pipeline of his own to do 
 the full build and test process. Very frequently we are making changes 
 that 
 have to be replicated across all pipelines, for example adding a new 
 material, adding an additional step, minor changes in the command syntax 
 of 
 one of the steps in pipeline. Since each developer is supposed to maintain 
 their own pipeline, they realize it only when the pipeline breaks because 
 of this change and we are in a continuous maintenance loop.

 Our need is to easily replicate these changes across all pipelines. 
 Most configuration is same except the name of the pipeline is different. 
 Whats the community recommendation for this. How else to manage this 
 situation?

 Warm regards,
 Pankaj

>

[go-cd] Re: Fan-In from different SCM material with deployment gate

[Jason Smyth]

Hi Joe,

I think you are on the right track with your manual trigger idea. It should 
be pretty straight-forward to put together a simple script/program that 
checks every minute or 2 to see:

1. Is there a new version of BUILD-A, -B or -C?
2. Is there currently an instance of BUILD-A, -B or -C running?

Then, if the answer to the first question is "yes" and the second question 
is "no", kick off the PACKAGE Pipeline.

Another option might be to design the PACKAGE process in such a way that it 
checks to see whether there is an instance of any of the BUILD Pipelines 
running and, if so, it fails the PACKAGE Pipeline. But depending on how 
often that situation comes up it might cause frustration in the user base 
if there are a lot of "failures" that aren't really failures.

Hope this helps somewhat.

Regards,
Jason


On Thursday, 16 December 2021 at 11:34:59 UTC-5 Joe H wrote:

> My application has several libraries each with their own git repository. 
> The repos provide material to individual pipelines that build the 
> libraries. 
>
> The build pipelines then feed to a revision packager pipeline that bundles 
> the libraries into a revision package which then feeds into a deployment 
> pipeline.
>
> REPO-A -> BUILD-A
> REPO-B -> BUILD-B
> REPO-C -> BUILD-C
>
> BUILD-A, BUILD-B, BUILD-C -> PACKAGE -> DEPLOY
>
> Let's say BUILD-A and BUILD-B take 20 seconds to complete, but BUILD-C 
> pipeline takes a few minutes to complete.
>
> What I'm seeing is that when commits come in on multiple repos somewhat 
> simultaneously, this results in multiple runs of the PACKAGE -> DEPLOY 
> pipelines.  I'm trying to set up a gate that waits for all running builds 
> to complete before continuing with PACKAGE.
>
> I understand why that's the default behavior if you view each path through 
> the graph as a unique run. The documentation also indicates that Fan-In 
> only works if the SCM material for the entry pipelines is the exact same. 
> So it seems I'm observing the documented behavior.
>
> Is there a way to achieve what I'm trying to do where I can Fan-In from 
> different SCM material?  I'm thinking like a way to override the "Fan-In 
> Key" where it could use an arbitrary key to drive the Fan-In behavior 
> instead of the repo URL.
>
> If not, is there an obvious way I'm not seeing to break up my graph such 
> that I can wait for all BUILD-* to complete with some kind of quiet period 
> gate before continuing with PACKAGE -> DEPLOY? For example, once any of the 
> BUILD-* pipelines have run, wait 45 seconds and if nothing else starts, 
> then run PACKAGE -> DEPLOY.
>
> Maybe the PACKAGE pipeline has to be manual trigger and ignores the 
> BUILD-* pipelines for scheduling, then the "gate" is really just a script 
> or something that uses the API to determine whether or not it's time to run?
>
> Looking for any ideas I'm not seeing...
> Thanks
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/85b2f312-e4fb-44b8-b921-68b4c39930d4n%40googlegroups.com.

[go-cd] Re: Need help with Gauge framework

[Jason Smyth]

Hello,

Would using an Environment Variable work?

You could set the variable either in GoCD at the Environment or Pipeline 
level, or on the Agent itself.

Hope this helps,
Jason


On Tuesday, 2 November 2021 at 07:27:53 UTC-4 s.saw...@gmail.com wrote:

> Is there a way to use CSV based on condition?
>
> My scenario is as below:
> i m using "Table" tab for csv
> have same csv in 2 location but csv path need to take based on environment
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/6d749a47-dbf3-46c6-9a5b-435135c95565n%40googlegroups.com.

[go-cd] Re: run bash as root in wsl

[Jason Smyth]

Hi Andy,

I have never tried running Agents inside WSL (or WSL2) but my guess is that 
the script will run as whatever account the Go Agent service runs under.

So if you want the script to run as root there are a few options:

1. Run the Go Agent service as root,
2. Grant the "go" user permission to run the script as root via sudo, 
either with password or without, or
3. Set the script owner to "root" and set the sticky bit so that whenever 
_anybody_ runs the script it runs as root. (This is probably only viable if 
the script is a permanent fixture inside the WSL installation. If the 
script is pulled from elsewhere at runtime then it will be owned by the 
"go" user and this just moves the problem around, forcing you to find a way 
to run chown and chmod as root instead of the script itself.)

Hope this helps,
Jason


On Thursday, 16 September 2021 at 07:03:53 UTC-4 Andy Bergh wrote:

>
> Hi,
>
> I installed gocd on wsl (not wsl2) on windows server 2019.
>
> When a job in a stage runs (test.sh for example), the bash file is run as 
> user 'go'.
> Is there a way for me to run the bash file as user root?
>
> I have been trying to find ways around it but run in so many other 
> problems. Like ssh keys, access to servers (from within the bash script) 
> etc.
>
> This will be much easier if I can somehow make gocd run the bash file as 
> root in wsl.
>
> Is this possible?
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/ae4863bd-ae81-4052-b325-7c34f05bab70n%40googlegroups.com.

[go-cd] Re: Exploring GoCD for bitbucket pipeline replacement

[Jason Smyth]

Hi V,

Here are some thoughts on the questions you raised:

1. Configure Pipeline without Git Material.
I believe GoCD requires that Pipelines have at least one Material. For the 
Pipeline model you describe, why not put the "pure shell script", as you 
describe it, into source control and then key the Pipeline off of that? 
Alternately, you could put the shell script into source control and use a 
GoCD Pipeline to test and then deploy the scripts to somewhere else to have 
them run outside GoCD. Regardless of the path chosen, I believe the scripts 
you alluded to should be under source control somewhere and, assuming they 
are, I don't see a need to create a Pipeline in GoCD without a Material.


2. Specify Git branch at Pipeline runtime.
The closest thing I can think of to this model would be to use Pipeline 
Parameters in GoCD. You can reference a Pipeline's Parameters in a Material 
definition so it should be possible to use a Parameter to specify the 
desired branch of a Git repository.

With that said, if I understand the situation you describe correctly, I do 
not recommend this approach. I think you are describing a pattern where you 
have multiple long-lived branches in Git and each one maps to a 
corresponding long-lived environment. So there is a "prod" branch that gets 
deployed to the "prod" environment; the "uat" branch gets deployed to the 
"uat" environment, and so on.

If this is accurate then using Parameters should be safe in that using a 
single Parameter to control both source control branch and destination 
environment would make it impossible to accidentally deploy the wrong code 
to the wrong environment. However, using a single Pipeline to manage all of 
your different environments would likely make it difficult to understand 
exactly what has been deployed where at any given point in time.

Instead, I recommend a templating pattern where you have a parameterized 
process defined somewhere (preferably in source control) and then reference 
that process in multiple Pipelines, with each Pipeline linking a particular 
source branch to its respective environment.


3a. Pass data from Pipeline 1 to Pipeline 2.
GoCD's Artifacts system works well for this. Configure Pipeline 1 so that 
it writes the key to a file that gets uploaded to GoCD Artifact repository. 
Create Pipeline 2 with Pipeline 1 as (one of) its Material(s). Configure 
Pipeline 2 to Fetch the saved Artifact from Pipeline 1, read the key from 
the file and does whatever work is required from there.

3b. Persist Pipeline data.
This happens automatically with GoCD's Artifacts system. Every time 
Pipeline 2 is run, it will run with a specific version of Pipeline 1 (and 
therefore a specific copy of the key) as its source. By default, this 
should be the most recent instance of Pipeline 1, but there is always the 
option of triggering with an older instance if that becomes necessary.


Hope you find some of this useful.

Regards,
Jason

On Monday, 23 August 2021 at 04:38:09 UTC-4 vig...@prolificatech.com wrote:

> Hi. I’m not sure if this is the right forum for it, if not please direct 
> me to right forum.
>
> We are exploring GoCD for our CI/CD pipeline. Currently we are using 
> Bitbucket Pipelines.
>
> I tried locally and have few questions regarding how-to,
>
> 1. Can we configure a pipeline without git dependency / material? We have 
> a pipeline which runs on a schedule and the pipeline is just pure shell 
> script (and it does not depend on any code from any repo).
>
> 2. How to configure multi-branch pipeline? For example, we have an 
> frontend repo with different branch for different environment, like dev, 
> prod, … so that when triggering the pipeline manually, we can select which 
> branch to trigger from.
>
> 3. Here is a complex scenarios which we trying to resolve. We have 2 repo, 
> 1 for frontend and 2 for backend. The backend pipeline will be dependent on 
> frontend pipeline. Whenever the frontend pipeline build is successful, it 
> should trigger backend pipeline. Also if there is any change in backend 
> repo, it should trigger backend pipeline. I’m able to configure this by 
> adding multiple dependencies for backend pipeline. The complexity here is, 
> we generated random key when building frontend (For eg.: KEY=hash-BUILD-1), 
> and the same needs to be passed to backend pipeline. 
>
> a. How to trigger backend pipeline with custom build parameter (KEY)? From 
> pipeline 1 it should pass parameter and value for pipeline 2 build. 
>
> b. How to persist param value (KEY) in backend pipeline, so that 
> subsequent build from backend repo (commits, poll scm) trigger will use the 
> param value (KEY) which is already set. Param value changes only when 
> frontend pipeline creates a new build.
>
> Thanks, V
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@google

[go-cd] Re: How to continue to next stage automatically , even if previous stage is failed

[Jason Smyth]

Hello Suman,

To the best of my knowledge there is no way to do this natively in GoCD. 
You could probably fake it by building something that continually checks 
for failed Stages and triggers the next Stage when it finds one but I 
question why someone would want to do this.

The way I tend to think about Stages in GoCD is that each one represents a 
self-contained process that proves that the release candidate passing 
through it is suitable for production in some way. Once a release candidate 
passes through all of the Stages in the Pipelines that drive it, we know it 
is good to release to production. If the candidate fails at any point, 
though, it will never be released to production so there is not much value 
in continuing with any Stages after the one that failed.

Could you elaborate a bit on your model and why you would want to run the 
next Stage even if the current Stage fails?

Regards,
Jason

On Monday, 2 August 2021 at 05:57:43 UTC-4 sumangs...@gmail.com wrote:

> Hi All,
>
> I am new to the GOCD , Can any one help me  on the below query,  
>
> Q) In a pipeline ,How can we  continue to next stage automatically  , when 
> previous stage is failed.
>
> Thanks
> Suman
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/251fff83-b21c-4c15-84dc-669dd7cb5a9cn%40googlegroups.com.

Re: [go-cd] Use Container Registry as Material?

[Jason Smyth]

Hi Chad,

Thank you for the feedback.

You raise a lot of good points.

I will need to think about this some more and come up with a pattern that 
fits our specific use cases.

Regards,
Jason


On Friday, 23 July 2021 at 01:39:08 UTC-4 Chad Wilson wrote:

> Hi Jason
>
> Personally, I'm not aware of there being a way to do this within the 
> material model of GoCD.
>
> From a design perspective I think there would be some challenges to a 
> general purpose approach here, since to my knowledge container registries 
> don't typically have any way to link together multiple versions of a 
> container to one another in a clear way compared to a VCS. 
>
>- A tag is just a tag; the same registry can have multiple different 
>variants of a container, but whose lineages are separate (e.g alpine and 
>debian variants of the same image).
>- Tags while useful and human-readable don't really imply ordering of 
>versions except by, say semver convention - and may or may not be immutable
>- Monitoring a mutable tag for changes in the underlying image hash 
>wouldn't necessarily allow the reproducibility that the GoCD material 
>subsystem aims for except by capturing and referring to the digest of the 
>image which might not be very friendly for usage in an environment since I 
>believe that you can't get the tag metadata along with that.
>
> Perhaps one would want something like OpenShift's image streams concept, 
> which IIRC is rather opinionated?
>
> The way we approached this problem (on ECR/AWS) was to have the pipelines 
> that produce or push docker images outside GoCD to also produce a small 
> manifest with the registry name and target tag etc, similar to 
> https://github.com/gocd/docker-registry-artifact-plugin (the plugin which 
> you could use if you are producing images inside GoCD and thus can use 
> pipeline dependency materials). The manifest was committed and pushed to a 
> Git Repo which could be monitored by a regular Git SCM material or a Git 
> Path Material (to avoid many-to-many repo explosion); with some simple 
> scripting to parse the manifest and decide what to do with that new image 
> (e.g override an image.tag in a Helm chart's values). By using a Git repo & 
> SCM material, we could pass along context from the originating system about 
> the changes included between one Docker image build and the next, and 
> provide more context for investigating issues and comparing VSMs, but this 
> need was really due to doing some custom Docker image builds outside of 
> GoCD rather than within.
>
> Another option, which I never evaluated but contemplated, would possibly 
> be to subscribe to the ECR repository's CloudWatch 'push' events and run a 
> lamba which invokes the GoCD API 
> <https://api.gocd.org/current/#scheduling-pipelines> to schedule the 
> relevant pipeline; passing the relevant image tag etc as an environment 
> variable. I felt this would be less manageable, externalized even more 
> "glue" outside GoCD, thus more opaque and difficult to decide whether a 
> trigger was necessary/desirable inside such lambdas, but was an option all 
> the same. 
>
> -Chad
>
> On Thu, Jul 15, 2021 at 3:48 AM Jason Smyth  
> wrote:
>
>> Hi everyone,
>>
>> Is it possible to configure GoCD to use a container registry (we are 
>> using ECR on AWS in particular) as a Material?
>>
>> The thought process here is that I would like to possibly trigger a 
>> Pipeline whenever a new version of a particular image is published.
>>
>> Thank you in advance for the feedback.
>>
>> Regards,
>> Jason Smyth
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go-cd+un...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/08105e21-3ffa-4167-acb2-2d39c3530961n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/go-cd/08105e21-3ffa-4167-acb2-2d39c3530961n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/dbd13395-f99f-4212-8134-3674b66cb39cn%40googlegroups.com.

[go-cd] Use Container Registry as Material?

[Jason Smyth]

Hi everyone,

Is it possible to configure GoCD to use a container registry (we are using 
ECR on AWS in particular) as a Material?

The thought process here is that I would like to possibly trigger a 
Pipeline whenever a new version of a particular image is published.

Thank you in advance for the feedback.

Regards,
Jason Smyth

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/08105e21-3ffa-4167-acb2-2d39c3530961n%40googlegroups.com.

Re: [go-cd] Re: Pipeline failing in Go Server

[Jason Smyth]

Hi Sushma,

I think we may need the actual error message if we are going to be able to 
help.

The situation you describe suggests that there is something specific to the 
configuration of these 2 Agents that is causing an issue but without the 
specific error it will be very difficult to determine the cause.

Regards,
Jason


On Monday, 21 June 2021 at 12:35:19 UTC-4 gangaia...@gmail.com wrote:

> Hi Jason,
>
> 1. Error on go-server is regarding ,not able to archive the 
> application.(but in agent manually and through command line  I can build 
> the app)
>
> 2.yes, we are trying to build android and iOS app, one of the agent is not 
> able to build android and the other iOS.
>
> 3.Yes,All the pipelines on this agents are getting failed with same error.
>
> Regards,
> Sushma
>
>
> On Mon, Jun 21, 2021, 9:56 PM Jason Smyth  
> wrote:
>
>> Hello Sushma,
>>
>> What errors are you seeing on the Agents? Is the error the same every 
>> time? Are both Agents failing with the same error or are the errors 
>> different? Are all Pipelines failing on these 2 Agents or just some of them?
>>
>> We are not likely to be able to help without more details around the 
>> specifics of your scenario and the issues that you are facing.
>>
>> Regards,
>> Jason
>>
>> On Monday, 21 June 2021 at 06:32:14 UTC-4 gangaia...@gmail.com wrote:
>>
>>> All,
>>>
>>> Hope you are doing fine.
>>>
>>> This is very strange scenario,
>>> We have around 8 agents with same configuration. We use pipeline as a 
>>> code to run pipeline from Go Server and all of our builds were running 
>>> successfully. But since few days only in 2 agents builds are failing 
>>> whereas to debug when we run the same command in agent using command line 
>>> or tools it is successful.
>>> Also when the same job is redirected to another resource(agent) its 
>>> being success. 
>>> Worried what I may be missing? and what may be the cause, which is 
>>> causing only failure from pipeline and not in local(agent where pipeline 
>>> builds job)
>>> Go version -19.9.0.
>>>
>>> Please advice.
>>>
>>> Regards,
>>> Sushma
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go-cd+un...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/bc4b57fe-8ce9-4eca-bd9a-9942d35e63f9n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/go-cd/bc4b57fe-8ce9-4eca-bd9a-9942d35e63f9n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/9e1bdfc0-a08b-43f6-bc57-6d67e49e7663n%40googlegroups.com.

[go-cd] Re: Pipeline failing in Go Server

[Jason Smyth]

Hello Sushma,

What errors are you seeing on the Agents? Is the error the same every time? 
Are both Agents failing with the same error or are the errors different? 
Are all Pipelines failing on these 2 Agents or just some of them?

We are not likely to be able to help without more details around the 
specifics of your scenario and the issues that you are facing.

Regards,
Jason

On Monday, 21 June 2021 at 06:32:14 UTC-4 gangaia...@gmail.com wrote:

> All,
>
> Hope you are doing fine.
>
> This is very strange scenario,
> We have around 8 agents with same configuration. We use pipeline as a code 
> to run pipeline from Go Server and all of our builds were running 
> successfully. But since few days only in 2 agents builds are failing 
> whereas to debug when we run the same command in agent using command line 
> or tools it is successful.
> Also when the same job is redirected to another resource(agent) its being 
> success. 
> Worried what I may be missing? and what may be the cause, which is causing 
> only failure from pipeline and not in local(agent where pipeline builds job)
> Go version -19.9.0.
>
> Please advice.
>
> Regards,
> Sushma
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/bc4b57fe-8ce9-4eca-bd9a-9942d35e63f9n%40googlegroups.com.

[go-cd] Re: trouble in db migration as part of GOCD version upgrade running in EKS

[Jason Smyth]

Hello Gupta,

Could you elaborate a bit on the specifics of your environment, what you 
are trying to accomplish and the challenges you are facing?

Regards,
Jason


On Saturday, 12 June 2021 at 07:21:23 UTC-4 Gupta wrote:

> Anyone can guide me with steps how to migrate db of gocd running in eks as 
> part of version upgrade, i don't find document useful for kubernetes.
> I stil want to keep it running with H2 db after migration .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/289a2c87-b9ee-4a15-9f1f-1909bf43a77fn%40googlegroups.com.

[go-cd] Re: trouble with getting-started instructions

[Jason Smyth]

Hello,

Did you check the permissions on the wrapper file referenced in the error 
message? Is it possible that GoCD was configured to run as an unprivileged 
service account that does not have permission to read the file?

Regards,
Jason

On Saturday, 12 June 2021 at 07:14:51 UTC-4 peter@gmail.com wrote:

>
>
> I'm struggling to "get started" with GoCD and was wondering if I missed 
> something.
>
>
> I tried the latest 21.2.0 on a Linux host, both the standalone `.zip` and 
> `apt` packages - both spat out the error message `/etc/init.d/go-server: 1: 
> eval: /usr/share/go-server/bin/../wrapper/wrapper-linux-x86-64: not found`.
>
>
> > The file is there - looks like a binary that was built with GCC. Not sure 
> > what's going on; seems too dumb to be a problem that only I've had.
>
>
> (Upgrading past Java11 knackered my debian-based host somehow, so, while it 
> reformats) I unpacked and ran the 20.5.0 server on my Windows PC fine, but, 
> when trying to launch the agent;
>
> - Invoking; `bin\go-agent.bat console -serverUrl=http://localhost:8153/go` 
> 
>
> - Produces; `Was passed main parameter 'console' but no main parameter was 
> defined in your arg class`
>
> (I posted this on gitter yesterday without realising that was a 
> "developers" chat, not "users" - sorry)
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/a236caeb-4076-4526-99d2-530fee5ff42cn%40googlegroups.com.

[go-cd] Re: GoCD for Azure Deployment using Terraform

[Jason Smyth]

Hello,

GoCD can interact with anything that has a command line interface, so 
deploying Azure resources into Azure using Terraform templates should be 
possible.

I would start with learning Terraform and how to manage a single 
Terraformed deployment from multiple client machines. I do not know a lot 
about Terraform, but I do know that this will involve, at least, choosing 
and implementing a remote location for storing Terraform state information. 
You will also need to decide how you will manage getting the Terraform 
secrets into the GoCD Agents.

Once you have a solid understanding of how Terraform works, I would think 
it should be fairly trivial to write some wrapper scripts for GoCD to 
leverage.

Hope this helps,
Jason Smyth


On Friday, 4 June 2021 at 08:54:46 UTC-4 phe...@gmail.com wrote:

> Hello all,
>
> Can anybody point me to "starter" documentation on how to deploy Azure 
> resources into Azure using GoCD and Terraform templates.
>
> Is this even possible with GoCD at the moment?
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/6bef3e33-7ddf-4877-9b48-575a9df50733n%40googlegroups.com.

Re: [go-cd] Java LTS Support in GoCD

[Jason Smyth]

Thank you Ashwanth, Ketan and Aravind.

Your replies and the resources you linked helped make things a lot clearer 
for me.

It seems GoCD's stance is "We expect the majority of our users to use the 
bundled version of Java. Because multiple versions of Java can co-exist on 
a system with each application running under the desired version, this 
should not be an issue." I think this makes sense and is a reasonable 
stance to take.

I am not clear how this affects the automatic update process for Static 
Agents, though. Will the automatic update also update the JRE or will it 
only update the GoCD Agent JARs?

Thank you again,
Jason

On Wednesday, 2 June 2021 at 10:54:31 UTC-4 Aravind SV wrote:

> Hello,
>
> There’s also a similar conversation which happened over at: 
> https://github.com/gocd/gocd/issues/8736
>
> Is it just too much work to test such an “old” version?
>
> I think, partially, yes.
>
> Sorry if this is not the right place to raise this but it didn’t seem 
> appropriate to raise a GitHub Issue at this point. I would love to hear 
> other people’s thoughts on this.
>
> No, this is the right place to bring it up.
>
> Regards,
> Aravind
>
> * From*: "'Ashwanth Kumar' via go-cd
> * Subject*: Re: [go-cd] Java LTS Support in GoCD
> * To*: go...@googlegroups.com
> * Date*: Wed, 2 Jun 2021 19:57:52 +0530
> Taking the notes from GoCD stance (
> https://www.gocd.org/2019/05/21/official-stance-on-java/) and looking at 
> IntelliJ blog post (
> https://blog.jetbrains.com/idea/2018/09/using-java-11-in-production-important-things-to-know/),
>  
> and OpenJDK releases page (https://jdk.java.net/archive/) and assuming my 
> understanding is correct, 
>
> It seems like you get LTS support only for paid version of Oracle JDK not 
> for the OpenJDK variants. 
>
> On Wed, Jun 2, 2021, 7:39 PM Jason Smyth  wrote:
>
>> Hi all,
>>
>> Based on what I read in this recent thread 
>> <https://groups.google.com/g/go-cd/c/GeH97RQAcQg>, it seems GoCD only 
>> supports Java v >= (latest-2), regardless of Long-Term Support status of 
>> any releases.
>>
>> This would seem to indicate that even though Java 11 is the most recent 
>> LTS version and the next LTS version will not be until Java 17, GoCD has 
>> already dropped support for Java 11 because Java 14 (not LTS) and 15 (not 
>> LTS) have already been released, so (latest-2) is now 13.
>>
>> This will effectively force all GoCD users to either go long periods of 
>> time without updating GoCD (i.e. do not run any versions of GoCD that 
>> support neither v11 nor v17) or switch to non-LTS versions of Java. Since 
>> non-LTS versions of Java are only supported for 6 months until the next 
>> version is released, this then forces all GoCD users to major Java version 
>> updates every 6 months if they want to run only supported versions.
>>
>> Am I missing something here? Why does GoCD not support the latest LTS 
>> version of Java? Why support Java 13 (support ended more than a year ago in 
>> March 2020) but not 11 (supported until at least 2024)? Is it just too much 
>> work to test such an "old" version?
>>
>> Sorry if this is not the right place to raise this but it didn't seem 
>> appropriate to raise a GitHub Issue at this point. I would love to hear 
>> other people's thoughts on this.
>>
>> Regards,
>> Jason Smyth
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go-cd+un...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/f64f8a9e-587a-498b-9c90-3af4ae23dc46n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/go-cd/f64f8a9e-587a-498b-9c90-3af4ae23dc46n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to go-cd+un...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/go-cd/CAD9m7CyS62bp0%2BtKKEL_mJrMy07kgM4LMics%3Df0zmV9DCTMUCQ%40mail.gmail.com
>  
> <https://groups.google.com/d/msgid/go-cd/CAD9m7CyS62bp0%2BtKKEL_mJrMy07kgM4LMics%3Df0zmV9DCTMUCQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/15d70f91-7062-4c36-a0ff-3d3383c65ca0n%40googlegroups.com.

[go-cd] Java LTS Support in GoCD

[Jason Smyth]

Hi all,

Based on what I read in this recent thread 
<https://groups.google.com/g/go-cd/c/GeH97RQAcQg>, it seems GoCD only 
supports Java v >= (latest-2), regardless of Long-Term Support status of 
any releases.

This would seem to indicate that even though Java 11 is the most recent LTS 
version and the next LTS version will not be until Java 17, GoCD has 
already dropped support for Java 11 because Java 14 (not LTS) and 15 (not 
LTS) have already been released, so (latest-2) is now 13.

This will effectively force all GoCD users to either go long periods of 
time without updating GoCD (i.e. do not run any versions of GoCD that 
support neither v11 nor v17) or switch to non-LTS versions of Java. Since 
non-LTS versions of Java are only supported for 6 months until the next 
version is released, this then forces all GoCD users to major Java version 
updates every 6 months if they want to run only supported versions.

Am I missing something here? Why does GoCD not support the latest LTS 
version of Java? Why support Java 13 (support ended more than a year ago in 
March 2020) but not 11 (supported until at least 2024)? Is it just too much 
work to test such an "old" version?

Sorry if this is not the right place to raise this but it didn't seem 
appropriate to raise a GitHub Issue at this point. I would love to hear 
other people's thoughts on this.

Regards,
Jason Smyth

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/f64f8a9e-587a-498b-9c90-3af4ae23dc46n%40googlegroups.com.

[go-cd] Re: Upgrading from 20.3.0 to 20.5.0

[Jason Smyth]

Hi Jo,

It should be possible to upgrade from 20.3.0 to 20.5.0 directly without 
stepping through 20.4.0 but I recommend thoroughly reading the release 
notes for each version to make sure you understand what has changed at each 
stage.

I know there were significant changes in some versions last year but I am 
not familiar enough with the details to know whether any of them could 
potentially affect the upgrade path.

Hope this helps,
Jason


On Tuesday, 16 March 2021 at 13:39:20 UTC-4 Jo G wrote:

> I found this instruction 
> https://docs.gocd.org/20.5.0/installation/upgrade_to_gocd_20.5.0.html.  
> Do I need to upgrade to 20.4.0 then do the DB migration or can I do the DB 
> migration on my existing 20.3.0 then upgrade to 20.5.0?  The doc makes it 
> sound like it's possible but I'm just looking for guidance.  Please advise.
>
> Thanks,
> Jo
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/ec4feb4b-aaea-442e-8d63-64cb3c9df616n%40googlegroups.com.

Re: [go-cd] how to save report at task level executed on sequence

[Jason Smyth]

Hello Sunaina,

I think the easiest way to do this is probably to configure all of the 
Tasks to output their reports to a specific location relative to the 
Pipeline's working directory (e.g., ./Reports).

You can then configure the Job to publish everything under ./Reports when 
the Job is over and you will get the reports from all Tasks at once.

Is there some reason you need to have the reports published as each Task 
completes that would prevent this solution from working for you?

Regards,
Jason


On Monday, 8 March 2021 at 04:48:32 UTC-5 s.saw...@gmail.com wrote:

> Do i need to add set of code in task execution script only as save report 
> in artifact will be done at job level, not task level.
>
> On Mon, Mar 8, 2021 at 2:31 PM Sushma Gangaiah  
> wrote:
>
>> Hi Sawadhiya,
>>
>> You can save the reports as an artifact in GoCD master for every task and 
>> fetch later.
>>
>> Regards,
>> Sushma
>>
>> On Mon, Mar 8, 2021, 2:22 PM s.saw...@gmail.com  
>> wrote:
>>
>>> Is there any way to save all tasks report in goCD
>>> I have multiple sequential tasks executing in one Job and i want to save 
>>> all the reports
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to go-cd+un...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/go-cd/5c55e85d-8413-4ebd-a804-40be33ee819bn%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go-cd+un...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/CAHDZtpQAe0SKoCTO-wRkPYW_%3DHfyQgAxyq3erA%3DG1yN2KabcpA%40mail.gmail.com
>>  
>> 
>> .
>>
>
>
> -- 
> Sunaina Awadhiya
> Bangalore
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/fca85bcb-2fa8-4492-8630-81944159a21dn%40googlegroups.com.

Re: [go-cd] Remove pipeline directory on agent after completion

[Jason Smyth]

Hi Joel,

Like Ashwanth, we also use a custom script to clean up Pipeline working 
directories. We run ours as a Windows Scheduled Task and leave the 5 most 
recent working directories. This creates an opportunity for race 
conditions, though, if the Agent happens to pick up a Job for a Pipeline 
while the cleanup script is in the middle of deleting that Pipeline's 
directory.

We haven't run into any issues yet but, if we do, we may have to look at 
adopting Ashwanth's idea of running cleanup in a dedicated Pipeline.

Cheers,
Jason


On Friday, 5 March 2021 at 21:20:48 UTC-5 ashwant...@gmail.com wrote:

> AFAIK there isn't any feature so far in GoCD. We currently use custom 
> script to delete all the other pipelines on an Agent by running a cleanup 
> pipeline periodically. Since each agent is exclusive to a pipeline run, you 
> can safely delete all the other pipelines in the working directory via the 
> cleanup pipeline without breaking any builds / deployment.
>
> Thanks,
>  
>
> On Fri, Mar 5, 2021 at 11:13 PM Joel Bodenmann  wrote:
>
>> Is there (by now) a feature available that allows to configure either on 
>> a per-agent or a per-pipeline level the behavior that a pipeline directory 
>> is automatically removed from the agent after finishting the pipeline job 
>> (in all cases, not just on success)?
>>
>> I believe that this older thread is asking for the same: 
>> https://groups.google.com/g/go-cd/c/fhlYvVcg5H4
>>
>>
>> Best regards,
>> ~ Joel
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go-cd+un...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/a1f351e5-c717-4153-be6e-7a1cfd77aa2dn%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
>
> Ashwanth Kumar / ashwanthkumar.in
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/03707baf-dbc6-4e5c-8315-c735343b2a13n%40googlegroups.com.

[go-cd] Re: Using GoCD API withing a Pipeline

[Jason Smyth]

Hi Labib,

1. I do not think so. I'm also not sure why you would want this. There are 
probably some APIs that Agents interact with that end users do not but what 
you're trying to accomplish is making the Agent call the end user APIs. Why 
would you want/need a second URL for this?

2. I do not believe GoCD passes the user context on to the Agent when a Job 
is started. The user's information is stored as a property on the Pipeline 
instance so that we know who triggered it but I believe that is the extent 
of it. The only way I can think of to pass user context to the Agent would 
be to use Properties or Environment Variables and have the user "Trigger 
with options" from the UI and manually set some values.

Hope this helps,
Jason


On Wednesday, 24 February 2021 at 18:05:07 UTC-5 la...@rubyplaynetwork.com 
wrote:

> Hey everyone,
>
> I am trying to create a pipeline that when triggered, it triggers certain 
> stages in a set of pipelines. This is not something I can do using 
> dependencies so I am using the GoCD API to do so. I have that all working 
> and fine but a couple of questions:
>
> 1- Is there an internal URL for the API that can be accessed by the agents 
> rather than using the public URL?
>
> 2- Since this is an internal request coming from a pipeline triggered 
> manually by a user, is there a way to pass that authentication to the API?
>
> Thanks,
> Labib
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/d14ad5ea-c3e4-4401-8289-69a7b08c6217n%40googlegroups.com.

Re: [go-cd] Mark Ready for Release - Go Pipes

[Jason Smyth]

Hi Nitesh,

You can get documentation for the APIs for your version of GoCD (17.4.0) 
here: https://api.gocd.org/17.4.0/#introduction.

With that said, unless you have some really compelling reasons for staying 
on 17.4, you should probably consider upgrading to a more current version.

Regards,
Jason

On Monday, 18 January 2021 at 11:08:04 UTC-5 nitesh...@gmail.com wrote:

> Hi All,
>
> Do we have any GO API which help us to get the latest version of the 
> Pipeline ?
>
> Regards, Nitesh 
>
> On Mon, Jan 18, 2021 at 7:01 PM kritik...@thoughtworks.com <
> kritik...@thoughtworks.com> wrote:
>
>> Hi Nitesh,
>>
>> > Thanks , but i think this is only available in latest version , i am 
>> using 17.4.0 , do we have available on this version ?
>>
>> No this API was introduced in `v18.11.0` .
>>
>> Regards
>> Kritika
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go-cd+un...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/5d8d7238-d47b-4be5-b2f1-de28af849dd3n%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Thanks&Regards
>
> Nitesh kumar
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/f53b8bcb-0cbe-4c3d-ade0-2a3198352d4cn%40googlegroups.com.

[go-cd] Re: Filesystem full /var/go

[Jason Smyth]

Hi Kalpesh,

Unless I am missing something, this does not seem to be a GoCD question at 
this point.

There is a server that has the role of a GoCD Agent. There is a directory 
on that server outside of the GoCD Agent directory that is full. The full 
directory appears to be the home directory of the user that runs the GoCD 
Agent but the only way to know that for sure is to review the server 
configuration. GoCD itself will not put anything outside of the working 
directory unless explicitly or implicitly directed to do so by the specific 
process being implemented. (For example, the process as implemented could 
require downloading npm packages to the GoCD account's home directory.)

If this server's only role is as a static GoCD Agent, I recommend 
decommissioning it and building a replacement. From GoCD's perspective, 
everything on the Agent should be disposable once a Pipeline run is 
complete so you lose nothing of value by rebuilding.

Alternately, take a backup of the entire /var/go directory and then delete 
whatever you like. In theory this is fine but I can't comfortably recommend 
it because I don't know nearly enough about your environment to be able to 
say whether or not deleting any given thing from the file system is safe.

Good luck,
Jason


On Friday, 18 December 2020 at 09:37:25 UTC-5 kalp...@gmail.com wrote:

> Hi Jason,
>
> Unfortunately we did some investigate and couldn't figure out which files 
> we need to get rid of. 
> There were lot of npm files, Cypress one's etc..,
>
> Regards,
> Kalpesh Malviya
>
> On Wednesday, 16 December 2020 at 21:08:29 UTC+5:30 
> jsm...@scimarketview.com wrote:
>
>> Hi  Kalpesh,
>>
>> From the output provided it looks like /var/go is some user's home 
>> directory.
>>
>> I recommend checking to see if the user "go" is still using this 
>> directory as its home directory. If it is, I recommend further analysis on 
>> the contents of the directory to determine what can be safely deleted.
>>
>> Hope this helps,
>> Jason
>>
>>
>> On Wednesday, 16 December 2020 at 01:52:21 UTC-5 kalp...@gmail.com wrote:
>>
>>> Hi Team,
>>>
>>> We have installed go-agent on the linux box.
>>>
>>> After successfully installation we changed the working dir by editing 
>>> the file
>>> `/usr/share/go-agent/wrapper-config/wrapper-properties.conf` to add the 
>>> following line  : 
>>>
>>> *`wrapper.working.dir=/data/go-agent` *Now after few months we have 
>>> recieved a mail from the linux team that there is space full on path* 
>>> /var/go*.  I'm not sure whether the files present under the dir 
>>> */var/go* is been used anymore as the current working dir is 
>>> */data/go-agent*. 
>>>
>>> Can you please check and confirm whether the contents of dir */var/go* 
>>> can be deleted
>>>
>>> I am attaching the contents of */var/go* below here.
>>> [ro...@y98641.x.xxx PROD:go]# pwd
>>> /var/go
>>> [ro...@y98641.xx.xxx PROD:go]# ls -latr
>>> total 20
>>> -rw-r--r-- 1  go  go  231  Aug 21  2019   .bashrc
>>> -rw-r--r-- 1  go  go  193  Aug 21  2019   
>>> .bash_profile
>>> -rw-r--r-- 1  go  go18  Aug 21  2019   
>>> .bash_logout
>>> drwxr-   3  go  go19  Jul  8 13:51   .pki
>>> drwxr-xr-x.  22  root   root 4096  Jul 1904:54   ..
>>> drwx--   3   go go25  Aug 12   12:47   .config
>>> drwxr-xr-x 2   go go  6  Nov  414:53   .node-gyp
>>> drwxr-xr-x 4   go go37  Nov  510:03   .cache
>>> drwxr-xr-x 7   go go  111  Dec  815:25   .npm
>>> drwxr-xr-x 4   go go31  Dec 10   11:48   blackduck
>>> drwxr-xr-x 2   go go4096  Dec 12   09:27   
>>> .oracle_jre_usage
>>> drwxr-xr-x 2   go go50  Dec 1410:26   
>>> .standard-cache
>>> drwx--  11  go go  211  Dec 1516:32   .
>>> drwx--2  go go60  Dec 15 16:32   .gnupg
>>>
>>>
>>>
>>> Below are the contents of dir */data/go-agent*  here :
>>> [ro...@y98641.xxx.xxx PROD:go-agent]# pwd
>>> /data/go-agent
>>> [r...@y98641.xxx.xxx PROD:go-agent]# ls -latr
>>> total 61660
>>> -rw-r--r-- 1go go 9660548 Jul 6  14:52 
>>> agent-launcher.jar
>>> -rw-r--r-- 1go go   26133287 Jul 6  14:52 
>>> agent.jar
>>> -rw-r--r-- 1go go   15290228 Jul 6  14:52 
>>> tfs-impl.jar
>>> drwxr-xr-x 2go go 35 Jul 6  14:52   
>>>   config
>>> drwxrwxrwx   13root  root 239 Dec 10  11:48 ..
>>> -rw-r--r-- 1go go12031656 Dec 10  14:03 
>>> agent-plugins.zip
>>> drwxrwxrwx 8 root root 248 Dec 10   14:03 .
>>> drwxr-xr-x 4 gogo 63 Dec 1014:03
>>> plugins
>>> drwxr-xr-x 4 gogo 88 Dec 1014:03
>>> plugin

[go-cd] Re: Empty agent list after Windows installation

[Jason Smyth]

Hi John,

If there are no rows on the Agents tab, that suggests that the Agent is not 
able to register itself with the server.

I recommend reviewing the Agent logs to identify the issue.

Regards,
Jason


On Wednesday, 16 December 2020 at 14:50:15 UTC-5 john.laur...@gmail.com 
wrote:

> I'm new to GoCD. I followed the installation guide for Server and Agent. 
> Yet, when I go to the Agents tab, there are no rows -- and no visible way 
> of creating one. I've tried the uninstall/re-install route with no success. 
> Any ideas about how to create or install an agent?
>
> John
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/f97db9ab-35b0-4066-a53b-cb1145cf29ben%40googlegroups.com.

[go-cd] Re: Filesystem full /var/go

[Jason Smyth]

Hi  Kalpesh,

>From the output provided it looks like /var/go is some user's home 
directory.

I recommend checking to see if the user "go" is still using this directory 
as its home directory. If it is, I recommend further analysis on the 
contents of the directory to determine what can be safely deleted.

Hope this helps,
Jason


On Wednesday, 16 December 2020 at 01:52:21 UTC-5 kalp...@gmail.com wrote:

> Hi Team,
>
> We have installed go-agent on the linux box.
>
> After successfully installation we changed the working dir by editing the 
> file
> `/usr/share/go-agent/wrapper-config/wrapper-properties.conf` to add the 
> following line  : 
>
> *`wrapper.working.dir=/data/go-agent` *Now after few months we have 
> recieved a mail from the linux team that there is space full on path* 
> /var/go*.  I'm not sure whether the files present under the dir */var/go* 
> is been used anymore as the current working dir is */data/go-agent*. 
>
> Can you please check and confirm whether the contents of dir */var/go* 
> can be deleted
>
> I am attaching the contents of */var/go* below here.
> [ro...@y98641.x.xxx PROD:go]# pwd
> /var/go
> [ro...@y98641.xx.xxx PROD:go]# ls -latr
> total 20
> -rw-r--r-- 1  go  go  231  Aug 21  2019   .bashrc
> -rw-r--r-- 1  go  go  193  Aug 21  2019   .bash_profile
> -rw-r--r-- 1  go  go18  Aug 21  2019   .bash_logout
> drwxr-   3  go  go19  Jul  8 13:51   .pki
> drwxr-xr-x.  22  root   root 4096  Jul 1904:54   ..
> drwx--   3   go go25  Aug 12   12:47   .config
> drwxr-xr-x 2   go go  6  Nov  414:53   .node-gyp
> drwxr-xr-x 4   go go37  Nov  510:03   .cache
> drwxr-xr-x 7   go go  111  Dec  815:25   .npm
> drwxr-xr-x 4   go go31  Dec 10   11:48   blackduck
> drwxr-xr-x 2   go go4096  Dec 12   09:27   
> .oracle_jre_usage
> drwxr-xr-x 2   go go50  Dec 1410:26   
> .standard-cache
> drwx--  11  go go  211  Dec 1516:32   .
> drwx--2  go go60  Dec 15 16:32   .gnupg
>
>
>
> Below are the contents of dir */data/go-agent*  here :
> [ro...@y98641.xxx.xxx PROD:go-agent]# pwd
> /data/go-agent
> [r...@y98641.xxx.xxx PROD:go-agent]# ls -latr
> total 61660
> -rw-r--r-- 1go go 9660548 Jul 6  14:52 
> agent-launcher.jar
> -rw-r--r-- 1go go   26133287 Jul 6  14:52 
> agent.jar
> -rw-r--r-- 1go go   15290228 Jul 6  14:52 
> tfs-impl.jar
> drwxr-xr-x 2go go 35 Jul 6  14:52 
> config
> drwxrwxrwx   13root  root 239 Dec 10  11:48 ..
> -rw-r--r-- 1go go12031656 Dec 10  14:03 
> agent-plugins.zip
> drwxrwxrwx 8 root root 248 Dec 10   14:03 .
> drwxr-xr-x 4 gogo 63 Dec 1014:03
> plugins
> drwxr-xr-x 4 gogo 88 Dec 1014:03
> plugins_work
> drwxr-xr-x 5 gogo 51 Dec 1014:03
> felix-cache
> drwxr-xr-x   10 gogo 4096 Dec 1014:18   data
> drwxr-xr-x 138 gogo 8192 Dec 1518:49   
> pipelines
> -rw-r--r--  1gogo0 Dec 16 
> 07:26  .tmp.file.list
> -rw-r--r--  1gogo0 Dec 16 
> 07:28  .agent-bootstrapper.running
>
>
>
> Regards,
> Kalpesh Malviya
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/cf622706-c4a9-4b65-9a81-6d0f5a93e3f7n%40googlegroups.com.

Re: [go-cd] Elastic agent registration problem from Windows docker swarm plugin

[Jason Smyth]

Hi there!

Glad to hear you got this sorted out.

Thank you for sharing your solution with the mailing list.

Regards,
Jason


On Tuesday, 8 December 2020 at 02:50:40 UTC-5 IXI wrote:

> Finally we got solution for this. You want beleave it, but it was just 
> about encoding of the file. We had UTF-8, then we changed to windows 
> default (to be honest, still not sure, which is windows default), and it 
> worked. So everything was setup correctly and only the encoding of the 
> agent-autoregister.properties was wrong. 
> Thank you all for all suggestions and help. :-)
>
> Dne středa 4. listopadu 2020 v 16:30:34 UTC+1 uživatel 
> jsm...@scimarketview.com napsal:
>
>> Hello,
>>
>> The host OS is less relevant than the container OS in this situation. 
>> What OS is your custom container image based on? This information will 
>> likely be necessary to "tunnel" into the container from the host.
>>
>> For troubleshooting, I recommend using docker's CLI to access one of your 
>> containers after GoCD spins it up. Since the containers are registering 
>> themselves as Static Agents, you should be able to see the Sandbox 
>> directory listed in the GoCD UI. Based on the information that Ram provided 
>> and what I see in our own Agents, you want to look for 
>> /config/autoregister.properties.
>>
>> My guess is that Ram is correct and this file is either missing or does 
>> not contain the expected information.
>>
>> Hope this helps,
>> Jason
>>
>> On Wednesday, 4 November 2020 at 09:25:46 UTC-5 IXI wrote:
>>
>>> Yes we do, what you mentioned. Path: 
>>> C:\Go-Agent\config\autoregister.properties  (please remember that we are 
>>> doing it on Windows Server 2016 machine)
>>>
>>> And just repeating question from my previous answer:  Is there a way to 
>>> debug our situation? It seems like a windows container problem.  
>>>
>>> Dne středa 4. listopadu 2020 v 13:18:33 UTC+1 uživatel srir...@gmail.com 
>>> napsal:
>>>
 On Wed, Nov 4, 2020 at 8:12 PM IXI  wrote:

> Yes we do. Is there a way to debug our situation? It seems like a 
> windows container problem.
>

 If your containers are coming up and getting registered as static 
 agents, then the agent startup is likely unable to locate 
 autoregister.properties.

 At which path are you installing the go-agent?
 You need to create a config sub-directory, and then place the 
 autoregister.properties file there (see 
 https://docs.gocd.org/current/advanced_usage/agent_auto_register.html)

 "On the remote GoCD Agent machine, create a file named 
 /config/autoregister.properties. In a native 
 agent installation this file is usually located on 
 /var/lib/go-agent/config/ directory. The config/ folder might not exist, 
 in 
 this case you should create it by yourself."

 -- Ram


  

>
> Dne středa 4. listopadu 2020 v 12:21:00 UTC+1 uživatel Aravind SV 
> napsal:
>
>> Hello,
>>
>> I see. You’re creating your own image.
>>
>> The docker image provided by GoCD (on DockerHub) does this 
>> 
>>  
>> to populate the autoregister.properties file correctly, based on 
>> environment variables passed in by the plugin when it wants to start an 
>> elastic agent. Do you do that in your image too? The values passed in 
>> should be used. That’s how the server / plugin knows that the agent it 
>> was 
>> waiting for has come up.
>>
>> Cheers,
>> Aravind
>>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to go-cd+un...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/go-cd/b1afe618-8db0-4023-9f12-f6578ddd9c88n%40googlegroups.com
>  
> 
> .
>


-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/b9589bfe-0681-4180-a771-bab13a4cb934n%40googlegroups.com.

[go-cd] Re: Error querying database. Cause: java.sql.SQLSyntaxErrorException: This version of MySQL doesn't yet support 'LIMIT & IN/ALL/ANY/SOME subquery'

[Jason Smyth]

Hello,

I could be wrong but this looks to me like the sort of thing that should be 
raised as an issue on GitHub:

https://github.com/gocd/gocd/issues

Regards,
Jason


On Wednesday, 2 December 2020 at 08:44:19 UTC-5 gdlai cn wrote:

>
> ### Error querying database. Cause: java.sql.SQLSyntaxErrorException: This 
> version of MySQL doesn't yet support 'LIMIT & IN/ALL/ANY/SOME subquery' ### 
> The error may exist in com/thoughtworks/go/server/dao/maps/Stage.xml ### 
> The error may involve defaultParameterMap ### The error occurred while 
> setting parameters ### SQL: SELECT stages.name as stageName, 
> stages.counter as stageCounter, stages.pipelineId as pipelineId, 
> pipelines.name as pipelineName, pipelines.counter as pipelineCounter, 
> pipelines.label as pipelineLabel, stages.approvedBy, stages.cancelledBy, 
> stages.createdTime, stages.orderId, stages.result as stageResult, 
> stages.approvalType, stages.completedByTransitionId as 
> completedByTransitionId, stages.state as stageState, stages.latestRun, 
> stages.fetchMaterials, stages.cleanWorkingDir, stages.rerunOfCounter, 
> stages.artifactsDeleted, stages.configVersion, stages.lastTransitionedTime, 
> builds.*, builds.id as buildId, buildstatetransitions.currentState, 
> buildstatetransitions.statechangetime, buildstatetransitions.id as 
> stateId FROM stages INNER JOIN pipelines ON pipelines.id = 
> stages.pipelineId AND pipelines.name = ? INNER JOIN builds ON stages.id = 
> builds.stageId AND builds.ignored != true LEFT JOIN buildstatetransitions 
> ON builds.id = buildstatetransitions.buildId WHERE stages.id in ( SELECT 
> stages.id FROM stages INNER JOIN pipelines ON stages.pipelineId = 
> pipelines.id AND stages.name = ? AND pipelines.name = ? AND result = 
> 'Passed' ORDER BY id LIMIT ? OFFSET ?) AND stages.result = 'Passed' ORDER 
> BY stages.id, builds.name, buildstatetransitions.statechangetime ### 
> Cause: java.sql.SQLSyntaxErrorException: This version of MySQL doesn't yet 
> support 'LIMIT & IN/ALL/ANY/SOME subquery' ; bad SQL grammar []; nested 
> exception is java.sql.SQLSyntaxErrorException: This version of MySQL 
> doesn't yet support 'LIMIT & IN/ALL/ANY/SOME subquery'
>
>
> MySql:8.0.22
> gocd: 20.10.0
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/dac79cea-294c-4b27-b7c7-52fb52998c4dn%40googlegroups.com.

Re: [go-cd] Cannot run agent in MSYS2

[Jason Smyth]

Hi Joel,

There is no way that I know of to specify a particular interpreter for GoCD 
Tasks but it isn't something I have ever looked into so it might be 
possible.

You are correct that each Task needs to explicitly invoke the child shell 
of choice. We use PowerShell rather than MSYS2 bash, but I suspect many of 
the concepts are the same.

The way we avoid having to do that for every command is with PowerShell 
scripts that we store in source control. Instead of having each command 
explicitly called from its own GoCD Task, a single GoCD Task launches a 
PowerShell script that does any number of things. This is nice because it 
eliminates a lot of otherwise redundant configuration in GoCD (how many 
times do I really want to type "powershell.exe -NonInteractive" into 
GoCD?). It also creates a reasonable boundary between our build/deploy 
processes (the scripts) and the orchestration tool (GoCD). If at some point 
we should decide that we want to look at other CD solutions, we already 
have all of the code necessary to execute our processes in a portable 
format.

If scripting isn't a viable option, I suggest comparing the filesystems 
between the working "Test Drive GoCD" environment and the broken agent-only 
environment. Maybe something will jump out as an obvious solution to the 
apparently missing wrapper problem.

Hope this helps,
Jason

On Friday, 27 November 2020 at 11:51:47 UTC-5 tec...@gmail.com wrote:

> Hello Jason,
>
> Unfortunately I made no progress on this. Furthermore, this is the only 
> thing I have to solve/get working before I can start transitioning to GoCD.
> The issue is exactly that the custom task command gets passed to `cmd.exe` 
> on Windows.
>
> Before continuing going down the rabbit hole: Is there a way within GoCD 
> to specify the terminal/shell to use for executing tasks? Something such as 
> `SHELL_BIN`?
>
> I am able to issue commands on cmd.exe which get executed in the MSYS2 
> bash. However, I have to do this for every command (eventually GoCD task). 
> You mentioned the possibility to launch the MSYS2 shell from cmd.exe (easy) 
> as a child process. Once the shell is up, how would I instruct GoCD to use 
> that child process shell?
>
>
> Best regards,
> ~ Joel
>
>
> On Wednesday, November 25, 2020 at 8:21:54 AM UTC+1 gangaia...@gmail.com 
> wrote:
>
>> Hi All,
>>
>> Very oftenly pipeline cannot be triggered and material update in master 
>> will get hunger for long.
>>
>> How can I resolve this?
>>
>> Regards,
>> Sushma
>>
>> On Tue, Nov 24, 2020, 10:15 PM Joel Bodenmann  wrote:
>>
>>> I've been testing GoCD today. One of my main requirements is that the 
>>> Windows agent(s) can perform builds within MSYS2 (unix environment).
>>>
>>> To get started I've opened an MSYS2 / MinGW64 terminal and pasted the 
>>> "Test Drive GoCD" line in there. Everything worked out of the box. The 
>>> script properly setup both the server and the agent and everything was 
>>> running within MSYS2. I was able to perform builds as per my requirements.
>>>
>>> The next step was to only get the agent running as the server itself is 
>>> running on a separate FreeBSD machine. However, when attempting to run the 
>>> agent within MSYS2 (MinGW64 shell) I'm presented with this message:
>>>
>>> $ ./bin/go-agent console
>>> Unable to locate any of the following binaries:
>>>   
>>> /home/joel/go-agent-20.10.0/bin/../wrapper/wrapper-mingw64_nt-10.0-19042-x86-64
>>>   
>>> /home/joel/go-agent-20.10.0/bin/../wrapper/wrapper-mingw64_nt-10.0-19042-x86-32
>>>   /home/joel/go-agent-20.10.0/bin/../wrapper/wrapper
>>>
>>> To me it seems that the java server wrapper is not finding the binaries 
>>> for my particular environment. However, the "Test Drive GoCD" 
>>> script/binaries ran without any issues.
>>>
>>> How can I solve this problem? Ultimately I just need to run the go-agent 
>>> within MSYS2.
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to go-cd+un...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/go-cd/027ae335-8701-4d90-afe9-09558a961c57n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/da3c8e3b-403c-4388-aab9-dd67dff3deb7n%40googlegroups.com.

Re: [go-cd] Pipeline not waiting for previous test pipeline to complete

[Jason Smyth]

Hi Joel,

I think the missing piece may be that *#{UPSTREAM_BUILD_PIPELINE} *does not 
need to be a direct parent of the Pipeline where the Fetch Artifact Task 
resides.

I recommend the following changes to Pipeline C:

   1. Change the value of *#{UPSTREAM_BUILD_PIPELINE}* from *Pipeline A* to 
*Pipeline 
   A/Pipeline B*.
   2. Remove the *Pipeline A* Material.

I modified your original XML document to show what I mean.

Hope this helps,
Jason

On Friday, 27 November 2020 at 10:43:15 UTC-5 tec...@gmail.com wrote:

> Thank you for your reply, Ketan.
>
> I've come across the fan-in & fan-out capabilities - which is one of the 
> main reasons I decided to give GoCD a try and hopefully switch to it 
> eventually.
> I've read the documentation you've linked prior to creating this 
> conversation. However, I'm unclear as to how to modify my pipeline C (or 
> rather template T_C) so there's no more explicit dependency on the upstream 
> pipeline. In my opinion I did this by having the *Fetch Artifact* task in 
> T_C which allows C which uses T_C to specify the upstream pipeline with 
> *#{UPSTREAM_BUILD_PIPELINE}*.
> What am I missing / doing wrong?
>
> In my pipeline C -> Materials I am unable to remove the dependency to the 
> upstream build pipeline A. As far as I can tell this has been added 
> automatically when creating pipeline A from template T_A as T_A requires 
> the *#{UPSTREAM_BUILD_PIPELINE}* parameter for the *Fetch Artifact* task.
>
> ~ Joel
>
>
> On Friday, November 27, 2020 at 4:16:38 PM UTC+1 ketanpad...@gmail.com 
> wrote:
>
>> If pipeline "C" requires a binary from "A", there's no need to add an 
>> explicit dependency. You can specify an upstream pipeline as part of your 
>> fetch artifact task.
>>
>> In either case, C should not trigger twice. Pipeline C should wait for B 
>> to be complete before kicking off.
>>
>> Also — what you're looking at is called "fan-in" in gocd terminology. This 
>> document  
>> might help.
>>
>> - Ketan
>>
>>
>>
>> On Fri, Nov 27, 2020 at 7:47 PM Joel Bodenmann  wrote:
>>
>>> Hello folks,
>>>
>>> Consider the following scenario:
>>>
>>>- Template *T_A*: Builds some binaries and stores them as an 
>>>artifact.
>>>- Template *T_B*: Fetches binaries from the upstream pipeline *T_A *and 
>>>runs tests. To achieve this,  *T_B* has a *Fetch Artifact *task with 
>>>the upstream pipeline as a parameter *#{UPSTREAM_BUILD_PIPELINE}.*
>>>- Template *T_C*: Fetches binaries from upstream pipeline *T_A* and 
>>>creates packages. To achieve this,  *T_C* has a *Fetch Artifact *task 
>>>with the upstream pipeline as a parameter 
>>>*#{UPSTREAM_BUILD_PIPELINE}.*
>>>- Pipeline *A*: Uses template *T_A*.
>>>- Pipeline *B*: Uses template *T_B**. *Sets *UPSTREAM_BUILD_PIPELINE  
>>>*to pipeline *A.*
>>>- Pipeline *C*: Uses template *T_C*. Sets *UPSTREAM_BUILD_PIPELINE  *
>>>to pipeline *A.*
>>>
>>> This results in the following VSM:
>>>
>>>
>>> The problem I have is that pipeline C will be triggered twice: Once 
>>> after A finishes and once after B finishes. The expected behavior (or the 
>>> behavior I wish to achieve) is that C will only run after B finishes 
>>> successfully.
>>>
>>> How can I make sure that C only runs after successful completion of B 
>>> (and therefore implicitly A)?
>>>
>>> Attached is the corresponding XML fragment of my templates & pipelines 
>>> where:
>>>
>>>- *T_A -> Build-CMake-OS-Compiler*
>>>- *T_B -> Test-OS*
>>>- *T_C -> Package-OS*
>>>- *A -> GPDS-Build-FreeBSD-GCC*
>>>- *B -> GPDS-Test-FreeBSD-GCC*
>>>- *C -> GPDS-Package-FreeBSD*
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to go-cd+un...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/go-cd/0a8a6dc4-0b49-4036-a573-44d0feca32b9n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/b3e4e221-1a00-424a-9d51-fef13e885b7fn%40googlegroups.com.
  

  
gcc
FreeBSD
g++10
Unix Makefiles
gcc10
4
  
  
https://github.com/simulton/gpds"; />
  



  
build/test/gpds-tests
FreeBSD
GPDS-Build-FreeBSD-GCC
  
  

  



  
GPDS-Build-FreeBSD-GCC/GPDS-Test-FreeBSD-GCC
FreeBSD
  
  

  

[go-cd] Re: Cannot run agent in MSYS2

[Jason Smyth]

Hi Joel,

I can't say for sure that it is possible because I am not familiar with 
MSYS2.

GoCD's Task is effectively a wrapper around a command that gets passed to 
(I think) cmd.exe on Windows. Assuming there is a way to launch MSYS2 (and 
pass it an instruction set) from the command line it should be fairly 
straight-forward to configure GoCD to launch it as a child process and 
execute whatever commands need to be executed in that environment.

Sorry I can't be more specific. I don't know the specific technology you 
depend on but the high-level concept I have in mind whenever I need to get 
GoCD to do something new is "I can make it do anything I can make the 
computer do from the command line."

Hope this helps,
Jason
On Tuesday, 24 November 2020 at 15:38:01 UTC-5 tec...@gmail.com wrote:

> Jason,
>
> Thank you for your reply.
>
> I am completely new to GoCD. If I can configure the agent to run all tasks 
> within the MSYS2 / MinGW64 shell then things will work and the agent itself 
> does not need to run inside MSYS2.
> The reason I attempted to run it inside MSYS2 is because I noticed that 
> then all tasks executed by the agent are executed in the MSYS2 / MinGW64 
> shell without any further configuration.
>
> If I'm able to configure the agent to run each task in the MSYS / MinGW64 
> shell I'm more than happy. How can I do that?
>
>
> Best regards,
> ~ Joel
>
> On Tuesday, November 24, 2020 at 8:48:45 PM UTC+1 jsm...@scimarketview.com 
> wrote:
>
>> Hello,
>>
>> I'm not familiar with MSYS2 so maybe I am missing an obvious point but 
>> why is it necessary that the Agent process itself run within MSYS2?
>>
>> Can the Agent process not just launch a child MSYS2 process if the Job 
>> requires it?
>>
>> Regards,
>> Jason
>>
>> On Tuesday, 24 November 2020 at 11:45:22 UTC-5 tec...@gmail.com wrote:
>>
>>> I've been testing GoCD today. One of my main requirements is that the 
>>> Windows agent(s) can perform builds within MSYS2 (unix environment).
>>>
>>> To get started I've opened an MSYS2 / MinGW64 terminal and pasted the 
>>> "Test Drive GoCD" line in there. Everything worked out of the box. The 
>>> script properly setup both the server and the agent and everything was 
>>> running within MSYS2. I was able to perform builds as per my requirements.
>>>
>>> The next step was to only get the agent running as the server itself is 
>>> running on a separate FreeBSD machine. However, when attempting to run the 
>>> agent within MSYS2 (MinGW64 shell) I'm presented with this message:
>>>
>>> $ ./bin/go-agent console
>>> Unable to locate any of the following binaries:
>>>   
>>> /home/joel/go-agent-20.10.0/bin/../wrapper/wrapper-mingw64_nt-10.0-19042-x86-64
>>>   
>>> /home/joel/go-agent-20.10.0/bin/../wrapper/wrapper-mingw64_nt-10.0-19042-x86-32
>>>   /home/joel/go-agent-20.10.0/bin/../wrapper/wrapper
>>>
>>> To me it seems that the java server wrapper is not finding the binaries 
>>> for my particular environment. However, the "Test Drive GoCD" 
>>> script/binaries ran without any issues.
>>>
>>> How can I solve this problem? Ultimately I just need to run the go-agent 
>>> within MSYS2.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/2939bcfa-4ec4-45b0-9084-e7f2f800af56n%40googlegroups.com.

[go-cd] Re: Cannot run agent in MSYS2

[Jason Smyth]

Hello,

I'm not familiar with MSYS2 so maybe I am missing an obvious point but why 
is it necessary that the Agent process itself run within MSYS2?

Can the Agent process not just launch a child MSYS2 process if the Job 
requires it?

Regards,
Jason

On Tuesday, 24 November 2020 at 11:45:22 UTC-5 tec...@gmail.com wrote:

> I've been testing GoCD today. One of my main requirements is that the 
> Windows agent(s) can perform builds within MSYS2 (unix environment).
>
> To get started I've opened an MSYS2 / MinGW64 terminal and pasted the 
> "Test Drive GoCD" line in there. Everything worked out of the box. The 
> script properly setup both the server and the agent and everything was 
> running within MSYS2. I was able to perform builds as per my requirements.
>
> The next step was to only get the agent running as the server itself is 
> running on a separate FreeBSD machine. However, when attempting to run the 
> agent within MSYS2 (MinGW64 shell) I'm presented with this message:
>
> $ ./bin/go-agent console
> Unable to locate any of the following binaries:
>   
> /home/joel/go-agent-20.10.0/bin/../wrapper/wrapper-mingw64_nt-10.0-19042-x86-64
>   
> /home/joel/go-agent-20.10.0/bin/../wrapper/wrapper-mingw64_nt-10.0-19042-x86-32
>   /home/joel/go-agent-20.10.0/bin/../wrapper/wrapper
>
> To me it seems that the java server wrapper is not finding the binaries 
> for my particular environment. However, the "Test Drive GoCD" 
> script/binaries ran without any issues.
>
> How can I solve this problem? Ultimately I just need to run the go-agent 
> within MSYS2.
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/59a60b7e-3330-43d3-becb-b701e8e22c47n%40googlegroups.com.

Re: [go-cd] Elastic agent registration problem from Windows docker swarm plugin

[Jason Smyth]

Hello,

The host OS is less relevant than the container OS in this situation. What 
OS is your custom container image based on? This information will likely be 
necessary to "tunnel" into the container from the host.

For troubleshooting, I recommend using docker's CLI to access one of your 
containers after GoCD spins it up. Since the containers are registering 
themselves as Static Agents, you should be able to see the Sandbox 
directory listed in the GoCD UI. Based on the information that Ram provided 
and what I see in our own Agents, you want to look for 
/config/autoregister.properties.

My guess is that Ram is correct and this file is either missing or does not 
contain the expected information.

Hope this helps,
Jason

On Wednesday, 4 November 2020 at 09:25:46 UTC-5 IXI wrote:

> Yes we do, what you mentioned. Path: 
> C:\Go-Agent\config\autoregister.properties  (please remember that we are 
> doing it on Windows Server 2016 machine)
>
> And just repeating question from my previous answer:  Is there a way to 
> debug our situation? It seems like a windows container problem.  
>
> Dne středa 4. listopadu 2020 v 13:18:33 UTC+1 uživatel srir...@gmail.com 
> napsal:
>
>> On Wed, Nov 4, 2020 at 8:12 PM IXI  wrote:
>>
>>> Yes we do. Is there a way to debug our situation? It seems like a 
>>> windows container problem.
>>>
>>
>> If your containers are coming up and getting registered as static agents, 
>> then the agent startup is likely unable to locate autoregister.properties.
>>
>> At which path are you installing the go-agent?
>> You need to create a config sub-directory, and then place the 
>> autoregister.properties file there (see 
>> https://docs.gocd.org/current/advanced_usage/agent_auto_register.html)
>>
>> "On the remote GoCD Agent machine, create a file named 
>> /config/autoregister.properties. In a native 
>> agent installation this file is usually located on 
>> /var/lib/go-agent/config/ directory. The config/ folder might not exist, in 
>> this case you should create it by yourself."
>>
>> -- Ram
>>
>>
>>  
>>
>>>
>>> Dne středa 4. listopadu 2020 v 12:21:00 UTC+1 uživatel Aravind SV napsal:
>>>
 Hello,

 I see. You’re creating your own image.

 The docker image provided by GoCD (on DockerHub) does this 
 
  
 to populate the autoregister.properties file correctly, based on 
 environment variables passed in by the plugin when it wants to start an 
 elastic agent. Do you do that in your image too? The values passed in 
 should be used. That’s how the server / plugin knows that the agent it was 
 waiting for has come up.

 Cheers,
 Aravind

>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to go-cd+un...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/go-cd/b1afe618-8db0-4023-9f12-f6578ddd9c88n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/5bf34784-5091-4678-b434-53ee5d1bbc89n%40googlegroups.com.

[go-cd] Re: GoCD with Bitbucket Cloud Issue

[Jason Smyth]

Hello,

I'm not familiar with Bitbucket cloud specifically but it seems to me you 
should have a few options here:


   1. Undelete the deleted user.
  - I have no idea if Bitbucket supports this or not but, if so, it is 
  almost certainly the easiest option.
   2. Recreate the deleted user:
  - Make sure that the username is exactly the same as the original.
  - Extract a copy of the user's public key from the GoCD server and/or 
  an Agent that used to work.
  - Upload the public key to the recreated user's Bitbucket profile.
   3. Recreate a working configuration:
  - Create a new user (or pick an existing one). Ideally this user 
  should be dedicated to GoCD and not associated with any individual in the 
  organization.
  - Create a keypair for the new user.
  - Install the private key on all GoCD servers and Agents that need 
  access to the Bitbucket repository.
  - Update the Material configuration of all Pipelines that access the 
  Bitbucket repository to use the new username.
   
Again, I am not familiar with Bitbucket, but these concepts should apply 
anywhere that SSH key authentication and authorization are being used.

Hope this helps,
Jason


On Friday, 23 October 2020 at 15:44:22 UTC-4 wson...@gmail.com wrote:

> I hope this is the right forum, please forgive me if it isn't. We use GoCD 
> for our CI/CD with git on Bitbucket cloud. Recently some brilliant dude 
> (me) removed the user in BB whose user and SSH keys were being used to run 
> all the git commands against our BB repo. So now nothing works. We're 
> looking for advice for reinstating a working SSH key and user to get our 
> builds working again.  Any advice would be so appreciated. 

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/8486eaa8-c2bd-41a7-9192-50d1338b5a46n%40googlegroups.com.

[go-cd] Re: timer-triggered pipeline not using latest dependency

[Jason Smyth]

Hi Julia,

The only thing I can think of that might cause an issue from what I can see 
in the Value Stream Maps that you posted is that the "ui_full" and 
"plugins" Pipelines seem to share a particular SVN source (coloured purple 
in the second photo). Are you sure that ui_full v2403 and plugins v189 both 
triggered using the same version of that Material?

Have you checked the logs on the GoCD server to see if there are any 
messages there relating to how the "apps" Pipeline is choosing the version 
of the plugins Pipeline to use?

I don't have any brilliant insights as to what the issue is at this point 
but hopefully this helps a little.

Regards,
Jason

On Friday, 23 October 2020 at 04:41:18 UTC-4 aquil...@gmx.de wrote:

>
> Hi everyone, 
>
> I'm struggling a bit with understanding the fan-in especially when using 
> timer-triggered pipelines. I hope that you can help me either to understand 
> or even to solve the following issue.
>
> There are 4 dependent pipelines as shown below, which are all 
> timer-triggered:
>
>- ui_full is triggered at 5pm
>- plugins and core are triggered at 6pm
>- apps is triggered at 7pm
>
> The timeslots are huge enough to ensure that the previous build has been 
> finished before the next dependent pipeline is started. The subversion 
> materials are usually subfolders of three different subversion repositories 
> (marked by color).
>
> Unfortunately, the apps pipeline consumes the very same version of the 
> plugins (e.g. 159) for every run, until the artifacts have been garbage 
> collected by GoCD and the pipeline fails. Then I have to trigger the apps 
> pipeline manually once, so that it uses the latest and greatest plugins 
> (e.g. 189). 
>
> Why doesn't the apps pipeline use the latest run of the plugins pipeline 
> anyways, while this is working for e.g. the core pipeline. How can I force 
> it to use the latest one? Why does it work when triggered manually, but not 
> when using the timer?
>
> There are no check-ins or manual triggers in between which would mess this 
> up. Any hint is appreciated.
>
> Regards,
> Julia
>
> Timer triggered:
> [image: timer-triggered.png]
>
> Manually triggered:
> [image: manually_triggered.png]
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/11b7dcc0-f1a9-452f-a81d-9b75b6ad52ben%40googlegroups.com.

Re: [go-cd] How to clear cached material source repo on gocd server?

[Jason Smyth]

Hi Bojan,

I think I understand now. The issue is that there is a Pipeline (or 
Pipelines) in GoCD that builds multiple source materials, including this 
3rd party library. As a result, because GoCD sees the 3rd party library 
source as "broken", the Pipeline doesn't trigger or doesn't build properly.

Since GoCD stores state in its database, I think there are a few ways you 
can deal with this situation:

1. Restore GoCD from a backup taken before the now-missing commit was 
introduced to the 3rd party library source. The obvious downside here is 
that you would lose anything that happened in GoCD since the backup was 
taken. Depending on the specific processes you have implemented this may or 
may not be a significant concern.
2. Manually trigger each affected Pipeline and specify the older version of 
the library's material. Keep doing this until a new commit is pushed into 
the library, which should resolve the current issue.
3. Change the material URL so that GoCD believes it is a new material. 
(Sounds like you have done this already but are unhappy with the result.)
4. Try to "hack" the GoCD database so that GoCD forgets about any commits 
that were deleted from the library's repo. I don't have nearly a good 
enough grasp of the database's structure to know how to do this, or even if 
it is realistically feasible.

In order to prevent this from recurring, you may want to rethink the way 
the build solution is architected. Instead of pulling directly from the 3rd 
party's git repo as part of your own build, could you not build the 3rd 
party library in its own Pipeline and then consume the output of that in 
other Pipelines? Alternately, if the nature of the code and the libraries 
is such that they must both be compiled from source at the same time, it 
may be worth creating your own fork of the 3rd party library on github and 
then building from that instead of directly from the 3rd party's repo. You 
would then be responsible for syncing the 3rd party's updates into your 
fork, but you would also have (almost) full control over your own source, 
which helps avoid this problem of "Someone on the internet did something 
and now my build(s) is (are) broken."

Hope this helps,
Jason


On Monday, 5 October 2020 at 12:16:00 UTC-4 Bojan S. wrote:

> Hi, Jason, 
> problem is that someone forced pushed on master on 3rd party source repo. 
> GoCD collected those commits, made build with our source, all was fine.
>
> Few days later, someone overwritten master. GoCD is stuck looking for 
> commit XYZ that was forced pushed, and does not collect latest master.
>
> Material for that 3rd party looks like below:
>
> {
> "branch": "master",
> "destination": "z3",
> "name": "z3",
> "type": "git",
> "url": "https://github.com/somecompany/library"; 
> }
>
> I "fixed" buy changing url to e.g. "
> https://github.com/somecompany/library.git";, and pushed, which works 
> fine, but because of our other libs, this issues long build for every 
> component, plus makes unnecessary commits to our code base.
>
>
> Please let me know if you need more details.
>
> Kind regards,
> Bojan
> On Monday, October 5, 2020 at 5:14:38 PM UTC+2 jsm...@scimarketview.com 
> wrote:
>
>> Hi Bojan,
>>
>> Could you elaborate on what the impact is to your process?
>>
>> It seems to me that if the source repo is not changing (no updates have 
>> been committed), then there should be nothing to build and it wouldn't be a 
>> problem that GoCD is "stuck" on the new version of master.
>>
>> I must be missing something on why this is an issue you need to solve.
>>
>> Regards,
>> Jason
>>
>> On Monday, 5 October 2020 at 08:19:01 UTC-4 Bojan S. wrote:
>>
>>> As I explained, it is 3rd party repo. I cannot make commit to source I 
>>> do not own.
>>>
>>> Please read the original note. Thank you.
>>> On Monday, October 5, 2020 at 1:13:07 PM UTC+2 ashwant...@gmail.com 
>>> wrote:
>>>
 Make a new commit to that repo. It should fix the problem. GoCD stores 
 the state in the DB. 

 On Mon, 5 Oct, 2020, 15:54 Bojan S.,  wrote:

> Hi all. 
>
> Source repo for 3rd party lib I am using has forced pushed some 
> changed to master, which our gocd accepted, but they removed those 
> commits 
> back to pre force pushed state.
>
> Now, my pipeline is stuck at "new" forced pushed state, and it cannot 
> collect latest master, which is basically old one.
>
> How can I clear git cache on gocd server to use current master state 
> always without getting stuck on git repo illegal state?
>
> Kind regards
> B
>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to go-cd+un...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/go-cd/629dc34a-2f02-4479-8cd9-b257b7ad247fn%40googlegroups.com

Re: [go-cd] How to clear cached material source repo on gocd server?

[Jason Smyth]

Hi Bojan,

Could you elaborate on what the impact is to your process?

It seems to me that if the source repo is not changing (no updates have 
been committed), then there should be nothing to build and it wouldn't be a 
problem that GoCD is "stuck" on the new version of master.

I must be missing something on why this is an issue you need to solve.

Regards,
Jason

On Monday, 5 October 2020 at 08:19:01 UTC-4 Bojan S. wrote:

> As I explained, it is 3rd party repo. I cannot make commit to source I do 
> not own.
>
> Please read the original note. Thank you.
> On Monday, October 5, 2020 at 1:13:07 PM UTC+2 ashwant...@gmail.com wrote:
>
>> Make a new commit to that repo. It should fix the problem. GoCD stores 
>> the state in the DB. 
>>
>> On Mon, 5 Oct, 2020, 15:54 Bojan S.,  wrote:
>>
>>> Hi all. 
>>>
>>> Source repo for 3rd party lib I am using has forced pushed some changed 
>>> to master, which our gocd accepted, but they removed those commits back to 
>>> pre force pushed state.
>>>
>>> Now, my pipeline is stuck at "new" forced pushed state, and it cannot 
>>> collect latest master, which is basically old one.
>>>
>>> How can I clear git cache on gocd server to use current master state 
>>> always without getting stuck on git repo illegal state?
>>>
>>> Kind regards
>>> B
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to go-cd+un...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/go-cd/629dc34a-2f02-4479-8cd9-b257b7ad247fn%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/b49aa30d-0abb-4f1e-a9d7-3ca65f0b5076n%40googlegroups.com.

[go-cd] Re: Running makefile within an agent

[Jason Smyth]

Hi Ryan,

You might want to take a look at these 2 threads that both seem to mention 
the same issue you describe:

https://groups.google.com/forum/#!topic/go-cd/fjOz211CIhY
https://groups.google.com/forum/#!topic/go-cd/Lejm7CCgJ_Q

Hope this helps,
Jason

On Thursday, 13 August 2020 07:25:02 UTC-4, Ryan Smith wrote:
>
> Hey everyone,
>
> I'm trying to configure one of my tasks to run a makefile like I would 
> when configuring it locally but I'm having a load of problems with it not 
> being a recognised command.
>
> Am I configuring this incorrectly or do I need to include in somewhere?
>
> Cheers,
> Ryan
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/b73d3677-229f-43f8-a424-01a015e46915o%40googlegroups.com.

[go-cd] Re: Artifact size limitation in GO 20.6.0?

[Jason Smyth]

Hi Chris,

Are there any quotas in place on the file system that the GoCD might be 
hitting? Are there any error messages in the server logs when the artifact 
upload fails? Are there any errors in Event Log when the artifact upload 
fails?

Unless there is a quota in place, it sounds like you're right about there 
being plenty of disk space so maybe the logs will have more information 
that can point you in the right direction.

Regards,
Jason

On Tuesday, 11 August 2020 11:14:50 UTC-4, Chris Payne wrote:
>
> Well we're on a Windows server. We have over 160GB free space on a 1TB 
> drive. Everything is stored on the single C drive. As far as I can tell, 
> everything looks fine on the server itself. But maybe there's something 
> hidden I'm missing.
>
> On Tuesday, August 11, 2020 at 11:10:39 AM UTC-4, Jason Smyth wrote:
>>
>> Hi Chris,
>>
>> The last time someone brought up a space issue like this, it turned out 
>> that the server had plenty of space but it was allocated elsewhere on the 
>> file system.
>>
>> Have you checked the server and confirmed that there are no volumes that 
>> are running out of space? I suggest looking specifically at the mount 
>> points that hold /tmp and GoCD's server directory.
>>
>> Regards,
>> Jason
>>
>> On Tuesday, 11 August 2020 11:05:54 UTC-4, Chris Payne wrote:
>>>
>>> We upgraded to 20.6.0 from 19.12 over the weekend and we're now getting 
>>> problems uploading large-ish artifacts (35MB+). This didn't happen 
>>> before... seeing these errors in our logs:
>>>
>>> [go] Artifact upload for file ### (Size: 38003857) was denied by the 
>>> server. This usually happens when server runs out of disk space.
>>> [go] Failed to upload #
>>>
>>> Is there some new limit imposed? Is there any way to configure this? My 
>>> searching on this turns up the problem reported with the Linux tmp 
>>> directory, but I'm not sure how that applies here at all. I don't see any 
>>> other settings in the config files or in the Tanuki wrapper that could 
>>> tweak this. The OS didn't change, and we have plenty of storage space on 
>>> the server (over 160GB). 
>>>
>>> Thanks!
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/12a9889a-9c30-4dab-a189-1630bed7708co%40googlegroups.com.

[go-cd] Re: Artifact size limitation in GO 20.6.0?

[Jason Smyth]

Hi Chris,

The last time someone brought up a space issue like this, it turned out 
that the server had plenty of space but it was allocated elsewhere on the 
file system.

Have you checked the server and confirmed that there are no volumes that 
are running out of space? I suggest looking specifically at the mount 
points that hold /tmp and GoCD's server directory.

Regards,
Jason

On Tuesday, 11 August 2020 11:05:54 UTC-4, Chris Payne wrote:
>
> We upgraded to 20.6.0 from 19.12 over the weekend and we're now getting 
> problems uploading large-ish artifacts (35MB+). This didn't happen 
> before... seeing these errors in our logs:
>
> [go] Artifact upload for file ### (Size: 38003857) was denied by the 
> server. This usually happens when server runs out of disk space.
> [go] Failed to upload #
>
> Is there some new limit imposed? Is there any way to configure this? My 
> searching on this turns up the problem reported with the Linux tmp 
> directory, but I'm not sure how that applies here at all. I don't see any 
> other settings in the config files or in the Tanuki wrapper that could 
> tweak this. The OS didn't change, and we have plenty of storage space on 
> the server (over 160GB). 
>
> Thanks!
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/16cc50a1-c309-4176-a0c7-f87d852cc8e1o%40googlegroups.com.

[go-cd] Re: GoCD - Deploy using an artefact rather using git checkout - URGENT please

[Jason Smyth]

Hello Prakash,

What happens if you add "echo" to the start of the command? Assuming it 
works as intended, the output of echo should make it clear whether or not 
the environment variables are expanding properly.

Another potential issue is with the way the command is constructed in the 
Pipeline definition. GoCD *may* be trying to execute a command called 
'/usr/bin/curl -L -k -v -u : -X GET 
"https://artifactory.com/service/rest/v1/search/assets/download?group=/xx-prj/master/${GO_PIPELINE_LABEL}&name=xx-prj/master/${GO_PIPELINE_LABEL}/${GO_REVISION_MYGIT}.zip";
 
-o "${GO_REVISION_MYGIT}.zip" && unzip -o "${GO_REVISION_MYGIT}.zip" -d .', 
rather than running curl with the rest of the string as arguments.

We've also had trouble getting quoting just right in our custom commands. 
Here's the XML definition of the command from my previous message:

  
/c
""#{msbuild}"

"#{projectRootDirectory}\#{projectName}\#{projectName}.proj"
/t:Release"

  

Note that each space-delimited argument is in its own  element, which 
translates to each on its own line if editing in the GUI. Note as well that 
each argument that could contain a space is enclosed in double quotes. 
Lastly, to get this to work, we had to add a double quote to the start of 
the second argument (first one after "/c") and to the end of the last 
argument so that the final argument list becomes */c ""msbuild" 
"path/to/project.proj" /t:Release".*

Hope this helps,
Jason

On Monday, 10 August 2020 10:31:19 UTC-4, Prakash Tirumalareddy wrote:
>
> Hello Jason
> Thanks for your email.
> If I updated variable manually the command works in agent.
>
> I am getting the following error: which does not tell me that is related 
> with populating the variable or not.. 
>
> *[go] Job Started: 2020-08-10 19:53:21 AEST*
>
> *[go] Start to prepare xx-prj/77/6-DeploySIT1/1/deploy on test.internal 
> [/var/lib/go-agent]*
>
> *[go] Cleaning working directory "/var/lib/go-agent/pipelines/ xx-prj" 
> since stage is configured to clean working directory*
>
> *[go] Skipping material update since stage is configured not to fetch 
> materials*
>
> *[go] Start to build xx-prj /77/6-DeploySIT1/1/deploy 
> on test.internal [/var/lib/go-agent]*
>
> *[go] Task: /usr/bin/curl -L -k -v -u : -X GET 
> "https://artifactory.com/service/rest/v1/search/assets/download?group=/xx-prj/master/${GO_PIPELINE_LABEL}&name=xx-prj/master/${GO_PIPELINE_LABEL}/${GO_REVISION_MYGIT}.zip
>  
> "
>  
> -o "${GO_REVISION_MYGIT}.zip" && unzip -o "${GO_REVISION_MYGIT}.zip" -d 
> .'took: 0.2s*
>
> *Error happened while attempting to : /usr/bin/curl -L -k -v -u 
> : -X GET 
> "https://artifactory.com/service/rest/v1/search/assets/download?group=/xx-prj/master/${GO_PIPELINE_LABEL}&name=xx-prj/master/${GO_PIPELINE_LABEL}/${GO_REVISION_MYGIT}.zip
>  
> "
>  
> -o "${GO_REVISION_MYGIT}.zip" && unzip -o "${GO_REVISION_MYGIT}.zip" -d .] 
> can be executed on this agent. *
>
> *[Debug Information] Environment variable PATH: 
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin*
>
> *[go] Task status: failed, took: 0.2s*
>
> *[go] Current job status: failed*
>
> *[go] Start to create properties xx-prj/77/6-DeploySIT1/1/deploy 
> on test.internal [/var/lib/go-agent]*
>
> *[go] Start to upload xx-prj/77/6-DeploySIT1/1/deploy on test.internal 
> [/var/lib/go-agent]*
>
> *[go] Job completed xx-prj/77/6-DeploySIT1/1/deploy 
> on test.internal [/var/lib/go-agent]*
>
> Regards
> Prakash
>
>
> On Tuesday, 11 August 2020 at 00:17:25 UTC+10 jsm...@scimarketview.com 
> wrote:
>
>> Hello Prakash,
>>
>> What do you see in the console output beyond the error message? I believe 
>> the raw command should get printed to console which would make 
>> troubleshooting easier.
>>
>> Here is an example from the raw output of one of our Pipelines that uses 
>> Custom Commands:
>>
>> !!|10:43:38.980 [go] Task: cmd /c 
>> ""C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe" 
>> "tfs\path\to\project.proj" "/t:Release\""
>>
>>
>> I suspect that the issue you are having is that GoCD tries to expand the 
>> environment variables used in your command before they have been defined 
>> but, again, you should be able to see that from the console output.
>>
>> If the console output looks right, including all of the environment 
>> variables, try running it from the command line on the agent to see what 
>> the error message is there.
>>
>> Hope this helps,
>> Jason
>>
>>
>>
>> On Monday, 10 August 2020 09:18:09 UTC-4, Prakash Tirumalareddy wrote:
>>>
>>> Hello Helge
>>>
>>> We t

[go-cd] Re: GoCD - Deploy using an artefact rather using git checkout - URGENT please

[Jason Smyth]

Hello Prakash,

What do you see in the console output beyond the error message? I believe 
the raw command should get printed to console which would make 
troubleshooting easier.

Here is an example from the raw output of one of our Pipelines that uses 
Custom Commands:

!!|10:43:38.980 [go] Task: cmd /c 
""C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe" 
"tfs\path\to\project.proj" "/t:Release\""


I suspect that the issue you are having is that GoCD tries to expand the 
environment variables used in your command before they have been defined 
but, again, you should be able to see that from the console output.

If the console output looks right, including all of the environment 
variables, try running it from the command line on the agent to see what 
the error message is there.

Hope this helps,
Jason


On Monday, 10 August 2020 09:18:09 UTC-4, Prakash Tirumalareddy wrote:
>
> Hello Helge
>
> We tried using the curl command:
>
> 6-DeploySIT1: 
> fetch_materials: no
> clean_workspace: yes
> approval: manual
> environment_variables:
> app_environment: nonprod
> verify_repo: verify-repo
> jobs:
> deploy:
> tasks: 
> - exec:
> command: /usr/bin/curl -L -k -v -u : -X 
> GET "
> https://artifactory.xys.com/service/rest/v1/search/assets/download?group=//master/${GO_PIPELINE_LABEL}&name=/master/${GO_PIPELINE_LABEL}/${GO_REVISION_MYGIT}.zip"
>  
> -o "${GO_REVISION_MYGIT}.zip" && unzip -o "${GO_REVISION_MYGIT}.zip" -d .'
> - exec: 
> arguments:
> - deploy
> command: make
>
> Error is:
> Check your command whether it can be executed on your agent.
>
> Regards
> Prakash
>
> On Saturday, 8 August 2020 at 00:06:00 UTC+10 bewal...@gmail.com wrote:
>
>> Hello Prakash,
>>
>> for an example see here: 
>> https://github.com/tomzo/gocd-yaml-config-plugin#example
>>
>> tasks: # ordered list of tasks to execute
>> - exec: # indicates type of task
>> command: make
>> arguments:
>> - "VERBOSE=true"
>>
>> You can specify your curl command here instead of the "make" call. The 
>> curl call is specific to your artifactory and the repository type you use 
>> -- so it's difficult for me to give you a neaningfull example.
>>
>> Regards,
>> Helge
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/ba64b612-6caf-4276-aecf-e1b5cf97cd26o%40googlegroups.com.

[go-cd] Re: GoCD - Deploy using an artefact rather using git checkout - URGENT please

[Jason Smyth]

Hello Prakash,

You wrote in your initial post:

I know there is a stage configuration which says not to checkout the 
> material. But when we enable this option then there is no way in that stage 
> to do required task.


Why can GoCD not do the required task if materials are not fetched from git?

I think once you know the answer to this question you should be well on 
your way to solving the issue.

Hope this helps,
Jason


On Wednesday, 5 August 2020 06:14:13 UTC-4, Prakash Tirumalareddy wrote:
>
> Hello Helge
>
> Thanks for your kind reply. 
> Either of the option is not solving the git-checkout at every stage.
> Is there way we can achieve to run stage by executing download the 
> artefacts from Artifactory. (Can we put curl command in the pipeline.yaml 
> itself? if so can you please give me some example?)
> Thanks again!!!
>
> Regards
> Prakash
>  
>
> On Wednesday, 5 August 2020 at 18:40:11 UTC+10 bewal...@gmail.com wrote:
>
>> Hello Prakash,
>>
>> there are 2 ways to achieve this:
>> 1. You can store the created artifact from the previous stage as a 
>> resulting artifact (
>> https://docs.gocd.org/20.4.0/configuration/managing_artifacts_and_reports.html)
>>  
>> and then fetch the artifact in the next stage with the "Fetch Artifact" 
>> task for further use.
>> 2. Upload the artifact to the Artifactory in the previous stage and 
>> retrieve the artifact in next stage using a simple curl task.
>> I would prefer the first option because all the artifact handling is done 
>> by GoCD internally.
>>
>> Regards,
>> Helge
>>
>>
>> Am Mittwoch, 5. August 2020 09:41:40 UTC+2 schrieb Prakash Tirumalareddy:
>>>
>>> Hello Everyone
>>>
>>> Is there anyway we can configure a particular stage not to do 
>>> git-checkout and instead download the artefacts from external repository 
>>> (such jfrog artifactory) which was built and pushed in previous stage?
>>>
>>> Currently what have seen that at every stage it does the git-checkout 
>>> for the commit hash. 
>>> I know there is a stage configuration which says not to checkout the 
>>> material. But when we enable this option then there is no way in that stage 
>>> to do required task.
>>>
>>> Your help will be greatly appreciated.
>>>
>>> Regards
>>> Prakash
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/5b437050-5134-4def-9f38-726bf39fabe4o%40googlegroups.com.

Re: [go-cd] Re: Can anyone help me with the steps to get reports in custom tab from all jobs running in parallel in one stage, AS i have noticed i am getting report only for 1 stage and other stage re

[Jason Smyth]

Hi Sunaina,

If you want to merge the HTML reports from multiple GoCD Jobs into a single 
HTML document, you will need a process that runs after all of the relevant 
Jobs has completed.

The simplest way to accomplish this in GoCD is to add a new Stage that can 
consume the artifacts generated in the previous Stage:

   1. Stage 1: Do stuff in multiple Jobs. Generate multiple HTML reports.
   2. Stage 2: Merge HTML reports from Stage 1.

Another way to accomplish this is to add a Job to the existing Stage where 
the first Task is to wait for all other Jobs in the Stage to complete:

   1. Job 1:
   1. Task 1: Do stuff
  2. Task 2: Create HTML report
  2. Job 2:
   1. Task 1: Do stuff
  2. Task 2: Create HTML report
  3. Job 3:
   1. Task 1: Wait for all other Jobs to complete
  2. Task 2: Merge HTML reports from all other Jobs
  
There are some downsides to this approach. Job 3 will often have a much 
longer runtime than is required to do its work. If you're not using Elastic 
Agents this means that Job 3 needlessly ties up one of your Agents while 
waiting for the other Jobs to complete. Depending on the specifics of your 
GoCD infrastructure configuration, it is even possible for Jobs to be 
scheduled in such a way that the Pipeline Stage gets blocked. Imagine this 
as a Template in a setup with just 2 static Agents and 2 Pipelines based on 
this Template. It is possible, in this scenario, that Pipeline 1 Job 3 gets 
started on Agent 1 and Pipeline 2 Job 3 gets started on Agent 2 before Job 
1/2 from either Pipeline, resulting in both Pipelines being blocked as Job 
3 runs forever waiting for the other Jobs to finish and the other Jobs 
stuck waiting for an Agent on which they can run.

A third way to do it is to add a Task to end of each Job that does the 
merge work if it is the last Job running for the Stage:

   1. Job 1:
  1. Task 1: Do stuff
  2. Task 2: Generate HTML report
  3. Task 3: Am I the only unfinished Job? If yes, merge all reports.
   2. Job 2:
  1. Task 1: Do stuff
  2. Task 2: Generate HTML report
  3. Task 3: Am I the only unfinished Job? If yes, merge all reports.
   3. ...
   4. Job N:
  1. Task 1: Do stuff
  2. Task 2: Generate HTML report
  3. Task 3: Am I the only unfinished Job? If yes, merge all reports.
   
This solution is subject to race conditions, though, because GoCD Jobs do 
not complete until all Artifacts have been uploaded and Artifacts are not 
uploaded until after all Tasks complete. This means that it is possible for 
the last Job to run its "am I the last?" check while the second-to-last Job 
is still uploading Artifacts. If this happens then the last Job will not 
recognize that it is the last one and will not run the HTML merge task.

Hope this helps.

Regards,
Jason


On Wednesday, 5 August 2020 06:13:08 UTC-4, Sunaina Awadhiya wrote:
>
> Thanks Sim,
>
> Really helpful information. I did create different folder path for html 
> report for 2 separate jobs which are executing in parallel and i have got 
> html reports successfully for both jobs but most of the times only 1 job 
> return html report. So just curious to know what could be the reason behind 
> and can you please help me how to merge all jobs report and put in one 
> single html.
>
> More detail steps with example would be more helpful.
>
> Thanks and Regards,
> Sunaina
>
> On Mon, Aug 3, 2020 at 1:17 AM Sasa Mitrovic  > wrote:
>
>> Hi,
>>
>> You need to create test artifacts for every single one you need. Path 
>> where to save it must be different file or folder. Important part is that 
>> this folder names becomes Artifact path after job finish and upload  
>> content to GOCD Server and you will see results in Artifacts Tab from UI. 
>> Now create Custom tabs and set source to point into this Artifacts. Your 
>> full nice report will be rendered inside custom tab. If you want to include 
>> only one report to show all of them, this report format need to include 
>> nice HTML page report created from all source jobs as one page with sub 
>> pages where all of them are in separate folder per each job you have. Use 
>> pure html or JS to create this summary page. This of this as HTML page with 
>> other source pages in different folders.
>>
>> I hope this will give you way and direction of thinking to better use 
>> Test Artifacts and Artifacts location on server after upload and 
>> combination with Custom Tabs. In this way Reporting becomes must powerful 
>> way how to do it from any other known CI/CD software. 
>>
>> P.S Not forget that all these paths are relative from Agent working 
>> directory so test it and learn it.
>>
>> B. Regards
>>
>> четвртак, 30. јул 2020. 19.20.26 UTC+2, Sunaina Awadhiya је написао/ла:
>>>
>>> Can anyone help me with the steps to get reports in custom tab from all 
>>> jobs running in parallel in one stage, AS i have noticed i am getting 
>>> report only for 1 stage and other stage reports are lost. I have c

[go-cd] Re: Can anyone help me with the steps to get reports in custom tab from all jobs running in parallel in one stage, AS i have noticed i am getting report only for 1 stage and other stage report

[Jason Smyth]

Hello Sunaia,

Could you provide an example that shows what you have done and maybe a 
mock-up explaining what you would like to accomplish?

My understanding is that Custom Tabs are created at the Job level, not at 
the Stage level, so I am having a hard time envisioning a workflow that 
might produce what you seem to be asking for.

Regards,
Jason

On Thursday, 30 July 2020 13:20:26 UTC-4, Sunaina Awadhiya wrote:
>
> Can anyone help me with the steps to get reports in custom tab from all 
> jobs running in parallel in one stage, AS i have noticed i am getting 
> report only for 1 stage and other stage reports are lost. I have created 
> custom tab for report
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/f1b5d0cc-9714-42ed-920e-2dd90856bc04o%40googlegroups.com.

Re: [go-cd] Re: I am looking for the step to follow for enabling email notification for one of GoCD pipeline, someone can please help

[Jason Smyth]

Hello Sunaina,

I recommend opening a new thread for your new question.

This will help others who may later come searching for answers to similar 
issues.

Regards,
Jason


On Wednesday, 29 July 2020 13:45:12 UTC-4, Sunaina Awadhiya wrote:
>
> Can anyone help me with the steps to get reports from all stages running 
> in parallel, AS i have noticed i am getting report only for 1 stage and 
> other stage reports are lost.
> I have created custom tab for report
>
> On Thu, May 7, 2020 at 4:22 PM Sunaina Awadhiya  > wrote:
>
>> Thanks Ashwanth for help, Could you please let me know below things:
>>
>> 1. which variable is set for total no of tests executed/Passed/Failed
>> 2. can we externalize test report generating inside.
>>
>> On Wed, May 6, 2020 at 9:57 AM 'Ashwanth Kumar' via go-cd <
>> go...@googlegroups.com > wrote:
>>
>>> Hello Sunaina,
>>>
>>> If you want more control on the notifications, consider adding a task 
>>> (preferably) as the last stage in the pipeline that can send you an email 
>>> with whatever template / attachments that's needed. This could be achieved 
>>> using a custom program 
>>>  / script 
>>> 
>>>  
>>> or if you're using a Linux based machine, you can also configure postfix 
>>> 
>>>  
>>> on the agent(s) and use mail  / mailx 
>>>  commands. 
>>>
>>> On Mon, May 4, 2020 at 10:12 PM Sunaina Awadhiya >> > wrote:
>>>
 Hi Kritika,

 Just curious to know if i can customize email notification default 
 template and report can be accessed without GoCD credentials. we generate 
 Html report in our application, is there any way to link such report.

 Thanks and Regards,
 Sunaina

 On Mon, May 4, 2020 at 9:10 AM kritik...@thoughtworks.com <
 kritik...@thoughtworks.com > wrote:

> Hey,
>
> This might be helpful to you: 
> https://docs.gocd.org/current/configuration/dev_notifications.html
>
> Thanks
> Kritika
>
> On Friday, May 1, 2020 at 9:48:34 PM UTC+5:30 s.saw...@gmail.com 
> wrote:
>
>> I am looking for the step to follow for enabling email notification 
>> for one of GoCD pipeline, someone can please help
>>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to go...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/go-cd/84501a74-8451-4bba-a2f3-8da079ea3810%40googlegroups.com
>  
> 
> .
>


 -- 
 Sunaina Awadhiya
 Bangalore

 -- 
 You received this message because you are subscribed to the Google 
 Groups "go-cd" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to go...@googlegroups.com .
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/go-cd/CA%2B7TbjKAKcKVai-_zGo14xXJQ1DEYqLoCRJW_f6hXbxE1BuxKw%40mail.gmail.com
  
 
 .

>>>
>>>
>>> -- 
>>>
>>> Ashwanth Kumar / ashwanthkumar.in
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to go...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/go-cd/CAD9m7CyCnMKTFrsiZvBuhYqcyrGC2EqXrjy9zkAz80aBw2Ohrw%40mail.gmail.com
>>>  
>>> 
>>> .
>>>
>>
>>
>> -- 
>> Sunaina Awadhiya
>> Bangalore
>>
>
>
> -- 
> Sunaina Awadhiya
> Bangalore
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/4532b59c-b410-41d5-877c-f3d5308afb01o%40googlegroups.com.

[go-cd] Re: Windows Go agent installation

[Jason Smyth]

Hi Balaji,

Is there some reason you need to install a version of the GoCD Agent that 
is so much older than the server?

I recommend installing using the installer for GoCD Agent version 19.8 and 
see if that resolves your issue.

Hope this helps,
Jason


On Thursday, 11 June 2020 10:59:36 UTC-4, Balaji Narasingapillai wrote:
>
> Hi Team,
>
> After go agent (17.2) installation in windows unable to view in go server 
> (19.8).
>
> in config folder agent.jks and guid.txt missing. any guidance please.
>
> Thanks,
> balaji 
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/9ee3c752-aaf4-4728-9d41-4bb992565f07o%40googlegroups.com.

Re: [go-cd] Re: Low database disk space warning message from Go Server

[Jason Smyth]

Hi Shivani,

>From the information provided in your initial post, it looks like most of 
the space on the root volume is consumed by /var/lib/go-server/artifacts 
(14GB).

My best guess is that this server was originally configured to use this 
directory for artifacts and then later changed to use /data/artifacts as 
shown in the screen shot. Assuming /var/lib/go-server/artifacts contains 
artifacts from old Pipeline runs, it should be safe to delete everything 
there.

I recommend reviewing the contents of /var/lib/go-server/artifacts and 
comparing them to what is currently in /data/artifacts. This should allow 
you to identify anything that does not exist in both locations and then 
make an educated decision about what is and is not safe to delete from 
/var/lib/go-server/artifacts.

Hope this helps,
Jason


On Friday, 17 April 2020 05:28:29 UTC-4, Ketan Padegaonkar wrote:
>
> Artifacts and database are different directories. It is not clear what 
> partition holds your DB directory.
>
> - Ketan
>
>
>
> On Fri, Apr 17, 2020 at 12:51 PM Shivani Shinde  > wrote:
>
>> Hi kritika,
>> Yes, I went through this link. But it states everything about 'artifacts 
>> low disk space'. I do not face the Artifact Disk Space issue, I faced 
>> *Database 
>> disk space low *issue.
>> Attaching screenshot for reference:
>>
>> [image: Screenshot from 2020-04-17 12-40-08.png]
>>
>>
>> I have a hard disk of 20GB attached at /data location for artifacts and 
>> have also followed Auto Delete Artifacts 
>> 
>>  configuration 
>> already:
>>
>> [image: Screenshot from 2020-04-17 12-47-47.png]
>>
>>
>> Is there something more that I could have missed?
>>
>> Thank you.
>>
>>
>> CONFIDENTIALITY. This email and any attachments are confidential to Alef 
>> Edge Inc., and may also be privileged, except where the email states it 
>> can be disclosed. If this email is received in error, please do not 
>> disclose the contents to anyone, notify the sender by return email, and 
>> delete this email (and any attachments) from your system.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/4a17d2fc-ebce-49b4-aacd-62f0e865243d%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/5c6dc9f0-ce4d-4091-b674-3758dbcd3778%40googlegroups.com.

[go-cd] Re: Changes in config.xml post upgrade - Not working

[Jason Smyth]

Hi Bala,

I recommend letting the GoCD upgrade facilities take care of this for you. 
I would do the following:


   1. Build a new server to replace your current 17.x one.
   2. Install an appropriate version of Java that will work with both GoCD 
   17.x and 19.x.
   3. Install GoCD server version 17.x (the same version as your current 
   server.
   4. Back up your current GoCD installation and restore it on the new 
   server.
   5. Upgrade the new server from 17.x to 19.x.

If you follow this process then you should end up with a version of 
config.xml that contains all of the information of your current 17.x 
installation but in the correct format for 19.x. This is essentially what 
we did when we upgraded from 15.x to 18.x late in 2018.

Hope this helps,
Jason


On Monday, 13 April 2020 07:50:24 UTC-4, Bala s wrote:
>
> Hello everyone,
>
> Recently i posted an upgrade approach and based on your inputs, i am able 
> to upgrade 19.x version from 17.x
> However I could see there are a considerable changes in 19.x config.xml 
> tags(as how the elements represent for e.g plugin, pipeline tags), which is 
> getting failed, if I try to restore my older config.xml file.
>
> Please note that we have more than 1000+ pipelines, hence the content of 
> the file is huge.
>
> Can you please suggest, how to migrate the config.xml to the new version. 
> Is there any tool or add-on available.
>
> Appreciate your inputs.
>
> Thanks,
> Bala
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/db6436a5-0929-4a2b-ba2b-0c6e8f168ded%40googlegroups.com.

Re: [go-cd] Passing variable to tasks

[Jason Smyth]

Hi Shivani,

I'm not sure I understand. Can you post specific examples of what is 
working and what isn't?

Another thing to try is:
bash
-c
make

The output of this should tell you whether or not the agent is able to find 
make on the path.

Regards,
Jason

On Thursday, 9 April 2020 10:08:52 UTC-4, Shivani Shinde wrote:
>
> HI Jason,
>
> Yes that did work and gave me correct path.
>
> But I have a weird observation:
> When I do not mention any environment variables, the command works 
> perfectly fine!
>
> And now when I add the environment variables, it give me the following 
> error:
> make: pwd: Command not found
> make: uname: Command not found
> /bin/sh: 1: basename: not found
> /bin/sh: 1: date: not found
> make: git: Command not found
>
> Any idea why shall this be happening?
>
>
> On Thursday, April 9, 2020 at 6:57:47 PM UTC+5:30, Jason Smyth wrote:
>>
>> Hi Shivani,
>>
>> The error message is saying that the "make" command is not found.
>>
>> I recommend trying the following as a troubleshooting step:
>> bash
>> -c
>> echo make build path="$(pwd)"
>>
>> The output from this should be: make build 
>> path="/path/to/pipeline/workingDirectory"
>>
>> If that works as intended then the next thing to troubleshoot is why make 
>> is not available on the path of your agent's shell.
>>
>> Regards,
>> Jason
>>
>> On Thursday, 9 April 2020 07:57:57 UTC-4, Shivani Shinde wrote:
>>>
>>> Hi Jason,
>>> I did try that. But I keep getting same error.
>>> Can you let me know commands need to mentioned from UI?
>>>
>>> On Wednesday, April 8, 2020 at 8:44:33 PM UTC+5:30, Jason Smyth wrote:
>>>>
>>>> Hi Shivani,
>>>>
>>>> If you want to wrap a call to anything in an explicit invocation of 
>>>> bash, the usual syntax is: bash -c "commandToRun with arguments".
>>>>
>>>> So to run the make command you specified, it would be: bash -c 'make 
>>>> build path="$(pwd)"'. Note the quotes around "$(pwd)" to deal with 
>>>> paths that contain spaces.
>>>>
>>>> Regards,
>>>> Jason
>>>>
>>>> On Wednesday, 8 April 2020 10:05:42 UTC-4, Shivani Shinde wrote:
>>>>>
>>>>> But I need to pass the path to my command which will be something like 
>>>>> this:
>>>>> make build path=$(pwd)
>>>>>
>>>>> Where I mention *make *in custom command field. How do you suggest to 
>>>>> use *bash -c * with this command?
>>>>>
>>>>> On Wed, Apr 8, 2020, 7:25 PM 'Ashwanth Kumar' via go-cd <
>>>>> go...@googlegroups.com> wrote:
>>>>>
>>>>>> Try using it like this 
>>>>>>
>>>>>> bash -c "echo $(pwd)" 
>>>>>>
>>>>>> On Wed, 8 Apr, 2020, 19:16 Shivani Shinde,  
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> Thank you for responding.
>>>>>>> I was initially using scripts only, but due to some reasons I need 
>>>>>>> to mention git URL in materials.
>>>>>>>
>>>>>>> I also tried using $(pwd) as suggested by you, but it gives me 
>>>>>>> output as:
>>>>>>>
>>>>>>> [go] Task: echo $(pwd)took: 0.2s
>>>>>>> $(pwd)
>>>>>>>
>>>>>>>
>>>>>>> I understand using scripts is better way to deal with variables when 
>>>>>>> trying to export them, but I was trying to find if any other way exists.
>>>>>>>
>>>>>>> Thank you.
>>>>>>>
>>>>>>> On Wednesday, April 8, 2020 at 5:31:16 PM UTC+5:30, Ashwanth Kumar 
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> One thing I've learned working with GoCD for a while is, you don't 
>>>>>>>> want to use absolute paths anywhere. Especially when agents are run 
>>>>>>>> across 
>>>>>>>> different hosts or as ephemeral containers somewhere. 
>>>>>>>>
>>>>>>>> To get the current pipeline path one can use something like $(pwd) 
>>>>>>>> or dirname etc.  and use relative paths from 

Re: [go-cd] Passing variable to tasks

[Jason Smyth]

Hi Shivani,

The error message is saying that the "make" command is not found.

I recommend trying the following as a troubleshooting step:
bash
-c
echo make build path="$(pwd)"

The output from this should be: make build 
path="/path/to/pipeline/workingDirectory"

If that works as intended then the next thing to troubleshoot is why make 
is not available on the path of your agent's shell.

Regards,
Jason

On Thursday, 9 April 2020 07:57:57 UTC-4, Shivani Shinde wrote:
>
> Hi Jason,
> I did try that. But I keep getting same error.
> Can you let me know commands need to mentioned from UI?
>
> On Wednesday, April 8, 2020 at 8:44:33 PM UTC+5:30, Jason Smyth wrote:
>>
>> Hi Shivani,
>>
>> If you want to wrap a call to anything in an explicit invocation of bash, 
>> the usual syntax is: bash -c "commandToRun with arguments".
>>
>> So to run the make command you specified, it would be: bash -c 'make 
>> build path="$(pwd)"'. Note the quotes around "$(pwd)" to deal with paths 
>> that contain spaces.
>>
>> Regards,
>> Jason
>>
>> On Wednesday, 8 April 2020 10:05:42 UTC-4, Shivani Shinde wrote:
>>>
>>> But I need to pass the path to my command which will be something like 
>>> this:
>>> make build path=$(pwd)
>>>
>>> Where I mention *make *in custom command field. How do you suggest to 
>>> use *bash -c * with this command?
>>>
>>> On Wed, Apr 8, 2020, 7:25 PM 'Ashwanth Kumar' via go-cd <
>>> go...@googlegroups.com> wrote:
>>>
>>>> Try using it like this 
>>>>
>>>> bash -c "echo $(pwd)" 
>>>>
>>>> On Wed, 8 Apr, 2020, 19:16 Shivani Shinde,  
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> Thank you for responding.
>>>>> I was initially using scripts only, but due to some reasons I need to 
>>>>> mention git URL in materials.
>>>>>
>>>>> I also tried using $(pwd) as suggested by you, but it gives me output 
>>>>> as:
>>>>>
>>>>> [go] Task: echo $(pwd)took: 0.2s
>>>>> $(pwd)
>>>>>
>>>>>
>>>>> I understand using scripts is better way to deal with variables when 
>>>>> trying to export them, but I was trying to find if any other way exists.
>>>>>
>>>>> Thank you.
>>>>>
>>>>> On Wednesday, April 8, 2020 at 5:31:16 PM UTC+5:30, Ashwanth Kumar 
>>>>> wrote:
>>>>>>
>>>>>> One thing I've learned working with GoCD for a while is, you don't 
>>>>>> want to use absolute paths anywhere. Especially when agents are run 
>>>>>> across 
>>>>>> different hosts or as ephemeral containers somewhere. 
>>>>>>
>>>>>> To get the current pipeline path one can use something like $(pwd) or 
>>>>>> dirname etc.  and use relative paths from that location. 
>>>>>>
>>>>>> Another point is Tasks within a job don't share any session on a 
>>>>>> shell of any kind among them. So if you export a variable and try to use 
>>>>>> it 
>>>>>> in the next task or later within the job it wouldn't work. They will 
>>>>>> have 
>>>>>> to be specified at the environment, pipeline, stage or at a job level. 
>>>>>> The 
>>>>>> specificity goes in the reverse order. Also as a general convention I've 
>>>>>> achieved best results when these scripts are NOT in GoCD tasks but 
>>>>>> instead 
>>>>>> in a separate shell script file and added as part of the source code or 
>>>>>> via 
>>>>>> a separate ops repo material(s). 
>>>>>>
>>>>>> HTH. 
>>>>>>
>>>>>> On Wed, 8 Apr, 2020, 17:13 Shivani Shinde,  
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>> I am having the hardest time dealing with variables for a task.
>>>>>>>
>>>>>>> What I want to achieve:
>>>>>>>
>>>>>>> 1. I want to pass a variable(pipeline's path) to the task which 
>>>>>>> generates builds. This is build command:
>>>>>>> make build path="/pipeline_path"

Re: [go-cd] Passing variable to tasks

[Jason Smyth]

Hi Shivani,

If you want to wrap a call to anything in an explicit invocation of bash, 
the usual syntax is: bash -c "commandToRun with arguments".

So to run the make command you specified, it would be: bash -c 'make build 
path="$(pwd)"'. Note the quotes around "$(pwd)" to deal with paths that 
contain spaces.

Regards,
Jason

On Wednesday, 8 April 2020 10:05:42 UTC-4, Shivani Shinde wrote:
>
> But I need to pass the path to my command which will be something like 
> this:
> make build path=$(pwd)
>
> Where I mention *make *in custom command field. How do you suggest to use 
> *bash 
> -c * with this command?
>
> On Wed, Apr 8, 2020, 7:25 PM 'Ashwanth Kumar' via go-cd <
> go...@googlegroups.com > wrote:
>
>> Try using it like this 
>>
>> bash -c "echo $(pwd)" 
>>
>> On Wed, 8 Apr, 2020, 19:16 Shivani Shinde, > > wrote:
>>
>>> Hi,
>>> Thank you for responding.
>>> I was initially using scripts only, but due to some reasons I need to 
>>> mention git URL in materials.
>>>
>>> I also tried using $(pwd) as suggested by you, but it gives me output as:
>>>
>>> [go] Task: echo $(pwd)took: 0.2s
>>> $(pwd)
>>>
>>>
>>> I understand using scripts is better way to deal with variables when 
>>> trying to export them, but I was trying to find if any other way exists.
>>>
>>> Thank you.
>>>
>>> On Wednesday, April 8, 2020 at 5:31:16 PM UTC+5:30, Ashwanth Kumar wrote:

 One thing I've learned working with GoCD for a while is, you don't want 
 to use absolute paths anywhere. Especially when agents are run across 
 different hosts or as ephemeral containers somewhere. 

 To get the current pipeline path one can use something like $(pwd) or 
 dirname etc.  and use relative paths from that location. 

 Another point is Tasks within a job don't share any session on a shell 
 of any kind among them. So if you export a variable and try to use it in 
 the next task or later within the job it wouldn't work. They will have to 
 be specified at the environment, pipeline, stage or at a job level. The 
 specificity goes in the reverse order. Also as a general convention I've 
 achieved best results when these scripts are NOT in GoCD tasks but instead 
 in a separate shell script file and added as part of the source code or 
 via 
 a separate ops repo material(s). 

 HTH. 

 On Wed, 8 Apr, 2020, 17:13 Shivani Shinde,  
 wrote:

> Hi everyone,
> I am having the hardest time dealing with variables for a task.
>
> What I want to achieve:
>
> 1. I want to pass a variable(pipeline's path) to the task which 
> generates builds. This is build command:
> make build path="/pipeline_path"
>
> I tried setting variable *pipelinepath*, in the environment variable 
> path, and using it in the task as specified in the document like:
> make build path=$pipelinepath
>
> or
>
> make build path="$pipelinepath"
>
> But it does not work!
>
> 2. I want to export certain variable assigning it a path of the 
> directory I git cloned. I have mentioned the git URL in Materials of 
> pipeline and mentioned destination directory as 'myProject'.
> I want to do the following:
> export myVar="/path/to/myProject"
>
>
> I want to achieve this at task level.
>
> How can I work on this? Any help will be appreciated!
>
> Thank you.
>
>
> CONFIDENTIALITY. This email and any attachments are confidential to 
> Alef Edge Inc., and may also be privileged, except where the email states 
> it can be disclosed. If this email is received in error, please do 
> not disclose the contents to anyone, notify the sender by 
> return email, and delete this email (and any attachments) from your 
> system.
>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to go...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/go-cd/083817de-2527-4e3e-a53f-36d4a1b407f4%40googlegroups.com
>  
> 
> .
>

>>> CONFIDENTIALITY. This email and any attachments are confidential to 
>>> Alef Edge Inc., and may also be privileged, except where the email states 
>>> it can be disclosed. If this email is received in error, please do not 
>>> disclose the contents to anyone, notify the sender by return email, and 
>>> delete this email (and any attachments) from your system.
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to go...@googlegroups.com .
>>> To view

Re: [go-cd] Need Help To Run Maven Builds Faster Setup on GoCD Kubernetes

[Jason Smyth]

Hello Mohit,

One option is to update the GoCD agent container image to include the local 
maven repository.

Another is to do what Ketan suggested and create a persistent volume that 
holds the local maven repo and gets mounted by the container at startup.

The specifics on how to do either of these will depend on your specific 
container implementation.

Regards,
Jason


On Wednesday, 25 March 2020 10:35:23 UTC-4, Mohit Garg wrote:
>
> Hi Ketan,
>
> Thanks for your quick response. I agree with your approach but problem is 
> that our GoCD agents created on the flyway as per demand. As soon as job 
> completed, container get killed and agent goes on stand by.
>
> Can you please suggest how can we solve this issue?
>
> On Wednesday, March 25, 2020 at 4:58:26 PM UTC+5:30, Ketan Padegaonkar 
> wrote:
>>
>> I've seen folks volume mounting the local maven/gradle repository in 
>> `~/.m2` or `~/.gradle` to get around this problem.
>>
>> - Ketan
>>
>>
>>
>> On Wed, Mar 25, 2020 at 3:37 PM 'Mohit Garg' via go-cd <
>> go...@googlegroups.com> wrote:
>>
>>> Hi GoCD Community,
>>>
>>> I am looking for some effective solution to fasten up our maven builds. 
>>> Each time a maven build is triggered, it downloads all dependencies present 
>>> in pom.xml which is taking too much time. Can someone please help me with 
>>> this?
>>>
>>> Regards,
>>> Mohit
>>>
>>> This message and its attachments are confidential (or legally 
>>> privileged) information and are meant solely for the addressee of such 
>>> message. Any unauthorized use of the message / its attachments is strictly 
>>> prohibited. 
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to go...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/go-cd/89763f47-21d1-4ba5-8796-141dba882a81%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/ae1d8a6f-d0e3-443a-a821-7332aef5fde1%40googlegroups.com.

Re: [go-cd] Unable to Run Pipeline After Updating Parameter-Based Material

[Jason Smyth]

Thank you for the reply, Aravind.

Although I could consistently reproduce this issue on Friday, as of just 
now I am unable to and the Pipeline runs normally when I change the 
parameter to support the current month.

I guess the issue resolved itself.

Cheers,
Jason


On Monday, 9 December 2019 08:58:40 UTC-5, Aravind SV wrote:
>
> Hello Jason, 
>
> It might be best to open an issue on GitHub with more snippets from the 
> log, or the whole log itself. 
>
> Cheers, 
> Aravind 
>
> On Fri, Dec 06, 2019 at 15:16:14 -0800, Jason Smyth wrote: 
> > Hello, 
> > 
> > We have a GoCD Pipeline that depends on a version-controlled zip archive 
> as 
> > one of its materials. To mitigate space issues caused by versioning a 
> large 
> > binary file, the zip archive is rotated monthly. The upstream process 
> that 
> > commits the archive timestamps it with -MM and every month we update 
> a 
> > Parameter on our GoCD Pipeline to ensure we pull the latest file. 
> > 
> > This has been working fine for the past 2+ years but this month GoCD 
> > started throwing errors as soon as we update the Parameter to the 
> current 
> > date. The error message we are seeing is: 
> > 
> > No row with the given identifier exists: 
> > [com.thoughtworks.go.domain.MaterialInstance#124725]; nested exception 
> is 
> > org.hibernate.ObjectNotFoundException: No row with the given identifier 
> > exists: [com.thoughtworks.go.domain.MaterialInstance#124725] 
> > 
> > As a result of this error: 
> > 
> >- GoCD fails when checking for a modification to the Material (error 
> as 
> >above). 
> >- GoCD fails to schedule the Pipeline (error as above). 
> >- GoCD fails to display Pipeline History in the UI 
> >
> > (http://gocd:8153/go/pipelineHistory.json?pipelineName= 
>
> >returns HTTP 500). 
> >- GoCD succeeds to display Pipeline History in the API 
> >(http://gocd:8153/go/api/pipelines//history works 
> as 
> >expected). 
> > 
> > In case it is relevant, the source control system is TFS. The Parameter 
> is 
> > used in the Material's Project Path ($/MyTeamProject/#{zipFile}) and 
> > Destination Directory (zipFile\#{zipFile}). 
> > 
> > As noted above, the errors start as soon as we change the date pattern 
> in 
> > the zipFile Parameter from 2019-11 to 2019-12. When we revert from 
> 2019-12 
> > back to 2019-11 they clear again. We have a test server that is a clone 
> of 
> > our production server and this Pipeline works fine in dev. 
> > 
> > Any thoughts on what we can do to resolve this issue? 
> > 
> > Any help is appreciated. 
> > 
> > Thank you, 
> > Jason 
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/274347da-7dd9-4979-b941-de8b4d3ebf87%40googlegroups.com.

[go-cd] Unable to Run Pipeline After Updating Parameter-Based Material

[Jason Smyth]

Hello,

We have a GoCD Pipeline that depends on a version-controlled zip archive as 
one of its materials. To mitigate space issues caused by versioning a large 
binary file, the zip archive is rotated monthly. The upstream process that 
commits the archive timestamps it with -MM and every month we update a 
Parameter on our GoCD Pipeline to ensure we pull the latest file.

This has been working fine for the past 2+ years but this month GoCD 
started throwing errors as soon as we update the Parameter to the current 
date. The error message we are seeing is:

No row with the given identifier exists: 
[com.thoughtworks.go.domain.MaterialInstance#124725]; nested exception is 
org.hibernate.ObjectNotFoundException: No row with the given identifier 
exists: [com.thoughtworks.go.domain.MaterialInstance#124725]

As a result of this error:

   - GoCD fails when checking for a modification to the Material (error as 
   above).
   - GoCD fails to schedule the Pipeline (error as above).
   - GoCD fails to display Pipeline History in the UI 
   (http://gocd:8153/go/pipelineHistory.json?pipelineName= 
   returns HTTP 500).
   - GoCD succeeds to display Pipeline History in the API 
   (http://gocd:8153/go/api/pipelines//history works as 
   expected).

In case it is relevant, the source control system is TFS. The Parameter is 
used in the Material's Project Path ($/MyTeamProject/#{zipFile}) and 
Destination Directory (zipFile\#{zipFile}).

As noted above, the errors start as soon as we change the date pattern in 
the zipFile Parameter from 2019-11 to 2019-12. When we revert from 2019-12 
back to 2019-11 they clear again. We have a test server that is a clone of 
our production server and this Pipeline works fine in dev.

Any thoughts on what we can do to resolve this issue?

Any help is appreciated.

Thank you,
Jason

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/eb069d28-0871-42f5-8cd6-db8da0113a0d%40googlegroups.com.

Re: [go-cd] Re: Substitutions in environment variable

[Jason Smyth]

Hi Maxim,

Maybe I am missing something but why do you want to do this?

Could you not reference the existing variable instead of creating a custom 
variable that contains the same data?

Regards,
Jason


On Monday, 30 September 2019 02:44:46 UTC-4, Maxim Shaev wrote:
>
> > Hey, are you running on windows or linux agent?
>
> Hey, It's Linux (Ubuntu 18.04 server) agent
>
> On Mon, Sep 30, 2019 at 5:02 AM Manuel Ortiz  > wrote:
>
>> Hey, are you running on windows or linux agent?
>>
>>
>>
>> El domingo, 29 de septiembre de 2019, 6:10:27 (UTC-6), Maxim Shaev 
>> escribió:
>>>
>>> Hi guys, Is it possible to define a custom environment variable based on 
>>> standard?
>>> e.g.
>>> ```
>>> 
>>>   
>>> ${GO_PIPELINE_LABEL}
>>>   
>>> ```
>>> That doesn't work as expected. The results:
>>> ```
>>> GO_PIPELINE_LABEL=dev-107
>>> APP_VERSION=${GO_PIPELINE_LABEL}
>>> ```
>>> If no, what's the best way to define that variable?
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/464f6d6b-8f40-4c34-9e8c-dffc7129cc75%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Best regards,
> Maxim Shaev
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/da5d61b6-390b-47d6-b4a7-ebd01499ac54%40googlegroups.com.

Re: [go-cd] GoCD Agent Failing to Clean Working Directory - "Unable to delete file"

[Jason Smyth]

Hi Ketan,

Thank you for the reply.

Using the sysinternals "handle" tool I was able to identify the process 
that opens the files (msbuild) but also confirmed that this process 
releases the files before the first Stage Job completes. Between when 
msbuild exits and the second Stage fails, handle shows no open handles for 
the a filter that includes a directory from the file that fails to delete.

I tried deleting the file via Windows Explorer and it also threw an error, 
but gave me the name of a process (.NET Core Host) that had the file locked.

Based on this it seems that what is happening is:


   1. Build phase (Stage 1) completes successfully and creates a .NET Core 
   Host that persists after the build completes.
   2. The test phase (Stage 2) somehow destroys the .NET Core Host that was 
   created in the build phase.

This behaviour seems both unintuitive and undesirable, but is obviously not 
a GoCD issue.

If anyone has any experience with building .NET Core applications on 
Windows that can shed some light on this it would be appreciated but 
otherwise we will continue our investigation off of this group.

Regards,
Jason

On Tuesday, 17 September 2019 01:17:06 UTC-4, Ketan Padegaonkar wrote:
>
> I'm assuming you're running on Windows. This can usually happen because of 
> open file handles. Could you use the sysinternals tools 
> <https://docs.microsoft.com/en-us/sysinternals/downloads/handle> to 
> identify processes that may be holding onto the files after the build 
> completes?
>
> - Ketan
>
>
>
> On Tue, Sep 17, 2019 at 2:21 AM Jason Smyth  > wrote:
>
>> Hello,
>>
>> We have several Pipelines that are failing during the "clean working 
>> directory" task of a Job. The only error message provided is "Unable to 
>> delete file".
>>
>> This just started recently but I believe that, until recently, these same 
>> Jobs did not have the "Clean working directory" attribute set so the issue 
>> has likely existed for some time and we are just discovering it now. The 
>> issue affects multiple Pipelines but all affected Pipelines follow the same 
>> pattern:
>>
>> Stage 1: 1 job
>> Stage 2: 2 jobs
>>
>> The issue only affects 1 Agent but this is the only Agent we have that is 
>> capable of running these jobs. The issue never seems to affect Stage 1 but 
>> what we see is that if Stage 1 completes and then Stage 2 for the same 
>> Pipeline starts immediately, both jobs fail with the same error message 
>> about being unable to delete a particular file.
>>
>> If we rerun Stage 2 a few minutes later, both jobs complete successfully. 
>> I have reviewed the Agent's log file and there is no additional information 
>> beyond the same "Unable to delete file" error message shown on the console.
>>
>> The file is inside GoCD's working directory and the GoCD user has full 
>> control of the entire directory tree so it should not be a permissions 
>> issue on the file. The file is a normal file, not a symlink that could lead 
>> outside the working directory to somewhere GoCD does not have access.
>>
>> How can we get additional information about why GoCD is failing to clean 
>> the working directory so that we can resolve the issue?
>>
>> At the moment we are working around the issue by manually re-running 
>> failed Stages but this is obviously far from ideal.
>>
>> Any insight here would be appreciated.
>>
>> Thank you,
>> Jason
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/go-cd/b060bcb6-7ac6-484a-8ae9-c1eb7136dcbd%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/go-cd/b060bcb6-7ac6-484a-8ae9-c1eb7136dcbd%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/ded8e7fa-0f52-41df-9cd3-4aed486231e4%40googlegroups.com.

[go-cd] GoCD Agent Failing to Clean Working Directory - "Unable to delete file"

[Jason Smyth]

Hello,

We have several Pipelines that are failing during the "clean working 
directory" task of a Job. The only error message provided is "Unable to 
delete file".

This just started recently but I believe that, until recently, these same 
Jobs did not have the "Clean working directory" attribute set so the issue 
has likely existed for some time and we are just discovering it now. The 
issue affects multiple Pipelines but all affected Pipelines follow the same 
pattern:

Stage 1: 1 job
Stage 2: 2 jobs

The issue only affects 1 Agent but this is the only Agent we have that is 
capable of running these jobs. The issue never seems to affect Stage 1 but 
what we see is that if Stage 1 completes and then Stage 2 for the same 
Pipeline starts immediately, both jobs fail with the same error message 
about being unable to delete a particular file.

If we rerun Stage 2 a few minutes later, both jobs complete successfully. I 
have reviewed the Agent's log file and there is no additional information 
beyond the same "Unable to delete file" error message shown on the console.

The file is inside GoCD's working directory and the GoCD user has full 
control of the entire directory tree so it should not be a permissions 
issue on the file. The file is a normal file, not a symlink that could lead 
outside the working directory to somewhere GoCD does not have access.

How can we get additional information about why GoCD is failing to clean 
the working directory so that we can resolve the issue?

At the moment we are working around the issue by manually re-running failed 
Stages but this is obviously far from ideal.

Any insight here would be appreciated.

Thank you,
Jason

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/b060bcb6-7ac6-484a-8ae9-c1eb7136dcbd%40googlegroups.com.

[go-cd] Re: Is it possible to limit an elastic agent profile to only allow one to run at a time?

[Jason Smyth]

We haven't started leveraging elastic agents yet so I can't comment on 
whether or not what you are looking to do is possible but if the 
restriction truly is for a single instance, why not use a single static 
agent instead of an elastic agent profile?

On Tuesday, 3 September 2019 09:43:55 UTC-4, Doug Lethin wrote:
>
> I have a use case where I would like to limit the number of parallel jobs 
> running on a particular elastic agent profile in our kubernetes cluster to 
> just one at a time.  If GoCD is currently executing a job with this elastic 
> again profile I would want all pending jobs associated with the elastic 
> agent to wait until the currently running agent finishes.  Is this current 
> possible?  If so, its not immediately obvious how to do it.I thought 
> maybe using kubernetes ResourceQuota's might solve the problem. I hadn't 
> tried that yet, but my guess is it wouldn't work causing the request to 
> spin up the next against to fail rather than wait.
>
> The use case I have is that the agent has a dependency on a fixed resource 
> that can't be shared.
>
> Thanks.
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/8720fded-59d2-4e29-afa3-5862a472a581%40googlegroups.com.

[go-cd] Re: External Artifact Propagation of Dependent Pipelines (aka Version Bumping)

[Jason Smyth]

Hi Anthony,

I think the only way to get GoCD to show these types of dependencies is to 
add Pipeline A as a Material for Pipeline B. This alone wouldn't handle 
automatically increasing the version of Library A in Application B but it 
would allow you to see all downstream applications that leverage Library A 
by looking at Pipeline A's VSM.

Depending on your library management solution you could potentially 
configure Application B to use the latest version of Library A instead of a 
specific version. I wouldn't recommend this, though, because you would lose 
the ability to track library version changes in Application B's source code 
and you would also lose the ability to recreate builds of Application B 
that were originally created with older versions of Library A.

A better solution for automating incrementing version might be to add a 
stage that does the version bump in Application B's source. This stage 
could exist either at the end of Pipeline A or as a completely separate 
Pipeline (C) whose sole purpose is to bump versions in applications that 
depend on Library A. You would need to maintain whatever process this new 
stage used to ensure it was aware of each new dependency that got added but 
I don't expect that would be a lot of work on top of manually adding the 
new downstream Pipelines.

With all of that said, I think the best solution might be the one you've 
already got. The existing implementation allows the maintainers of each 
downstream application to decide for themselves when they want to migrate 
to a new version of Library A. By extension, this means that nobody has to 
do any work to ensure that Application Q, which is working just fine in 
production and hasn't been updated in over a year, works with the latest 
version of Library A, which may include intentionally breaking changes.

I'm curious what issues you are encountering that are making you want to 
force library updates in otherwise unchanged applications.

This is just my opinion. I'm no expert but hopefully you find some of this 
helpful.

Cheers,
Jason


On Friday, 16 August 2019 13:59:11 UTC-4, Anthony Abate wrote:
>
> Hello
>
> I am curious what is the best practice is for the following scenario:
>
> Library A is used by Program B via an External Artifact Repository (
> nuget.org, npm, etc)
>
> It seems like a 'simple scenario involving 2 pipelines:
>
> Pipeline A (Library)
> Source -> Build -> Test -> Deploy Artifact
>
> Pipeline B (Program that use Library A)
> Source -> Build -> Test -> Deploy To Production
>
> In this example, there is no 'GoCD dependency' between pipeline A and B.  
> If A's source is changed, A.2 is built and eventually deployed to the 
> External Artifact Repo.   In order for program B to use it, source code is 
> edited (ie C# packageReference or Package.json) to link to the A.2.  A new 
> build B.2 is generated and if it passes, it is deployed.  The VSM for 
> Pipeline B.2 has no indication that Pipeline A.2 was involved / consumed.
>
> It feels like the Up / Down stream + Package Repo along with the VSM 
> capabilities of GoCD should be able to handle this scenario.  However, in 
> my experimentation, things break down due to fact that the build has to 
> modify a file under source control that tells Program B to switch from  
> Library A.1 to A.2.  Changing the file locally during the build stage seems 
> like cheating since this won't become part of the official source history.  
> (version control of the library references is very important)
>
> Like above the example above, all of my pipelines are 'silos'  If i want 
> to deploy a new version of the top most dependency, I have to do a lot of 
> source code changes to propagate a version bump downstream manually.  Now 
> granted this is extremely 'safe' since the change might be breaking - it 
> just can be very time consuming if there are 20 downstream projects.  If 
> this were automated I could in theory publish a new 'top-level' dependency 
> and have the entire downstream build process detect any breaking changes 
> via build or test failures.
>
> In addition automation, for any built (or deployed) artifacts, I would 
> want to see a VSM that shows the state of all upstream pipelines that were 
> involved / consumed so I can see at a glance what upstream labels are part 
> of a downstream artifacts.  
>
> 1. Does GoCD have a way of showing / managing these type of dependencies?
>
> 2. Does GoCD have a way to bump versions (change source code) references 
> to match upstream labels? Or is this burden on me?  
>
> 3. Are all upstream pipeline labels available as environment variables to 
> downstream pipelines?  
> (I could in theory create a parallel dependency pipeline - which does 
> nothing but bump versions. However, the further downstream, the wider the 
> scope of version bumping becomes as C depends on B and A, not just B 
> depends on A)
>
> In any case, was curious if some how the current functionality of GoCD 
> address

[go-cd] Re: Verify GoCD Version

[Jason Smyth]

Hi CaSac,

This sounds like an unusual situation but I believe the version reported in 
the GoCD UI should be correct. It almost seems as if GoCD was installed 
from an RPM and then updated without one. Or possibly there is an RPM 
version of GoCD and a non-RPM version installed on the same server.

Cheers,
Jason

On Tuesday, 30 July 2019 08:31:48 UTC-4, CaSac wrote:
>
> Dear, executing rpm -qa |grep go-server, the rpm that I found is 17.5
>
> rpm -qa | grep go
> *go-server-17.5.0*-5095.noarch
>
> While at the end of the UI I see:
>
> Go Version:* 18.2.0* (6228-fbb379a451871b29aa47aef907a78dc64379233f).
>
> What's the right version?
>
> Thank You
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/8461834c-79a9-47e0-95f8-9dfb86cd4960%40googlegroups.com.

[go-cd] Re: Associated gocd agent to a pipeline group

[Jason Smyth]

Hello,

To the best of my knowledge there is no way to link Pipeline Groups to 
Agents in GoCD. You would need to use either Environments or Resources to 
enforce the association between the individual Pipelines and the Agents you 
want them to run on.

I'm not sure why you would want to do this, though. It seems to me that 
there would be greater value in having all 5 Agents available to run any 
pipeline to reduce the chance of resource (Agent) contention when multiple 
Pipelines within a single Group need to run around the same time.

Regards,
Jason

On Thursday, 13 June 2019 01:15:02 UTC-4, Priyanka Makke wrote:
>
> I have 5 gocd agents connected to my gocd server.
>
> The use-case is I want to run a particular pipeline belonging to a 
> particular pipeline group to a specific go-agent every time.
>
> Example: all pipelines having pipeline-group-1 should run on the agent 
> agent-4.
>
> Can we achieve this using GOCD ?
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/aaf8fc68-f122-49d9-9192-dcfa32c35645%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-cd] Re: Agent is unable to communicate with server through load balancer

[Jason Smyth]

Hello,

I think this section of the stack trace you posted explains what is 
happening in your situation:

   - There is possibly a reverse proxy (or load balancer) that is 
> terminating SSL. Hint: use port 8154 of the GoCD server. See 
> https://docs.gocd.org/19.3.0/installation/configure-reverse-proxy.html#agents-and-reverse-proxies
>  
> 
>  for 
> details.


>From the GoCD documentation section linked in the stack strace:

 The GoCD server requires that the agents connect to it directly without 
> any reverse-proxies in between that perform SSL termination. This is 
> because GoCD agent-server communication is authenticated using SSL/TLS 
> client certificates, a reverse-proxy will be interpreted as a MITM 
> (man-in-the-middle-attack) and the agents will not be able to connect to 
> the server.


I suspect you will need to set up a separate AWS Load-Balancer that does 
not do SSL termination but rather forwards port 443 to port 8154 on the 
GoCD server.

Cheers,
Jason  
 

On Wednesday, 15 May 2019 09:40:30 UTC-4, Abhi Muthukumar wrote:
>
> Hi,
>
> We have a GoCd Server running behind an aws load-balancer.  Our setup goes 
> as below.
>
> Domain: abc.xyz.cloud
> Route53 record: abc.xyz.cloud --> Load balancer
> Load Balancer : any (HTTP) requests to abc.xyz.cloud at port 80 will be 
> forwarded to port 443 and any (HTTPS) requests to abc.xyz.cloud at 443 will 
> be forwarded to gocd-server-target-group
> Target Group: gocd-server-target-group uses HTTP protocol to hit our 
> gocd-server ec2-instance on port 8153 (as it is a HTTP request)
> GoCd Agents: There are 2 gocd agents running in private subnet and tries 
> to connect with server using the server url - https://abc.xyz.cloud/go
>
> Load Balancer takes care of stripping the certificate and makes HTTP 
> request to go server
>
> I am able to launch gocd server through browser and view my pipelines.  
> From agent, I'm able to curl server's health end point.  But I'm not able 
> to see the agents in UI and I see the blow error in go-agent log.
>
> 2019-05-15 13:32:25,120 ERROR [scheduler-2] AgentHTTPClientController:100 
> - Error occurred when agent tried to ping server:
> org.springframework.remoting.RemoteAccessException: Could not access HTTP 
> invoker remote service at [
> https://ci.receptiviti.cloud/go/remoting/remoteBuildRepository]; nested 
> exception is org.apache.http.client.ClientProtocolException: The server 
> returned status code 403. Possible reasons include:
>- This agent has been deleted from the configuration
>- This agent is pending approval
>- There is possibly a reverse proxy (or load balancer) that is 
> terminating SSL. Hint: use port 8154 of the GoCD server. See 
> https://docs.gocd.org/19.3.0/installation/configure-reverse-proxy.html#agents-and-reverse-proxies
>  
> for details.
> at 
> org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:226)
> at 
> org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:153)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at 
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
> at com.sun.proxy.$Proxy8.ping(Unknown Source)
> at 
> com.thoughtworks.go.agent.AgentHTTPClientController.ping(AgentHTTPClientController.java:95)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
> at 
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.http.client.ClientProtocolException: The server 
> returned status code 403. Possible reasons include:
>- This agent has been deleted from the configuration
>- This agent is pending

Re: [go-cd] Can a GoCD Agent's Token be Recovered?

[Jason Smyth]

Hi Ketan,

My mistake. I missed the fact that the command you provided included a 
substitution string that needed to be replaced with a value taken from my 
configuration file.

I ran the command again after making the substitution and we successfully 
reconnected the agent to our server.

Thank you again for your help.

Cheers,
Jason


On Monday, 22 April 2019 23:35:13 UTC-4, Ketan Padegaonkar wrote:
>
> Make sure your guid.txt file does not have a trailing newline/whitespace.
>
> - Ketan
>
>
>
> On Fri, Apr 19, 2019 at 12:46 AM Jason Smyth  > wrote:
>
>> Hi Ketan,
>>
>> Thank you for your reply.
>>
>> I tried what you suggested (including removing the new line from the end 
>> of the file) but we are still getting the same error.
>>
>> Any other suggestions?
>>
>> If it's relevant, running a script directly against a copy of the 
>> database is a viable option.
>>
>> Cheers,
>> Jason
>>
>>
>> On Thursday, 18 April 2019 12:02:41 UTC-4, Ketan Padegaonkar wrote:
>>>
>>> Try:
>>>
>>> cat guid.txt | openssl dgst -sha256 -hmac 
>>> "TOKEN_GENERATION_KEY_FROM_CONFIG_XML" -binary | base64 > token
>>>
>>> You may need to remove the newline from the generated `token` file.
>>>
>>> - Ketan
>>>
>>>
>>>
>>> On Thu, Apr 18, 2019 at 8:48 PM Jason Smyth  
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> For some unknown reason one of our agents re-registered with a new 
>>>> GUID. We would like to revert it to its original GUID so that its history 
>>>> is maintained but we do not have a backup copy of the token.
>>>>
>>>> We restored the original GUID but now see token errors in the log when 
>>>> we try to bring the agent online:
>>>>
>>>> A token has already been issued for this agent.
>>>>
>>>> Is there a way we can recover the original token from the server or do 
>>>> we need to just accept the loss of agent history?
>>>>
>>>> Thanks in advance,
>>>> Jason
>>>>
>>>> -- 
>>>> You received this message because you are subscribed to the Google 
>>>> Groups "go-cd" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>> an email to go...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-cd] Can a GoCD Agent's Token be Recovered?

[Jason Smyth]

Hi Ketan,

Thank you for your reply.

I tried what you suggested (including removing the new line from the end of 
the file) but we are still getting the same error.

Any other suggestions?

If it's relevant, running a script directly against a copy of the database 
is a viable option.

Cheers,
Jason


On Thursday, 18 April 2019 12:02:41 UTC-4, Ketan Padegaonkar wrote:
>
> Try:
>
> cat guid.txt | openssl dgst -sha256 -hmac 
> "TOKEN_GENERATION_KEY_FROM_CONFIG_XML" -binary | base64 > token
>
> You may need to remove the newline from the generated `token` file.
>
> - Ketan
>
>
>
> On Thu, Apr 18, 2019 at 8:48 PM Jason Smyth  > wrote:
>
>> Hello,
>>
>> For some unknown reason one of our agents re-registered with a new GUID. 
>> We would like to revert it to its original GUID so that its history is 
>> maintained but we do not have a backup copy of the token.
>>
>> We restored the original GUID but now see token errors in the log when we 
>> try to bring the agent online:
>>
>> A token has already been issued for this agent.
>>
>> Is there a way we can recover the original token from the server or do we 
>> need to just accept the loss of agent history?
>>
>> Thanks in advance,
>> Jason
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to go...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-cd] Can a GoCD Agent's Token be Recovered?

[Jason Smyth]

Hello,

For some unknown reason one of our agents re-registered with a new GUID. We 
would like to revert it to its original GUID so that its history is 
maintained but we do not have a backup copy of the token.

We restored the original GUID but now see token errors in the log when we 
try to bring the agent online:

A token has already been issued for this agent.

Is there a way we can recover the original token from the server or do we 
need to just accept the loss of agent history?

Thanks in advance,
Jason

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-cd] Re: Choose Bundled Java Bitness for Chocolatey GoCD Agent Install

[Jason Smyth]

Thank you both for your feedback. The --x86 switch was what we are looking 
for.

Now I just need to figure out the correct syntax to pass arguments in via 
--ia and we will have our first step towards a more automated process for 
setting up our static agents.

Cheers,
Jason

On Wednesday, 10 April 2019 21:18:58 UTC-4, Aravind SV wrote:
>
> On Wed, Apr 10, 2019 at 15:04:33 -0700, Varsha Varadarajan wrote: 
> > The gocd-chocolatey repo  
> hasn't 
> > been updated to reflect that gocd no longer has the 32 bit installer. 
> With 
> > chocolatey, you can use the --x86 flag to force 32 bit installations. 
> But I 
> > don't think that'll work on versions after 18.11.0. But, haven't been 
> able 
> > to verify that. 
>
> Yes, for version 18.11: choco install gocdagent --version 18.11.0 --x86 
>
> ... should work fine. 
>
> Beyond that, anyway, there are no 32-bit installers, as Jason said. 
>
> Cheers, 
> Aravind 
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-cd] Re: Choose Bundled Java Bitness for Chocolatey GoCD Agent Install

[Jason Smyth]

Hi Varsha,

Thank you for your reply.

What you say makes sense, however we were hoping to continue using the 
bundled Java.

In looking at the Chocolatey package for 18.11 
<https://chocolatey.org/packages/gocdagent/18.11.0> I can see that it 
includes both the 32- and 64-bit setup executable files. What's not clear, 
however, is how Chocolatey determines which setup file to run when the 
package is installed.

Is there an argument that can be passed to specify or is it automatically 
chosen based on either the OS or the parent caller?

Regards,
Jason


On Wednesday, 10 April 2019 16:01:03 UTC-4, Varsha Varadarajan wrote:
>
> Hi,
>
> If you have java installed on the nodes already, you can specify 
> `GO_AGENT_JAVA_HOME` as an argument during installation. This will override 
> the default bundled jre and use the one you specify. The gocd agent 
> chocolatey package supports the same args as specified in the silent 
> installation section in the docs - 
> https://docs.gocd.org/19.3.0/installation/install/agent/windows.html#silent-headless-installation
> . 
>
> - Varsha
>
> On Wednesday, 10 April 2019 12:04:07 UTC-7, Jason Smyth wrote:
>>
>> Hello,
>>
>> We are looking at the possibility of moving to Chocolatey as an 
>> installation source for our Windows GoCD agents. Currently we are running 
>> 32-bit Java on all of our servers and would like to preserve this.
>>
>> Is there a way we can specify the bitness of Java when installing the 
>> GoCD agent package from Chocolatey? Since GoCD does not ship with 32-bit 
>> Java as of 18.12 we would of course need to install 18.11 and have the 
>> agents update themselves from that.
>>
>> Thank you for your time,
>> Jason
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-cd] Choose Bundled Java Bitness for Chocolatey GoCD Agent Install

[Jason Smyth]

Hello,

We are looking at the possibility of moving to Chocolatey as an 
installation source for our Windows GoCD agents. Currently we are running 
32-bit Java on all of our servers and would like to preserve this.

Is there a way we can specify the bitness of Java when installing the GoCD 
agent package from Chocolatey? Since GoCD does not ship with 32-bit Java as 
of 18.12 we would of course need to install 18.11 and have the agents 
update themselves from that.

Thank you for your time,
Jason

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-cd] How to Show Relative Time on Dashboard?

[Jason Smyth]

We recently updated from GoCD v16.11 to 19.01. With this update one of the 
changes is that the trigger time shown on the Dashboard (pipelines) page is 
now absolute (e.g. "05 Apr, 2019 at 10:35:28 Local Time") rather than 
relative (e.g. "4 days ago").

Feedback from our users is that the question they usually want answered is 
"How long ago did this happen?" so they would prefer to see time elapsed 
rather than a timestamp.

Is there a configuration option that will allow us to have elapsed time 
displayed on our dashboard page?

Thank you for your time,
Jason

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.