Re: Storing job configurations with source code in VCS

[Patrick van der Velde]

That sounds like a promising approach. I will go investigate that

On Saturday, 9 April 2016 01:16:58 UTC+12, Martin d'Anjou wrote:
>
> You could look at the Simple Build For Pipeline Plugin 
> , combined 
> with some TBD way of automatically creating job configurations (perhaps 
> something like the BitBucket Branch Source Plugin 
> ).
>  
> The Simple Build For Pipeline has a base class that can be expanded to 
> support a custom mini-DSL code job declaration, and implement a Pipeline 
> DSL flow inside a plugin of your own writing. With that plugin, users are 
> presented with a high level declarative view of their flow, so they can 
> turn knobs on their flow. This declarative view could be stored in their 
> source code repo (git) as a "Jenkinsfile". The BitBucket Branch Source 
> Plugin can scan BitBucket repositories and create jobs for projects that 
> contain a special file literally named the "Jenkinsfile" (optionally one 
> job per branch, even one job per pull-request if you wish). This 
> Jenkinsfile could contain the call to the extension of Simple Build. The 
> extension of Simple Build has to be a plugin that is uploaded to Jenkins. 
> It provides many advantages: it can be unit tested as a Jenkins plugin on 
> the command line (mvn test), and the flow becomes available as global 
> variable in Jenkins (thus making it available to any job configuration in 
> Jenkins). Conceptually, you could have multiple custom Simple Build 
> extensions (each one as a different plugin of your own, or many in the same 
> plugin of your own): one per type of job you need to run.
>
> Conceptually, that is the closest I have come to being able to test the 
> job configurations before I deploy them, the closest I have come to give 
> users the ability tweak their build parameters in their own source code 
> without making everyone an administrator. But I am still lacking the 
> ability to create multiple jobs per project (i.e. multiple Jenkins jobs for 
> each user (git) repository). Unfortunately, there can only be one 
> Jenkingfile per branch, I wish I could have many since I don't want my 
> pipeline all in the same job config.
>
> Conceptually, if the Simple Build plugin extension is kept backwards 
> compatible (which should be doable with good enough unit tests on the 
> custom plugin), rerunning older builds, even years later, should be 
> possible.
>
> It is still a work in progress for me.
>
> Hope this helps,
> Martin
>
> On Wednesday, March 9, 2016 at 11:04:13 PM UTC-5, Patrick van der Velde 
> wrote:
>>
>> Hi All
>>
>> We are trying to improve our build server setup and are trying to solve 
>> the problem that users can change the configuration of their builds through 
>> the UI, but those changes are not stored in source control, so it is hard 
>> to determine what has changed and who made those changes. Additionally (and 
>> maybe even more importantly) it is hard to rebuild an old version of the 
>> software because that may require a different job configuration. I know 
>> there are several plugins to handle backing up configuration changes to VCS 
>> / etc. but those backup to their own repositories (if they work) and we 
>> want the configuration for a build job to be stored with the source code 
>> for that build job, not in a separate repository. Because if we store the 
>> definition for the build job and the source code in the same location we 
>> could easily recreate a build environment as it was for that specific 
>> commit.
>>
>> One option would be to use the Job-DSL plugin to create the jobs from a 
>> configuration file that is in VCS. The only catch with that would be that 
>> editing the configuration would require users to update the configuration 
>> text file which I suspect would increase the chance of errors and also 
>> makes it harder for the users to test configuration changes. Ideally we 
>> would like to keep the UI as the way to edit configurations but then just 
>> have that write to the configuration file.
>>
>> I suspect this problem cannot be solved with our current version of 
>> jenkins (latest LTS 1.642.2) but I'm wondering if there are any plugins / 
>> combination of plugins / other apps that could make this possible?
>>
>> thanks
>>
>> Petrik
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/538efdfd-a2aa-4a0b-b92f-0746b83466bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Storing job configurations with source code in VCS

[Martin d'Anjou]

You could look at the Simple Build For Pipeline Plugin 
, combined 
with some TBD way of automatically creating job configurations (perhaps 
something like the BitBucket Branch Source Plugin 
). 
The Simple Build For Pipeline has a base class that can be expanded to 
support a custom mini-DSL code job declaration, and implement a Pipeline 
DSL flow inside a plugin of your own writing. With that plugin, users are 
presented with a high level declarative view of their flow, so they can 
turn knobs on their flow. This declarative view could be stored in their 
source code repo (git) as a "Jenkinsfile". The BitBucket Branch Source 
Plugin can scan BitBucket repositories and create jobs for projects that 
contain a special file literally named the "Jenkinsfile" (optionally one 
job per branch, even one job per pull-request if you wish). This 
Jenkinsfile could contain the call to the extension of Simple Build. The 
extension of Simple Build has to be a plugin that is uploaded to Jenkins. 
It provides many advantages: it can be unit tested as a Jenkins plugin on 
the command line (mvn test), and the flow becomes available as global 
variable in Jenkins (thus making it available to any job configuration in 
Jenkins). Conceptually, you could have multiple custom Simple Build 
extensions (each one as a different plugin of your own, or many in the same 
plugin of your own): one per type of job you need to run.

Conceptually, that is the closest I have come to being able to test the job 
configurations before I deploy them, the closest I have come to give users 
the ability tweak their build parameters in their own source code without 
making everyone an administrator. But I am still lacking the ability to 
create multiple jobs per project (i.e. multiple Jenkins jobs for each user 
(git) repository). Unfortunately, there can only be one Jenkingfile per 
branch, I wish I could have many since I don't want my pipeline all in the 
same job config.

Conceptually, if the Simple Build plugin extension is kept backwards 
compatible (which should be doable with good enough unit tests on the 
custom plugin), rerunning older builds, even years later, should be 
possible.

It is still a work in progress for me.

Hope this helps,
Martin

On Wednesday, March 9, 2016 at 11:04:13 PM UTC-5, Patrick van der Velde 
wrote:
>
> Hi All
>
> We are trying to improve our build server setup and are trying to solve 
> the problem that users can change the configuration of their builds through 
> the UI, but those changes are not stored in source control, so it is hard 
> to determine what has changed and who made those changes. Additionally (and 
> maybe even more importantly) it is hard to rebuild an old version of the 
> software because that may require a different job configuration. I know 
> there are several plugins to handle backing up configuration changes to VCS 
> / etc. but those backup to their own repositories (if they work) and we 
> want the configuration for a build job to be stored with the source code 
> for that build job, not in a separate repository. Because if we store the 
> definition for the build job and the source code in the same location we 
> could easily recreate a build environment as it was for that specific 
> commit.
>
> One option would be to use the Job-DSL plugin to create the jobs from a 
> configuration file that is in VCS. The only catch with that would be that 
> editing the configuration would require users to update the configuration 
> text file which I suspect would increase the chance of errors and also 
> makes it harder for the users to test configuration changes. Ideally we 
> would like to keep the UI as the way to edit configurations but then just 
> have that write to the configuration file.
>
> I suspect this problem cannot be solved with our current version of 
> jenkins (latest LTS 1.642.2) but I'm wondering if there are any plugins / 
> combination of plugins / other apps that could make this possible?
>
> thanks
>
> Petrik
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/7b2284dd-84de-4cc6-9b82-7ab96d1d7684%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Storing job configurations with source code in VCS

[Craig Rodrigues]

Hi,

Take a look at the SCM Sync configuration plugin
https://wiki.jenkins-ci.org/display/JENKINS/SCM+Sync+configuration+plugin
It allows users to modify their job configuration, but keeps track of the
changes in Git or Subversion.

--
Craig

On Wed, Mar 9, 2016 at 8:04 PM, Patrick van der Velde <
petrikvanderve...@gmail.com> wrote:

> Hi All
>
> We are trying to improve our build server setup and are trying to solve
> the problem that users can change the configuration of their builds through
> the UI, but those changes are not stored in source control, so it is hard to
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CAG%3DrPVd2sDUZB_K5a6dBGTNLQoEz1%2BC6B5t6uLM4571XDErJEw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Storing job configurations with source code in VCS

[Tom Moore]

On Tuesday, March 22, 2016 at 3:25:17 PM UTC-4, Patrick van der Velde wrote:
>
> Hi Victor
>
> Well that's a good start. Being able to verify that the configuration 
> works is good. However I would really prefer having some kind of UI to 
> create the configurations to make it easier for users to edit their 
> configurations. I'm guessing there are no plug-ins that allow me to achieve 
> this.
>
> Thanks
>
> We do a variant of this where our makefiles contain a build timestamp 
which is needed by our application and we want it captured so if we ever 
rebuild, we can exactly reproduce the binary (sha1 match on binary)

1) Setup your build job as normal, except do NOT use the delete workspace 
before building option.  If you do, this strategy wont work.
2) Run the build once to initialize your workspace to the scm you are 
connecting to.
3) Update your build job to have a pre-scm step that compares the job's 
config.xml with a copy that exists in your workspace repository:
   If the config.xml does not exist in the workspace, copy it local and 
commit it to your repository.
   If the config,xml exists in the workspace, and is different, then copy 
it local and commit the changes to the repository.
   If the config.xml exists and is the same as the workspace, do nothing.

After adding the pre-scm step, your config.xml will be evaluated and saved 
to the repository if modified.  Since this happens before the SCM step, 
then  the revision that actually gets built will contain the correct 
config.xml for the build.


-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/34b8a930-1934-4a12-b3ab-6addfdb65ef1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Storing job configurations with source code in VCS

[Patrick van der Velde]

Hi Victor

Well that's a good start. Being able to verify that the configuration works 
is good. However I would really prefer having some kind of UI to create the 
configurations to make it easier for users to edit their configurations. 
I'm guessing there are no plug-ins that allow me to achieve this.

Thanks


On Thursday, 10 March 2016 20:02:31 UTC+13, Victor Martinez wrote:
>
> Hi,
> You can easily add some tests to verify your jobDSL jobs locally before 
> committing/pushing them.
>
> Look at the below entry 
> https://groups.google.com/forum/m/#!topic/job-dsl-plugin/xBOxlCMc6Qg
>
> What I also do it is running a local jenkins docker container and apply 
> those changes as the final validation 
>
> I hope it helps
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/b7129d2a-6367-4885-a9b4-a9fe5843c259%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Storing job configurations with source code in VCS

[Victor Martinez]

Hi,
You can easily add some tests to verify your jobDSL jobs locally before 
committing/pushing them.

Look at the below entry 
https://groups.google.com/forum/m/#!topic/job-dsl-plugin/xBOxlCMc6Qg

What I also do it is running a local jenkins docker container and apply those 
changes as the final validation 

I hope it helps

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/98b15160-4554-44e0-9946-5809917d1594%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Storing job configurations with source code in VCS

[Patrick van der Velde]

Hi All

We are trying to improve our build server setup and are trying to solve the 
problem that users can change the configuration of their builds through the 
UI, but those changes are not stored in source control, so it is hard to 
determine what has changed and who made those changes. Additionally (and 
maybe even more importantly) it is hard to rebuild an old version of the 
software because that may require a different job configuration. I know 
there are several plugins to handle backing up configuration changes to VCS 
/ etc. but those backup to their own repositories (if they work) and we 
want the configuration for a build job to be stored with the source code 
for that build job, not in a separate repository. Because if we store the 
definition for the build job and the source code in the same location we 
could easily recreate a build environment as it was for that specific 
commit.

One option would be to use the Job-DSL plugin to create the jobs from a 
configuration file that is in VCS. The only catch with that would be that 
editing the configuration would require users to update the configuration 
text file which I suspect would increase the chance of errors and also 
makes it harder for the users to test configuration changes. Ideally we 
would like to keep the UI as the way to edit configurations but then just 
have that write to the configuration file.

I suspect this problem cannot be solved with our current version of jenkins 
(latest LTS 1.642.2) but I'm wondering if there are any plugins / 
combination of plugins / other apps that could make this possible?

thanks

Petrik

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/50ec9175-61a6-4189-8a94-71c082c89a8c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.