Re: Multiple pipelines in Jenkinsfile

[Mike Rooney]

I have not found a solution yet but I'd love to know if there is one.
Current best practice seems to be to have a Jenkinsfile which triggers Job
DSL to process DSL files located in a subdirectory, so at least they are
all organized in your repo and Jenkinsfile serves as the single point of
entry. You can still have Jenkinsfile perform your "primary build" natively
in this case, just add a Job DSL stage somewhere in it.

On Thu, Mar 23, 2017 at 5:32 PM, Joshua Noble  wrote:

> Has anyone been able to make any progress on this?
>
> I have a new Jenkins 2 cluster up and running, and I've created
> declarative Jenkinsfile pipelines for each app repo. This works excellent
> for building all app branches and PR's. However, we have some more generic
> parameterized deploy jobs that we'd like to add. The Github Organization
> plugin is also currently used for automatically scanning all of our repos,
> and when Jenkinsfile's are found, jobs are automatically built - but under
> the org/repo/branch namespace.
>
> We have a generic jenkins repo that hosts some legacy Jenkins Job DSL
> plugin scripts. Ideally, we'd like to add a Jenkinsfile to this repo, and
> have that job define these generic deploy and utility jobs. (perhaps via
> groovy loads) I would prefer to use the newer Jenkinsfile-based syntax -
> declarative or scripted is fine. I am hoping that I don't have to resort
> back to the Jenkins Jobs DSL plugin though.
>
> Is there any simple way to accomplish this?
>
> On Monday, October 3, 2016 at 5:40:49 AM UTC-4, Sorin Ionuț Sbârnea wrote:
>>
>> Did you finish your implementation? Groovy is far from being my native
>> language and it would be really helpful if you could share a snippet that
>> is doing the subdirectory Jenkinsfile discovery and loading.
>>
>> On Thursday, July 21, 2016 at 1:43:18 AM UTC+1, slide wrote:
>>>
>>> The way I am planning on doing this is with the findFiles and load
>>> functions. I'll use findFiles in my Jenkinsfile to look for other build
>>> files further in the repo and create jobs from those to run. The other
>>> files will not necessarily be the same setup as a Jenkinsfile, but will use
>>> the pipeline syntax.
>>>

 --
> You received this message because you are subscribed to a topic in the
> Google Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/jenkinsci-users/nQ6p7xw5jqI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> jenkinsci-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/jenkinsci-users/a219dde2-d90c-4ab9-a738-51ab6487a800%40googlegroups.
> com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CADya-sbapk1zxP-CgYvZz5gXumTtwzf3g2EL-74hFtt%2BigfuQQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: git included/excluded regions not working with pipeline jobs

[Mike Rooney]

Ah, thanks for the pointer Mark. I actually found that one but assumed it
was specific to multibranch. I updated it a bit to be more generic :)

On Thu, Jul 21, 2016 at 6:08 PM, Mark Waite 
wrote:

> There is a bug report noting that pipeline jobs don't honor the ignore
> paths.
>
> https://issues.jenkins-ci.org/browse/JENKINS-36836
>
> Mark Waite
>
> On Thu, Jul 21, 2016 at 2:16 PM Mike Rooney  wrote:
>
>> Hello there!
>>
>> We've got some Pipeline jobs that are using "Polling ignores commits in
>> certain paths", plus the required "Force polling using workspace" option.
>>
>> It looks like we've got this set up correctly based on documentation and
>> seeing this in the polling log: "Ignored commit
>> 742b406b1d31bc04cff22c463561b381af0377fc: Found only excluded paths: "
>> However, then it seems to build it anyway. I'm wondering if this has
>> something to do with how Pipeline jobs don't seem to have a direct
>> workspace for the whole thing?
>>
>> Any tips or thoughts? Here's the full polling log:
>>
>> Polling for changes in
>> > git rev-parse origin/master^{commit} # timeout=10
>> > git log --full-history --no-abbrev --format=raw -M -m --raw
>> acde0988ce12d03c232baab96b0ca0fcd5c268fb..742b406b1d31bc04cff22c463561b381af0377fc
>> # timeout=10
>> Ignored commit 742b406b1d31bc04cff22c463561b381af0377fc: Found only
>> excluded paths:
>> Using strategy: Default
>> [poll] Last Built Revision: Revision
>> acde0988ce12d03c232baab96b0ca0fcd5c268fb (origin/master)
>> using .gitcredentials to set credentials
>> > git --version # timeout=10
>> > git init /var/cache/tomcat8/temp/hudson7613875610483359225tmp #
>> timeout=10
>> > git config --local credential.username redacted # timeout=10
>> > git config --local credential.helper store
>> --file=/var/cache/tomcat8/temp/git7992680616771925108.credentials #
>> timeout=10
>> > git -c core.askpass=true ls-remote -h
>> https://github.com/redacted/redacted.git # timeout=10
>> > git config --local --remove-section credential # timeout=10
>> Found 10 remote heads on https://github.com/redacted/redacted.git
>> [poll] Latest remote head revision on refs/heads/master is:
>> 742b406b1d31bc04cff22c463561b381af0377fc
>> Done. Took 1.6 sec
>> Changes found
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Jenkins Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to jenkinsci-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/jenkinsci-users/adcfd101-c025-4be9-ac25-493449476ec4%40googlegroups.com
>> <https://groups.google.com/d/msgid/jenkinsci-users/adcfd101-c025-4be9-ac25-493449476ec4%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jenkinsci-users/wNIMGhJ76rA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> jenkinsci-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtETNKH3-daX3FbgHEwmX5Zsy9%2BYtvgtNpARNa2enXXQHA%40mail.gmail.com
> <https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtETNKH3-daX3FbgHEwmX5Zsy9%2BYtvgtNpARNa2enXXQHA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CADya-sa%3DtQPTeR%2BOPJ1G_SG7OKq5HXgG-cE-sdC3PnVH7ZddiA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Multiple pipelines in Jenkinsfile

[Mike Rooney]

On Thu, Jul 21, 2016 at 11:42 AM, alex kessinger 
wrote:

> I've tried the seed job-dsl method previously. Some automation was better
> then no automation, but I think the Jenkinsfile in repo is even better. If
> I make a change to the Jenkinsfile, that change can be isolated in each
> environment/branch until it has been promoted to next step. If I have one
> master seed job it's harder for me to control the promotion between
> environments.
>

I agree, being able to have multiple Jenkinsfiles or specify multiple jobs
in one definitely feels like the right and superior option. I suppose you
could work around it by having your Jenkinsfile only run on master (either
via an option in the job or by checking env.BRANCH_NAME in the
Jenkinsfile), and then that Jenkinsfile runs job-dsl, which creates
Pipeline jobs that run for every branch/PR as desired. But, it will
probably be organizationally inferior and everyone will be having to
reinvent some logic to create and organize these sub-jobs that should be
maintained in one place.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CADya-sZwfJSZuOL1cJRz7Zv%2BMyU2-5%2BmG7QmBF%3DzYzPGsitwuw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

git included/excluded regions not working with pipeline jobs

[Mike Rooney]

Hello there!

We've got some Pipeline jobs that are using "Polling ignores commits in 
certain paths", plus the required "Force polling using workspace" option.

It looks like we've got this set up correctly based on documentation and 
seeing this in the polling log: "Ignored commit 
742b406b1d31bc04cff22c463561b381af0377fc: Found only excluded paths: " 
However, then it seems to build it anyway. I'm wondering if this has 
something to do with how Pipeline jobs don't seem to have a direct 
workspace for the whole thing?

Any tips or thoughts? Here's the full polling log:

Polling for changes in
> git rev-parse origin/master^{commit} # timeout=10
> git log --full-history --no-abbrev --format=raw -M -m --raw 
acde0988ce12d03c232baab96b0ca0fcd5c268fb..742b406b1d31bc04cff22c463561b381af0377fc
 
# timeout=10
Ignored commit 742b406b1d31bc04cff22c463561b381af0377fc: Found only 
excluded paths: 
Using strategy: Default
[poll] Last Built Revision: Revision 
acde0988ce12d03c232baab96b0ca0fcd5c268fb (origin/master)
using .gitcredentials to set credentials
> git --version # timeout=10
> git init /var/cache/tomcat8/temp/hudson7613875610483359225tmp # timeout=10
> git config --local credential.username redacted # timeout=10
> git config --local credential.helper store 
--file=/var/cache/tomcat8/temp/git7992680616771925108.credentials # 
timeout=10
> git -c core.askpass=true ls-remote -h 
https://github.com/redacted/redacted.git # timeout=10
> git config --local --remove-section credential # timeout=10
Found 10 remote heads on https://github.com/redacted/redacted.git
[poll] Latest remote head revision on refs/heads/master is: 
742b406b1d31bc04cff22c463561b381af0377fc
Done. Took 1.6 sec
Changes found

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/adcfd101-c025-4be9-ac25-493449476ec4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Multiple pipelines in Jenkinsfile

[Mike Rooney]

This need makes a lot of sense to us, where we have a couple related 
sub-projects (as sub directories) in a single repository. It makes sense 
that they each have their own pipeline jobs and can run on different 
schedules. I've also seen cases similar to Alex's (hi Alex!) where there 
are different tasks you want to do with a single repo that don't make sense 
as one pipeline job that runs together (building/testing versus a nightly 
cron-type task that runs in the repo).

It is reasonable that a Jenkinsfile corresponds to a single Pipeline job, 
because these are often associated with and run via a Pipeline job which 
isn't a logical "parent" of these seed jobs. However, a great place for 
this enhancement would be the Github Org / Bitbucket plugins that scan 
repositories for Jenkinsfiles and are already in the place of creating 
multiple Pipeline jobs.

My proposal would be: add a configuration option for the Github and 
Bitbucket plugins which scan organizations for Jenkinsfiles. So, "Project 
Recognizers -> Pipeline Jenkinsfile" would get a box for this which 
defaults to "Jenkinsfile". Some logical configuration examples might be, 
"Jenkinsfiles/*", "**/Jenkinsfile", "Jenkinsfile-*". Then the 
Github/Bitbucket plugins can be pointed at an org, or just one repository, 
and multiple Jenkinsfiles can exist which define different Pipeline jobs.

Bartłomiej and Alex, would something like this satisfy your use cases as 
well?

- Michael

On Sunday, May 29, 2016 at 12:47:40 PM UTC-5, Bartłomiej Sacharski wrote:
>
> I'm really hyped about the Jenkinsfiles - they make it much much easier to 
> document and preserve project configuration.
> However, all the examples that I've seen seem to use single pipeline.
> I've tried to define different stages in separate node blocks, however 
> they still were seen as a single pipeline.
>
> Is it possible to define multiple pipelines in a single Jenkinsfile? Or 
> maybe there's undocumented functionality for .jenkinsfile extension to 
> handle such cases?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/41de15f2-5341-4a62-aac3-6d3040c7119b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

getting only an NPE when building a Jenkinsfile job, how to debug?

[Mike Rooney]

Hello there! We're using Jenkins 2.9 with the Bitbucket Branch Source 
Plugin 1.5. It picks up on a branch with a Jenkinsfile which is neat, but 
the only build output we get is:

Started by user m a
java.lang.NullPointerException
at org.eclipse.jgit.lib.ObjectId.fromString(ObjectId.java:231)
at 
jenkins.plugins.git.AbstractGitSCMSource$SpecificRevisionBuildChooser.(AbstractGitSCMSource.java:388)
at 
com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.build(BitbucketSCMSource.java:411)
at 
org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:78)
at 
org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:206)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Finished: FAILURE


The Jenkinsfile looks like this:
node {
  stage "Checkout Git"
  checkout scm

  stage "Fake build"
  echo "My branch is: ${env.BRANCH_NAME}"
  sh "ls"
}


I'm at a loss of how to debug this. I've tried changing the Jenkinsfile and 
pushing, but it isn't clear that Jenkins is using that new Jenkinsfile, so 
I'm not sure how to see which Jenkinsfile it uses (or does it always grab 
it from the repo for each run?) This error doesn't seem to provide any 
context so I'm not sure if this is coming from a stage in my Jenkinsfile or 
somewhere else earlier. Any tips? For what it is worth, here is the 
sanitized Jenkins log:

Jun 16, 2016 4:53:48 PM 
com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient 
getHttpClient

INFO: Jenkins proxy: XXX

Jun 16, 2016 4:53:48 PM 
com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient 
getHttpClient

INFO: Using proxy authentication (user=XXX)

Jun 16, 2016 4:53:48 PM 
com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient 
getHttpClient

INFO: Jenkins proxy: XXX

Jun 16, 2016 4:53:48 PM 
com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient 
getHttpClient

INFO: Using proxy authentication (user=XXX)

Jun 16, 2016 4:53:49 PM org.jenkinsci.plugins.workflow.job.WorkflowRun 
finish

INFO: stash-mps/XXX/feature%2FCM-10898-android-use-a-jenkinsfile-to-test #8 
completed: FAILURE


-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/55fe6eb9-1f9e-4965-ac5a-db28738748a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Getting One Job Invocation Per Git Commit

[Mike Rooney]

A late reply, but as this thread appears on Google for trying to solve this 
problem, I figured I'd post my solution. It isn't great as you can't use 
the Git plugin, so a new BuildChooser would be superior, but it does result 
in one build per git commit across all branches.

First, modify your job to echo the last built commit to somewhere in the 
workspace (such as last_built_commit), or preferably throw it in S3 or 
something more persistent.

Now, use the ScriptTrigger plugin and create a script to see if the 
last_built_commit is still the latest, exiting 0 if not. Here's my example:

LAST_BUILT=`cat $WORKSPACE/last_built_commit` # or curl from S3 or wherever
> NEXT_COMMIT=`cd $WORKSPACE/repo && git rev-list --all --remotes --reverse 
> -n 100 | grep -A 1 $LAST_BUILT | tail -n 1`
> echo $NEXT_COMMIT > $WORKSPACE/next_build_commit
> # If the last built commit is not the newest, exit 0, meaning we need a 
> new build.
> [[ $LAST_BUILT != $NEXT_COMMIT ]]


Finally, handle git clone / pulling yourself in a build step: "if [ ! -e 
repo ]; then git clone repo.git; fi; cd repo && git fetch --all && git 
checkout `cat $WORKSPACE/next_build_commit`"

Ta-da! As far as I can tell, this works well, since a Jenkins restart or 
job failure will result in another attempt at that commit. Depending on 
where you write your last_built_commit you can control whether it will 
retry commits or skip failures.

I'd definitely love to see a BuildChooser that handled this, including for 
a branch specifier of **; that would be quite wonderful. However for now I 
figured this solution might help others, or maybe someone will point out a 
better way for me :)

- Mike

On Friday, July 8, 2011 3:24:59 PM UTC-4, Randall R Schulz wrote:
>
> Hi,
>
> I'm using Jenkins to drive automated tests triggered by Git commits. 
> Because of the duration of the tests and the frequency of commits (at 
> certain times of the day), often multiple commits occur while the tests 
> from an earlier commit are running.
>
> I need to perform a full test run for each commit without this "batching" 
> or "clumping" of multiple commits arriving during a test run.
>
> How might I achieve that?
>
>
> Randall Schulz
>