Re: Is Jenkins the right tool for hundreds of repositories

[Jonathan Rogers]

Chris Marks wrote:
>
> I haven't used Jenkins in a large Git environment, but perhaps those
> who have problems with the performance could file bugs related to
> their experiences in specific situations if they haven't already?  It
> would benefit the community to have a bullet-proof Git plugin that
> performed well at scale.
>

The fact that this hasn't gotten any attention in three years is what
convinced me to dump the git plugin:
https://issues.jenkins-ci.org/browse/JENKINS-5724
>

-- 
Jonathan Rogers

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is Jenkins the right tool for hundreds of repositories

[Jonathan Rogers]

William Soula wrote:
> Look into the github trigger where pushes to a repo causes the webhook to hit 
> the url for the job and it will check for changes and build.  Thus no polling 
> is involved.
>
> http://kohsuke.org/2011/12/01/polling-must-die-triggering-jenkins-builds-from-a-git-hook/
>
Unfortunately, the configuration described there does still poll. The
URL trigger invokes the polling rather than a timer. To get decent
performance, I use a URL trigger with a branch name parameter and all
git operations are handled by custom scripts rather than anything within
Jenkins itself.

-- 
Jonathan Rogers

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

RE: Is Jenkins the right tool for hundreds of repositories

[William Soula]

Look into the github trigger where pushes to a repo causes the webhook to hit 
the url for the job and it will check for changes and build.  Thus no polling 
is involved.

http://kohsuke.org/2011/12/01/polling-must-die-triggering-jenkins-builds-from-a-git-hook/

I'd like to suggest the Jenkins Job Builder for creating a ton of jobs, bonus 
is you get to put them into source control.

http://ci.openstack.org/jenkins-job-builder/

Will

From: jenkinsci-users@googlegroups.com [jenkinsci-users@googlegroups.com] on 
behalf of Jonathan Rogers [jonathanrrog...@gmail.com]
Sent: Friday, May 10, 2013 2:38 PM
To: jenkinsci-users@googlegroups.com
Subject: Re: Is Jenkins the right tool for hundreds of repositories

Les Mikesell wrote:
> On Fri, May 10, 2013 at 12:38 PM, JonathanRRogers
>  wrote:
>> Jenkins' git plugin performs poorly with hundreds of branches in a single
>> repository. I haven't tried it with hundreds of repositories, but I wouldn't
>> be surprised if that also performed poorly. I now avoid the git plugin
>> entirely and use git commands directly.
> Do you know if this is specific to git or is it just the nature of
> polling for a whole bunch of different jobs on the same time cycle?
> Or perhaps made worse by polling the same repository with many jobs at
> the same time?

The problem is with Jenkins' git plugin. When I configured it to poll a
repository without limiting branches (the default configuration), one
Jenkins thread would use 100% of a CPU for many minutes just to figure
out what jobs to trigger. No git operations take anywhere close to that
when using the native commands.

--
Jonathan Rogers

--
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is Jenkins the right tool for hundreds of repositories

[Jonathan Rogers]

Les Mikesell wrote:
> On Fri, May 10, 2013 at 12:38 PM, JonathanRRogers
>  wrote:
>> Jenkins' git plugin performs poorly with hundreds of branches in a single
>> repository. I haven't tried it with hundreds of repositories, but I wouldn't
>> be surprised if that also performed poorly. I now avoid the git plugin
>> entirely and use git commands directly.
> Do you know if this is specific to git or is it just the nature of
> polling for a whole bunch of different jobs on the same time cycle?
> Or perhaps made worse by polling the same repository with many jobs at
> the same time?

The problem is with Jenkins' git plugin. When I configured it to poll a
repository without limiting branches (the default configuration), one
Jenkins thread would use 100% of a CPU for many minutes just to figure
out what jobs to trigger. No git operations take anywhere close to that
when using the native commands.

-- 
Jonathan Rogers

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is Jenkins the right tool for hundreds of repositories

[Les Mikesell]

On Fri, May 10, 2013 at 12:38 PM, JonathanRRogers
 wrote:
>
> Jenkins' git plugin performs poorly with hundreds of branches in a single
> repository. I haven't tried it with hundreds of repositories, but I wouldn't
> be surprised if that also performed poorly. I now avoid the git plugin
> entirely and use git commands directly.

Do you know if this is specific to git or is it just the nature of
polling for a whole bunch of different jobs on the same time cycle?
Or perhaps made worse by polling the same repository with many jobs at
the same time?

--
   Les Mikesell
 lesmikes...@gmail.com

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is Jenkins the right tool for hundreds of repositories

[Chris Marks]

You may also want to look into the Jenkins job dsl plugin (
https://github.com/jenkinsci/job-dsl-plugin/) for providing an easier way
to create and manage jobs in Jenkins.

I haven't used Jenkins in a large Git environment, but perhaps those who
have problems with the performance could file bugs related to their
experiences in specific situations if they haven't already?  It would
benefit the community to have a bullet-proof Git plugin that performed well
at scale.

Thanks,
Chris

topher1...@gmail.com


On Fri, May 10, 2013 at 12:41 PM, Benjamin Lau wrote:

> I'm not using post commit hooks... But those would probably work well for
> your situation. For us we don't build on every commit currently. But if you
> take a look at the remote api you can probably make it work that way pretty
> easily.
>
> You can use cURL to trigger it here's one of our urls to trigger a build
> with parameters:
>
>
> http://buildserver:8080/job/buildServer/buildWithParameters?delay=0sec&server_git_branch=featureBranch
>
> Ben
> On May 10, 2013 2:10 PM, "Jon Drukman"  wrote:
>
>> On Fri, May 10, 2013 at 10:44 AM, Benjamin Lau 
>> wrote:
>>
>>> I also use git commands directly. If you make it so your build is
>>> identical for all of your repositories you could have a single
>>> parameterized job which has parameters for the repo and branch (unless
>>> everything is always in master).
>>>
>>
>> I was thinking of going down this route.  Can you elaborate on how you
>> set it up?  Are you using git post-commit hooks to trigger the Jenkins
>> project with the name of the repo & branch that got committed?
>>
>> The builds are all basically identical so being able to parameterize a
>> single project seems like the correct approach.
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "Jenkins Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to jenkinsci-users+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is Jenkins the right tool for hundreds of repositories

[Benjamin Lau]

I'm not using post commit hooks... But those would probably work well for
your situation. For us we don't build on every commit currently. But if you
take a look at the remote api you can probably make it work that way pretty
easily.

You can use cURL to trigger it here's one of our urls to trigger a build
with parameters:

http://buildserver:8080/job/buildServer/buildWithParameters?delay=0sec&server_git_branch=featureBranch

Ben
On May 10, 2013 2:10 PM, "Jon Drukman"  wrote:

> On Fri, May 10, 2013 at 10:44 AM, Benjamin Lau 
> wrote:
>
>> I also use git commands directly. If you make it so your build is
>> identical for all of your repositories you could have a single
>> parameterized job which has parameters for the repo and branch (unless
>> everything is always in master).
>>
>
> I was thinking of going down this route.  Can you elaborate on how you set
> it up?  Are you using git post-commit hooks to trigger the Jenkins project
> with the name of the repo & branch that got committed?
>
> The builds are all basically identical so being able to parameterize a
> single project seems like the correct approach.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is Jenkins the right tool for hundreds of repositories

[Jon Drukman]

On Fri, May 10, 2013 at 10:44 AM, Benjamin Lau wrote:

> I also use git commands directly. If you make it so your build is
> identical for all of your repositories you could have a single
> parameterized job which has parameters for the repo and branch (unless
> everything is always in master).
>

I was thinking of going down this route.  Can you elaborate on how you set
it up?  Are you using git post-commit hooks to trigger the Jenkins project
with the name of the repo & branch that got committed?

The builds are all basically identical so being able to parameterize a
single project seems like the correct approach.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is Jenkins the right tool for hundreds of repositories

[Benjamin Lau]

I also use git commands directly. If you make it so your build is identical
for all of your repositories you could have a single parameterized job
which has parameters for the repo and branch (unless everything is always
in master).

To make this easier to work with I also wrote some shell scripts to make it
easy to start any of the builds. Though in my case we're working with
thousands of branches.

Ben
On May 10, 2013 1:38 PM, "JonathanRRogers" 
wrote:

> On Friday, May 10, 2013 1:23:41 PM UTC-4, Jon Drukman wrote:
>>
>> I am looking into using Jenkins as a tool for continuous deployment.  A
>> commit to a git repository triggers building of rpms, updating a yum repo,
>> possibly triggering production machines to do a yum update, etc.  I've set
>> up a test project, and Jenkins seems like it provides a lot of useful
>> functionality.
>>
>> Unfortunately the way our company is set up, we have hundreds of git
>> repositories and more added every week (they are all separate ad campaigns
>> for various clients).  Some questions I have:
>>
>> Does Jenkins performance suffer with potentially hundreds of
>> repositories?  (As of today, 352.  Most are not changed on a daily basis.
>>  Maybe 5-10.  Eventually we may have 500 or more simultaneous active
>> campaigns.)
>>
>
> Jenkins' git plugin performs poorly with hundreds of branches in a single
> repository. I haven't tried it with hundreds of repositories, but I
> wouldn't be surprised if that also performed poorly. I now avoid the git
> plugin entirely and use git commands directly.
>
>
>>
>> Is there an API to create projects?  Manually adding 352 is not going to
>> be a good time.  I can use Fake or something to script a web browser but
>> that is distasteful.  Plus we're going to have to automatically add the new
>> campaigns as they are created.
>>
>>
> There is an excellent API as described in the wiki:
> https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API>
>
> I generate and manipulate projects with Python scripts using
> http://pythonhosted.org/jenkinsapi/>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is Jenkins the right tool for hundreds of repositories

[JonathanRRogers]

On Friday, May 10, 2013 1:23:41 PM UTC-4, Jon Drukman wrote:
>
> I am looking into using Jenkins as a tool for continuous deployment.  A 
> commit to a git repository triggers building of rpms, updating a yum repo, 
> possibly triggering production machines to do a yum update, etc.  I've set 
> up a test project, and Jenkins seems like it provides a lot of useful 
> functionality.
>
> Unfortunately the way our company is set up, we have hundreds of git 
> repositories and more added every week (they are all separate ad campaigns 
> for various clients).  Some questions I have:
>
> Does Jenkins performance suffer with potentially hundreds of repositories? 
>  (As of today, 352.  Most are not changed on a daily basis.  Maybe 5-10. 
>  Eventually we may have 500 or more simultaneous active campaigns.)
>

Jenkins' git plugin performs poorly with hundreds of branches in a single 
repository. I haven't tried it with hundreds of repositories, but I 
wouldn't be surprised if that also performed poorly. I now avoid the git 
plugin entirely and use git commands directly.
 

>
> Is there an API to create projects?  Manually adding 352 is not going to 
> be a good time.  I can use Fake or something to script a web browser but 
> that is distasteful.  Plus we're going to have to automatically add the new 
> campaigns as they are created.
>
>
There is an excellent API as described in the wiki:
https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API>

I generate and manipulate projects with Python scripts using
http://pythonhosted.org/jenkinsapi/>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Is Jenkins the right tool for hundreds of repositories

[Jon Drukman]

I am looking into using Jenkins as a tool for continuous deployment.  A 
commit to a git repository triggers building of rpms, updating a yum repo, 
possibly triggering production machines to do a yum update, etc.  I've set 
up a test project, and Jenkins seems like it provides a lot of useful 
functionality.

Unfortunately the way our company is set up, we have hundreds of git 
repositories and more added every week (they are all separate ad campaigns 
for various clients).  Some questions I have:

Does Jenkins performance suffer with potentially hundreds of repositories? 
 (As of today, 352.  Most are not changed on a daily basis.  Maybe 5-10. 
 Eventually we may have 500 or more simultaneous active campaigns.)

Is there an API to create projects?  Manually adding 352 is not going to be 
a good time.  I can use Fake or something to script a web browser but that 
is distasteful.  Plus we're going to have to automatically add the new 
campaigns as they are created.

Let me know what you think.  Thanks.
-jsd-

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.