Re: WO, Git and Jenkins: Impedance mismatch?
Just in case anyone is following along at home… On 01/08/2012, at 9:35 AM, Paul Hoadley wrote: > _However_, I'm not convinced this works reliably. I'm currently debugging a > situation where pushing commits to multiple Eclipse projects within the one > repo is triggering only the _first_ Jenkins job. Obviously that can be > disastrous if the project dependency is the other way around. (I haven't got > to the bottom of it yet—something like that is happening, though I can't > prove it's the fault of Jenkins.) I was able to reproduce this at will. Say you have FooApplication dependent on FooFramework, and they're both in the Foo repository. (I'm using Bitbucket, but I presume that doesn't matter.) FooApplication and FooFramework are both Jenkins jobs, with the latter triggering the former. You can restrict FooApplication to just be triggered by commits to "FooApplication/.*", and similarly with the framework, which is nice in theory. I had polling set to 5 minutes. Pushing commits just to FooFramework triggered that project to build, as expected. But pushing a group of commits that included commits to _both_ projects triggered a build in only _FooApplication_, and it was reproducible. A workaround is to use some sort of push notification from the repository, and this is what I have done. (Simple polling was nicer, as you can keep your Jenkins server isolated from inbound traffic.) -- Paul Hoadley http://logicsquad.net/ ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: WO, Git and Jenkins: Impedance mismatch?
On 22/06/2012, at 9:04 AM, Paul Hoadley wrote: > I'm looking at moving some projects from Subversion to Git. We use Jenkins > as a build server. I'm interested to know some broad details about the > WO-Git-Jenkins setups that others are using. > > In the past, we've used Subversion repositories that are somewhere in the > middle of the range from "monolithic" (where absolutely all of your code is > in one repo) to "atomic" (where it's basically one Eclipse project per repo). > For example, an "application" repo might contain three Eclipse projects: one > app, and a couple of frameworks, say. I had intended to basically continue > this arrangement with the move to Git, mostly because it's pretty easy to > just dump out an entire repo with git-svn. But I've already run into at > least the following problems: > > 1. Unlike Subversion, where I can pick a point in the tree from which to > begin the checkout (say, FooProject/FooApp), with Git I need to clone the > whole repo into the Jenkins workspace. I suspect I can work around this with > some changes to my workspace setup script. > > 2. Related, say I'm building Jenkins jobs FooApp and FooFramework from > within the FooProject repo—now I'm cloning the entire repo into the workspace > of each of those jobs. > > Neither of these are show-stoppers, but it seems messy. I _could_ get around > this by having a strict single-project-per-repo policy—but then repo > proliferation becomes an issue, and I no longer have the convenience of > grouping related projects together in one repo. > > Can anyone share their broad approach to these issues? A month has passed, and I'm still only marginally closer to a satisfactory solution here. There seems to be no simple way around the issue of cloning the entire repo for each Jenkins job, though the Git plugin at least allows you to specify "Included Regions", which is a regex with which you can nominate, say, 'FooApplication/.*' so that commits only to FooApplication trigger builds for the FooApplication job. _However_, I'm not convinced this works reliably. I'm currently debugging a situation where pushing commits to multiple Eclipse projects within the one repo is triggering only the _first_ Jenkins job. Obviously that can be disastrous if the project dependency is the other way around. (I haven't got to the bottom of it yet—something like that is happening, though I can't prove it's the fault of Jenkins.) Trying to build on multiple branches is very awkward. My workspace setup script (derived from the original Subversion-based script that Mike Schrag wrote years ago) is not branch-aware, and the best way to make it so is not obvious. (The main problem is that, as with the original script, it copies frameworks into the workspace based on the project's .classpath. There's no way to denote in the .classpath that I want the 'develop' branch of FooFramework.framework, and then there would only be convention to rely upon to find that as a Jenkins job anyway. How is all this best generalised?) So at the moment, I add an extra build step that deletes the frameworks copied over from their 'master' branch projects, and copies over those frameworks from the required branch. Seriously, I'd love to hear from anyone else who is doing any of this—especially building multiple branches of the same projects from Git repos, using Jenkins and the Git plugin (or any other plugin). Does anyone have anything they can share? Has anyone looked at Atlassian's Bamboo? -- Paul Hoadley http://logicsquad.net/ ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: WO, Git and Jenkins: Impedance mismatch?
On 22/06/2012, at 12:25 PM, Kieran Kelleher wrote: > As you well know Paul, I don't have a Git-Jenkins setup (yet), but off the > top of my head, if I was concerned about the network latency and space > consumption of cloning a full git repo across the network for every Jenkins > job execution, I would approach my setup as follows: > > (1) First, on the Jenkins build server, setup a single bare repo mirror of > the remote repo. There are a number of ways to mirror a git repo such as > executing a 'git remote fetch --all' on the mirror side or adding a > post-commit hook on the remote side to push all commits to the mirror as they > come in, so just google git repo mirroring and pick a method that works best > for you. > > (2) For your Jenkins workspace clone operation one or more of the git clone > operations that gives the fastest clone and is suitable for a Jenkins > workspace clone (see --local, --shared, --reference, --no-checkout, > --single-branch and --depth options), some of which will give you a minimal, > hard-link-no-copy or a shallow single-branch clone that might be useful only > to use as a short-lived skeleton repo useful only for a for a Jenkins build > of a single commit snapshot of a working copy. Thanks Kieran, I'll look at those ideas. I was also going to look at sparse checkout [1], though I think that might be beyond the ability of the current state of the Jenkins Git plugin. [1] http://schacon.github.com/git/git-read-tree.html#_sparse_checkout -- Paul Hoadley http://logicsquad.net/ ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: WO, Git and Jenkins: Impedance mismatch?
As you well know Paul, I don't have a Git-Jenkins setup (yet), but off the top of my head, if I was concerned about the network latency and space consumption of cloning a full git repo across the network for every Jenkins job execution, I would approach my setup as follows: (1) First, on the Jenkins build server, setup a single bare repo mirror of the remote repo. There are a number of ways to mirror a git repo such as executing a 'git remote fetch --all' on the mirror side or adding a post-commit hook on the remote side to push all commits to the mirror as they come in, so just google git repo mirroring and pick a method that works best for you. (2) For your Jenkins workspace clone operation one or more of the git clone operations that gives the fastest clone and is suitable for a Jenkins workspace clone (see --local, --shared, --reference, --no-checkout, --single-branch and --depth options), some of which will give you a minimal, hard-link-no-copy or a shallow single-branch clone that might be useful only to use as a short-lived skeleton repo useful only for a for a Jenkins build of a single commit snapshot of a working copy. HTH, Kieran On Jun 21, 2012, at 7:34 PM, Paul Hoadley wrote: > Hello, > > I'm looking at moving some projects from Subversion to Git. We use Jenkins > as a build server. I'm interested to know some broad details about the > WO-Git-Jenkins setups that others are using. > > In the past, we've used Subversion repositories that are somewhere in the > middle of the range from "monolithic" (where absolutely all of your code is > in one repo) to "atomic" (where it's basically one Eclipse project per repo). > For example, an "application" repo might contain three Eclipse projects: one > app, and a couple of frameworks, say. I had intended to basically continue > this arrangement with the move to Git, mostly because it's pretty easy to > just dump out an entire repo with git-svn. But I've already run into at > least the following problems: > > 1. Unlike Subversion, where I can pick a point in the tree from which to > begin the checkout (say, FooProject/FooApp), with Git I need to clone the > whole repo into the Jenkins workspace. I suspect I can work around this with > some changes to my workspace setup script. > > 2. Related, say I'm building Jenkins jobs FooApp and FooFramework from > within the FooProject repo—now I'm cloning the entire repo into the workspace > of each of those jobs. > > Neither of these are show-stoppers, but it seems messy. I _could_ get around > this by having a strict single-project-per-repo policy—but then repo > proliferation becomes an issue, and I no longer have the convenience of > grouping related projects together in one repo. > > Can anyone share their broad approach to these issues? > > > -- > Paul Hoadley > http://logicsquad.net/ > > > > ___ > Do not post admin requests to the list. They will be ignored. > Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) > Help/Unsubscribe/Update your Subscription: > https://lists.apple.com/mailman/options/webobjects-dev/kelleherk%40gmail.com > > This email sent to kelleh...@gmail.com ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: WO, Git and Jenkins: Impedance mismatch?
On 2012-06-21, at 4:34 PM, Paul Hoadley wrote: > Hello, > > I'm looking at moving some projects from Subversion to Git. We use Jenkins > as a build server. I'm interested to know some broad details about the > WO-Git-Jenkins setups that others are using. > > In the past, we've used Subversion repositories that are somewhere in the > middle of the range from "monolithic" (where absolutely all of your code is > in one repo) to "atomic" (where it's basically one Eclipse project per repo). > For example, an "application" repo might contain three Eclipse projects: one > app, and a couple of frameworks, say. I had intended to basically continue > this arrangement with the move to Git, mostly because it's pretty easy to > just dump out an entire repo with git-svn. But I've already run into at > least the following problems: > > 1. Unlike Subversion, where I can pick a point in the tree from which to > begin the checkout (say, FooProject/FooApp), with Git I need to clone the > whole repo into the Jenkins workspace. I suspect I can work around this with > some changes to my workspace setup script. > > 2. Related, say I'm building Jenkins jobs FooApp and FooFramework from > within the FooProject repo—now I'm cloning the entire repo into the workspace > of each of those jobs. > > Neither of these are show-stoppers, but it seems messy. I _could_ get around > this by having a strict single-project-per-repo policy—but then repo > proliferation becomes an issue, and I no longer have the convenience of > grouping related projects together in one repo. > > Can anyone share their broad approach to these issues? + 1 :-) David > > > -- > Paul Hoadley > http://logicsquad.net/ > > > > ___ > Do not post admin requests to the list. They will be ignored. > Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) > Help/Unsubscribe/Update your Subscription: > https://lists.apple.com/mailman/options/webobjects-dev/programmingosx%40mac.com > > This email sent to programming...@mac.com ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
WO, Git and Jenkins: Impedance mismatch?
Hello, I'm looking at moving some projects from Subversion to Git. We use Jenkins as a build server. I'm interested to know some broad details about the WO-Git-Jenkins setups that others are using. In the past, we've used Subversion repositories that are somewhere in the middle of the range from "monolithic" (where absolutely all of your code is in one repo) to "atomic" (where it's basically one Eclipse project per repo). For example, an "application" repo might contain three Eclipse projects: one app, and a couple of frameworks, say. I had intended to basically continue this arrangement with the move to Git, mostly because it's pretty easy to just dump out an entire repo with git-svn. But I've already run into at least the following problems: 1. Unlike Subversion, where I can pick a point in the tree from which to begin the checkout (say, FooProject/FooApp), with Git I need to clone the whole repo into the Jenkins workspace. I suspect I can work around this with some changes to my workspace setup script. 2. Related, say I'm building Jenkins jobs FooApp and FooFramework from within the FooProject repo—now I'm cloning the entire repo into the workspace of each of those jobs. Neither of these are show-stoppers, but it seems messy. I _could_ get around this by having a strict single-project-per-repo policy—but then repo proliferation becomes an issue, and I no longer have the convenience of grouping related projects together in one repo. Can anyone share their broad approach to these issues? -- Paul Hoadley http://logicsquad.net/ ___ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list (Webobjects-dev@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
