Greg, Thank you for responding, I am somewhat new to git and had no idea about the --reference option. Your solution looks like basically the best way to do this.
I still wonder why there are no options out-of-the-box to: 1) Set up only one cloned repository per project, and 2) Refuse to check out repositories at the top level (on the machine which is conducting the build) Or, if there are options to do this but I don't know where they are. It seems like the setup you and I have should be the default! The only thing I figure is that for generality's sake, each build should have a unique clone of the repo. Thanks. *Also, my terminology is too loose - I meant "clone" not "checkout". On Mon, Dec 16, 2013 at 12:59 PM, Gergely Nagy <gsz...@gmail.com> wrote: > Not sure how have the exact same "checkout" multiple times is useful (most > of my jobs modify the workspace so I need to isolate them from each other > -> separate workspace == separate "checkouts"). > > OTOH, I was also interested to save on disk space when it comes to large > git repos cloned multiple times. Note that I mean "clone" not "checkout". > > So, an easy/safe enough solution to reduce waste of duplicate clones: > 1) have 1 proper clone of the big repo on each slave - just by create a > dumb "mirror" job with no build steps, just to keep the clone up to date > 2) the actual jobs use the "Reference" repo option: although the main > clone location can be still be the central server, you point to that > workspace of the mirror job as a reference (see --reference > https://www.kernel.org/pub/software/scm/git/docs/git-clone.html, and the > Advanced git plugin option). > > My results using this is quite dramatic, the repo (.git) sizes under the > worker jobs are just a few Mb, even the actually repo is 1.5G+ in my case. > Also the "clone" for each worker job will just take a fraction of time. > > Now I'd like Jenkins (plugin?) to provide a setup like this > out-of-the-box, eliminating the manual fiddling (I needed to add site-wide > variables overridden for each slave, etc..). Anyway it's good enough for me > now. > > Of course the actual "checkout" sizes are the same, but I see no > safe&straightforward way around that, that's just the nature of > building/testing code. > So, not sure if this helps but good luck anyway. > Greg > > > > On 16 December 2013 17:06, <qbd...@vt.edu> wrote: > >> Has anyone considered a modification to the git plugin (or other SCM >> plugins), to reduce the total number of checkouts physically required on a >> slave machine? >> >> We run 6 mult-configuration projects (debug and release configurations) >> on a handful of computers. Each configuration requires one code checkout >> (2.1 G). That's 2.1*2*6 = 25.2 G on each slave machine, for the source >> code only. The master build machine requires another 12.6 G, because it >> checks out additional copies *outside* the scope of build directories (no >> one knows why). Forget about leaving "Advanced Project Options" > "Restrict >> where this project can be run" unchecked, because if the build is started >> on a different computer each time, the code has to be checked out at the >> top level (potentially adding the 12.6 G to each slave machine, if the load >> is shared). >> >> Perhaps these issues have already been addressed; we run Jenkins version >> 1.524. >> >> Suggestions: >> a) Live with redundant code checkouts >> b) Stop using the git plugin, and have my shell script handle code >> checkout >> c) Rewrite the git plugin, or add an option, to allow one checkout per >> project on a slave. Technically this is the minimum number of checkouts >> allowed, because each project is allowed to access a unique sha1 hash. >> d) Add an option to the Jenkins core to disregard the top-level code >> checkouts (which I gather are done for some kind of consistency check...we >> can live without that check I think). >> e) I don't know Java and don't have much free time, otherwise I would >> write a completely new git plugin. >> >> Thanks for any suggestions, especially if I am missing something obvious >> here. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jenkinsci-users+unsubscr...@googlegroups.com. >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Jenkins Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jenkinsci-users/IwgQSKqrC6g/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > jenkinsci-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
