On Mon, Aug 4, 2014 at 3:02 PM, Brian Moffat <crd...@gmail.com> wrote:
> Hi Mark, > Ok - I can see that my use-case is not universal and is likely on the > "very simple" end of the spectrum. I am currently not using distributed > builds. My builds of multiple portals use the same code base and are > serialized - therefore one checkout is all that is needed to build all of > them. In the short term, I'm thinking of doing the following - please tell > me if this is an abomination in the Jenkins world. > Not an abomination, but it seems complicated to me. > I will create a new project that is not a matrix job and is triggered by > an SCM change. All it will do is the git checkout, and call a single > downstream project, which will be the existing matrix project (without a > trigger) that has the git section deleted and simply invokes the downstream > build project iterating through the matrix elements. My assumption is that > since the SCM-triggered project will only do one checkout (explicitly or > implicitly - since it is not a matrix project), and since the downstream > projects do not have any git configurations, they will not do any checkouts > either. > I'm not sure what benefit you gain from a matrix job in that case. Since builds are serialized, and able to use a single project which iterates through the matrix elements one at a time, why not just create a single freestyle project which calls a script that iterates through the projects and builds each project? Using a starter job to do the checkout, then a matrix job to build the configurations seems like an interesting way to iterate through the configurations, but aren't the configurations already expressed in the source code, so you can write a script (MSBuild, perl, python, Java, etc.) which iterates through those configurations and builds each of them. > Feels like a kludge, but i think the result will be a complete build in 11 > minutes as opposed to 30. > > Do you agree? > > Thanks, > Brian > > On Monday, August 4, 2014 2:47:01 PM UTC-4, Mark Waite wrote: >> >> >> On Mon, Aug 4, 2014 at 12:24 PM, Brian Moffat <crd...@gmail.com> wrote: >> >>> Thanks Mark. >>> I have a couple of follow-up questions, if you don't mind. >>> >>> 1. I've seen it work - meaning that the matrix project would do one >>> checkout and the downstream projects would run one after the other with no >>> lengthy pause between. So I'm left feeling like there should be a way to >>> "get back to that" without the suggestions you made - since I've not >>> modified the project configs since seeing it work. It may be that at the >>> time it was an "unintended feature" that got removed by a plug-in update, >>> but that's pure conjecture on my part. >>> >> >> I'm reasonably sure that it has always behaved that way. Each job in a >> matrix (multi-configuration) project may be on a separate machine, so each >> job performs a "git fetch" followed by a "git checkout". However, if you >> can find versions of the git plugin and the git client plugin which behave >> the way you want, you could compare the source code of the current plugin >> and the version behaving the way you want to see if you can identify the >> cause of the behavioral change. >> >> >>> 2. Why does the matrix job always do a "get a full copy" of the >>> repository before invoking the downstream job? If that's what I wanted, I >>> would have the downstream job do the pull. To me it looks like a bug, not >>> an enhancement request. What am I missing? >>> >> >> I believe that each job in the matrix job needs an independent copy of >> the repository so that it can work independent of the other jobs in the >> matrix. >> >> >>> -Brian >>> >>> On Monday, August 4, 2014 1:41:08 PM UTC-4, Mark Waite wrote: >>>> >>>> >>>> >>>> >>>> On Mon, Aug 4, 2014 at 10:11 AM, Brian Moffat <crd...@gmail.com> wrote: >>>> >>>>> I have a matrix project to build different C# projects/solutions. >>>>> There is one git repository for all solutions. >>>>> >>>>> The matrix job is configured for Git to checkout the master branch to >>>>> a local directory, and then invokes a downstream project for each matrix >>>>> element (solution). That downstream project has not Git configuration, >>>>> and >>>>> simply builds the solution using MSBuild. >>>>> >>>>> What I am observing (looking at the console output for the Matrix >>>>> build job instance) is the matrix job performs the git checkout, which >>>>> takes about 4 minutes. It then says "Triggering <matrix element>", but >>>>> that downstream project doesn't show up in the build list for another 4-5 >>>>> minutes. Based on noticing that the content of the checkout directory is >>>>> changing (total size differences) during the period between when it says >>>>> "Triggering ..." and when that downstream job actually starts, my guess is >>>>> that another git checkout is occurring behind the scenes. >>>>> >>>> >>>> Your guess is correct. Each instance gets a full copy of the >>>> repository. >>>> >>>> If your repository is large, you could add the "Additional behaviours" >>>> for "Advanced clone options" and use a reference repository on the build >>>> machine(s). For example, if you created a bare repository copy of your >>>> central git repo on each build machine at c:\bare\your-git-repo.git, then >>>> you could list c:\bare\your-git-repo.git as the reference repository >>>> location. Git will then reference the contents of that bare repository and >>>> use that to reduce the "fetch" time. >>>> >>>> That won't reduce the "checkout" time, just the "fetch" time. If you >>>> also need to reduce the checkout time, then you might consider using >>>> "Sparse checkout" to only checkout those subdirectories which are actually >>>> required for your build. >>>> >>>> Another alternative would be to implement "git-work-dir" support in the >>>> git plugin, and submit a pull request for it. Refer to >>>> https://issues.jenkins-ci.org/browse/JENKINS-23594 for an enhancement >>>> request asking for that support. That enhancement request seems to match >>>> your use case quite well. >>>> >>>> >>>>> Obviously, this adds a significant amount of time to the total length >>>>> of the matrix build. The builds of each solution take under 1 minute. So >>>>> for the 7 solutions, I would expect the length of time to be 4+(7*1) or >>>>> roughly 11 minutes, but with the 4-5 minute delay between each matrix >>>>> element, it takes (4*7) + (7*1) or roughly 30 minutes. Our CI/CD >>>>> processes >>>>> are maturing, and the lengthy delay between checkin at deploy to staging >>>>> is >>>>> now slowing us down. >>>>> >>>>> Can anyone provide me guidance on what is going on behind the scenes >>>>> (nothing in any Jenkins log that I can find), or tell me how I can >>>>> configure the matrix job "properly" to not do this? >>>>> >>>>> This has run fast, sometimes - but I think perhaps it stopped working >>>>> after updating plugins, or may somehow be related to the plugins installed >>>>> - I just don't know where to start to look. >>>>> >>>>> The git section of the matrix job has: >>>>> - Additional Behaviors : Checkout to a subdirectory: >>>>> C:\Build\Source\WebPortal\Master\Dev >>>>> - Advanced sub-module behavior: Recursively update submodules (checked) >>>>> - Prune stale remote-tracking branches (not sure this is necessary?) >>>>> >>>>> Here is the console log for the matrix job: >>>>> >>>>> Started by user < <http://autobuild.blueport.com/user/bmoffat>me> >>>>> [EnvInject] - Loading node environment variables. >>>>> Building in workspace C:\Tools\Jenkins\jobs\BuildAllWebPortals\workspace >>>>> > C:\Tools\Git\cmd\git.exe rev-parse --is-inside-work-tree >>>>> Fetching changes from the remote Git repository >>>>> > C:\Tools\Git\cmd\git.exe config remote.origin.url <our git repository >>>>> url> >>>>> >>>>> Pruning obsolete local branches >>>>> Fetching upstream changes from <our git repository url> >>>>> >>>>> > C:\Tools\Git\cmd\git.exe --version >>>>> > C:\Tools\Git\cmd\git.exe fetch --tags --progress <our git repository >>>>> url> +refs/heads/master:refs/remotes/origin/master --prune >>>>> > C:\Tools\Git\cmd\git.exe rev-parse "origin/master^{commit}" >>>>> Checking out Revision b18aa8cff0aa9da9166f740877845cef27b612ff >>>>> (origin/master) >>>>> > C:\Tools\Git\cmd\git.exe config core.sparsecheckout >>>>> > C:\Tools\Git\cmd\git.exe checkout -f >>>>> b18aa8cff0aa9da9166f740877845cef27b612ff >>>>> > C:\Tools\Git\cmd\git.exe rev-list >>>>> b18aa8cff0aa9da9166f740877845cef27b612ff >>>>> > C:\Tools\Git\cmd\git.exe remote >>>>> > C:\Tools\Git\cmd\git.exe submodule init >>>>> > C:\Tools\Git\cmd\git.exe submodule sync >>>>> > C:\Tools\Git\cmd\git.exe config --get remote.origin.url >>>>> > C:\Tools\Git\cmd\git.exe submodule update --init --recursive >>>>> Email was triggered for: Before Build >>>>> Sending email for trigger: Before Build >>>>> Sending email to: <distribution list> >>>>> Triggering MobileASI >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileASI/> >>>>> >>>>> MobileASI >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileASI/> >>>>> completed with result SUCCESS >>>>> Triggering Furniture >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=Furniture/> >>>>> >>>>> Furniture >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=Furniture/> >>>>> completed with result SUCCESS >>>>> Triggering RetailPortal >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=RetailPortal/> >>>>> >>>>> RetailPortal >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=RetailPortal/> >>>>> completed with result SUCCESS >>>>> Triggering MobileLeons >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileLeons/> >>>>> >>>>> MobileLeons >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileLeons/> >>>>> completed with result SUCCESS >>>>> Triggering Leons >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=Leons/> >>>>> >>>>> Leons >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=Leons/> >>>>> completed with result SUCCESS >>>>> Triggering MobileVCF >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileVCF/> >>>>> >>>>> MobileVCF >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=MobileVCF/> >>>>> completed with result SUCCESS >>>>> Triggering RetailTool >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=RetailTool/> >>>>> >>>>> RetailTool >>>>> <http://autobuild.blueport.com/job/BuildAllWebPortals/Portal=RetailTool/> >>>>> completed with result SUCCESS >>>>> No emails were triggered. >>>>> Finished: SUCCESS >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Jenkins Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to jenkinsci-use...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks! >>>> Mark Waite >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jenkinsci-use...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Thanks! >> Mark Waite >> > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Thanks! Mark Waite -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
