Re: Git Plugin: Clean after checkout exclusions?
James, job-private local Maven repositories do work for me. I don't mind the overhead in disk space. By not using option Clean after checkout and trimming the workspace by script instead, I am able to reuse a populated private local repository from one job run to the next. I also like to see Maven to support layered repositories. -Max On 02/14/2014 08:03 AM, James Nord (jnord) wrote: You can't clean in the maven pom - as the maven pom is what will use what you are trying to clean :-o However - even if Max could selectively clean the repo I don't think it would scale. With Gerrit - you can end up with many changes submitted at the same time - and you don't want to run them serially (feedback loop is too long). As such you need multiple workspaces - and you need multiple executors as well. As no one writes well behaved unit tests the best practice is to use 1 executor per slave. At this point the likely hood of getting a slave that has the workspace that Jenkins wants is possibly slim. So even if you could - you may end up getting wildly different build times in various cases. There is likely more to be gained (long term) from using shallow clones and reference repositories. If you have a good maven chache close to you build server - the time to download isn't the dominating factor anymore (and downloads can be performed in parallel, and modules can be built in parallel with Maven3 - so the download time end up not blocking/contributing to the build time by any significant amount) What possibly would help is layered local maven repository support (something that Jason has mooted several times on the Maven lists) - but nothing is available yet. (but that then assumes that all projects use the same maven settings (for all mirrors and have a mirror of * fallback). /James *From:*jenkinsci-users@googlegroups.com [mailto:jenkinsci-users@googlegroups.com] *On Behalf Of *Mark Waite *Sent:* 13 February 2014 23:53 *To:* jenkinsci-users@googlegroups.com *Subject:* Re: Git Plugin: Clean after checkout exclusions? If the Clean after checkout option you are referencing is the option from the git plugin, then I would rather suggest that you should adapt and maintain some form of build specific clean within your maven pom.xml, rather than complicating the git plugin with inclusions and exclusions for the operation. The Clean after checkout option in the git plugin is conceptually mapped to git clean -xfd. Your exclusion / inclusion proposal has no matching concept in the git command line nor in JGit. I'd rather not see the git plugin stray from its conceptual mapping to git commands. It is complicated enough already. Mark Waite On Thu, Feb 13, 2014 at 4:34 PM, Max Spring m2spr...@springdot.org mailto:m2spr...@springdot.org wrote: I'm using the Clean after checkout option a lot with my Gerrit verification Maven jobs. I also have these jobs use a local repository, sitting in the job's workspace. Because of the Clean after checkout option set, the job's local repository gets wiped and the subsequent build has to time-consumingly repopulate it with lots of artifacts. I'd like to speed this up. Ideally, I'd like to exclude certain directories/files from being removed when using the Clean after checkout option is set. Sort of: exclude=$WORKSPACE/.repository In addition to an exclude option for Clean after checkout, I would like to have an include option, where I would then specify certain hierarchies within the private local repository to be cleaned, like include=$WORKSPACE/.repository/org/myorg/. -Max -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com mailto:jenkinsci-users%2bunsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- Thanks! Mark Waite -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com mailto:jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
RE: Git Plugin: Clean after checkout exclusions?
You can't clean in the maven pom - as the maven pom is what will use what you are trying to clean :-o However - even if Max could selectively clean the repo I don't think it would scale. With Gerrit - you can end up with many changes submitted at the same time - and you don't want to run them serially (feedback loop is too long). As such you need multiple workspaces - and you need multiple executors as well. As no one writes well behaved unit tests the best practice is to use 1 executor per slave. At this point the likely hood of getting a slave that has the workspace that Jenkins wants is possibly slim. So even if you could - you may end up getting wildly different build times in various cases. There is likely more to be gained (long term) from using shallow clones and reference repositories. If you have a good maven chache close to you build server - the time to download isn't the dominating factor anymore (and downloads can be performed in parallel, and modules can be built in parallel with Maven3 - so the download time end up not blocking/contributing to the build time by any significant amount) What possibly would help is layered local maven repository support (something that Jason has mooted several times on the Maven lists) - but nothing is available yet. (but that then assumes that all projects use the same maven settings (for all mirrors and have a mirror of * fallback). /James From: jenkinsci-users@googlegroups.com [mailto:jenkinsci-users@googlegroups.com] On Behalf Of Mark Waite Sent: 13 February 2014 23:53 To: jenkinsci-users@googlegroups.com Subject: Re: Git Plugin: Clean after checkout exclusions? If the Clean after checkout option you are referencing is the option from the git plugin, then I would rather suggest that you should adapt and maintain some form of build specific clean within your maven pom.xml, rather than complicating the git plugin with inclusions and exclusions for the operation. The Clean after checkout option in the git plugin is conceptually mapped to git clean -xfd. Your exclusion / inclusion proposal has no matching concept in the git command line nor in JGit. I'd rather not see the git plugin stray from its conceptual mapping to git commands. It is complicated enough already. Mark Waite On Thu, Feb 13, 2014 at 4:34 PM, Max Spring m2spr...@springdot.orgmailto:m2spr...@springdot.org wrote: I'm using the Clean after checkout option a lot with my Gerrit verification Maven jobs. I also have these jobs use a local repository, sitting in the job's workspace. Because of the Clean after checkout option set, the job's local repository gets wiped and the subsequent build has to time-consumingly repopulate it with lots of artifacts. I'd like to speed this up. Ideally, I'd like to exclude certain directories/files from being removed when using the Clean after checkout option is set. Sort of: exclude=$WORKSPACE/.repository In addition to an exclude option for Clean after checkout, I would like to have an include option, where I would then specify certain hierarchies within the private local repository to be cleaned, like include=$WORKSPACE/.repository/org/myorg/. -Max -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.commailto:jenkinsci-users%2bunsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- Thanks! Mark Waite -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.commailto:jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Git Plugin: Clean after checkout exclusions?
Hi Mark,
good feedback. Haven't used git clean before.
I turned off Clean after checkout and do this before the actual build:
( set +x
git clean -n -xfd \
| grep -v ' \.repository/' \
| while read f; do \
p=${f#Would remove }
(set -x; rm -rf $WORKSPACE/$p)
done
)
rm -rf $WORKSPACE/.repository/org/myorg
That gives me what I want.
(Yes, I could have use some | cut ... | xargs rm pipe to achieve the same, but
I like the console output I get with my way.)
Regards,
-Max
On 02/13/2014 03:53 PM, Mark Waite wrote:
If the Clean after checkout option you are referencing is the option from the
git plugin, then I would rather suggest that you should adapt and maintain some form of
build specific clean within your maven pom.xml, rather than complicating the git plugin
with inclusions and exclusions for the operation.
The Clean after checkout option in the git plugin is conceptually mapped to git
clean -xfd. Your exclusion / inclusion proposal has no matching concept in the git command
line nor in JGit. I'd rather not see the git plugin stray from its conceptual mapping to git
commands. It is complicated enough already.
Mark Waite
On Thu, Feb 13, 2014 at 4:34 PM, Max Spring m2spr...@springdot.org
mailto:m2spr...@springdot.org wrote:
I'm using the Clean after checkout option a lot with my Gerrit
verification Maven jobs.
I also have these jobs use a local repository, sitting in the job's
workspace.
Because of the Clean after checkout option set, the job's local
repository gets wiped and
the subsequent build has to time-consumingly repopulate it with lots of
artifacts.
I'd like to speed this up.
Ideally, I'd like to exclude certain directories/files from being removed when using
the Clean after checkout option is set.
Sort of: exclude=$WORKSPACE/.__repository
In addition to an exclude option for Clean after checkout,
I would like to have an include option, where I would then specify
certain hierarchies within the private local repository to be cleaned,
like include=$WORKSPACE/.__repository/org/myorg/.
-Max
--
You received this message because you are subscribed to the Google Groups
Jenkins Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscribe@__googlegroups.com
mailto:jenkinsci-users%2bunsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/__groups/opt_out
https://groups.google.com/groups/opt_out.
--
Thanks!
Mark Waite
--
You received this message because you are subscribed to the Google Groups Jenkins
Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups Jenkins
Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Git Plugin: Clean after checkout exclusions?
I'm using the Clean after checkout option a lot with my Gerrit verification Maven jobs. I also have these jobs use a local repository, sitting in the job's workspace. Because of the Clean after checkout option set, the job's local repository gets wiped and the subsequent build has to time-consumingly repopulate it with lots of artifacts. I'd like to speed this up. Ideally, I'd like to exclude certain directories/files from being removed when using the Clean after checkout option is set. Sort of: exclude=$WORKSPACE/.repository In addition to an exclude option for Clean after checkout, I would like to have an include option, where I would then specify certain hierarchies within the private local repository to be cleaned, like include=$WORKSPACE/.repository/org/myorg/. -Max -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Git Plugin: Clean after checkout exclusions?
If the Clean after checkout option you are referencing is the option from the git plugin, then I would rather suggest that you should adapt and maintain some form of build specific clean within your maven pom.xml, rather than complicating the git plugin with inclusions and exclusions for the operation. The Clean after checkout option in the git plugin is conceptually mapped to git clean -xfd. Your exclusion / inclusion proposal has no matching concept in the git command line nor in JGit. I'd rather not see the git plugin stray from its conceptual mapping to git commands. It is complicated enough already. Mark Waite On Thu, Feb 13, 2014 at 4:34 PM, Max Spring m2spr...@springdot.org wrote: I'm using the Clean after checkout option a lot with my Gerrit verification Maven jobs. I also have these jobs use a local repository, sitting in the job's workspace. Because of the Clean after checkout option set, the job's local repository gets wiped and the subsequent build has to time-consumingly repopulate it with lots of artifacts. I'd like to speed this up. Ideally, I'd like to exclude certain directories/files from being removed when using the Clean after checkout option is set. Sort of: exclude=$WORKSPACE/.repository In addition to an exclude option for Clean after checkout, I would like to have an include option, where I would then specify certain hierarchies within the private local repository to be cleaned, like include=$WORKSPACE/.repository/org/myorg/. -Max -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- Thanks! Mark Waite -- You received this message because you are subscribed to the Google Groups Jenkins Users group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
