[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title
zy zhang commented on JENKINS-19022
Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Hi, you can use the below groovy script to delete git revisions for the current build. import jenkins.model.* jenkinsInstance = jenkins.model.Jenkins.get() def job = jenkinsInstance.getItemByFullName(JOB_NAME); def build = job.getBuild(BUILD_NUMBER) def prj = build.project def gitActions = build.getActions(hudson.plugins.git.util.BuildData.class) if (gitActions != null) { for (action in gitActions) { build.actions.remove(action) //build.actions.add(action) build.save() } }
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.150469.1375284228000.6420.1569567241384%40Atlassian.JIRA.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Jim D commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Thanks for the update! Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.150469.1375284228000.9319.1561552081292%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Mark Waite commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Jim D this issue won't be fixed in 4.0.0. The incompatibilities from the BuildDetails change were too great for the community. The accidental release of git plugin 4.0.0-rc to the production update centers showed incompatibilities that I had missed in my testing and that others had missed in their testing. BuildData will be the same bloated memory user in 4.0.0 that it is in 3.x. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.150469.1375284228000.8940.1561499402387%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Jim D commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Mark Waite, I'm glad I found the comments in this issue, thank you! When the final 4.0.0 is released with the changes for this issue, will it make sure to handle both BuildDetails and BuildData for previous build runs? We updated to 4.0.0-rc some time back, and I just came across this thread recently trying to resolve a BuildData issue, and rolled back to 3.9.3. At this point, Git plugin is working and creating BuildData for each build run, but all those old builds done with 4.0.0-rc don't have any kind of "Git Build Data" in the Jenkins UI anymore, and no BuildData or BuildDetails when retrieved programmatically from WorkflowRun.getAllActions. I think it must be that 3.9.3 isn't able to read the new BuildDetails object info. Will 4.0.0 be able to read both, from both 3.9.X builds and 4.0.0 builds? Thanks again. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.150469.1375284228000.8782.1561495441189%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Jacob Keller commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields tzafrir, the gitlab plugin probably just needs to be updated to look for BuildDetails instead of BuildData. As for the rebuilding issue, that is likely due to the fact that the new plugin no longer maintains history of "I built this already" if the build which did it was deleted. If the build hasn't been deleted it's plausible there is a bug in how we look up the old data again It may be worth more seriously considering the XmlFile approach. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Stefan Hengelein commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Thanks for your quick responses! Connor Tumbleson The downgrade to the previous version (3.9.1) fixed the issue for now. Just took some time to identify the culprit. Mark Waite No problem mate, mistakes happen. I'll have a closer look on monday so hopefully I can help to provide steps to reproduce the issue! < /td> Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title tzafrir commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields The buildData is also required for gitlab plugin. gitlab plugin pipeline step "gitlabCommitStatus" isn't working anymore with 4.0.0-rc It does not find any BuildData object. Had to downgrade to 3.9.3 where all was working fine. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Connor Tumbleson commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Thanks Mark Waite - we reverted 3.9.2, then re-upgraded to 3.9.3. Unfortunately our builds were in quite a messed up shape. New commits to a branch were triggering all branches that still lived on the remote. We tried deleting the repos from the filesystem of Jenkins, hoping a new clone would resolve that - It did not. For other googlers who stumble upon this page, we had to restore a server backup and lose about 2 days of history on builds. We lost minor configurations, but it was way better than having our system bloated with thousands of builds and building nearly everything on each commit. Mistakes happen mate, no worries and I'll try and file a bug to keep these bugs organized. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Mark Waite commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields You should downgrade to git plugin 3.9.2 then install git plugin 3.9.3 which includes a fix for an agent / tool interaction issue that was first introduced in git plugin 3.9.2. The git plugin 4.0.0-rc is only a release candidate, not a production ready release. I made a mistake when I chose the version number. I assumed that the "-rc" suffix would deliver the plugin only from the experimental update center and not from the general update center. I was wrong. I apologize sincerely for my mistake. The issues detected with git plugin 4.0.0-rc indicate that it was not as close to release as my testing indicated. More testing and more fixes will be needed before git plugin 4.0.0 will be released for general availability. You can help assure that the problems you've found are fixed in git plugin 4.0.0-rc by providing a bug report which contains numbered steps that describe how someone else can see the same bug that you are seeing. The descriptions in the comments of this specific bug report seem to indicate at least two different bugs which are outside of this bug report and should be reported separately so that they can be tracked separately to resolution. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Connor Tumbleson commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Tagging onto this ticket, because something here caught my eye that Stefan Hengelein posted > With this change now our jenkins tries to rebuild ALL OF OUR BRANCHES from the last years. With the upgrade of (git-plugin) 4.0.0-rc, all our branches are being rebuilt (years old branches) after a commit to the project. This is blowing up reports, status reports, slack messages and more. As you can guess, years of builds rerunning is creating the quite the panic. I'm not even sure if the issue is resolved once this backlog of builds is done, because it might take days to get there. What can I do to help? My first instinct is to downgrade back to 3.9.2 (git-plugin) but not sure if that will be worth it. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Stefan Hengelein commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Jesse Glick What? I've just cited what Kevin Phillips wrote and later expanded on issues we're seeing. I'm looking for a strategy to migrate to the new version of the plugin that doesn't involve our CI running for weeks to catch up the mess that was created here. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Jesse Glick commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields does anyone know what other plugins / behaviours might be affected I would start with this, though I would not expect that to be an exhaustive list of affected functionality. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Stefan Hengelein commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields >> Kevin Phillips added a comment - 2016-08-03 17:07 > Question: does anyone know what other plugins / behaviours might be affected by the purging of this Git build data from the build logs? I am trying to figure out what other subsystems > might be affected so I can better test the impact of running this script before rolling it out into a production environment. This "fix" broke SCM polling for our use case: We use an GIT SCM build trigger with a pattern, so all branches that match "FOOBAR-*" are built. With this change now our jenkins tries to rebuild ALL OF OUR BRANCHES from the last years. As you can imagine that is quite the joke and would take weeks until all of our builds ran through. Do you have a solution how to fix this? I'm not so keen on having to stay on the old plugin versions forever. Probably not related but another issue: "ERROR: [GitHub Commit Status Setter] - Cannot retrieve Git metadata for the build, setting build result to FAILURE" Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Mark Waite commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields I have completed my testing of the pull request from Jacob Keller. I need to write some documentation of the change for the release notes, then it will be ready to merge. A week or less after that merge is completed, I intend to deliver the git client plugin 3.0.0-rc and the git plugin 4.0.0-rc builds. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Jacob Keller commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Make sure that commits aren't rebuilt when they've already been covered by a build. Make sure that job start time isn't significantly worse. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Mark Waite commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields The biggest concern (for me) is that you enable the extension and watch for regressions in other areas of the git plugin. Does change history still display as expected? Are builds triggered as expected? Are there places where you detect the absence of the build data? Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Benoit Sigoure commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Thanks for the link, I've installed the plugin [version: 3.9.0-SNAPSHOT (private-69892a21-jenkins)] and disabled the daily job that runs the groovy script. Anything special I should look out for? The issue with the bloat is going to take time to manifest itself again as usually it requires thousands of build.xml files to accumulate. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Jacob Keller commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Yes, please. Additional testing would be a huge benefit. You can even use the one compiled by the pull request build tester found at https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fgit-plugin/detail/PR-578/11/artifacts Specifically https://ci.jenkins.io/job/Plugins/job/git-plugin/job/PR-578/11/artifact/target/git.hpi Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Mark Waite commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Benoit Sigoure have you tried the build from the pull request in your environment? It has been running well for Jacob Keller and would be good to have additional users report their results. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Benoit Sigoure commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields There is a bit of movement on the pull request on GitHub, which gives me hope, but for those stuck with the groovy script to do periodic cleanups, I've measured that the cost is about 65ms per build on a typical dedicated server with 2 spinning drives in a RAID1. This means that the periodic cleanups quickly get increasingly expensive once thousands of builds start to accumulate. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Mark Waite commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields A warning is reported to the log by git plugin 3.1.0 released 4 Mar 2017. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title SCM/JIRA link daemon commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Code changed in jenkins User: Mark Waite Path: src/main/java/hudson/plugins/git/GitSCM.java http://jenkins-ci.org/commit/git-plugin/cae9fb61ffcd2144278aec2255cef897cca569d0 Log: Merge pull request #472 from jglick/buildsByBranchName-JENKINS-19022 JENKINS-19022 Print a warning to the build log when the job seems to be in trouble due to buildsByBranchName bloat Compare: https://github.com/jenkinsci/git-plugin/compare/16e366e146c6...cae9fb61ffcd Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title SCM/JIRA link daemon commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/plugins/git/GitSCM.java http://jenkins-ci.org/commit/git-plugin/5bb7eed6a9231af15e7c0f5a964f3044381a979d Log: JENKINS-19022 Print a warning to the build log when the job seems to be in trouble due to buildsByBranchName bloat. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Arnaud Héritier commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Jesse Glick ok. Thx Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Jesse Glick reopened an issue No, the fix was reverted. Jenkins / JENKINS-19022 GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Change By: Jesse Glick Resolution: Fixed Status: Resolved Reopened Assignee: Nicolas De Loof Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message bec
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Arnaud Héritier resolved as Fixed Was fixed by Nicolas De Loof in Git plugin >= 2.4.0 https://github.com/jenkinsci/git-plugin/pull/312 Jenkins / JENKINS-19022 GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Change By: Arnaud Héritier Status: Open Resolved Assignee: Nicolas De Loof Resolution: Fixed Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Kevin Phillips commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Question: does anyone know what other plugins / behaviours might be affected by the purging of this Git build data from the build logs? I am trying to figure out what other subsystems might be affected so I can better test the impact of running this script before rolling it out into a production environment. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Kevin Phillips commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields I have successfully tested a variation of the groovy scripts mentioned previously to remove the 90% bloat in some of our build logs. One modification I did make was to only process builds other than the last build. This appears to prevent the premature build triggers mentioned earlier since the plugin appears to cache all of the relevant build history for every branch built previously in the last build. So by excluding that one build from the script we prevent jobs from rebuilding every branch immediately afterwards. I believe this solution is preferable to preserving the last built revision being done above. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Arnaud Héritier assigned an issue to Unassigned Jenkins / JENKINS-19022 GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Change By: Arnaud Héritier Assignee: Nicolas De Loof Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Laine Walker-Avina updated an issue Jenkins / JENKINS-19022 GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Change By: Laine Walker-Avina Labels: branch build.xml builddata disk git multi-job performance Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Laine Walker-Avina updated an issue Jenkins / JENKINS-19022 GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Change By: Laine Walker-Avina Labels: branch build.xml builddata git multi-job performance Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title
Laine Walker-Avina edited a comment on JENKINS-19022
Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
+1 To what Kevin said. We have a MultiJob that 3-levels deep with other MultiJobs and MatrixJobs, and new runs typically have 4-5 MB build.xml files with most of that being GitBuildData. We're using the StashBuildTrigger plugin so wiping the build data is ok for us, and we saw the disk usage go from 3GB to 187MB after purging the Git BuildData with a modified version of the script above. As a workaround we have to have the jenkns master on a server with 32 GB of RAM and hope that everything fits into the page cache or heap. {code:java}import hudson.matrix.*import hudson.model.*import com.tikal.jenkins.plugins.multijob.*hudsonInstance = hudson.model.Hudson.instanceallItems = hudsonInstance.getAllItems(AbstractProject.class);// Iterate over all jobs and find the ones that have a hudson.plugins.git.util.BuildData// as an action. We then clean it by removing the useless array action.buildsByBranchName//def numJobs = 0;def runcounter = 0;def cleanGit;cleanGit = { build -> gitActions = build.getActions(hudson.plugins.git.util.BuildData.class); if (gitActions != null) {for (action in gitActions) { action.buildsByBranchName = new HashMap(); hudson.plugins.git.Revision r = action.getLastBuiltRevision(); if (r != null) {for (branch in r.getBranches()) { action.buildsByBranchName.put(branch.getName(), action.lastBuild) } } build.actions.remove(action) build.actions.add(action) build.save(); runcounter++;} }};for (job in allItems) { numJobs++; def counter = 0; for (locbuild in job.getBuilds()) {// It is possible for a build to have multiple BuildData actions// since we can use the Mulitple SCM plugin.def gitActions = locbuild.getActions(hudson.plugins.git.util.BuildData.class)if (gitActions != null) { for (action in gitActions) {counter++; }}if (job instanceof MatrixProject) { runcounter = 0; for (run in locbuild.getRuns()) {cleanGit(run); } if (runcounter > 0) {println(" -->> cleaned: " + runcounter + " runs"); }}if (job instanceof MultiJobProject) { runcounter = 0; cleanGit(locbuild); def recurseSubBuild; recurseSubBuild = { sb -> for(bld in sb) {if(bld.build != null){ cleanGit(bld.build); if(bld.build instanceof MultiJobBuild) {if(bld.build.getSubBuilds().size() != 0) { recurseSubBuild(bld.build.getSubBuilds());} }} } }; recurseSubBuild(locbuild.getSubBuilds()); println("***"); if (runcounter > 0) {println(" -->> cleaned: " + runcounter + " runs"); }}if (counter > 0) { println("-- cleaned: " + counter + " builds");} }}{code}
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title
Laine Walker-Avina commented on JENKINS-19022
Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
+1 To what Kevin said. We have a MultiJob that 3-levels deep with other MultiJobs and MatrixJobs, and new runs typically have 4-5 MB build.xml files with most of that being GitBuildData. We're using the StashBuildTrigger plugin so wiping the build data is ok for us, and we saw the disk usage go from 3GB to 187MB after purging the Git BuildData with a modified version of the script above.
import hudson.matrix.*
import hudson.model.*
import com.tikal.jenkins.plugins.multijob.*
hudsonInstance = hudson.model.Hudson.instance
allItems = hudsonInstance.getAllItems(AbstractProject.class);
// Iterate over all jobs and find the ones that have a hudson.plugins.git.util.BuildData
// as an action.
//
// We then clean it by removing the useless array action.buildsByBranchName
//
def numJobs = 0;
def runcounter = 0;
def cleanGit;
cleanGit = { build ->
gitActions = build.getActions(hudson.plugins.git.util.BuildData.class);
if (gitActions != null) {
for (action in gitActions) {
action.buildsByBranchName = new HashMap();
hudson.plugins.git.Revision r = action.getLastBuiltRevision();
if (r != null) {
for (branch in r.getBranches()) {
action.buildsByBranchName.put(branch.getName(), action.lastBuild)
}
}
build.actions.remove(action)
build.actions.add(action)
build.save();
runcounter++;
}
}
};
for (job in allItems) {
numJobs++;
def counter = 0;
for (locbuild in job.getBuilds()) {
// It is possible for a build to have multiple BuildData actions
// since we can use the Mulitple SCM plugin.
def gitActions = locbuild.getActions(hudson.plugins.git.util.BuildData.class)
if (gitActions != null) {
for (action in gitActions) {
counter++;
}
}
if (job instanceof MatrixProject) {
runcounter = 0;
for (run in locbuild.getRuns()) {
cleanGit(run);
}
if (runcounter > 0) {
println(" -->> cleaned: " + runcounter + " runs");
}
}
if (job instanceof MultiJobProject) {
runcounter = 0;
cleanGit(locbuild);
def recurseSubBuild;
recurseSubBuild = { sb ->
for(bld in sb)
{
if(bld.build != null)
{
cleanGit(bld.build);
if(bld.build instanceof MultiJobBuild)
{
if(bld.build.getSubBuilds().size() != 0) {
recurseSubBuild(bld.build.getSubBuilds());
}
}
}
}
};
recurseSubBuild(locbuild.getSubBuilds());
println("***");
if (runcounter > 0) {
println(" -->> cleaned: " + runcounter + " runs");
}
}
if (counter > 0) {
println("-- cleaned: " + counter + " builds");
}
}
}
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Kevin Phillips edited a comment on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Also, as a corollary to the previous comment, when using the MultiJob plugin the problem being discussed here is further exacerbated because a copy of the Git BuildData node is being stored for each child job (recursively) of the parent multi-job. This results in very large log files being generated (i.e.: 5-10MB or more) in cases when there are many Git branches being reported and many sub-jobs managed by a single multijob. In our test cases approximately 90% of that data is consumed by the Git BuildData sections. Also, in such cases , when a multijob has any more than 20 or 30 builds in the history the Jenkins master can take minutes to load the parent job on the dashboard, and in some extreme cases (i.e.: with hundreds or thousands of builds in the history) it can actually cause the entire dashboard to become unresponsive and time out.This defect has severe consequences and, imo, must be fixed sooner rather than later ... and with an update to the Git plugin, not with a hack or workaround like a script that gets run out-of-process. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Kevin Phillips edited a comment on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Also, as a corollary to the previous comment, when using the MultiJob plugin the problem being discussed here is further exacerbated because a copy of the Git BuildData node is being stored for each child job of the parent multi-job. This results in very large log files being generated (i.e.: 5-10MB or more) in cases when there are many Git branches being reported and many sub-jobs managed by a single multijob. Also, in such cases, when a multijob has any more than 20 or 30 builds in the history the Jenkins master can take minutes to load the parent job on the dashboard, and in some extreme cases (i.e.: with hundreds or thousands of builds in the history) it can actually cause the entire dashboard to become unresponsive and time out.This defect has severe consequences and, imo, must be fixed sooner rather than later ... and with an update to the Git plugin, not with a hack or workaround like a script that gets run out-of-process. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-19022) GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields
Title: Message Title Kevin Phillips commented on JENKINS-19022 Re: GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields Also, as a corollary to the previous comment, when using the MultiJob plugin the problem being discussed here is further exacerbated because a copy of the Git BuildData node is being stored for each child job of the parent multi-job. This results in very large log files being generated (i.e.: 5-10MB or more) in cases when there are many Git branches being reported and many sub-jobs managed by a single multijob. Also, in such cases, when a multijob has any more than 20 or 30 builds in the history the Jenkins master can take minutes to load the parent job on the dashboard, and in some extreme cases it can actually cause the dashboard to become unresponsive and time out. This defect has severe consequences and, imo, must be fixed sooner rather than later ... and with an update to the Git plugin, not with a hack or workaround like a script that gets run out-of-process. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
