Re: Fingerprinting performance
I removed the include pattern "jobs/*/promotions/*/config.xml", now the fingerprinting takes no time :D. I think I can conclude that the scan pattern really slows things down when there are many jobs. Something interesting is, the scandir is triggered by FingerPrint.save(), SCM sync config plugin may be too aggressive here when SaveableListener.fireOnChange being triggered. I think my problem is solved. Thank your very much Daniel for the great help. -jv Am 11.07.2013 17:16, schrieb John Vacz: The SCM sync configuration: Am 11.07.2013 17:10, schrieb John Vacz: Executor #1 for Slave16 : executing SD-2112 #3 "Executor #1 for comitdev16 : executing SD-2112 #3" Id=62 Group=main RUNNABLE at java.io.UnixFileSystem.list(Native Method) at java.io.File.list(File.java:973) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1257) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1211) at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:1030) at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:909) - locked org.apache.tools.ant.DirectoryScanner@34cf6c49 at hudson.plugins.scm_sync_configuration.strategies.model.PatternsEntityMatcher.matchingFilesFrom(PatternsEntityMatcher.java:41) at hudson.plugins.scm_sync_configuration.strategies.model.PatternsEntityMatcher.matches(PatternsEntityMatcher.java:25) at hudson.plugins.scm_sync_configuration.strategies.AbstractScmSyncStrategy.isSaveableApplicable(AbstractScmSyncStrategy.java:53) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationPlugin.getStrategyForSaveable(ScmSyncConfigurationPlugin.java:277) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationSaveableListener.onChange(ScmSyncConfigurationSaveableListener.java:22) at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:78) at hudson.model.Fingerprint.save(Fingerprint.java:862) - locked hudson.model.Fingerprint@281c2370 at hudson.model.Fingerprint.(Fingerprint.java:597) at hudson.model.FingerprintMap.create(FingerprintMap.java:90) at hudson.model.FingerprintMap.create(FingerprintMap.java:45) at hudson.util.KeyedDataStorage.get(KeyedDataStorage.java:156) at hudson.model.FingerprintMap.get(FingerprintMap.java:79) at hudson.model.FingerprintMap.get(FingerprintMap.java:45) at hudson.util.KeyedDataStorage.getOrCreate(KeyedDataStorage.java:108) at hudson.model.FingerprintMap.getOrCreate(FingerprintMap.java:65) at hudson.tasks.Fingerprinter$1Record.addRecord(Fingerprinter.java:210) at hudson.tasks.Fingerprinter.record(Fingerprinter.java:254) at hudson.tasks.Fingerprinter.perform(Fingerprinter.java:133) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776) at hudson.model.Build$BuildExecution.post2(Build.java:183) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726) at hudson.model.Run.execute(Run.java:1601) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:241) Thats the one segment of the therad dump. Looks like scm sync config plugin is scanning directories a lot? Am 11.07.2013 16:45, schrieb John Vacz: The artifacts: 30MB x 1, 6.5MB x 9, 1MB x 1, others are 5 plain text files between 0.1 ~ 10KB. I will look into the threadDump... Am 11.07.2013 16:33, schrieb Daniel Beck: How big are these artifacts? At /threadDump, you can access live stack traces. Maybe look for something fingerprint related during those 5 minutes, it could give you a hint what is taking so long. Copy artifact works without fingerprinting in the source project, but always calculates its own when copying. See JENKINS-12134 and JENKINS-18653. On 11.07.2013, at 15:06, John Vacz wrote: Recently the fingerprinting of our jubs is becoming very slow.Per build we have 16 artifacts to be fingerprinted, now that alone lasts ~5 minutes. Since we are using copy artifact plugin very heavily, the situation is becoming even worse - the slowness adds up. Unfortunately I cannot tell from which Jenkins version this happens, I only noticed this slowness in several weeks ~ around version 1.51x, but the p
Re: Fingerprinting performance
The SCM sync configuration: Am 11.07.2013 17:10, schrieb John Vacz: Executor #1 for Slave16 : executing SD-2112 #3 "Executor #1 for comitdev16 : executing SD-2112 #3" Id=62 Group=main RUNNABLE at java.io.UnixFileSystem.list(Native Method) at java.io.File.list(File.java:973) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1257) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1211) at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:1030) at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:909) - locked org.apache.tools.ant.DirectoryScanner@34cf6c49 at hudson.plugins.scm_sync_configuration.strategies.model.PatternsEntityMatcher.matchingFilesFrom(PatternsEntityMatcher.java:41) at hudson.plugins.scm_sync_configuration.strategies.model.PatternsEntityMatcher.matches(PatternsEntityMatcher.java:25) at hudson.plugins.scm_sync_configuration.strategies.AbstractScmSyncStrategy.isSaveableApplicable(AbstractScmSyncStrategy.java:53) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationPlugin.getStrategyForSaveable(ScmSyncConfigurationPlugin.java:277) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationSaveableListener.onChange(ScmSyncConfigurationSaveableListener.java:22) at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:78) at hudson.model.Fingerprint.save(Fingerprint.java:862) - locked hudson.model.Fingerprint@281c2370 at hudson.model.Fingerprint.(Fingerprint.java:597) at hudson.model.FingerprintMap.create(FingerprintMap.java:90) at hudson.model.FingerprintMap.create(FingerprintMap.java:45) at hudson.util.KeyedDataStorage.get(KeyedDataStorage.java:156) at hudson.model.FingerprintMap.get(FingerprintMap.java:79) at hudson.model.FingerprintMap.get(FingerprintMap.java:45) at hudson.util.KeyedDataStorage.getOrCreate(KeyedDataStorage.java:108) at hudson.model.FingerprintMap.getOrCreate(FingerprintMap.java:65) at hudson.tasks.Fingerprinter$1Record.addRecord(Fingerprinter.java:210) at hudson.tasks.Fingerprinter.record(Fingerprinter.java:254) at hudson.tasks.Fingerprinter.perform(Fingerprinter.java:133) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776) at hudson.model.Build$BuildExecution.post2(Build.java:183) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726) at hudson.model.Run.execute(Run.java:1601) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:241) Thats the one segment of the therad dump. Looks like scm sync config plugin is scanning directories a lot? Am 11.07.2013 16:45, schrieb John Vacz: The artifacts: 30MB x 1, 6.5MB x 9, 1MB x 1, others are 5 plain text files between 0.1 ~ 10KB. I will look into the threadDump... Am 11.07.2013 16:33, schrieb Daniel Beck: How big are these artifacts? At /threadDump, you can access live stack traces. Maybe look for something fingerprint related during those 5 minutes, it could give you a hint what is taking so long. Copy artifact works without fingerprinting in the source project, but always calculates its own when copying. See JENKINS-12134 and JENKINS-18653. On 11.07.2013, at 15:06, John Vacz wrote: Recently the fingerprinting of our jubs is becoming very slow.Per build we have 16 artifacts to be fingerprinted, now that alone lasts ~5 minutes. Since we are using copy artifact plugin very heavily, the situation is becoming even worse - the slowness adds up. Unfortunately I cannot tell from which Jenkins version this happens, I only noticed this slowness in several weeks ~ around version 1.51x, but the problem might well be irrelevant to the jenkins version. Some more background information: Jenkins 1.518 on Debian 6 64bit and built-in Winstone we have a standard job template, each git branch has one jenkins job respectively, at the moment we have ~240 jobs (active + disabled). If one branch is done, the jenkins job is disabled but not deleted. So we have many jobs with multiple builds, and we do limit the perserved artifacts (max. 2 builds per job). Beside that, we have 2 long-live jobs, together
Re: Fingerprinting performance
Executor #1 for Slave16 : executing SD-2112 #3 "Executor #1 for comitdev16 : executing SD-2112 #3" Id=62 Group=main RUNNABLE at java.io.UnixFileSystem.list(Native Method) at java.io.File.list(File.java:973) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1257) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287) at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1211) at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:1030) at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:909) - locked org.apache.tools.ant.DirectoryScanner@34cf6c49 at hudson.plugins.scm_sync_configuration.strategies.model.PatternsEntityMatcher.matchingFilesFrom(PatternsEntityMatcher.java:41) at hudson.plugins.scm_sync_configuration.strategies.model.PatternsEntityMatcher.matches(PatternsEntityMatcher.java:25) at hudson.plugins.scm_sync_configuration.strategies.AbstractScmSyncStrategy.isSaveableApplicable(AbstractScmSyncStrategy.java:53) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationPlugin.getStrategyForSaveable(ScmSyncConfigurationPlugin.java:277) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationSaveableListener.onChange(ScmSyncConfigurationSaveableListener.java:22) at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:78) at hudson.model.Fingerprint.save(Fingerprint.java:862) - locked hudson.model.Fingerprint@281c2370 at hudson.model.Fingerprint.(Fingerprint.java:597) at hudson.model.FingerprintMap.create(FingerprintMap.java:90) at hudson.model.FingerprintMap.create(FingerprintMap.java:45) at hudson.util.KeyedDataStorage.get(KeyedDataStorage.java:156) at hudson.model.FingerprintMap.get(FingerprintMap.java:79) at hudson.model.FingerprintMap.get(FingerprintMap.java:45) at hudson.util.KeyedDataStorage.getOrCreate(KeyedDataStorage.java:108) at hudson.model.FingerprintMap.getOrCreate(FingerprintMap.java:65) at hudson.tasks.Fingerprinter$1Record.addRecord(Fingerprinter.java:210) at hudson.tasks.Fingerprinter.record(Fingerprinter.java:254) at hudson.tasks.Fingerprinter.perform(Fingerprinter.java:133) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776) at hudson.model.Build$BuildExecution.post2(Build.java:183) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726) at hudson.model.Run.execute(Run.java:1601) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:241) Thats the one segment of the therad dump. Looks like scm sync config plugin is scanning directories a lot? Am 11.07.2013 16:45, schrieb John Vacz: The artifacts: 30MB x 1, 6.5MB x 9, 1MB x 1, others are 5 plain text files between 0.1 ~ 10KB. I will look into the threadDump... Am 11.07.2013 16:33, schrieb Daniel Beck: How big are these artifacts? At /threadDump, you can access live stack traces. Maybe look for something fingerprint related during those 5 minutes, it could give you a hint what is taking so long. Copy artifact works without fingerprinting in the source project, but always calculates its own when copying. See JENKINS-12134 and JENKINS-18653. On 11.07.2013, at 15:06, John Vacz wrote: Recently the fingerprinting of our jubs is becoming very slow.Per build we have 16 artifacts to be fingerprinted, now that alone lasts ~5 minutes. Since we are using copy artifact plugin very heavily, the situation is becoming even worse - the slowness adds up. Unfortunately I cannot tell from which Jenkins version this happens, I only noticed this slowness in several weeks ~ around version 1.51x, but the problem might well be irrelevant to the jenkins version. Some more background information: Jenkins 1.518 on Debian 6 64bit and built-in Winstone we have a standard job template, each git branch has one jenkins job respectively, at the moment we have ~240 jobs (active + disabled). If one branch is done, the jenkins job is disabled but not deleted. So we have many jobs with multiple builds, and we do limit the perserved artifacts (max. 2 builds per job). Beside that, we have 2 long-live jobs, together ~ 700 builds. The number of artifacts sum up could be quite large. Alth
Re: Fingerprinting performance
The artifacts: 30MB x 1, 6.5MB x 9, 1MB x 1, others are 5 plain text files between 0.1 ~ 10KB. I will look into the threadDump... Am 11.07.2013 16:33, schrieb Daniel Beck: How big are these artifacts? At /threadDump, you can access live stack traces. Maybe look for something fingerprint related during those 5 minutes, it could give you a hint what is taking so long. Copy artifact works without fingerprinting in the source project, but always calculates its own when copying. See JENKINS-12134 and JENKINS-18653. On 11.07.2013, at 15:06, John Vacz wrote: Recently the fingerprinting of our jubs is becoming very slow.Per build we have 16 artifacts to be fingerprinted, now that alone lasts ~5 minutes. Since we are using copy artifact plugin very heavily, the situation is becoming even worse - the slowness adds up. Unfortunately I cannot tell from which Jenkins version this happens, I only noticed this slowness in several weeks ~ around version 1.51x, but the problem might well be irrelevant to the jenkins version. Some more background information: Jenkins 1.518 on Debian 6 64bit and built-in Winstone we have a standard job template, each git branch has one jenkins job respectively, at the moment we have ~240 jobs (active + disabled). If one branch is done, the jenkins job is disabled but not deleted. So we have many jobs with multiple builds, and we do limit the perserved artifacts (max. 2 builds per job). Beside that, we have 2 long-live jobs, together ~ 700 builds. The number of artifacts sum up could be quite large. Although I do delete (linux shell) the artifacts periodically (every several months to ~1 year) , I didnt touch the fingerprints/ directory ever since we first adopted Hudson (5+ years). Now the fingerprints/ contains 245M data. I suspect that the size of the fingerprint database may be the main culprit, but thats only my speculation without any hard evidence. It seems that Jenkins garbage collects them [1] if builds are deleted within/through Jenkins. But is the fingerprint database being generally maintained? Does the size of the fingerprint database really matter? If yes, can I just delete the whole fingerprints/ without breaking the copy-artifact plugin (the ability to deploy a previous build using copy-artifact is crucial for us)? Or how can I reduce the size? I might be looking at a complete wrong direction, so any help/idea is very much appreciated. -jv [1] https://issues.jenkins-ci.org/browse/JENKINS-18417 -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Fingerprinting performance
How big are these artifacts? At /threadDump, you can access live stack traces. Maybe look for something fingerprint related during those 5 minutes, it could give you a hint what is taking so long. Copy artifact works without fingerprinting in the source project, but always calculates its own when copying. See JENKINS-12134 and JENKINS-18653. On 11.07.2013, at 15:06, John Vacz wrote: > Recently the fingerprinting of our jubs is becoming very slow.Per build we > have 16 artifacts to be fingerprinted, now that alone lasts ~5 minutes. Since > we are using copy artifact plugin very heavily, the situation is becoming > even worse - the slowness adds up. > > Unfortunately I cannot tell from which Jenkins version this happens, I only > noticed this slowness in several weeks ~ around version 1.51x, but the > problem might well be irrelevant to the jenkins version. > > Some more background information: > Jenkins 1.518 on Debian 6 64bit and built-in Winstone > we have a standard job template, each git branch has one jenkins job > respectively, at the moment we have ~240 jobs (active + disabled). If one > branch is done, the jenkins job is disabled but not deleted. So we have many > jobs with multiple builds, and we do limit the perserved artifacts (max. 2 > builds per job). Beside that, we have 2 long-live jobs, together ~ 700 > builds. The number of artifacts sum up could be quite large. Although I do > delete (linux shell) the artifacts periodically (every several months to ~1 > year) , I didnt touch the fingerprints/ directory ever since we first adopted > Hudson (5+ years). Now the fingerprints/ contains 245M data. > > I suspect that the size of the fingerprint database may be the main culprit, > but thats only my speculation without any hard evidence. It seems that > Jenkins garbage collects them [1] if builds are deleted within/through > Jenkins. But is the fingerprint database being generally maintained? > > Does the size of the fingerprint database really matter? If yes, can I just > delete the whole fingerprints/ without breaking the copy-artifact plugin (the > ability to deploy a previous build using copy-artifact is crucial for us)? Or > how can I reduce the size? > > I might be looking at a complete wrong direction, so any help/idea is very > much appreciated. > > -jv > > [1] https://issues.jenkins-ci.org/browse/JENKINS-18417 > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
