Re: Slow dashboard loading for jobs with many compiler warnings

[Ruben van Staveren]

> cppcheck JENKINS-17363
>  Ludicrously slow load time [with 
> lazyloading]
>
actually I meant 
JENKINS-19437
 Implement load on demand functionality in 
Cppcheck

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Slow dashboard loading for jobs with many compiler warnings

[Ruben van Staveren]

Hi Ullrich, 

Thank you for your fast answer!

On Wednesday, October 30, 2013 6:47:33 PM UTC+1, Ullrich Hafner wrote:
>
>
> This seems to be the problem! The build.xml file should be as small as 
> possible since these files are always read (and are always kept in memory). 
> Seems that these plug-ins do not use an additional file to store the data: 
> this is a bug (or at least a bad design). Can you please file a new bug 
> report for both plug-ins? 
>

I found 2 earlier bug reports, and added to them

cppcheck JENKINS-17363  
Ludicrously 
slow load time [with 
lazyloading]
sloccount JENKINS-4769  
Memory 
consumption is huge 
 

>
> Is the performance better if you disable the cppcheck and sloccount 
> plugins? What other warnings are you aggregating in the analysis.xml file? 
> Are these produced by the warnings plugin?
>

I removed the sloccount data and trimmed the cppcheck data as it was too 
valuable to toss away, but won't be checking for style errors there 
anymore. The build.xml files now only max out at 300K instead of 13MB.

After reloading Jenkins I saw that the heap was reduced from ~700MB to 
~300MB and the "All" view is fast again. As you suggested, the compiler 
warning with it's separate analysis.xml is not the culprit and follows the 
right approach. I suggested in those two issues that these plugins also 
follow that strategy.

Best Regards,
Ruben




-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Slow dashboard loading for jobs with many compiler warnings

[Ulli Hafner]

Am 30.10.2013 um 16:47 schrieb Ruben van Staveren :

> Hi,
> 
> I am a Jenkins administrator for a site with not that many jobs, but these 
> jobs keep a constant rate of massive compiler warnings (~9000 - 1).
> For a particular job for a software with multiple versions the per build 
> build.xml log file that is generated and kept by Jenkins is around ~13MB. The 
> analysis.xml containing the compiler warnings is around ~ 8.5 MB. With 14 
> jobs each keeping 14 days of history brings the total Jenkins has to scan to 
> ~2.1 G.
> 

What dashboards are visible? Most (if not all) compiler warnings graphs do not 
read the analysis.xml file at all. These files are only read when a user drills 
down to a warning. 


> Only be removing jobs from the dashboard view made it responsive again. I 
> still observed some lag switching between other views, but when I had reduced 
> the amount of jobs it seemed Jenkins was able to keep everything in core. The 
> "All" pane is still unusable due to this.
> 
> The bulk of the build.xml seems to consist out of 
> ~ 89000 lines of cppcheck data
> ~ 87000 lines of sloccount data
> 

This seems to be the problem! The build.xml file should be as small as possible 
since these files are always read (and are always kept in memory). Seems that 
these plug-ins do not use an additional file to store the data: this is a bug 
(or at least a bad design). Can you please file a new bug report for both 
plug-ins? 

Is the performance better if you disable the cppcheck and sloccount plugins? 
What other warnings are you aggregating in the analysis.xml file? Are these 
produced by the warnings plugin?
> 
> Trimming these sections from the build.xml leaves it only ~ 430 lines
> 
> The analysis.xml is around ~183500 lines.
> 
> Except for the team fixing the compiler warnings, reducing the amount of jobs 
> per dashboard view, shorter retention periods for build data, no graphs in 
> dashboards, deleting old jobs, is there any way in where Jenkins can be tuned 
> to handle these warning heavy jobs ? 
> 
> Best Regards,
>  Ruben van Staveren
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



signature.asc
Description: Message signed with OpenPGP using GPGMail

Slow dashboard loading for jobs with many compiler warnings

[Ruben van Staveren]

Hi,

I am a Jenkins administrator for a site with not that many jobs, but these 
jobs keep a constant rate of massive compiler warnings (~9000 - 1).
For a particular job for a software with multiple versions the per build 
build.xml log file that is generated and kept by Jenkins is around ~13MB. 
The analysis.xml containing the compiler warnings is around ~ 8.5 MB. With 
14 jobs each keeping 14 days of history brings the total Jenkins has to 
scan to ~2.1 G.

Only be removing jobs from the dashboard view made it responsive again. I 
still observed some lag switching between other views, but when I had 
reduced the amount of jobs it seemed Jenkins was able to keep everything in 
core. The "All" pane is still unusable due to this.

The bulk of the build.xml seems to consist out of 

~ 89000 lines of cppcheck data

~ 87000 lines of sloccount data

Trimming these sections from the build.xml leaves it only ~ 430 lines


The analysis.xml is around ~183500 lines.


Except for the team fixing the compiler warnings, reducing the amount of 
jobs per dashboard view, shorter retention periods for build data, no 
graphs in dashboards, deleting old jobs, is there any way in where Jenkins 
can be tuned to handle these warning heavy jobs ? 


Best Regards,

 Ruben van Staveren




-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.