Hum, It might not be stuck, actually, but just spending very long time to compress the log.
In theory, the log compression is not waited for, though. Note that if you stop a build that is waiting for a lost deferred, this will have no effect as you describe. You got absolutly no exception in twisted.log? Pierre Le ven. 26 août 2016 à 17:06, Neil Gilmore <ngilm...@grammatech.com> a écrit : > I'm currently looking at a step with 3 logs: > 81531 lines > 489285 lines > 489311 lines > > An earlier successful run would have that first log at 244080 lines. > > This particular build is stuck, though. :( (which is why I'm looking at > it.) (A bit off-topic, but I tried stopping it. The last step is marked > cancelled, but that's the only effect.) > > > Neil Gilmore > grammatech.com > > > On 8/26/2016 6:25 AM, Pierre Tardy wrote: > > > The ram looks like sufficient, it might be a good test to try and increase > the number of cpu for that VM. > In your trace, I can see the use of up to 7 threads at the same time, so > you might gain by going 8 CPUs > > Also, make sure that your VM host is not overbooked. In my experience of > using VMware VMs provided by IT, overbooking has been a source of > inexplicable performance issues. > > 12k lines is a lot, but buildbot shall support this kind of load without > issue. > > Le ven. 26 août 2016 à 12:05, Francesco Di Mizio < > francescodimi...@gmail.com> a écrit : > >> It's a vmware virtual machine with 4 GIGs RAM and 4 CPUs@3Ghz. It runs, >> among other marginal things, 2 docker containers - one for the buildbot and >> one for the postgres db. >> >> The most beefy logs have around 12K lines. Is it too much? >> Also some other logs are read from the worker's filesystem and added as >> additional logs. >> >> On Fri, Aug 26, 2016 at 11:51 AM, Pierre Tardy <tar...@gmail.com> wrote: >> >>> Cool! >>> I can indeed see 3 spikes. >>> >>> Looks related to logs and logs compression. >>> >>> What is the HW spec of your master machine? >>> How much log does your build generate? >>> >>> Pierre >>> >>> Le ven. 26 août 2016 à 11:42, Francesco Di Mizio < >>> francescodimi...@gmail.com> a écrit : >>> >>>> Pierre, >>>> >>>> I enabled it, waited 1 min and saw the spike, then stopped after a few >>>> secs. Attached the json. >>>> Awesome tool btw, work wonders! >>>> >>>> On Thu, Aug 25, 2016 at 1:14 PM, Pierre Tardy <tar...@gmail.com> wrote: >>>> >>>>> You can try to hit the button in the morning with a two hours gather >>>>> period, and hope that you see the spike during that period.. >>>>> >>>>> >>>>> Le jeu. 25 août 2016 à 12:17, Francesco Di Mizio < >>>>> francescodimi...@gmail.com> a écrit : >>>>> >>>>>> Thanks a lot! Pierre I will def will give it a shot. I am not sure >>>>>> I'll be able to smash that 'start recording' button as the UI >>>>>> isusuallystuck when the CPU spikes. Updates to come! >>>>>> >>>>>> On Thu, Aug 25, 2016 at 10:45 AM, Pierre Tardy <tar...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Francesco, >>>>>>> >>>>>>> I spent some time in order to implement a profiler plugin for >>>>>>> buildbot >>>>>>> >>>>>>> You can give it a look, and send your profile.json file if you need >>>>>>> more analysis from me. >>>>>>> https://github.com/tardyp/buildbot_profiler >>>>>>> >>>>>>> Regards, >>>>>>> Pierre >>>>>>> >>>>>>> >>>>>>> Le mer. 24 août 2016 à 22:43, Francesco Di Mizio < >>>>>>> francescodimi...@gmail.com> a écrit : >>>>>>> >>>>>>>> I've tried and it's not an easy task because of my Win into Vagrant >>>>>>>> into Docker setup. >>>>>>>> I'll try again soon when I get a Linux box! >>>>>>>> >>>>>>>> On Fri, Aug 19, 2016 at 5:54 PM, Vasily <just.one....@yandex.ru> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Umm, no. VTune has Python support starting 2017 Beta, and, well, >>>>>>>>> it was my team (at Intel) work actually :-) >>>>>>>>> >>>>>>>>> P.S. I'm from Intel, too. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Vasily >>>>>>>>> 19 авг. 2016 г. 18:17 пользователь "Francesco Di Mizio" < >>>>>>>>> francescodimi...@gmail.com> написал: >>>>>>>>> >>>>>>>>> I had thought you were making fun of Intel somehow ;) >>>>>>>>>> >>>>>>>>>> On Aug 19, 2016 5:07 PM, "Pierre Tardy" <tar...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> ahah >>>>>>>>>> >>>>>>>>>> I though this was a taunt on me being employed by Intel. >>>>>>>>>> I actually had mitigated experience with vtune few years ago, and >>>>>>>>>> didn't know they had python support until then. >>>>>>>>>> Being an opensource guy, I usually neglegate to look at >>>>>>>>>> proprietary stuff. >>>>>>>>>> >>>>>>>>>> Pierre >>>>>>>>>> >>>>>>>>>> Le ven. 19 août 2016 à 12:18, Vasily <just.one....@yandex.ru> a >>>>>>>>>> écrit : >>>>>>>>>> >>>>>>>>>>> I'm again suggesting to look into Python profiling capabilities >>>>>>>>>>> of Intel® VTune™ Amplifier. It could run statistical profiling for >>>>>>>>>>> a long >>>>>>>>>>> time and display CPU usage over time, so the developer can look at >>>>>>>>>>> specific >>>>>>>>>>> time range where CPU usage was too high and see which functions were >>>>>>>>>>> executed. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Vasily >>>>>>>>>>> 19 авг. 2016 г. 11:57 пользователь "Pierre Tardy" < >>>>>>>>>>> tar...@gmail.com> написал: >>>>>>>>>>> >>>>>>>>>>> Hi Francesco, >>>>>>>>>>>> >>>>>>>>>>>> Your described setup looks sane to me. >>>>>>>>>>>> >>>>>>>>>>>> The problems we are trying to catch are cpu spikes, as far as I >>>>>>>>>>>> understand, which does not happen for very long, but are very >>>>>>>>>>>> annoying for >>>>>>>>>>>> users, as it is blocking the reactor. >>>>>>>>>>>> >>>>>>>>>>>> This problem is not easy to see in the profile you sent, as >>>>>>>>>>>> this profile is over long time, so we see the average of each >>>>>>>>>>>> method during >>>>>>>>>>>> the day and not the spikes. >>>>>>>>>>>> >>>>>>>>>>>> What would really be needed is a on-demand profiler which would >>>>>>>>>>>> detect cpu spikes and only log the stack traces during those times. >>>>>>>>>>>> >>>>>>>>>>>> Here is a nice blog pst explaining why statistic profiling is >>>>>>>>>>>> cool and easy to implement in python. >>>>>>>>>>>> https://nylas.com/blog/performance >>>>>>>>>>>> >>>>>>>>>>>> For 0.9.1 I want to concentrate on scalability, and write a >>>>>>>>>>>> debugging ui plugin based on those ideas (and probably code) >>>>>>>>>>>> >>>>>>>>>>>> That would be great if your team can help on that matter. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Pierre >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>> >> > > _______________________________________________ > users mailing > listusers@buildbot.nethttps://lists.buildbot.net/mailman/listinfo/users > > _______________________________________________ > users mailing list > users@buildbot.net > https://lists.buildbot.net/mailman/listinfo/users
_______________________________________________ users mailing list users@buildbot.net https://lists.buildbot.net/mailman/listinfo/users