Pierre many thanks for that. I do not remember why the force scheduler api didnt suite my needs, that was my first approach. I will try the new snippet when I get to work in a few hours.
On Wed, Aug 10, 2016 at 11:18 PM, Pierre Tardy <tar...@gmail.com> wrote: > Hi, > > After carefully looking at your code, it does not look to have obvious > performance flaw. > I think it would have been cheaper for you to just use the forcescheduler > REST api, but if that code works for you this is totally fine. > > I saw that you did not manage to get sample in your prod as well, so I did > look at my code again, and now I am able to see the samples. > > statprof does not like to stop before start > > Please try again with this code: > > > import statprof > from twisted.application.internet import TimerService > from buildbot.util.service import BuildbotService > class BuildbotTimerService(TimerService, BuildbotService): > name = "timer" > def __init__(self): > BuildbotService.__init__(self) > TimerService.__init__(self, 10, self.dump_stats) > self.started = False > def dump_stats(self): > if not self.started: > statprof.start() > self.started = True > return > statprof.stop() > with open("profile.txt", "w") as f: > statprof.display(f) > statprof.display() > statprof.start() > > c['services'] = [ BuildbotTimerService() ] > > > > Le mer. 10 août 2016 à 14:14, Francesco Di Mizio < > francescodimi...@gmail.com> a écrit : > >> Many thanks. Find it attached. >> >> It's a tiny http server. People use a web page to send an HTTP request to >> it to kick off builds. >> >> >> On Wed, Aug 10, 2016 at 1:55 PM, Pierre Tardy <tar...@gmail.com> wrote: >> >>> I'll take a look. >>> >>> Le mer. 10 août 2016 12:45, Francesco Di Mizio < >>> francescodimi...@gmail.com> a écrit : >>> >>>> I am starting to think it may be due to my own custom scheduler. If I >>>> uploaded the code somewhere could anybody take a look? it's a 150 lines py >>>> script. >>>> >>>> On Wed, Aug 10, 2016 at 12:10 PM, Francesco Di Mizio < >>>> francescodimi...@gmail.com> wrote: >>>> >>>>> Thanks for that Pierre. I have it on on production and the >>>>> performances do not degrade indeed. >>>>> >>>>> buildbot@e2123a50d1fb:~$ cat profile.txt >>>>> No samples recorded. >>>>> >>>>> I need to study statprof a bit now ;) >>>>> >>>>> On Wed, Aug 10, 2016 at 11:20 AM, Vasily <vassn...@gmail.com> wrote: >>>>> >>>>>> One can also use Intel® VTune™, you know... :-) >>>>>> >>>>>> Thanks, >>>>>> Vasily >>>>>> 09 авг. 2016 г. 23:56 пользователь "Pierre Tardy" <tar...@gmail.com> >>>>>> написал: >>>>>> >>>>>>> You can add following snippet to your master.cfg. It looks statprof >>>>>>> does not work on osx, so I cannot test it to the end, but I know similar >>>>>>> did work on my prod >>>>>>> >>>>>>> >>>>>>> import statprof >>>>>>> from twisted.application.internet import TimerService >>>>>>> from buildbot.util.service import BuildbotService >>>>>>> class BuildbotTimerService(TimerService, BuildbotService): >>>>>>> name = "timer" >>>>>>> def __init__(self): >>>>>>> BuildbotService.__init__(self) >>>>>>> TimerService.__init__(self, 10, self.dump_stats) >>>>>>> def dump_stats(self): >>>>>>> statprof.stop() >>>>>>> with open("profile.txt", "w") as f: >>>>>>> statprof.display(f) >>>>>>> statprof.start() >>>>>>> >>>>>>> c['services'] = [ BuildbotTimerService() ] >>>>>>> >>>>>>> >>>>>>> Le mar. 9 août 2016 à 21:44, Francesco Di Mizio < >>>>>>> francescodimi...@gmail.com> a écrit : >>>>>>> >>>>>>>> Pierre, >>>>>>>> >>>>>>>> if you can enlighten on how you intend to use statprof, I will be >>>>>>>> happy to give it a try. >>>>>>>> This problem here is on top of my list right now as it's making my >>>>>>>> prod env unusable. >>>>>>>> >>>>>>>> On Tue, Aug 9, 2016 at 8:25 PM, Pierre Tardy <tar...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Please note that cprofile is very intrusive and will lead to about >>>>>>>>> 3x slower code. >>>>>>>>> >>>>>>>>> So I would not run that in prod. This is why I prefer statprof >>>>>>>>> which is using a non intrusive method to get the stats >>>>>>>>> >>>>>>>>> Le mar. 9 août 2016 19:15, Francesco Di Mizio < >>>>>>>>> francescodimi...@gmail.com> a écrit : >>>>>>>>> >>>>>>>>>> On a side note I have managed to get twistd to dump a blob I can >>>>>>>>>> then load and analyze. Got to see if I can run this in production. >>>>>>>>>> If anybody is curious the following worked for me: >>>>>>>>>> twistd --savestats -n --profiler=cprofile --profile=/crcdata/ >>>>>>>>>> profile.stats -y ./buildbot.tac >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Aug 9, 2016 at 5:21 PM, Francesco Di Mizio < >>>>>>>>>> francescodimi...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Well that's expected to an extent - p4 poller is running 'p4 >>>>>>>>>>> changes' every pollinterval seconds. >>>>>>>>>>> >>>>>>>>>>> Anyway just as a test I've tried to disable the poller and still >>>>>>>>>>> at times I am seeing the CPU spiking up to above 100%. I believe I >>>>>>>>>>> really >>>>>>>>>>> need to profile this somehow. >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 9, 2016 at 5:08 PM, Dan Kegel <d...@kegel.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Also watch 'top' and see if poller processes are hogging >>>>>>>>>>>> resources. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> users@buildbot.net >>>>>>> https://lists.buildbot.net/mailman/listinfo/users >>>>>>> >>>>>> >>>>> >>>> >>
_______________________________________________ users mailing list users@buildbot.net https://lists.buildbot.net/mailman/listinfo/users