>Regarding the profiler, I also had some problems running it for a longer time, so if you find a fix, please share it with me. It was a bytes versus string issue indeed in the python parts.
I did struggle to build the project again as expected, but managed to hack something eventually. see https://pypi.org/project/buildbot-profiler/1.3.1/ I did verify the profile duration is now working. Regards Pierre Le mer. 13 janv. 2021 à 12:49, Vlad Bogolin <v...@mariadb.org> a écrit : > Hi, > > So the changes issue I was referring to doesn't seem to be fixed (I > checked the latest Buildbot code) and looked back over my changes. The > problem that I identified is here > https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L320 > because is some cases there would be an unmatched_filter or > unmatched_order. *So if this is indeed the case, you should see the > warning in the buildbot log file:* > > "Warning: limited data api query is not backed by db because of following > filters..." as defined here > https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L322 > > There are two reasons why this happens. One, an incomplete definition of > the fieldMapping for the data/changes.py and this would be the fix: > > +++ b/master/buildbot/data/changes.py > @@ -42,6 +42,19 @@ class FixerMixin: > return change > fieldMapping = { > 'changeid': 'changes.id', > + 'author': 'changes.author', > + 'committer': 'changes.committer', > + 'comments': 'changes.comments', > + 'branch': 'changes.branch', > + 'revision': 'changes.revision', > + 'revlink': 'changes.revlink', > + 'when_timestamp': 'changes.when_timestamp', > + 'category': 'changes.category', > + 'repository': 'changes.repository', > + 'codebase': 'changes.codebase', > + 'project': 'changes.project', > + 'sourcestampid': 'changes.sourcestampid', > + 'parent_changeids': 'changes.parent_changeids', > } > > and two not having all the columns in the select statement in order to be > able to check for matched/unmatched filtering or ordering as it is done > here > https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L269. > My solution was to get all the fields from the changes table from the > database (don't know if this is the best approach) as you can see here: > > +++ b/master/buildbot/db/changes.py > @@ -249,7 +249,7 @@ class > ChangesConnectorComponent(base.DBConnectorComponent): > def thd(conn): > # get the changeids from the 'changes' table > changes_tbl = self.db.model.changes > - q = sa.select([changes_tbl.c.changeid]) > + q = changes_tbl.select() > > However, since only the changeid would be needed, don't sure if this would > be the right approach. Also, I would suspect some missing tests because I > would expect this to be a pretty common use case. > > Regarding the profiler, I also had some problems running it for a longer > time, so if you find a fix, please share it with me. > > Cheers, > Vlad > > On Wed, Jan 13, 2021 at 1:33 PM Pierre Tardy <tar...@gmail.com> wrote: > >> >> I am not sure why this wouldn't work. I vaguely recall there was an issue >> there, but can't figure it out staring at the code. >> As this is quite ancient, I am not sure the JS part will even build >> anymore :-/ >> >> You can change the default values at that line in the python >> >> https://github.com/tardyp/buildbot_profiler/blob/master/buildbot_profiler/api.py#L193 >> I think you will be able to force them by editing this file inside your >> virtualenv.. >> >> >> Regards >> Pierre >> >> >> Le mer. 13 janv. 2021 à 11:46, Yngve N. Pettersen <yn...@vivaldi.com> a >> écrit : >> >>> > I insist on the buildbot profiler. What I was saying before is that you >>> > need to hit the record button before the problem appears, and put a >>> large >>> > enough record time to be sure to catch a spike. >>> > Then, you will be able to zoom to the cpu spike and catch the issue >>> > precisely. >>> > >>> > If the spike is in the order of minutes like you said, you can >>> configure >>> > it >>> > like this and get enough samples to get enough evidence to where the >>> code >>> > is actually spending time: >>> > >>> > ProfilerService(frequency=500, gatherperiod=60 * 60, mode='virtual', >>> > basepath=None, wantBuilds=100 >>> >>> I tried configuring this with the settings dropdown in the WebGUI >>> plugin, >>> but AFAICT it is not working, it only gathers info for 30 seconds. >>> >>> I guess I must be holding it incorrectly. >>> >>> > This will record for one hour, and mitigate the memory used if you >>> worry >>> > about it. >>> >>> -- >>> Sincerely, >>> Yngve N. Pettersen >>> Vivaldi Technologies AS >>> >> > > -- > Vlad >
_______________________________________________ users mailing list users@buildbot.net https://lists.buildbot.net/mailman/listinfo/users