[Wikidata-bugs] [Maniphest] [Commented On] T227450: Many `request has exceeded memory limit` fatal errors for wikidata jobrunner

Ladsgroup added a comment. This is the metrics for this: F29965420: image.png https://logstash.wikimedia.org/goto/2a2c595208225df6466c104592f40ede TASK DETAIL https://phabricator.wikimedia.org/T227450 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T227450: Many `request has exceeded memory limit` fatal errors for wikidata jobrunner

alaa_wmde added a comment. waiting for this to be verified in production. expectation is that error rate drops in the few days after deploying this https://logstash.wikimedia.org/app/kibana#/dashboard/AWra4yyim2VjIW0682f0?_g=h@8b5b71a&_a=h@21362d2 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T227450: Many `request has exceeded memory limit` fatal errors for wikidata jobrunner

gerritbot added a comment. Change 525084 **merged** by jenkins-bot: [mediawiki/extensions/WikibaseQualityConstraints@master] Add entity lookup without cache service and use it in SymmetryChecker https://gerrit.wikimedia.org/r/525084 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T227450: Many `request has exceeded memory limit` fatal errors for wikidata jobrunner

gerritbot added a comment. Change 525084 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup): [mediawiki/extensions/WikibaseQualityConstraints@master] Set the entity lookup to retrieve the value and do not cache it into memory https://gerrit.wikimedia.org/r/525084 TASK

[Wikidata-bugs] [Maniphest] [Commented On] T227450: Many `request has exceeded memory limit` fatal errors for wikidata jobrunner

Ladsgroup added a comment. So we can call the mediawiki's API (wbgetclaim) for symmetry check, the doesn't reduce the total amount of memory needed, it just distributes them in two places. I think I found the underlying problem, it seems that WBQC requests entity lookup that caches the

[Wikidata-bugs] [Maniphest] [Commented On] T227450: Many `request has exceeded memory limit` fatal errors for wikidata jobrunner

Michael added a comment. Other than resolving the root cause, logging the frequency of this error to Grafana to monitor it and preventing it from going to Logstash could be a sensible way forward, too. TASK DETAIL https://phabricator.wikimedia.org/T227450 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T227450: Many `request has exceeded memory limit` fatal errors for wikidata jobrunner

Michael added a comment. No idea what changed in our config, but the error message seems to be somewhat different now: [XSw5rgpAAE0AAICPh8QL] /rpc/RunSingleJob.php PHP Fatal Error from line 139 of

[Wikidata-bugs] [Maniphest] [Commented On] T227450: Many `request has exceeded memory limit` fatal errors for wikidata jobrunner

Lucas_Werkmeister_WMDE added a comment. In T227450#5312623 , @Ladsgroup wrote: > The symmetry check can be done in SPARQL but I'm not sure if this is the direction we want to move towards (maybe we should? I don't know) No, we

[Wikidata-bugs] [Maniphest] [Commented On] T227450: Many `request has exceeded memory limit` fatal errors for wikidata jobrunner

Ladsgroup added a comment. And almost all are SymmetricChecker, it seems it tries to load a huge item and it fails, the number of fatal error is not a big issue because when a job fails, it gets retried lots of times so the number is a little bit bigger than the actual cases. The symmetry