Addshore created this task. Addshore added projects: wdwb-tech, Wikidata, Wikimedia-production-error. Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION While looking at logstash for some reason @Silvan_WMDE & I noticed there were quite some errors relating to https://www.wikidata.org/wiki/Special:EntityData/Q30.rdf - Timeouts `[5649a640-5f42-4d22-aaec-36ffdfb053de] /wiki/Special:EntityData/Q30.rdf Wikibase\DataModel\Services\Lookup\EntityLookupException: The maximum execution time of 60 seconds was exceeded` - OOMs `[0891912f-89b3-4cbc-a675-bf37bff44910] PHP Fatal error: Allowed memory size of 698351616 bytes exhausted (tried to allocate 134217736 bytes) in /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/purtle/src/RdfWriterBase.php:229` - EntityLookupException `[5649a640-5f42-4d22-aaec-36ffdfb053de] 2021-04-27 13:57:05: Fatal exception of type "Wikibase\DataModel\Services\Lookup\EntityLookupException"` (due to timeout) Seemingly this has been occurring for the complete time range of existing logstash logs F34429833: image.png <https://phabricator.wikimedia.org/F34429833> Searching phabricator I came across T62003: HTTP 503 error when requesting linked data for large entities <https://phabricator.wikimedia.org/T62003> **but I do not think this is related** as I also get a 500 error while requesting these pages directly from a mwapp server inside the cluster. addshore@deploy1002:~$ curl -k -L -H "Host: www.wikidata.org" -I https://mw1405.eqiad.wmnet/wiki/Special:EntityData/Q30.rdf HTTP/1.1 500 Internal Server Error Date: Tue, 27 Apr 2021 13:57:23 GMT Server: mw1405.eqiad.wmnet X-Powered-By: PHP/7.2.31-1+0~20200514.41+debian9~1.gbpe2a56b+wmf1+buster1 X-Request-Id: 5e4f152f-8a45-4d31-a77a-b9a112319b25 Backend-Timing: D=53005048 t=1619531843710982 Content-Type: text/html; charset=UTF-8 X-Envoy-Upstream-Service-Time: 53005 Transfer-Encoding: chunked The stack traces vary. An example for OOM would be: #0 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/purtle/src/RdfWriterBase.php(229): unknown() #1 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/purtle/src/XmlRdfWriter.php(69): Wikimedia\Purtle\RdfWriterBase->write() #2 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/purtle/src/XmlRdfWriter.php(221): Wikimedia\Purtle\XmlRdfWriter->tag() #3 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/purtle/src/RdfWriterBase.php(451): Wikimedia\Purtle\XmlRdfWriter->writeText() #4 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Rdf/TermsRdfBuilder.php(92): Wikimedia\Purtle\RdfWriterBase->text() #5 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Rdf/TermsRdfBuilder.php(183): Wikibase\Repo\Rdf\TermsRdfBuilder->addLabels() #6 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(480): Wikibase\Repo\Rdf\TermsRdfBuilder->addEntityStub() #7 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(444): Wikibase\Repo\Rdf\RdfBuilder->addEntityStub() #8 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(225): Wikibase\Repo\Rdf\RdfBuilder->resolveMentionedEntities() #9 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(170): Wikibase\Repo\LinkedData\EntityDataSerializationService->rdfSerialize() #10 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(529): Wikibase\Repo\LinkedData\EntityDataSerializationService->getSerializedData() #11 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(275): Wikibase\Repo\LinkedData\EntityDataRequestHandler->showData() #12 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Specials/SpecialEntityData.php(143): Wikibase\Repo\LinkedData\EntityDataRequestHandler->handleRequest() #13 /srv/mediawiki/php-1.37.0-wmf.1/includes/specialpage/SpecialPage.php(646): Wikibase\Repo\Specials\SpecialEntityData->execute() #14 /srv/mediawiki/php-1.37.0-wmf.1/includes/specialpage/SpecialPageFactory.php(1381): SpecialPage->run() #15 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(313): MediaWiki\SpecialPage\SpecialPageFactory->executePath() #16 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(916): MediaWiki->performRequest() #17 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(550): MediaWiki->main() #18 /srv/mediawiki/php-1.37.0-wmf.1/index.php(53): MediaWiki->run() #19 /srv/mediawiki/php-1.37.0-wmf.1/index.php(46): wfIndexMain() #20 /srv/mediawiki/w/index.php(3): require() And an example for timeout could be: from /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/request-timeout/src/Detail/ExcimerTimerWrapper.php(97) #0 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/request-timeout/src/Detail/ExcimerTimerWrapper.php(72): Wikimedia\RequestTimeout\Detail\ExcimerTimerWrapper->onTimeout(integer) #1 /srv/mediawiki/php-1.37.0-wmf.1/vendor/data-values/serialization/src/Deserializers/DataValueDeserializer.php(115): Wikimedia\RequestTimeout\Detail\ExcimerTimerWrapper->Wikimedia\RequestTimeout\Detail\{closure}(integer) #2 /srv/mediawiki/php-1.37.0-wmf.1/vendor/data-values/serialization/src/Deserializers/DataValueDeserializer.php(91): DataValues\Deserializers\DataValueDeserializer->getDeserialization(array) #3 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakDeserializer.php(128): DataValues\Deserializers\DataValueDeserializer->deserialize(array) #4 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakDeserializer.php(117): Wikibase\DataModel\Deserializers\SnakDeserializer->deserializeDataValue(array) #5 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakDeserializer.php(100): Wikibase\DataModel\Deserializers\SnakDeserializer->newValueSnak(array) #6 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakDeserializer.php(82): Wikibase\DataModel\Deserializers\SnakDeserializer->getDeserialized(array) #7 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakListDeserializer.php(60): Wikibase\DataModel\Deserializers\SnakDeserializer->deserialize(array) #8 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakListDeserializer.php(41): Wikibase\DataModel\Deserializers\SnakListDeserializer->getDeserialized(array) #9 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ReferenceDeserializer.php(77): Wikibase\DataModel\Deserializers\SnakListDeserializer->deserialize(array) #10 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ReferenceDeserializer.php(67): Wikibase\DataModel\Deserializers\ReferenceDeserializer->deserializeSnaks(array) #11 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ReferenceDeserializer.php(57): Wikibase\DataModel\Deserializers\ReferenceDeserializer->getDeserialized(array) #12 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ReferenceListDeserializer.php(53): Wikibase\DataModel\Deserializers\ReferenceDeserializer->deserialize(array) #13 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ReferenceListDeserializer.php(40): Wikibase\DataModel\Deserializers\ReferenceListDeserializer->getDeserialized(array) #14 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/StatementDeserializer.php(161): Wikibase\DataModel\Deserializers\ReferenceListDeserializer->deserialize(array) #15 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/StatementDeserializer.php(124): Wikibase\DataModel\Deserializers\StatementDeserializer->setReferencesFromSerialization(array, Wikibase\DataModel\Statement\Statement) #16 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/StatementDeserializer.php(100): Wikibase\DataModel\Deserializers\StatementDeserializer->getDeserialized(array) #17 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/StatementListDeserializer.php(60): Wikibase\DataModel\Deserializers\StatementDeserializer->deserialize(array) #18 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/StatementListDeserializer.php(41): Wikibase\DataModel\Deserializers\StatementListDeserializer->getDeserialized(array) #19 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ItemDeserializer.php(130): Wikibase\DataModel\Deserializers\StatementListDeserializer->deserialize(array) #20 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ItemDeserializer.php(85): Wikibase\DataModel\Deserializers\ItemDeserializer->setStatementListFromSerialization(array, Wikibase\DataModel\Entity\Item) #21 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ItemDeserializer.php(77): Wikibase\DataModel\Deserializers\ItemDeserializer->getDeserialized(array) #22 /srv/mediawiki/php-1.37.0-wmf.1/vendor/serialization/serialization/src/Deserializers/DispatchingDeserializer.php(42): Wikibase\DataModel\Deserializers\ItemDeserializer->deserialize(array) #23 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/internal-serialization/src/Deserializers/EntityDeserializer.php(42): Deserializers\DispatchingDeserializer->deserialize(array) #24 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/EntityContentDataCodec.php(253): Wikibase\InternalSerialization\Deserializers\EntityDeserializer->deserialize(array) #25 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityDataLoader.php(82): Wikibase\Lib\Store\EntityContentDataCodec->decodeEntity(string, NULL) #26 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(236): Wikibase\Lib\Store\Sql\WikiPageEntityDataLoader->loadEntityDataFromWikiPageRevision(MediaWiki\Revision\RevisionStoreRecord, string, integer) #27 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(122): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->loadEntity(stdClass, string) #28 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(54): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string) #29 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/data-access/src/ByTypeDispatchingEntityRevisionLookup.php(55): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string) #30 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(54): Wikibase\DataAccess\ByTypeDispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string) #31 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(104): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string) #32 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(87): Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string) #33 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(104): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string) #34 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(87): Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string) #35 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/RevisionBasedEntityLookup.php(46): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string) #36 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-services/src/Lookup/RedirectResolvingEntityLookup.php(51): Wikibase\Lib\Store\RevisionBasedEntityLookup->getEntity(Wikibase\DataModel\Entity\ItemId) #37 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(438): Wikibase\DataModel\Services\Lookup\RedirectResolvingEntityLookup->getEntity(Wikibase\DataModel\Entity\ItemId) #38 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(225): Wikibase\Repo\Rdf\RdfBuilder->resolveMentionedEntities(Wikibase\DataModel\Services\Lookup\RedirectResolvingEntityLookup) #39 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(170): Wikibase\Repo\LinkedData\EntityDataSerializationService->rdfSerialize(Wikibase\Lib\Store\EntityRevision, NULL, array, Wikibase\Repo\Rdf\RdfBuilder, NULL) #40 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(529): Wikibase\Repo\LinkedData\EntityDataSerializationService->getSerializedData(string, Wikibase\Lib\Store\EntityRevision, NULL, array, NULL) #41 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(275): Wikibase\Repo\LinkedData\EntityDataRequestHandler->showData(WebRequest, OutputPage, string, Wikibase\DataModel\Entity\ItemId, integer) #42 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Specials/SpecialEntityData.php(143): Wikibase\Repo\LinkedData\EntityDataRequestHandler->handleRequest(string, WebRequest, OutputPage) #43 /srv/mediawiki/php-1.37.0-wmf.1/includes/specialpage/SpecialPage.php(646): Wikibase\Repo\Specials\SpecialEntityData->execute(string) #44 /srv/mediawiki/php-1.37.0-wmf.1/includes/specialpage/SpecialPageFactory.php(1381): SpecialPage->run(string) #45 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(313): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext) #46 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(916): MediaWiki->performRequest() #47 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(550): MediaWiki->main() #48 /srv/mediawiki/php-1.37.0-wmf.1/index.php(53): MediaWiki->run() #49 /srv/mediawiki/php-1.37.0-wmf.1/index.php(46): wfIndexMain() #50 /srv/mediawiki/w/index.php(3): require(string) #51 {main} While investigating the situation a little more I discovered that: - The JSON api is not impacted, and loads quickly https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&ids=Q30 - The JSON Special:EntityData page loads just fine https://www.wikidata.org/wiki/Special:EntityData/Q30.json (including in the cluster `curl -k -L -H "Host: www.wikidata.org" -o -I https://mw1405.eqiad.wmnet/wiki/Special:EntityData/Q30.json`) - The TTL Special:EntityData page also has this issue for Q30 https://www.wikidata.org/wiki/Special:EntityData/Q30.ttl (including in the cluster `curl -k -L -H "Host: www.wikidata.org" -o -I https://mw1405.eqiad.wmnet/wiki/Special:EntityData/Q30.ttl`) - The TTL flavour=dump format for Special:EntityData doesn't appear to have this issue? https://www.wikidata.org/wiki/Special:EntityData/Q30.ttl?flavor=dump (BUT IT DOES not going via the edge? `curl -k -L -H "Host: www.wikidata.org" -o -I https://mw1405.eqiad.wmnet/wiki/Special:EntityData/Q30.ttl?flavour=dump`) - This could end up impacting the WDQS, but right now it appears to have the latest data https://query.wikidata.org/#select%20%2a%20where%20%7B%20wd%3AQ30%20schema%3Aversion%20%3Fv%20%7D - The item page itself seems to load just fine (even when busting the parser cache) https://www.wikidata.org/wiki/Q30, and infact loads / renders very quickly Expanding my logstash search it appears this is happening to some other entities too (I'm sure there would be more of these if I looked harder): - https://www.wikidata.org/wiki/Special:EntityData/Q183.ttl - https://www.wikidata.org/wiki/Special:EntityData/Q423.ttl - https://www.wikidata.org/wiki/Special:EntityData/Q81068910.ttl TASK DETAIL https://phabricator.wikimedia.org/T281272 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Tarrow, WMDE-leszek, Jakob_WMDE, Lydia_Pintscher, Addshore, Silvan_WMDE, Aklapper, Invadibot, maantietaja, Akuckartz, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
