[Wikidata-bugs] [Maniphest] T301167: Service implementation for wdqs20[09,10,11,12]
jcrespo added a comment. One tip to avoid having people on call (like me) worrying about pending implementation services is to add the hiera key `profile::monitoring::notifications_enabled: false`. This is not promoted much because most people handle stateless services that are easy and fast to setup. If you, like me, handle statful ones that can take a lot to load its data and be fully setup, this flag is useful until they are fully productionized. Hope this tip is found useful. Thank you and sorry for the previous ping! TASK DETAIL https://phabricator.wikimedia.org/T301167 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, jcrespo Cc: jcrespo, bking, Aklapper, RKemper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload
jcrespo added a comment. I was told by @Gehel that it was unrelated to this, but related to T301167 <https://phabricator.wikimedia.org/T301167>. Sorry for the confussion. TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, RKemper, bking, Oravrattas, Aklapper, Gehel, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, EBjune, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T323096: WDQS Data Reload
jcrespo added a comment. Sorry if it is the wrong ticket, but several services of wdqs2010, wdqs2011 and wdqs2012 are alerting. The sevice is returnin 400 commands. My guess is this is due to this ongoing data reload (no issue). If that is the case, could the alerts "WDQS SPARQL" and other failing checks be acknowledeged on icinga, to prevent alert spam? Thank you! TASK DETAIL https://phabricator.wikimedia.org/T323096 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, RKemper, bking, Oravrattas, Aklapper, Gehel, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, EBjune, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T307586: wbsearchentities produces no results on 1.39.0-wmf.10
jcrespo added a comment. In T307586#7908045 <https://phabricator.wikimedia.org/T307586#7908045>, @Quiddity wrote: > For Tech News purposes, how should this entry be described? IIUC from the description, something like this? > >> There was a problem with Wikidata's search autocomplete last week. This has now been fixed. > > Please approve that wording, or suggest a better summary. Thanks! Looks good to me :-) TASK DETAIL https://phabricator.wikimedia.org/T307586 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: brennen, jcrespo Cc: Quiddity, EBernhardson, Zabe, dcausse, brennen, hashar, jcrespo, Raymond, Moebeus, Lucas_Werkmeister_WMDE, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, R4356th, Bebiezaza, EhsanKhandowa, maantietaja, CBogen, ItamarWMDE, Akuckartz, darthmon_wmde, Rosalie_WMDE, PatsagornY, Viztor, Nandana, Amorymeltzer, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, JJMC89, _jensen, rosalieper, Scott_WUaS, Johan, Luke081515, Verdy_p, Wikidata-bugs, aude, TheDJ, Jdforrester-WMF, Addshore, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T307586: wbsearchentities produces no results on 1.39.0-wmf.10
jcrespo added a comment. Thanks for such a quick reaction, BTW. TASK DETAIL https://phabricator.wikimedia.org/T307586 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: brennen, hashar, jcrespo, Raymond, Moebeus, Lucas_Werkmeister_WMDE, Aklapper, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, R4356th, Bebiezaza, EhsanKhandowa, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, darthmon_wmde, Rosalie_WMDE, PatsagornY, Kent7301, joker88john, Viztor, CucyNoiD, Nandana, Gaboe420, Amorymeltzer, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, EBjune, LawExplorer, Lewizho99, JJMC89, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Johan, Luke081515, Verdy_p, Wikidata-bugs, aude, TheDJ, Jdforrester-WMF, Addshore, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T307586: wbsearchentities produces no results on 1.39.0-wmf.10
jcrespo updated the task description. jcrespo added a project: User-notice. TASK DETAIL https://phabricator.wikimedia.org/T307586 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Raymond, Moebeus, Lucas_Werkmeister_WMDE, Aklapper, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, R4356th, Bebiezaza, EhsanKhandowa, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, darthmon_wmde, Rosalie_WMDE, PatsagornY, Kent7301, joker88john, Viztor, CucyNoiD, Nandana, Gaboe420, Amorymeltzer, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, EBjune, LawExplorer, Lewizho99, JJMC89, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Johan, Luke081515, Verdy_p, Wikidata-bugs, aude, TheDJ, Jdforrester-WMF, Addshore, Mbch331, Jay8g, valerio.bozzolan ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T307586: wbsearchentities produces no results on 1.39.0-wmf.10
jcrespo added a comment. In addition to the fix/rollback- could some integration test or heuristic production monitoring also be able to be implemented, for future faster detection? TASK DETAIL https://phabricator.wikimedia.org/T307586 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Raymond, Moebeus, Lucas_Werkmeister_WMDE, Aklapper, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, darthmon_wmde, Rosalie_WMDE, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, EBjune, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Addshore, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
jcrespo added a comment. Terminated Jobs: JobId Level FilesBytes Status FinishedName 396417 Full 108,32011.70 G OK 13-Dec-21 09:34 graphite1004.eqiad.wmnet-Weekly-Mon-production-srv-carbon-whisper-daily 396418 Full 108,32011.70 G OK 13-Dec-21 09:35 graphite2003.codfw.wmnet-Weekly-Mon-production-srv-carbon-whisper-daily TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi, jcrespo Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
jcrespo added a comment. Running Jobs: Console connected using TLS at 13-Dec-21 09:20 JobId Type Level Files Bytes Name Status == 396417 Back Full 4,568412.9 M graphite1004.eqiad.wmnet-Weekly-Mon-production-srv-carbon-whisper-daily is running 396418 Back Full 0 0 graphite2003.codfw.wmnet-Weekly-Mon-production-srv-carbon-whisper-daily is running TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi, jcrespo Cc: Lydia_Pintscher, LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
jcrespo added a comment. Let me give it a deeper look, while the patch by itself looks good as is, I want to check if a different (non-default) backup policy would be more advantageous in frequency and space. :-) TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, 786, Suran38, Biggs657, Invadibot, Lalamarie69, Devnull, maantietaja, lmata, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Robin.guo, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
jcrespo added a comment. I don't have the answer to that question, but whenever any of you have the servers and path(s), you can follow the instructions at https://wikitech.wikimedia.org/wiki/Bacula#Adding_a_new_client to send a preliminary backup proposal to Puppet, and I will assist you to merge it with the proper setup (e.g. schedule, day, etc.) - I think it will be more useful to discuss the details over a patch :-). TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: LSobanski, jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
jcrespo added a comment. One more question, to finally decide if setting up weekly full backups or daily but incremental- do all files mostly change completely, or only a subset of them? Incrementals are able to be done with file granularity only (it will backup fully files as long as its path or hash has changed), if value.wsp changes every minute, and there is only 1 per value, we will do "weekly only full", otherwise the daily incrementals may be preferred. If we end up doing weekly fulls, 11GB * 12 weeks of retention = 130 GB, which we can handle with no issue. TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, LSobanski, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T294355: Several Wikidata Grafana boards missing data before October 2021
jcrespo added projects: bacula, Data-Persistence-Backup, Data-Persistence. jcrespo added a comment. number of files are (within reason) a non-blocker for bacula, as files are packaged into volumes. It is true that each file is stored as a mysql record, but that should be able to scale until dozens of (US) billons, although it may be slow to recover when rebuilding metadata. Most limiting factor would be the overall size + backup frequency for capacity planning. We don't have a lot of temporal data backed up, so not sure if we could come up with a strategy that saves space (e.g. if data is immutable, we may want to avoid full backups every day). What is the file/directory structure? If data is below e.g. 100GB I would consider it "small" and not requiring optimization. The typical backup schedule is incrementals of a set of paths every day, differentials every fortnite, and fulls monthly- however it is highly customizable per job. TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, LSobanski, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T186716: enable fine grained usage tracking on Commons
jcrespo added a project: Data-Persistence (Consultation). TASK DETAIL https://phabricator.wikimedia.org/T186716 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Kormat, Michael, Ladsgroup, Marostegui, RP88, Mike_Peel, Aklapper, Lydia_Pintscher, karapayneWMDE, Invadibot, maantietaja, Y.ssk, Muchiri124, Hazizibinmahdi, CBogen, Akuckartz, Nandana, Lahi, Gq86, Ramsey-WMF, GoranSMilovanovic, QZanden, LawExplorer, Poyekhali, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Ixocactus, Wong128hk, Wikidata-bugs, aude, El_Grafo, Dinoguy1000, Steinsplitter, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T186716: enable fine grained usage tracking on Commons
jcrespo added a subscriber: Kormat. jcrespo added a comment. > the DBAs to approve this That should be @Kormat and/or @Marostegui (he is on vacations right now). TASK DETAIL https://phabricator.wikimedia.org/T186716 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Kormat, Michael, Ladsgroup, jcrespo, Marostegui, RP88, Mike_Peel, Aklapper, Lydia_Pintscher, karapayneWMDE, Invadibot, maantietaja, Y.ssk, Muchiri124, Hazizibinmahdi, CBogen, Akuckartz, Nandana, Lahi, Gq86, Ramsey-WMF, GoranSMilovanovic, QZanden, LawExplorer, Poyekhali, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Ixocactus, Wong128hk, Wikidata-bugs, aude, El_Grafo, Dinoguy1000, Steinsplitter, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T48643: [Story] Dispatching via job queue (instead of cron script)
jcrespo added a comment. At 16:02-16:06, which would fit with the deployment of https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/725019/ logstash baseline number of messages increased around a 50%: https://grafana.wikimedia.org/d/00561/logstash?orgId=1=2=1633013879325=1633021930229=eqiad%20prometheus%2Fops=dlq%2Fmain My guess (without checking the nature of the new messages) is this normal and expected due to increase debugging/informational logs in the job system, but I wanted to give a heads up, and hopefully someone can verify it is as I said, in case it is not normal and expected to confirm there are not an increase in the number of errors. TASK DETAIL https://phabricator.wikimedia.org/T48643 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, toan, Tarrow, jijiki, alaa_wmde, Ladsgroup, Joe, Addshore, Aklapper, Tobi_WMDE_SW, JanZerebecki, Wikidata-bugs, Abraham, Nemo_bis, Denny, aude, Ricordisamoa, Lydia_Pintscher, daniel, hoo, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, R4356th, EhsanKhandowa, maantietaja, Juan90264, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, PatsagornY, Kent7301, joker88john, Michael, Viztor, CucyNoiD, Nandana, Gaboe420, lucamauri, Amorymeltzer, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, JJMC89, Maathavan, _jensen, rosalieper, Scott_WUaS, Johan, Luke081515, TheDJ, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T281480: Cannot access the database: Too many connections
jcrespo triaged this task as "Unbreak Now!" priority. jcrespo added a comment. This should be a blocker- es traffic has grown almost grown 100x since 14 april, correlates strongly with the 19h deploy: F34434387: es_issue.png <https://phabricator.wikimedia.org/F34434387> TASK DETAIL https://phabricator.wikimedia.org/T281480 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Urbanecm, Aklapper, LarsWirzenius, Invadibot, Devnull, LSobanski, maantietaja, Akuckartz, Iflorez, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Minhnv-2809, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Addshore, Mbch331, Jay8g, Krenair, RhinosF1, AMooney ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T276762: Wikibase\Lib\Store\Sql\SiteLinkTable::getLinks can take over a minute to execute
jcrespo added a comment. Offtopic- and feel free to PM in private. What is a good way to report database-related issues to wikidata development team? I am a bit intimidated by the amount of tags and dashboards (which will probably reflect your internal organization, but I am not too familiar with it). I trust you already that any tag (even incorrect!) I use always end up being attended (and I am happy with it, thank you!)- but please provide feedback if I can make your life easier for you when triaging (who to CC? tags to use?, etc.) Thank you! TASK DETAIL https://phabricator.wikimedia.org/T276762 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Addshore, Marostegui, hoo, Aklapper, jcrespo, maantietaja, Akuckartz, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, abian, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T276762: Wikibase\Lib\Store\Sql\SiteLinkTable::getLinks can take over a minute to execute
jcrespo updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T276762 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Marostegui, hoo, Aklapper, jcrespo, maantietaja, Akuckartz, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, abian, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T276762: Wikibase\Lib\Store\Sql\SiteLinkTable::getLinks can take over a minute to execute
jcrespo created this task. jcrespo added projects: Wikidata, Wikibase. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION This weekend, while an ongoing incident was being handled, I checked and saw several badly performing queries running. These didn't have (I believe) any relation with the incident, but any web request that takes more than 1 or a few seconds (in the worse possible case) is worrying, as they can pile up and create load issues on the database. While there was several queries with bad performance (mostly related to recentchanges and API), the worst occurring query right now is: SELECT /* Wikibase\Lib\Store\Sql\SiteLinkTable::getLinks */ count(*) FROM `wb_items_per_site` WHERE ips_site_id = '?'; where '?' can be any production wiki, but in particular, for enwiki, it selects over **8 million rows**. We saw over 30 of those, over all s8 servers, today Mar 8 at 8:18 UTC. This not only takes over 1 minute to run, but is likely to be not useful due to the amount of memory consumed. While reading large number of rows can be useful for something like a dump/analytics, it is unlikely to be useful for a web request. Moreso, the application server will struggle to handle 2GB of data at a time, even in memory, that this query returns. Is it possible some extra WHERE or LIMIT was missing from the query? TASK DETAIL https://phabricator.wikimedia.org/T276762 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Marostegui, hoo, Aklapper, jcrespo, maantietaja, Akuckartz, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, abian, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium
jcrespo added a subscriber: Marostegui. jcrespo added a comment. Handover to @Marostegui for him to comment, as he will be the person to know if this continues happening or now. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Marostegui, Addshore, Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, jannee_e, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, gnosygnu, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T272571: Updating's Wikidata property suggester caused replica lag on all wikidata databases
jcrespo created this task. jcrespo added projects: Wikidata, MediaWiki-extensions-PropertySuggester, Wikimedia-production-error. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION We got an alert on `#wikimedia-databases` IRC saying: PROBLEM - MariaDB sustained replica lag on db is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/00273/mysql?orgId=1=db=9104 However, the spike of lag was seen on all servers,it just happened db was more sensitive to be reported: F34010823: Screenshot from 2021-01-21 10-57-14.png <https://phabricator.wikimedia.org/F34010823> Logstash indicated thousands of client errors due to lag: F34010825: Screenshot from 2021-01-21 10-59-06.png <https://phabricator.wikimedia.org/F34010825> Log around that time seems to indicate maintenance to Wikidata property suggester: 09:44:09 !log Updated the Wikidata property suggester with data from the 2021-01-11 JSON dump and applied the T132839 workarounds hoo on IRC seemed to agree that it was likely the cause: 09:48:10 The maintenance script rebuilds the entire table (yuck...) We can have temporary lag on one server and we are able to cope with that, but on all servers is quite imapacting for editors/recentchanges/etc. Let's try to avoid production impact by one of: refactoring the script, adding pauses (e.g. waitForReplica()) or avoiding its run, or any other method, up to devel team. TASK DETAIL https://phabricator.wikimedia.org/T272571 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Marostegui, hoo, Aklapper, jcrespo, Akuckartz, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Verdy_p, Wikidata-bugs, aude, Sjoerddebruin, Jdforrester-WMF, Mbch331, Rxy, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T165726: [Hackathon doc sprint] Improve deployment documentation
jcrespo closed subtask T165756: Create summary templates on Wikitech wiki to stop writing the same things everywhere, everytime as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T165726 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Aklapper, Dereckson, Akuckartz, apaskulin, CptViraj, Pavithraes, darthmon_wmde, Dibya, 94rain, DannyS712, Nandana, Tks4Fish, Cpaulf30, Lahi, Gq86, GoranSMilovanovic, Ivana_Isadora, Jayprakash12345, QZanden, Kizule, LawExplorer, Puik, DatGuy, Devwaker, Niklitov, _jensen, Urbanecm, rosalieper, xSavitar, Envlh, JEumerus, Scott_WUaS, Ananthsubray, Superzerocool, Tulsi_Bhagat, Wong128hk, Luke081515, SimmeD, Wikidata-bugs, Snowolf, aude, Tobias1984, Dcljr, Dinoguy1000, Matanya, Mbch331, Rxy, Jay8g, Krenair, RhinosF1 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T86530: Replace wb_terms table with more specialized mechanisms for terms (tracking)
jcrespo added a comment. To be fair, technically, this is resolved because wb_terms has been replaced, AFAIK, with a more specialized mechanism (several smaller and normalised tables) :-) TASK DETAIL https://phabricator.wikimedia.org/T86530 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Mholloway, Addshore, jcrespo, Ricordisamoa, Lydia_Pintscher, adrianheine, thiemowmde, Tobi_WMDE_SW, JeroenDeDauw, JanZerebecki, aude, Aklapper, Liuxinyu970226, daniel, hoo, Akuckartz, darthmon_wmde, Nickleh, Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Volker_E, Wikidata-bugs, GWicke, Dinoguy1000, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T171928: Wikidata and dewiki databases locked
jcrespo closed subtask T172490: Monitor swap/memory usage on databases as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T171928 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: gerritbot, mark, Marostegui, Elitre, Joe, jcrespo, greg, Mbch331, Smalyshev, MisterSynergy, Jay8g, Lydia_Pintscher, Aklapper, Esc3300, PokestarFan, lmata, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, Viztor, Legado_Shulgin, CucyNoiD, Nandana, jijiki, Gaboe420, Amorymeltzer, Giuliamocci, Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Imarlier, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Lewizho99, Zppix, JJMC89, Maathavan, elukey, _jensen, rosalieper, Scott_WUaS, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Bawolff, ArielGlenn, faidon, He7d3r, TheDJ, Jdforrester-WMF, Rxy, akosiaris, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246232: Bad Wikidata query format in DatabaseTermInLangIdsResolver::selectTermsViaJoin
jcrespo added a comment. There is this procedure: https://wikitech.wikimedia.org/wiki/MariaDB/query_performance which could be done for Wikidata by people with root database privileges, however it is quite involved. I will ask if maybe Manuel wants to perform soon such a thing for other main reasons (e.g. MCR changes checking) and maybe this could be checked at the same time. There is also performance_schema storing digests <https://dev.mysql.com/doc/refman/5.7/en/sys-statement-analysis.html>, but it is unreliable due to the large amount of different query patterns mw/wikibase gets (it gets quickly out of memory after a server gets started). If the query was stored there, it howerver can reliably say when was the last time it was executed. Note that due to volume, all queries are almost impossible to be checked, only a high sigma of it, probabilistically. TASK DETAIL https://phabricator.wikimedia.org/T246232 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, jcrespo Cc: Michael, Lucas_Werkmeister_WMDE, Addshore, Aklapper, Marostegui, Ladsgroup, jcrespo, Hazizibinmahdi, Akuckartz, Iflorez, darthmon_wmde, alaa_wmde, Nandana, jijiki, Klaas_Z4us_V, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T246232: Bad Wikidata query format in DatabaseTermInLangIdsResolver::selectTermsViaJoin
jcrespo added a comment.
> based on that sentence, I would even today vote against the change Amir
just uploaded if it wasn’t for this ticket, because it seems to fly directly in
the face of this documentation: why use $join_conds if the documentation says
it’s unnecessary except for a LEFT JOIN that we don’t want?
That documentation is, IMHO, wrong. And I can bring authoritative references
if needed: implicit joins are not ANSI (aka not real "SQL" standard), and are
deprecated in several database engines ("considered harmful" XD). MySQL keeps
supporting it, as it does with other set of bad practices- and I don't advocate
necessarily to remove the option, but I do to discourage it- again based on
actual past outages that it has caused on Wikimedia by mistaken deploys, as I
mentioned on my initial comment.
TASK DETAIL
https://phabricator.wikimedia.org/T246232
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ladsgroup, jcrespo
Cc: Lucas_Werkmeister_WMDE, Addshore, Aklapper, Marostegui, Ladsgroup, jcrespo,
Alter-paule, Hazizibinmahdi, Beast1978, Un1tY, Akuckartz, Hook696, Iflorez,
darthmon_wmde, Kent7301, alaa_wmde, joker88john, CucyNoiD, Nandana, jijiki,
Klaas_Z4us_V, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan,
GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, Maathavan, elukey, _jensen,
rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331,
Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run
jcrespo added a comment. One thing that may be relevant here is that the query may have worked once or twice in the last 6 months due to this underlying issue. Last update that completed successfully was 27 March. In effect, because this bug, this wasn't disabled officially but it was already (for the most part) not-working (but still causing issues before the merge). TASK DETAIL https://phabricator.wikimedia.org/T238199 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: SilentSpike, WMDE-leszek, ArielGlenn, Lea_Lacroix_WMDE, jcrespo, Addshore, Lydia_Pintscher, Aklapper, Ladsgroup, Marostegui, darthmon_wmde, Nandana, jijiki, Amorymeltzer, Imarlier, Lahi, Gq86, Lsherwinforone, GoranSMilovanovic, Jayprakash12345, QZanden, LawExplorer, Sethakill, elukey, _jensen, rosalieper, Scott_WUaS, Wong128hk, Wikidata-bugs, aude, Bawolff, He7d3r, Jdforrester-WMF, Mbch331, Rxy, Jay8g, akosiaris ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T252952: Wikidata dispatching slow and maxlag high on Wikidata due to db1101 replication lag
jcrespo added a comment.
{P11212}
TASK DETAIL
https://phabricator.wikimedia.org/T252952
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Addshore, jcrespo
Cc: WMDE-leszek, darthmon_wmde, Lea_Lacroix_WMDE, Marostegui, jcrespo,
Ladsgroup, Lydia_Pintscher, Pasleim, Addshore, Liuxinyu970226, Aklapper,
Bugreporter, Blissjay007, Oblanco79, Alter-paule, Beast1978, Un1tY, Hook696,
Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, Iflorez, Kent7301, alaa_wmde,
Meekrab2012, joker88john, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris,
Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985,
Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Darkminds3113, Bsandipan,
Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6,
QZanden, LawExplorer, WSH1906, Lewizho99, Minhnv-2809, Zppix, Maathavan,
_jensen, rosalieper, Scott_WUaS, Jonas, Wong128hk, Wikidata-bugs, aude, faidon,
Mbch331, Rxy, Jay8g, Krenair, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run
jcrespo added a project: Wikimedia-Incident. TASK DETAIL https://phabricator.wikimedia.org/T238199 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Lea_Lacroix_WMDE, jcrespo, Addshore, Lydia_Pintscher, Aklapper, Ladsgroup, Marostegui, darthmon_wmde, Nandana, jijiki, Amorymeltzer, Imarlier, Lahi, Gq86, Lsherwinforone, GoranSMilovanovic, Jayprakash12345, QZanden, LawExplorer, Sethakill, elukey, _jensen, rosalieper, Scott_WUaS, Wong128hk, Wikidata-bugs, aude, Bawolff, ArielGlenn, He7d3r, Jdforrester-WMF, Mbch331, Rxy, Jay8g, akosiaris ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run
jcrespo added a comment. Over 50K user errors: https://logstash.wikimedia.org/goto/e78777bdaa039396e05bca8099294c91 TASK DETAIL https://phabricator.wikimedia.org/T238199 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Lea_Lacroix_WMDE, jcrespo, Addshore, Lydia_Pintscher, Aklapper, Ladsgroup, Marostegui, darthmon_wmde, Nandana, Amorymeltzer, Lahi, Gq86, Lsherwinforone, GoranSMilovanovic, Jayprakash12345, QZanden, LawExplorer, Sethakill, _jensen, rosalieper, Scott_WUaS, Wong128hk, Wikidata-bugs, aude, Mbch331, Rxy ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T171928: Wikidata and dewiki databases locked
jcrespo closed subtask T172489: Monitor read_only on all databases, make it page on masters as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T171928 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: gerritbot, mark, Marostegui, Elitre, Joe, jcrespo, greg, Mbch331, Smalyshev, MisterSynergy, TerraCodes, Jay8g, Lydia_Pintscher, Aklapper, Esc3300, PokestarFan, Blissjay007, Oblanco79, Alter-paule, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, Viztor, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, jijiki, Gaboe420, Versusxo, Majesticalreaper22, Amorymeltzer, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Techguru.pc, Imarlier, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Zppix, JJMC89, Maathavan, elukey, _jensen, rosalieper, Scott_WUaS, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Bawolff, ArielGlenn, faidon, He7d3r, TheDJ, Jdforrester-WMF, Rxy, akosiaris, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Blocker] T171928: Wikidata and dewiki databases locked
jcrespo changed the status of subtask T172489: Monitor read_only on all databases, make it page on masters from Stalled to Open. TASK DETAIL https://phabricator.wikimedia.org/T171928 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: gerritbot, mark, Marostegui, Elitre, Joe, jcrespo, greg, Mbch331, Smalyshev, MisterSynergy, TerraCodes, Jay8g, Lydia_Pintscher, Aklapper, Esc3300, PokestarFan, Blissjay007, Oblanco79, Alter-paule, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, Viztor, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, jijiki, Gaboe420, Versusxo, Majesticalreaper22, Amorymeltzer, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Techguru.pc, Imarlier, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Zppix, JJMC89, Maathavan, elukey, _jensen, rosalieper, Scott_WUaS, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Bawolff, ArielGlenn, faidon, He7d3r, TheDJ, Jdforrester-WMF, Rxy, akosiaris, fgiunchedi, RhinosF1 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Blocker] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language
jcrespo changed the status of subtask T239238: Switchover s8 primary database master db1109 - db1104 - Date TBD from Open to Stalled. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui, jcrespo Cc: Marostegui, jcrespo, alaa_wmde, Addshore, Aklapper, Ladsgroup, Iflorez, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Rxy, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T249565: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views
jcrespo added a comment. I think people that worked for a long time since yesterday are taking a well-deserved rest and may be unresponsive. From the data recovery/database perspective everything is done and no longer UBN. But I think it should be someone familiar with Wikidata that should give the last ok, as there may be those cache issues still affecting some pages served, with a long tail. TASK DETAIL https://phabricator.wikimedia.org/T249565 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, jcrespo Cc: jcrespo, CDanis, Marostegui, Darwinius, Zaiva_Igor, Kazrok4545, Neolexx, Amorymeltzer, Rowingbohe, Lea_Lacroix_WMDE, matej_suchanek, Laurent04000, Giku, doctaxon, Robby, Ixocactus, pywikibot-bugs-list, JAnD, Lofhi, Ltrlg, Mike_Peel, Premeditated, Jony, Majavah, Jar, SCP-2000, Mahir256, alanajjar, hoo, Soul_Train, Mr.Ibrahem, Krinkle, NickK, Sakretsu, Urbanecm, MisterSynergy, Christian75, Fuzheado, Emaus, RexxS, Paladox, MaxSem, BRPever, AntiCompositeNumber, Coffeeandcrumbs, ppelberg, Catrope, Esanders, Jdforrester-WMF, MMiller_WMF, Jdlrobson, Piramidion, kolbert, Pigsonthewing, Wittylama, PKM, Erutuon, Moebeus, Zoranzoki21, MusikAnimal, Thibaut120094, Pcoombe, RoySmith, Bugreporter, Addshore, Miraclepine, Ladsgroup, Can_I_Log_In, Krenair, Ahmad252, Dreamy_Jazz, Tgr, Koavf, DannyS712, Aklapper, Hazizibinmahdi, Iflorez, darthmon_wmde, alaa_wmde, RhinosF1, Viztor, Nandana, jijiki, Imarlier, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, JJMC89, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, Johan, Wong128hk, Luke081515, Wikidata-bugs, aude, Bawolff, ArielGlenn, Lydia_Pintscher, He7d3r, TheDJ, Mbch331, Rxy, Jay8g, akosiaris ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T68025: [Story] Monitor size of some Wikidata database tables
jcrespo added a comment. > perhaps it might be an idea for more people to have access to this zarcillo thing? Stress on "The reasons why those are not more public is that we were told not to put those on public prometheus by security as they could compromise the anonymity of certain users on smaller wikis". Please involve them before making anything public, as they warned us in the past. Also be careful with size- compression could made the report on I_S very wrong, depending on the used page size (don't remember the details, but I know that is a thing). We should sync more to make sure no missleading values are used for anything important. And indeed, that we don't work twice. :-D TASK DETAIL https://phabricator.wikimedia.org/T68025 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Marostegui, Ladsgroup, Reedy, ArielGlenn, Liuxinyu970226, GoranSMilovanovic, Addshore, jcrespo, Ricordisamoa, hoo, Aklapper, daniel, JanZerebecki, Lydia_Pintscher, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, Iflorez, darthmon_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Minhnv-2809, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T246159: Setting on read new for items causes increased worst response time and number of open connections
jcrespo added a comment. Looks like CPU exhaustion which could be, in part, due to T232446 <https://phabricator.wikimedia.org/T232446>. TASK DETAIL https://phabricator.wikimedia.org/T246159 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, jcrespo Cc: jcrespo, Marostegui, Aklapper, ArielGlenn, Addshore, WMDE-leszek, Ladsgroup, Hazizibinmahdi, Iflorez, darthmon_wmde, alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T235265: Slow query on SpecialMostLinked creating lag on wikidata slaves
jcrespo added a comment. T238199 <https://phabricator.wikimedia.org/T238199> is not a duplicate, but certainly it is the same theme and effect (just caused lag on wikidata replicas). TASK DETAIL https://phabricator.wikimedia.org/T235265 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Masumrezarock100, Aklapper, Ladsgroup, Anomie, Marostegui, darthmon_wmde, Nandana, Amorymeltzer, Lahi, Gq86, Darkminds3113, Lsherwinforone, GoranSMilovanovic, Jayprakash12345, QZanden, LawExplorer, Vali.matei, Sethakill, _jensen, rosalieper, Scott_WUaS, Pchelolo, Volker_E, Wong128hk, Wikidata-bugs, aude, GWicke, Dinoguy1000, Mbch331, Jay8g ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run
jcrespo edited projects, added MediaWiki-Special-pages; removed MediaWiki-General. TASK DETAIL https://phabricator.wikimedia.org/T238199 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Addshore, Lydia_Pintscher, Aklapper, matmarex, Ladsgroup, Marostegui, darthmon_wmde, Nandana, Amorymeltzer, Lahi, Gq86, Lsherwinforone, GoranSMilovanovic, Jayprakash12345, QZanden, LawExplorer, Sethakill, _jensen, rosalieper, Scott_WUaS, Jonas, Wong128hk, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Rxy, Jay8g, Krenair, Adidsone1, DannyS712, Phukettaxigroup ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run
jcrespo added a project: Wikimedia-production-error. jcrespo added a comment. https://logstash.wikimedia.org/goto/060d35b0c4c75be0a47e14bb8602ac14 TASK DETAIL https://phabricator.wikimedia.org/T238199 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Addshore, Lydia_Pintscher, Aklapper, matmarex, Ladsgroup, Marostegui, Adidsone1, darthmon_wmde, DannyS712, Nandana, Phukettaxigroup, Lahi, Gq86, GoranSMilovanovic, Jayprakash12345, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Rxy, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T238199: SpecialFewestRevisions::reallyDoQuery takes more than 9h to run
jcrespo added a comment. I killed SpecialFewestRevisions::reallyDoQuery because it was causing the host (s8, db1101) where it was running to lag for 1-2 seconds, enough for queries waiting for replication to take a dive in performance, and logging hundreds of thousands of db errors. Long running queries are ok on a vslow server, but I wonder if we could make them in smaller batches (slow overal) to not tank the writes. TASK DETAIL https://phabricator.wikimedia.org/T238199 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Addshore, Lydia_Pintscher, Aklapper, matmarex, Ladsgroup, Marostegui, Adidsone1, darthmon_wmde, DannyS712, Nandana, Phukettaxigroup, Lahi, Gq86, GoranSMilovanovic, Jayprakash12345, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Rxy ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T246232: Bad Wikidata query format, review process broken?
jcrespo updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246232 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Aklapper, Marostegui, Ladsgroup, jcrespo, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T246232: Bad Wikidata query format, review process broken?
jcrespo created this task.
jcrespo added a project: Wikidata.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
At T246159 <https://phabricator.wikimedia.org/T246159> we can observe a
really badly formatted query:
SELECT /*
Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsResolver::selectTermsViaJoin
*/ wbtl_id, wbtl_type_id, wbxl_language, wbx_text, wbit_item_id FROM
`wbt_term_in_lang`, `wbt_text_in_lang`, `wbt_text`, `wbt_item_terms` WHERE
(wbtl_text_in_lang_id=wbxl_id) AND (wbxl_text_id=wbx_id) AND wbit_item_id IN
(...) AND wbxl_language IN ('hu', 'en') AND (`wbit_term_in_lang_id` = wbtl_id)
Compare this with the equivalent:
SELECT /*
Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsResolver::selectTermsViaJoin
*/
wbtl_id, wbtl_type_id, wbxl_language, wbx_text, wbit_item_id
FROM wbt_term_in_lang
JOIN wbt_text_in_lang
ON wbtl_text_in_lang_id = wbxl_id
JOIN wbt_text
ON wbxl_text_id = wbx_id
JOIN wbt_item_terms
ON wbit_term_in_lang_id = wbtl_id
WHERE wbit_item_id IN (...) AND wbxl_language IN ('hu', 'en')
(ignore the quotation and justification, and I don't know my heart the right
columns of each table). It is funny that a method called `selectTermsViaJoin`
didn't contain any join!
While this could be disregarded as a minor issue, I would like to direct your
attention to this issue for 2 reasons:
- A badly formatted query like this **has been the direct cause of outages at
wikimedia database infrastructure**. By convention (outside of Wikimedia, but
specially on Wikimedia) implicit JOINS are highly discouraged. I don't know how
even the wikimedia abstraction layer allows them.
JOINS should be explicit and be used with its ON clause. In the past, the
removal of an ON clause but not removal from the implicit list of tables to
join caused overload an outage. As a preventive measure, this was considered a
bad practice and discouraged. This is also a standard among any serious
professional SQL work. This is hard to read, to the point I could not work with
the query. **JOIN ... ON is way more legible**. You know with a single view
which tables are involved and differenciate filtering from joining comparisons,
preventing mistakes when changing the query.
- This issue would have been caught immediately by DBAs or other developers
on review and asked to be changed immediately. This leads to the more important
issue that **review process may be broken** because not even the most obvious
formatting issue has been corrected on review. I am not looking to assign any
blame here, I am asking to review the reviewing process for database layer
changes so obvious mistakes like these are caught on time.
TASK DETAIL
https://phabricator.wikimedia.org/T246232
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Aklapper, Marostegui, Ladsgroup, jcrespo, darthmon_wmde, Nandana, Lahi,
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS,
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T246232: Bad Wikidata query format, review process broken?
jcrespo updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T246232 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Aklapper, Marostegui, Ladsgroup, jcrespo, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T246159: Setting on read new for items causes increased worst response time and number of open connections
jcrespo added a comment. so 200K per minute overall is approximately 20/60/3 (db1126 has a third of the main, non special traffic) = 1000 QPS max per server. I would categorize this as frequent, normally those should have a latency of 0.01 seconds or less in average (and for that, that meas not reading more than 1000 rows). There are not hard limits, this is approximately the aim to go for the average query. I believe up to 1000 writes and 1 rows read can read efficiently on that time. This will put between 20 and 50 the batch size. The 9 advice was related to bad plan being chosen due to eq_dive_range_limit being 10, but as far I can see, it has no effect here due to the type of query. The issue is that if that amount of reads have to happen anyway, just they will be sent in batches, unless we slow down the query rate somehow, this will fix nothing-- it will make the database happy, but will move the "slowdown" of the overal HTTP call to the app server. Please take my advice as a very relative general one, it is the result on server performance what will put the limits, not me, I just can make predictions based on the data provided :-D TASK DETAIL https://phabricator.wikimedia.org/T246159 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, jcrespo Cc: jcrespo, Marostegui, Aklapper, ArielGlenn, Addshore, WMDE-leszek, Ladsgroup, Hazizibinmahdi, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, Iflorez, darthmon_wmde, alaa_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T246159: Setting on read new for items causes increased worst response time and number of open connections
jcrespo added a comment. I am almost made up my mind to give a suggestion, last question, I promise: could you provide an idea of frequency of this query (normally & at peak times). This will be the determinant factor on how much latency we can "spend" on it. I don't need exact values, just an order of magnitude (100, 1000, 1000?) per second. TASK DETAIL https://phabricator.wikimedia.org/T246159 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, jcrespo Cc: jcrespo, Marostegui, Aklapper, ArielGlenn, Addshore, WMDE-leszek, Ladsgroup, Hazizibinmahdi, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, Iflorez, darthmon_wmde, alaa_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T246159: Setting on read new for items causes increased worst response time and number of open connections
jcrespo added a comment. Do you have latency data for each run? Normally clients report query execution time, which you should already have when getting the handlers. TASK DETAIL https://phabricator.wikimedia.org/T246159 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, jcrespo Cc: jcrespo, Marostegui, Aklapper, ArielGlenn, Addshore, WMDE-leszek, Ladsgroup, Hazizibinmahdi, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, Iflorez, darthmon_wmde, alaa_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T208425: [EPIC] Kill the wb_terms table
jcrespo added a comment. Thanks a lot for the work on this- may I suggest a step before the next step (after rebuilding) of "checking all data, old and new, is consistent". This is a lot of data, and even on well thought processes missing rows were discovered after I requested a comparison on other well-though migration, which happened due to mistakes/existing inconsistencies/aborts. May I request such a step, which could be as fast as a simple join query (<5m to run) between old an new to check no rows are missing or extra, and have equivalent data? TASK DETAIL https://phabricator.wikimedia.org/T208425 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: ArielGlenn, jcrespo, Joe, Gehel, alaa_wmde, Marostegui, Jdforrester-WMF, Addshore, Aklapper, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T243701: Wikidata maxlag repeatedly over 5s since Jan20, 2020 (primarily caused by the query service)
jcrespo added a comment. > API maxlag has been repeatedly declared to always be <5s by several people in the past That would sound about right for Mediawiki API (5-10 seconds) ... > If WDQS lag is high, API maxlag is also high ... but this sounds worrying if true- I would wait for wikidata experts to comment on that, which is new to me. That dependency does indeed look like a real problem to me, as I don't think WDQS can keep up with the same promise at the moment based on my casual observation and understanding of the architecture. Thanks for bringing this up, as this has deeper infrastructure implications due to service dependencies. While I understand the need of "slowing bot edits" in case of lag, I can see a problem making MW api and WDQS equivalent in SLA at the moment. > Where have you found WDQS lag promises? I searched on mediawiki.org and Phabricator to see relevant discussions about expected uptime and latency & lag. For the specific issue you are facing, I may be suggest to review SLA expectations about the api (any of it)- and timing out and erroring quickly rather than waiting in case of lag for non-interactive tests. TASK DETAIL https://phabricator.wikimedia.org/T243701 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Addshore, jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, Gehel, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, Manybubbles, Lydia_Pintscher, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T243701: Wikidata queryservice lag repeatedly over 5s since Jan20, 2020
jcrespo added a comment. I am not part of the Wikidata QS team, so I don't have answers, just questions :-D Only chiming in because my team was been tagged on this ticket- please understand we (SREs) are not in direct charge of this service and that someone else should answer with first hand knowledge. Could you provide the documentation of where it is guaranteed that Wikidata Query Lag is going to be < 5s ? I searched and the only thing I found on a draft saying: > Seconds or even a minute or two lag seems acceptable at this point and > As anyone is free to use this endpoint, the traffic sees a lot a variability and thus the performance of the endpoint can vary quite a lot. Discussions at T199228 <https://phabricator.wikimedia.org/T199228>, which would be a good place to provide feedback/requirements, talk about setting it on <5 minutes of lag with some number of nines, not 5 seconds. I would like to know if WDQ team has promised <5s lag, as I would be surprised, given that canonical data storage of all wikis (mariadb), including Wikidata, is only considered lagged starting at 5-10 seconds of delay, and that would be a lower bound (blocker) before indexing and postprocessing, not including downtime. Please note that I am not saying this is invalid or impossible, I am just asking if this should be a feature request rather than a bug report. As far I understand, an interactive, pseudo-real time query interface could be technically possible (?), but would require large architecture refactoring, but AFAIK it was not the goal of the current setup (?). Please someone with actual knowledge, feel free to correct me. You mention: > Pywikibot test environments and CIs But I don't see how pywikibot depending on an external service (WDQS) for CI would be a good idea, could you elaborate on what you are trying to achieve, so maybe there is a better understanding on what the requirements are? Thank you! TASK DETAIL https://phabricator.wikimedia.org/T243701 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Dvorapa, Aklapper, Strainu, darthmon_wmde, ET4Eva, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Vali.matei, Avner, Zppix, Gehel, _jensen, rosalieper, Scott_WUaS, Jonas, FloNight, Xmlizer, Volker_E, Wong128hk, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, GWicke, Dinoguy1000, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T178445: flapping monitoring for recommendation_api on scb
jcrespo edited projects, added Core Platform Team; removed Core Platform Team Legacy (Watching / External). jcrespo added a comment. This is flapping very frequently, but with a 500, not a 429 (scb1002 only, for example, twice per hour). Should I close this and open a new one, or can this be handled here? TASK DETAIL https://phabricator.wikimedia.org/T178445 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, brennen, Joe, Volans, mobrovac, Smalyshev, Gehel, Stashbot, Aklapper, Dzahn, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, ET4Eva, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, LawExplorer, Avner, Zppix, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, FloNight, Wong128hk, Eevans, Hardikj, Wikidata-bugs, aude, Capt_Swing, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T241418: wdqs1006 /srv low on disk space
jcrespo added a comment. This is solved to me, thanks @Gehel https://grafana.wikimedia.org/d/00489/wikidata-query-service?orgId=1=8=1577192533117=1577210803000 TASK DETAIL https://phabricator.wikimedia.org/T241418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Zbyszko, dcausse, Mathew.onipe, Gehel, Aklapper, jcrespo, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T241418: wdqs1006 /srv low on disk space
jcrespo added a comment. Issue started the 22 Dec at around 2:16 https://grafana.wikimedia.org/d/00377/host-overview?orgId=1=wdqs1006=eqiad%20prometheus%2Fops=wdqs=1576541970193=1577197748564 TASK DETAIL https://phabricator.wikimedia.org/T241418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Zbyszko, dcausse, Mathew.onipe, Gehel, Aklapper, jcrespo, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T213191: Some queries causes wdqs-blazegraph on wdqs1006 to crash and restart
jcrespo added a comment. Issue started the 22 Dec at around 2:16 https://grafana.wikimedia.org/d/00377/host-overview?orgId=1=wdqs1006=eqiad%20prometheus%2Fops=wdqs=1576541970193=1577197748564 TASK DETAIL https://phabricator.wikimedia.org/T213191 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Liuxinyu970226, Gehel, Smalyshev, Mathew.onipe, Aklapper, darthmon_wmde, Legado_Shulgin, Nandana, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T241418: wdqs1006 /srv low on disk space
jcrespo created this task. jcrespo added projects: Wikidata-Query-Service, Wikidata. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION 24 Dec 13:38 PROBLEM - Disk space on wdqs1006 is CRITICAL: DISK CRITICAL - free space: /srv 53349 MB (3% inode=99%) TASK DETAIL https://phabricator.wikimedia.org/T241418 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Mathew.onipe, Gehel, Aklapper, jcrespo, darthmon_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T237984: Some property labels are not displayed on Item pages
jcrespo added a comment. I have provided @addshore with the transaction that deleted the `wbx_id = 110010309` row. It happened on 2019-12-05 11:09:27 (although the transaction, and its data context, started in the previous second). TASK DETAIL https://phabricator.wikimedia.org/T237984 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, jcrespo Cc: jcrespo, Marostegui, abian, JAnD, Ash_Crow, Addshore, PKM, Moebeus, alaa_wmde, VIGNERON, Aklapper, Lydia_Pintscher, Ladsgroup, Lea_Lacroix_WMDE, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, Iflorez, darthmon_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T238084: Wikibase/repo/includes/Parsers/MwTimeIsoParser.php: preg_match(): Unknown modifier 'p'
jcrespo created this task.
jcrespo added a project: Wikidata.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
While this error is just a warning (not fatal), please triage to see if it
breaks some intended logic:
{
"_index": "logstash-deploy-2019.11.12",
"_type": "mediawiki",
"_id": "AW5fuzghKWrIH1QRNJlb",
"_version": 1,
"_score": null,
"_source": {
"exception": {
"trace": "#0 [internal function]:
MWExceptionHandler::handleError(integer, string, string, integer, array)\n#1
/srv/mediawiki/php-1.35.0-wmf.5/extensions/Wikibase/repo/includes/Parsers/MwTimeIsoParser.php(162):
preg_match(string, string, NULL)\n#2
/srv/mediawiki/php-1.35.0-wmf.5/extensions/Wikibase/repo/includes/Parsers/MwTimeIsoParser.php(130):
Wikibase\\Repo\\Parsers\\MwTimeIsoParser->parseFromOutputString(Language,
string, integer, string)\n#3
/srv/mediawiki/php-1.35.0-wmf.5/extensions/Wikibase/repo/includes/Parsers/MwTimeIsoParser.php(100):
Wikibase\\Repo\\Parsers\\MwTimeIsoParser->reconvertOutputString(string,
Language)\n#4
/srv/mediawiki/php-1.35.0-wmf.5/vendor/data-values/common/src/ValueParsers/StringValueParser.php(46):
Wikibase\\Repo\\Parsers\\MwTimeIsoParser->stringParse(string)\n#5
/srv/mediawiki/php-1.35.0-wmf.5/vendor/data-values/common/src/ValueParsers/DispatchingValueParser.php(58):
ValueParsers\\StringValueParser->parse(string)\n#6
/srv/mediawiki/php-1.35.0-wmf.5/extensions/Wikibase/repo/includes/Api/ParseValue.php(198):
ValueParsers\\DispatchingValueParser->parse(string)\n#7
/srv/mediawiki/php-1.35.0-wmf.5/extensions/Wikibase/repo/includes/Api/ParseValue.php(110):
Wikibase\\Repo\\Api\\ParseValue->parseStringValue(ValueParsers\\DispatchingValueParser,
string, NULL)\n#8
/srv/mediawiki/php-1.35.0-wmf.5/includes/api/ApiMain.php(1603):
Wikibase\\Repo\\Api\\ParseValue->execute()\n#9
/srv/mediawiki/php-1.35.0-wmf.5/includes/api/ApiMain.php(539):
ApiMain->executeAction()\n#10
/srv/mediawiki/php-1.35.0-wmf.5/includes/api/ApiMain.php(510):
ApiMain->executeActionWithErrorHandling()\n#11
/srv/mediawiki/php-1.35.0-wmf.5/api.php(83): ApiMain->execute()\n#12
/srv/mediawiki/w/api.php(3): require(string)\n#13 {main}",
"code": 0,
"file":
"/srv/mediawiki/php-1.35.0-wmf.5/extensions/Wikibase/repo/includes/Parsers/MwTimeIsoParser.php:162",
"message": "PHP Warning: preg_match(): Unknown modifier 'p'",
"class": "ErrorException"
},
"server": "www.wikidata.org",
"phpversion": "7.2.22-1+0~20190902.26+debian9~1.gbpd64eb7+wmf1",
"wiki": "wikidatawiki",
"channel": "error",
"exception_id": "Xcqu5wpAEDcAAD1qehUAAACK",
"program": "mediawiki",
"type": "mediawiki",
"caught_by": "mwe_handler",
"exception_url":
"/w/api.php?action=wbparsevalue=json=time=%1F1994=%7B%22lang%22%3A%22fr%22%7D",
"http_method": "GET",
"host": "mw1290",
"@version": 1,
"shard": "s8",
"timestamp": "2019-11-12T13:08:55+00:00",
"severity": "err",
"unique_id": "Xcqu5wpAEDcAAD1qehUAAACK",
"level": "ERROR",
"ip": "10.64.32.67",
"mwversion": "1.35.0-wmf.5",
"logsource": "mw1290",
"message": "[Xcqu5wpAEDcAAD1qehUAAACK]
/w/api.php?action=wbparsevalue=json=time=%1F1994=%7B%22lang%22%3A%22fr%22%7D
ErrorException from line 162 of
/srv/mediawiki/php-1.35.0-wmf.5/extensions/Wikibase/repo/includes/Parsers/MwTimeIsoParser.php:
PHP Warning: preg_match(): Unknown modifier 'p'",
"normalized_message": "[{exception_id}] {exception_url}
ErrorException from line 162 of
/srv/mediawiki/php-1.35.0-wmf.5/extensions/Wikibase/repo/includes/Parsers/MwTimeIsoParser.php:
PHP Warning: preg_match(): Unknown modifier 'p'",
"url":
"/w/api.php?action=wbparsevalue=json=time=%1F1994=%7B%22lang%22%3A%22fr%22%7D",
"reqId": "Xcqu5wpAEDcAAD1qehUAAACK",
"tags": [
"input-kafka-rsyslog-udp-localhost",
"rsyslog-udp-localhost",
"kafka",
"es"
],
"referrer": "https://www.wikidata.org/wiki/Q74674585;,
"@timestamp": "2019-11-12T13:08:55.358Z",
[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr
jcrespo added a comment. Unless there is a last time change, this issue is for me fixed. Obviously, please do any data checks/refill that you think is necessary (if any) to fix potential problems caused by this. Thanks for working on this. TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, jcrespo Cc: Krinkle, brennen, mmodell, Lucas_Werkmeister_WMDE, Addshore, #dba, Liuxinyu970226, LarsWirzenius, Lydia_Pintscher, alaa_wmde, Jdforrester-WMF, WMDE-leszek, Marostegui, Ladsgroup, Aklapper, jcrespo, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, Iflorez, darthmon_wmde, Meekrab2012, joker88john, DannyS712, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Pablo-WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Assigned] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language
jcrespo assigned this task to Marostegui. TASK DETAIL https://phabricator.wikimedia.org/T237120 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Marostegui, jcrespo Cc: Marostegui, jcrespo, alaa_wmde, Addshore, Aklapper, Ladsgroup, Iflorez, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, _jensen, rosalieper, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language
jcrespo triaged this task as "Normal" priority. jcrespo moved this task from Triage to Next on the DBA board. jcrespo added a comment. Thanks, acking this, but I hope this is not an emergency, as we may take a bit more than usual to apply it, as 50% of persistence team is on unavailable ATM. Would potentially starting next week be good enough (first on the queue)? TASK DETAIL https://phabricator.wikimedia.org/T237120 WORKBOARD https://phabricator.wikimedia.org/project/board/1060/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Marostegui, jcrespo, alaa_wmde, Addshore, Aklapper, Ladsgroup, Iflorez, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Minhnv-2809, _jensen, rosalieper, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr
jcrespo added a comment. FYI There seems to be a 50% or more write penalty throughput when the rebuild script is running: https://grafana.wikimedia.org/d/00273/mysql?var-server=db1109=9104=eqiad%20prometheus%2Fops=1=1572439082952=1572460682953 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, jcrespo Cc: Addshore, #dba, Liuxinyu970226, LarsWirzenius, Lydia_Pintscher, alaa_wmde, Jdforrester-WMF, WMDE-leszek, Marostegui, Ladsgroup, Aklapper, jcrespo, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, Iflorez, darthmon_wmde, Meekrab2012, joker88john, DannyS712, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Pablo-WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)
jcrespo added a comment. > The maintenance script restarted every 30 mins Small correction, every hour at XX:30 minutes. Maybe you already knew that, but just in case it could lead to missleading analysis. TASK DETAIL https://phabricator.wikimedia.org/T236928 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, jcrespo Cc: jcrespo, Ladsgroup, Liuxinyu970226, Addshore, Aklapper, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, joker88john, DannyS712, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T236928: Wikidata editing via API (and UI) slow (timeouts)
jcrespo added a comment. > We are seeing 5-15s lag on db slaves. Offtopic- I don't believe the lag was real- it just that there was so much overload that lag checking reads/connections by mw would take so much time to finish and return results. This is irrelevant for this ticket (the state was bad), but something that maybe could be optimized or could be missleading in other scenarios in the lag checking code? TASK DETAIL https://phabricator.wikimedia.org/T236928 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, jcrespo Cc: jcrespo, Ladsgroup, Liuxinyu970226, Addshore, Aklapper, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, darthmon_wmde, Meekrab2012, joker88john, DannyS712, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T236464: Deadlock found when trying to get lock
jcrespo added a parent task: T30599: Deadlock tracking bug (tracking). TASK DETAIL https://phabricator.wikimedia.org/T236464 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Liuxinyu970226, #dba, Aklapper, LarsWirzenius, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Agabi10, Pchelolo, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T236464: Deadlock found when trying to get lock
jcrespo added a comment. Is this on Wikidata? T234948 <https://phabricator.wikimedia.org/T234948> TASK DETAIL https://phabricator.wikimedia.org/T236464 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Liuxinyu970226, #dba, Aklapper, LarsWirzenius, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Agabi10, Pchelolo, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr
jcrespo added a comment. We found a possible cause even if partial: maintenance queries for Special page update like `SpecialMostLinked::reallyDoQuery` causing lag and general slowdown on other servers, probably you can check what those do and if they break or make the wikidata migration or make it more difficult. For now manuel killed the ongoing queries. TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: alaa_wmde, jcrespo Cc: alaa_wmde, Jdforrester-WMF, WMDE-leszek, Marostegui, Ladsgroup, Aklapper, jcrespo, Iflorez, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr
jcrespo added a comment. Now there seems to be also many duplicate insert errors. TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: alaa_wmde, jcrespo Cc: alaa_wmde, Jdforrester-WMF, WMDE-leszek, Marostegui, Ladsgroup, Aklapper, jcrespo, Iflorez, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr
jcrespo added a comment. It came back https://logstash.wikimedia.org/goto/de0f6611dde37e811edcba4f530131c0 TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: alaa_wmde, jcrespo Cc: alaa_wmde, Jdforrester-WMF, WMDE-leszek, Marostegui, Ladsgroup, Aklapper, jcrespo, Iflorez, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; tr
jcrespo added a comment. > @jcrespo what rate should we be concerned about? (to plan/monitor ahead) As I said above, with the current rate it is not of immediate concern to me, but my suggestion is to understand why it is happening and what are the consequences, to be 100% sure data is not getting lost or it is affecting the performance of some process (I don't even know where that query is from) so someone else says "we can live with it". From a DBA point of view, the only thing I can see is that this is the top database error by approximately a factor of 10 but I don't have good visibility on the actual application logic impact. TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: alaa_wmde, jcrespo Cc: alaa_wmde, Jdforrester-WMF, WMDE-leszek, Marostegui, Ladsgroup, Aklapper, jcrespo, Iflorez, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; try res
jcrespo added a parent task: T30599: Deadlock tracking bug (tracking). TASK DETAIL https://phabricator.wikimedia.org/T234948 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: WMDE-leszek, Marostegui, Ladsgroup, Aklapper, jcrespo, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T234948: New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; try res
jcrespo created this task.
jcrespo added projects: Wikidata, Wikimedia-database-error,
Wikimedia-production-error.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
Before 2019-10-08T00:00:02 there was almost no wikidata deadlocks, since then
there are 650-1250 per hours at a relatively consistent rate:
https://logstash.wikimedia.org/goto/654dbe1fd799f62f062798cb82e12e23
Happening when executing:
INSERT INTO `wbt_item_terms` (wbit_item_id,wbit_term_in_lang_id) VALUES
('
There is no apparent version change or configuration change at the time, it
could have started at the same time as dumps or other automated process.
The rate is not of immediate concern, but reporting as it could be a
regression from the almost 0 rate we recently achieved.
TASK DETAIL
https://phabricator.wikimedia.org/T234948
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Marostegui, Ladsgroup, Aklapper, jcrespo, darthmon_wmde, DannyS712,
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen,
rosalieper, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T68025: [Story] Monitor size of some Wikidata database tables
jcrespo added a comment. We already have sizes of all uncompressed and compressed tables on zarcillo, those are planned to be shown in a dashboard. The reasons why those are not more public is that we were told not to put those on public prometheus by security as they could compromise the anonymity of certain users on smaller wikis. Please talk to security before doing it. Please talk to us DBAs befere reimplementing an existing feature. TASK DETAIL https://phabricator.wikimedia.org/T68025 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: ArielGlenn, Liuxinyu970226, GoranSMilovanovic, Addshore, jcrespo, Ricordisamoa, hoo, Aklapper, daniel, JanZerebecki, Lydia_Pintscher, darthmon_wmde, Nandana, AndyTan, Lahi, Gq86, QZanden, Marostegui, LawExplorer, Minhnv-2809, _jensen, rosalieper, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)
jcrespo added a comment. Yes, we were worried about the frequency of it (it was the most common DB-related error for some time) because of the performance implications, not that it happens sometimes. I am guessing that this is due to multiple edits happening at the same time about the same subject/item/related item? Sadly there is limited capabilities to make the jobqueue smarter, I think. Maybe there could be strict partitioning on ranges to prevent collision? I don't know. TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Michael, Lydia_Pintscher, jcrespo, Ladsgroup, hoo, Addshore, Marostegui, Aklapper, Krinkle, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T226084: Wikidata server slow
jcrespo added a comment. I am going away now for this week, but if I happen to be right about my previous comment, please don't close the ticket, and make this a child of T220735 <https://phabricator.wikimedia.org/T220735>. If the issues is sill persisting, it is something else still to be identified (but I wasn't able to reproduce it after returning to .8). TASK DETAIL https://phabricator.wikimedia.org/T226084 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Addshore, alaa_wmde, Lea_Lacroix_WMDE, WMDE-leszek, Aklapper, Magnus, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T226084: Wikidata server slow
jcrespo added a comment. > since yesterday Could you try again? There was a regression on .10 that now has been reverted, and the timing matches the .10 deployment to wikidata. This is still a bug if it no longer happens, and we need to notify so it doesn't reappear (we are now on 0.8 again, and I suspect it no longer happens). TASK DETAIL https://phabricator.wikimedia.org/T226084 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Addshore, alaa_wmde, Lea_Lacroix_WMDE, WMDE-leszek, Aklapper, Magnus, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)
jcrespo added a comment. Apparently there are jobqueue issues, potentially impacting to this: T226109 <https://phabricator.wikimedia.org/T226109> TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Michael, Lydia_Pintscher, jcrespo, Ladsgroup, hoo, Addshore, Marostegui, Aklapper, Krinkle, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)
jcrespo added a comment. Thanks, that looks good. I saw no significant impact impact on db infra, if it does more queries/transactions it had not noticeable issue. Did it impact lag/performance on your side? I see a gap here last week- https://logstash.wikimedia.org/goto/b4170e5964398cfb6ca2769f59479faa issue or unrelated? TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Michael, Lydia_Pintscher, jcrespo, Ladsgroup, hoo, Addshore, Marostegui, Aklapper, Krinkle, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T226084: Wikidata server slow
jcrespo added a comment. Could you add the request url/post data used at T226084#5268060 <https://phabricator.wikimedia.org/T226084#5268060> ? Having 1 or more specific endpoints simplifies debugging. TASK DETAIL https://phabricator.wikimedia.org/T226084 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Addshore, alaa_wmde, Lea_Lacroix_WMDE, WMDE-leszek, Aklapper, Magnus, darthmon_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T68108: [Epic] Store media information for files on Wikimedia Commons as structured data
jcrespo added a comment. So just to be clear- based on growth projections I recently got from SDC team of wikibase on commons, the separation is not only convenient or needed for performance, literally s4 would not be able to fit except the initial deployment of data, or do it for very short term. Disk usage is close to 2 TB right now, with and additional 2TB of structured data (metadata only). IOPS would be close to wikidata, which requires a dedicated cluster. This is based on their statistics and projected growth, on top of the current growth and usage. We can scale those out with already budgeted hw, but we need software support. Extra content data would not be a concern as we already planned ES expansion for next fiscal. I think we should plan a bit the different high level tasks related to the database, to schedule them properly, both the ones that directly affect SDC (implementation, wikibase dedicated db, wb_terms migration, MCR) and the ones that affect it indirectly by increase or release of used resources (actor, comment, *links migrations). TASK DETAIL https://phabricator.wikimedia.org/T68108 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Ladsgroup, MarkTraceur, WMDE-leszek, jcrespo, Marostegui, AfroThundr3007730, Stashbot, _jensen, SandraF_WMF, Ramsey-WMF, CCicalese_WMF, PokestarFan, Saerdnaer, Juandev, Elitre, Wesalius, Zppix, NMaia, Mattias_Ostmar-WMSE, Sadads, Poyekhali, -jem-, Deskana, Tfinc, Smalyshev, Jheald, LikeLifer, Yann, intracer, Spinster, Orofarne, Filceolaire, MZMcBride, bzimport, TheDJ, zhuyifei1999, DixonD, Bugreporter, RP88, Aklapper, Matanya, waldyrious, El_Grafo, Daniel_Mietchen, Jdforrester-WMF, GPHemsley, Bene, Legoktm, Nemo_bis, Lokal_Profil, Tobi_WMDE_SW, He7d3r, Petrb, Gilles, jayvdb, Kelson, Steinsplitter, JeroenDeDauw, iecetcwcpggwqpgciazwvzpfjpwomjxn, revi, JanZerebecki, JeanFred, Ricordisamoa, Snowolf, Keegan, Rillke, Bawolff, Fabrice_Florin, Multichill, Liuxinyu970226, Ainali, Tgr, Lydia_Pintscher, jeremyb, Stryn, Ltrlg, daniel, Dereckson, JohnLewis, darthmon_wmde, Nandana, JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, GoranSMilovanovic, QZanden, Tramullas, Acer, V4switch, LawExplorer, Salgo60, Silverfish, rosalieper, Taiwania_Justo, Susannaanas, Wong128hk, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Dinoguy1000, Raymond, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T68108: [Epic] Store media information for files on Wikimedia Commons as structured data
jcrespo added a comment. > The move would likely require some changes to the Wikibase code Could you clarify why? As all other hosts seem to be ok with wikibase server for wikidata being on a separate database? Does MCR use wikibase differently or is it something else? Maybe having 2 wikibase services to use? Note again we notified of this need months in advance during planning phase, as new features require usually extra resources. > Wikibase-specific tables there is wbc_changes, and maybe other wikibase client tables- we don't have a problem with those, as those exist locally on all (wikidata-enabled) wikis. The ones we are worried about are the wikibase server ones (aka the equivalent of s8 on s4). I am not sure we should wait for the refactoring, but as long as the tables on s4 are empty or almost empty (with I think a single row on wb_id_counters), we are flexible. What we don't want is lots of data there that later is more complex to migrate away (we are assuming there will be a large amount of updates there when SDC is at full steam). A few hosts were budgeted for the split for FY2019-2020, we are not in a rush, but it would be nice to have some planning in place for next fiscal year so there are no unexpected delays, specially given persistence team is at 50% capacity ATM. TASK DETAIL https://phabricator.wikimedia.org/T68108 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Ladsgroup, MarkTraceur, WMDE-leszek, jcrespo, Marostegui, AfroThundr3007730, Stashbot, _jensen, SandraF_WMF, Ramsey-WMF, CCicalese_WMF, PokestarFan, Saerdnaer, Juandev, Elitre, Wesalius, Zppix, NMaia, Mattias_Ostmar-WMSE, Sadads, Poyekhali, -jem-, Deskana, Tfinc, Smalyshev, Jheald, LikeLifer, Yann, intracer, Spinster, Orofarne, Filceolaire, MZMcBride, bzimport, TheDJ, zhuyifei1999, DixonD, Bugreporter, RP88, Aklapper, Matanya, waldyrious, El_Grafo, Daniel_Mietchen, Jdforrester-WMF, GPHemsley, Bene, Legoktm, Nemo_bis, Lokal_Profil, Tobi_WMDE_SW, He7d3r, Petrb, Gilles, jayvdb, Kelson, Steinsplitter, JeroenDeDauw, iecetcwcpggwqpgciazwvzpfjpwomjxn, revi, JanZerebecki, JeanFred, Ricordisamoa, Snowolf, Keegan, Rillke, Bawolff, Fabrice_Florin, Multichill, Liuxinyu970226, Ainali, Tgr, Lydia_Pintscher, jeremyb, Stryn, Ltrlg, daniel, Dereckson, JohnLewis, darthmon_wmde, Nandana, JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, GoranSMilovanovic, QZanden, Tramullas, Acer, V4switch, LawExplorer, Salgo60, Silverfish, rosalieper, Taiwania_Justo, Susannaanas, Wong128hk, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Dinoguy1000, Raymond, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T221577: Wikimedia\Rdbms\LBFactory::getEmptyTransactionTicket: LinksUpdate does not have outer scope
jcrespo added a comment. https://logstash.wikimedia.org/goto/fa3d82b03d920e028b05682a499c9f8b Thanks a lot to everybody! TASK DETAIL https://phabricator.wikimedia.org/T221577 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron, jcrespo Cc: kchapman, DannyS712, WMDE-leszek, alaa_wmde, Ladsgroup, Lydia_Pintscher, daniel, akosiaris, JTannerWMF, Krinkle, Jdforrester-WMF, Marostegui, aaron, TerraCodes, Liuxinyu970226, Aklapper, jcrespo, E.S.A-Sheild, darthmon_wmde, Premeditated, joker88john, CucyNoiD, Nandana, NebulousIris, Banyek, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, V4switch, LawExplorer, Vali.matei, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Jonas, Wong128hk, Niharika, Wikidata-bugs, matthiasmullie, aude, Dinoguy1000, Fabrice_Florin, MaxSem, Matanya, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T221577: Wikimedia\Rdbms\LBFactory::getEmptyTransactionTicket: LinksUpdate does not have outer scope
jcrespo added a comment. Just to be clear, this is on HEAD, but not yet deployed/live, correct? https://logstash.wikimedia.org/goto/9eb41e92236727f9095b28489dc244a6 No problem if I am right, just to make sure nothing is being missed. TASK DETAIL https://phabricator.wikimedia.org/T221577 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron, jcrespo Cc: kchapman, DannyS712, WMDE-leszek, alaa_wmde, Ladsgroup, Lydia_Pintscher, daniel, akosiaris, JTannerWMF, Krinkle, Jdforrester-WMF, Marostegui, aaron, TerraCodes, Liuxinyu970226, Aklapper, jcrespo, E.S.A-Sheild, darthmon_wmde, Premeditated, joker88john, CucyNoiD, Nandana, NebulousIris, Banyek, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, V4switch, LawExplorer, Vali.matei, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Jonas, Wong128hk, Niharika, Wikidata-bugs, matthiasmullie, aude, Dinoguy1000, Fabrice_Florin, MaxSem, Matanya, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)
jcrespo added a comment. > and DBAs to babysit the deployment for a couple of days I can do that if you tell me anyting specific -metrics- to check for (other than typical error count and general stability. Specially if initial deployment happens at the beginning of next week for example, near EU timezone. TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Lydia_Pintscher, jcrespo, Ladsgroup, hoo, Addshore, Marostegui, Aklapper, Krinkle, darthmon_wmde, Premeditated, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T138208: Connections to all db servers for wikidata as wikiadmin from snapshot, terbium
jcrespo added a comment. Because this issue, or T143870 <https://phabricator.wikimedia.org/T143870>, and/or long running connections due to mw connection handler, there was a connection issue at https://logstash.wikimedia.org/goto/286304e84262d2fe3335acd5eed135bb and there is likely to be another one. This may or may not create isues on wikidata dumps/exports, depending if retries are done on the latest configuration. TASK DETAIL https://phabricator.wikimedia.org/T138208 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: Lydia_Pintscher, daniel, hoo, ArielGlenn, jcrespo, Zppix, darthmon_wmde, Premeditated, Nandana, Lahi, Gq86, GoranSMilovanovic, Lunewa, QZanden, LawExplorer, _jensen, rosalieper, gnosygnu, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T205045: Exception from LinksUpdate: Deadlock found in database query (from Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages)
jcrespo added a comment. FYI, because in the past I have been told by people they were not aware of the impact. I am **not** too worried about this because I don't think it creates user impact (it impacts only the jobqueue), but this is probably at the moment the most frequent Wikibase database error, and one of the top ones on the wikimedia database infrastructure (probably the top except T221577 <https://phabricator.wikimedia.org/T221577>): https://logstash.wikimedia.org/goto/a224c66bbf0cc9969cfc4e71bc3c2e7a TASK DETAIL https://phabricator.wikimedia.org/T205045 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Ladsgroup, hoo, Addshore, Marostegui, Aklapper, Krinkle, darthmon_wmde, Premeditated, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Jdforrester-WMF, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T68108: [Epic] Store media information for files on Wikimedia Commons as structured data
jcrespo added subscribers: Marostegui, jcrespo. jcrespo added a comment. I've been told wikibase tables have been created on s4. We would like to have been notified of this- we are not sure wikibase for commons should live on s4, if the growth of structured data is as large of it was for wikidata, we should create a separate cluster, dedicated to it (s4, like s1 and s8 are quite bloated). Changing it before or at the beginning is easy, doing it later is more complicated. Please talk to DBAs to understand hw needs. I prefer to request more hw that we need and later not buy it that needing it an not having the budget available. CC @Marostegui. I asked to talk to us months before that deployment happened. TASK DETAIL https://phabricator.wikimedia.org/T68108 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Marostegui, AfroThundr3007730, Stashbot, _jensen, SandraF_WMF, Ramsey-WMF, CCicalese_WMF, PokestarFan, Saerdnaer, Juandev, Elitre, Wesalius, Zppix, NMaia, Mattias_Ostmar-WMSE, Sadads, Poyekhali, -jem-, Deskana, Tfinc, Smalyshev, Jheald, LikeLifer, Yann, intracer, Spinster, Orofarne, Filceolaire, MZMcBride, bzimport, TheDJ, zhuyifei1999, DixonD, Bugreporter, RP88, Aklapper, Matanya, waldyrious, El_Grafo, Daniel_Mietchen, Jdforrester-WMF, GPHemsley, Bene, Legoktm, Nemo_bis, Lokal_Profil, Tobi_WMDE_SW, He7d3r, Petrb, Gilles, jayvdb, Kelson, Steinsplitter, JeroenDeDauw, Micru, revi, JanZerebecki, JeanFred, Ricordisamoa, Snowolf, Keegan, Rillke, Bawolff, Fabrice_Florin, Multichill, Liuxinyu970226, Ainali, Tgr, Lydia_Pintscher, jeremyb, Stryn, Ltrlg, daniel, Dereckson, JohnLewis, alaa_wmde, Nandana, JKSTNK, Lahi, PDrouin-WMF, Gq86, E1presidente, Cparle, Anooprao, GoranSMilovanovic, QZanden, Tramullas, Acer, V4switch, LawExplorer, Silverfish, rosalieper, Susannaanas, Wong128hk, Jane023, Wikidata-bugs, Base, matthiasmullie, aude, Dinoguy1000, Raymond, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T219450: "wikitext" content is not allowed on page … in slot "Main"
jcrespo added a comment. @Yann That is known, it has been on purpose left like that so an actual fix can be tested there first. TASK DETAIL https://phabricator.wikimedia.org/T219450 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: WMDE-leszek, Addshore, jcrespo, Jdforrester-WMF, TerraCodes, Liuxinyu970226, Lucas_Werkmeister_WMDE, Yann, NicoScribe, Fae, Jane023, Clpo13, Aklapper, MB-one, alaa_wmde, CucyNoiD, Nandana, NebulousIris, JKSTNK, jijiki, Gaboe420, Cindylee124, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Poyekhali, _jensen, rosalieper, Wong128hk, Wikidata-bugs, aude, El_Grafo, Dinoguy1000, ArielGlenn, Lydia_Pintscher, He7d3r, Steinsplitter, Mbch331, Jay8g, Krenair, akosiaris, Keegan ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T219450: "wikitext" content is not allowed on page … in slot "Main"
jcrespo added projects: Wikimedia-Incident, Wikimedia-production-error. TASK DETAIL https://phabricator.wikimedia.org/T219450 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Jdforrester-WMF, TerraCodes, Liuxinyu970226, Lucas_Werkmeister_WMDE, Yann, NicoScribe, Fae, Jane023, Clpo13, Aklapper, MB-one, alaa_wmde, CucyNoiD, Nandana, NebulousIris, JKSTNK, jijiki, Gaboe420, Cindylee124, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Poyekhali, _jensen, rosalieper, Wong128hk, Wikidata-bugs, aude, El_Grafo, Dinoguy1000, ArielGlenn, Lydia_Pintscher, He7d3r, Steinsplitter, Mbch331, Jay8g, Krenair, akosiaris, Keegan ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T219450: "wikitext" content is not allowed on page … in slot "Main"
jcrespo added a comment. We have several confirmations this is working now. TASK DETAIL https://phabricator.wikimedia.org/T219450 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Jdforrester-WMF, TerraCodes, Liuxinyu970226, Lucas_Werkmeister_WMDE, Yann, NicoScribe, Fae, Jane023, Clpo13, Aklapper, MB-one, alaa_wmde, CucyNoiD, Nandana, NebulousIris, JKSTNK, Gaboe420, Cindylee124, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Poyekhali, _jensen, rosalieper, Wong128hk, Wikidata-bugs, aude, El_Grafo, Dinoguy1000, Lydia_Pintscher, Steinsplitter, Mbch331, Keegan ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T208425: [EPIC] Kill the wb_terms table
jcrespo added subscribers: Gehel, Joe, jcrespo. jcrespo added a comment. I see you are going to ask DBAs at T219145 <https://phabricator.wikimedia.org/T219145>, that is great. As a heads up, because I saw there is a chance of starting using other data stores (which by itself is not an issue), instead of or in addition to MySQL to give a heads up to service operations SREs and probably search too- in the past there has been misunderstandings with sending important/data that has to be persisted to datastores that ops cannot guarantee persistance/are not properly replicated between dcs (e.g. redis). Please do not go beyond the research phase without taking SREs with your proposal to avoid future misunderstandings about storage requirements. CC @Joe @Gehel TASK DETAIL https://phabricator.wikimedia.org/T208425 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, Joe, Gehel, alaa_wmde, Marostegui, Jdforrester-WMF, Addshore, Aklapper, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack
jcrespo added a comment. labsdb1009.mgmt (stress on management interface) is down according to icinga for 14 hours (around net maintenance), maybe a loose cable or misconfiguration? Not a huge blocker, but better make sure it is not an intended/known state. TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, jcrespo Cc: jcrespo, Tgr, Dbrant, JoeWalsh, Bstorm, thcipriani, akosiaris, Joe, fgiunchedi, hashar, ArielGlenn, jijiki, Addshore, Catrope, elukey, Marostegui, Stashbot, Paladox, gerritbot, Aklapper, BBlack, Cmjohnson, ayounsi, alaa_wmde, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, kostajh, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Soum213, Taiwania_Justo, Thibaut120094, Wong128hk, Wikidata-bugs, aude, Southparkfan, mark, Lydia_Pintscher, Darkdadaah, faidon, Nikerabbit, Arrbee, santhosh, KartikMistry, Jdforrester-WMF, Mbch331, Jay8g, Ltrlg ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T212690: DBQueryTimeoutError on Wikidata's Special:Nuke
jcrespo added a subtask: T188679: Nuke should batch the deletions. TASK DETAIL https://phabricator.wikimedia.org/T212690 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, gerritbot, Addshore, Marostegui, Aklapper, abian, alaa_wmde, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Glaisher, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T212690: DBQueryTimeoutError on Wikidata's Special:Nuke
jcrespo added a comment. I am going to be bold and say that maybe the solution to timeouts (or at least part of the solution for timeouts) like this one reported on enwiki: "PHP fatal error: entire web request took longer than 60 seconds and timed out" would be T188679 <https://phabricator.wikimedia.org/T188679> TASK DETAIL https://phabricator.wikimedia.org/T212690 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jcrespo Cc: jcrespo, gerritbot, Addshore, Marostegui, Aklapper, abian, alaa_wmde, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g, Glaisher, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T215616: Improve interlingual links across wikis through Wikidata IDs
jcrespo added a comment. the API works good for query specific pages/entities, not for example to know which pages that existing in X_wiki are missing on the Y_wiki. Sure, and you are free to either: a) Setup your own database from dumps or even expose it to other people b) Do a feature request for Mediawiki to support that I think a) has more chances to go through, similar to T59617, but if you want b), you are not adding the right people here https://www.mediawiki.org/wiki/RFCTASK DETAILhttps://phabricator.wikimedia.org/T215616EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Tbayer, jcrespo, EBernhardson, Halfak, Nuria, JAllemandou, diego, Nandana, Akovalyov, Banyek, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Avner, Minhnv-2809, _jensen, Luke081515, Wikidata-bugs, aude, Capt_Swing, Dinoguy1000, Mbch331, Jay8g, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T215616: Improve interlingual links across wikis through Wikidata IDs
jcrespo added a comment. diego added a project: DBA. I don't understand what is the actionable here for us. Without context, I would say that: (1) querying the wb_items_per_site table in the wikidatawiki on MariaDB, or (2) through the sitelinks on Wikidata Json dumps. That is not accurate, wikidata has an api for that: https://www.wikidata.org/w/api.php?action=""> Which returns a JSON to further analyze. Of course, you cannot join that with anything else, but those would be features outside of mediawiki, and that an analytics-focus data store could handle, or your own scratch data? Unless you are suggesting mediawiki (the software) should have that complex queries out of the box, I am not sure how a #DBAs could help you?TASK DETAILhttps://phabricator.wikimedia.org/T215616EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, EBernhardson, Halfak, Nuria, JAllemandou, diego, Nandana, Akovalyov, Banyek, AndyTan, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, Avner, Minhnv-2809, _jensen, Luke081515, Wikidata-bugs, aude, Capt_Swing, Dinoguy1000, Mbch331, Jay8g, Krenair, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T202764: Wikidata produces a lot of failed requests for recentchanges API
jcrespo added a subtask: T104459: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves. TASK DETAILhttps://phabricator.wikimedia.org/T202764EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Stashbot, Marostegui, Banyek, Reedy, gerritbot, Krinkle, Addshore, Yurik, jcrespo, Imarlier, Ladsgroup, Lydia_Pintscher, WMDE-leszek, Aklapper, Gehel, Smalyshev, Legado_Shulgin, Nandana, thifranc, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T202764: Wikidata produces a lot of failed requests for recentchanges API
jcrespo removed a parent task: T104459: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves. TASK DETAILhttps://phabricator.wikimedia.org/T202764EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Stashbot, Marostegui, Banyek, Reedy, gerritbot, Krinkle, Addshore, Yurik, jcrespo, Imarlier, Ladsgroup, Lydia_Pintscher, WMDE-leszek, Aklapper, Gehel, Smalyshev, Legado_Shulgin, Nandana, thifranc, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T214644: DBQueryErrors when trying to create Wikidata Items
jcrespo added a comment. Thanks @Addshore !TASK DETAILhttps://phabricator.wikimedia.org/T214644EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Ladsgroup, Addshore, Aklapper, abian, Zeroth, Legado_Shulgin, Nandana, thifranc, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Marostegui, LawExplorer, Minhnv-2809, Zppix, _jensen, Wong128hk, Luke081515, Wikidata-bugs, aude, faidon, Jdforrester-WMF, Mbch331, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T214644: DBQueryErrors when trying to create Wikidata Items
jcrespo added a comment. The edit rate not being affected means this was a probably relatively localized problem https://grafana.wikimedia.org/d/00208/edit-count?orgId=1=1548357720872=1548374190181 and not something that happens regularly.TASK DETAILhttps://phabricator.wikimedia.org/T214644EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Ladsgroup, Addshore, Aklapper, abian, Zeroth, Legado_Shulgin, Nandana, thifranc, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Marostegui, LawExplorer, Minhnv-2809, Zppix, _jensen, Wong128hk, Luke081515, Wikidata-bugs, aude, faidon, Jdforrester-WMF, Mbch331, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T214644: DBQueryErrors when trying to create Wikidata Items
jcrespo added subscribers: Ladsgroup, jcrespo.jcrespo triaged this task as "Low" priority.jcrespo moved this task from Triage to Blocked external/Not db team on the DBA board.jcrespo added a comment.
Sorry you suffered this issues. I can see a few hundred Wikibase\UpsertSqlIdGenerator::upsertId errors at that time.
In particular to:
INSERT INTO `wb_id_counters` (id_type,id_value) VALUES ('wikibase-item','1') ON DUPLICATE KEY UPDATE id_value = LAST_INSERT_ID(id_value + 1)
I will send this information to wikidata team and CC @Ladsgroup , as that count could be a contention point. However the error didn't repeat itself in the next 17 hours, and corrected itself after a few minutes, so maybe it was just the typical bot going out of hand creating new items and not leaving anyones else resources to do the same (something that it is common on Wikidata). It could also be related to a temporary issue with the new version deployment: https://wikitech.wikimedia.org/w/index.php?title=Deployments=1814619#deploycal-item-20190124T2000
I can see an increase activity on enwiki, and notification messages. I can also see that the automatic overload checkers kicked in and solved the issue automatically in a few minutes, so, - in a way - something was already done to "fix" it (the query killer takes care of bad database states).TASK DETAILhttps://phabricator.wikimedia.org/T214644WORKBOARDhttps://phabricator.wikimedia.org/project/board/1060/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Ladsgroup, Addshore, Aklapper, abian, Zeroth, Legado_Shulgin, Nandana, thifranc, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Marostegui, LawExplorer, Minhnv-2809, Zppix, _jensen, Wong128hk, Luke081515, Wikidata-bugs, aude, faidon, Jdforrester-WMF, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
