subject:"\[Wikidata\-bugs\] \[Maniphest\] T280627\: Investigation of remaining page

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-09-15 Thread Maintenance_bot

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Maintenance_bot
Cc: Tarrow, Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, maantietaja, Y.ssk, 
Muchiri124, Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, alaa_wmde, 
holger.knust, EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, 
GoranSMilovanovic, QZanden, LawExplorer, Poyekhali, _jensen, rosalieper, 
Agabi10, Taiwania_Justo, Scott_WUaS, Pchelolo, Ixocactus, Wong128hk, 
Wikidata-bugs, aude, El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Suran38, 
Biggs657, Lalamarie69, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, 
Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, 
Bsandipan, Lewizho99, Maathavan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-09-15 Thread gerritbot

gerritbot added a comment.


  Change 721083 **merged** by jenkins-bot:
  
  [mediawiki/extensions/Wikibase@REL1_36] Fix: When a sitelink change happens, 
always dispatch to the site
  
  https://gerrit.wikimedia.org/r/721083

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, gerritbot
Cc: Tarrow, Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Suran38, Biggs657, Invadibot, 
Lalamarie69, maantietaja, Y.ssk, Juan90264, Muchiri124, Alter-paule, 
Hazizibinmahdi, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Iflorez, 
WDoranWMF, Kent7301, alaa_wmde, holger.knust, EvanProdromou, joker88john, 
CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Ramsey-WMF, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, 
Maathavan, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, Scott_WUaS, 
Pchelolo, Ixocactus, Wong128hk, Wikidata-bugs, aude, El_Grafo, Dinoguy1000, 
Steinsplitter, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-09-15 Thread gerritbot

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, gerritbot
Cc: Tarrow, Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Suran38, Biggs657, Invadibot, 
Lalamarie69, maantietaja, Y.ssk, Juan90264, Muchiri124, Alter-paule, 
Hazizibinmahdi, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Iflorez, 
WDoranWMF, Kent7301, alaa_wmde, holger.knust, EvanProdromou, joker88john, 
CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Ramsey-WMF, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Lewizho99, 
Maathavan, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, Scott_WUaS, 
Pchelolo, Ixocactus, Wong128hk, Wikidata-bugs, aude, El_Grafo, Dinoguy1000, 
Steinsplitter, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-09-15 Thread gerritbot

gerritbot added a comment.


  Change 721083 had a related patch set uploaded (by Addshore; author: 
Addshore):
  
  [mediawiki/extensions/Wikibase@REL1_36] Fix: When a sitelink change happens, 
always dispatch to the site
  
  https://gerrit.wikimedia.org/r/721083

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, gerritbot
Cc: Tarrow, Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, maantietaja, Y.ssk, 
Muchiri124, Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, alaa_wmde, 
holger.knust, EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, 
GoranSMilovanovic, QZanden, LawExplorer, Poyekhali, _jensen, rosalieper, 
Agabi10, Taiwania_Justo, Scott_WUaS, Pchelolo, Ixocactus, Wong128hk, 
Wikidata-bugs, aude, El_Grafo, Dinoguy1000, Steinsplitter, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-12 Thread Maintenance_bot

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Maintenance_bot
Cc: Tarrow, Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, maantietaja, Muchiri124, 
Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, alaa_wmde, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, GoranSMilovanovic, QZanden, 
LawExplorer, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan, Lalamarie69, 
Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, 
Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-12 Thread gerritbot

gerritbot added a comment.


  Change 686768 **merged** by jenkins-bot:
  
  [mediawiki/extensions/Wikibase@master] Fix: When a sitelink change happens, 
always dispatch to the site
  
  https://gerrit.wikimedia.org/r/686768

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, gerritbot
Cc: Tarrow, Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, Lalamarie69, maantietaja, 
Muchiri124, Alter-paule, Hazizibinmahdi, Beast1978, CBogen, Un1tY, Akuckartz, 
Hook696, Iflorez, WDoranWMF, Kent7301, alaa_wmde, holger.knust, EvanProdromou, 
joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Ramsey-WMF, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, 
Lewizho99, Maathavan, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-11 Thread Addshore

Addshore closed this task as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Addshore
Cc: Tarrow, Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, Lalamarie69, maantietaja, 
Muchiri124, Alter-paule, Hazizibinmahdi, Beast1978, CBogen, Un1tY, Akuckartz, 
Hook696, Iflorez, WDoranWMF, Kent7301, alaa_wmde, holger.knust, EvanProdromou, 
joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Ramsey-WMF, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, 
Lewizho99, Maathavan, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-11 Thread Addshore

Addshore added a comment.


  Jakon on mattermost:
  
  > amazing, I can even reproduce it locally:
  >
  > create client page
  > create repo item
  > add sitelink from repo to the client page. make sure not to touch the 
client page after this.
  > wb_changes entries appear
  > dispatchChanges.php does nothing, runJobs.php --wiki client does nothing
  > edit the client page
  >  dispatching suddenly works

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Addshore
Cc: Tarrow, Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, Lalamarie69, maantietaja, 
Muchiri124, Alter-paule, Hazizibinmahdi, Beast1978, CBogen, Un1tY, Akuckartz, 
Hook696, Iflorez, WDoranWMF, Kent7301, alaa_wmde, holger.knust, EvanProdromou, 
joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Ramsey-WMF, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, 
Lewizho99, Maathavan, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-11 Thread Tarrow

Tarrow added a comment.


  Looks like this investigation can probably be moved to done then? Or are we 
waiting for a second reproduction on test? @Jakob_WMDE / @Michael

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Tarrow
Cc: Tarrow, Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, Lalamarie69, maantietaja, 
Muchiri124, Alter-paule, Hazizibinmahdi, Beast1978, CBogen, Un1tY, Akuckartz, 
Hook696, Iflorez, WDoranWMF, Kent7301, alaa_wmde, holger.knust, EvanProdromou, 
joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Ramsey-WMF, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, 
Lewizho99, Maathavan, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-08 Thread Addshore

Addshore added a comment.


  We can use `page.page_links_updated` to see when the page links / a refresh 
links job has last been run on the page!
  (thanks again @Krinkle)
  In PHP we can do ` WikiPage::factory(Title::newFromText('Category:Interior of 
Chapelle des Anges de Neunhoffen à Dambach (Bas-Rhin)'))->getLinksTimestamp() `
  
  If we run the job synchronously on a debug server, we can see the timestamp 
get bumped correctly.
  
>>> WikiPage::factory(Title::newFromText('Category:Interior of Chapelle des 
Anges de Neunhoffen à Dambach (Bas-Rhin)'))->getLinksTimestamp();
=> "20210505193823"
>>> (new RefreshLinksJob( Title::newFromText('Category:Interior of Chapelle 
des Anges de Neunhoffen à Dambach (Bas-Rhin)'), 
['rootJobTimestamp'=>wfTimestampNow()] ))->run();
=> true
>>> WikiPage::factory(Title::newFromText('Category:Interior of Chapelle des 
Anges de Neunhoffen à Dambach (Bas-Rhin)'))->getLinksTimestamp();
=> "20210505195510"
  
  Trying to schedule a job in the queue manually (and wait for it to execute) 
for the commons category:
  
>>> WikiPage::factory(Title::newFromText('Category:Interior of Chapelle des 
Anges de Neunhoffen à Dambach (Bas-Rhin)'))->getLinksTimestamp();
=> "20210505195510"
>>> wfTimestampNow();
=> "20210505195834"
>>> $pu->scheduleRefreshLinks( [ Title::newFromText('Category:Interior of 
Chapelle des Anges de Neunhoffen à Dambach (Bas-Rhin)') ], 
['rootJobTimestamp'=>wfTimestampNow()], 'update', 'uid:addshore' );
=> null

  
  It takes a while and at the time of writing this comment I am still waiting 
for it to execute...
  
  We could see the job enter the kafka queue:
  
addshore@stat1007:~$ kafkacat -b kafka-main1001.eqiad.wmnet -p 0 -t 
'eqiad.mediawiki.job.refreshLinks' -o -100 | grep commons.wikimedia.org | 
grep Neunhoffen

{"$schema":"/mediawiki/job/1.0.0","meta":{"uri":"https://placeholder.invalid/wiki/Special:Badtitle","request_id":"766bd8c164f5e0761ffc3a89","id":"9f61fcb8-85db-4e66-906c-155a16fddb16","dt":"2021-05-05T19:58:17Z","domain":"commons.wikimedia.org","stream":"mediawiki.job.refreshLinks"},"database":"commonswiki","type":"refreshLinks","sha1":"3a14ea46843e2f7b3b687bdf45112fbca14ac89a","params":{"rootJobTimestamp":"20210505195817","causeAction":"update","causeAgent":"uid:addshore","namespace":14,"title":"Interior_of_Chapelle_des_Anges_de_Neunhoffen_à_Dambach_(Bas-Rhin)","requestId":"766bd8c164f5e0761ffc3a89"},"mediawiki_signature":"32c72f98713180f9aefc8059c33016bb221d6950"}
  
  It took a while so at the same time I tried on testwiki:
  
>>> 
WikiPage::factory(Title::newFromText('PagePropsies'))->getLinksTimestamp();
=> "20210505124631"
>>> wfTimestampNow();
=> "20210505201007"
>>> $pu->scheduleRefreshLinks( [ Title::newFromText('PagePropsies') ], 
['rootJobTimestamp'=>wfTimestampNow()], 'update', 'uid:addshore' );
=> null
...
>>> 
WikiPage::factory(Title::newFromText('PagePropsies'))->getLinksTimestamp();
=> "20210505202436"
  
  So on testwiki this took 14 mins 30 seconds (roughly what @Jakob_WMDE 
@Michael & I saw testing).
  And a commonswiki category took TBA a time when it get executed from the 
queue
  
  --
  
  Some more relevant conversation from IRC
  
  > 9:37 PM  addshore: is your job refreshLinks or 
refreshLinksPrioritized?
  > 9:38 PM  How do I know that? I was assuming refreshLinks, but 
perhaps that's where I went wrong?
  > 9:38 PM  since those 2 have VERY different guarantees.. for 
prioritize one mean delay is a few seconds, while for regular it's 6 hours
  > 9:45 PM  kafkacat -b kafka-main1001.equad.wmnet
  > 9:45 PM  kafka-jumbo is an analytics cluster where jobs were 
replicated at some point, but ottomata is in the process of stopping the 
mirroring
  > 9:47 PM  so, with kafka-main you should see you job in 
non-prioritized non-partitioned topic. e.g kafkacat -b 
kafka-main1001.eqiad.wmnet:9092 -p 0 -t eqiad.mediawiki.job.refreshLinks
  > 9:47 PM  if you are sending it via wikibase WikiPageUpdater like 
in your paste on phab
  
  
  
  --
  
  So applying this to investigate another one of these seemingly more "randon" 
failures:
  
  - I grabbed a commons category page from 
https://www.wikidata.org/w/index.php?title=Special:Contributions&offset=20210503061116&limit=500&target=Pi+bot&namespace=all&tagfilter=&newOnly=1&start=&end=
  - 2 May 18:48 the sitelink was added 
https://www.wikidata.org/w/index.php?title=Q106686114&diff=1413255927&oldid=1413255916
  - I confirm with 
`(MediaWiki\MediaWikiServices::getInstance()->getParserCache())->get(Title::newFromText('Category:Views_from_Palacio_Municipal_(Montevideo)')->toPageRecord(),ParserOptions::newCanonical())->getProperty('wikibase_item');`
 that the property exists in the parser output
  - I confirm with 
`PageProps::getInstance()->getProperties([Title::newFromText('Category:Views_from_Palacio_Municipal_(Montevideo)')],'wikibas

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-07 Thread Addshore

Addshore added a comment.


  In T280627#7070564 , 
@Addshore wrote:
  
  > Another learning.
  > Chained greps on `kafkacat` seem to break and give different results, so 
should be avoided...
  
  
  
  > 9:23 PM  Is there something I should know about grepping the 
output of kafkacat twice? I seem to not get matches if I pipe it through 2 
greps? https://phabricator.wikimedia.org/T280627#7070564
  > 9:39 PM  addshore: i'd guess something about flushing, try 
--line-buffered on the first grep
  > 10:15 PM  ebernhardson: yup, that solves it!

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Addshore
Cc: Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, Lalamarie69, maantietaja, 
Muchiri124, Alter-paule, Hazizibinmahdi, Beast1978, CBogen, Un1tY, Akuckartz, 
Hook696, Iflorez, WDoranWMF, Kent7301, alaa_wmde, holger.knust, EvanProdromou, 
joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Ramsey-WMF, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, 
Lewizho99, Maathavan, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-07 Thread Addshore

Addshore moved this task from Doing to Peer Review on the Wikidata-Campsite 
(Wikidata-Campsite-Iteration-∞) board.
Addshore added a comment.


  Moving the investigation to peer review.
  Though I think the parent ticket should be picked up "in full" now for a 
proper fix and tests / cleanup

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

WORKBOARD
  https://phabricator.wikimedia.org/project/board/3539/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Addshore
Cc: Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, Lalamarie69, maantietaja, 
Muchiri124, Alter-paule, Hazizibinmahdi, Beast1978, CBogen, Un1tY, Akuckartz, 
Hook696, Iflorez, WDoranWMF, Kent7301, alaa_wmde, holger.knust, EvanProdromou, 
joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Ramsey-WMF, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, 
Lewizho99, Maathavan, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-07 Thread gerritbot

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, gerritbot
Cc: Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, Lalamarie69, maantietaja, 
Muchiri124, Alter-paule, Hazizibinmahdi, Beast1978, CBogen, Un1tY, Akuckartz, 
Hook696, Iflorez, WDoranWMF, Kent7301, alaa_wmde, holger.knust, EvanProdromou, 
joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Ramsey-WMF, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, 
Lewizho99, Maathavan, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-07 Thread gerritbot

gerritbot added a comment.


  Change 686768 had a related patch set uploaded (by Addshore; author: 
Addshore):
  
  [mediawiki/extensions/Wikibase@master] Fix: When a sitelink change happens, 
always dispatch to the site
  
  https://gerrit.wikimedia.org/r/686768

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, gerritbot
Cc: Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, maantietaja, Muchiri124, 
Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, alaa_wmde, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, GoranSMilovanovic, QZanden, 
LawExplorer, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-07 Thread Addshore

Addshore added a comment.


  I couldn't resist poking @daniel with the findings.
  
  > 9:34 PM  I'm trying to remember whether saving the wikidata page 
would add subscriptions for all sites that have sitelinks. 
  > 9:35 PM  onParserCacheSaveComplete
  > 9:36 PM  another possibility is that we used to dispatch to all 
sites that had sitelinks or something?
  > 9:36 PM  I think we forced dispatching to the site that was just 
added, if the event was "add a sitelink".
  
  We went digging around and found exactly what daniel was referring to
  
  > 9:44 PM  So that code path depends on this change being an 
`ItemChange` and i dont see where that is made right now
  > 9:45 PM  I think it'S because only ItemChanges have site links. Or 
at least, that's how it used to be.
  > 9:46 PM  Wiring for 'WikibaseClient.EntityChangeFactory' puts a 
mapping into EntityChangeFactory, which has Item::ENTITY_TYPE => 
ItemChange::class,
  > 9:47 PM  so the only other thing I see that could go wrong is 
this diff condition
  > 9:47 PM  
https://github.com/wikimedia/Wikibase/blob/33c8fb8a6fd09d15cc7a3c47bdc6c60e398ce0e2/repo/includes/ChangeDispatcher.php#L362
  > 9:54 PM  i got it
  > 9:55 PM  >>> 
MediaWiki\MediaWikiServices::getInstance()->getService('WikibaseRepo.Store')->getEntityChangeLookup()->loadByChangeIds([547009])[0]
  > 9:55 PM  => Wikibase\Repo\Notifications\RepoItemChange {#5576}
  > 9:55 PM  But the condition is `if ( $change instanceof ItemChange 
) {`
  
  The object being returned by the factory was changed 
https://github.com/wikimedia/Wikibase/commit/d05a0c7ee31f62dd00de583646c45470891f5204#diff-ee6dbb8537c6e9d555c439b164d9ccc2e4d40aedcff12ec86331e5c5637bf7ebR748
  but this condition in dispatching was not changed.

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Addshore
Cc: Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, maantietaja, Muchiri124, 
Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, alaa_wmde, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, GoranSMilovanovic, QZanden, 
LawExplorer, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-07 Thread Addshore

Addshore added a comment.


  Another learning.
  Chained greps on `kafkacat` seem to break and give different results, so 
should be avoided...
  
  See below where I would expect the job to be shown in both greps
  
addshore@stat1007:~$ kafkacat -C -b kafka-main1001.eqiad.wmnet -p 0 -t 
'eqiad.mediawiki.job.ChangeNotification' -o -10 | grep 
858d2fa00061557d1f5f64a2b650c7b61b604994

{"$schema":"/mediawiki/job/1.0.0","meta":{"uri":"https://placeholder.invalid/wiki/Special:Badtitle","request_id":"59d70f2de736400dbaf3d953","id":"58fd9760-325d-426e-9f39-a0d3b20d9e88","dt":"2021-05-07T19:44:22Z","domain":"test-commons.wikimedia.org","stream":"mediawiki.job.ChangeNotification"},"database":"testcommonswiki","type":"ChangeNotification","params":{"repo":false,"changeIds":[547010],"rootJobTimestamp":"20210507194421"},"mediawiki_signature":"858d2fa00061557d1f5f64a2b650c7b61b604994"}
% Reached end of topic eqiad.mediawiki.job.ChangeNotification [0] at offset 
319708891
^C
addshore@stat1007:~$ kafkacat -C -b kafka-main1001.eqiad.wmnet -p 0 -t 
'eqiad.mediawiki.job.ChangeNotification' -o -10 | grep 
858d2fa00061557d1f5f64a2b650c7b61b604994 | grep test
% Reached end of topic eqiad.mediawiki.job.ChangeNotification [0] at offset 
319708898

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Addshore
Cc: Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, maantietaja, Muchiri124, 
Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, alaa_wmde, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, GoranSMilovanovic, QZanden, 
LawExplorer, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-07 Thread Addshore

Addshore added a comment.


  **Reproduction case:**
  
  - Come up with some easy grepable string, such as "IMGREPABLE87654"
  - Create a new item on testwikidatawiki with this string as the label
  - Create a new page on testcommonswiki with this string as the title
  - Close the page on testcommonswiki and do not re open it or query the api 
etc about this page at all
  - Add a sitelink to testcommonswiki and the page you create
  - Wait and do not touch anything for some time.
  - You have reproduced the issue
  
  **Reproduction verification:**
  
  - You can see that a `wb_changes` entry was made for the item using something 
like `select * from wb_changes_subscription where cs_subscriber_id = 
"testcommonswiki" AND cs_entity_id = "Q215220";`
  - You can assume that the dispatch process did look at this wb_changes entry 
(not code snippet provided but I tested this separately)
  - You can see that no `ChangeNotificationJob` was scheduled for the entry in 
`wb_changes` using something like `kafkacat -C -b kafka-main1001.eqiad.wmnet -p 
0 -t 'eqiad.mediawiki.job.ChangeNotification' -o -1 | grep test | grep 
547009` where the last grep is the ID of the `wb_changes` row.
  - As this job was not scheduled you would also not expect to see the html 
cache update job or the links update job.
  - You can confirm that a html cache update job has not run, as the page still 
has a parser cache entry 
`(MediaWiki\MediaWikiServices::getInstance()->getParserCache())->get(Title::newFromText($t)->toPageRecord(),ParserOptions::newCanonical())->getCacheTime();`
 where $t is the page title.
  - You can also confirm nothing is in the page_props table 
`ageProps::getInstance()->getProperties([Title::newFromText($t)],'wikibase_item');`
  - You can also confirm that the page on testcommonswiki is NOT subscribed to 
the Item `select * from wb_changes_subscription where cs_subscriber_id = 
"testcommonswiki" AND cs_entity_id = "Q215220";`
  
  **The issue**
  The dispatch process relies on clients being subscribed to entities in order 
for the `ChangeNotificationJobs` to be sent.
  This can be seen in 
https://github.com/wikimedia/Wikibase/blob/master/repo/includes/ChangeDispatcher.php#L400
 
  Which ultimately ends up querying this `wb_changes_suvscription` table
  
https://github.com/wikimedia/Wikibase/blob/33c8fb8a6fd09d15cc7a3c47bdc6c60e398ce0e2/client/includes/Usage/Sql/SqlSubscriptionManager.php#L100
  This is ONLY written to by a job on client, on parser output save
  
https://github.com/wikimedia/Wikibase/blob/33c8fb8a6fd09d15cc7a3c47bdc6c60e398ce0e2/client/includes/Hooks/DataUpdateHookHandler.php#L195
  Thus if either of the following are not the case, the dispatch process will 
not happen for the client page:
  
  - The page is not already subscribed for some reason
  - The page does not get it's parser output generated for some reason between 
the wikidata edit and the dispatch process processing the change
  
  **Making the process work with 1 more edit:**
  
  - It's now time to load the commons page again and purge the page with 
`action=purge`
  - This will trigger the job adding the subscription, which can be seen with 
something like `select * from wb_changes_subscription where cs_subscriber_id = 
"testcommonswiki" AND cs_entity_id = "Q215220";`
  - We can now make another edit to the item (adding a description for example) 
https://test.wikidata.org/w/index.php?title=Q215220&diff=540854&oldid=540853
  - Which should make a new change that we can grep for a job for and see such 
as `kafkacat -C -b kafka-main1001.eqiad.wmnet -p 0 -t 
'eqiad.mediawiki.job.ChangeNotification' -o -10 | grep 547010`
  - This will in turn schedule the 2 jobs we expect (htmlcacheupdate and links 
update)
  - We can see that these have happened as now the parsercache timestamp will 
be bumped again, and the page_props table will be updated shortly ( a few mins 
at most)
  
  **Suspicion**
  This code has existed in this way forever (ish)
  At some point Wikibase used to fully purge the pages on client, maybe even in 
a slightly different place in code? at some point this changed, and this flow 
and assumptions of order was broken somehow?
  
  Will wait until next week for @Jakob_WMDE and or @Michael to confirm this 
reproduction, then I think we could close the investigation and come up with a 
plan of attack

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Addshore
Cc: Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, maantietaja, Muchiri124, 
Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, alaa_wmde, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, GoranSMilovanovic, QZanden, 
LawExplorer

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-06 Thread Addshore

Addshore closed subtask T282160: Enable testcommonswiki as a client of 
testwikidatawiki with dispatching running as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Addshore
Cc: Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, maantietaja, Muchiri124, 
Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, alaa_wmde, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, GoranSMilovanovic, QZanden, 
LawExplorer, Poyekhali, _jensen, rosalieper, Agabi10, Taiwania_Justo, 
Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, Wikidata-bugs, aude, 
El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-05 Thread Addshore

Addshore added a comment.


  I chatted a little with @Krinkle and @daniel and ...
  
  Firstly:
  
  - The `purgeWebCache` correctly triggers a `UpdateHtmlCacheJob` which runs 
quickly and "dirties" the parsercache for the page being dispatched to
  - This leads to cache misses when using a debug line like 
`(MediaWiki\MediaWikiServices::getInstance()->getParserCache())->get(Title::newFromText('Category:Interior
 of Chapelle des Anges de Neunhoffen à Dambach 
(Bas-Rhin)')->toPageRecord(),ParserOptions::newCanonical())->getCacheTime();`
  - A user view will parse the page correctly, and the page property for 
wikibase_item will be visible in the parser output with 
`(MediaWiki\MediaWikiServices::getInstance()->getParserCache())->get(Title::newFromText('Category:Interior
 of Chapelle des Anges de Neunhoffen à Dambach 
(Bas-Rhin)')->toPageRecord(),ParserOptions::newCanonical())->getProperty('wikibase_item');`
  - At this stage the page_prop will not be in the page_props table.
  
  Secondly:
  
  - The `scheduleRefreshLinks` should trigger a `RefreshLinksJob`. (These take 
a little longer to run in most cases)
  - At some unknown time in the future the `page_props` table should be 
populated with the `wikibase_item` page property. (we saw this happen a few 
times)
  - You'd be able to see this in the DB, or using something like 
`PageProps::getInstance()->getProperties([Title::newFromText('Category:Interior 
of Chapelle des Anges de Neunhoffen à Dambach (Bas-Rhin)')],'wikibase_item');`
  
  -
  
  Now trying to apply this again to the random failures seen in T233520: 
page_props wikibase_item is sometimes not added to client pages when a sitelink 
is added on a repo  where 
`page_props` is seemingly never populated
  
  One can make a WikiPageUpdater for a client with:
  
$services = MediaWiki\MediaWikiServices::getInstance();
$pu = new \Wikibase\Client\Changes\WikiPageUpdater( 
JobQueueGroup::singleton(), \Wikibase\Client\WikibaseClient::getLogger( 
$services ), $services->getStatsdDataFactory());
  
  And then trigger a web cache job with something like:
  
$pu->purgeWebCache( [ Title::newFromText('Category:Interior of Chapelle des 
Anges de Neunhoffen à Dambach (Bas-Rhin)') ], 
['rootJobTimestamp'=>wfTimestampNow()], 'update', 'uid:addshore' );
  
  As with the bullet point lists above, we see the cache start missing at this 
point, and a web view will re parse the page.
  
  When trying to do the same with the page for the refreshlinks job, I can do 
the following:
  
$pu->scheduleRefreshLinks( [ Title::newFromText('Category:Interior of 
Chapelle des Anges de Neunhoffen à Dambach (Bas-Rhin)') ], 
['rootJobTimestamp'=>wfTimestampNow()], 'update', 'uid:addshore' );
  
  In theory this should schedule the job, and no error occurs.
  But in both of these job cases I do not see anything matching the job I 
scheduled in kafka (perhaps I'm looking in the wrong place).
  I waited 10-30 minutes and did not see the `page_props` for a category page 
on commons end up updating. (could be the job was still in the queue?)
  
  I then manually ran the job on the debug server:
  
(new RefreshLinksJob( Title::newFromText('Category:Interior of Chapelle des 
Anges de Neunhoffen à Dambach (Bas-Rhin)'), 
['rootJobTimestamp'=>wfTimestampNow()] ))->run();
  
  And the page_prop now instantly appears in the page_props table
  
>>> 
PageProps::getInstance()->getProperties([Title::newFromText('Category:Interior 
of Chapelle des Anges de Neunhoffen à Dambach (Bas-Rhin)')],'wikibase_item');
=> [
 103128055 => "Q106684122",
   ]
  
  
  
  -
  
  So either:
  
  1. The call to scheduleRefreshLinks is not actually scheduling a job? and 
thus the page_props table is never being updated?
  2. The jobs for the pages that appear to not have page_props just have not 
executed yet?
  3. something else?
  
  Looking at 
https://grafana.wikimedia.org/d/CbmStnlGk/jobqueue-job?orgId=1&var-dc=eqiad%20prometheus%2Fk8s&var-job=refreshLinks
 though these jobs shoul only take a few hours at most?
  But really this is ~15 mins currently.
  This lines up with what @Michael @Jakob_WMDE and I were seeing between 
testwiki and testwikidata, but not with what is seen in T233520 
 on commonswiki?

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Addshore
Cc: Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, maantietaja, Muchiri124, 
Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, alaa_wmde, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, GoranSMilovanovic, QZanden, 
LawExplorer, P

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-05 Thread Addshore

Addshore added subscribers: Jakob_WMDE, daniel, Krinkle.
Addshore added a comment.


  So after the look that we (@Jakob_WMDE @Michael & I) took yesterday we think 
we figured out some things.
  The reproduction case in T233520#6898755 
 half works, but we seem to 
consistently see the page_props table eventually get populated with the needed 
value.
  
  Seemingly the following is happening:
  
  - Create a sitelink
- Dispatching process
  - The sitelink edit kicks off the dispatching process for getting the 
change to the needed clients
  - Within a few seconds we see the parser cache / output timestamp change 
to the current time (implying that dispatching is done and the page was purged 
and now reparsed). This can be seen in a HTML comment of the page on the client.
  - This lines up with delay in dispatching (which is now very short, only 
a few seconds) https://www.wikidata.org/wiki/Special:DispatchStats and the fact 
that the jobs on the client also get executed quickly 
https://grafana.wikimedia.org/d/CbmStnlGk/jobqueue-job?orgId=1&var-dc=eqiad%20prometheus%2Fk8s&var-job=ChangeNotification
- ParserOutput generation after dispatch
  - We see the page reparse as the timestamp for ParserOutput generation 
change to the current time.
  - Using `shell.php` we can see that the page_props added by wikibase hook 
are immediately available in the new `ParserOutput` object stored in the cache.
- This can be checked with something like 
`(MediaWiki\MediaWikiServices::getInstance()->getParserCache())->get(Title::newFromText('PagePropsies')->toPageRecord(),ParserOptions::newCanonical())->getProperty('wikibase_item');`
  - Using the API or looking directly in the DB we do not see the page 
property for example 
https://test.wikipedia.org/w/api.php?action=query&format=json&prop=pageprops&titles=PagePropsies
- Some time later the value seems to appear in the page_props table (the 
parser output timestamp does not change here)
  
  I have enquired to @daniel to see if this gap between generating 
`ParserOutput` with our property in it, and population of the `page_props` 
table is normal or to be expected (to try and fully understand the flow here).
  @Krinkle also popped in on IRC and noted that we could be seeing a user view 
cause the initial parse? which would lead to the `page_props` population occur 
in a job after (todo check this)
  
  Another thing that we then discussed was the fact that this dispatching 
process now only takes a few seconds.
  It could be a reasonable assumption that the issue seen in T233520: 
page_props wikibase_item is sometimes not added to client pages when a sitelink 
is added on a repo  is now caused by 
a race condition due to this fast process.
  The ticket was created in 2019 and dispatching got fast in 2018 (see T205865: 
Investigate decrease in wikidata dispatch times due to eqiad -> codfw DC switch 
). Throughout multiple 
investigations we have not really found any other reason for this change in 
behaviour.
  
  **The possible race condition here would be:**
  
  - Edit adding the sitelink to an Item happens on wikidata
  - `wb_changes` table is initially populated via `notifyOnPageModified` during 
the main request (see 
https://www.mediawiki.org/wiki/Manual:Hooks/RevisionFromEditComplete)
  - In `ItemHandler::getSecondaryDataUpdates` we add updates to call 
`saveLinksOfItem` or `deleteLinksOfItem` which can be executed in a deferred 
fashion (post request), and I believe this can also land in a job?
  
  This can lead to wb_changes being written some period of time before 
sitelinks end up in the secondary store, which is where the client reads from 
to populate the page_properties of the client page.
  
  A similar race condition probably leads to what we see in T248984: Client 
recentchanges entries sometimes don't have their wb_changes.change_id reference 
set 
  `$changeStore->saveChange` is called in `onRecentChangeSave` that adds recent 
change info to the already stored wb_changes row.
  But this row may already have been read by the client due to fast 
dispatching, thus this data ends up missing from the client.

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, Addshore
Cc: Krinkle, daniel, Jakob_WMDE, Mike_Peel, Aklapper, Jheald, 
Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, aaron, tstarling, LibrErli, 
Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, Bencemac, Tacsipacsi, 
Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, maantietaja, Muchiri124, 
Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, alaa_wmde, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, GoranSMilovanovic, QZanden, 
LawExplorer, Poyekhali, _j

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-03 Thread Michael

Michael added a comment.


  incremental progress update:
  I managed to connect the tentative starting point ChangeHandler 

 (you can see ChangeHandlers bigger context in the new arch docs 
)
 with the likely endpoint ClientParserOutputDataUpdater 
.
 The assumption is that the problem is somewhere between those two places in 
the code.
  
  - `\Wikibase\Client\Changes\ChangeHandler::handleChange` is where we start
  - One of that methods main actions is to call 
`\Wikibase\Client\Changes\WikiPageUpdater::scheduleRefreshLinks`
  - which creates a new `\RefreshLinksJob` and adds it to the job queue via 
`\JobQueueGroup::lazyPush`
- In that constructor a field `removeDuplicates` is probably set to true in 
our case - maybe our job gets deduplicated with one that doesn't do the trick?
  - at some point that job is `\RefreshLinksJob::run` and executes the private 
method `\RefreshLinksJob::runForTitle`
  - that `runForTitle` method then requests `\RefreshLinksJob::getParserOutput`
  - This method eventually gets fresh parser output, but checks a couple of 
conditions first, one looks particularly suspect:
  
$cachedOutput = $this->getParserOutputFromCache( $parserCache, 
$page, $revision, $stats );
if ( $cachedOutput ) {
return $cachedOutput;
}
  
  - however,
1. we should have dealt with that by unsetting rootJobTimestamp 

2. This should not have been an issue in the first place, because if the 
timestamp that this job was generated is older than the time when the 
ParserOutput was last cached, then that means that the the ParserOutput has 
been generated in the meantime and the changes should have been picked at that 
moment anyway
  - That being said, if there is seemingly no cached output, then it should be 
generated by `\MediaWiki\Revision\RenderedRevision::getRevisionParserOutput` 
which is too complex for me to dive into, until I have a good reason to suspect 
that the issue lies somewhere inside
  - but I assume at some point `\AbstractContent::getParserOutput` gets called
  - which runs the hook `onContentAlterParserOutput` (c.f. official docs for 
the hook ContentAlterParserOutput 
)
  - And we have a handler registered for that hook: 
`\Wikibase\Client\Hooks\ParserOutputUpdateHookHandler::onContentAlterParserOutput`
  - That handler ultimately calls the last step 
`\Wikibase\Client\ParserOutput\ClientParserOutputDataUpdater::updateItemIdProperty`
 which is supposed to modify the page props
  
  Something in that chain above is probably broken under some conditions.
  
  Preliminary learning/confusion: We don't actually care about the main 
functionality of `RefreshLinksJob`, we care about the ParserOutput being 
recreated.

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael
Cc: Mike_Peel, Aklapper, Jheald, Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, 
aaron, tstarling, LibrErli, Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, 
Bencemac, Tacsipacsi, Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, 
maantietaja, Muchiri124, Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, 
alaa_wmde, holger.knust, EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, 
GoranSMilovanovic, QZanden, LawExplorer, Poyekhali, _jensen, rosalieper, 
Agabi10, Taiwania_Justo, Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, 
Wikidata-bugs, aude, El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-05-03 Thread Michael

Michael claimed this task.
Michael moved this task from To Do (prioritised from top to bottom) to Doing on 
the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞) board.
Restricted Application added a project: User-Michael.

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

WORKBOARD
  https://phabricator.wikimedia.org/project/board/3539/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael
Cc: Mike_Peel, Aklapper, Jheald, Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, 
aaron, tstarling, LibrErli, Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, 
Bencemac, Tacsipacsi, Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, 
maantietaja, Muchiri124, Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, 
alaa_wmde, holger.knust, EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, 
GoranSMilovanovic, QZanden, LawExplorer, Poyekhali, _jensen, rosalieper, 
Agabi10, Taiwania_Justo, Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, 
Wikidata-bugs, aude, El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-04-21 Thread Lydia_Pintscher

Lydia_Pintscher moved this task from Prioritized Product (prioritised from top 
to bottom) to Wikidata-Campsite-Iteration-∞ on the Wikidata-Campsite board.
Lydia_Pintscher edited projects, added Wikidata-Campsite 
(Wikidata-Campsite-Iteration-∞); removed Wikidata-Campsite.

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

WORKBOARD
  https://phabricator.wikimedia.org/project/board/3402/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: Mike_Peel, Aklapper, Jheald, Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, 
aaron, tstarling, LibrErli, Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, 
Bencemac, Tacsipacsi, Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, 
maantietaja, Muchiri124, Hazizibinmahdi, CBogen, Akuckartz, Iflorez, WDoranWMF, 
alaa_wmde, holger.knust, EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, 
GoranSMilovanovic, QZanden, LawExplorer, Poyekhali, _jensen, rosalieper, 
Agabi10, Taiwania_Justo, Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, 
Wikidata-bugs, aude, El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

2021-04-20 Thread Lydia_Pintscher

Lydia_Pintscher created this task.
Lydia_Pintscher added projects: Wikidata, Commons, User-Addshore, 
User-Ladsgroup, Platform Engineering, MW-1.35-notes (1.35.0-wmf.22; 
2020-03-03), Wikidata-Campsite, wdwb-tech.

TASK DESCRIPTION
  We want to investigate the cause of the remaining issues with updating 
page_props on the client as lined out in T233520 
. For context, steps how to 
reproduce and previous discussions see T233520 
.
  
  Goal of the investigation: Figure out the cause of the remaining update issue 
and propose one or more solutions.

TASK DETAIL
  https://phabricator.wikimedia.org/T280627

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: Mike_Peel, Aklapper, Jheald, Lucas_Werkmeister_WMDE, Addshore, WMDE-leszek, 
aaron, tstarling, LibrErli, Lea_Lacroix_WMDE, Ladsgroup, RolandUnger, Urbanecm, 
Bencemac, Tacsipacsi, Kizule, CCicalese_WMF, Lydia_Pintscher, Invadibot, 
maantietaja, Muchiri124, Hazizibinmahdi, CBogen, Akuckartz, WDoranWMF, 
holger.knust, EvanProdromou, Nandana, Lahi, Gq86, Ramsey-WMF, 
GoranSMilovanovic, QZanden, LawExplorer, Poyekhali, _jensen, rosalieper, 
Agabi10, Taiwania_Justo, Scott_WUaS, Pchelolo, Jonas, Ixocactus, Wong128hk, 
Wikidata-bugs, aude, El_Grafo, Dinoguy1000, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

[Wikidata-bugs] [Maniphest] T280627: Investigation of remaining page_props update issue [timebox: 2 days]

24 matches

Site Navigation

Mail list logo

Footer information