[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE, Gehel Cc: Gehel, Lucas_Werkmeister_WMDE, tfmorris, CBogen, EBernhardson, Ladsgroup,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Lucas_Werkmeister_WMDE moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞) board. Lucas_Werkmeister_WMDE added a comment. Alright, great! (Leaving open in case @dcausse isn’t done with this Needs Reporting column yet.) TASK DETAIL

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Ladsgroup added a comment. I see. The impact doesn't show up in the term store and db, where it matters the most. So it's good to consider done IMO. TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Lucas_Werkmeister_WMDE added a comment. No, we still try to look the content up in the parser cache, I think – the change was that if the parser cache isn’t hit, then we generate parser output without HTML and don’t store it in the parser cache anymore. TASK DETAIL

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Ladsgroup added a comment. It shouldn't hit ParserCache anymore but it seems we got it again? https://grafana.wikimedia.org/d/00106/parser-cache?viewPanel=8=now-12h=1=now=wikibase_item=wikibase_property TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

dcausse assigned this task to Lucas_Werkmeister_WMDE. dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T239931 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-01-13T12:09:46Z] Synchronized wmf-config/InitialiseSettings.php: T239931 : Revert "Disable sanity check cirrus jobs for Wikidata" (duration: 01m 16s) TASK DETAIL

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

gerritbot added a comment. Change 655389 **merged** by jenkins-bot: [operations/mediawiki-config@master] Revert "Disable sanity check cirrus jobs for Wikidata" https://gerrit.wikimedia.org/r/655389 TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

gerritbot added a comment. Change 655389 had a related patch set uploaded (by DCausse; owner: DCausse): [operations/mediawiki-config@master] Revert "Disable sanity check cirrus jobs for Wikidata" https://gerrit.wikimedia.org/r/655389 TASK DETAIL

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

dcausse added a comment. In T239931#6736120 , @Lucas_Werkmeister_WMDE wrote: > Agreed. Should we (Wikidata team) do the config change or leave it to you? :) We can ship the config change no worries :) TASK DETAIL

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Lucas_Werkmeister_WMDE added a comment. Agreed. Should we (Wikidata team) do the config change or leave it to you? :) TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE Cc:

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

dcausse added a comment. In T239931#6719994 , @EBernhardson wrote: > With the holidays over and everyone back, i think we can turn this on? Sounds good to me! TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

EBernhardson added a comment. With the holidays over and everyone back, i think we can turn this on? TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Gehel,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

dcausse added a comment. The sanitizer is working OK with increased concurrency (T266762 ), we might try to enable it again on wikidata and sees how it performs. TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

CBogen set the point value for this task to "2". TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: CBogen Cc: Gehel, Lucas_Werkmeister_WMDE, tfmorris, CBogen, EBernhardson, Ladsgroup, Addshore,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Lucas_Werkmeister_WMDE added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞). TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE Cc: Gehel,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Gehel removed projects: Wikibase wb_terms leftovers 2020, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞). Gehel added a comment. Saneitizer is running slower than usual at the moment, this needs to be investigated first. See T266762 and

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

dcausse added a comment. Moving to blocked until we know what's causing T266762 TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc:

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Lucas_Werkmeister_WMDE added a comment. To summarize: the sanitizer was turned off on Wikidata because it caused excessive database load; during the wb_terms cleanup hike, we optimized it to not do as much unneeded work (mainly rEWBA01de21628b23: Override

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Lucas_Werkmeister_WMDE changed the task status from "Stalled" to "Open". Lucas_Werkmeister_WMDE moved this task from Stalled/Waiting to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞) board. Lucas_Werkmeister_WMDE added a comment. Alright,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

EBernhardson added a comment. Saneitizer was turned back on last week, everything there is working well and wikidata can be reenabled any time. TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

WMDE-leszek added a project: Wikibase wb_terms leftovers 2020. TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: WMDE-leszek Cc: Lucas_Werkmeister_WMDE, tfmorris, Pintoch, CBogen, EBernhardson,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Lucas_Werkmeister_WMDE set Due Date to Tue, Oct 27, 12:00 AM. TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE Cc: Lucas_Werkmeister_WMDE, tfmorris, Pintoch, CBogen,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Lucas_Werkmeister_WMDE added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞). TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE Cc: Lucas_Werkmeister_WMDE,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Lucas_Werkmeister_WMDE changed the task status from "Open" to "Stalled". Lucas_Werkmeister_WMDE added a comment. Alright, then we can try turning on the sanitizer on Wikidata either next week (after wmf.14 is rolled out) or the week after that (probably safer). TASK DETAIL

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

gerritbot added a comment. Change 633170 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@master] Override EntityHandler::getParserOutputForIndexing https://gerrit.wikimedia.org/r/633170 TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

EBernhardson added a comment. All seems reasonable to me. Note that the saneitizer is globally turned off as of a week ago due to a separate incident. Expecting to turn that back on this week. TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

gerritbot added a comment. Change 632752 **merged** by jenkins-bot: [mediawiki/core@master] Clarify HTML generation for indexing in ContentHandler https://gerrit.wikimedia.org/r/632752 TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Addshore added a comment. In T239931#6536170 , @Lucas_Werkmeister_WMDE wrote: > If we no longer use the HTML in the sanitizer (see the above changes), do we still need a gradual ramp-up when reenabling the sanitizer or is it okay to

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Ladsgroup added a comment. Agreed. TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup Cc: Lucas_Werkmeister_WMDE, tfmorris, Pintoch, CBogen, EBernhardson, Ladsgroup, Addshore,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Lucas_Werkmeister_WMDE added a comment. In T239931#6331040 , @Ladsgroup wrote: > In T239931#6330953 , @CBogen wrote: > >> @addshore do you know if the load has been reduced enough

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

gerritbot added a comment. Change 632751 **abandoned** by Lucas Werkmeister (WMDE): [mediawiki/extensions/Jade@master] Override getParserOutputForIndexing to ensure HTML Reason: No longer needed with PS2 of I50f3a530f2, where generating HTML is no longer skipped by default.

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

gerritbot added a comment. Change 633170 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)): [mediawiki/extensions/Wikibase@master] Override EntityHandler::getParserOutputForIndexing https://gerrit.wikimedia.org/r/633170 TASK DETAIL

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Lucas_Werkmeister_WMDE added a comment. The above two changes attempt to implement the optimization outlined in the IRC log in the task description: to stop generating HTML for search indexing by setting the `'generate-html'` hint. (Wikibase seems to respect this hint, as far as I could

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: tfmorris, Pintoch, CBogen, EBernhardson, Ladsgroup, Addshore, Aklapper, dcausse,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

gerritbot added a comment. Change 632752 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)): [mediawiki/core@master] Don’t generate HTML in getParserOutputForIndexing https://gerrit.wikimedia.org/r/632752 TASK DETAIL

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

gerritbot added a comment. Change 632751 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)): [mediawiki/extensions/Jade@master] Override getParserOutputForIndexing to ensure HTML https://gerrit.wikimedia.org/r/632751 TASK DETAIL

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: tfmorris, Pintoch, CBogen, EBernhardson,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

CBogen merged a task: T264566: Redirected entity still present in search results after 6 months. CBogen added subscribers: Pintoch, tfmorris. TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

CBogen triaged this task as "Medium" priority. TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: CBogen Cc: CBogen, EBernhardson, Ladsgroup, Addshore, Aklapper, dcausse, Wilmanbeno, Akuckartz,

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

Ladsgroup added a comment. In T239931#6330953 , @CBogen wrote: > @addshore do you know if the load has been reduced enough that this is no longer an issue? The load has been reduced (we dropped the old term store and the new one is

[Wikidata-bugs] [Maniphest] T239931: Reduce the impact of the sanitizer on wikidata

CBogen added a comment. @addshore do you know if the load has been reduced enough that this is no longer an issue? TASK DETAIL https://phabricator.wikimedia.org/T239931 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: CBogen Cc: CBogen,