Addshore closed this task as "Resolved". Addshore added a comment.
Other than the work that has been done here I'm going to close this with the following comment that i left somewhere else In T219123#5924185 <https://phabricator.wikimedia.org/T219123#5924185>, @Addshore wrote: > Spent a lot of time chatting with @jcrespo today. > After some experimenting it seems that the main issue that we have been seeing was probably caused by disk reads due to data not being in the db cache yet. > This slowness of disk reads was also added to as these db tables are compressed (at least on db1126). > I got to Q6 million for reads from clients 30 mins or so ago and everything seems stable. > I turned off the batch write migration this morning but turned it back on 10 mins ago (batch size 100) and everything still seems to be indicating that it is okay. > The code snippet for warming the cache of db1126 is below for the range 2-4 million, and can be run from the mwmaint servers. > > for i in {2000000..4000000..100000}; do echo "$i..."; time sql wikidatawiki --host db1126 -- -e "SELECT wbit_item_id as id, wby_name as type, wbxl_language as language, wbx_text as text FROM wbt_item_terms LEFT JOIN wbt_term_in_lang ON wbit_term_in_lang_id = wbtl_id LEFT JOIN wbt_type ON wbtl_type_id = wby_id LEFT JOIN wbt_text_in_lang ON wbtl_text_in_lang_id = wbxl_id LEFT JOIN wbt_text ON wbxl_text_id = wbx_id WHERE wbit_item_id BETWEEN $i AND ($i + 100000)" > /dev/null; done > > This cache warmer should be performed twice on each range shortly before turning on reads from the new store for that range. > The cache efficiency can be seen in the below snapshot where: > a) cache warm 2-4 million pass 1 > b) cache warm 2-4 million pass 2 > c) cache warm 4-6 million pass 1 > d) cache warm 4-6 million pass 2 > F31647226: image.png <https://phabricator.wikimedia.org/F31647226> > This can also be seen in the disk IO > F31647231: image.png <https://phabricator.wikimedia.org/F31647231> > Although I have started an initial cache warm for Q6-8 million I probably won't bump the read level higher today. TASK DETAIL https://phabricator.wikimedia.org/T246159 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup, Addshore Cc: jcrespo, Marostegui, Aklapper, ArielGlenn, Addshore, WMDE-leszek, Ladsgroup, Hazizibinmahdi, Iflorez, darthmon_wmde, alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
