[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-16 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-17T06:26:18Z] <_joe_> stop 
testing on mw1281, T260329 

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Majavah, JMeybohm, Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, 
ArielGlenn, RhinosF1, Joe, lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-16 Thread Joe 
Joe added a comment.


  To test the hypothesis that this is related to firejail use, we're sending 1 
req/s to one appserver to use pygments, to see if that has any particularly ill 
effect.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: Majavah, JMeybohm, Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, 
ArielGlenn, RhinosF1, Joe, lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-16 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-17T05:28:00Z] <_joe_> 
depooling mw1281 for testing for T260329 


TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Majavah, JMeybohm, Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, 
ArielGlenn, RhinosF1, Joe, lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-14 Thread JMeybohm 
JMeybohm added a comment.


  Looking at the values today it's pretty clear that mw1382 wins and mw1381 
takes the second place.
  The overall memory usage looks like it's safe to leave it this way over the 
weekend. On Monday we should reboot the clusters again, with 
"cgroup.memory=nokmem".

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JMeybohm
Cc: JMeybohm, Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, 
ArielGlenn, RhinosF1, Joe, lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread JMeybohm 
JMeybohm updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JMeybohm
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread JMeybohm 
JMeybohm updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JMeybohm
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-13T14:34:04Z] <_joe_> 
rebooting mw1381 with a newer kernel, mw1383 as control with the old kernel 
T260329 

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-13T14:31:37Z] <_joe_> 
installing kernel 4.19.0-0.bpo.9 on mw1381 T260329 


TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-13T13:10:47Z] <_joe_> 
forcing a puppet run on the api appservers in eqiad  T260329 


TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Maintenance_bot 
Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, 
joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, 
Lewizho99, Maathavan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-13T13:07:45Z] 
 Synchronized wmf-config/CommonSettings.php: revert 
enabling of lilypond (again) T257091 T260329 
 (duration: 00m 59s)

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Alter-paule, Beast1978, Un1tY, 
Akuckartz, Hook696, darthmon_wmde, WDoranWMF, Kent7301, holger.knust, 
EvanProdromou, joker88john, Legado_Shulgin, CucyNoiD, Nandana, Klaas_Z4us_V, 
Gaboe420, Giuliamocci, Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, 
LawExplorer, Lewizho99, Zppix, Maathavan, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread gerritbot 
gerritbot added a comment.


  Change 619623 **merged** by jenkins-bot:
  [operations/mediawiki-config@master] Revert "Re-enable LilyPond/Score in safe 
mode (3rd attempt)"
  
  https://gerrit.wikimedia.org/r/619623

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Alter-paule, Beast1978, Un1tY, 
Akuckartz, Hook696, darthmon_wmde, WDoranWMF, Kent7301, holger.knust, 
EvanProdromou, joker88john, Legado_Shulgin, CucyNoiD, Nandana, Klaas_Z4us_V, 
Gaboe420, Giuliamocci, Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, 
LawExplorer, Lewizho99, Zppix, Maathavan, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread gerritbot 
gerritbot added a comment.


  Change 619623 had a related patch set uploaded (by Giuseppe Lavagetto; owner: 
Giuseppe Lavagetto):
  [operations/mediawiki-config@master] Revert "Re-enable LilyPond/Score in safe 
mode (3rd attempt)"
  
  https://gerrit.wikimedia.org/r/619623

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread gerritbot 
gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Alter-paule, Beast1978, Un1tY, 
Akuckartz, Hook696, darthmon_wmde, WDoranWMF, Kent7301, holger.knust, 
EvanProdromou, joker88john, Legado_Shulgin, CucyNoiD, Nandana, Klaas_Z4us_V, 
Gaboe420, Giuliamocci, Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, 
LawExplorer, Lewizho99, Zppix, Maathavan, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread JMeybohm 
JMeybohm updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JMeybohm
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread JMeybohm 
JMeybohm updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JMeybohm
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-13T11:05:30Z]  
depool mw1380 for downgrade of 
poppler-utils,libpoppler-glib8,libpoppler64,curl,libcurl3,libcurl3-gnutls,libpython3.5,python3.5,libpython3.5-stdlib,python3.5-minimal,libpython3.5-minimal,imagemagick-6-common,libmagickcore-6.q16-3,libmagickwand-6.q16-3,imagemagick-6.q16,imagemagick,e2fslibs,e2fsprogs,libcomerr2,libss2
 and reboot - T260329 

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-13T10:17:13Z]  
depool mw1379 for downgrade of 
poppler-utils,libpoppler-glib8,libpoppler64,curl,libcurl3,libcurl3-gnutls,libpython3.5,python3.5,libpython3.5-stdlib,python3.5-minimal,libpython3.5-minimal,imagemagick-6-common,libmagickcore-6.q16-3,libmagickwand-6.q16-3,imagemagick-6.q16,imagemagick,e2fslibs,e2fsprogs,libcomerr2,libss2
 and reboot - T260329 

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread JMeybohm 
JMeybohm updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JMeybohm
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Joe 
Joe added a comment.


  In T260329#6382296 , 
@Ladsgroup wrote:
  
  > For the wikibase part, I highly doubt it, the php entry point calls 
`wfLoadExtension` internally.
  
  Hi strongly doubt it has any effect as well, but I'd prefer to test our 
assumptions first.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Ladsgroup 
Ladsgroup added a comment.


  For the wikibase part, I highly doubt it, the php entry point calls 
`wfLoadExtension` internally.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup
Cc: Ladsgroup, Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, 
RhinosF1, Joe, lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-13T08:45:28Z] <_joe_> 
downgrading imagemagick on mw1378 T260329 


TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Tarrow, Addshore, CDanis, Aklapper, jijiki, ArielGlenn, RhinosF1, Joe, 
lmata, wkandek, JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, 
LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-13T07:55:24Z] <_joe_> 
downgrading curl/libcurl3/libcurl3-gnutls on mw1377 T260329 


TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: CDanis, Aklapper, jijiki, ArielGlenn, RhinosF1, Joe, lmata, wkandek, 
JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, 
Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, 
elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, 
Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Joe 
Joe updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: CDanis, Aklapper, jijiki, ArielGlenn, RhinosF1, Joe, lmata, wkandek, 
JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, 
Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, 
elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, 
Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Joe 
Joe added a comment.


  The list of software updated that day on the appservers is at P12221 


TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: CDanis, Aklapper, jijiki, ArielGlenn, RhinosF1, Joe, lmata, wkandek, 
JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, 
Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, 
elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, 
Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260329: Figure what change caused the ongoing memleak on mw appservers

2020-08-13 Thread Joe 
Joe created this task.
Joe added projects: serviceops, Operations, Sustainability (Incident Followup), 
Platform Engineering, Wikidata.

TASK DESCRIPTION
  Something induced a progressive memory leak on all servers serving MediaWiki 
starting on august 4th between 8 and 13 UTC.
  
  The problem is more evident in servers that receive more traffic, which seems 
to correlate the memleak with something happening in MediaWiki.
  
  However, these are the patches that were released during that timeframe:
  
  - enabling Score in production again 
https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/618039
  - this change to how wikibase is loaded 
https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/618266/
  
  both seem like extremely simple changes we could just rollback easily.
  
  At the same time, some software packages were updated, the most relevant of 
which is imagemagik
  
  Plan of action:
  
  - Rollback those two changes
  - Check if memory growth continues
  - rollback imagemagick on one server, reboot it, see how it behaves 
afterwards.

TASK DETAIL
  https://phabricator.wikimedia.org/T260329

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: CDanis, Aklapper, jijiki, ArielGlenn, RhinosF1, Joe, lmata, wkandek, 
JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, 
Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, 
elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, 
Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs