[Wikidata-bugs] [Maniphest] T281480: Cannot access the database: Too many connections

[Joe]

Joe added a comment.


  There is definitely something going very wrong with memcached:
  
  
https://grafana.wikimedia.org/d/00316/memcache?viewPanel=60&orgId=1&from=now-30d&to=now
  
  shows misses increasing across the board

TASK DETAIL
  https://phabricator.wikimedia.org/T281480

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: Pchelolo, Joe, jcrespo, Urbanecm, Aklapper, LarsWirzenius, Invadibot, 
Devnull, LSobanski, maantietaja, Akuckartz, Iflorez, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
Marostegui, LawExplorer, Minhnv-2809, _jensen, rosalieper, Agabi10, Scott_WUaS, 
Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Addshore, Mbch331, Jay8g, 
Krenair, AMooney
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T281480: Cannot access the database: Too many connections

[Pchelolo]

Pchelolo added a comment.


  > A better candidate for changing something is probably 
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/677299. @Pchelolo is looking 
into it.
  
  This one is a partial revert of a previously added optimization that was not 
needed, and is fixing PoolCounter - before PoolCounter couldn't fetch the 
parsed content after waiting for a lock. But we can try reverting it if there's 
no other guesses

TASK DETAIL
  https://phabricator.wikimedia.org/T281480

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Pchelolo
Cc: Pchelolo, Joe, jcrespo, Urbanecm, Aklapper, LarsWirzenius, Invadibot, 
Devnull, LSobanski, maantietaja, Akuckartz, Iflorez, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
Marostegui, LawExplorer, Minhnv-2809, _jensen, rosalieper, Agabi10, Scott_WUaS, 
Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Addshore, Mbch331, Jay8g, 
Krenair, AMooney
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T281480: Cannot access the database: Too many connections

[Joe]

Joe added a subscriber: Pchelolo.
Joe added a comment.


  In T281480#7046160 , @Joe 
wrote:
  
  > Given we only make requests to external storage when parsercache has a 
miss, it seemed sensible to look for corresponding patterns in parsercache.
  >
  > I see we introduced a new category of misses on the same date 
"miss_absent_metadata", see 
https://grafana-rw.wikimedia.org/d/00106/parser-cache?viewPanel=7&orgId=1&from=now-30d&to=now
 which seems related.
  
  that's probably a red herring, it's a bugfix Daniel Kinzler made in 
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/677346 and looks like a 
bugfix for the counter.
  
  A better candidate for changing something is probably 
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/677299. @Pchelolo is looking 
into it.

TASK DETAIL
  https://phabricator.wikimedia.org/T281480

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: Pchelolo, Joe, jcrespo, Urbanecm, Aklapper, LarsWirzenius, Invadibot, 
Devnull, LSobanski, maantietaja, Akuckartz, Iflorez, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
Marostegui, LawExplorer, Minhnv-2809, _jensen, rosalieper, Agabi10, Scott_WUaS, 
Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Addshore, Mbch331, Jay8g, 
Krenair, AMooney
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T281480: Cannot access the database: Too many connections

[Joe]

Joe added a comment.


  Given we only make requests to external storage when parsercache has a miss, 
it seemed sensible to look for corresponding patterns in parsercache.
  
  I see we introduced a new category of misses on the same date 
"miss_absent_metadata", see 
https://grafana-rw.wikimedia.org/d/00106/parser-cache?viewPanel=7&orgId=1&from=now-30d&to=now
 which seems related.

TASK DETAIL
  https://phabricator.wikimedia.org/T281480

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: Joe, jcrespo, Urbanecm, Aklapper, LarsWirzenius, Invadibot, Devnull, 
LSobanski, maantietaja, Akuckartz, Iflorez, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
Marostegui, LawExplorer, Minhnv-2809, _jensen, rosalieper, Agabi10, Scott_WUaS, 
Pchelolo, Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Addshore, Mbch331, 
Jay8g, Krenair, AMooney
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T281480: Cannot access the database: Too many connections

[LarsWirzenius]

LarsWirzenius added a parent task: T278347: 1.37.0-wmf.3 deployment blockers.

TASK DETAIL
  https://phabricator.wikimedia.org/T281480

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: LarsWirzenius
Cc: jcrespo, Urbanecm, Aklapper, LarsWirzenius, Invadibot, Devnull, LSobanski, 
maantietaja, Akuckartz, Iflorez, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, 
LawExplorer, Minhnv-2809, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Addshore, Mbch331, Jay8g, 
Krenair, AMooney
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T281480: Cannot access the database: Too many connections

[LarsWirzenius]

LarsWirzenius added a comment.


  ACK, I'll make it a train blocker.

TASK DETAIL
  https://phabricator.wikimedia.org/T281480

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: LarsWirzenius
Cc: jcrespo, Urbanecm, Aklapper, LarsWirzenius, Invadibot, Devnull, LSobanski, 
maantietaja, Akuckartz, Iflorez, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, 
LawExplorer, Minhnv-2809, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Addshore, Mbch331, Jay8g, 
Krenair, AMooney
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T281480: Cannot access the database: Too many connections

[jcrespo]

jcrespo triaged this task as "Unbreak Now!" priority.
jcrespo added a comment.


  This should be a blocker- es traffic has grown almost grown 100x since 14 
april, correlates strongly with the 19h deploy:
  
  F34434387: es_issue.png 

TASK DETAIL
  https://phabricator.wikimedia.org/T281480

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: jcrespo
Cc: jcrespo, Urbanecm, Aklapper, LarsWirzenius, Invadibot, Devnull, LSobanski, 
maantietaja, Akuckartz, Iflorez, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, 
LawExplorer, Minhnv-2809, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Addshore, Mbch331, Jay8g, 
Krenair, RhinosF1, AMooney
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T281480: Cannot access the database: Too many connections

[Urbanecm]

Urbanecm added a project: DBA.
Urbanecm added a comment.


  Tagging #dba , as they might be 
able to offer some guidance on finding the issue here.

TASK DETAIL
  https://phabricator.wikimedia.org/T281480

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Urbanecm
Cc: Urbanecm, Aklapper, LarsWirzenius, Invadibot, Devnull, LSobanski, 
maantietaja, Akuckartz, Iflorez, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, Marostegui, 
LawExplorer, Minhnv-2809, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Addshore, Mbch331, Jay8g, 
Krenair, valerio.bozzolan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T281480: Cannot access the database: Too many connections

[LarsWirzenius]

LarsWirzenius removed a parent task: T278347: 1.37.0-wmf.3 deployment blockers.

TASK DETAIL
  https://phabricator.wikimedia.org/T281480

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: LarsWirzenius
Cc: Aklapper, LarsWirzenius, Invadibot, maantietaja, Akuckartz, darthmon_wmde, 
WDoranWMF, holger.knust, EvanProdromou, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Addshore, Mbch331, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T281480: Cannot access the database: Too many connections

[LarsWirzenius]

LarsWirzenius created this task.
LarsWirzenius added projects: Wikimedia-production-error, Wikidata, 
MediaWiki-Revision-backend, Platform Engineering.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: wdwb-tech.

TASK DESCRIPTION
  Error
  -
  
  - mwversion: `1.37.0-wmf.1`
  - reqId: `93b8ccbd-f566-4569-94f4-7aa3266003cb`
  
  name=normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBConnectionError: Cannot 
access the database: Too many connections (10.64.32.102) (10.64.32.102)
  
  name=exception.trace
from 
/srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1510)
#0 
/srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/loadbalancer/LoadBalancer.php(997):
 Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 
/srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/loadbalancer/LoadBalancer.php(962):
 Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#2 
/srv/mediawiki/php-1.37.0-wmf.1/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1101):
 Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#3 
/srv/mediawiki/php-1.37.0-wmf.1/includes/externalstore/ExternalStoreDB.php(168):
 Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, string, integer)
#4 
/srv/mediawiki/php-1.37.0-wmf.1/includes/externalstore/ExternalStoreDB.php(312):
 ExternalStoreDB->getReplica(string)
#5 
/srv/mediawiki/php-1.37.0-wmf.1/includes/externalstore/ExternalStoreDB.php(66): 
ExternalStoreDB->fetchBlob(string, string, boolean)
#6 
/srv/mediawiki/php-1.37.0-wmf.1/includes/externalstore/ExternalStoreAccess.php(52):
 ExternalStoreDB->fetchFromURL(string)
#7 /srv/mediawiki/php-1.37.0-wmf.1/includes/Storage/SqlBlobStore.php(509): 
ExternalStoreAccess->fetchFromURL(string, array)
#8 
/srv/mediawiki/php-1.37.0-wmf.1/includes/libs/objectcache/wancache/WANObjectCache.php(1714):
 MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, 
array, NULL, array)
#9 
/srv/mediawiki/php-1.37.0-wmf.1/includes/libs/objectcache/wancache/WANObjectCache.php(1542):
 WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#10 /srv/mediawiki/php-1.37.0-wmf.1/includes/Storage/SqlBlobStore.php(513): 
WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#11 /srv/mediawiki/php-1.37.0-wmf.1/includes/Storage/SqlBlobStore.php(430): 
MediaWiki\Storage\SqlBlobStore->expandBlob(string, array, string)
#12 /srv/mediawiki/php-1.37.0-wmf.1/includes/Storage/SqlBlobStore.php(286): 
MediaWiki\Storage\SqlBlobStore->fetchBlobs(array, integer)
#13 
/srv/mediawiki/php-1.37.0-wmf.1/includes/libs/objectcache/wancache/WANObjectCache.php(1714):
 MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, 
array, NULL, array)
#14 
/srv/mediawiki/php-1.37.0-wmf.1/includes/libs/objectcache/wancache/WANObjectCache.php(1542):
 WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#15 /srv/mediawiki/php-1.37.0-wmf.1/includes/Storage/SqlBlobStore.php(291): 
WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#16 
/srv/mediawiki/php-1.37.0-wmf.1/includes/Revision/RevisionStore.php(1191): 
MediaWiki\Storage\SqlBlobStore->getBlob(string, integer)
#17 
/srv/mediawiki/php-1.37.0-wmf.1/includes/Revision/RevisionStore.php(1463): 
MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord,
 NULL, NULL, NULL, integer)
#18 [internal function]: 
MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#19 /srv/mediawiki/php-1.37.0-wmf.1/includes/Revision/SlotRecord.php(324): 
call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#20 
/srv/mediawiki/php-1.37.0-wmf.1/includes/Revision/RevisionRecord.php(164): 
MediaWiki\Revision\SlotRecord->getContent()
#21 /srv/mediawiki/php-1.37.0-wmf.1/includes/parser/Parser.php(3697): 
MediaWiki\Revision\RevisionRecord->getContent(string)
#22 /srv/mediawiki/php-1.37.0-wmf.1/includes/parser/Parser.php(3547): 
Parser->statelessFetchTemplate(Title, Parser)
#23 /srv/mediawiki/php-1.37.0-wmf.1/includes/parser/Parser.php(3415): 
Parser->fetchTemplateAndTitle(Title)
#24 /srv/mediawiki/php-1.37.0-wmf.1/includes/parser/Parser.php(3157): 
Parser->getTemplateDom(Title)
#25 /srv/mediawiki/php-1.37.0-wmf.1/includes/parser/PPFrame_Hash.php(263): 
Parser->braceSubstitution(array, PPFrame_Hash)
#26 /srv/mediawiki/php-1.37.0-wmf.1/includes/parser/Parser.php(2879): 
PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#27 /srv/mediawiki/php-1.37.0-wmf.1/includes/parser/Parser.php(1549): 
Parser->replaceVariables(string)
#28 /srv/mediawiki/php-1.37.0-wmf.1/includes/parser/Parser.php(639): 
Parser->internalParse(string)
#29 
/srv/mediawiki/php-1.37.0-wmf.1/includes/content/WikitextContent.php(375): 
Parser->parse(string, Title, ParserOptions,