Nuria has submitted this change and it was merged.
Change subject: Block mysql consumer if the queue is too big
......................................................................
Block mysql consumer if the queue is too big
Adds a maximum size for the mysql insertion queue.
If this limit is reached, the consumer should wait
until the insertion worker empties the queue.
Bug: T115667
Change-Id: I99d6e74168efc71793ed466fda32cef75887e1a8
---
M server/eventlogging/handlers.py
1 file changed, 8 insertions(+), 1 deletion(-)
Approvals:
Nuria: Verified; Looks good to me, approved
diff --git a/server/eventlogging/handlers.py b/server/eventlogging/handlers.py
index e667cd7..d841ecf 100644
--- a/server/eventlogging/handlers.py
+++ b/server/eventlogging/handlers.py
@@ -270,11 +270,18 @@
# that the connection is alive, and reconnect if necessary.
dbapi_connection.ping(True)
try:
- batch_size = 400
+ batch_size = 5000
batch_time = 300 # in seconds
+ # Max number of batches pending insertion.
+ queue_size = 1000
+ sleep_seconds = 5
# Link the main thread to the worker thread so we
# don't keep filling the queue if the worker died.
while worker.is_alive():
+ # If the queue is too big, wait for the worker to empty it.
+ while len(events_batch) > queue_size:
+ logger.info('Sleeping %d seconds', sleep_seconds)
+ time.sleep(sleep_seconds)
event = (yield)
# Break the event stream by schema (and revision)
scid = (event['schema'], event['revision'])
--
To view, visit https://gerrit.wikimedia.org/r/246796
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I99d6e74168efc71793ed466fda32cef75887e1a8
Gerrit-PatchSet: 5
Gerrit-Project: mediawiki/extensions/EventLogging
Gerrit-Branch: master
Gerrit-Owner: Mforns <mfo...@wikimedia.org>
Gerrit-Reviewer: Madhuvishy <mviswanat...@wikimedia.org>
Gerrit-Reviewer: Mforns <mfo...@wikimedia.org>
Gerrit-Reviewer: Milimetric <dandree...@wikimedia.org>
Gerrit-Reviewer: Nuria <nu...@wikimedia.org>
Gerrit-Reviewer: Ori.livneh <o...@wikimedia.org>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
