Ottomata has uploaded a new change for review. https://gerrit.wikimedia.org/r/163669
Change subject: Reset queue_buffering_max_ms to default 1000 ms for varnishkafka ...................................................................... Reset queue_buffering_max_ms to default 1000 ms for varnishkafka We increased this in attempt to avoid produce errors during kafka <-> zookeeper timeouts. However, this just caused the number of buffered messages in varnishkafka to increase, but did not actually help wtih produce errors. As far as I can tell currently, varnishkafka and librdkafka are not responsible for lost messages during zookeeper timeouts. http://cl.ly/image/3N3i2L04161r shows the msgq_cnt buffer increase on esams bits since I started tweaking this setting. You can read more about the bug we are trying to figure out here: https://bugzilla.wikimedia.org/show_bug.cgi?id=69667 Change-Id: If6a9d5e58c2917f2e37666ff6f70c44f2526ec15 --- M manifests/role/cache.pp 1 file changed, 1 insertion(+), 6 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/69/163669/1 diff --git a/manifests/role/cache.pp b/manifests/role/cache.pp index 526b16a..b02a0fb 100644 --- a/manifests/role/cache.pp +++ b/manifests/role/cache.pp @@ -521,12 +521,7 @@ # At ~6000 msgs per second, 500000 messages is over 1 minute # of buffering, which should be more than enough. queue_buffering_max_messages => 500000, - # Broker <-> Zookeeper timeout is currently set at - # 16 seconds, and it can take a few seconds for leadership - # change metadata to propagate to down to varnishkafka. - # Setting this to 30 seconds. - queue_buffering_max_ms => 30000, - # bits varnishes do about 6000 reqs / sec each. + # bits varnishes can do about 6000 reqs / sec each. # We want to send batches at least once a second. batch_num_messages => 6000, # large timeout to account for potential cross DC latencies -- To view, visit https://gerrit.wikimedia.org/r/163669 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If6a9d5e58c2917f2e37666ff6f70c44f2526ec15 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Ottomata <o...@wikimedia.org> _______________________________________________ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
