KevinYang21 commented on issue #7269: [AIRFLOW-6651] Add Redis Heartbeat option URL: https://github.com/apache/airflow/pull/7269#issuecomment-590627923 @ashb Good question. I don't have an exact performance A/B test result number for this change( we're still building that framework). But to give a bit more context on the motivation behind this change: for all the past DB incident we had, despite different variaty of the root causes( e.g. high CPU/ram usage caused by bad query or spiky query load), connections from heartbeats always worsen the issue and ended up bringing down the DB with connection flood. When reviewing the incidents, many of the root causes turned out to be not severe enough to bring down the DB by themselves, but caused heartbeats to be slow and heartbeats brought down the DB. We already have done a lot query optimizations, which is awesome. We believe with this change would give us a lot more headroom operating the metadata DB. And I would argue that the added complexity is quite affordable for the value.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
