xiaohui-sun commented on a change in pull request #5208: [TE] Disable alerts if
it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r407146946
##########
File path:
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
##########
@@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo
monitorTaskInfo) {
// update detection health
updateDetectionHealth();
+
+ // disable alerts that failed consecutively for a long time
+ disableLongFailedAlerts();
+
} catch (Exception e) {
LOG.error("Exception in monitor update task", e);
}
}
+ /**
+ * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but
there is no success run since then.
+ */
+ private void disableLongFailedAlerts() {
Review comment:
Currently we don't have a good way to test it. The only way would be
integration test. But having more than one integration tests running may cause
conflicts. That's why I have to add sequence number in our local integration
test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
