On 13 February 2016 at 17:24, Sam Ruby <[email protected]> wrote: > I'm thinking that the value of generating infrastructure alerts for > individual transient LDAP LDAP failures has run its course. > > With that in mind, and considering the following commit; perhaps it > would be worth revisiting the question as to what monitoring levels > should do what. > > For example, would it make sense to define some levels as simply > sending emails to the dev@whimsical, and only generate infrastructure > alerts for higher levels?
Yes, that would be fine by me. > - Sam Ruby > > > On Fri, Feb 12, 2016 at 1:00 PM, Sebastian Bazley <[email protected]> wrote: >> Commit 1998570a194ea0b0ef7b08ff8819b7c72e5cfd5a: >> Log warning if cronjob was missed recently >> >> >> Branch: refs/heads/master >> Author: Sebb <[email protected]> >> Committer: Sebb <[email protected]> >> Pusher: sebb <[email protected]> >> >> ------------------------------------------------------------ >> www/status/monitors/public_json.rb | +++++++++++ >> --- >> ------------------------------------------------------------ >> 14 changes: 11 additions, 3 deletions. >> ------------------------------------------------------------ >> >> >> diff --git a/www/status/monitors/public_json.rb >> b/www/status/monitors/public_json.rb >> index 42e8738..b8b5936 100644 >> --- a/www/status/monitors/public_json.rb >> +++ b/www/status/monitors/public_json.rb >> @@ -3,7 +3,9 @@ >> # >> >> def Monitor.public_json(previous_status) >> - grace_period = 86_400 # one day >> + danger_period = 86_400 # one day >> + >> + warning_period = 5400 # 1.5 hours >> >> logs = File.expand_path('../../www/logs/public-*') >> >> @@ -40,9 +42,15 @@ def Monitor.public_json(previous_status) >> end >> >> # Check to see if the log has been updated recently >> - if Time.now - File.mtime(log) > grace_period >> + if Time.now - File.mtime(log) > warning_period >> + status[name].merge! level: 'warning', >> + data: "Last updated: #{File.mtime(log).to_s} (more than 1.5 hours >> old)" >> + end >> + >> + # Check to see if the log has been updated recently >> + if Time.now - File.mtime(log) > danger_period >> status[name].merge! level: 'danger', >> - data: "Last updated: #{File.mtime(log).to_s}" >> + data: "Last updated: #{File.mtime(log).to_s} (more than 24 hours >> old)" >> end >> >> # Treat everything left as an error to be reported
