[tor-dev] monitoring significant drops of flags in dirauth votes
Hi Damian and Tom, Roger discovered that dannenberg did not include any exit flags in certain votes anymore [1]. It would be great if we would detect and notify about such events in the future. I see two places where this could be added: DocTor: a new check that alerts on events where a certain dir auth does either no longer include certain flags (guard, hsdir, exit, ..) at all or better: if the amount of relays with a certain flag significantly dropped by xx % from one vote to the next. consenus-health graphs: we have nice graphs per dirauth and bwauth, if we would have per-dirauthvote-per-flag (mainly guard, exit, hsdir - we have already running) graphs as well we could spot such events (and even trends) better. (btw: what caused there recent flat-line in graphs on 2018-02-03 - 2018-02-05) What do you think? thanks for considering it, nusenu [1] https://lists.torproject.org/pipermail/tor-relays/2018-February/014480.html -- https://mastodon.social/@nusenu twitter: @nusenu_ signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] monitoring significant drops of flags in dirauth votes
Thanks nusenu! Nice idea, added it to DocTor... https://gitweb.torproject.org/doctor.git/commit/?id=8945013 It gives a notice if flags issued by an authority are 50% different from the conensus. Presently there's only one instance of that... [consensus-health] NOTICE: moria1 had 756 HSDir flags in its vote but the consensus had 2583 On Sun, Feb 11, 2018 at 1:21 AM, nusenu wrote: > Hi Damian and Tom, > > Roger discovered that dannenberg did not include any exit flags > in certain votes anymore [1]. > > It would be great if we would detect and notify about such events in the > future. > > I see two places where this could be added: > > DocTor: > a new check that alerts on events where a certain dir auth does > either no longer include certain flags (guard, hsdir, exit, ..) at all > or better: if the amount of relays with a certain flag significantly dropped > by xx % from one vote to the next. > > consenus-health graphs: > we have nice graphs per dirauth and bwauth, if we would have > per-dirauthvote-per-flag (mainly guard, exit, hsdir - we have already running) > graphs as well we could spot such events (and even trends) > better. (btw: what caused there recent flat-line in graphs on 2018-02-03 - > 2018-02-05) > > What do you think? > > > thanks for considering it, > nusenu > > > [1] > https://lists.torproject.org/pipermail/tor-relays/2018-February/014480.html > > > -- > https://mastodon.social/@nusenu > twitter: @nusenu_ > ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] monitoring significant drops of flags in dirauth votes
> Thanks nusenu! Nice idea, added it to DocTor... thanks for implementing the new check so fast. > https://gitweb.torproject.org/doctor.git/commit/?id=8945013 > > It gives a notice if flags issued by an authority are 50% different > from the conensus. Presently there's only one instance of that... This is also very useful but slightly different from what I had in mind, because it would not trigger if dirauths upgrade from A to B in the same hour and most exits, guards or hsdirs are gone due to a bug in version B. > NOTICE: moria1 had 756 HSDir flags in its vote but > the consensus had 2583 I tried to find something related to this in the 0.3.3.x changelogs because moria1 (the affected dirauth) is the only one running tor alpha but I didn't find anything related to a change in what is required to earn the HSDir flag. Has there been any change related to how HSDir is assigned that would explain that significant difference? -- https://mastodon.social/@nusenu twitter: @nusenu_ signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] monitoring significant drops of flags in dirauth votes
I think the doctor notification is the best mechanism. I'm not opposed to adding more graphs to consensus-health, but I think I'd want to coordinate with the metrics team. There was talk about them absorbing consensus health in some capacity, so I'd prefer to avoid doing a lot of work on graphs if it's going to be redone or throw away. The host running depictor was down for several days, which explains the gap in data. Thanks for the thoughts! -tom ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] monitoring significant drops of flags in dirauth votes
> thanks for implementing the new check so fast. No problem! Thanks for suggesting it. > This is also very useful but slightly different from what I had in mind, > because it would not trigger if dirauths upgrade from A to B in the > same hour and most exits, guards or hsdirs are gone due to a bug in version B. This should catch a bug with B unless every authority upgrades to B in the same hour. Otherwise we'd get an alert - either because the majority is B and the remaining A votes are out of band, or the consensus is made with A and authorities that upgraded to B are different. Is there another check in particular that you'd like? One gotcha is that checks that require state (such as comparing with the last hour's consensus) is a bit more work. > I tried to find something related to this in the 0.3.3.x changelogs > because moria1 (the affected dirauth) is the only one running tor alpha > but I didn't find anything related to a change in what is required > to earn the HSDir flag. Has there been any change related to how > HSDir is assigned that would explain that significant difference? For what it's worth I started with alarming when authorities differed more than 20% from the consensus but it was a bit noisier... [consensus-health] NOTICE: longclaw had 3100 HSDir flags in its vote but the consensus had 2583 [consensus-health] NOTICE: moria1 had 756 HSDir flags in its vote but the consensus had 2583 [consensus-health] NOTICE: moria1 had 1397 Guard flags in its vote but the consensus had 1761 ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] monitoring significant drops of flags in dirauth votes
On 11 Feb (21:21:00), nusenu wrote: > > Thanks nusenu! Nice idea, added it to DocTor... > > thanks for implementing the new check so fast. > > > https://gitweb.torproject.org/doctor.git/commit/?id=8945013 > > > > It gives a notice if flags issued by an authority are 50% different > > from the conensus. Presently there's only one instance of that... > > This is also very useful but slightly different from what I had in mind, > because it would not trigger if dirauths upgrade from A to B in the > same hour and most exits, guards or hsdirs are gone due to a bug in version B. > > > > NOTICE: moria1 had 756 HSDir flags in its vote but > > the consensus had 2583 > > I tried to find something related to this in the 0.3.3.x changelogs > because moria1 (the affected dirauth) is the only one running tor alpha > but I didn't find anything related to a change in what is required > to earn the HSDir flag. Has there been any change related to how > HSDir is assigned that would explain that significant difference? This is because moria1 is running an experimental patch that drastically cut down the number of voted HSDir. It has been doing that for a while now. I can't recall the ticket but this was an attempt a while back to see how much it would be bad to vote down for the most stable relays on the network to be HSDir. Roger can probably explain it better but all in all nothing to worried about there as it is expected. Actually, from time to time, you should expect moria1 to behave strangely sometimes since it is often running alpha code :). Cheers! David -- And8vxUcJVOn9srRjJ3mpKMUC5pScfYMRq9Qv9yt54Y= signature.asc Description: PGP signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] monitoring significant drops of flags in dirauth votes
Damian Johnson: >> thanks for implementing the new check so fast. > > No problem! Thanks for suggesting it. > >> This is also very useful but slightly different from what I had in mind, >> because it would not trigger if dirauths upgrade from A to B in the >> same hour and most exits, guards or hsdirs are gone due to a bug in version >> B. > > This should catch a bug with B unless every authority upgrades to B in > the same hour. Otherwise we'd get an alert - either because the > majority is B and the remaining A votes are out of band, or the > consensus is made with A and authorities that upgraded to B are > different. > > Is there another check in particular that you'd like? Yes, but not directly related to this thread. I will file it via trac.tpo. > One gotcha is > that checks that require state (such as comparing with the last hour's > consensus) is a bit more work. Yes, that is what I was wondering if DocTor keeps any state at all already. >> I tried to find something related to this in the 0.3.3.x changelogs >> because moria1 (the affected dirauth) is the only one running tor alpha >> but I didn't find anything related to a change in what is required >> to earn the HSDir flag. Has there been any change related to how >> HSDir is assigned that would explain that significant difference? > > For what it's worth I started with alarming when authorities differed > more than 20% from the consensus but it was a bit noisier... > > [consensus-health] NOTICE: longclaw had 3100 HSDir flags in its vote > but the consensus had 2583 > [consensus-health] NOTICE: moria1 had 756 HSDir flags in its vote but > the consensus had 2583 > [consensus-health] NOTICE: moria1 had 1397 Guard flags in its vote but > the consensus had 1761 I assume this has not been deployed - 50% or maybe 40% are fine I guess. To come up with good threshold values one would need to look at historic data for the past few months. -- https://mastodon.social/@nusenu twitter: @nusenu_ signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] monitoring significant drops of flags in dirauth votes
> I assume this has not been deployed - 50% or maybe 40% are fine I guess. > To come up with good threshold values > one would need to look at historic data for the past few months. Nope, it is deployed (if by 'deployed' you mean DocTor is presently performing this check). From David's reply about moria1 it sounds like any check of this sort may be a red herring since they experiment with moria1, but I'll leave that up to you guys. Just let me know what kind of check you want. ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] monitoring significant drops of flags in dirauth votes
On Mon, Feb 12, 2018 at 08:31:13AM -0800, Damian Johnson wrote: > Nope, it is deployed (if by 'deployed' you mean DocTor is presently > performing this check). From David's reply about moria1 it sounds like > any check of this sort may be a red herring since they experiment with > moria1, but I'll leave that up to you guys. Just let me know what kind > of check you want. It might be smartest to just put in an exception for moria1's HSDir votes, since we know it's being different. --Roger ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] monitoring significant drops of flags in dirauth votes
Roger Dingledine: > It might be smartest to just put in an exception for moria1's > HSDir votes, since we know it's being different. yes, please :) and it would also be nice to have: https://trac.torproject.org/projects/tor/ticket/25222 so we can filter for those emails that we care about most (or filter those that we do not care about) -- https://mastodon.social/@nusenu twitter: @nusenu_ signature.asc Description: OpenPGP digital signature ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] monitoring significant drops of flags in dirauth votes
> It might be smartest to just put in an exception for moria1's > HSDir votes, since we know it's being different. Suppressed any notices for HSDir flags. Also fixed the time based suppression for the check (it should have sent one notice a day rather than one an hour). ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
