So I’ve just had a conversation with dgoulet on IRC, which I will reformat and subedit here as a conversation regarding OnionBalance and issues in 2.6 and 2.7 when a recently rebooted HS publishes a fresh descriptor:
[…] alecm: consider OnionBalance which - being a bunch of daemons on a bunch of servers - will be a lot more prone to intermittent failures of 1+ daemons yielding a lot of republishing alecm: we tend to move services around, and daemons will be killed in one place and resurrected elsewhere, and then we'll have to bundle up a new descriptor and ship it out dgoulet: hrm so with that new 027 cache behavior, as long as the IP are usable, the descriptor will be kept, if they all become unusable, a new descriptor fetch is triggered and then those IPs will be tried alecm: There's a mandatory refresh [of the descriptor] after N minutes? dgoulet: we'll retry 3 times and after that all HSDir are in timeout for 15 minutes (I think, I'll have to validate) before retrying any HSDirs alecm: I wonder if descriptors should publish a recommended TTL - [number of seconds to live before refresh] dgoulet: yeah we have an idea for a "revision-counter" in the descriptor being incremented at each new version for the 24 hours period dgoulet: a TTL could be useful for load balancing though! alecm: so, here's a scenario: imagine that we run 10 daemons, alecm: call these daemons: A B C D E F G H I J - they all have random onion addresses alecm: we steal one IP from each daemon, and bundle the 10 stolen IPs together to make an onionbalance site descriptor and publish it alecm: people pull that descriptor, it's quite popular alecm: we then lose power in a datacentre, which takes out half of our onions - say, A through E alecm: we reboot the datacentre and restart A-E merely 10 minutes later alecm: everyone who has already loaded our onionbalance site descriptor tests A B C D E and finds them all dead, because the old IPs for A-E are invalid alecm: so they all move to F G H I J - which get overloaded even though (new) A B C D E are back up alecm: and this persists for up to 244, even though the outage was only 10 minutes alecm: net result: large chunks of the world (anyone with an old descriptor + anyone randomly choosing F-J) have a shitty experience, which is not what high-availability is all about :-) dgoulet: that will be what's going to happen - having a TTL in the desc. would help here indeed, I see the issue dgoulet: TTL would be one thing to add, here we could also add a mechanism for a client retrying IPs that failed in the situation where some of the IPs are still working, or making client balance themself randomly could be also an idea dgoulet: definitely there is some content here for tor-dev - I don't have a good answer but it should definitely be addressed alecm: proper random selection of IP would be beneficial for load-balancing; not perfect, but in the long run, helpful — Alec Muffett Security Infrastructure Facebook Engineering London
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
