I work on a system that handles 10s of billions of events per day, and we do a lot of queueing. Big +1 on having bounded queues. Unbounded in-memory queues aren't, they just have a bound you have no direct control over and that blows up the world when its hit.
The only reason to have a queue size greater than 1 is to handle spikes in the producer, short outages in the consumer, or a bit of out-of-phaseness between producers and consumers. If your consumers habitually can't keep up with your producers, you're screwed regardless, but unbounded queues make your problem grow from simply being 'consumer can't keep up' to 'consumer can't keep up, I'm OOMing, and I just lost a boatload of messages'. If the only option that's available are unbounded queues, there are going to be a lot of explicit semaphores sitting around in production Rust code. I would also hope that Rust doesn't decide to silently drop messages on the floor. It's very true that systems that can tolerate dropped messages are more robust, but not all messages are created equal, and not all messages can just be dropped (i.e., if they come from a 3rd party). I need to have control over what happens to a message that I can't send, whether that means slow-pathing it to something on-disk, passing an error back to my source, or just deciding to drop it. The twitter conversation has discussion of blocking sends leading to deadlocks: yes, blocking indefinitely is also bad, so don't do that. If it's blocking it should have a timeout, or as others have mentioned you can keep the sends non-blocking and just have them fail if the queue is full. On Wed, Dec 18, 2013 at 9:29 PM, Tony Arcieri <basc...@gmail.com> wrote: > Some context: > > https://twitter.com/mentalguy/status/284776872452173824 > > As someone who knows a lot of people who use Erlang in production (Erlang > has unbounded mailboxes), and the maintainer of my own actor-based > concurrency framework (Celluloid, which started with unbounded mailboxes > and is in the process of moving to bounded ones), I have come to the Hard > Won Knowledge(TM) that unbounded queues/mailboxes/channels are a bad idea. > > In terms of production users of these sorts of systems, my personal > experience is that nobody likes unboundedness and most have experienced > some sort of production outage because of it. This isn't just a minor > nitpick. This is the sort of decision that makes or breaks the reliability > of systems under load. > > The main problem with unboundedness is that users systems based on > unbounded queues fail to adequately build mechanisms for providing > backpressure into their code. They then start flooding their systems with > messages, and get confused why they're performing so poorly when the answer > is they have a huge backlog of unprocessed messages. Processes overloaded > with too many messages will slow down and grow in memory until they > eventually exhaust system resources and crash. > > Adding bounds to a channel doesn't require that sends block, and I think > Rust is doing the Right Thing(TM) here in regard to non-blocking sends and > I would never ask you to change that. There are other options for bounding > channels which don't involve a blocking send though: > > 1) Drop messages on the floor: This falls into the category of "at most > once" message semantics that actor systems are typically described as > having (although there's a fun discussion about this right now on the friam > mailing list). My personal opinion is that systems that can tolerate the > loss of messages are more robust by design > > 2) Crash the sender: This works similarly to the above in that it results > in the messages being discarded, but can loop in Erlang-style fault > tolerance to recover from an overloaded system. I definitely find this less > preferable than simply dropping messages on the floor though. This is a > particularly invasive option that I think doesn't translate well to > distributed systems. > > 3) Make sends to a full channel an error: I'm really not a fan of this > option at all but I'm listing it for completeness. We could make everyone > check every message send for success. Ick. No thanks. > > Bounded channels offer a lot of advantages over unbounded ones, IMO. > Fixed-size data structures confer a natural performance advantage over > elastic ones that need to be resized to accomodate growing numbers of > messages. Depending on what data structure you use, you either take an > up-front performance hit to provide variable capacity, or take an invisible > performance hit whenever you hit some cap and need to resize. Adding a > bound makes it easier to reason about the use cases and requirements and > will generally allow you to leverage better data structures (e.g. ring > buffers) that will confer maximum performance. > > -- > Tony Arcieri > > _______________________________________________ > Rust-dev mailing list > Rust-dev@mozilla.org > https://mail.mozilla.org/listinfo/rust-dev > >
_______________________________________________ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
