I am not making any assumptions other than Rabbit needs to maintain the state of the consumers. As the Kafka docs point out this is the fundamental difference between most providers in the space and Kafka.
Thinking of a high throughput stream of messages and many active consumers of different speeds, I am struggling with how Rabbit can avoid random I/O with all the acks. Each consumer’s state is certainly not linearly stored on disk so there would have to be seeks. Further log-structured merge trees are used in NoSQL stores like Cassandra and are optimized for random read access. Why do you feel ‘Rabbit does not do lots of random IO?’ Looking at some docs on the Rabbit site they seem to mention that performance degrades as the size of the persistent message store increases. Too much random I/O could certainly explain this degradation. http://www.rabbitmq.com/blog/2011/09/24/sizing-your-rabbits/ The use case I have been talking about all along is a continuous firehose of data with throughput in the 100s of thousands messages per second. You will have 10-20 consumers of different speeds ranging from real-time (Storm) to batch (Hadoop). This means the message store is in the 100s GBs to terabytes range at all times. -Jonathan On Sat, Jun 8, 2013 at 2:09 PM, Alexis Richardson < alexis.richard...@gmail.com> wrote: > Jonathan > > I am aware of the difference between sequential writes and other kinds > of writes ;p) > > AFAIK the Kafka docs describe a sort of platonic alternative system, > eg "normally people do this.. Kafka does that..". This is a good way > to explain design decisions. However, I think you may be assuming > that Rabbit is a lot like the generalised other system. But it is not > - eg Rabbit does not do lots of random IO. I'm led to understand that > Rabbit's msg store is closer to log structured storage (a la > Log-Structured Merge Trees) in some ways. However, Rabbit does do > more synchronous I/O, and has a different caching strategy (AFAIK). > "It's complicated" > > In order to help provide useful info to the community, please could > you describe a concrete test that we could discuss? I think that > would really help. You mentioned a scenario with one large data set > being streamed into the broker(s), and then consumed (in full?) by 2+ > consumers of wildly varying speeds. Could you elaborate please? > > alexis > > > Also, this is probably OT but I have never grokked this in the Design Doc: > > "Consumer rebalancing is triggered on each addition or removal of both > broker nodes and other consumers within the same group. For a given > topic and a given consumer group, broker partitions are divided evenly > among consumers within the group." > > When a new consumer and/or partition appears, can messages in the > broker get "moved" from one partition to another? > > > On Sat, Jun 8, 2013 at 12:53 PM, Jonathan Hodges <hodg...@gmail.com> > wrote: > > On Sat, Jun 8, 2013 at 2:09 AM, Jonathan Hodges <hodg...@gmail.com> > wrote: > >> Thanks so much for your replies. This has been a great help > understanding > >> Rabbit better with having very little experience with it. I have a few > >> follow up comments below. > > > > Happy to help! > > > > I'm afraid I don't follow your arguments below. Rabbit contains many > > optimisations too. I'm told that it is possible to saturate the disk > > i/o, and you saw the message rates I quoted in the previous email. > > YES of course there are differences, mostly an accumulation of things. > > For example Rabbit spends more time doing work before it writes to > > disk. > > > > It would be great if you can you detail some of the optimizations? It > > would seem to me Rabbit has much more overhead due to maintaining state > of > > the consumers as well as general messaging processing which makes it > > impossible to manage the same write throughput as Kafka when you need to > > persist large amounts of data to disk. I definitely believe you that > > Rabbit can saturate the disk but it is much more seek centric i.e. random > > access read/writes vs sequential read/writes. Kafka saturates the disk > > too, but since it leverages sequential disk I/O is orders of magnitude > more > > efficient persisting to disk than random access. > > > > > > You said: > > > > "Since Rabbit must maintain the state of the > > consumers I imagine it’s subjected to random data access patterns on disk > > as opposed to sequential." > > > > I don't follow the logic here, sorry. > > > > Couple of side comments: > > > > * In your Hadoop vs RT example, Rabbit would deliver the RT messages > > immediately and write the rest to disk. It can do this at high rates > > - I shall try to get you some useful data here. > > > > * Bear in mind that write speed should be orthogonal to read speed. > > Ask yourself - how would Kafka provide a read cache, and when might > > that be useful? > > > > * I'll find out what data structure Rabbit uses for long term > persistence. > > > > What I am saying here is when Rabbit needs to retrieve and persist each > > consumer’s state from its internal DB this information isn’t linearly > > persisted on disk so it requires disk seeks which is in much less > > inefficient than sequential access. You do get the difference here, > > correct? Sequential reads from disk are nearly 1.5x faster than random > > reads from memory and 4-5 orders of magnitude faster than random reads > from > > disk (http://queue.acm.org/detail.cfm?id=1563874). > > > > As was detailed at length in my previous post Kafka uses the OS > > pagecache/sendfile which is much more efficient than memory or > applications > > cache. > > > > That would be awesome if you can confirm what Rabbit is using as a > > persistent data structure. More importantly, whether it is BTree or > > something else, is the disk i/o random or linear? > > > > > > "Quoting the Kafka design page ( > > http://kafka.apache.org/07/design.html) performance of sequential > writes on > > a 6 7200rpm SATA RAID-5 array is about 300MB/sec but the performance of > > random writes is only about 50k/sec—a difference of nearly 10000X." > > > > Depending on your use case, I'd expect 2x-10x overall throughput > > differences, and will try to find out more info. As I said, Rabbit > > can saturate disk i/o. > > > > This is only speaking of the use case of high throughput with persisting > > large amounts of data to disk where there is 4 orders of magnitude more > > than 10x difference. It all comes down to random vs sequential > > writes/reads to disk as I mentioned above. > > > > > > On Sat, Jun 8, 2013 at 2:07 AM, Alexis Richardson < > > alexis.richard...@gmail.com> wrote: > > > >> Jonathan > >> > >> On Sat, Jun 8, 2013 at 2:09 AM, Jonathan Hodges <hodg...@gmail.com> > wrote: > >> > Thanks so much for your replies. This has been a great help > >> understanding > >> > Rabbit better with having very little experience with it. I have a > few > >> > follow up comments below. > >> > >> Happy to help! > >> > >> I'm afraid I don't follow your arguments below. Rabbit contains many > >> optimisations too. I'm told that it is possible to saturate the disk > >> i/o, and you saw the message rates I quoted in the previous email. > >> YES of course there are differences, mostly an accumulation of things. > >> For example Rabbit spends more time doing work before it writes to > >> disk. > >> > >> You said: > >> > >> "Since Rabbit must maintain the state of the > >> consumers I imagine it’s subjected to random data access patterns on > disk > >> as opposed to sequential." > >> > >> I don't follow the logic here, sorry. > >> > >> Couple of side comments: > >> > >> * In your Hadoop vs RT example, Rabbit would deliver the RT messages > >> immediately and write the rest to disk. It can do this at high rates > >> - I shall try to get you some useful data here. > >> > >> * Bear in mind that write speed should be orthogonal to read speed. > >> Ask yourself - how would Kafka provide a read cache, and when might > >> that be useful? > >> > >> * I'll find out what data structure Rabbit uses for long term > persistence. > >> > >> > >> "Quoting the Kafka design page ( > >> http://kafka.apache.org/07/design.html) performance of sequential > writes > >> on > >> a 6 7200rpm SATA RAID-5 array is about 300MB/sec but the performance of > >> random writes is only about 50k/sec—a difference of nearly 10000X." > >> > >> Depending on your use case, I'd expect 2x-10x overall throughput > >> differences, and will try to find out more info. As I said, Rabbit > >> can saturate disk i/o. > >> > >> alexis > >> > >> > >> > >> > >> > > >> >> While you are correct the payload is a much bigger concern, managing > the > >> >> metadata and acks centrally on the broker across multiple clients at > >> scale > >> >> is also a concern. This would seem to be exasperated if you have > >> > consumers > >> >> at different speeds i.e. Storm and Hadoop consuming the same topic. > >> >> > >> >> In that scenario, say storm consumes the topic messages in real-time > and > >> >> Hadoop consumes once a day. Let’s assume the topic consists of 100k+ > >> >> messages/sec throughput so that in a given day you might have 100s > GBs > >> of > >> >> data flowing through the topic. > >> >> > >> >> To allow Hadoop to consume once a day, Rabbit obviously can’t keep > 100s > >> > GBs > >> >> in memory and will need to persist this data to its internal DB to be > >> >> retrieved later. > >> > > >> > I am not sure why you think this is a problem? > >> > > >> > For a fixed number of producers and consumers, the pubsub and delivery > >> > semantics of Rabbit and Kafka are quite similar. Think of Rabbit as > >> > adding an in-memory cache that is used to (a) speed up read > >> > consumption, (b) obviate disk writes when possible due to all client > >> > consumers being available and consuming. > >> > > >> > > >> > Actually I think this is the main use case that sets Kafka apart from > >> > Rabbit and speaks to the poster’s ‘Arguments for Kafka over RabbitMQ’ > >> > question. As you mentioned Rabbit is a general purpose messaging > system > >> > and along with that has a lot of features not found in Kafka. There > are > >> > plenty of times when Rabbit makes more sense than Kafka, but not when > you > >> > are maintaining large message stores and require high throughput to > disk. > >> > > >> > Persisting 100s GBs of messages to disk is a much different problem > than > >> > managing messages in memory. Since Rabbit must maintain the state of > the > >> > consumers I imagine it’s subjected to random data access patterns on > disk > >> > as opposed to sequential. Quoting the Kafka design page ( > >> > http://kafka.apache.org/07/design.html) performance of sequential > >> writes on > >> > a 6 7200rpm SATA RAID-5 array is about 300MB/sec but the performance > of > >> > random writes is only about 50k/sec—a difference of nearly 10000X. > >> > > >> > They go on to say persistent data structure used in messaging systems > >> > metadata is often a BTree. BTrees are the most versatile data > structure > >> > available, and make it possible to support a wide variety of > >> transactional > >> > and non-transactional semantics in the messaging system. They do come > >> with > >> > a fairly high cost, though: Btree operations are O(log N). Normally > O(log > >> > N) is considered essentially equivalent to constant time, but this is > not > >> > true for disk operations. Disk seeks come at 10 ms a pop, and each > disk > >> can > >> > do only one seek at a time so parallelism is limited. Hence even a > >> handful > >> > of disk seeks leads to very high overhead. Since storage systems mix > very > >> > fast cached operations with actual physical disk operations, the > observed > >> > performance of tree structures is often superlinear. Furthermore > BTrees > >> > require a very sophisticated page or row locking implementation to > avoid > >> > locking the entire tree on each operation. The implementation must > pay a > >> > fairly high price for row-locking or else effectively serialize all > >> reads. > >> > Because of the heavy reliance on disk seeks it is not possible to > >> > effectively take advantage of the improvements in drive density, and > one > >> is > >> > forced to use small (< 100GB) high RPM SAS drives to maintain a sane > >> ratio > >> > of data to seek capacity. > >> > > >> > Intuitively a persistent queue could be built on simple reads and > appends > >> > to files as is commonly the case with logging solutions. Though this > >> > structure would not support the rich semantics of a BTree > implementation, > >> > but it has the advantage that all operations are O(1) and reads do not > >> > block writes or each other. This has obvious performance advantages > since > >> > the performance is completely decoupled from the data size--one server > >> can > >> > now take full advantage of a number of cheap, low-rotational speed > 1+TB > >> > SATA drives. Though they have poor seek performance, these drives > often > >> > have comparable performance for large reads and writes at 1/3 the > price > >> and > >> > 3x the capacity. > >> > > >> > Having access to virtually unlimited disk space without penalty means > >> that > >> > we can provide some features not usually found in a messaging system. > For > >> > example, in kafka, instead of deleting a message immediately after > >> > consumption, we can retain messages for a relative long period (say a > >> week). > >> > > >> > Our assumption is that the volume of messages is extremely high, > indeed > >> it > >> > is some multiple of the total number of page views for the site > (since a > >> > page view is one of the activities we process). Furthermore we assume > >> each > >> > message published is read at least once (and often multiple times), > hence > >> > we optimize for consumption rather than production. > >> > > >> > There are two common causes of inefficiency: too many network > requests, > >> and > >> > excessive byte copying. > >> > > >> > To encourage efficiency, the APIs are built around a "message set" > >> > abstraction that naturally groups messages. This allows network > requests > >> to > >> > group messages together and amortize the overhead of the network > >> roundtrip > >> > rather than sending a single message at a time. > >> > > >> > The MessageSet implementation is itself a very thin API that wraps a > byte > >> > array or file. Hence there is no separate serialization or > >> deserialization > >> > step required for message processing, message fields are lazily > >> > deserialized as needed (or not deserialized if not needed). > >> > > >> > The message log maintained by the broker is itself just a directory of > >> > message sets that have been written to disk. This abstraction allows a > >> > single byte format to be shared by both the broker and the consumer > (and > >> to > >> > some degree the producer, though producer messages are checksumed and > >> > validated before being added to the log). > >> > > >> > Maintaining this common format allows optimization of the most > important > >> > operation: network transfer of persistent log chunks. Modern unix > >> operating > >> > systems offer a highly optimized code path for transferring data out > of > >> > pagecache to a socket; in Linux this is done with the sendfile system > >> call. > >> > Java provides access to this system call with the > FileChannel.transferTo > >> > api. > >> > > >> > To understand the impact of sendfile, it is important to understand > the > >> > common data path for transfer of data from file to socket: > >> > > >> > 1. The operating system reads data from the disk into pagecache in > >> kernel > >> > space > >> > 2. The application reads the data from kernel space into a > user-space > >> > buffer > >> > 3. The application writes the data back into kernel space into a > socket > >> > buffer > >> > 4. The operating system copies the data from the socket buffer to > the > >> NIC > >> > buffer where it is sent over the network > >> > > >> > This is clearly inefficient, there are four copies, two system calls. > >> Using > >> > sendfile, this re-copying is avoided by allowing the OS to send the > data > >> > from pagecache to the network directly. So in this optimized path, > only > >> the > >> > final copy to the NIC buffer is needed. > >> > > >> > We expect a common use case to be multiple consumers on a topic. Using > >> the > >> > zero-copy optimization above, data is copied into pagecache exactly > once > >> > and reused on each consumption instead of being stored in memory and > >> copied > >> > out to kernel space every time it is read. This allows messages to be > >> > consumed at a rate that approaches the limit of the network > connection. > >> > > >> > > >> > So in the end it would seem Kafka’s specialized nature to write data > >> first > >> > really shines over Rabbit when your use case requires a very high > >> > throughput unblocking firehose with large data persistence to disk. > >> Since > >> > this is only one use case this by no means is saying Kafka is better > than > >> > Rabbit or vice versa. I think it is awesome there are more options to > >> > choose from so you can pick the right tool for the job. Thanks open > >> source! > >> > > >> > As always YMMV. > >> > > >> > > >> > > >> > On Fri, Jun 7, 2013 at 4:40 PM, Alexis Richardson < > >> > alexis.richard...@gmail.com> wrote: > >> > > >> >> Jonathan, > >> >> > >> >> > >> >> On Fri, Jun 7, 2013 at 7:03 PM, Jonathan Hodges <hodg...@gmail.com> > >> wrote: > >> >> > Hi Alexis, > >> >> > > >> >> > I appreciate your reply and clarifications to my misconception > about > >> >> > Rabbit, particularly on the copying of the message payloads per > >> consumer. > >> >> > >> >> Thank-you! > >> >> > >> >> > >> >> > It sounds like it only copies metadata like the consumer state > i.e. > >> >> > position in the topic messages. > >> >> > >> >> Basically yes. Of course when a message is delivered to N>1 > >> >> *machines*, then there will be N copies, one per machine. > >> >> > >> >> Also, for various reasons, very tiny (<60b) messages do get copied as > >> >> you'd assumed. > >> >> > >> >> > >> >> > I don’t have experience with Rabbit and > >> >> > was basing this assumption based on Google searches like the > >> following - > >> >> > > >> >> > >> > http://ilearnstack.com/2013/04/16/introduction-to-amqp-messaging-with-rabbitmq/ > >> >> . > >> >> > It seems to indicate with topic exchanges that the messages get > >> copied > >> >> to > >> >> > a queue per consumer, but I am glad you confirmed it is just the > >> >> metadata. > >> >> > >> >> Yup. > >> >> > >> >> That's a fairly decent article but even the good stuff uses words > like > >> >> "copy" without a fixed denotation. Don't believe the internets! > >> >> > >> >> > >> >> > While you are correct the payload is a much bigger concern, > managing > >> the > >> >> > metadata and acks centrally on the broker across multiple clients > at > >> >> scale > >> >> > is also a concern. This would seem to be exasperated if you have > >> >> consumers > >> >> > at different speeds i.e. Storm and Hadoop consuming the same topic. > >> >> > > >> >> > In that scenario, say storm consumes the topic messages in > real-time > >> and > >> >> > Hadoop consumes once a day. Let’s assume the topic consists of > 100k+ > >> >> > messages/sec throughput so that in a given day you might have 100s > >> GBs of > >> >> > data flowing through the topic. > >> >> > > >> >> > To allow Hadoop to consume once a day, Rabbit obviously can’t keep > >> 100s > >> >> GBs > >> >> > in memory and will need to persist this data to its internal DB to > be > >> >> > retrieved later. > >> >> > >> >> I am not sure why you think this is a problem? > >> >> > >> >> For a fixed number of producers and consumers, the pubsub and > delivery > >> >> semantics of Rabbit and Kafka are quite similar. Think of Rabbit as > >> >> adding an in-memory cache that is used to (a) speed up read > >> >> consumption, (b) obviate disk writes when possible due to all client > >> >> consumers being available and consuming. > >> >> > >> >> > >> >> > I believe when large amounts of data need to be persisted > >> >> > is the scenario described in the earlier posted Kafka paper ( > >> >> > > >> >> > >> > http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf > >> >> ) > >> >> > where Rabbit’s performance really starts to bog down as compared to > >> >> Kafka. > >> >> > >> >> Not sure what parts of the paper you mean? > >> >> > >> >> I read that paper when it came out. I found it strongest when > >> >> describing Kafka's design philosophy. I found the performance > >> >> statements made about Rabbit pretty hard to understand. This is not > >> >> meant to be a criticism of the authors! I have seen very few > >> >> performance papers about messaging that I would base decisions on. > >> >> > >> >> > >> >> > This Kafka paper is looks to be a few years old > >> >> > >> >> Um.... Lots can change in technology very quickly :-) > >> >> > >> >> Eg.: At the time this paper was published, Instagram had 5m users. > >> >> Six months earlier in Dec 2010, it had 1m. Since then it grew huge > >> >> and got acquired. > >> >> > >> >> > >> >> > >> >> > so has something changed > >> >> > within the Rabbit architecture to alleviate this issue when large > >> amounts > >> >> > of data are persisted to the internal DB? > >> >> > >> >> Rabbit introduced a new internal flow control system which impacted > >> >> performance under steady load. This may be relevant? I couldn't say > >> >> from reading the paper. > >> >> > >> >> I don't have a good reference for this to hand, but here is a post > >> >> about external flow control that you may find amusing: > >> >> > >> >> > >> > http://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory-throughput-latency-and-bandwidth/ > >> >> > >> >> > >> >> > Do the producer and consumer > >> >> > numbers look correct? If no, maybe you can share some Rabbit > >> benchmarks > >> >> > under this scenario, because I believe it is the main area where > Kafka > >> >> > appears to be the superior solution. > >> >> > >> >> This is from about one year ago: > >> >> > >> >> > >> > http://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-measurements-part-2/ > >> >> > >> >> Obviously none of this uses batching, which is an easy trick for > >> >> increasing throughput. > >> >> > >> >> YMMV. > >> >> > >> >> Is this helping? > >> >> > >> >> alexis > >> >> > >> >> > >> >> > >> >> > Thanks for educating me on these matters. > >> >> > > >> >> > -Jonathan > >> >> > > >> >> > > >> >> > > >> >> > On Fri, Jun 7, 2013 at 6:54 AM, Alexis Richardson < > >> ale...@rabbitmq.com > >> >> >wrote: > >> >> > > >> >> >> Hi > >> >> >> > >> >> >> Alexis from Rabbit here. I hope I am not intruding! > >> >> >> > >> >> >> It would be super helpful if people with questions, observations > or > >> >> >> moans posted them to the rabbitmq list too :-) > >> >> >> > >> >> >> A few comments: > >> >> >> > >> >> >> * Along with ZeroMQ, I consider Kafka to be one of the interesting > >> and > >> >> >> useful messaging projects out there. In a world of cruft, Kafka > is > >> >> >> cool! > >> >> >> > >> >> >> * This is because both projects come at messaging from a specific > >> >> >> point of view that is *different* from Rabbit. OTOH, many other > >> >> >> projects exist that replicate Rabbit features for fun, or NIH, or > due > >> >> >> to misunderstanding the semantics (yes, our docs could be better) > >> >> >> > >> >> >> * It is striking how few people describe those differences. In a > >> >> >> nutshell they are as follows: > >> >> >> > >> >> >> *** Kafka writes all incoming data to disk immediately, and then > >> >> >> figures out who sees what. So it is much more like a database > than > >> >> >> Rabbit, in that new consumers can appear well after the disk write > >> and > >> >> >> still subscribe to past messages. Instead, Rabbit which tries to > >> >> >> deliver to consumers and buffers otherwise. Persistence is > optional > >> >> >> but robust and a feature of the buffer ("queue") not the upstream > >> >> >> machinery. Rabbit is able to cache-on-arrival via a plugin, but > this > >> >> >> is a design overlay and not particularly optimal. > >> >> >> > >> >> >> *** Kafka is a client server system with end to end semantics. It > >> >> >> defines order to include processing order, and keeps state on the > >> >> >> client to do this. Group management is via a 3rd party service > >> >> >> (Zookeeper? I forget which). Rabbit is a server-only protocol > based > >> >> >> system which maintains order on the server and through completely > >> >> >> language neutral protocol semantics. This makes Rabbit perhaps > more > >> >> >> natural as a 'messaging service' eg for integration and other > >> >> >> inter-app data transfer. > >> >> >> > >> >> >> *** Rabbit is a general purpose messaging system with extras like > >> >> >> federation. It speaks many protocols, and has core features like > HA, > >> >> >> transactions, management, etc. Everything can be switched on or > off. > >> >> >> Getting all this to work while keeping the install light and > fast, is > >> >> >> quite fiddly. Kafka by contrast comes from a specific set of use > >> >> >> cases, which are interesting certainly. I am not sure if Kafka > wants > >> >> >> to be a general purpose messaging system, but it will become a bit > >> >> >> more like Rabbit if that is the goal. > >> >> >> > >> >> >> *** Both approaches have costs. In the case of Rabbit the cost is > >> >> >> that more metadata is stored on the broker. Kafka can get > >> performance > >> >> >> gains by storing less such data. But we are talking about some N > >> >> >> thousands of MPS versus some M thousands. At those speeds the > >> clients > >> >> >> are usually the bottleneck anyway. > >> >> >> > >> >> >> * Let me also clarify some things: > >> >> >> > >> >> >> *** Rabbit does NOT store multiple copies of the same message > across > >> >> >> queues, unless they are very small (<60b, iirc). A message > delivered > >> >> >> to >1 queue on 1 machine is stored once. Metadata about that > message > >> >> >> may be stored more than once, but, at scale, the big cost is the > >> >> >> payload. > >> >> >> > >> >> >> *** Rabbit's vanilla install does store some index data in memory > >> when > >> >> >> messages flow to disk. You can change this by using a plugin, but > >> >> >> this is a secret-menu undocumented feature. Very very few people > >> need > >> >> >> any such thing. > >> >> >> > >> >> >> *** A Rabbit queue is lightweight. It's just an ordered > consumption > >> >> >> buffer that can persist and ack. Don't assume things about Rabbit > >> >> >> queues based on what you know about IBM MQ, JMS, and so forth. > >> Queues > >> >> >> in Rabbit and Kafka are not the same. > >> >> >> > >> >> >> *** Rabbit does not use mnesia for message storage. It has its > own > >> >> >> DB, optimised for messaging. You can use other DBs but this is > >> >> >> Complicated. > >> >> >> > >> >> >> *** Rabbit does all kinds of batching and bulk processing, and can > >> >> >> batch end to end. If you see claims about batching, buffering, > etc., > >> >> >> find out ALL the details before drawing conclusions. > >> >> >> > >> >> >> I hope this is helpful. > >> >> >> > >> >> >> Keen to get feedback / questions / corrections. > >> >> >> > >> >> >> alexis > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> On Fri, Jun 7, 2013 at 2:09 AM, Marc Labbe <mrla...@gmail.com> > >> wrote: > >> >> >> > We also went through the same decision making and our arguments > for > >> >> Kafka > >> >> >> > where in the same lines as those Jonathan mentioned. The fact > that > >> we > >> >> >> have > >> >> >> > heterogeneous consumers is really a deciding factor. Our > >> requirements > >> >> >> were > >> >> >> > to avoid loosing messages at all cost while having multiple > >> consumers > >> >> >> > reading the same data at a different pace. On one side, we have > a > >> few > >> >> >> > consumers being fed with data coming in from most, if not all, > >> >> topics. On > >> >> >> > the other side, we have a good bunch of consumers reading only > >> from a > >> >> >> > single topic. The big guys can take their time to read while the > >> >> smaller > >> >> >> > ones are mostly for near real-time events so they need to keep > up > >> the > >> >> >> pace > >> >> >> > of incoming messages. > >> >> >> > > >> >> >> > RabbitMQ stores data on disk only if you tell it to while Kafka > >> >> persists > >> >> >> by > >> >> >> > design. From the beginning, we decided we would try to use the > >> queues > >> >> the > >> >> >> > same way, pub/sub with a routing key (an exchange in RabbitMQ) > or > >> >> topic, > >> >> >> > persisted to disk and replicated. > >> >> >> > > >> >> >> > One of our scenario was to see how the system would cope with > the > >> >> largest > >> >> >> > consumer down for a while, therefore forcing the brokers to keep > >> the > >> >> data > >> >> >> > for a long period. In the case of RabbitMQ, this consumer has it > >> owns > >> >> >> queue > >> >> >> > and data grows on disk, which is not really a problem if you > plan > >> >> >> > consequently. But, since it has to keep track of all messages > read, > >> >> the > >> >> >> > Mnesia database used by RabbitMQ as the messages index also > grows > >> >> pretty > >> >> >> > big. At that point, the amount of RAM necessary becomes very > large > >> to > >> >> >> keep > >> >> >> > the level of performance we need. In our tests, we found that > this > >> an > >> >> >> > adverse effect on ALL the brokers, thus affecting all consumers. > >> You > >> >> can > >> >> >> > always say that you'll monitor the consumers to make sure it > won't > >> >> >> happen. > >> >> >> > That's a good thing if you can. I wasn't ready to make that bet. > >> >> >> > > >> >> >> > Another point is the fact that, since we wanted to use pub/sub > >> with a > >> >> >> > exchange in RabbitMQ, we would have ended up with a lot data > >> >> duplication > >> >> >> > because if a message is read by multiple consumers, it will get > >> >> >> duplicated > >> >> >> > in the queue of each of those consumer. Kafka wins on that side > too > >> >> since > >> >> >> > every consumer reads from the same source. > >> >> >> > > >> >> >> > The downsides of Kafka were the language issues (we are using > >> mostly > >> >> >> Python > >> >> >> > and C#). 0.8 is very new and few drivers are available at this > >> point. > >> >> >> Also, > >> >> >> > we will have to try getting as close as possible to > >> once-and-only-once > >> >> >> > guarantee. There are two things where RabbitMQ would have given > us > >> >> less > >> >> >> > work out of the box as opposed to Kafka. RabbitMQ also provides > a > >> >> bunch > >> >> >> of > >> >> >> > tools that makes it rather attractive too. > >> >> >> > > >> >> >> > In the end, looking at throughput is a pretty nifty thing but > being > >> >> sure > >> >> >> > that I'll be able to manage the beast as it grows will allow me > to > >> >> get to > >> >> >> > sleep way more easily. > >> >> >> > > >> >> >> > > >> >> >> > On Thu, Jun 6, 2013 at 3:28 PM, Jonathan Hodges < > hodg...@gmail.com > >> > > >> >> >> wrote: > >> >> >> > > >> >> >> >> We just went through a similar exercise with RabbitMQ at our > >> company > >> >> >> with > >> >> >> >> streaming activity data from our various web properties. Our > use > >> >> case > >> >> >> >> requires consumption of this stream by many heterogeneous > >> consumers > >> >> >> >> including batch (Hadoop) and real-time (Storm). We pointed out > >> that > >> >> >> Kafka > >> >> >> >> acts as a configurable rolling window of time on the activity > >> stream. > >> >> >> The > >> >> >> >> window default is 7 days which allows for supporting clients of > >> >> >> different > >> >> >> >> latencies like Hadoop and Storm to read from the same stream. > >> >> >> >> > >> >> >> >> We pointed out that the Kafka brokers don't need to maintain > >> consumer > >> >> >> state > >> >> >> >> in the stream and only have to maintain one copy of the stream > to > >> >> >> support N > >> >> >> >> number of consumers. Rabbit brokers on the other hand have to > >> >> maintain > >> >> >> the > >> >> >> >> state of each consumer as well as create a copy of the stream > for > >> >> each > >> >> >> >> consumer. In our scenario we have 10-20 consumers and with the > >> scale > >> >> >> and > >> >> >> >> throughput of the activity stream we were able to show Rabbit > >> quickly > >> >> >> >> becomes the bottleneck under load. > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> On Thu, Jun 6, 2013 at 12:40 PM, Dragos Manolescu < > >> >> >> >> dragos.manole...@servicenow.com> wrote: > >> >> >> >> > >> >> >> >> > Hi -- > >> >> >> >> > > >> >> >> >> > I am preparing to make a case for using Kafka instead of > Rabbit > >> MQ > >> >> as > >> >> >> a > >> >> >> >> > broker-based messaging provider. The context is similar to > that > >> of > >> >> the > >> >> >> >> > Kafka papers and user stories: the producers publish > monitoring > >> >> data > >> >> >> and > >> >> >> >> > logs, and a suite of subscribers consume this data (some > store > >> it, > >> >> >> others > >> >> >> >> > perform computations on the event stream). The requirements > are > >> >> >> typical > >> >> >> >> of > >> >> >> >> > this context: low-latency, high-throughput, ability to deal > with > >> >> >> bursts > >> >> >> >> and > >> >> >> >> > operate in/across multiple data centers, etc. > >> >> >> >> > > >> >> >> >> > I am familiar with the performance comparison between Kafka, > >> >> Rabbit MQ > >> >> >> >> and > >> >> >> >> > Active MQ from the NetDB 2011 paper< > >> >> >> >> > > >> >> >> >> > >> >> >> > >> >> > >> > http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf > >> >> >> >> >. > >> >> >> >> > However in the two years that passed since then the number of > >> >> >> production > >> >> >> >> > Kafka installations increased, and people are using it in > >> different > >> >> >> ways > >> >> >> >> > than those imagined by Kafka's designers. In light of these > >> >> >> experiences > >> >> >> >> one > >> >> >> >> > can use more data points and color when contrasting to > Rabbit MQ > >> >> >> (which > >> >> >> >> by > >> >> >> >> > the way also evolved since 2011). (And FWIW I know I am not > the > >> >> first > >> >> >> one > >> >> >> >> > to walk this path; see for example last year's OSCON session > on > >> the > >> >> >> State > >> >> >> >> > of MQ<http://lanyrd.com/2012/oscon/swrcz/>.) > >> >> >> >> > > >> >> >> >> > I would appreciate it if you could share measurements, > results, > >> or > >> >> >> even > >> >> >> >> > anecdotal evidence along these lines. How have you avoided > the > >> >> "let's > >> >> >> use > >> >> >> >> > Rabbit MQ because everybody else does it" route when solving > >> >> problems > >> >> >> for > >> >> >> >> > which Kafka is a better fit? > >> >> >> >> > > >> >> >> >> > Thanks, > >> >> >> >> > > >> >> >> >> > -Dragos > >> >> >> >> > > >> >> >> >> > >> >> >> > >> >> > >> >
