Yes, this is single machine, I think that is the most directly comparable case to 0.7. It would be good to rerun with a replicated topic too.
-Jay On Thu, Oct 25, 2012 at 5:44 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > Single machine performance looks great. I'm assuming this was done > with replication factor = 1, right ? > > In reality, the only factor that can affect this number, is the time > it takes for the followers to fetch the data. > This is important for end-to-end latency since until the follower > replicas fetch the data, the leader will not increment the > highwatermark to expose that data to the consumer, thereby affecting > end-to-end latency. > > Thanks, > Neha > > On Thu, Oct 25, 2012 at 5:08 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > I ran the end-to-end latency test on 0.8 branch and it actually looks > > pretty good even with the remaining 0.8 perf bugs. I get an average 0.29 > ms > > for the producer=>broker=>consumer roundtrip over localhost. > > > > Some background for those not following along: > > > > End-to-end latency is the time it takes for a message sent from the > > producer to arrive at the consumer. It actually does make sense to > measure > > this over localhost since the networks contribution is going to > > vary enormously based on the network. > > > > In 0.7 there was a pretty strong tradeoff between latency and throughput, > > you could have great throughput with high latency (hundreds of ms to > > seconds) or bad throughput with low latency (bad at least in terms of > disk > > capabilities, though comparable to other popular messaging systems). > There > > were two causes for this: (1) we were using the disk flush to guarantee > > durability and so we wouldn't hand out messages to consumers until the > > flush occurred, and (2) the consumer would backoff whenever it reached > the > > end of the log to avoid "spin waiting" and polling for new data in a > tight > > loop. > > > > Both these issues are resolved in 0.8 which is what leads to the > > improvement: > > - In 0.8 we have replication which gives a stronger durability guarantee > > without requiring that we block on the disk flush. > > - We added "long poll" for the consumer which ensures immediate message > > delivery without adding any polling overhead. > > > > -Jay >
