I think what you are asking for is backpressure from the broker to the producer. I.e. when the broker got close to full it would start to slow down the producer to let the consumer catch up. This is a fairly typical thing for a message broker to do.
Our approach is different though. We have found that most use cases cannot tolerate backpressure because the production of messages comes from a live service that cannot stop. So we have instead focused on scaling the data that is retained and the consumption. In Kafka it is very reasonable to retain 5TB per server and you can have lots of servers to provide sufficient buffer. If you feel like there are use cases where backpressure is preferrable it would probably be good for us to hear about them because right now we haven't seen a need for it. -Jay On Wed, Dec 11, 2013 at 11:42 PM, xingcan <[email protected]> wrote: > Guozhang, > > Thanks for your prompt replay. I got two 300GB SAS disks for each borker. > At peak time, the produce speed for each broker is about 70MB/s. > Apparently, > this speed is already restricted by network. While, the consume speed is > lower > for some topics are consumed by more than one group. Under this > circumstance, > if the peak time lasts for hours, my disks will be fully used. > > > On Thu, Dec 12, 2013 at 2:00 PM, Guozhang Wang <[email protected]> wrote: > > > One possible approach is to change the retention policy on broker. > > > > How large your messages can accumulate on brokers at peak time? > > > > Guozhang > > > > > > On Wed, Dec 11, 2013 at 9:09 PM, xingcan <[email protected]> wrote: > > > > > Hi, > > > > > > In my application, the produce speed could be very high at some > specific > > > time in a day while return > > > to a low speed at the rest of time. Frequently, my data logs are > flushed > > > away before they are being > > > consumed by clients due to lacking disk space during the busy times. > > > > > > Increasing consume speed seems to be difficult and adding space for log > > > files are also not a ultimate solution. Any suggestions for this > problem? > > > Are there any speed control mechanism in Kafka? > > > > > > Thanks. > > > > > > -- > > > *Xingcan* > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > *Xingcan* >
