Millions of messages per day (with each message being few bytes) is not really 'Big Data'. Kafka has been tested for a million message per second.
The answer to all your question IMO is "It depends". You can start with a single instance (Single machine installation). Let your producer send messages. Keep one broker. Increase to N brokers. When you touch the upper limit add a server and repeat all the stuff. Bench marking and scalability are aspects which you should try on your own by playing with Kafka. Every use case is different. So performance metric of one is not a global answer. For your question on Topic or Queue, please read something about Distributed Computing Pub/Sub, Message Queue's and other patterns which are generic concepts and has nothing to do with Kafka. It again depends on your use case. Please read as to what topics in Kafka are? If you just go through the definition of topics you would yourself answer your question within a minute. Replications and all would be next steps once you are done with a single running instance of Kafka. So go ahead and get your hands dirty. You will love Kafka :) And yes, the most important thing: Please read the documentation first (bit of theory) and then dive. There is no silver bullet. Cheers, Yavar http://lnkd.in/GRrrDJ On Mon, Jul 22, 2013 at 4:27 PM, <anantha.muru...@wipro.com> wrote: > Hi, > > > > I am planning to use Apache Kafka 0.8 to handle millions of messages per > day. Now I need to form the environment, like > > > > (i) How many Topics to be created? > (ii) How many partitions/replications to be created? > (iii) How many Brokers to be created? > (iv) How many consumer instances in consumer group? > > (v) Topic or Queue? If topic whether we need to create multiple group Id > as supposed to single one? > > > > How we can go about it? Please clarify. > > Thanks & Regards, > Anantha > > Please do not print this email unless it is absolutely necessary. > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) and > may contain proprietary, confidential or privileged information. If you are > not the intended recipient, you should not disseminate, distribute or copy > this e-mail. Please notify the sender immediately and destroy all copies of > this message and any attachments. > > WARNING: Computer viruses can be transmitted via email. The recipient > should check this email and any attachments for the presence of viruses. > The company accepts no liability for any damage caused by any virus > transmitted by this email. > > www.wipro.com >
