Messages are around 100 per second. A message size is around 4 KB. This is for source topic. >From this, it is keyed to an intermediate topic and aggregated to a table. Keyed message will be around 1 KB or so.
On Thu, Dec 1, 2016 at 9:44 PM, Tauzell, Dave <dave.tauz...@surescripts.com> wrote: > Do you have some idea of the size and number of messages per second you'll > put onto the topics at peak? > > -Dave > > -----Original Message----- > From: Sachin Mittal [mailto:sjmit...@gmail.com] > Sent: Thursday, December 1, 2016 9:44 AM > To: users@kafka.apache.org > Subject: Re: I need some help with the production server architecture > > And what about my brokers. Should I hedge them as well. > > Like say put 2 zk on nodejs server and 1 on db server. > Put 2 brokers on db server and 1 on nodejs server, something like that. > > Thanks > Sachin > > > On Thu, Dec 1, 2016 at 8:59 PM, Tauzell, Dave < > dave.tauz...@surescripts.com> > wrote: > > > For low volume zookeeper doesn't seem to use many resources. I would > put > > it on nodejs server as that will have less IO and heavy IO could > > impact zookeeper. Or, you could put some ZK nodes on nodejs and some on > DB > > servers to hedge your bets. As always, you'll find out a lot once you > > actually start running it in production. > > > > -Dave > > > > -----Original Message----- > > From: Sachin Mittal [mailto:sjmit...@gmail.com] > > Sent: Thursday, December 1, 2016 6:03 AM > > To: users@kafka.apache.org > > Subject: Re: I need some help with the production server architecture > > > > Folks any help on this. > > > > Just to put it in simple terms, since we have limited resources > > available to us what is better option 1. run zookeeper on servers > > running the nodejs web server or db server. > > 2. what about kafka brokers. > > > > Thanks > > Sachin > > > > > > On Tue, Nov 29, 2016 at 1:06 PM, Sachin Mittal <sjmit...@gmail.com> > wrote: > > > > > Hi, > > > Sometime back i was informed on the group that in production we > > > should never run kafka on same physical machine. So based on that I > > > have a question on how to divide the server nodes we have to run > > > zookeper and kafka brokers. > > > > > > I have a following setup > > > Data center 1 > > > Lan 1 (3 VMs) > > > 192.168.xx.yy1 > > > 192.168.xx.yy2 > > > 192.168.xx.yy3 > > > Right now here we are running a cluster of 3 nodejs web servers. > > > These collect data from web and write to kafka queue. Each VM has 70 > > > GB of space. > > > > > > Lan 2 (3 VMs) > > > 192.168.zz.aa1 > > > 192.168.zz.aa2 > > > 192.168.zz.aa3 > > > These are served the cluster of our database server. Each VM has 400 > > > GB of space. > > > > > > Date center 2 > > > Lan 1 (3 VMs) > > > 192.168.yy.bb1 > > > 192.168.yy.bb2 > > > 192.168.yy.bb3 > > > Three new machines where we plan to run a cluster of new database to > > > be served as sink of kafka stream applications. Each VM has 400 GB > > > of > > space. > > > These have connectivity only between Lan 2 of Data center 1 with a > > > 100MBs of data transfer rate. > > > > > > Each VM has a 4 core processor and 16 GB of RAM. They all run linux. > > > > > > Now I would like my topics to be replicated with a factor of 3. > > > Since we don't foresee much volume of data, I don't want it to be > partitioned. > > > > > > Also we would like one server to be used as streaming application > > > server, where we can run one or more kafka stream applications to > > > process the topics and write to the new database. > > > > > > So please let me know what is a suitable division to run brokers > > > and zookeeper. > > > > > > > > > Thanks > > > Sachin > > > > > > > > > > > > > > This e-mail and any files transmitted with it are confidential, may > > contain sensitive information, and are intended solely for the use of > > the individual or entity to whom they are addressed. If you have > > received this e-mail in error, please notify the sender by reply > > e-mail immediately and destroy all copies of the e-mail and any > attachments. > > > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of the > individual or entity to whom they are addressed. If you have received this > e-mail in error, please notify the sender by reply e-mail immediately and > destroy all copies of the e-mail and any attachments. >
