Re: Multiple brokers - do they share the load?
Ben wrote: > If you set these configs at broker start they will automatically replicate > newly created topics: > > default.replication.factor > num.partitions=3 Thanks. (I assume the first config is missing a trailing "=3" bit). Final question (I think). If I restart my brokers with those configuration changes, will existing topics acquire the desired replication factor, or will I need to do something else to tweak existing topics? Skip > Ben > > On Fri, Dec 1, 2017 at 1:41 PM Skip Montanaro <skip.montan...@gmail.com> > wrote: > >> Are dynamically created topics thus never replicated? >> >> On Nov 30, 2017 6:52 PM, "Tauzell, Dave" <dave.tauz...@surescripts.com> >> wrote: >> >> > You then also need to set this up for each topic you create: >> > >> > > bin/kafka-topics.sh --create --zookeeper localhost:2181 >> > --replication-factor 3 --partitions 3 --topic my-replicated-topic >> > >> > -Dave >> > >> > -Original Message- >> > From: Skip Montanaro [mailto:skip.montan...@gmail.com] >> > Sent: Thursday, November 30, 2017 4:20 PM >> > To: users@kafka.apache.org >> > Subject: Re: Multiple brokers - do they share the load? >> > >> > > If you create a partitioned topic with at least 3 partitions then you >> > will see your client connect to all of the brokers. The client decides >> > which partition a message should go to and then sends it directly to the >> > broker that is the leader for that partition. If you have replicated >> > topics, then the brokers themselves will also be connected to one-another >> > in order to replication messages. >> > >> > Thanks. So if I create three brokers all pointing to the same Zookeeper >> > instance, then define >> > >> > offsets.topic.replication.factor=3 >> > transaction.state.log.replication.factor=3 >> > >> > the brokers will find each other via Zookeeper and dynamically created >> > topics (which is all I've used so far) will be properly replicated? >> > Should I also bump transaction.state.log.min.isr above 1? >> > >> > Apologies for such basic questions. I'm still trying to absorb the >> > documentation related to overall architecture and broker specifics. >> > >> > Skip >> > >> > This e-mail and any files transmitted with it are confidential, may >> > contain sensitive information, and are intended solely for the use of the >> > individual or entity to whom they are addressed. If you have received >> this >> > e-mail in error, please notify the sender by reply e-mail immediately and >> > destroy all copies of the e-mail and any attachments. >> > >> > > -- > > > This email, including attachments, is private and confidential. If you have > received this email in error please notify the sender and delete it from > your system. Emails are not secure and may contain viruses. No liability > can be accepted for viruses that might be transferred by this email or any > attachment. Any unauthorised copying of this message or unauthorised > distribution and publication of the information contained herein are > prohibited. > > 7digital Group plc. Registered office: 69 Wilson Street, London EC2A 2BB. > Registered in England and Wales. Registered No. 04843573.
Re: Multiple brokers - do they share the load?
If you set these configs at broker start they will automatically replicate newly created topics: default.replication.factor num.partitions=3 Ben On Fri, Dec 1, 2017 at 1:41 PM Skip Montanaro <skip.montan...@gmail.com> wrote: > Are dynamically created topics thus never replicated? > > On Nov 30, 2017 6:52 PM, "Tauzell, Dave" <dave.tauz...@surescripts.com> > wrote: > > > You then also need to set this up for each topic you create: > > > > > bin/kafka-topics.sh --create --zookeeper localhost:2181 > > --replication-factor 3 --partitions 3 --topic my-replicated-topic > > > > -Dave > > > > -Original Message- > > From: Skip Montanaro [mailto:skip.montan...@gmail.com] > > Sent: Thursday, November 30, 2017 4:20 PM > > To: users@kafka.apache.org > > Subject: Re: Multiple brokers - do they share the load? > > > > > If you create a partitioned topic with at least 3 partitions then you > > will see your client connect to all of the brokers. The client decides > > which partition a message should go to and then sends it directly to the > > broker that is the leader for that partition. If you have replicated > > topics, then the brokers themselves will also be connected to one-another > > in order to replication messages. > > > > Thanks. So if I create three brokers all pointing to the same Zookeeper > > instance, then define > > > > offsets.topic.replication.factor=3 > > transaction.state.log.replication.factor=3 > > > > the brokers will find each other via Zookeeper and dynamically created > > topics (which is all I've used so far) will be properly replicated? > > Should I also bump transaction.state.log.min.isr above 1? > > > > Apologies for such basic questions. I'm still trying to absorb the > > documentation related to overall architecture and broker specifics. > > > > Skip > > > > This e-mail and any files transmitted with it are confidential, may > > contain sensitive information, and are intended solely for the use of the > > individual or entity to whom they are addressed. If you have received > this > > e-mail in error, please notify the sender by reply e-mail immediately and > > destroy all copies of the e-mail and any attachments. > > > -- This email, including attachments, is private and confidential. If you have received this email in error please notify the sender and delete it from your system. Emails are not secure and may contain viruses. No liability can be accepted for viruses that might be transferred by this email or any attachment. Any unauthorised copying of this message or unauthorised distribution and publication of the information contained herein are prohibited. 7digital Group plc. Registered office: 69 Wilson Street, London EC2A 2BB. Registered in England and Wales. Registered No. 04843573.
RE: Multiple brokers - do they share the load?
Are dynamically created topics thus never replicated? On Nov 30, 2017 6:52 PM, "Tauzell, Dave" <dave.tauz...@surescripts.com> wrote: > You then also need to set this up for each topic you create: > > > bin/kafka-topics.sh --create --zookeeper localhost:2181 > --replication-factor 3 --partitions 3 --topic my-replicated-topic > > -Dave > > -Original Message- > From: Skip Montanaro [mailto:skip.montan...@gmail.com] > Sent: Thursday, November 30, 2017 4:20 PM > To: users@kafka.apache.org > Subject: Re: Multiple brokers - do they share the load? > > > If you create a partitioned topic with at least 3 partitions then you > will see your client connect to all of the brokers. The client decides > which partition a message should go to and then sends it directly to the > broker that is the leader for that partition. If you have replicated > topics, then the brokers themselves will also be connected to one-another > in order to replication messages. > > Thanks. So if I create three brokers all pointing to the same Zookeeper > instance, then define > > offsets.topic.replication.factor=3 > transaction.state.log.replication.factor=3 > > the brokers will find each other via Zookeeper and dynamically created > topics (which is all I've used so far) will be properly replicated? > Should I also bump transaction.state.log.min.isr above 1? > > Apologies for such basic questions. I'm still trying to absorb the > documentation related to overall architecture and broker specifics. > > Skip > > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of the > individual or entity to whom they are addressed. If you have received this > e-mail in error, please notify the sender by reply e-mail immediately and > destroy all copies of the e-mail and any attachments. >
RE: Multiple brokers - do they share the load?
You then also need to set this up for each topic you create: > bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor > 3 --partitions 3 --topic my-replicated-topic -Dave -Original Message- From: Skip Montanaro [mailto:skip.montan...@gmail.com] Sent: Thursday, November 30, 2017 4:20 PM To: users@kafka.apache.org Subject: Re: Multiple brokers - do they share the load? > If you create a partitioned topic with at least 3 partitions then you will > see your client connect to all of the brokers. The client decides which > partition a message should go to and then sends it directly to the broker > that is the leader for that partition. If you have replicated topics, then > the brokers themselves will also be connected to one-another in order to > replication messages. Thanks. So if I create three brokers all pointing to the same Zookeeper instance, then define offsets.topic.replication.factor=3 transaction.state.log.replication.factor=3 the brokers will find each other via Zookeeper and dynamically created topics (which is all I've used so far) will be properly replicated? Should I also bump transaction.state.log.min.isr above 1? Apologies for such basic questions. I'm still trying to absorb the documentation related to overall architecture and broker specifics. Skip This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Multiple brokers - do they share the load?
> If you create a partitioned topic with at least 3 partitions then you will > see your client connect to all of the brokers. The client decides which > partition a message should go to and then sends it directly to the broker > that is the leader for that partition. If you have replicated topics, then > the brokers themselves will also be connected to one-another in order to > replication messages. Thanks. So if I create three brokers all pointing to the same Zookeeper instance, then define offsets.topic.replication.factor=3 transaction.state.log.replication.factor=3 the brokers will find each other via Zookeeper and dynamically created topics (which is all I've used so far) will be properly replicated? Should I also bump transaction.state.log.min.isr above 1? Apologies for such basic questions. I'm still trying to absorb the documentation related to overall architecture and broker specifics. Skip
RE: Multiple brokers - do they share the load?
If you create a partitioned topic with at least 3 partitions then you will see your client connect to all of the brokers. The client decides which partition a message should go to and then sends it directly to the broker that is the leader for that partition. If you have replicated topics, then the brokers themselves will also be connected to one-another in order to replication messages. -Dave -Original Message- From: Skip Montanaro [mailto:skip.montan...@gmail.com] Sent: Tuesday, November 28, 2017 8:06 AM To: users@kafka.apache.org Subject: Multiple brokers - do they share the load? Apologies for the rather long set-up... I've been using Kafka as a client for a few months now. The setup I've been using has three brokers on separate servers, all listening to port 9092. My consumers always connected to server1:9092. I've ignored server2 and server3. Now I'm starting to mess around a bit with setting up my own itsy-bitsy cluster. Step one is a single instance at host1:9092. Next step in the instructions (I'm following the recipe laid out in the documentation) will be to add two more brokers at host1:9093 and host1:9094. My question: If every consumer connects to host1:9092 will the brokers listening to the other ports starve for attention, or does the connection process somehow redirect clients to the other brokers so the three (or more) brokers get fairly equitable loads? Thanks, Skip Montanaro This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Multiple brokers - do they share the load?
You are connecting to a single seed node - your kafka library will then under the hood connect to the partition leaders for each partition you subscribe or post to. The load is not different compared to if you gave all nodes as connect parameter. However if your seed node crashes then your client cannot connect to the cluster.,. 2017-11-28 15:06 GMT+01:00 Skip Montanaro: > Apologies for the rather long set-up... > > I've been using Kafka as a client for a few months now. The setup I've > been using has three brokers on separate servers, all listening to > port 9092. My consumers always connected to server1:9092. I've ignored > server2 and server3. > > Now I'm starting to mess around a bit with setting up my own > itsy-bitsy cluster. Step one is a single instance at host1:9092. Next > step in the instructions (I'm following the recipe laid out in the > documentation) will be to add two more brokers at host1:9093 and > host1:9094. > > My question: If every consumer connects to host1:9092 will the brokers > listening to the other ports starve for attention, or does the > connection process somehow redirect clients to the other brokers so > the three (or more) brokers get fairly equitable loads? > > Thanks, > > Skip Montanaro >