-On adding a new Kafka Broker(B) to the cluster. Will all data present
on broker A will be distributed automatically? If not what I need to do
distribute the data.
Automatic distribution doesn't happen. Use kafka-reassign-partitions.sh
command to distribute load manually.
- Not let's suppose somehow the case! is solved my data is distributed
on both the brokers. Now due to some maintenance issue, I want to take
down
the server B.
- How to transfer the data of Broker B to the already existing broker
A or to a new Broker C. Use kafka-reassign-partitions.sh command to
distribute load manually.
- How can I increase the replication factor of my brokers at runtime. RF
of topics can be increased in runtime. Default RF can be set in broker
config for topics created without any RF config.
- How can I change the zookeeper IPs present in Kafka Broker Config at
runtime without restarting Kafka? Not sure if this is ideal to do.
- How can I dynamically change the Kafka Configuration at runtime Can't
change broker config dynamically at runtime. Can change topic config using
kafka-topics.sh command.
- Regarding Kafka Client:
- Do I need to specify all Kafka broker IP to kafkaClient for
connection? Not necessary. But to be highly available, configure more
nodes.
- And each and every time a broker is added or removed does I need to
add or remove my IP in Kafka Client connection String. As it will
always
require to restart my producer and consumers? Not necessary when
there are available nodes in producer config. But also take a look at RF
and min.insync.replicas in such cases when one node goes down if the
producer fails to produce.
-Can I have two brokers at the same time running at multiple versions
just for Kafka version upgrade? Yes, you can. But recommended is to keep
all brokers running the same version. During rolling restart, its ok to
have one node upgraded while other nodes are still not.
*NOTE: Follow documentation <https://kafka.apache.org/documentation/> for
better understanding.*
Thanks.
Suman
On Fri, Dec 7, 2018 at 1:06 PM Abhimanyu Nagrath <abhimanyunagr...@gmail.com>
wrote:
> Hi Vahid,
>
> I missed one more point.
>
>
> - Can I have two brokers at the same time running at multiple versions
> just for Kafka version upgrade?
>
> Regards,
> Abhimanyu
>
> On Fri, Dec 7, 2018 at 1:03 PM Abhimanyu Nagrath <
> abhimanyunagr...@gmail.com>
> wrote:
>
> > Thanks for clarifying all the points . Just a small doubt regarding my
> > second query:
> >
> >
> > - *Not let's suppose somehow the case! is solved my data is
> > distributed on both the brokers. Now due to some maintenance issue, I
> want
> > to take down the server B.*
> > - *How to transfer the data of Broker B to the already existing
> > broker A or to a new Broker C.*
> >
> > Is there any command through which I can say that move all data from
> > broker B to broker A.
> >
> > Regards,
> > Abhimanyu
> >
> > On Fri, Dec 7, 2018 at 12:33 PM Vahid Hashemian <
> vahid.hashem...@gmail.com>
> > wrote:
> >
> >> Hi Abhimanyu,
> >>
> >> I have answered your questions inline, but before that I just want to
> >> emphasize the notion of topics and partitions that are critical to
> Kafka's
> >> resiliency and scalability.
> >>
> >> Topics in Kafka can have multiple partitions. Each partition can be
> stored
> >> on one broker only. But the number of partitions can grow over time to
> >> make
> >> topics scalable.
> >> Each topic partition can be configured to replicate on multiple brokers,
> >> and this is how data becomes resilient to failures and outages.
> >> You can find more detailed information in the highly recommended Kafka
> >> Documentation (https://kafka.apache.org/documentation/).
> >>
> >> I hope you find the answers below helpful.
> >>
> >> --Vahid
> >>
> >> On Thu, Dec 6, 2018 at 10:19 PM Abhimanyu Nagrath <
> >> abhimanyunagr...@gmail.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > I have a use case I want to set up a Kafka cluster initially at the
> >> > starting I have 1 Kafka Broker(A) and 1 Zookeeper Node. So below
> >> mentioned
> >> > are my queries:
> >> >
> >> > - On adding a new Kafka Broker(B) to the cluster. Will all data
> >> present
> >> > on broker A will be distributed automatically? If not what I need
> to
> >> do
> >> > distribute the data.
> >> >
> >>
> >> When you add a new broker, existing data will not automatically move. In
> >> order to have the new broker receive data, existing partitions need to
> >> manually move to that broker. Kafka provides a command line tool for
> >> (re)assigning partitions to broker (kafka-reassign-partitions example).
> >> As new topic partitions are added to the cluster they will be
> distributed
> >> in a way that keeps all brokers busy.
> >>
> >>
> >> > - Not let's suppose somehow the case! is solved my data is
> >> distributed
> >> > on both the brokers. Now due to some maintenance issue, I want to
> >> take
> >> > down
> >> > the server B.
> >> > - How to transfer the data of Broker B to the already existing
> >> broker
> >> > A or to a new Broker C.
> >> >
> >>
> >> You can use the reassign partition tools again to achieve that. If
> broker
> >> B
> >> is going to join the cluster again, you may not need to do anything,
> >> assuming you have created your topics (partitions) with resiliency in
> mind
> >> (with enough replicas). Kafka will take care of partition movements for
> >> you.
> >>
> >>
> >> > - How can I increase the replication factor of my brokers at
> runtime
> >> >
> >>
> >> Again, you can use the same tool and increase the number of brokers
> >> assigned to each partition (
> >>
> >>
> https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor
> >> )
> >>
> >>
> >> > - How can I change the zookeeper IPs present in Kafka Broker Config
> >> at
> >> > runtime without restarting Kafka?
> >> >
> >>
> >> This is not a supported operation. Ideally you are supporting your Kafka
> >> cluster with a ZooKeeper ensemble, that is resilient too to some
> failures
> >> and maintenance outages.
> >>
> >>
> >> > - How can I dynamically change the Kafka Configuration at runtime
> >> >
> >>
> >> Thanks to KIP-226 (
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration
> >> )
> >> there are some broker configurations that can be modified without a
> broker
> >> restart. Details are in that document.
> >>
> >>
> >> > - Regarding Kafka Client:
> >> > - Do I need to specify all Kafka broker IP to kafkaClient for
> >> > connection?
> >> >
> >>
> >> You need to provide enough broker IPs that guarantees the client can
> >> connect to at least one of them. As long as the client can talk to one
> >> broker it can obtain all the information it needs (by polling the
> >> metadata)
> >> to function.
> >>
> >>
> >> > - And each and every time a broker is added or removed does I
> >> need to
> >> > add or remove my IP in Kafka Client connection String. As it
> will
> >> > always
> >> > require to restart my producer and consumers?
> >> >
> >>
> >> Other than providing a few brokers, a more robust solution is to refresh
> >> the list of available brokers at runtime. A basic approach is querying
> >> ZooKeeper to compile a list of available brokers, to configure the
> client.
> >>
> >>
> >> >
> >> > *Note:*
> >> >
> >> > - Kafka Version: 2.0.0
> >> > - Zookeeper: 3.4.9
> >> > - Broker Size : (2 core, 8 GB RAM) [4GB for Kafka and 4 GB for OS]
> >> >
> >> > Regards,
> >> > Abhimanyu
> >> >
> >>
> >>
> >> --
> >>
> >> Thanks!
> >> --Vahid
> >>
> >
>
--
*Suman*
*OlaCabs*
