Re: Change Apache Kafka properties dynamically without restarting servers using Archaius

[Tom Crayford]

inline

On Mon, Aug 15, 2016 at 5:49 PM, VIJJU CH  wrote:

> Hello,
>
> I have some questions related to properties of Apache Kafka. We have a four
> node Kafka Cluster which is on Amazon EMR. Currently, in order to change
> any properties we do restart the servers to pick the new changes made. For
> the following:
>
> 1. Can we change the topic name that a producer is writing to
>

Specified in the message by your code, so no problem.


> 2. Can we change the bootstrap server list
>

Not without restarting the producer/consumer objects.


> 3. Can we change the consumer group name
>

Not without restarting the consumer object.


> 4. Can we change the number of consumers etc. and be able to verify and
> measure that the change is picked up and did not result in lost or
> duplicate message nor introduce any consumer lag
>

This is simply the number of client objects you have running. It's under
your control.

Note that "did not result in lost or duplicate message" is not entirely
possible in the presence of failure. You have to pick one and optimize for
that (Kafka's docs often refer to this as "at least once" or "at most once"
messaging).

Thanks

Tom Crayford
Heroku Kafka


>
> Is it possible to change these properties without restarting the servers ?
> I mean can we change these dynamically using Archaius (
> https://github.com/Netflix/archaius/wiki)?
>
> Reply me at your earliest convenience.
>
> Thanks,
> Vijju
>

Change Apache Kafka properties dynamically without restarting servers using Archaius

[VIJJU CH]

Hello,

I have some questions related to properties of Apache Kafka. We have a four
node Kafka Cluster which is on Amazon EMR. Currently, in order to change
any properties we do restart the servers to pick the new changes made. For
the following:

1. Can we change the topic name that a producer is writing to
2. Can we change the bootstrap server list
3. Can we change the consumer group name
4. Can we change the number of consumers etc. and be able to verify and
measure that the change is picked up and did not result in lost or
duplicate message nor introduce any consumer lag

Is it possible to change these properties without restarting the servers ?
I mean can we change these dynamically using Archaius (
https://github.com/Netflix/archaius/wiki)?

Reply me at your earliest convenience.

Thanks,
Vijju