*if the consuming of all produced data is not important
On Tue, Oct 29, 2019 at 12:42 PM Daniyar Kulakhmetov <
dkulakhme...@liftoff.io> wrote:
> As a solution, you may have multiple Kafka clusters. For example, 1 main
> Kafka cluster + 7 secondary (mirrored) Kafka clusters.
> The logic would be a
As a solution, you may have multiple Kafka clusters. For example, 1 main
Kafka cluster + 7 secondary (mirrored) Kafka clusters.
The logic would be as follows:
1) Messages are produced to the main Kafka cluster
2) Secondary Kafka clusters consume data from the main Kafka cluster
3) 7K consumers are
I have following Kafka cluster:
- Broker#: 13 (1 Broker : 14 cores & 36GB memory )
- Kafka cluster version: 2.0.0
- Kafka Java client version: 2.0.0
- Number topics: ~15 (3 replica, 2 min replica, 8 partitions)
- Number of consumers: 7K (all independent and manually assigned all
Everything has impact. You cannot keep churning loads of messages under the
same operating condition, and expect nothing to change.
You have know find out (via load testing) an optimum operating condition
(e.g. partition, batch.size etc.) for you producer/consumer to work
correctly. Remember that
I wanted to understand whether broker will be unstable with large number of
consumers or will consume face some issue like lag will increase?
On Mon, Oct 21, 2019 at 6:55 PM Shyam P wrote:
> What are you trying to do here ? whats your objective ?
>
> On Sat, Oct 19, 2019 at 8:45 PM Hrishikesh M
What are you trying to do here ? whats your objective ?
On Sat, Oct 19, 2019 at 8:45 PM Hrishikesh Mishra
wrote:
> Can anyone please help me with this?
>
> On Fri, 18 Oct 2019 at 2:58 PM, Hrishikesh Mishra
> wrote:
>
> > Hi all,
> >
> > I wanted to understand, having large numbers of consumers
Can anyone please help me with this?
On Fri, 18 Oct 2019 at 2:58 PM, Hrishikesh Mishra
wrote:
> Hi all,
>
> I wanted to understand, having large numbers of consumers on
> producer latency and brokers. I have around 7K independent consumers. Each
> consumer is consuming all partitions of a topic.
Hi all,
I wanted to understand, having large numbers of consumers on
producer latency and brokers. I have around 7K independent consumers. Each
consumer is consuming all partitions of a topic. I have manually assigned
partitions of a topic to a consumer, not using consumer groups. Each
consumer is
