Re: Some questions on Kafka on order of messages with mutiple partitions
Thanks Peter.
This is my modified Json file
{
"version":1,
"partitions":[
{"topic":"md","partition":0,"replicas":[1,3,7]},
{"topic":"md","partition":1,"replicas":[2,8,9]},
{"topic":"md","partition":2,"replicas":[7,10,12]},
{"topic":"md","partition":3,"replicas":[1,12,9]},
{"topic":"md","partition":4,"replicas":[7,9,11]},
{"topic":"md","partition":5,"replicas":[3,11,1]},
{"topic":"md","partition":6,"replicas":[10,1,7]},
{"topic":"md","partition":7,"replicas":[8,7,3]},
{"topic":"md","partition":8,"replicas":[2,3,10]}
]
}
Running below
kafka-reassign-partitions.sh --bootstrap-server rhes75:9092
--reassignment-json-file ./reduce_replication_factor2.json --execute
and
kafka-topics.sh --describe --bootstrap-server rhes75:9092 --topic md
I get
Topic: md TopicId: UfQly87bQPCbVKoH-PQheg PartitionCount: 9
ReplicationFactor: 3Configs: segment.bytes=1073741824,retention.ms
=1000,retention.bytes=1073741824
Topic: md Partition: 0Leader: 1 Replicas: 1,3,7
Isr: 1,3,7
Topic: md Partition: 1Leader: 2 Replicas: 2,8,9
Isr: 2,8,9
Topic: md Partition: 2Leader: 7 Replicas: 7,10,12
Isr: 10,7,12
Topic: md Partition: 3Leader: 1 Replicas: 1,12,9
Isr: 1,9,12
Topic: md Partition: 4Leader: 7 Replicas: 7,9,11
Isr: 9,7,11
Topic: md Partition: 5Leader: 3 Replicas: 3,11,1
Isr: 1,3,11
Topic: md Partition: 6Leader: 10 Replicas: 10,1,7
Isr: 10,1,7
Topic: md Partition: 7Leader: 8 Replicas: 8,7,3
Isr: 7,3,8
Topic: md Partition: 8Leader: 2 Replicas: 2,3,10
Isr: 10,2,3
I trust this is correct?
Thanks
view my Linkedin profile
Re: Some questions on Kafka on order of messages with mutiple partitions
It looks like you successfully removed replicas from partitions 0, 1, and 2,
but partitons 3 - 8 still show 9 replicas. You probably intended to remove them
from all 9 partitions? You’ll need to create another json file with partitions
3 - 8 to complete the task.
—
Peter
> On May 17, 2023, at 12:41 AM, Mich Talebzadeh
> wrote:
>
> Thanks Miguel, I did that
>
> Based on the following Json file
>
> {
> "version":1,
> "partitions":[
> {"topic":"md","partition":0,"replicas":[1,3,7]},
> {"topic":"md","partition":1,"replicas":[2,8,9]},
> {"topic":"md","partition":2,"replicas":[7,10,12]}
> ]
> }
>
> I ran this command
>
> kafka-reassign-partitions.sh --bootstrap-server rhes75:9092
> --reassignment-json-file ./reduce_replication_factor2.json --execute
>
> Current partition replica assignment
> {"version":1,"partitions":[{"topic":"md","partition":0,"replicas":[1,3,7],"log_dirs":["any","any","any"]},{"topic":"md","partition":1,"replicas":[2,8,9],"log_dirs":["any","any","any"]},{"topic":"md","partition":2,"replicas":[7,10,12],"log_dirs":["any","any","any"]}]}
> Save this to use as the --reassignment-json-file option during rollback
> Successfully started partition reassignments for md-0,md-1,md-2
>
> kafka-topics.sh --describe --bootstrap-server rhes75:9092 --topic md
>
> Topic: md TopicId: UfQly87bQPCbVKoH-PQheg PartitionCount: 9
> ReplicationFactor: 3Configs: segment.bytes=1073741824,retention.ms
> =1000,retention.bytes=1073741824
>Topic: md Partition: 0Leader: 1 Replicas: 1,3,7
> Isr: 1,3,7
>Topic: md Partition: 1Leader: 2 Replicas: 2,8,9
> Isr: 2,8,9
>Topic: md Partition: 2Leader: 7 Replicas: 7,10,12
>Isr: 10,7,12
>Topic: md Partition: 3Leader: 1 Replicas:
> 1,12,9,11,7,3,10,8,2 Isr: 10,1,9,2,12,7,3,11,8
>Topic: md Partition: 4Leader: 7 Replicas:
> 7,9,11,1,3,10,8,2,12 Isr: 10,1,9,2,12,7,3,11,8
>Topic: md Partition: 5Leader: 3 Replicas:
> 3,11,1,7,10,8,2,12,9 Isr: 10,1,9,2,12,7,3,11,8
>Topic: md Partition: 6Leader: 10 Replicas:
> 10,1,7,3,8,2,12,9,11 Isr: 10,1,9,2,12,7,3,11,8
>Topic: md Partition: 7Leader: 8 Replicas:
> 8,7,3,10,2,12,9,11,1 Isr: 10,1,9,2,12,7,3,11,8
>Topic: md Partition: 8Leader: 2 Replicas:
> 2,3,10,8,12,9,11,1,7 Isr: 10,1,9,2,12,7,3,11,8
>
>
> Mich
>
>
> view my Linkedin profile
>
Re: Some questions on Kafka on order of messages with mutiple partitions
Thanks Miguel, I did that
Based on the following Json file
{
"version":1,
"partitions":[
{"topic":"md","partition":0,"replicas":[1,3,7]},
{"topic":"md","partition":1,"replicas":[2,8,9]},
{"topic":"md","partition":2,"replicas":[7,10,12]}
]
}
I ran this command
kafka-reassign-partitions.sh --bootstrap-server rhes75:9092
--reassignment-json-file ./reduce_replication_factor2.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"md","partition":0,"replicas":[1,3,7],"log_dirs":["any","any","any"]},{"topic":"md","partition":1,"replicas":[2,8,9],"log_dirs":["any","any","any"]},{"topic":"md","partition":2,"replicas":[7,10,12],"log_dirs":["any","any","any"]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for md-0,md-1,md-2
kafka-topics.sh --describe --bootstrap-server rhes75:9092 --topic md
Topic: md TopicId: UfQly87bQPCbVKoH-PQheg PartitionCount: 9
ReplicationFactor: 3Configs: segment.bytes=1073741824,retention.ms
=1000,retention.bytes=1073741824
Topic: md Partition: 0Leader: 1 Replicas: 1,3,7
Isr: 1,3,7
Topic: md Partition: 1Leader: 2 Replicas: 2,8,9
Isr: 2,8,9
Topic: md Partition: 2Leader: 7 Replicas: 7,10,12
Isr: 10,7,12
Topic: md Partition: 3Leader: 1 Replicas:
1,12,9,11,7,3,10,8,2 Isr: 10,1,9,2,12,7,3,11,8
Topic: md Partition: 4Leader: 7 Replicas:
7,9,11,1,3,10,8,2,12 Isr: 10,1,9,2,12,7,3,11,8
Topic: md Partition: 5Leader: 3 Replicas:
3,11,1,7,10,8,2,12,9 Isr: 10,1,9,2,12,7,3,11,8
Topic: md Partition: 6Leader: 10 Replicas:
10,1,7,3,8,2,12,9,11 Isr: 10,1,9,2,12,7,3,11,8
Topic: md Partition: 7Leader: 8 Replicas:
8,7,3,10,2,12,9,11,1 Isr: 10,1,9,2,12,7,3,11,8
Topic: md Partition: 8Leader: 2 Replicas:
2,3,10,8,12,9,11,1,7 Isr: 10,1,9,2,12,7,3,11,8
Mich
view my Linkedin profile
Re: Some questions on Kafka on order of messages with mutiple partitions
Hi Mich,
You can use the script `kafka-reassign-partitions.sh` to re-locate or
change the number of replicas
Regards
Miguel
El mar, 16 may 2023 a las 18:44, Mich Talebzadeh ()
escribió:
> Thanks Peter. I meant reduce replication from 9 to 3 and Not partitions.
> Apologies for any confusion
>
>
> Cheers
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Tue, 16 May 2023 at 17:38, Peter Bukowinski wrote:
>
> > Mich,
> >
> > It is not possible to reduce the number of partitions for a kafka topic
> > without deleting and recreating the topic. What previous responders to
> your
> > inquiry noted is that your topic replication of 9 is high. What you want
> to
> > do is reduce your replication, not the partitions. You can do this using
> > the same json file you had the first time, with all 9 partitions. Just
> > remove 6 of the 9 broker ids from the replicas array, e.g.
> >
> > cat reduce_replication_factor.json
> > {
> > "version":1,
> > "partitions":[
> > {"topic":"md","partition":0,"replicas":[12,10,8]},
> > {"topic":"md","partition":1,"replicas":[9,8,2]},
> > {"topic":"md","partition":2,"replicas":[11,2,12]},
> > {"topic":"md","partition":3,"replicas":[1,12,9]},
> > {"topic":"md","partition":4,"replicas":[7,9,11]},
> > {"topic":"md","partition":5,"replicas":[3,11,1]}
> > ]
> > }
> >
> > You may want to adjust where the replicas sit to achieve a better balance
> > across the cluster, but this arrangement only truncates the last 6
> replicas
> > from the list, so should complete quickly as no replica data would move,
> > only be deleted.
> >
> > —
> > Peter Bukowinski
> >
> >
> >
> > > On May 12, 2023, at 1:24 PM, Mich Talebzadeh <
> mich.talebza...@gmail.com>
> > wrote:
> > >
> > > My bad. Only need 3 partitions
> > >
> > > {
> > > "version":1,
> > > "partitions":[
> > > {"topic":"md","partition":0,"replicas":[12,10,8,2,9,11,1,7,3]},
> > > {"topic":"md","partition":1,"replicas":[9,8,2,12,11,1,7,3,10]},
> > > {"topic":"md","partition":2,"replicas":[11,2,12,9,1,7,3,10,8]}
> > > ]
> > > }
> > >
> > > Mich Talebzadeh,
> > > Lead Solutions Architect/Engineering Lead
> > > Palantir Technologies Limited
> > > London
> > > United Kingdom
> > >
> > >
> > > view my Linkedin profile
> > >
Re: Some questions on Kafka on order of messages with mutiple partitions
Thanks Peter. I meant reduce replication from 9 to 3 and Not partitions.
Apologies for any confusion
Cheers
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Tue, 16 May 2023 at 17:38, Peter Bukowinski wrote:
> Mich,
>
> It is not possible to reduce the number of partitions for a kafka topic
> without deleting and recreating the topic. What previous responders to your
> inquiry noted is that your topic replication of 9 is high. What you want to
> do is reduce your replication, not the partitions. You can do this using
> the same json file you had the first time, with all 9 partitions. Just
> remove 6 of the 9 broker ids from the replicas array, e.g.
>
> cat reduce_replication_factor.json
> {
> "version":1,
> "partitions":[
> {"topic":"md","partition":0,"replicas":[12,10,8]},
> {"topic":"md","partition":1,"replicas":[9,8,2]},
> {"topic":"md","partition":2,"replicas":[11,2,12]},
> {"topic":"md","partition":3,"replicas":[1,12,9]},
> {"topic":"md","partition":4,"replicas":[7,9,11]},
> {"topic":"md","partition":5,"replicas":[3,11,1]}
> ]
> }
>
> You may want to adjust where the replicas sit to achieve a better balance
> across the cluster, but this arrangement only truncates the last 6 replicas
> from the list, so should complete quickly as no replica data would move,
> only be deleted.
>
> —
> Peter Bukowinski
>
>
>
> > On May 12, 2023, at 1:24 PM, Mich Talebzadeh
> wrote:
> >
> > My bad. Only need 3 partitions
> >
> > {
> > "version":1,
> > "partitions":[
> > {"topic":"md","partition":0,"replicas":[12,10,8,2,9,11,1,7,3]},
> > {"topic":"md","partition":1,"replicas":[9,8,2,12,11,1,7,3,10]},
> > {"topic":"md","partition":2,"replicas":[11,2,12,9,1,7,3,10,8]}
> > ]
> > }
> >
> > Mich Talebzadeh,
> > Lead Solutions Architect/Engineering Lead
> > Palantir Technologies Limited
> > London
> > United Kingdom
> >
> >
> > view my Linkedin profile
> >
Re: Some questions on Kafka on order of messages with mutiple partitions
Mich,
It is not possible to reduce the number of partitions for a kafka topic without
deleting and recreating the topic. What previous responders to your inquiry
noted is that your topic replication of 9 is high. What you want to do is
reduce your replication, not the partitions. You can do this using the same
json file you had the first time, with all 9 partitions. Just remove 6 of the 9
broker ids from the replicas array, e.g.
cat reduce_replication_factor.json
{
"version":1,
"partitions":[
{"topic":"md","partition":0,"replicas":[12,10,8]},
{"topic":"md","partition":1,"replicas":[9,8,2]},
{"topic":"md","partition":2,"replicas":[11,2,12]},
{"topic":"md","partition":3,"replicas":[1,12,9]},
{"topic":"md","partition":4,"replicas":[7,9,11]},
{"topic":"md","partition":5,"replicas":[3,11,1]}
]
}
You may want to adjust where the replicas sit to achieve a better balance
across the cluster, but this arrangement only truncates the last 6 replicas
from the list, so should complete quickly as no replica data would move, only
be deleted.
—
Peter Bukowinski
> On May 12, 2023, at 1:24 PM, Mich Talebzadeh
> wrote:
>
> My bad. Only need 3 partitions
>
> {
> "version":1,
> "partitions":[
> {"topic":"md","partition":0,"replicas":[12,10,8,2,9,11,1,7,3]},
> {"topic":"md","partition":1,"replicas":[9,8,2,12,11,1,7,3,10]},
> {"topic":"md","partition":2,"replicas":[11,2,12,9,1,7,3,10,8]}
> ]
> }
>
> Mich Talebzadeh,
> Lead Solutions Architect/Engineering Lead
> Palantir Technologies Limited
> London
> United Kingdom
>
>
> view my Linkedin profile
>
Re: Some questions on Kafka on order of messages with mutiple partitions
zookeeper not starting, what could be the problem see screen shots below [image: image.png] [image: image.png] Please Help! On Fri, May 12, 2023 at 1:47 PM Matthias J. Sax wrote: > This email was sent from an external source so please treat with caution. > > > Does having 9 partitions with 9 replication factors make sense here? > > A replication factor of 9 sounds very high. For production, replication > factor of 3 is recommended. > > How many partitions you want/need is a different question, and cannot be > answered in a general way. > > > "Yes" to all other questions. > > > -Matthias > > > > On 5/12/23 9:50 AM, Mich Talebzadeh wrote: > > Hi, > > > > I have used Apache Kafka in conjunction with Spark as a messaging > > source. This rather dated diagram describes it > > > > I have two physical hosts each 64 GB, running RHES 7.6, these are called > > rhes75 and rhes76 respectively. The Zookeeper version is 3.7.1 and kafka > > version is 3.4.0 > > > > > > image.png > > I have a topic md -> MarketData that has been defined as below > > > > kafka-topics.sh --create --bootstrap-server > > > rhes75:9092,rhes75:9093,rhes75:9094,rhes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096, > rhes76:9097 --replication-factor 9 --partitions 9 --topic md > > > > kafka-topics.sh --describe --bootstrap-server > > > rhes75:9092,rhes75:9093,rhes75:9094,rhes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096, > rhes76:9097 --topic md > > > > > > This is working fine > > > > Topic: md TopicId: UfQly87bQPCbVKoH-PQheg PartitionCount: 9 > > ReplicationFactor: 9Configs: segment.bytes=1073741824 > > Topic: md Partition: 0Leader: 12 Replicas: > > 12,10,8,2,9,11,1,7,3 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 1Leader: 9 Replicas: > > 9,8,2,12,11,1,7,3,10 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 2Leader: 11 Replicas: > > 11,2,12,9,1,7,3,10,8 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 3Leader: 1 Replicas: > > 1,12,9,11,7,3,10,8,2 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 4Leader: 7 Replicas: > > 7,9,11,1,3,10,8,2,12 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 5Leader: 3 Replicas: > > 3,11,1,7,10,8,2,12,9 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 6Leader: 10 Replicas: > > 10,1,7,3,8,2,12,9,11 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 7Leader: 8 Replicas: > > 8,7,3,10,2,12,9,11,1 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 8Leader: 2 Replicas: > > 2,3,10,8,12,9,11,1,7 Isr: 10,1,9,2,12,7,3,11,8 > > > > However, I have a number of questions > > > > 1. Does having 9 partitions with 9 replication factors make sense here? > > 2. As I understand the parallelism is equal to the number of partitions > > for a topic. > > 3. Kafka only provides a total order over messages *within a > > partition*, not between different partitions in a topic and in > > this case I have one topic > > 4. > > > > Data within a Partition will be stored in the order in which it is > > written, therefore, data read from a Partition will be read in order > > for that partition? > > > > 5. > > > > Finally if I want to get messages in order across multiple all 9 > > partitionss, then I need to group messages with a key, so that > > messages with the samekey goto the samepartition and withinthat > > partition the messages are ordered > > > > Thanks > > > > > > *Disclaimer:* Use it at your own risk.Any and all responsibility for any > > loss, damage or destruction of data or any other property which may > > arise from relying on this email's technical content is explicitly > > disclaimed. The author will in no case be liable for any monetary > > damages arising from such loss, damage or destruction. > > > > About Ascential plc Ascential delivers specialist information, analytics and ecommerce optimisation platforms to the world's leading consumer brands and their ecosystems. Our world-class businesses improve performance and solve problems for our customers by delivering immediately actionable information combined with visionary longer-term thinking across Digital Commerce, Product Design and Marketing. We also serve customers across Retail & Financial Services. With over 3,000 employees across five continents, we combine local expertise with a global footprint for clients in over 120 countries. Ascential is listed on the London Stock Exchange. The information in or attached to this email is confidential and may be legally privileged. If you are not the intended recipient of this message, any use, disclosure, copying, distribution or any action taken in reliance on it is prohibited and may be unlawful. If you have received this message in error, please notify the sender immediately by r
Re: Some questions on Kafka on order of messages with mutiple partitions
My bad. Only need 3 partitions
{
"version":1,
"partitions":[
{"topic":"md","partition":0,"replicas":[12,10,8,2,9,11,1,7,3]},
{"topic":"md","partition":1,"replicas":[9,8,2,12,11,1,7,3,10]},
{"topic":"md","partition":2,"replicas":[11,2,12,9,1,7,3,10,8]}
]
}
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
Re: Some questions on Kafka on order of messages with mutiple partitions
This json file seemed to work
cat reduce_replication_factor.json
{
"version":1,
"partitions":[
{"topic":"md","partition":0,"replicas":[12,10,8,2,9,11,1,7,3]},
{"topic":"md","partition":1,"replicas":[9,8,2,12,11,1,7,3,10]},
{"topic":"md","partition":2,"replicas":[11,2,12,9,1,7,3,10,8]},
{"topic":"md","partition":3,"replicas":[1,12,9,11,7,3,10,8,2]},
{"topic":"md","partition":4,"replicas":[7,9,11,1,3,10,8,2,12]},
{"topic":"md","partition":5,"replicas":[3,11,1,7,10,8,2,12,9]}
]
}
kafka-reassign-partitions.sh --bootstrap-server
rhes75:9092,rhes75:9093,rhes75:9094,rhes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096,
rhes76:9097 --reassignment-json-file ./reduce_replication_factor.json
--execute
The output
Successfully started partition reassignments for
md-0,md-1,md-2,md-3,md-4,md-5
I guess it is going to take sometime before it is completed.
Thanks
On Fri, 12 May 2023 at 20:16, Mich Talebzadeh
wrote:
> Thanks Matthias.
>
> with regard to your point below:
>
> A replication factor of 9 sounds very high. For production, a replication
> factor of 3 is recommended.
>
> Is it possible to dynamically reduce this number to 3 when the topic is
> actively consumed)?
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Fri, 12 May 2023 at 19:38, Matthias J. Sax wrote:
>
>> > Does having 9 partitions with 9 replication factors make sense here?
>>
>> A replication factor of 9 sounds very high. For production, replication
>> factor of 3 is recommended.
>>
>> How many partitions you want/need is a different question, and cannot be
>> answered in a general way.
>>
>>
>> "Yes" to all other questions.
>>
>>
>> -Matthias
>>
>>
>>
>> On 5/12/23 9:50 AM, Mich Talebzadeh wrote:
>> > Hi,
>> >
>> > I have used Apache Kafka in conjunction with Spark as a messaging
>> > source. This rather dated diagram describes it
>> >
>> > I have two physical hosts each 64 GB, running RHES 7.6, these are
>> called
>> > rhes75 and rhes76 respectively. The Zookeeper version is 3.7.1 and
>> kafka
>> > version is 3.4.0
>> >
>> >
>> > image.png
>> > I have a topic md -> MarketData that has been defined as below
>> >
>> > kafka-topics.sh --create --bootstrap-server
>> >
>> rhes75:9092,rhes75:9093,rhes75:9094,rhes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096,
>> rhes76:9097 --replication-factor 9 --partitions 9 --topic md
>> >
>> > kafka-topics.sh --describe --bootstrap-server
>> >
>> rhes75:9092,rhes75:9093,rhes75:9094,rhes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096,
>> rhes76:9097 --topic md
>> >
>> >
>> > This is working fine
>> >
>> > Topic: md TopicId: UfQly87bQPCbVKoH-PQheg PartitionCount: 9
>> > ReplicationFactor: 9Configs: segment.bytes=1073741824
>> > Topic: md Partition: 0Leader: 12 Replicas:
>> > 12,10,8,2,9,11,1,7,3 Isr: 10,1,9,2,12,7,3,11,8
>> > Topic: md Partition: 1Leader: 9 Replicas:
>> > 9,8,2,12,11,1,7,3,10 Isr: 10,1,9,2,12,7,3,11,8
>> > Topic: md Partition: 2Leader: 11 Replicas:
>> > 11,2,12,9,1,7,3,10,8 Isr: 10,1,9,2,12,7,3,11,8
>> > Topic: md Partition: 3Leader: 1 Replicas:
>> > 1,12,9,11,7,3,10,8,2 Isr: 10,1,9,2,12,7,3,11,8
>> > Topic: md Partition: 4Leader: 7 Replicas:
>> > 7,9,11,1,3,10,8,2,12 Isr: 10,1,9,2,12,7,3,11,8
>> > Topic: md Partition: 5Leader: 3 Replicas:
>> > 3,11,1,7,10,8,2,12,9 Isr: 10,1,9,2,12,7,3,11,8
>> > Topic: md Partition: 6Leader: 10 Replicas:
>> > 10,1,7,3,8,2,12,9,11 Isr: 10,1,9,2,12,7,3,11,8
>> > Topic: md Partition: 7Leader: 8 Replicas:
>> > 8,7,3,10,2,12,9,11,1 Isr: 10,1,9,2,12,7,3,11,8
>> > Topic: md Partition: 8Leader: 2 Replicas:
>> > 2,3,10,8,12,9,11,1,7 Isr: 10,1,9,2,12,7,3,11,8
>> >
>> > However, I have a number of questions
>> >
>> > 1. Does having 9 partitions with 9 replication factors make sense
>> here?
>> > 2. As I understand the parallelism is equal to the number of partitions
>> > for a topic.
>> > 3. Kafka only provides a total order over messages *within a
>> > partition*, not between different partitions in a topic and in
>> > this case I have one topic
>> > 4.
>> >
>> > Data within a Partition will be stored in the order in which it is
>> > written, therefore, data read from a Partition will be read in order
>> > for that partition?
>> >
>> > 5.
>> >
>> > Finally if I want to get messages in order across multiple all 9
>> > partitionss, then I need to group messages with a key, so that
>> > messages with the samekey goto the samepartition and withinthat
>> > par
Re: Some questions on Kafka on order of messages with mutiple partitions
Thanks Matthias. with regard to your point below: A replication factor of 9 sounds very high. For production, a replication factor of 3 is recommended. Is it possible to dynamically reduce this number to 3 when the topic is actively consumed)? *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 12 May 2023 at 19:38, Matthias J. Sax wrote: > > Does having 9 partitions with 9 replication factors make sense here? > > A replication factor of 9 sounds very high. For production, replication > factor of 3 is recommended. > > How many partitions you want/need is a different question, and cannot be > answered in a general way. > > > "Yes" to all other questions. > > > -Matthias > > > > On 5/12/23 9:50 AM, Mich Talebzadeh wrote: > > Hi, > > > > I have used Apache Kafka in conjunction with Spark as a messaging > > source. This rather dated diagram describes it > > > > I have two physical hosts each 64 GB, running RHES 7.6, these are called > > rhes75 and rhes76 respectively. The Zookeeper version is 3.7.1 and kafka > > version is 3.4.0 > > > > > > image.png > > I have a topic md -> MarketData that has been defined as below > > > > kafka-topics.sh --create --bootstrap-server > > > rhes75:9092,rhes75:9093,rhes75:9094,rhes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096, > rhes76:9097 --replication-factor 9 --partitions 9 --topic md > > > > kafka-topics.sh --describe --bootstrap-server > > > rhes75:9092,rhes75:9093,rhes75:9094,rhes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096, > rhes76:9097 --topic md > > > > > > This is working fine > > > > Topic: md TopicId: UfQly87bQPCbVKoH-PQheg PartitionCount: 9 > > ReplicationFactor: 9Configs: segment.bytes=1073741824 > > Topic: md Partition: 0Leader: 12 Replicas: > > 12,10,8,2,9,11,1,7,3 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 1Leader: 9 Replicas: > > 9,8,2,12,11,1,7,3,10 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 2Leader: 11 Replicas: > > 11,2,12,9,1,7,3,10,8 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 3Leader: 1 Replicas: > > 1,12,9,11,7,3,10,8,2 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 4Leader: 7 Replicas: > > 7,9,11,1,3,10,8,2,12 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 5Leader: 3 Replicas: > > 3,11,1,7,10,8,2,12,9 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 6Leader: 10 Replicas: > > 10,1,7,3,8,2,12,9,11 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 7Leader: 8 Replicas: > > 8,7,3,10,2,12,9,11,1 Isr: 10,1,9,2,12,7,3,11,8 > > Topic: md Partition: 8Leader: 2 Replicas: > > 2,3,10,8,12,9,11,1,7 Isr: 10,1,9,2,12,7,3,11,8 > > > > However, I have a number of questions > > > > 1. Does having 9 partitions with 9 replication factors make sense here? > > 2. As I understand the parallelism is equal to the number of partitions > > for a topic. > > 3. Kafka only provides a total order over messages *within a > > partition*, not between different partitions in a topic and in > > this case I have one topic > > 4. > > > > Data within a Partition will be stored in the order in which it is > > written, therefore, data read from a Partition will be read in order > > for that partition? > > > > 5. > > > > Finally if I want to get messages in order across multiple all 9 > > partitionss, then I need to group messages with a key, so that > > messages with the samekey goto the samepartition and withinthat > > partition the messages are ordered > > > > Thanks > > > > > > *Disclaimer:* Use it at your own risk.Any and all responsibility for any > > loss, damage or destruction of data or any other property which may > > arise from relying on this email's technical content is explicitly > > disclaimed. The author will in no case be liable for any monetary > > damages arising from such loss, damage or destruction. > > >
Re: Some questions on Kafka on order of messages with mutiple partitions
Does having 9 partitions with 9 replication factors make sense here? A replication factor of 9 sounds very high. For production, replication factor of 3 is recommended. How many partitions you want/need is a different question, and cannot be answered in a general way. "Yes" to all other questions. -Matthias On 5/12/23 9:50 AM, Mich Talebzadeh wrote: Hi, I have used Apache Kafka in conjunction with Spark as a messaging source. This rather dated diagram describes it I have two physical hosts each 64 GB, running RHES 7.6, these are called rhes75 and rhes76 respectively. The Zookeeper version is 3.7.1 and kafka version is 3.4.0 image.png I have a topic md -> MarketData that has been defined as below kafka-topics.sh --create --bootstrap-server rhes75:9092,rhes75:9093,rhes75:9094,rhes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096, rhes76:9097 --replication-factor 9 --partitions 9 --topic md kafka-topics.sh --describe --bootstrap-server rhes75:9092,rhes75:9093,rhes75:9094,rhes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096, rhes76:9097 --topic md This is working fine Topic: md TopicId: UfQly87bQPCbVKoH-PQheg PartitionCount: 9 ReplicationFactor: 9 Configs: segment.bytes=1073741824 Topic: md Partition: 0 Leader: 12 Replicas: 12,10,8,2,9,11,1,7,3 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 1 Leader: 9 Replicas: 9,8,2,12,11,1,7,3,10 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 2 Leader: 11 Replicas: 11,2,12,9,1,7,3,10,8 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 3 Leader: 1 Replicas: 1,12,9,11,7,3,10,8,2 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 4 Leader: 7 Replicas: 7,9,11,1,3,10,8,2,12 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 5 Leader: 3 Replicas: 3,11,1,7,10,8,2,12,9 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 6 Leader: 10 Replicas: 10,1,7,3,8,2,12,9,11 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 7 Leader: 8 Replicas: 8,7,3,10,2,12,9,11,1 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 8 Leader: 2 Replicas: 2,3,10,8,12,9,11,1,7 Isr: 10,1,9,2,12,7,3,11,8 However, I have a number of questions 1. Does having 9 partitions with 9 replication factors make sense here? 2. As I understand the parallelism is equal to the number of partitions for a topic. 3. Kafka only provides a total order over messages *within a partition*, not between different partitions in a topic and in this case I have one topic 4. Data within a Partition will be stored in the order in which it is written, therefore, data read from a Partition will be read in order for that partition? 5. Finally if I want to get messages in order across multiple all 9 partitionss, then I need to group messages with a key, so that messages with the samekey goto the samepartition and withinthat partition the messages are ordered Thanks *Disclaimer:* Use it at your own risk.Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
Some questions on Kafka on order of messages with mutiple partitions
Hi, I have used Apache Kafka in conjunction with Spark as a messaging source. This rather dated diagram describes it I have two physical hosts each 64 GB, running RHES 7.6, these are called rhes75 and rhes76 respectively. The Zookeeper version is 3.7.1 and kafka version is 3.4.0 [image: image.png] I have a topic md -> MarketData that has been defined as below kafka-topics.sh --create --bootstrap-server rhes75:9092,rhes75:9093,rhes75:9094,rhes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096, rhes76:9097 --replication-factor 9 --partitions 9 --topic md kafka-topics.sh --describe --bootstrap-server rhes75:9092,rhes75:9093,rhes75:9094,rhes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096, rhes76:9097 --topic md This is working fine Topic: md TopicId: UfQly87bQPCbVKoH-PQheg PartitionCount: 9 ReplicationFactor: 9Configs: segment.bytes=1073741824 Topic: md Partition: 0Leader: 12 Replicas: 12,10,8,2,9,11,1,7,3 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 1Leader: 9 Replicas: 9,8,2,12,11,1,7,3,10 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 2Leader: 11 Replicas: 11,2,12,9,1,7,3,10,8 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 3Leader: 1 Replicas: 1,12,9,11,7,3,10,8,2 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 4Leader: 7 Replicas: 7,9,11,1,3,10,8,2,12 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 5Leader: 3 Replicas: 3,11,1,7,10,8,2,12,9 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 6Leader: 10 Replicas: 10,1,7,3,8,2,12,9,11 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 7Leader: 8 Replicas: 8,7,3,10,2,12,9,11,1 Isr: 10,1,9,2,12,7,3,11,8 Topic: md Partition: 8Leader: 2 Replicas: 2,3,10,8,12,9,11,1,7 Isr: 10,1,9,2,12,7,3,11,8 However, I have a number of questions 1. Does having 9 partitions with 9 replication factors make sense here? 2. As I understand the parallelism is equal to the number of partitions for a topic. 3. Kafka only provides a total order over messages *within a partition*, not between different partitions in a topic and in this case I have one topic 4. Data within a Partition will be stored in the order in which it is written, therefore, data read from a Partition will be read in order for that partition? 5. Finally if I want to get messages in order across multiple all 9 partitionss, then I need to group messages with a key, so that messages with the same key go to the same partition and within that partition the messages are ordered Thanks *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
