As I mentioned in the implementation of PIP, we will plug-in the partition
assignment strategy.
However, in the same cluster, it is impossible for some Brokers to use
consistent hashing and some Brokers to use round robin.
On 2023/04/11 07:37:19 Xiangying Meng wrote:
> Hi Linlin,
> > This is
Hi Linlin,
> This is an incompatible modification, so the entire cluster needs to be
upgraded, not just a part of the nodes
Appreciate your contribution to the new feature in PIP-255.
I have a question regarding the load-balancing aspect of this feature.
You mentioned that this is an
> Do we expect that the bundles should have the same loads?
Yes.If the bundle load is similar, it will be easier to achieve a balanced
state during subsequent load balancing.
If the bundle loads are different or there are hot bundles, it will trigger
bundle split or unload.
Unload the hotspot
Hi Lin,
> The load managed by each Bundle is not even. Even if the number of
partitions managed
by each bundle is the same, there is no guarantee that the sum of the
loads of these partitions
will be the same.
Do we expect that the bundles should have the same loads? The bundle is the
base
> This appears to be the "round-robin topic-to-bundle mapping" option in
> the `fundBundle` function. Is this the only place that needs an update? Can
> you list what change is required?
In this PIP, we only discuss topic-to-bundle mapping
Change is required:
1)
When lookup, partitions is
The namespace level bundle-unload can be performed in
NamespaceService#splitAndOwnBundleOnceAndRetry
A new judgment will be added here.
After splitting the bundle, it should determine whether to unload at the
namespace level.
On 2023/03/19 09:53:07 lifepuzzlefun wrote:
> I'm interest on the
Hi, I see. I have follow-up questions below.
- This appears to be the "round-robin topic-to-bundle mapping" option in
the `fundBundle` function. Is this the only place that needs an update? Can
you list what change is required?
- How do we enable this "round-robin topic-to-bundle mapping option"
Thanks for joining this discussion
> 1. where is the partition to bundle mapping stored？
We don't need to store the mapping relationship, it can be calculated
dynamically. The first is the starting bundle, partition-0 is calculated
directly through consistent hashing. Subsequent partitions
I'm interest on the implementation details.
1. where is the partition to bundle mapping stored？when upgrade origin logic to
the new round robin logic. how the current code distinguish partition assigned
by origin logic and the new created topic assign by round robin logic.
2. can you explain
Hi, pulsar community. I'm insterest in this topic and want to join the
discussion : - )
I test to create one partitioned topic with 1024 partitions in a namespace with
256 bundle.
the distribution for this topic is post below.
key is the total assigned partition number to the bundle.
value
Thanks for your review.
> Could you clarify the limitation of the current logic?
The current logic cannot guarantee that the traffic of each bundle is the same,
and must be balanced through split.
However, the load of the topic is not always the same, and the traffic of the
business changes
Hi,
Thank you for sharing this.
In general, I think this can be another good option for Pulsar load
assignment logic.
However, I have some comments below.
> The load managed by each Bundle is not even.
Even if the number of partitions managed by each bundle is the same,
there is no guarantee
12 matches
Mail list logo
