sergey-chugunov-1985 commented on code in PR #13714:
URL: https://github.com/apache/ignite/pull/13714#discussion_r4349875054
##########
docs/_docs/data-rebalancing.adoc:
##########
@@ -28,6 +28,83 @@ In pure in-memory clusters, the default behavior is to start
rebalancing immedia
In clusters with persistence, the baseline topology has to be changed manually
(default behavior), or can be changed automatically when
link:clustering/baseline-topology#baseline-topology-autoadjustment[automatic
baseline adjustment] is enabled.
====
+== Rebalance Strategy [[rebalance-strategy]]
+
+When a node joins or rejoins the cluster, each partition is restored using one
of two strategies:
+
+- *Full rebalance* — the node downloads the entire partition content from
another server node that holds a copy of the partition.
+- *Historical rebalance* — the node restores its own data from the local disk
and downloads only the delta (WAL entries written while it was absent) from the
source node that holds a copy of the partition.
+
+The strategy is selected per partition based on the following rules:
+
+. A full rebalance is used if the node has no local data for the partition
(update counter is zero).
++
+This applies both to a *new node joining for the first time* (it has no data
on disk at all) and to a node that was *evicted from the baseline* and rejoined
(its local partition data was invalidated). In both cases the full partition
content must be downloaded from another node.
+
+. A full rebalance is used if the delta exceeds the total number of entries in
the partition — transferring the full content is more efficient in that case.
++
+Example: a partition contains 100 entries. While the node was absent, 500
writes hit this partition. Downloading 500 delta entries is more expensive than
just downloading all 100 entries → full rebalance.
+
+. A full rebalance is used if the partition size is below the
`historical.rebalance.threshold` (DMS property, default 500 entries), unless
the `IGNITE_PREFER_WAL_REBALANCE` system property is set to `true`.
++
+Example: a partition contains 10 entries. While the node was absent, 3 writes
hit this partition. The delta (3) is smaller than the partition (10), but 10 <
500 — the partition is too small to justify WAL replay overhead → full
rebalance. With `IGNITE_PREFER_WAL_REBALANCE=true` this threshold check is
bypassed, and a historical rebalance is used regardless of the partition size
(provided the other rules above are not triggered).
+
+. A historical rebalance is used when all of the above conditions are not met:
the node has local data, the delta is smaller than the partition, and the
partition exceeds the threshold.
++
+Example: a partition contains 10,000 entries. While the node was absent, 30
writes hit this partition. Downloading 30 delta entries is far cheaper than all
10,000 → historical rebalance.
+
+[NOTE]
+====
+If both nodes report the same update counter for a partition, no rebalance is
performed for that partition.
Review Comment:
```suggestion
If both nodes report the same update counter for a partition, rebalance
doesn't start for that partition.
```
##########
docs/_docs/data-rebalancing.adoc:
##########
@@ -28,6 +28,83 @@ In pure in-memory clusters, the default behavior is to start
rebalancing immedia
In clusters with persistence, the baseline topology has to be changed manually
(default behavior), or can be changed automatically when
link:clustering/baseline-topology#baseline-topology-autoadjustment[automatic
baseline adjustment] is enabled.
====
+== Rebalance Strategy [[rebalance-strategy]]
+
+When a node joins or rejoins the cluster, each partition is restored using one
of two strategies:
+
+- *Full rebalance* — the node downloads the entire partition content from
another server node that holds a copy of the partition.
+- *Historical rebalance* — the node restores its own data from the local disk
and downloads only the delta (WAL entries written while it was absent) from the
source node that holds a copy of the partition.
+
+The strategy is selected per partition based on the following rules:
+
+. A full rebalance is used if the node has no local data for the partition
(update counter is zero).
++
+This applies both to a *new node joining for the first time* (it has no data
on disk at all) and to a node that was *evicted from the baseline* and rejoined
(its local partition data was invalidated). In both cases the full partition
content must be downloaded from another node.
Review Comment:
```suggestion
This applies both to a *new node joining for the first time* (it has no data
on disk at all) and to a node that was *removed from the baseline* and than
added back (its local partition data was invalidated). In both cases the full
partition content must be downloaded from another node.
```
##########
docs/_docs/data-rebalancing.adoc:
##########
@@ -28,6 +28,83 @@ In pure in-memory clusters, the default behavior is to start
rebalancing immedia
In clusters with persistence, the baseline topology has to be changed manually
(default behavior), or can be changed automatically when
link:clustering/baseline-topology#baseline-topology-autoadjustment[automatic
baseline adjustment] is enabled.
====
+== Rebalance Strategy [[rebalance-strategy]]
Review Comment:
```suggestion
== Rebalancing Strategy [[rebalance-strategy]]
```
##########
docs/_docs/data-rebalancing.adoc:
##########
@@ -28,6 +28,83 @@ In pure in-memory clusters, the default behavior is to start
rebalancing immedia
In clusters with persistence, the baseline topology has to be changed manually
(default behavior), or can be changed automatically when
link:clustering/baseline-topology#baseline-topology-autoadjustment[automatic
baseline adjustment] is enabled.
====
+== Rebalance Strategy [[rebalance-strategy]]
+
+When a node joins or rejoins the cluster, each partition is restored using one
of two strategies:
+
+- *Full rebalance* — the node downloads the entire partition content from
another server node that holds a copy of the partition.
+- *Historical rebalance* — the node restores its own data from the local disk
and downloads only the delta (WAL entries written while it was absent) from the
source node that holds a copy of the partition.
+
+The strategy is selected per partition based on the following rules:
+
+. A full rebalance is used if the node has no local data for the partition
(update counter is zero).
++
+This applies both to a *new node joining for the first time* (it has no data
on disk at all) and to a node that was *evicted from the baseline* and rejoined
(its local partition data was invalidated). In both cases the full partition
content must be downloaded from another node.
+
+. A full rebalance is used if the delta exceeds the total number of entries in
the partition — transferring the full content is more efficient in that case.
++
+Example: a partition contains 100 entries. While the node was absent, 500
writes hit this partition. Downloading 500 delta entries is more expensive than
just downloading all 100 entries → full rebalance.
+
+. A full rebalance is used if the partition size is below the
`historical.rebalance.threshold` (DMS property, default 500 entries), unless
the `IGNITE_PREFER_WAL_REBALANCE` system property is set to `true`.
Review Comment:
```suggestion
. A full rebalance is used if the partition size is below the
`historical.rebalance.threshold` (a property stored in Distributed MetaStorage,
default 500 entries), unless the `IGNITE_PREFER_WAL_REBALANCE` system property
is set to `true`.
```
##########
modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcRebalancingTest.java:
##########
@@ -91,16 +100,644 @@ public void testFullRebalanceCrossDC() throws Exception {
/** */
@Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
public void testHistoricalRebalanceSameDC() throws Exception {
checkHistoricalRebalance(false);
}
/** */
@Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
public void testHistoricalRebalanceCrossDC() throws Exception {
checkHistoricalRebalance(true);
}
+ /**
+ * Verifies that no rebalance demand is sent when a node restarts without
any
Review Comment:
```suggestion
* Verifies that no rebalance request is sent when a node restarts
without any
```
##########
modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcRebalancingTest.java:
##########
Review Comment:
A lot of duplicated code shared between tests: start few nodes, activate
nodes, create cache config, create cache, populate cache. Is it possible to
extract at least some of these scenarios into separate methods?
##########
docs/_docs/data-rebalancing.adoc:
##########
@@ -28,6 +28,83 @@ In pure in-memory clusters, the default behavior is to start
rebalancing immedia
In clusters with persistence, the baseline topology has to be changed manually
(default behavior), or can be changed automatically when
link:clustering/baseline-topology#baseline-topology-autoadjustment[automatic
baseline adjustment] is enabled.
====
+== Rebalance Strategy [[rebalance-strategy]]
+
+When a node joins or rejoins the cluster, each partition is restored using one
of two strategies:
+
+- *Full rebalance* — the node downloads the entire partition content from
another server node that holds a copy of the partition.
+- *Historical rebalance* — the node restores its own data from the local disk
and downloads only the delta (WAL entries written while it was absent) from the
source node that holds a copy of the partition.
+
+The strategy is selected per partition based on the following rules:
+
+. A full rebalance is used if the node has no local data for the partition
(update counter is zero).
++
+This applies both to a *new node joining for the first time* (it has no data
on disk at all) and to a node that was *evicted from the baseline* and rejoined
(its local partition data was invalidated). In both cases the full partition
content must be downloaded from another node.
+
+. A full rebalance is used if the delta exceeds the total number of entries in
the partition — transferring the full content is more efficient in that case.
++
+Example: a partition contains 100 entries. While the node was absent, 500
writes hit this partition. Downloading 500 delta entries is more expensive than
just downloading all 100 entries → full rebalance.
+
+. A full rebalance is used if the partition size is below the
`historical.rebalance.threshold` (DMS property, default 500 entries), unless
the `IGNITE_PREFER_WAL_REBALANCE` system property is set to `true`.
++
+Example: a partition contains 10 entries. While the node was absent, 3 writes
hit this partition. The delta (3) is smaller than the partition (10), but 10 <
500 — the partition is too small to justify WAL replay overhead → full
rebalance. With `IGNITE_PREFER_WAL_REBALANCE=true` this threshold check is
bypassed, and a historical rebalance is used regardless of the partition size
(provided the other rules above are not triggered).
+
+. A historical rebalance is used when all of the above conditions are not met:
the node has local data, the delta is smaller than the partition, and the
partition exceeds the threshold.
++
+Example: a partition contains 10,000 entries. While the node was absent, 30
writes hit this partition. Downloading 30 delta entries is far cheaper than all
10,000 → historical rebalance.
+
+[NOTE]
+====
+If both nodes report the same update counter for a partition, no rebalance is
performed for that partition.
+
+A node reports counters only for partitions it has written to. If the dataset
is small relative to the partition count, most partitions will have a zero
counter and will not be included in the rebalance.
+====
+
+[NOTE]
+====
+*Historical rebalance requires the delta to be present in the WAL of the
source node.* If the WAL segments that contain the delta were already archived
or deleted by the time the node rejoins — for example, the node was absent for
a long time, or WAL history and archiving are configured with a small retention
— the source node cannot read the delta. In that case Ignite does not lose
data: the supplier detects the failure to iterate over the WAL, marks the last
checkpoint as not applicable for WAL rebalance, marks the affected partitions
as missed, and the node falls back to a *full rebalance*.
Review Comment:
```suggestion
*Historical rebalance requires the delta to be present in the WAL of the
source node.* If the WAL segments that contain the delta were already archived
or deleted by the time node lagged behind rejoins — for example, the node was
absent for a long time, or WAL history and archiving are configured with a
small retention — the source node cannot supply the delta. In that case Ignite
does not lose data: the supplier detects the failure to iterate over the WAL,
marks the last checkpoint as not applicable for WAL rebalance, marks the
affected partitions as missed, and the node falls back to a *full rebalance*.
```
##########
docs/_docs/data-rebalancing.adoc:
##########
@@ -28,6 +28,83 @@ In pure in-memory clusters, the default behavior is to start
rebalancing immedia
In clusters with persistence, the baseline topology has to be changed manually
(default behavior), or can be changed automatically when
link:clustering/baseline-topology#baseline-topology-autoadjustment[automatic
baseline adjustment] is enabled.
====
+== Rebalance Strategy [[rebalance-strategy]]
+
+When a node joins or rejoins the cluster, each partition is restored using one
of two strategies:
+
+- *Full rebalance* — the node downloads the entire partition content from
another server node that holds a copy of the partition.
+- *Historical rebalance* — the node restores its own data from the local disk
and downloads only the delta (WAL entries written while it was absent) from the
source node that holds a copy of the partition.
+
+The strategy is selected per partition based on the following rules:
+
+. A full rebalance is used if the node has no local data for the partition
(update counter is zero).
++
+This applies both to a *new node joining for the first time* (it has no data
on disk at all) and to a node that was *evicted from the baseline* and rejoined
(its local partition data was invalidated). In both cases the full partition
content must be downloaded from another node.
+
+. A full rebalance is used if the delta exceeds the total number of entries in
the partition — transferring the full content is more efficient in that case.
++
+Example: a partition contains 100 entries. While the node was absent, 500
writes hit this partition. Downloading 500 delta entries is more expensive than
just downloading all 100 entries → full rebalance.
+
+. A full rebalance is used if the partition size is below the
`historical.rebalance.threshold` (DMS property, default 500 entries), unless
the `IGNITE_PREFER_WAL_REBALANCE` system property is set to `true`.
++
+Example: a partition contains 10 entries. While the node was absent, 3 writes
hit this partition. The delta (3) is smaller than the partition (10), but 10 <
500 — the partition is too small to justify WAL replay overhead → full
rebalance. With `IGNITE_PREFER_WAL_REBALANCE=true` this threshold check is
bypassed, and a historical rebalance is used regardless of the partition size
(provided the other rules above are not triggered).
+
+. A historical rebalance is used when all of the above conditions are not met:
the node has local data, the delta is smaller than the partition, and the
partition exceeds the threshold.
++
+Example: a partition contains 10,000 entries. While the node was absent, 30
writes hit this partition. Downloading 30 delta entries is far cheaper than all
10,000 → historical rebalance.
+
+[NOTE]
+====
+If both nodes report the same update counter for a partition, no rebalance is
performed for that partition.
+
+A node reports counters only for partitions it has written to. If the dataset
is small relative to the partition count, most partitions will have a zero
counter and will not be included in the rebalance.
+====
+
+[NOTE]
+====
+*Historical rebalance requires the delta to be present in the WAL of the
source node.* If the WAL segments that contain the delta were already archived
or deleted by the time the node rejoins — for example, the node was absent for
a long time, or WAL history and archiving are configured with a small retention
— the source node cannot read the delta. In that case Ignite does not lose
data: the supplier detects the failure to iterate over the WAL, marks the last
checkpoint as not applicable for WAL rebalance, marks the affected partitions
as missed, and the node falls back to a *full rebalance*.
+====
+
+[CAUTION]
+====
+The `rebalanceDelay` (deprecated) setting also prevents a historical
rebalance: when a delay is configured, the rebalance is scheduled outside of
the partition map exchange context, and the historical path (which is tied to
that context) is never taken — every partition is restored with a full
rebalance regardless of the delta size and regardless of
`IGNITE_PREFER_WAL_REBALANCE`.
+====
+
+In a link:clustering/multi-data-center[multi-data-center] setup, the preloader
prefers a node from the same data center that holds a copy of the partition. If
no such node is available, it falls back to a node from a remote data center.
This behavior applies to both full and historical rebalance.
Review Comment:
```suggestion
In a link:clustering/multi-data-center[multi-data-center] setup, the node
demanding the data prefers a node from the same data center that holds a copy
of the partition. If no such node is available, it falls back to a node from a
remote data center. This behavior applies to both full and historical rebalance.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
