devmadhuu commented on PR #10074:
URL: https://github.com/apache/ozone/pull/10074#issuecomment-4325120080
> Thanks a lot @devmadhuu for the detailed explanation and continuous
efforts in handling this.
>
> **General comments:**
>
> 1. IIUC, `Full snapshot` is a costly operation and the cost vary
depends on the SCM database size, I'd suggest `not to trigger automatic
FULL_SNAP `instead analyze the overhead of snapshot datatransfer, snapshot
loading, GC Pressure & potential system resource utilization based on existing
known SCM database size(say if the database size is 275GB and 300M containers).
Then will think about the solution and proposal to the users.
>
> 2. Could you please come up with the set of meaningful metrics to
understand the TARGETED_SYNC. For example, per state based metrics which can be
used by the users/customers to determine the nonOpenDrift acceptable
discrepancies and tune the TARGETED_SYNC interval. Another one can be the time
taken etc.
>
> 3. Can we think of a warning or alert if the decrecepencies grows
beyond certain threshold or limit?
>
>
> Something like, non_open_container_decrecepencies <= 10K and the
TARGETED_SYNC interval is 1hr
>
> ```
> Apr 22nd 00:01 AM : non_open_container_decrecepencies 10K
> Apr 22nd 00:02 AM : non_open_container_decrecepencies 5K
> Apr 22nd 00:03 AM : non_open_container_decrecepencies 15K WARN
> Apr 22nd 00:04 AM : non_open_container_decrecepencies 12K WARN
> Apr 22nd 00:05 AM : non_open_container_decrecepencies 2K
> Apr 22nd 00:06 AM : non_open_container_decrecepencies 2K
> Apr 22nd 00:07 AM : non_open_container_decrecepencies 2K
> ```
Yes, full SCM DB snapshot download is a time consuming with high latency
operation. As we have experienced, some large clusters can even reach with 500+
GB of SCM DB and continue to grow, so it is always good to avoid this full SCM
DB download situation and keep doing `TARGETED_SYNC` at regular intervals with
default interval of 1H which is configurable.
**`How we arrive at every 1 hour check of determining sync action ->
`TARGETED_SYNC` or FULL SCM DB SNAPSHOT download. ?`**
— We arrived based on below math for 100 million containers and impact of
syncing in Recon by making RPC calls from Recon to SCM every 1 hour and why it
is justified.
— In high concurrent clusters with high workload, if we increase the
frequency from 1 hour to 2 hours, we may observe the increase in discrepancy of
containers between SCM and Recon in worst case, so this may force full snapshot
download which we want to avoid.
**`Why we need this `TARGETED_SYNC` every 1H when Recon is already having
logic of processing of DN container reports which is an existing behvaior in
Recon` and why those existing ICR and FCRs are not sufficient for SCM-only
containers.?**
- ICR and periodic FCR do not guarantee Recon will learn about every SCM
container.
- A container can exist in SCM but never appear in DN reports if the client
fails before any replica is created or before DNs persist/report it.
- In that case, Recon can miss an OPEN empty container entirely.
- Later, SCM may keep that container OPEN for some time and eventually move
it toward deletion, while Recon still has no signal from DN-side reporting.
- That is exactly the class of drift this SCM-driven sync is meant to repair.
**# Recon SCM Sync Latency Projections**
## Scope
This note derives projected latency, RPC count, transfer volume, and SCM
heap impact for Recon syncing container IDs from SCM using a batch size of
`500,000` container IDs per RPC.
The math is based on the following observed log sample from Recon:
```text
/var/log/hadoop-ozone/ozone-recon.log:2026-03-18 10:41:40,896 WARN
[pool-53-thread-1]-org.apache.hadoop.ozone.recon.scm.ReconStorageContainerSyncHelper:
BASELINE: Starting sync - totalContainers=123409, batchSize=123409
/var/log/hadoop-ozone/ozone-recon.log:2026-03-18 10:41:41,269 WARN
[pool-53-thread-1]-org.apache.hadoop.ozone.recon.scm.ReconStorageContainerSyncHelper:
BASELINE: Sync complete - totalTime=516ms, rpcCalls=1, avgRpcTime=283ms,
totalRpcTime=283ms, newContainers=0
```
## Code References
The sizing assumption for one serialized `ContainerID` comes from:
-
[ReconStorageContainerSyncHelper.java](/Users/deveshsingh/Downloads/Ozone/workspace/fork/ozone/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerSyncHelper.java:81)
The batch-size logic comes from:
-
[ReconStorageContainerSyncHelper.java](/Users/deveshsingh/Downloads/Ozone/workspace/fork/ozone/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerSyncHelper.java:714)
The SCM RPC serving the state-filtered container ID list comes from:
-
[SCMClientProtocolServer.java](/Users/deveshsingh/Downloads/Ozone/workspace/fork/ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java:1563)
The SCM side state-map list creation comes from:
-
[ContainerStateMap.java](/Users/deveshsingh/Downloads/Ozone/workspace/fork/ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/states/ContainerStateMap.java:270)
The protobuf response building on SCM comes from:
-
[StorageContainerLocationProtocolServerSideTranslatorPB.java](/Users/deveshsingh/Downloads/Ozone/workspace/fork/ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocolServerSideTranslatorPB.java:1442)
## Assumptions
1. Each serialized `ContainerID` on the wire is approximately `12 bytes`.
Formula:
```text
size_per_container_id = 12 bytes
```
2. The proposed application-level batch size is:
```text
batch_size = 500,000 container IDs per RPC
```
3. The latency sample is treated as a baseline for linear projection:
```text
baseline_container_count = 123,409
baseline_rpc_calls = 1
baseline_avg_rpc_time = 283 ms
baseline_total_time = 516 ms
```
4. Two time models are projected:
- `RPC time only`
Uses `avgRpcTime=283ms` from the log.
- `End-to-end sync time`
Uses `totalTime=516ms` from the log.
5. These projections are for the state-list RPC that transfers container IDs,
not full `ContainerWithPipeline` metadata payloads.
## Maximum Theoretical IDs Per 128 MB RPC
If we only consider the serialized wire size:
```text
max_rpc_message_size = 128 MB
= 128 * 1024 * 1024
= 134,217,728 bytes
max_container_ids_per_128mb_rpc =
134,217,728 / 12
= 11,184,810.67
```
Therefore:
```text
max_container_ids_per_128mb_rpc ~= 11.18 million
```
This is only a wire-size upper bound. It is not a recommended operational
batch size because SCM serialization cost, protobuf object creation, and
transient heap pressure grow with the list size.
## Proposed Operational Batch Size
Proposed batch size:
```text
batch_size = 500,000
```
This yields per-RPC wire payload:
```text
payload_per_rpc_bytes = 500,000 * 12
= 6,000,000 bytes
```
Conversions:
```text
payload_per_rpc_decimal_mb = 6,000,000 / 1,000,000 = 6.0 MB
payload_per_rpc_binary_mib = 6,000,000 / 1,048,576 = 5.72 MiB
```
So:
```text
payload_per_rpc ~= 6.0 MB ~= 5.72 MiB
```
## Baseline-Derived Time Per Container
### RPC Time Only
Using:
```text
baseline_avg_rpc_time = 283 ms
baseline_container_count = 123,409
```
Per-container RPC time:
```text
rpc_time_per_container =
283 ms / 123,409
= 0.0022931878 ms
= 2.2931878 microseconds
```
### End-to-End Sync Time
Using:
```text
baseline_total_time = 516 ms
baseline_container_count = 123,409
```
Per-container end-to-end time:
```text
total_time_per_container =
516 ms / 123,409
= 0.0041804077 ms
= 4.1804077 microseconds
```
## Time Per 500K RPC
### RPC Time Only
```text
rpc_time_per_500k =
500,000 * 0.0022931878 ms
= 1,146.5939 ms
= 1.1466 s
```
### End-to-End Sync Time
```text
total_time_per_500k =
500,000 * 0.0041804077 ms
= 2,090.2039 ms
= 2.0902 s
```
## General Projection Formulas
For any container count `N`:
### Number of RPC Calls
```text
rpc_calls(N) = ceil(N / 500,000)
```
### Total Transfer Volume
```text
transfer_bytes(N) = N * 12
```
### Projected RPC Time Only
```text
projected_rpc_time_ms(N) =
N * (283 / 123,409)
```
### Projected End-to-End Sync Time
```text
projected_total_time_ms(N) =
N * (516 / 123,409)
```
## 4 Million Containers
### RPC Calls
```text
rpc_calls =
ceil(4,000,000 / 500,000)
= 8
```
### Transfer Volume
```text
transfer_bytes =
4,000,000 * 12
= 48,000,000 bytes
transfer_decimal_mb = 48,000,000 / 1,000,000 = 48 MB
transfer_binary_mib = 48,000,000 / 1,048,576 = 45.78 MiB
```
### RPC Time Only
```text
projected_rpc_time_ms =
4,000,000 * (283 / 123,409)
= 9,172.7511 ms
= 9.17 s
```
### End-to-End Sync Time
```text
projected_total_time_ms =
4,000,000 * (516 / 123,409)
= 16,721.6312 ms
= 16.72 s
```
## 10 Million Containers
### RPC Calls
```text
rpc_calls =
ceil(10,000,000 / 500,000)
= 20
```
### Transfer Volume
```text
transfer_bytes =
10,000,000 * 12
= 120,000,000 bytes
transfer_decimal_mb = 120 MB
transfer_binary_mib = 114.44 MiB
```
### RPC Time Only
```text
projected_rpc_time_ms =
10,000,000 * (283 / 123,409)
= 22,931.8778 ms
= 22.93 s
```
### End-to-End Sync Time
```text
projected_total_time_ms =
10,000,000 * (516 / 123,409)
= 41,804.0779 ms
= 41.80 s
```
## 50 Million Containers
### RPC Calls
```text
rpc_calls =
ceil(50,000,000 / 500,000)
= 100
```
### Transfer Volume
```text
transfer_bytes =
50,000,000 * 12
= 600,000,000 bytes
transfer_decimal_mb = 600 MB
transfer_binary_mib = 572.20 MiB
```
### RPC Time Only
```text
projected_rpc_time_ms =
50,000,000 * (283 / 123,409)
= 114,659.3891 ms
= 114.66 s
= 1.91 min
```
### End-to-End Sync Time
```text
projected_total_time_ms =
50,000,000 * (516 / 123,409)
= 209,020.3893 ms
= 209.02 s
= 3.48 min
```
## 100 Million Containers
### RPC Calls
```text
rpc_calls =
ceil(100,000,000 / 500,000)
= 200
```
### Transfer Volume
```text
transfer_bytes =
100,000,000 * 12
= 1,200,000,000 bytes
transfer_decimal_gb = 1.2 GB
transfer_binary_gib = 1,200,000,000 / 1,073,741,824 = 1.118 GiB
```
### RPC Time Only
```text
projected_rpc_time_ms =
100,000,000 * (283 / 123,409)
= 229,318.7783 ms
= 229.32 s
= 3.82 min
```
### End-to-End Sync Time
```text
projected_total_time_ms =
100,000,000 * (516 / 123,409)
= 418,040.7788 ms
= 418.04 s
= 6.97 min
```
## Summary Table
| Containers | RPC Calls | Payload / RPC | Total Payload | Est. RPC Time
Only | Est. End-to-End Sync Time |
|---|---:|---:|---:|---:|---:|
| `4M` | `8` | `~6.0 MB` | `~48 MB` | `~9.2 s` | `~16.7 s` |
| `10M` | `20` | `~6.0 MB` | `~120 MB` | `~22.9 s` | `~41.8 s` |
| `50M` | `100` | `~6.0 MB` | `~600 MB` | `~114.7 s` | `~209.0 s` |
| `100M` | `200` | `~6.0 MB` | `~1.2 GB` | `~229.3 s` | `~418.0 s` |
## SCM Heap Estimate
The `12 bytes` figure is only the serialized protobuf wire size for one
`ContainerID`. It is not the Java heap size of all transient objects involved
in serving the RPC.
Important implementation details:
1. SCM obtains the IDs from the state map and collects them into a `List`.
2. SCM then converts each `ContainerID` into protobuf and adds it to the
response builder.
This means SCM heap cost per RPC is mostly:
- the list of references returned for the batch
- protobuf response objects / builder state
- transient serialization buffers
It is not equivalent to allocating `500,000 * 12 bytes` on heap.
### Practical Per-RPC Heap Estimate for 500K IDs
Wire payload:
```text
500,000 * 12 = 6,000,000 bytes ~= 6 MB
```
Reasonable transient heap estimate at SCM:
- list/reference overhead: `~2 MB to 4 MB`
- protobuf payload/builder/buffers: `~6 MB+`
- additional temporary object overhead: `several MB`
Practical planning range:
```text
estimated_transient_heap_per_500k_rpc ~= 10 MB to 25 MB
```
A good midpoint estimate is:
```text
estimated_transient_heap_per_500k_rpc ~= 15 MB
```
## Key Takeaways
1. A `128 MB` RPC envelope could theoretically carry about `11.18 million`
container IDs if only wire size is considered.
2. A `500K` operational batch is much safer and produces only `~6 MB` of
payload per RPC.
3. For `100 million` containers:
```text
rpc_calls = 200
total_transfer_volume = 1.2 GB
estimated_rpc_time_only = 229.32 s ~= 3.82 min
estimated_end_to_end_sync_time = 418.04 s ~= 6.97 min
```
4. SCM transient heap should remain bounded per call with `500K` batching and
is expected to be in the `10 MB to 25 MB` range per RPC rather than
scaling
toward the full `100 million` container population.
## Caveats
These numbers are first-order projections, not guarantees.
They assume near-linear scaling from the observed baseline. Real behavior may
shift due to:
- SCM CPU saturation
- protobuf serialization overhead
- JVM GC behavior
- Recon consumer-side processing
- network bandwidth and latency variability
- concurrent SCM load from other clients
**On your 3rd and last point related to alerts/notifications :**
Currently, we don't have any alert framework inherently supported in Recon.
I discussed and raised this point earlier, but since users can have various
external ways also based on hadoop metrics, so we didn't carry on with this
idea.
So based on existing hadoop metrics, we can have following metrics:
1. Count of occurrences in cumulative way, how many events of full snapshot
got hit rather than real execution of full snapshot.
2. in last such event of full snapshot, size of last drift of that last
event. If value comes out 50K or any other value.
3. Time between last 2 such events of full SCM DB snapshot download.
4. We can add separate metrics per state drift. CLOSED, QUASI_CLOSED etc,
and size of drift.
5. Status of Targeted sync if it is in progress , success, failure.
6. Timetaken by last targeted sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
