[
https://issues.apache.org/jira/browse/HDDS-12834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108624#comment-18108624
]
Chia-Chuan Ho edited comment on HDDS-12834 at 8/27/26 3:48 AM:
---------------------------------------------------------------
Thanks [~ayarovoy] , that is a good point.
OM process CPU utilization is a composite signal: it can come from RPC/gRPC
request processing, Ratis apply, RocksDB reads/writes, GC, and other background
services.
Therefore, CPU utilization alone does not tell us whether OM is busy serving
foreground requests.
A bottommost RocksDB compaction performs SST reads, merging/filtering,
decompression/compression, checksum generation, and SST writes. It can be CPU
intensive, but it can also be primarily I/O-bound depending on the compression,
dataset, storage, and page cache. We should measure this instead of assuming
CPU is the dominant resource.
I propose doing a signal study before implementing the gate. On a 3-OM HA
setup, I will compare idle, read-heavy, and write/delete-heavy workloads, both
with and without manual bottommost compaction. Per OM, I will collect:
* foreground request throughput and latency;
* existing OM protocol request concurrency;
* Hadoop RPC queue/thread-pool pressure;
* Ratis apply and OM double-buffer flush pressure;
* process CPU, GC, and disk I/O;
* RocksDB compaction duration and bytes read/written.
The HA distinction is important: a follower may have almost no client requests
while still being busy applying replicated transactions and flushing RocksDB.
Request rate or handler utilization alone could therefore classify a busy
follower as idle.
Based on the results, we can choose the signal that best predicts foreground
latency impact. My current candidate is request concurrency or queue pressure
for the leader, supplemented by Ratis/double-buffer pressure for followers,
rather than a fixed CPU or requests/sec threshold. I will share the
measurements before finalizing the implementation approach.
was (Author: JIRAUSER310086):
Thanks [~ayarovoy] , that is a good point.
OM process CPU utilization is a composite signal: it can come from RPC/gRPC
request processing, Ratis apply, RocksDB reads/writes, GC, and other background
services.
Therefore, CPU utilization alone does not tell us whether OM is busy serving
foreground requests.
A bottommost RocksDB compaction performs SST reads, merging/filtering,
decompression/compression, checksum generation, and SST writes. It can be CPU
intensive, but it can also be primarily I/O-bound depending on the compression,
dataset, storage, and page cache. We should measure this instead of assuming
CPU is the dominant resource.
I propose doing a signal study before implementing the gate. On a 3-OM HA
setup, I will compare idle, read-heavy, and write/delete-heavy workloads, both
with and without manual bottommost compaction. Per OM, I will collect:
* foreground request throughput and latency;
* existing OM protocol request concurrency;
* Hadoop RPC queue/thread-pool pressure;
* Ratis apply and OM double-buffer flush pressure;
* process CPU, GC, and disk I/O;
* RocksDB compaction duration and bytes read/written.
The HA distinction is important: a follower may have almost no client requests
while still being busy applying replicated transactions and flushing RocksDB.
Request rate or handler utilization alone could therefore classify a busy
follower as idle.
Based on the results, we can choose the signal that best predicts foreground
latency impact. My current candidate is request concurrency or queue pressure
for the leader, supplemented by Ratis/double-buffer pressure for followers,
rather than a fixed CPU or requests/sec threshold. I will share the
measurements before finalizing the implementation approach.
> Improve performance of auto-compaction of tables by the background service.
> ---------------------------------------------------------------------------
>
> Key: HDDS-12834
> URL: https://issues.apache.org/jira/browse/HDDS-12834
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Tejaskriya Madhan
> Priority: Major
>
> HDDS-12819 aims to introduce a background service that bottommost-compacts a
> set of column families. This process can be optimized if it is aware of the
> CPU utilization of the service that that point in time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
