jiacai2050 commented on code in PR #141: URL: https://github.com/apache/horaedb-docs/pull/141#discussion_r1818284257
########## content/en/docs/design/compaction_offload.md: ########## @@ -0,0 +1,92 @@ +--- +title: "Compaction Offload" +--- + +**Note: This feature is still in development.** + +This chapter discusses compaction offload, which is designed to separate the compaction workload from the local horaedb nodes and delegate it to external compaction nodes. + +## Overview + +```plaintext +┌─────────────────────────────────────────────────────────────────────────┐ +│ │ +│ HoraeMeta Cluster │ +│ │ +└─────────────────────────────────────────────────────────────────────────┘ + ▲ ▲ | | + │ │1.Fetch compaction │(Monitor compaction│ + │ │ node info │node) │ + | │ ▼ ▼ +┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ +│ │ │ │2.Offload Task│ │ │ │ +│ HoraeDB │ │ HoraeDB │ ─────────▶ │ Compaction │ │ Compaction │ +│ │ │ │ ◀───────── │ Node │ │ Node │ +└────────────┘ └────────────┘ 4.Ret Result └────────────┘ └────────────┘ + | | | | + │ 5.Update the │ │ 3.Compact │ + │ SSTable │ │ │ + ▼ ▼ ▼ ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ ┌─────────────────────┐ │ +│ Object Storage │ Temporary Workspace │ │ +│ └─────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +The diagram above describes the architecture of cluster for compaction offload, where some key concepts need to be explained: + +- `Compaction Node`: Takes responsibility to handle offloaded compaction tasks. The compaction node receives the compaction task and performs the actual merging of SSTables, then sends back the task result to HoraeDB. +- `HoraeMeta Cluster`: HoraeMeta acts as a compaction nodes manager in the compaction offload scenario. It monitors the compaction nodes cluster and schedule the compaction nodes. + +The procedure of remote compaction based above architecture diagram is: + +1. HoraeDB fetchs the information of suitable compaction node from the HoraeMeta. +2. HoraeDB distributes the compaction task to the remote compaction node, according to the information fetch from HoraeMeta. Review Comment: ```suggestion 2. HoraeDB submit the compaction task to the remote compaction node, according to the information fetched from HoraeMeta. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
