This is an automated email from the ASF dual-hosted git repository.
dinglei pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
The following commit(s) were added to refs/heads/develop by this push:
new 648cac3 [RIP-9] Add english document of storage design(#809)
648cac3 is described below
commit 648cac3eaddafd54016f685084c7b50aa583c723
Author: vonzhou <[email protected]>
AuthorDate: Thu Feb 21 16:48:29 2019 +0800
[RIP-9] Add english document of storage design(#809)
[RIP-9] Add english document of storage design
---
docs/en/Design_Store.md | 36 ++++++++++++++++++++++++++++++
docs/en/images/rocketmq_storage_arch.png | Bin 0 -> 111170 bytes
docs/en/images/rocketmq_storage_flush.png | Bin 0 -> 22281 bytes
3 files changed, 36 insertions(+)
diff --git a/docs/en/Design_Store.md b/docs/en/Design_Store.md
new file mode 100644
index 0000000..ac14022
--- /dev/null
+++ b/docs/en/Design_Store.md
@@ -0,0 +1,36 @@
+# Message Storage
+
+
+
+
+Message storage is the most complicated and important part of RocketMQ. This
section will describe the three aspects of RocketMQ: message storage
architecture, PageCache and memory mapping, and RocketMQ's two different disk
flushing methods.
+
+## 1. Message Storage Architecture
+
+
+The message storage architecture diagram consists of 3 files related to
message storage: `CommitLog` file, `ConsumeQueue` file, and `IndexFile`.
+
+
+* `CommitLog`：The `CommitLog` file stores message body and metadata sent by
producer, and the message content is not fixed length. The default size of one
`CommitLog` file is 1G, the length of the file name is 20 digits, the left side
is zero padded, and the remaining is the starting offset. For example,
`00000000000000000000` represents the first file, the starting offset is 0, and
the file size is 1G=1073741824, when the first `CommitLog` file is full, the
second `CommitLog` file is `0 [...]
+* `ConsumeQueue`: The `ConsumeQueue` is used to improve the performance of
message consumption. Since RocketMQ uses topic-based subscription mode, message
consumption is specific to the topic. Traversing the commitlog file to retrieve
messages of one topic is very inefficient. The consumer can find the messages
to be consumed according to the `ConsumeQueue`. The `ConsumeQueue`(logic
consume queue) as an index of the consuming message stores the starting
physical offset `offset` in `Commi [...]
+* `IndexFile`: The `IndexFile` provides a way to query messages by key or time
interval. The path of the `IndexFile` is `$HOME/store/index/${fileName}`, the
file name `fileName` is named after the timestamp when it was created. One
IndexFile's size is about 400M, and it can store 2000W indexes. The underlying
storage of `IndexFile` is designed to implement the `HashMap` structure in the
file system, so RocketMQ's index file is a hash index.
+
+
+From the above architecture of the RocketMQ message storage, we can see
RocketMQ uses a hybrid storage structure, that is, all the queues in an
instance of the broker share a single log file `CommitLog` to store messages.
RocketMQ's hybrid storage structure(messages of multiple topics are stored in
one CommitLog) uses a separate storage structure for the data and index parts
for Producer and Consumer respectively. The Producer sends the message to the
Broker, then the Broker persists the [...]
+
+## 2 PageCache and memory map
+
+PageCache is a cache of files by the operating system to speed up the reading
and writing of files. In general, the speed of sequential read and write files
is almost the same as the speed of read and write memory. The main reason is
that the OS uses a portion of the memory as PageCache to optimize the
performance of the read and write operations. For data writing, the OS will
first write to the Cache, and then the `pdflush` kernel thread asynchronously
flush the data in the Cache to the [...]
+
+In RocketMQ, the logic consumption queue `ConsumeQueue` stores less data and
is read sequentially. With the help of prefetch of the page cache mechanism,
the read performance of the `ConsumeQueue` file is almost close to the memory
read, even in the case of message accumulation, it does not affect performance.
But for the log data file `CommitLog`, it will generate many random access
reads when reading the message content, which seriously affects the
performance. If you choose the approp [...]
+
+
+In addition, RocketMQ mainly reads and writes files through
`MappedByteBuffer`. `MappedByteBuffer` uses the `FileChannel` model in NIO to
directly map the physical files on the disk to the memory address in user space
(`Mmap` method reduces the performance overhead of traditional IO copying disk
file data back and forth between the buffer in kernel space and the buffer in
user space), it converts the file operation into direct memory address
manipulation, which greatly improves the effic [...]
+
+## 3 Message Disk Flush
+
+
+
+
+* synchronous flush: As shown above, the RocketMQ's Broker will return a
successful `ACK` response to the Producer after the message is truly persisted
to disk. Synchronous flushing is a good guarantee for the reliability of MQ
messages, but it will have a big impact on performance. Generally, it is
suitable for financial business applications.
+* asynchronous flush: Asynchronous flushing can take full advantage of the
PageCache of the OS, as long as the message is written to the PageCache, the
successful `ACK` can be returned to the Producer. The message flushing is
performed by the background asynchronous thread, which reduces the read and
write delay and improves the performance and throughput of the MQ.
\ No newline at end of file
diff --git a/docs/en/images/rocketmq_storage_arch.png
b/docs/en/images/rocketmq_storage_arch.png
new file mode 100644
index 0000000..8c719e1
Binary files /dev/null and b/docs/en/images/rocketmq_storage_arch.png differ
diff --git a/docs/en/images/rocketmq_storage_flush.png
b/docs/en/images/rocketmq_storage_flush.png
new file mode 100644
index 0000000..1610ae0
Binary files /dev/null and b/docs/en/images/rocketmq_storage_flush.png differ
