[
https://issues.apache.org/jira/browse/IGNITE-26279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Polovtsev reassigned IGNITE-26279:
--------------------------------------------
Assignee: Aleksandr Polovtsev
> Implement segment index files and a checkpointer thread
> -------------------------------------------------------
>
> Key: IGNITE-26279
> URL: https://issues.apache.org/jira/browse/IGNITE-26279
> Project: Ignite
> Issue Type: Improvement
> Reporter: Ivan Bessonov
> Assignee: Aleksandr Polovtsev
> Priority: Major
> Labels: ignite-3
>
> Index file is a persistent analogue to
> https://issues.apache.org/jira/browse/IGNITE-26272's mem-tables.
> h2. Files
> First of all, the general data format:
> {code:java}
> +--------+---------------------+-----+---------------------+
> | Header | Payload for group 1 | ... | Payload for group N |
> +--------+---------------------+-----+---------------------+ {code}
> Header should look like this:
> {noformat}
> +---------------------+-----------------------------------------+-----+
> | Common meta | Raft group 1 meta | ... |
> +---------------------+-----------------------------------------+-----+
> | Magic | Version | N | Group ID | Flags | Offset | Start | End | ... |
> +---------------------+-----------------------------------------+-----+{noformat}
> * {{Magic number}} is for checking that this is the right file. 4 bytes.
> * {{Version}} is for format extensibility. 4 bytes.
> * {{N}} is a number of raft groups in this file. 4 bytes.
> * {{GroupID}} is a raft group identifier. 8 bytes.
> * {{Flags}} means additional information about the group, for example “this
> is the last index for this group” flag. 4 bytes.
> * {{Offset}} is an offset within this index file that points to "Payload for
> group X" block. 4 bytes.
> * {{Start}} and {{End}} here mean the range of raft indexes in this index
> file. 8 bytes each.
> Payload should look like this:
> {code:java}
> +----------+-----+----------+
> | Offset 1 | ... | Offset M |
> +----------+-----+----------+ {code}
> * {{Offset}} is an offset in a corresponding segment file that points to a
> corresponding log entry.
> * There are exactly {{End-Start}} offsets there.
> h2. Checkpointer
> We can chose other name if this one is confusing.
> The checkpointer thread has to be awaken when log storage switches to the
> next segment. It does the following:
> * Collect mem-tables from all stripes and make them immutable. New
> mem-tables will be allocated automatically.
> * {{sync}} the segment file.
> * Write an index file that matches the mem-table.
> * [Out of scope for now] Create an in-memory metadata structures for the
> index.
> * Remove the mem-table from memory.
> * {{sync}} the index file.
> * Rename {{.bin.tmp}} file to the {{{}.bin{}}}.
> * Wait for a next {{{}mem-table{}}}.
> There might be a queue of unwritten mem-tables. We should limit its size.
> Throttling is out of scope.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
