[
https://issues.apache.org/jira/browse/IGNITE-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Bessonov updated IGNITE-23240:
-----------------------------------
Attachment: Screenshot from 2024-09-20 10-42-49.png
> Ignite 3 new log storage
> ------------------------
>
> Key: IGNITE-23240
> URL: https://issues.apache.org/jira/browse/IGNITE-23240
> Project: Ignite
> Issue Type: Epic
> Reporter: Ivan Bessonov
> Priority: Major
> Labels: ignite-3
> Attachments: Screenshot from 2024-09-20 10-38-53.png, Screenshot from
> 2024-09-20 10-38-57.png, Screenshot from 2024-09-20 10-42-49.png, Screenshot
> from 2024-09-20 10-43-09.png
>
>
> h1. Preface
> Current implementation, based on {{{}RocksDB{}}}, is known to be way slower
> then it should be. There are multiple obvious reasons for that:
> * Writing into WAL +and+ memtable
> * Creating unique keys for every record
> * Inability to efficiently serialize data, we must have an intermediate
> state before we pass data into {{{}RocksDB{}}}'s API.
> h1. Benchmarks
> h3. Local benchmarks
> Local benchmarks ({{{}LogStorageBenchmarks{}}}) have been performed on my
> local environment with fsync disabled. I got the following results:
> * {{{}Logit{}}}:
>
> {noformat}
> Test write:
> Log number : 1024000
> Log Size : 16384
> Batch Size : 100
> Cost time(s) : 23.541
> Total size : 16777216000
> Throughput(bps) : 712680684
> Throughput(rps) : 43498
> Test read:
> Log number : 1024000
> Log Size : 16384
> Batch Size : 100
> Cost time(s) : 3.808
> Total size : 16777216000
> Throughput(bps) : 4405781512
> Throughput(rps) : 268907
> Test done!{noformat}
> * {{{}RocksDB{}}}:
> {noformat}
> Test write:
> Log number : 1024000
> Log Size : 16384
> Batch Size : 100
> Cost time(s) : 178.785
> Total size : 16777216000
> Throughput(bps) : 93840176
> Throughput(rps) : 5727
> Test read:
> Log number : 1024000
> Log Size : 16384
> Batch Size : 100
> Cost time(s) : 13.572
> Total size : 16777216000
> Throughput(bps) : 1236163866
> Throughput(rps) : 75449
> Test done!{noformat}
> While testing on local environment is not optimal, is still shows a huge
> improvement in writing speed (7.5x) and reading speed (3.5x). Enabling
> {{fsync}} sort-of equalizes writing speed, but we still expect that simpler
> log implementation would be faster dues to smaller overall overhead.
> h3. Integration testing
> Benchmark for 3 servers and 1 client writing data in multiple threads show
> the following.
> {{{}RocksDB{}}}:
>
>
> {{{}Logit{}}}:
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
