lengyuexuexuan opened a new issue, #2210:
URL: https://github.com/apache/incubator-pegasus/issues/2210
## Bug Report
1. What did you do?
There are two places that may cause this issus.
The first is copying prepare_list.
1. copy prepare_list
1.
https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L305-L308
2.
https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/prepare_list.cpp#L52-L57
3. During the process of copying the prepare_list, write operations
were not stopped, so mutation_cache and last_commit_decree will continue to
grow.
4. Since last_commit_decree is copied later, it is possible for
last_committed_decree to be greater than the actual committed_decree in the
prepare_list.
5. Therefore, this may lead to data loss.
2. plog append
1.
https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L370-L378
2.
https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L1359-L1366
3. Currently, when the child replicates mutations, there are also
asynchronous writes coming from the client, and both operations will perform
plog append.
4. The plog does not have a method to distinguish the order.
2. soulution
1. For the first issue, it can be resolved by copying the mutation_cache
again after copying the last_commited_decree.
2. For the second issue, there is currently no good solution in mind.
3. Stopping client writes before the copy_prepare_list process seems
like a good approach to prevent conflicts and ensure consistency.
4. By stopping client writes before the copy_prepare_list process, you
can address both of the issues.
3. Here is the test for the write-stop duration
数据量 1M
| 分片数量 | 单分片阻写时间(min) | 单分片阻写时间(max) | 单分片阻写时间(avg) | 总体阻写时间 |
|--------------|---------------------|---------------------|---------------------|--------------|
| 32->64 | 1.1s | 2.1s | 1.9s
| 85s |
| 64->128 | 1.5s | 61s | 4.2s
| 83s |
| 128->256 | 1.3s | 5.9s | 3.2s
| 85s |
| 256->512 | 1.3s | 32s | 3.1s
| 89s |
| 512->1024 | 1.3s | 60s | 7.1s
| 110s |
数据量 172G
| 分片数量 | 单分片阻写时间(min) | 单分片阻写时间(max) | 单分片阻写时间(avg) | 总体阻写时间 |
|--------------|---------------------|---------------------|---------------------|--------------|
| 64->128 | 1.2s | 121.5s | 20.9s
| 141s |
| 128->256 | 1.8s | 81.5s | 14.9s
| 122s |
| 256->512 | 2s | 66.1s | 11s
| 144s |
| 512->1024 | (内存被打满 OOM) | |
| |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
