Chen Luo created ASTERIXDB-2540:
-----------------------------------
Summary: Optimize Performance Stability of Storage
Key: ASTERIXDB-2540
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2540
Project: Apache AsterixDB
Issue Type: Improvement
Components: STO - Storage
Reporter: Chen Luo
Assignee: Chen Luo
This is one of a series of improvements to optimize the performance stability
of our storage subsystem, which suffers from a number of problems. The end
result is that there are periodically write stalls during data ingestion, even
though the ingestion speed is relatively low.
This improvement will deal with the following issues:
1. Bypass all queuing of disk writes during LSM flush and merge operations.
Queuing (by BufferCache and IOManager) will cause serious problems to the
fairness of disk writes. Thus, a small flush operation could be severely
interfered with a large merge operation and would take a much longer time to
finish.
2. Perform regular disk forces during flush and merge operations (16MB by
default). This is very helpful to limit the I/O queue length of the file system
and provides fairness to queries and other writers. This optimization has been
implemented in most storage systems today, including Couchbase Server.
3. Optionally, add the support for rate limiting of disk writes to ensure the
performance stability of queries. The user can configure the maximum disk write
bandwidth for each dataset. This ensures that the system can provide stable
performance for both queries and writes, even with large background merges.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
