I attached the histogram of the latency in the JIRA. The x-axis is the latency while the y-axis is the cumulative distribution. We can see that about 30% insertion can be finished in 20ms, and 60% insertion can be finished in 40ms even though the IoTDB instance is serving for a heavy workload... So, eliminating the long tail insertion can make the average latency far better.
If someone is working on the refactor_overflow or refactor_bufferwrite, please pay attention to the code branch for this issue. Best, ----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 xiangdong Huang (JIRA) <j...@apache.org> 于2019年6月4日周二 下午11:00写道： > xiangdong Huang created IOTDB-112: > ------------------------------------- > > Summary: Avoid long tail insertion which is caused by > synchronized close-bufferwrite > Key: IOTDB-112 > URL: https://issues.apache.org/jira/browse/IOTDB-112 > Project: Apache IoTDB > Issue Type: Improvement > Reporter: xiangdong Huang > > > In our test, IoTDB has a good insertion performance, and the average > latency can be ~200 ms in a given workload and hardware. > > However, when we draw the histogram of the latency, we find that 97.5% > latencies are less than 200 ms, while 2.7% latencies are greater. The > result shows that there are some long tail latency. > > Then we find that some insertion latencies are about 30 seconds... (but > the ratio is less than 0.5%). Indeed, for each connection, a long tail > insertion appears per 1 or 2 minutes.... > > By reading source codes, I think it is because that in the insertion > function, > > `private void insertBufferWrite(FileNodeProcessor fileNodeProcessor, long > timestamp, > boolean isMonitor, TSRecord tsRecord, String deviceId)`, > > if the corresponding TsFile is too large, the function is blocked until > the memtable is flushed on disk and the TsFile is sealed (we call it as > closing a TsFile). The latencies of the long tail insertions are very close > to the time cost of flushing and sealing a TsFile. > > So, if we set the closing function using the async mode, we can avoid the > long tail insertion. > > However, there are some side effects we have to fix: > # At the same time, if a new insertion comes, then a new memtable should > be assigned, and a new unsealed TsFile is created; > # That means that there are more than 1 unsealed TsFiles if the system is > crashed before the closing function is finished. So, we have to modify the > startup process to recover these files. > > Is there any other side effect that I have to pay attention to? > > > > > > > > > > > > > > > > -- > This message was sent by Atlassian JIRA > (v7.6.3#76005) >
