Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22215 )
Change subject: IMPALA-13478: Sync tuple cache files to disk asynchronously ...................................................................... IMPALA-13478: Sync tuple cache files to disk asynchronously When a tuple cache entry is first being written, we want to sync the contents to disk. Currently, that happens on the fast path and delays the query results, sometimes significantly. This moves the Sync() call off of the fast path by passing the work to a thread pool. The threads in the pool open the file, sync it to disk, then close the file. If anything goes wrong, the cache entry is evicted. The tuple cache can generate writes very quickly, so this needs a backpressure mechanism to avoid overwhelming the disk. In particular, it needs to avoid accumulating dirty buffers to the point that the OS throttles new writes, delaying the query fast path. This implements a limit on outstanding writes (i.e. writes that have not been flushed to disk). To enforce it, writers now call UpdateWriteSize() to reserve space before writing. UpdateWriteSize() can fail if it hits the limit on outstanding writes or if this particular cache entry has hit the maximum size. When it fails, the writer should abort writing the cache entry. Since UpdateWriteSize() is updating the charge in the cache, the outstanding writes are being counted against the capacity, triggering evictions. This improves the tuple cache's adherence to the capacity limit. The outstanding writes limits is configured via the tuple_cache_outstanding_write_limit startup flag, which is either a specific size string (e.g. 1GB) or a percentage of the process memory limit. To avoid updating the cache charge very frequently, this has an update chunk size specified by tuple_cache_outstanding_write_chunk_bytes. This adds counters at the daemon level: - outstanding write bytes - number of writes halted due to backpressure - number of sync calls that fail (due to IO errors) - number of sync calls dropped due to queue backpressure The runtime profile adds a NumTupleCacheBackpressureHalted counter that is set when a write hits the outstanding write limit. This has a startup option to add randomness to the tuple cache keys to make it easy to test a scenario with no cache hits. Testing: - Added unit tests to tuple-cache-mgr-test - Testing with TPC-DS on a cluster with fast NVME SSDs showed a significant improvement in the first-run times due to the asynchronous syncs. - Testing with TPC-H on a system with a slow disk and zero cache hits showed improved behavior with the backpressure Change-Id: I646bb56300656d8b8ac613cb8fe2f85180b386d3 Reviewed-on: http://gerrit.cloudera.org:8080/22215 Reviewed-by: Joe McDonnell <[email protected]> Reviewed-by: Michael Smith <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/tuple-cache-node.cc M be/src/exec/tuple-cache-node.h M be/src/exec/tuple-file-read-write-test.cc M be/src/exec/tuple-file-writer.cc M be/src/exec/tuple-file-writer.h M be/src/runtime/exec-env.cc M be/src/runtime/tuple-cache-mgr-test.cc M be/src/runtime/tuple-cache-mgr.cc M be/src/runtime/tuple-cache-mgr.h M be/src/service/query-options.cc M common/thrift/generate_error_codes.py M common/thrift/metrics.json 12 files changed, 722 insertions(+), 75 deletions(-) Approvals: Joe McDonnell: Looks good to me, but someone else must approve Michael Smith: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/22215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I646bb56300656d8b8ac613cb8fe2f85180b386d3 Gerrit-Change-Number: 22215 Gerrit-PatchSet: 13 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]>
