Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-3202: implement spill-to-disk in new buffer pool ......................................................................
IMPALA-3202: implement spill-to-disk in new buffer pool See https://goo.gl/0zuy97 for a high-level summary of the design. Unpinned pages can now be written to disk to free up memory. After unpinning, pages enter a "dirty" state. Each client initiates asynchronous writes for dirty page to free up memory to allocate more buffers. After the write completes, pages are "clean" and can be evicted from memory by any client that needs the buffer. This is implemented by moving pages between lists in ClientImpl (a new internal class that stores the client's state) and BufferPool. I/O: ---- The mechanics of I/O to scratch files is handled by the TmpFileMgr mechanisms introduced in earlier IMPALA-3202 patches. The decision to start a write I/O is based on two factors. First, each client maintains the invariant that bytes of the dirty pages should not exceed the unused reservation. This is to ensure that any client's reservation can always be fulfilled by evicting a clean page. Second, additional writes are initiated to proactively write data to disk, so that clean pages are available when needed. The buffer pool tries to keep enough writes in flight to keep all disks busy. The buffer pool avoids read I/O whenever possible: if an unpinned page is pinned again and its buffer is still in memory, no I/O is required to read back the data. Locking: -------- Concurrency is managed using client, page, and clean page list locks. The design uses intrusive doubly-linked lists to track pages. This patch adds a LockType parameter to InternalQueue and a FakeLock type to support a non-concurrent InternalList type that is more convenient for the buffer pool's locking scheme. Testing: -------- Added some basic unit tests to BufferPoolTest that exercise the new code paths. More test coverage will be added later with system tests and porting of tests from BufferedBlockMgrTest - see the draft test plan for IMPALA-3200. Change-Id: I8c6119a4557da853fefa02e17c49d8be0e9fbe01 Reviewed-on: http://gerrit.cloudera.org:8080/5584 Reviewed-by: Tim Armstrong <tarmstr...@cloudera.com> Tested-by: Impala Public Jenkins --- M be/src/common/compiler-util.h M be/src/runtime/bufferpool/buffer-pool-counters.h A be/src/runtime/bufferpool/buffer-pool-internal.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/bufferpool/buffer-pool.h M be/src/runtime/bufferpool/suballocator-test.cc M be/src/runtime/bufferpool/suballocator.cc M be/src/runtime/bufferpool/suballocator.h M be/src/runtime/disk-io-mgr.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/testutil/death-test-util.h M be/src/util/aligned-new.h A be/src/util/fake-lock.h M be/src/util/internal-queue.h 16 files changed, 1,133 insertions(+), 387 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/5584 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I8c6119a4557da853fefa02e17c49d8be0e9fbe01 Gerrit-PatchSet: 19 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple <jbapple-imp...@apache.org> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
