Tim Armstrong has uploaded a new patch set (#2). Change subject: IMPALA-3282: TBinaryProtocol buffers consume unbounded memory ......................................................................
IMPALA-3282: TBinaryProtocol buffers consume unbounded memory This patch removes the builtin string buffer in TBinaryProtocol. It was used to buffer large strings being read via thrift. The buffer is never shrunk, so each thrift connection ends up consuming memory equal to the largest string that was received by the connection. This can result in a lot of memory, given a large number of connections and large strings. The string buffer is actually unnecessary, since the buffer contents are always copied into the string, so the buffer simply adds an extra copy. This may not be a big win, since string::resize() implicitly does a memset(). The fix is to resize the string then write directly to the string's memory. The C++11 standard guarantees that a std::string's memory is contiguous. Change-Id: I58bdc899e604bde2f5b2d5360f79537ce25715ee --- M buildall.sh A source/thrift/thrift-0.9.0-patches/0008-IMPALA-3582-string-buffer-memory.patch 2 files changed, 74 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Toolchain refs/changes/01/3201/2 -- To view, visit http://gerrit.cloudera.org:8080/3201 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I58bdc899e604bde2f5b2d5360f79537ce25715ee Gerrit-PatchSet: 2 Gerrit-Project: Toolchain Gerrit-Branch: master Gerrit-Owner: Tim Armstrong <[email protected]>
