Alex Behm has posted comments on this change. Change subject: [PREVIEW] Use native allocation while building catalog updates ......................................................................
Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/7955/1/be/src/catalog/catalog.cc File be/src/catalog/catalog.cc: PS1, Line 113: free(buf); > Switching the JVM to allocate huge chunks of memory via malloc is going to 1. This code path is catalogd only. 2. Roughly the same amount of untracked memory is already allocated today when creating the thrift topic update to the statestore (data lives in std::string). Imo, this patch makes the untracked memory problem no worse than it already is - fair point that it does not make it better either. 3. The 2GB JVM array limit means the catalogd goes down for large catalogs, and is very hard to recover, so we really need to solve this issue. I can imagine alternative solutions that involve copying large amounts of data back-and-forth through JNI (e.g. implementing a chunked ByteArrayOutputStream) Do you have an alternative proposal? -- To view, visit http://gerrit.cloudera.org:8080/7955 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I383684effa9524734ce3c6c0fb7ed37de0e15782 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-HasComments: Yes
