Dimitris Tsirogiannis has posted comments on this change. Change subject: [PREVIEW] Use native allocation while building catalog updates ......................................................................
Patch Set 3: (7 comments) http://gerrit.cloudera.org:8080/#/c/7955/3//COMMIT_MSG Commit Message: PS3, Line 20: 4-5GB You need to comment about the limit (4GB) that currently thrift is imposing. http://gerrit.cloudera.org:8080/#/c/7955/3/fe/src/main/java/org/apache/impala/service/JniCatalog.java File fe/src/main/java/org/apache/impala/service/JniCatalog.java: PS3, Line 142: This method serializes : * the output of CatalogService#getCatallogObjects() into native memory allocated : * using a custom TNativeSerializer and returns the corresponding TNativeByteBuffer : * serialized into bytes. Alternatively: "Uses a custom serializer (TNativeSerializer) to serialize the catalog objects into a byte buffer that is backed by native memory." http://gerrit.cloudera.org:8080/#/c/7955/3/fe/src/main/java/org/apache/impala/thrift/NativeByteArrayOutputStream.java File fe/src/main/java/org/apache/impala/thrift/NativeByteArrayOutputStream.java: PS3, Line 36: 512MB update value Line 54: // Limit on the size to which the underlying buffer can grow Comment why that limit exists PS3, Line 103: off offset PS3, Line 115: if (bufferLen_ >= BUFFER_DOUBLING_RESIZE_LIMIT) { : while (newBufferSize < bytesWritten_ + len) { : newBufferSize += BUFFER_RESIZE_INCREMENTS; : } : } else { : while (newBufferSize < bytesWritten_ + len) { : newBufferSize <<= 1; : } : } I think if you put the if/else block inside the while block it may be a bit easier to follow plus it will give you a slightly better behavior because you'll be doubling as long as newBufferSize < 1GB and then you'll switch to incremental increases if you cross the 1GB boundary. PS3, Line 135: public synchronized void reset() { : } single line (here and for some of the other functions as well). -- To view, visit http://gerrit.cloudera.org:8080/7955 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I383684effa9524734ce3c6c0fb7ed37de0e15782 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com> Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com> Gerrit-Reviewer: Mostafa Mokhtar <mmokh...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-HasComments: Yes
