Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8825 )
Change subject: IMPALA-5990: End-to-end compression of metadata ...................................................................... Patch Set 3: (5 comments) Thanks! http://gerrit.cloudera.org:8080/#/c/8825/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/8825/2//COMMIT_MSG@17 PS2, Line 17: - A single catalog object cannot be larger than ~2GB. > It turns out the 1GB limit is fixed in some jdk7 release. openjdk7 has the Ack http://gerrit.cloudera.org:8080/#/c/8825/2//COMMIT_MSG@23 PS2, Line 23: Testing: Ran existing tests. A test for compressing and decompressing > To produce it I modified the code and manually added two 1.95GB objects. It Probably not, but it may be useful to have around, depending on how you did it. Up to you. http://gerrit.cloudera.org:8080/#/c/8825/2/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/8825/2/be/src/catalog/catalog-server.cc@384 PS2, Line 384: VLOG(1) << "Publishing " << (deleted ? "deletion: " : "update: ") << item.key << > Added size into the logging. The backend doesn't know the version so the ve Ack http://gerrit.cloudera.org:8080/#/c/8825/2/be/src/catalog/catalog-util.cc File be/src/catalog/catalog-util.cc: http://gerrit.cloudera.org:8080/#/c/8825/2/be/src/catalog/catalog-util.cc@139 PS2, Line 139: Status CompressCatalogObject(const uint8_t* buf, uint32_t size, string* output) { > We can change DecompressCatalogObject() to using string. I'm not sure if we Ok. http://gerrit.cloudera.org:8080/#/c/8825/2/fe/src/main/java/org/apache/impala/util/TByteBuffer.java File fe/src/main/java/org/apache/impala/util/TByteBuffer.java: http://gerrit.cloudera.org:8080/#/c/8825/2/fe/src/main/java/org/apache/impala/util/TByteBuffer.java@26 PS2, Line 26: * ByteBuffer-backed implementation of TTransport. > I copied it from thrift and removed the unused functions. I'm not sure what It's definitely ok to copy, but I'm not sure if need to do more. At the very least, let's make sure to indicate here that it's a copy, so when we upgrade Thrift, we can rip this out. -- To view, visit http://gerrit.cloudera.org:8080/8825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a8819cad734b3a416eef6c954e55b73cc6023ae Gerrit-Change-Number: 8825 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang <tw...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com> Gerrit-Reviewer: Philip Zeyliger <phi...@cloudera.com> Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com> Gerrit-Comment-Date: Fri, 15 Dec 2017 19:57:51 +0000 Gerrit-HasComments: Yes
