sunithabeeram commented on a change in pull request #4127: Declare byte array
size when initializing ByteArrayOutputStream
URL: https://github.com/apache/incubator-pinot/pull/4127#discussion_r276315910
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/common/datatable/DataTableImplV2.java
##########
@@ -346,6 +354,17 @@ public void addException(@Nonnull ProcessingException
processingException) {
return byteArrayOutputStream.toByteArray();
}
+ private int getTotalByteArraySize(byte[] metadataBytes, byte[]
dictionaryMapBytes, byte[] dataSchemaBytes) {
+ int metadataSize = metadataBytes.length + 1;
+ int dictionarySize = dictionaryMapBytes != null ?
dictionaryMapBytes.length + 1 : 2;
+ int dataSchemaSize = dataSchemaBytes != null ? dataSchemaBytes.length + 1
: 2;
+ int fixedSizeDataSize = _fixedSizeDataBytes != null ?
_fixedSizeDataBytes.length + 1 : 2;
+ int variableSizeDataSize = _variableSizeDataBytes != null ?
_variableSizeDataBytes.length + 1 : 2;
+
+ // total size that byte array needs. 3 denotes version, number of rows and
columns.
+ return 3 + metadataSize + dictionarySize + dataSchemaSize +
fixedSizeDataSize + variableSizeDataSize;
Review comment:
Could we use static constants instead of magic numbers above? Its not
immediately clear why something is a 2 vs 1 etc. Looking through the actual
serialization gives a clue, but thats not evident here.
I am assuming that even if the estimate here is off, the buffer will still
get resized and we'll not encounter issues. If so, do the estimates need to be
precise? Also, would it be an option to use a thread-local BAOS so the object
is not allocated everytime?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
