homatthew commented on code in PR #3751:
URL: https://github.com/apache/gobblin/pull/3751#discussion_r1306072770
##########
gobblin-modules/gobblin-orc/src/main/java/org/apache/gobblin/writer/GobblinBaseOrcWriter.java:
##########
@@ -183,9 +241,52 @@ public void commit()
throws IOException {
closeInternal();
super.commit();
+ properties.setProp(ORC_WRITER_ESTIMATED_RECORD_SIZE,
String.valueOf(estimatedRecordSize));
+ properties.setProp(ORC_WRITER_ESTIMATED_BYTES_ALLOCATED_CONVERTER_MEMORY,
String.valueOf(this.converterMemoryManager.getConverterBufferTotalSize()));
}
/**
+ * Modifies the size of the writer buffer based on the average size of the
records written so far.
+ * If the new batch size is smaller than size of the current batch, the
current batch is flushed.
Review Comment:
Is this comment correct? The current code seems to:
1. Calculate the new batch size
2. If the new batch size is 20% greater or smaller, then we flush
3. and then ensure the new row batch is of the new batchsize
We seem to flush for larger and smaller
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
