Will-Lo commented on code in PR #3751:
URL: https://github.com/apache/gobblin/pull/3751#discussion_r1306184285
##########
gobblin-modules/gobblin-orc/src/main/java/org/apache/gobblin/writer/GobblinBaseOrcWriter.java:
##########
@@ -183,9 +241,52 @@ public void commit()
throws IOException {
closeInternal();
super.commit();
+ properties.setProp(ORC_WRITER_ESTIMATED_RECORD_SIZE,
String.valueOf(estimatedRecordSize));
+ properties.setProp(ORC_WRITER_ESTIMATED_BYTES_ALLOCATED_CONVERTER_MEMORY,
String.valueOf(this.converterMemoryManager.getConverterBufferTotalSize()));
}
/**
+ * Modifies the size of the writer buffer based on the average size of the
records written so far.
+ * If the new batch size is smaller than size of the current batch, the
current batch is flushed.
Review Comment:
Yeah I used to have it only for smaller but realized it's easier to manage
certain edge cases if it just flushes every time it resizes the batch, will
update the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
