Nikita-Shupletsov commented on code in PR #23483:
URL: https://github.com/apache/kafka/pull/23483#discussion_r4031220377
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/ChunkedByteBufferOutputStream.java:
##########
@@ -241,11 +262,28 @@ private void releaseUnusedChunks() {
for (ByteBuffer chunk : unused)
pool.deallocate(chunk);
}
+ for (ByteBuffer chunk : unused)
Review Comment:
shouldn't we do this check before we actually call pool.deallocate? this
this order we can deallocate something that doesn't belong to the pool. I think
that's the point of having poolAllocatedChunks? please correct me if I am wrong
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/ChunkedRecordAccumulator.java:
##########
@@ -365,6 +356,23 @@ protected ProducerBatch createProducerBatch(TopicPartition
tp, MemoryRecordsBuil
return new ChunkedProducerBatch(tp, recordsBuilder, nowMs);
}
+ /**
+ * Upper bound on the bytes the batch's first record will write, used both
to pre-size the batch's
+ * chunks and as its write-limit basis. Starts from the uncompressed
record-size upper bound
+ * ({@link AbstractRecords#estimateSizeInBytesUpperBound}, which ignores
compression), then for a
+ * compressed codec inflates by the same {@link
MemoryRecordsBuilder#COMPRESSION_RATE_ESTIMATION_FACTOR}
+ * that {@link MemoryRecordsBuilder#estimatedBytesWrittenAfter} applies.
Without the inflation that
+ * first-record check (in {@link ChunkedProducerBatch#tryAppend}) can
demand ~5% more than was
+ * reserved and throw for large compressed records.
+ */
+ private int initialChunkedBatchSize(byte[] key, byte[] value, Header[]
headers) {
Review Comment:
I assume the logic was taken from
MemoryRecordsBuilder.estimatedBytesWritten. but there we also use
CompressionRatioEstimator.estimation, which tracks previous compression rates
for the topic.
I don't necessarily think we need to do it here, but I am also not sure if
adding these 5% on top will do much for us here
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/ChunkedByteBufferOutputStream.java:
##########
@@ -158,8 +161,25 @@ private void advanceWhileCurrentChunkFull() {
*/
private void advanceToNextChunk() {
if (currentChunkIndex + 1 >= chunks.size()) {
- // TODO: KAFKA-20579. With compression support, grow here instead
of throwing.
- throw new IllegalStateException("write exceeded the stream's
remaining chunk capacity");
+ ByteBuffer next = null;
+ try {
+ List<ByteBuffer> chunk = pool.allocateChunks(chunkSize, 0);
+ next = chunk.get(0);
+ } catch (BufferExhaustedException e) {
+ // pool out of memory — fall through to heap
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ // fall through to heap
Review Comment:
I don't think we fall through here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
