FrankChen021 commented on code in PR #20127:
URL: https://github.com/apache/druid/pull/20127#discussion_r3881187559
##########
embedded-tests/src/test/java/org/apache/druid/testing/embedded/kinesis/KinesisResource.java:
##########
@@ -178,14 +175,33 @@ public void publishRecordsToTopic(String topic,
List<byte[]> records, Map<String
public void publishRecordsToTopicPartition(String topic, String
partitionKey, List<byte[]> records)
{
- for (byte[] record : records) {
- kinesisClient.putRecord(
- PutRecordRequest.builder()
- .streamName(topic)
- .partitionKey(partitionKey)
- .data(SdkBytes.fromByteArray(record))
- .build()
+ publishRecordsInBatches(topic, records, record -> partitionKey);
+ }
+
+ private void publishRecordsInBatches(
+ String topic,
+ List<byte[]> records,
+ Function<byte[], String> partitionKeyFunction
+ )
+ {
+ for (int start = 0; start < records.size(); start +=
PUT_RECORDS_BATCH_SIZE) {
Review Comment:
I reviewed the shared `KinesisResource` and its four current test call paths
(`StreamIndexTestBase`, `StreamIndexDataFormatsTestBase`,
`StreamIndexFaultToleranceTest`, and `KinesisFaultToleranceTest`). The current
callers publish fixed-size Wikipedia events: at most 1,000 records per call,
split into two 500-record requests; the data-format tests publish 60 records
per call. A 500-record request would need records averaging roughly 20 KiB to
reach the current 10 MiB aggregate limit, while these generated records are
only a few hundred bytes.
Therefore, the current tests cannot hit the aggregate-size cap, and this is
not a regression caused by this PR. We do not need to add size-aware batching
here; that can be handled separately if a future caller sends large records.
Also, the limit in the original finding should be 10 MiB rather than 5 MiB.
Reviewed 5 relevant files.
<!-- mergelens:review -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
