cameronlee314 commented on a change in pull request #947: SAMZA-2120: Enable
custom handling of ConsumerRecords consumed by Kafka
URL: https://github.com/apache/samza/pull/947#discussion_r264370317
##########
File path:
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaConsumerProxy.java
##########
@@ -328,9 +326,21 @@ private void fetchMessages() {
return results;
}
- private int getRecordSize(ConsumerRecord<K, V> r) {
- int keySize = (r.key() == null) ? 0 : r.serializedKeySize();
- return keySize + r.serializedValueSize();
+ /**
+ * Convert a {@link ConsumerRecord} to an {@link IncomingMessageEnvelope}.
+ *
+ * This has a protected visibility so that {@link KafkaConsumerProxy} can be
extended to add special handling logic
+ * for custom Kafka systems.
+ *
+ * @param consumerRecord {@link ConsumerRecord} from Kafka that was consumed
+ * @param systemStreamPartition {@link SystemStreamPartition} corresponding
to the record
+ * @return {@link IncomingMessageEnvelope} corresponding to the {@code
consumerRecord}
+ */
+ protected IncomingMessageEnvelope handleNewRecord(ConsumerRecord<K, V>
consumerRecord,
Review comment:
I originally considered a method name like that, but I felt that this hook
could include some other custom logic as well, such as managing managing
offsets for committing them back to the kafka broker.
We could use `toIncomingMessageEnvelope` for now and then change it later,
but I figured `handleNewRecord` leaves it a bit more open.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
