pratyakshsharma commented on a change in pull request #1652:
URL: https://github.com/apache/hudi/pull/1652#discussion_r430560815
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -207,6 +208,11 @@ public KafkaOffsetGen(TypedProperties props) {
maxEventsToReadFromKafka = (maxEventsToReadFromKafka == Long.MAX_VALUE ||
maxEventsToReadFromKafka == Integer.MAX_VALUE)
? Config.maxEventsFromKafkaSource : maxEventsToReadFromKafka;
long numEvents = sourceLimit == Long.MAX_VALUE ? maxEventsToReadFromKafka
: sourceLimit;
+
+ if (numEvents < toOffsets.size()) {
Review comment:
Interesting! I guess both of you have valid points but am somewhat more
inclined towards actually throwing the exception. The reason being throwing
this exception will help users detect misconfigs around sourceLimit, for
example, if someone sets sourceLimit to very small number in production setup
by mistake :).
Also would like to hear from @vinothchandar on this :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
