[
https://issues.apache.org/jira/browse/HUDI-340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17141627#comment-17141627
]
wangxianghu edited comment on HUDI-340 at 6/22/20, 2:13 AM:
------------------------------------------------------------
[~vinoth] [~Pratyaksh] yes, I agree user sets sourceLimit to a huge value, then
we can assume he is purposely doing that and he has a strong reasoning, and we
can allow that.
Besides, I take the lottery example is trying to say this line is unnecessary,
just as VC marked:
{code:java}
// this looks like a lottery
maxEventsToReadFromKafka = (maxEventsToReadFromKafka == Long.MAX_VALUE ||
maxEventsToReadFromKafka == Integer.MAX_VALUE)
? Config.maxEventsFromKafkaSource : maxEventsToReadFromKafka;
{code}
was (Author: wangxianghu):
[~vinoth] [~Pratyaksh] yes, I agree user sets sourceLimit to a huge value, then
we can assume he is purposely doing that and he has a strong reasoning, and we
can allow that.
Besides, I take the lottery example is trying to say this line is unnecessary,
just as VC marked:
{code:java}
// Same as VC said
maxEventsToReadFromKafka = (maxEventsToReadFromKafka == Long.MAX_VALUE ||
maxEventsToReadFromKafka == Integer.MAX_VALUE)
? Config.maxEventsFromKafkaSource : maxEventsToReadFromKafka;
{code}
> Increase Default max events to read from kafka source
> -----------------------------------------------------
>
> Key: HUDI-340
> URL: https://issues.apache.org/jira/browse/HUDI-340
> Project: Apache Hudi
> Issue Type: Improvement
> Components: DeltaStreamer
> Reporter: Pratyaksh Sharma
> Assignee: Pratyaksh Sharma
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Right now, DEFAULT_MAX_EVENTS_TO_READ is set to 1M in case of kafka source in
> KafkaOffsetGen.java class. DeltaStreamer can handle much more incoming
> records than this.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
