[
https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585107#comment-15585107
]
Amit Sela commented on BEAM-744:
--------------------------------
That's Correct, I updated the JIRA, thanks!
> A runner should be able to override KafkaIO max wait properties.
> ----------------------------------------------------------------
>
> Key: BEAM-744
> URL: https://issues.apache.org/jira/browse/BEAM-744
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
> Reporter: Amit Sela
>
> KafkaIO has two "wait" properties:
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}},
> default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}},
> default: 10 msec.
> [~rangadi] mentioned some of these were set to due to limitations of the
> DirectRunner, and I can add that they are now limiting the Spark runner
> (which reads in defined time frames, which may be smaller then the wait time
> and so never actually read).
> This feels like defaults should be set for optimal read from Kafka, while a
> runner may override those if it needs to.
> [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which
> may be passed when creating the reader.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
