[
https://issues.apache.org/jira/browse/BEAM-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726486#comment-16726486
]
Raghu Angadi commented on BEAM-6285:
------------------------------------
Could you expand a bit more on the warning? Is it because of randomized
group_id? May be simpler solution is to let user provide a group id for 'offset
consumer'?
In addition to Backlog, watermarks are also dependent on offset consumer (e.g.
logAppendTimestampPolicy advances timestamp if the reader is caught up).
> option to disable offsetConsumer in KafkaIO
> -------------------------------------------
>
> Key: BEAM-6285
> URL: https://issues.apache.org/jira/browse/BEAM-6285
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
> Reporter: Xu Mingmin
> Assignee: Xu Mingmin
> Priority: Major
>
> Add an option in KafkaIO.read() to avoid using `offsetConsumer`.
> *What's the problem:*
> When security is enabled, function
> `KafkaUnboundedReader.updateLatestOffsets()` always fails with lots of WARN
> log.
> *What's the cause:*
> `offsetConsumer` uses a mocked `ConsumerConfig.GROUP_ID_CONFIG`, and it
> doesn't work as expected when security is enabled. In our case,
> `ConsumerConfig.GROUP_ID_CONFIG` is the key to validate whether this consumer
> is approved.
> *What's the solution:*
> Add an option to disable `offsetConsumer` by developers;
> *What's the side effect:*
> With `offsetConsumer` disabled, backlog is unknown.
>
> [~rangadi] any comments?
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
