[
https://issues.apache.org/jira/browse/BEAM-6285?focusedWorklogId=190135&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-190135
]
ASF GitHub Bot logged work on BEAM-6285:
----------------------------------------
Author: ASF GitHub Bot
Created on: 25/Jan/19 18:33
Start Date: 25/Jan/19 18:33
Worklog Time Spent: 10m
Work Description: rangadi commented on pull request #7598: [BEAM-6285]
add parameters for offsetConsumer in KafkaIO.read()
URL: https://github.com/apache/beam/pull/7598#discussion_r251088725
##########
File path:
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
##########
@@ -164,6 +156,18 @@ public boolean start() throws IOException {
// since CPU consumed on the workers would be low and will likely avoid
unnecessary upscale.
offsetConsumerConfig.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG,
"read_uncommitted");
+ Object groupId =
spec.getConsumerConfig().get(ConsumerConfig.GROUP_ID_CONFIG);
+ // override group_id and disable auto_commit so that it does not interfere
with main consumer
+ String offsetGroupId =
+ String.format(
+ "%s_offset_consumer_%d_%s",
+ name, (new Random()).nextInt(Integer.MAX_VALUE), (groupId == null
? "none" : groupId));
+ offsetConsumerConfig.put(ConsumerConfig.GROUP_ID_CONFIG, offsetGroupId);
+
+ if (spec.getOffsetConsumerConfig() != null) {
Review comment:
Another side effect is that this could overwrite 'ISOLATION_LEVEL_CONFIG'
too, if the user's config has that (intentionally or not). May be we don't want
that.
I think it will be simpler to move this if block to after line 152 (line
number on master).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 190135)
Time Spent: 2h (was: 1h 50m)
> add parameters for offsetConsumer in KafkaIO.read()
> ---------------------------------------------------
>
> Key: BEAM-6285
> URL: https://issues.apache.org/jira/browse/BEAM-6285
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
> Reporter: Xu Mingmin
> Assignee: Xu Mingmin
> Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Add an option in KafkaIO.read() to avoid using `offsetConsumer`.
> *What's the problem:*
> When security is enabled, function
> `KafkaUnboundedReader.updateLatestOffsets()` always fails with lots of WARN
> log.
> *What's the cause:*
> `offsetConsumer` uses a mocked `ConsumerConfig.GROUP_ID_CONFIG`, and it
> doesn't work as expected when security is enabled. In our case,
> `ConsumerConfig.GROUP_ID_CONFIG` is the key to validate whether this consumer
> is approved.
> *What's the solution:*
> Add an option to disable `offsetConsumer` by developers;
> *What's the side effect:*
> With `offsetConsumer` disabled, backlog is unknown.
>
> [~rangadi] any comments?
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
