tombentley commented on code in PR #11781:
URL: https://github.com/apache/kafka/pull/11781#discussion_r896737764
##########
connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java:
##########
@@ -1327,30 +1334,39 @@ public WorkerTask doBuild(Task task,
connectorClientConfigOverridePolicy, kafkaClusterId);
KafkaProducer<byte[], byte[]> producer = new
KafkaProducer<>(producerProps);
- TopicAdmin topicAdmin;
+ // Prepare to create a topic admin if the task requires one, but
do not actually create an instance
+ // until/unless one is needed
+ final AtomicReference<TopicAdmin> topicAdmin = new
AtomicReference<>();
+ final Supplier<TopicAdmin> topicAdminCreator = () ->
topicAdmin.updateAndGet(existingAdmin -> {
+ if (existingAdmin != null) {
+ return existingAdmin;
+ }
+ Map<String, Object> adminOverrides =
adminConfigs(id.connector(), "connector-adminclient-" + id, config,
+ sourceConfig, connectorClass,
connectorClientConfigOverridePolicy, kafkaClusterId, ConnectorType.SOURCE);
+ Admin adminClient = Admin.create(adminOverrides);
+ return new
TopicAdmin(adminOverrides.get(BOOTSTRAP_SERVERS_CONFIG), adminClient);
+ });
+
Map<String, TopicCreationGroup> topicCreationGroups;
if (config.topicCreationEnable() &&
sourceConfig.usesTopicCreation()) {
topicCreationGroups =
TopicCreationGroup.configuredGroups(sourceConfig);
// Create a topic admin that the task can use for topic
creation
- Map<String, Object> adminOverrides =
adminConfigs(id.connector(), "connector-adminclient-" + id, config,
- sourceConfig, connectorClass,
connectorClientConfigOverridePolicy, kafkaClusterId, ConnectorType.SOURCE);
- topicAdmin = new TopicAdmin(adminOverrides);
+ topicAdminCreator.get();
Review Comment:
Looking at the tests it doesn't seem we have any that assert that the
`Admin` actually gets closed. `WorkerSourceTaskTest` asserts this, but for this
code we're reliant that every invocation of `doBuild` always returns a
`WorkerSourceTask` which is always stopped. I can see this is pretty hard to
test though.
##########
connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java:
##########
@@ -1327,30 +1334,39 @@ public WorkerTask doBuild(Task task,
connectorClientConfigOverridePolicy, kafkaClusterId);
KafkaProducer<byte[], byte[]> producer = new
KafkaProducer<>(producerProps);
- TopicAdmin topicAdmin;
+ // Prepare to create a topic admin if the task requires one, but
do not actually create an instance
+ // until/unless one is needed
+ final AtomicReference<TopicAdmin> topicAdmin = new
AtomicReference<>();
+ final Supplier<TopicAdmin> topicAdminCreator = () ->
topicAdmin.updateAndGet(existingAdmin -> {
+ if (existingAdmin != null) {
+ return existingAdmin;
+ }
+ Map<String, Object> adminOverrides =
adminConfigs(id.connector(), "connector-adminclient-" + id, config,
+ sourceConfig, connectorClass,
connectorClientConfigOverridePolicy, kafkaClusterId, ConnectorType.SOURCE);
+ Admin adminClient = Admin.create(adminOverrides);
+ return new
TopicAdmin(adminOverrides.get(BOOTSTRAP_SERVERS_CONFIG), adminClient);
+ });
+
Map<String, TopicCreationGroup> topicCreationGroups;
if (config.topicCreationEnable() &&
sourceConfig.usesTopicCreation()) {
topicCreationGroups =
TopicCreationGroup.configuredGroups(sourceConfig);
// Create a topic admin that the task can use for topic
creation
- Map<String, Object> adminOverrides =
adminConfigs(id.connector(), "connector-adminclient-" + id, config,
- sourceConfig, connectorClass,
connectorClientConfigOverridePolicy, kafkaClusterId, ConnectorType.SOURCE);
- topicAdmin = new TopicAdmin(adminOverrides);
+ topicAdminCreator.get();
Review Comment:
The conditions under which we actually end up calling
`topicAdminCreator.get()` are:
```java
sourceConfig.offsetsTopic() != null &&
config.connectorOffsetsTopicsPermitted() // connector-specific store
|| config.topicCreationEnable() && sourceConfig.usesTopicCreation() // task
needs for topic creation
```
These are the only circumstances in which `topicAdmin.get()` would return
non-null.
So I wonder if we should just put that logic directly in `doBuild()`, and
have `offsetStoreForRegularSourceTask()` accept a nullable `topicAdminSupplier`
(which will be guaranteed non-null in the case where it's actually called).
Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
