[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408:
URL: https://github.com/apache/kafka/pull/14408#discussion_r1331958841
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java:
##
@@ -523,9 +526,38 @@ public
CompletableFuture> groupsByPartition = new HashMap<>();
+for (String groupId : groupIds) {
+final int partition = partitionFor(groupId);
+final List groupList =
groupsByPartition.getOrDefault(partition, new ArrayList<>());
+groupList.add(groupId);
+groupsByPartition.put(partition, groupList);
+}
+
+final
List>
futures = new ArrayList<>();
+for (Map.Entry> entry :
groupsByPartition.entrySet()) {
+int partition = entry.getKey();
+List groupList = entry.getValue();
+
CompletableFuture
future =
+runtime.scheduleWriteOperation("delete-group",
+new
TopicPartition(Topic.GROUP_METADATA_TOPIC_NAME, partition),
+coordinator -> coordinator.deleteGroups(context,
groupList));
+futures.add(future);
+}
+
+final CompletableFuture allFutures =
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]));
Review Comment:
Yeah, it makes sense. I'm not sure what to return if there's an exception in
the write operation, since we can't set an error code for a
`DeletableGroupResultCollection`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408:
URL: https://github.com/apache/kafka/pull/14408#discussion_r1331960774
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java:
##
@@ -523,9 +526,38 @@ public
CompletableFuture> groupsByPartition = new HashMap<>();
+for (String groupId : groupIds) {
+final int partition = partitionFor(groupId);
+final List groupList =
groupsByPartition.getOrDefault(partition, new ArrayList<>());
+groupList.add(groupId);
+groupsByPartition.put(partition, groupList);
+}
+
+final
List>
futures = new ArrayList<>();
+for (Map.Entry> entry :
groupsByPartition.entrySet()) {
+int partition = entry.getKey();
+List groupList = entry.getValue();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408:
URL: https://github.com/apache/kafka/pull/14408#discussion_r1331961151
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java:
##
@@ -523,9 +526,38 @@ public
CompletableFuture> groupsByPartition = new HashMap<>();
+for (String groupId : groupIds) {
+final int partition = partitionFor(groupId);
+final List groupList =
groupsByPartition.getOrDefault(partition, new ArrayList<>());
+groupList.add(groupId);
+groupsByPartition.put(partition, groupList);
+}
+
+final
List>
futures = new ArrayList<>();
+for (Map.Entry> entry :
groupsByPartition.entrySet()) {
+int partition = entry.getKey();
+List groupList = entry.getValue();
+
CompletableFuture
future =
+runtime.scheduleWriteOperation("delete-group",
+new
TopicPartition(Topic.GROUP_METADATA_TOPIC_NAME, partition),
+coordinator -> coordinator.deleteGroups(context,
groupList));
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408:
URL: https://github.com/apache/kafka/pull/14408#discussion_r1331961609
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java:
##
@@ -523,9 +526,38 @@ public
CompletableFuture> groupsByPartition = new HashMap<>();
+for (String groupId : groupIds) {
+final int partition = partitionFor(groupId);
+final List groupList =
groupsByPartition.getOrDefault(partition, new ArrayList<>());
+groupList.add(groupId);
+groupsByPartition.put(partition, groupList);
+}
+
+final
List>
futures = new ArrayList<>();
+for (Map.Entry> entry :
groupsByPartition.entrySet()) {
+int partition = entry.getKey();
+List groupList = entry.getValue();
+
CompletableFuture
future =
+runtime.scheduleWriteOperation("delete-group",
+new
TopicPartition(Topic.GROUP_METADATA_TOPIC_NAME, partition),
+coordinator -> coordinator.deleteGroups(context,
groupList));
+futures.add(future);
+}
+
+final CompletableFuture allFutures =
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]));
+return allFutures.thenApply(v -> {
+final DeleteGroupsResponseData.DeletableGroupResultCollection res
= new DeleteGroupsResponseData.DeletableGroupResultCollection();
+for
(CompletableFuture
future : futures) {
+try {
+DeleteGroupsResponseData.DeletableGroupResultCollection
result = future.get();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408:
URL: https://github.com/apache/kafka/pull/14408#discussion_r1331969154
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorShard.java:
##
@@ -262,6 +267,29 @@ public HeartbeatResponseData genericGroupHeartbeat(
);
}
+public
CoordinatorResult deleteGroups(
+RequestContext context,
+List groupIds
+) throws ApiException {
+
+
CoordinatorResult groupDeleteCoordinatorResult =
+groupMetadataManager.groupDelete(context, groupIds);
+
+List validGroupIds = new ArrayList<>();
+for (DeleteGroupsResponseData.DeletableGroupResult result :
groupDeleteCoordinatorResult.response()) {
+if (result.errorCode() == Errors.NONE.code()) {
+validGroupIds.add(result.groupId());
+}
+}
+
+CoordinatorResult
deleteOffsetCoordinatorResult =
+offsetMetadataManager.deleteAllOffsets(context, validGroupIds);
+
+final List records = groupDeleteCoordinatorResult.records();
+records.addAll(deleteOffsetCoordinatorResult.records());
+return new CoordinatorResult<>(records,
groupDeleteCoordinatorResult.response());
Review Comment:
I rearranged this part. Now we loop over the group ids and process them one
by one -- validate, populate record list with offset tombstones, add group
tombstone, and add response.
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##
@@ -3071,6 +3072,41 @@ private void removeCurrentMemberFromGenericGroup(
group.remove(member.memberId());
}
+public
CoordinatorResult groupDelete(
+RequestContext context,
+List groupIds
+) throws ApiException {
+final DeleteGroupsResponseData.DeletableGroupResultCollection
resultCollection =
+new DeleteGroupsResponseData.DeletableGroupResultCollection();
+final List records = new ArrayList<>();
+
+groupIds.forEach(groupId -> {
+DeleteGroupsResponseData.DeletableGroupResult result =
+new
DeleteGroupsResponseData.DeletableGroupResult().setGroupId(groupId);
+try {
+validateGroupDelete(groupId);
+
records.add(RecordHelpers.newGroupMetadataTombstoneRecord(groupId));
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408:
URL: https://github.com/apache/kafka/pull/14408#discussion_r1332009903
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java:
##
@@ -333,6 +348,98 @@ public CoordinatorResult
commitOffset(
return new CoordinatorResult<>(records, response);
}
+/**
+ * Handles an OffsetDelete request.
+ *
+ * @param context The request context.
+ * @param request The OffsetDelete request.
+ *
+ * @return A Result containing the OffsetDeleteResponseData response and
+ * a list of records to update the state machine.
+ */
+public CoordinatorResult deleteOffsets(
+RequestContext context,
+OffsetDeleteRequestData request
+) throws ApiException {
+final List records = new ArrayList<>();
+final OffsetDeleteResponseData.OffsetDeleteResponseTopicCollection
responseTopicCollection =
+new
OffsetDeleteResponseData.OffsetDeleteResponseTopicCollection();
+OffsetDeleteResponseData response = new OffsetDeleteResponseData();
+try {
+Group group = validateOffsetDelete(request);
+
+request.topics().forEach(topic -> {
+final
OffsetDeleteResponseData.OffsetDeleteResponsePartitionCollection
responsePartitionCollection =
+new
OffsetDeleteResponseData.OffsetDeleteResponsePartitionCollection();
+final boolean subscribedToTopic =
group.isSubscribedToTopic(topic.name());
+
+topic.partitions().forEach(partition -> {
+records.add(RecordHelpers.newOffsetCommitTombstoneRecord(
+request.groupId(),
+topic.name(),
+partition.partitionIndex()
+));
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408:
URL: https://github.com/apache/kafka/pull/14408#discussion_r1332010169
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java:
##
@@ -333,6 +348,98 @@ public CoordinatorResult
commitOffset(
return new CoordinatorResult<>(records, response);
}
+/**
+ * Handles an OffsetDelete request.
+ *
+ * @param context The request context.
+ * @param request The OffsetDelete request.
+ *
+ * @return A Result containing the OffsetDeleteResponseData response and
+ * a list of records to update the state machine.
+ */
+public CoordinatorResult deleteOffsets(
+RequestContext context,
+OffsetDeleteRequestData request
+) throws ApiException {
+final List records = new ArrayList<>();
+final OffsetDeleteResponseData.OffsetDeleteResponseTopicCollection
responseTopicCollection =
+new
OffsetDeleteResponseData.OffsetDeleteResponseTopicCollection();
+OffsetDeleteResponseData response = new OffsetDeleteResponseData();
+try {
+Group group = validateOffsetDelete(request);
+
+request.topics().forEach(topic -> {
+final
OffsetDeleteResponseData.OffsetDeleteResponsePartitionCollection
responsePartitionCollection =
+new
OffsetDeleteResponseData.OffsetDeleteResponsePartitionCollection();
+final boolean subscribedToTopic =
group.isSubscribedToTopic(topic.name());
+
+topic.partitions().forEach(partition -> {
+records.add(RecordHelpers.newOffsetCommitTombstoneRecord(
+request.groupId(),
+topic.name(),
+partition.partitionIndex()
+));
+
+OffsetDeleteResponseData.OffsetDeleteResponsePartition
responsePartition =
+new
OffsetDeleteResponseData.OffsetDeleteResponsePartition().setPartitionIndex(partition.partitionIndex());
+if (subscribedToTopic) {
+responsePartition =
responsePartition.setErrorCode(Errors.GROUP_SUBSCRIBED_TO_TOPIC.code());
+}
+responsePartitionCollection.add(responsePartition);
+});
+
+final OffsetDeleteResponseData.OffsetDeleteResponseTopic
responseTopic =
+new
OffsetDeleteResponseData.OffsetDeleteResponseTopic().setPartitions(responsePartitionCollection);
+responseTopicCollection.add(responseTopic);
+});
+response = response.setTopics(responseTopicCollection);
+} catch (ApiException ex) {
+response = response.setErrorCode(Errors.forException(ex).code());
+}
+return new CoordinatorResult<>(records, response);
+}
+
+/**
+ * Deletes all the offsets of the given groups to handle a GroupDelete
request.
+ * Validations are done in groupDelete method in GroupMetadataManager.
+ *
+ * @param context The request context.
+ * @param groupIds The list of group ids of the given groups.
+ * @return A Result containing the OffsetDeleteResponseData response and
+ * a list of records to update the state machine.
+ */
+public CoordinatorResult
deleteAllOffsets(
Review Comment:
Fixed
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorShard.java:
##
@@ -262,6 +267,29 @@ public HeartbeatResponseData genericGroupHeartbeat(
);
}
+public
CoordinatorResult deleteGroups(
+RequestContext context,
+List groupIds
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
