[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408: URL: https://github.com/apache/kafka/pull/14408#discussion_r1331958841 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java: ## @@ -523,9 +526,38 @@ public CompletableFuture> groupsByPartition = new HashMap<>(); +for (String groupId : groupIds) { +final int partition = partitionFor(groupId); +final List groupList = groupsByPartition.getOrDefault(partition, new ArrayList<>()); +groupList.add(groupId); +groupsByPartition.put(partition, groupList); +} + +final List> futures = new ArrayList<>(); +for (Map.Entry> entry : groupsByPartition.entrySet()) { +int partition = entry.getKey(); +List groupList = entry.getValue(); + CompletableFuture future = +runtime.scheduleWriteOperation("delete-group", +new TopicPartition(Topic.GROUP_METADATA_TOPIC_NAME, partition), +coordinator -> coordinator.deleteGroups(context, groupList)); +futures.add(future); +} + +final CompletableFuture allFutures = CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])); Review Comment: Yeah, it makes sense. I'm not sure what to return if there's an exception in the write operation, since we can't set an error code for a `DeletableGroupResultCollection`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408: URL: https://github.com/apache/kafka/pull/14408#discussion_r1331960774 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java: ## @@ -523,9 +526,38 @@ public CompletableFuture> groupsByPartition = new HashMap<>(); +for (String groupId : groupIds) { +final int partition = partitionFor(groupId); +final List groupList = groupsByPartition.getOrDefault(partition, new ArrayList<>()); +groupList.add(groupId); +groupsByPartition.put(partition, groupList); +} + +final List> futures = new ArrayList<>(); +for (Map.Entry> entry : groupsByPartition.entrySet()) { +int partition = entry.getKey(); +List groupList = entry.getValue(); Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408: URL: https://github.com/apache/kafka/pull/14408#discussion_r1331961151 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java: ## @@ -523,9 +526,38 @@ public CompletableFuture> groupsByPartition = new HashMap<>(); +for (String groupId : groupIds) { +final int partition = partitionFor(groupId); +final List groupList = groupsByPartition.getOrDefault(partition, new ArrayList<>()); +groupList.add(groupId); +groupsByPartition.put(partition, groupList); +} + +final List> futures = new ArrayList<>(); +for (Map.Entry> entry : groupsByPartition.entrySet()) { +int partition = entry.getKey(); +List groupList = entry.getValue(); + CompletableFuture future = +runtime.scheduleWriteOperation("delete-group", +new TopicPartition(Topic.GROUP_METADATA_TOPIC_NAME, partition), +coordinator -> coordinator.deleteGroups(context, groupList)); Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408: URL: https://github.com/apache/kafka/pull/14408#discussion_r1331961609 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java: ## @@ -523,9 +526,38 @@ public CompletableFuture> groupsByPartition = new HashMap<>(); +for (String groupId : groupIds) { +final int partition = partitionFor(groupId); +final List groupList = groupsByPartition.getOrDefault(partition, new ArrayList<>()); +groupList.add(groupId); +groupsByPartition.put(partition, groupList); +} + +final List> futures = new ArrayList<>(); +for (Map.Entry> entry : groupsByPartition.entrySet()) { +int partition = entry.getKey(); +List groupList = entry.getValue(); + CompletableFuture future = +runtime.scheduleWriteOperation("delete-group", +new TopicPartition(Topic.GROUP_METADATA_TOPIC_NAME, partition), +coordinator -> coordinator.deleteGroups(context, groupList)); +futures.add(future); +} + +final CompletableFuture allFutures = CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])); +return allFutures.thenApply(v -> { +final DeleteGroupsResponseData.DeletableGroupResultCollection res = new DeleteGroupsResponseData.DeletableGroupResultCollection(); +for (CompletableFuture future : futures) { +try { +DeleteGroupsResponseData.DeletableGroupResultCollection result = future.get(); Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408: URL: https://github.com/apache/kafka/pull/14408#discussion_r1331969154 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorShard.java: ## @@ -262,6 +267,29 @@ public HeartbeatResponseData genericGroupHeartbeat( ); } +public CoordinatorResult deleteGroups( +RequestContext context, +List groupIds +) throws ApiException { + + CoordinatorResult groupDeleteCoordinatorResult = +groupMetadataManager.groupDelete(context, groupIds); + +List validGroupIds = new ArrayList<>(); +for (DeleteGroupsResponseData.DeletableGroupResult result : groupDeleteCoordinatorResult.response()) { +if (result.errorCode() == Errors.NONE.code()) { +validGroupIds.add(result.groupId()); +} +} + +CoordinatorResult deleteOffsetCoordinatorResult = +offsetMetadataManager.deleteAllOffsets(context, validGroupIds); + +final List records = groupDeleteCoordinatorResult.records(); +records.addAll(deleteOffsetCoordinatorResult.records()); +return new CoordinatorResult<>(records, groupDeleteCoordinatorResult.response()); Review Comment: I rearranged this part. Now we loop over the group ids and process them one by one -- validate, populate record list with offset tombstones, add group tombstone, and add response. ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java: ## @@ -3071,6 +3072,41 @@ private void removeCurrentMemberFromGenericGroup( group.remove(member.memberId()); } +public CoordinatorResult groupDelete( +RequestContext context, +List groupIds +) throws ApiException { +final DeleteGroupsResponseData.DeletableGroupResultCollection resultCollection = +new DeleteGroupsResponseData.DeletableGroupResultCollection(); +final List records = new ArrayList<>(); + +groupIds.forEach(groupId -> { +DeleteGroupsResponseData.DeletableGroupResult result = +new DeleteGroupsResponseData.DeletableGroupResult().setGroupId(groupId); +try { +validateGroupDelete(groupId); + records.add(RecordHelpers.newGroupMetadataTombstoneRecord(groupId)); Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408: URL: https://github.com/apache/kafka/pull/14408#discussion_r1332009903 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java: ## @@ -333,6 +348,98 @@ public CoordinatorResult commitOffset( return new CoordinatorResult<>(records, response); } +/** + * Handles an OffsetDelete request. + * + * @param context The request context. + * @param request The OffsetDelete request. + * + * @return A Result containing the OffsetDeleteResponseData response and + * a list of records to update the state machine. + */ +public CoordinatorResult deleteOffsets( +RequestContext context, +OffsetDeleteRequestData request +) throws ApiException { +final List records = new ArrayList<>(); +final OffsetDeleteResponseData.OffsetDeleteResponseTopicCollection responseTopicCollection = +new OffsetDeleteResponseData.OffsetDeleteResponseTopicCollection(); +OffsetDeleteResponseData response = new OffsetDeleteResponseData(); +try { +Group group = validateOffsetDelete(request); + +request.topics().forEach(topic -> { +final OffsetDeleteResponseData.OffsetDeleteResponsePartitionCollection responsePartitionCollection = +new OffsetDeleteResponseData.OffsetDeleteResponsePartitionCollection(); +final boolean subscribedToTopic = group.isSubscribedToTopic(topic.name()); + +topic.partitions().forEach(partition -> { +records.add(RecordHelpers.newOffsetCommitTombstoneRecord( +request.groupId(), +topic.name(), +partition.partitionIndex() +)); Review Comment: Fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] dongnuo123 commented on a diff in pull request #14408: Kafka 14506: Implement DeleteGroups API and OffsetDelete API
dongnuo123 commented on code in PR #14408: URL: https://github.com/apache/kafka/pull/14408#discussion_r1332010169 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java: ## @@ -333,6 +348,98 @@ public CoordinatorResult commitOffset( return new CoordinatorResult<>(records, response); } +/** + * Handles an OffsetDelete request. + * + * @param context The request context. + * @param request The OffsetDelete request. + * + * @return A Result containing the OffsetDeleteResponseData response and + * a list of records to update the state machine. + */ +public CoordinatorResult deleteOffsets( +RequestContext context, +OffsetDeleteRequestData request +) throws ApiException { +final List records = new ArrayList<>(); +final OffsetDeleteResponseData.OffsetDeleteResponseTopicCollection responseTopicCollection = +new OffsetDeleteResponseData.OffsetDeleteResponseTopicCollection(); +OffsetDeleteResponseData response = new OffsetDeleteResponseData(); +try { +Group group = validateOffsetDelete(request); + +request.topics().forEach(topic -> { +final OffsetDeleteResponseData.OffsetDeleteResponsePartitionCollection responsePartitionCollection = +new OffsetDeleteResponseData.OffsetDeleteResponsePartitionCollection(); +final boolean subscribedToTopic = group.isSubscribedToTopic(topic.name()); + +topic.partitions().forEach(partition -> { +records.add(RecordHelpers.newOffsetCommitTombstoneRecord( +request.groupId(), +topic.name(), +partition.partitionIndex() +)); + +OffsetDeleteResponseData.OffsetDeleteResponsePartition responsePartition = +new OffsetDeleteResponseData.OffsetDeleteResponsePartition().setPartitionIndex(partition.partitionIndex()); +if (subscribedToTopic) { +responsePartition = responsePartition.setErrorCode(Errors.GROUP_SUBSCRIBED_TO_TOPIC.code()); +} +responsePartitionCollection.add(responsePartition); +}); + +final OffsetDeleteResponseData.OffsetDeleteResponseTopic responseTopic = +new OffsetDeleteResponseData.OffsetDeleteResponseTopic().setPartitions(responsePartitionCollection); +responseTopicCollection.add(responseTopic); +}); +response = response.setTopics(responseTopicCollection); +} catch (ApiException ex) { +response = response.setErrorCode(Errors.forException(ex).code()); +} +return new CoordinatorResult<>(records, response); +} + +/** + * Deletes all the offsets of the given groups to handle a GroupDelete request. + * Validations are done in groupDelete method in GroupMetadataManager. + * + * @param context The request context. + * @param groupIds The list of group ids of the given groups. + * @return A Result containing the OffsetDeleteResponseData response and + * a list of records to update the state machine. + */ +public CoordinatorResult deleteAllOffsets( Review Comment: Fixed ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorShard.java: ## @@ -262,6 +267,29 @@ public HeartbeatResponseData genericGroupHeartbeat( ); } +public CoordinatorResult deleteGroups( +RequestContext context, +List groupIds Review Comment: Fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org