feyman2016 commented on a change in pull request #9270:
URL: https://github.com/apache/kafka/pull/9270#discussion_r494424452
##########
File path: core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala
##########
@@ -1040,21 +1040,36 @@ class GroupCoordinator(val brokerId: Int,
group.currentState match {
case Stable =>
- info(s"Static member joins during Stable stage will not trigger
rebalance.")
- group.maybeInvokeJoinCallback(member, JoinGroupResult(
- members = List.empty,
- memberId = newMemberId,
- generationId = group.generationId,
- protocolType = group.protocolType,
- protocolName = group.protocolName,
- // We want to avoid current leader performing trivial assignment
while the group
- // is in stable stage, because the new assignment in leader's next
sync call
- // won't be broadcast by a stable group. This could be guaranteed by
- // always returning the old leader id so that the current leader
won't assume itself
- // as a leader based on the returned message, since the new
member.id won't match
- // returned leader id, therefore no assignment will be performed.
- leaderId = currentLeader,
- error = Errors.NONE))
+ // check if group's selectedProtocol of next generation will change,
if not, simply store group to persist the
+ // updated static member, if yes, rebalance should be triggered to let
the group's assignment and selectProtocol consistent
+ val selectedProtocolOfNextGeneration = group.selectProtocol
+ if (group.protocolName.contains(selectedProtocolOfNextGeneration)) {
+ info(s"Static member which joins during Stable stage and doesn't
affect selectProtocol will not trigger rebalance.")
+ val groupAssignment: Map[String, Array[Byte]] =
group.allMemberMetadata.map(member => member.memberId ->
member.assignment).toMap
+ groupManager.storeGroup(group, groupAssignment, error => {
+ group.inLock {
+ if (error != Errors.NONE) {
+ warn(s"Failed to persist metadata for group ${group.groupId}:
${error.message}")
+ }
+ }
+ })
+ group.maybeInvokeJoinCallback(member, JoinGroupResult(
+ members = List.empty,
+ memberId = newMemberId,
+ generationId = group.generationId,
+ protocolType = group.protocolType,
+ protocolName = group.protocolName,
+ // We want to avoid current leader performing trivial assignment
while the group
+ // is in stable stage, because the new assignment in leader's next
sync call
+ // won't be broadcast by a stable group. This could be guaranteed
by
+ // always returning the old leader id so that the current leader
won't assume itself
+ // as a leader based on the returned message, since the new
member.id won't match
+ // returned leader id, therefore no assignment will be performed.
+ leaderId = currentLeader,
+ error = Errors.NONE))
+ } else {
+ maybePrepareRebalance(group, s"Group's selectedProtocol will change
because static member ${member.memberId} with instance id $groupInstanceId
joined with change of protocol")
Review comment:
For example, if previously there are one leader + one follower(both
static) in a group, the protocols are: `List(("range", metadata),
("roundrobin", metadata))` for both, and current selected protocol is `range`,
if later the follower join again with protocols: `List(("roundrobin",
metadata))`, and the selectedProtocol should be `roundrobin`, now, the
selectedProtocol and the actual assignment is not consistent, here I let it
rebalance to make sure that the selectedProtocol and actual assignment are
consistent. On the other way around, if we don't do the rebalance, we cannot
successfully persist the group since this line `val metadata =
memberMetadata.metadata(protocol)` in
`kafka.coordinator.group.GroupMetadataManager#groupMetadataValue` will fail.
##########
File path: core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala
##########
@@ -1040,21 +1040,36 @@ class GroupCoordinator(val brokerId: Int,
group.currentState match {
case Stable =>
- info(s"Static member joins during Stable stage will not trigger
rebalance.")
- group.maybeInvokeJoinCallback(member, JoinGroupResult(
- members = List.empty,
- memberId = newMemberId,
- generationId = group.generationId,
- protocolType = group.protocolType,
- protocolName = group.protocolName,
- // We want to avoid current leader performing trivial assignment
while the group
- // is in stable stage, because the new assignment in leader's next
sync call
- // won't be broadcast by a stable group. This could be guaranteed by
- // always returning the old leader id so that the current leader
won't assume itself
- // as a leader based on the returned message, since the new
member.id won't match
- // returned leader id, therefore no assignment will be performed.
- leaderId = currentLeader,
- error = Errors.NONE))
+ // check if group's selectedProtocol of next generation will change,
if not, simply store group to persist the
+ // updated static member, if yes, rebalance should be triggered to let
the group's assignment and selectProtocol consistent
+ val selectedProtocolOfNextGeneration = group.selectProtocol
+ if (group.protocolName.contains(selectedProtocolOfNextGeneration)) {
+ info(s"Static member which joins during Stable stage and doesn't
affect selectProtocol will not trigger rebalance.")
+ val groupAssignment: Map[String, Array[Byte]] =
group.allMemberMetadata.map(member => member.memberId ->
member.assignment).toMap
+ groupManager.storeGroup(group, groupAssignment, error => {
+ group.inLock {
+ if (error != Errors.NONE) {
Review comment:
Indeed, here need to be revised, will update~
##########
File path: core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala
##########
@@ -1040,21 +1040,36 @@ class GroupCoordinator(val brokerId: Int,
group.currentState match {
case Stable =>
- info(s"Static member joins during Stable stage will not trigger
rebalance.")
- group.maybeInvokeJoinCallback(member, JoinGroupResult(
- members = List.empty,
- memberId = newMemberId,
- generationId = group.generationId,
- protocolType = group.protocolType,
- protocolName = group.protocolName,
- // We want to avoid current leader performing trivial assignment
while the group
- // is in stable stage, because the new assignment in leader's next
sync call
- // won't be broadcast by a stable group. This could be guaranteed by
- // always returning the old leader id so that the current leader
won't assume itself
- // as a leader based on the returned message, since the new
member.id won't match
- // returned leader id, therefore no assignment will be performed.
- leaderId = currentLeader,
- error = Errors.NONE))
+ // check if group's selectedProtocol of next generation will change,
if not, simply store group to persist the
+ // updated static member, if yes, rebalance should be triggered to let
the group's assignment and selectProtocol consistent
+ val selectedProtocolOfNextGeneration = group.selectProtocol
+ if (group.protocolName.contains(selectedProtocolOfNextGeneration)) {
+ info(s"Static member which joins during Stable stage and doesn't
affect selectProtocol will not trigger rebalance.")
+ val groupAssignment: Map[String, Array[Byte]] =
group.allMemberMetadata.map(member => member.memberId ->
member.assignment).toMap
+ groupManager.storeGroup(group, groupAssignment, error => {
+ group.inLock {
+ if (error != Errors.NONE) {
+ warn(s"Failed to persist metadata for group ${group.groupId}:
${error.message}")
+ }
+ }
+ })
+ group.maybeInvokeJoinCallback(member, JoinGroupResult(
+ members = List.empty,
+ memberId = newMemberId,
+ generationId = group.generationId,
+ protocolType = group.protocolType,
+ protocolName = group.protocolName,
+ // We want to avoid current leader performing trivial assignment
while the group
+ // is in stable stage, because the new assignment in leader's next
sync call
+ // won't be broadcast by a stable group. This could be guaranteed
by
+ // always returning the old leader id so that the current leader
won't assume itself
+ // as a leader based on the returned message, since the new
member.id won't match
+ // returned leader id, therefore no assignment will be performed.
+ leaderId = currentLeader,
+ error = Errors.NONE))
+ } else {
+ maybePrepareRebalance(group, s"Group's selectedProtocol will change
because static member ${member.memberId} with instance id $groupInstanceId
joined with change of protocol")
Review comment:
For example, if previously there are one leader + one follower(both
static) in a group, the protocols are: `List(("range", metadata),
("roundrobin", metadata))` for both, and current selected protocol is `range`,
if later the follower join again with protocols: `List(("roundrobin",
metadata))`, and the selectedProtocol should be `roundrobin`, now, the
selectedProtocol and the actual assignment is not consistent, here I let it
rebalance to make sure that the selectedProtocol and actual assignment are
consistent. On the other way around, if we don't do the rebalance, we cannot
successfully persist the group since this line `val metadata =
memberMetadata.metadata(protocol)` in
`kafka.coordinator.group.GroupMetadataManager#groupMetadataValue` will fail.
Test
`staticMemberRejoinWithUnknownMemberIdAndChangeOfProtocolWithSelectedProtocolChanged
` is for this case.
##########
File path:
core/src/test/scala/unit/kafka/coordinator/group/GroupCoordinatorTest.scala
##########
@@ -3882,6 +3942,21 @@ class GroupCoordinatorTest {
Await.result(responseFuture, Duration(rebalanceTimeout + 100,
TimeUnit.MILLISECONDS))
}
+ private def staticJoinGroupWithPersistence(groupId: String,
+ memberId: String,
Review comment:
Fixed, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
