[GitHub] [kafka] guozhangwang commented on a diff in pull request #12035: KAFKA-13217: Reconsider skipping the LeaveGroup on close() or add an overload that does so
guozhangwang commented on code in PR #12035:
URL: https://github.com/apache/kafka/pull/12035#discussion_r878743871
##
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##
@@ -1498,6 +1516,61 @@ public synchronized boolean close(final Duration
timeout) throws IllegalArgument
return close(timeoutMs);
}
+/**
+ * Shutdown this {@code KafkaStreams} by signaling all the threads to
stop, and then wait up to the timeout for the
+ * threads to join.
+ * @param options contains timeout to specify how long to wait for the
threads to shutdown, and a flag leaveGroup to
+ * trigger consumer leave call
+ * @return {@code true} if all threads were successfully
stopped{@code false} if the timeout was reached
+ * before all threads stopped
+ * Note that this method must not be called in the {@link
StateListener#onChange(KafkaStreams.State, KafkaStreams.State)} callback of
{@link StateListener}.
+ * @throws IllegalArgumentException if {@code timeout} can't be
represented as {@code long milliseconds}
+ */
+public synchronized boolean close(final CloseOptions options) throws
IllegalArgumentException {
+final String msgPrefix =
prepareMillisCheckFailMsgPrefix(options.timeout, "timeout");
+final long timeoutMs = validateMillisecondDuration(options.timeout,
msgPrefix);
+if (timeoutMs < 0) {
+throw new IllegalArgumentException("Timeout can't be negative.");
+}
+
+final long startMs = time.milliseconds();
+
+final boolean closeStatus = close(timeoutMs);
+
+final Optional groupInstanceId = clientSupplier
+
.getConsumer(applicationConfigs.getGlobalConsumerConfigs(clientId))
+.groupMetadata()
+.groupInstanceId();
+
+final long remainingTimeMs = timeoutMs - (time.milliseconds() -
startMs);
Review Comment:
It's rare but possible that the `remainingTimeMs` here could be negative,
let's guard it with `max(0, ...)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12035: KAFKA-13217: Reconsider skipping the LeaveGroup on close() or add an overload that does so
guozhangwang commented on code in PR #12035:
URL: https://github.com/apache/kafka/pull/12035#discussion_r858263213
##
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##
@@ -1498,6 +1516,37 @@ public synchronized boolean close(final Duration
timeout) throws IllegalArgument
return close(timeoutMs);
}
+/**
+ * Shutdown this {@code KafkaStreams} by signaling all the threads to
stop, and then wait up to the timeout for the
+ * threads to join.
+ * @param options contains timeout to specify how long to wait for the
threads to shutdown, and a flag leaveGroup to
+ * trigger consumer leave call
+ * @return {@code true} if all threads were successfully
stopped{@code false} if the timeout was reached
+ * before all threads stopped
+ * Note that this method must not be called in the {@link
StateListener#onChange(KafkaStreams.State, KafkaStreams.State)} callback of
{@link StateListener}.
+ * @throws IllegalArgumentException if {@code timeout} can't be
represented as {@code long milliseconds}
+ */
+public synchronized boolean close(final CloseOptions options) throws
IllegalArgumentException {
+final String msgPrefix =
prepareMillisCheckFailMsgPrefix(options.timeout, "timeout");
+final long timeoutMs = validateMillisecondDuration(options.timeout,
msgPrefix);
+if (timeoutMs < 0) {
+throw new IllegalArgumentException("Timeout can't be negative.");
+}
+
+if (options.leaveGroup) {
+log.debug("Sending leave group trigger to removing instance from
consumer group");
+//removing instance from consumer group
+adminClient.removeMembersFromConsumerGroup(
Review Comment:
Actually I got it wrong at the beginning :P The
`RemoveMembersFromConsumerGroupOptions` only allows use to remove the static
members, via the `groupInstanceId`, which is a static config not a dynamic
value, so we just need to pass in that from the config, and not need to rely on
`KafkaConsumer#groupMetadata`.
Sorry for causing the confusion..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12035: KAFKA-13217: Reconsider skipping the LeaveGroup on close() or add an overload that does so
guozhangwang commented on code in PR #12035:
URL: https://github.com/apache/kafka/pull/12035#discussion_r857910667
##
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##
@@ -1498,6 +1516,37 @@ public synchronized boolean close(final Duration
timeout) throws IllegalArgument
return close(timeoutMs);
}
+/**
+ * Shutdown this {@code KafkaStreams} by signaling all the threads to
stop, and then wait up to the timeout for the
+ * threads to join.
+ * @param options contains timeout to specify how long to wait for the
threads to shutdown, and a flag leaveGroup to
+ * trigger consumer leave call
+ * @return {@code true} if all threads were successfully
stopped{@code false} if the timeout was reached
+ * before all threads stopped
+ * Note that this method must not be called in the {@link
StateListener#onChange(KafkaStreams.State, KafkaStreams.State)} callback of
{@link StateListener}.
+ * @throws IllegalArgumentException if {@code timeout} can't be
represented as {@code long milliseconds}
+ */
+public synchronized boolean close(final CloseOptions options) throws
IllegalArgumentException {
+final String msgPrefix =
prepareMillisCheckFailMsgPrefix(options.timeout, "timeout");
+final long timeoutMs = validateMillisecondDuration(options.timeout,
msgPrefix);
+if (timeoutMs < 0) {
+throw new IllegalArgumentException("Timeout can't be negative.");
+}
+
+if (options.leaveGroup) {
+log.debug("Sending leave group trigger to removing instance from
consumer group");
+//removing instance from consumer group
+adminClient.removeMembersFromConsumerGroup(
Review Comment:
Hi @sayantanu-dey
1) Yeah I agree that if we wait for the future to get resolved before
calling `consumer.close` it may be rejected. Actually, I think we should call
`close` first, then using the admin to call remove-member.
2) What I meant is that we should consider only returning this
`KafkaStreams#close` call after the future from the remove-member has
completed, but also bounded by the timeoutMs. I.e. after the inner `close`
returns we subtract the time taken and then use the remaining to wait for the
future to get resolved. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12035: KAFKA-13217: Reconsider skipping the LeaveGroup on close() or add an overload that does so
guozhangwang commented on code in PR #12035:
URL: https://github.com/apache/kafka/pull/12035#discussion_r857908050
##
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##
@@ -1498,6 +1516,37 @@ public synchronized boolean close(final Duration
timeout) throws IllegalArgument
return close(timeoutMs);
}
+/**
+ * Shutdown this {@code KafkaStreams} by signaling all the threads to
stop, and then wait up to the timeout for the
+ * threads to join.
+ * @param options contains timeout to specify how long to wait for the
threads to shutdown, and a flag leaveGroup to
+ * trigger consumer leave call
+ * @return {@code true} if all threads were successfully
stopped{@code false} if the timeout was reached
+ * before all threads stopped
+ * Note that this method must not be called in the {@link
StateListener#onChange(KafkaStreams.State, KafkaStreams.State)} callback of
{@link StateListener}.
+ * @throws IllegalArgumentException if {@code timeout} can't be
represented as {@code long milliseconds}
+ */
+public synchronized boolean close(final CloseOptions options) throws
IllegalArgumentException {
+final String msgPrefix =
prepareMillisCheckFailMsgPrefix(options.timeout, "timeout");
+final long timeoutMs = validateMillisecondDuration(options.timeout,
msgPrefix);
+if (timeoutMs < 0) {
+throw new IllegalArgumentException("Timeout can't be negative.");
+}
+
+if (options.leaveGroup) {
+log.debug("Sending leave group trigger to removing instance from
consumer group");
+//removing instance from consumer group
+adminClient.removeMembersFromConsumerGroup(
+
applicationConfigs.getString(StreamsConfig.APPLICATION_ID_CONFIG),
+new RemoveMembersFromConsumerGroupOptions()
+);
Review Comment:
Hi @dajac thanks for your comments.
1) I think your call out for not specifying the member id in
`RemoveMembersFromConsumerGroupOptions` is right (and sorry I missed that in my
pass). This can be retrieved from `KafkaConsumer#groupMetadata`.
2) Regarding the "leave-group" behavior, the thing is that with static
member today we would disable sending the leave-group request (this is an
internal config of Consumer, named as `LEAVE_GROUP_ON_CLOSE_CONFIG`, used by
Streams). And since it's a config, we cannot re-enable it on the fly --- and
that's why I once talked to you about having `KafkaConsumer#close` to expose a
similar mechanism to indicate whether or not send leave-group as well --- so
the only way left to enforcing the member to be removed immediately is via the
admin client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12035: KAFKA-13217: Reconsider skipping the LeaveGroup on close() or add an overload that does so
guozhangwang commented on code in PR #12035:
URL: https://github.com/apache/kafka/pull/12035#discussion_r854451233
##
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##
@@ -1498,6 +1516,37 @@ public synchronized boolean close(final Duration
timeout) throws IllegalArgument
return close(timeoutMs);
}
+/**
+ * Shutdown this {@code KafkaStreams} by signaling all the threads to
stop, and then wait up to the timeout for the
+ * threads to join.
+ * @param options contains timeout to specify how long to wait for the
threads to shutdown, and a flag leaveGroup to
+ * trigger consumer leave call
+ * @return {@code true} if all threads were successfully
stopped{@code false} if the timeout was reached
+ * before all threads stopped
+ * Note that this method must not be called in the {@link
StateListener#onChange(KafkaStreams.State, KafkaStreams.State)} callback of
{@link StateListener}.
+ * @throws IllegalArgumentException if {@code timeout} can't be
represented as {@code long milliseconds}
+ */
+public synchronized boolean close(final CloseOptions options) throws
IllegalArgumentException {
+final String msgPrefix =
prepareMillisCheckFailMsgPrefix(options.timeout, "timeout");
+final long timeoutMs = validateMillisecondDuration(options.timeout,
msgPrefix);
+if (timeoutMs < 0) {
+throw new IllegalArgumentException("Timeout can't be negative.");
+}
+
+if (options.leaveGroup) {
+log.debug("Sending leave group trigger to removing instance from
consumer group");
+//removing instance from consumer group
+adminClient.removeMembersFromConsumerGroup(
Review Comment:
Should we also wait on the returned result of remove-member to complete
within the `timeoutMs` as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
