[GitHub] [kafka] feyman2016 commented on a change in pull request #9270: KAFKA-10284: Group membership update due to static member rejoin should be persisted
feyman2016 commented on a change in pull request #9270:
URL: https://github.com/apache/kafka/pull/9270#discussion_r499196092
##
File path: core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala
##
@@ -1040,21 +1040,36 @@ class GroupCoordinator(val brokerId: Int,
group.currentState match {
case Stable =>
-info(s"Static member joins during Stable stage will not trigger
rebalance.")
-group.maybeInvokeJoinCallback(member, JoinGroupResult(
- members = List.empty,
- memberId = newMemberId,
- generationId = group.generationId,
- protocolType = group.protocolType,
- protocolName = group.protocolName,
- // We want to avoid current leader performing trivial assignment
while the group
- // is in stable stage, because the new assignment in leader's next
sync call
- // won't be broadcast by a stable group. This could be guaranteed by
- // always returning the old leader id so that the current leader
won't assume itself
- // as a leader based on the returned message, since the new
member.id won't match
- // returned leader id, therefore no assignment will be performed.
- leaderId = currentLeader,
- error = Errors.NONE))
+// check if group's selectedProtocol of next generation will change,
if not, simply store group to persist the
+// updated static member, if yes, rebalance should be triggered to let
the group's assignment and selectProtocol consistent
+val selectedProtocolOfNextGeneration = group.selectProtocol
+if (group.protocolName.contains(selectedProtocolOfNextGeneration)) {
+ info(s"Static member which joins during Stable stage and doesn't
affect selectProtocol will not trigger rebalance.")
+ val groupAssignment: Map[String, Array[Byte]] =
group.allMemberMetadata.map(member => member.memberId ->
member.assignment).toMap
+ groupManager.storeGroup(group, groupAssignment, error => {
+group.inLock {
+ if (error != Errors.NONE) {
Review comment:
Thanks for the review, @vvcephei . You're right for the current commit,
but as @abbccdda mentioned, the current commit does have issues, I'm revising
it to `reset` the `static member` in the group and return the error if
`storeGroup` encountered any errors. Code is ready, but met some local issues
when writing unittests, will update once it's ready.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KAFKA-10569) Running aggregate queries on KSQL client side is getting to ERROR Shutdown broker because all log dirs in ...
[ https://issues.apache.org/jira/browse/KAFKA-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206860#comment-17206860 ] Matthias J. Sax commented on KAFKA-10569: - Thanks for chiming in [~agavra]! > Running aggregate queries on KSQL client side is getting to ERROR Shutdown > broker because all log dirs in ... > - > > Key: KAFKA-10569 > URL: https://issues.apache.org/jira/browse/KAFKA-10569 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 2.5.0 > Environment: local >Reporter: Petre Gordan >Priority: Critical > Attachments: KSQLDBServerSideErrors.txt, KafkaClusterLogs.txt, > ProductsOrders.txt, ZiikeeperSideLog.txt, kafka-server-start.bat, ksql, > ksql-server-start, ksql-server.properties, schema-registry.properties, > server.properties, zookeeper-server-start.bat, zookeeper.properties > > > Working on Windows 10 and confluent-5.5.0-2.12.zip and kafka_2.12-2.5.0. I'm > running locally: > * in powershell, zookeeper with: *bin\windows\zookeeper-server-start.bat > config\zookeeper.properties* > * in powershell, kafka-server > *bin\windows\kafka-server-start.bat config\server.properties* > * in bash (with ubuntu) ksqldb server > sudo bin/ksql-server-start etc/ksqldb/ksql-server.properties > * in bash (with ubuntu) ksql client > sudo bin/ksql [http://0.0.0.0:8088|http://0.0.0.0:8088/] > After all of these are sorted, than, I'm starting to practice the Kafka. So, > I'm creating tables, streams, making inserts, and all good. I can do small > queries like: > select * from products emit changes;, etc. > All good until at this step. > When, I'm trying to run every type of aggregate query, than is showing the > result after a while, but into the end, after I will press Ctrl+C to > terminate that and to do other query, everything is down. > For example, see the attached .sql script, and after I will run that script > the products table and orders stream are created with success and populated > with success. > After that if I run this query: > select ProductRowKey, count(ProductRowKey) from orders group by ProductRowKey > emit changes; > I can see the results, all good, but into the end if I will press Ctrl + C, > than everything is down. > > Looking into the logs and taking based on the time history the main raised > warning and issues are: > * first is raised this: > Query terminated due to exception:org.eclipse.jetty.io.EofException > (io.confluent.ksql.rest.server.resources.streaming.QueryStreamWriter:95) > * than this: INFO stream-client [_confluent-ksql-default_transient State > transition from RUNNING to PENDING_SHUTDOWN > (org.apache.kafka.streams.KafkaStreams:285) > * than these: > INFO stream-thread [_confluent-ksql-default_transient Informed to shut down > (org.apache.kafka.streams.processor.internals.StreamThread:1116) > State transition from RUNNING to PENDING_SHUTDOWN > (org.apache.kafka.streams.processor.internals.StreamThread:221) > Shutdown complete > (org.apache.kafka.streams.processor.internals.StreamThread:1150) > * than this: > INFO stream-thread [qtp2032891036-47] Deleting obsolete state directory 0_0 > for task 0_0 as 1ms has elapsed > * than this: > WARN Could not clean up the schema registry for query: > _confluent-ksql-default_transient_ > * than this: > WARN [Producer clientId=producer-1] Connection to node 0 > (localhost/127.0.0.1:9092) could not be established. Broker may not be > available. (org.apache.kafka.clients.NetworkClient:763) > all of these from above on the KSQLDB server side logs, > * finally this on the Kafka cluster side: > ERROR Shutdown broker because all log dirs in have failed > (kafka.log.LogManager) > > And after that everything is down. Please see all the attached files to get > all the info. > > Please help me with these. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-10569) Running aggregate queries on KSQL client side is getting to ERROR Shutdown broker because all log dirs in ...
[ https://issues.apache.org/jira/browse/KAFKA-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-10569. - Resolution: Invalid > Running aggregate queries on KSQL client side is getting to ERROR Shutdown > broker because all log dirs in ... > - > > Key: KAFKA-10569 > URL: https://issues.apache.org/jira/browse/KAFKA-10569 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 2.5.0 > Environment: local >Reporter: Petre Gordan >Priority: Critical > Attachments: KSQLDBServerSideErrors.txt, KafkaClusterLogs.txt, > ProductsOrders.txt, ZiikeeperSideLog.txt, kafka-server-start.bat, ksql, > ksql-server-start, ksql-server.properties, schema-registry.properties, > server.properties, zookeeper-server-start.bat, zookeeper.properties > > > Working on Windows 10 and confluent-5.5.0-2.12.zip and kafka_2.12-2.5.0. I'm > running locally: > * in powershell, zookeeper with: *bin\windows\zookeeper-server-start.bat > config\zookeeper.properties* > * in powershell, kafka-server > *bin\windows\kafka-server-start.bat config\server.properties* > * in bash (with ubuntu) ksqldb server > sudo bin/ksql-server-start etc/ksqldb/ksql-server.properties > * in bash (with ubuntu) ksql client > sudo bin/ksql [http://0.0.0.0:8088|http://0.0.0.0:8088/] > After all of these are sorted, than, I'm starting to practice the Kafka. So, > I'm creating tables, streams, making inserts, and all good. I can do small > queries like: > select * from products emit changes;, etc. > All good until at this step. > When, I'm trying to run every type of aggregate query, than is showing the > result after a while, but into the end, after I will press Ctrl+C to > terminate that and to do other query, everything is down. > For example, see the attached .sql script, and after I will run that script > the products table and orders stream are created with success and populated > with success. > After that if I run this query: > select ProductRowKey, count(ProductRowKey) from orders group by ProductRowKey > emit changes; > I can see the results, all good, but into the end if I will press Ctrl + C, > than everything is down. > > Looking into the logs and taking based on the time history the main raised > warning and issues are: > * first is raised this: > Query terminated due to exception:org.eclipse.jetty.io.EofException > (io.confluent.ksql.rest.server.resources.streaming.QueryStreamWriter:95) > * than this: INFO stream-client [_confluent-ksql-default_transient State > transition from RUNNING to PENDING_SHUTDOWN > (org.apache.kafka.streams.KafkaStreams:285) > * than these: > INFO stream-thread [_confluent-ksql-default_transient Informed to shut down > (org.apache.kafka.streams.processor.internals.StreamThread:1116) > State transition from RUNNING to PENDING_SHUTDOWN > (org.apache.kafka.streams.processor.internals.StreamThread:221) > Shutdown complete > (org.apache.kafka.streams.processor.internals.StreamThread:1150) > * than this: > INFO stream-thread [qtp2032891036-47] Deleting obsolete state directory 0_0 > for task 0_0 as 1ms has elapsed > * than this: > WARN Could not clean up the schema registry for query: > _confluent-ksql-default_transient_ > * than this: > WARN [Producer clientId=producer-1] Connection to node 0 > (localhost/127.0.0.1:9092) could not be established. Broker may not be > available. (org.apache.kafka.clients.NetworkClient:763) > all of these from above on the KSQLDB server side logs, > * finally this on the Kafka cluster side: > ERROR Shutdown broker because all log dirs in have failed > (kafka.log.LogManager) > > And after that everything is down. Please see all the attached files to get > all the info. > > Please help me with these. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kafka] Fleshgrinder opened a new pull request #9369: KAFKA-4715: Ignore case of CompressionType and OffsetResetStrategy
Fleshgrinder opened a new pull request #9369: URL: https://github.com/apache/kafka/pull/9369 This allows the configuration values for CompressionType in the ProducerConfig and the OffsetResetStrategy in the ConsumerConfig to be specified regardless of their case. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-9234) Consider using @Nullable and @Nonnull annotations
[
https://issues.apache.org/jira/browse/KAFKA-9234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206735#comment-17206735
]
Richard Fussenegger commented on KAFKA-9234:
There are no standard Java {{@Nullable}}, {{@Nonnull}}, or {{@NotNull}}
annotations. We would need to add a dependency on something (Checker, Lombok,
JetBrains, …).
> Consider using @Nullable and @Nonnull annotations
> -
>
> Key: KAFKA-9234
> URL: https://issues.apache.org/jira/browse/KAFKA-9234
> Project: Kafka
> Issue Type: Improvement
> Components: admin, clients, consumer, KafkaConnect, producer ,
> streams, streams-test-utils
>Reporter: Matthias J. Sax
>Assignee: Manasvi Gupta
>Priority: Minor
> Labels: beginner, newbie
>
> Java7 was dropped some time ago, and we might want to consider usein Java8
> `@Nullable` and `@Nonnull` annotations for all public facing APIs instead of
> documenting it in JavaDocs only.
> This tickets should be broken down in a series of smaller PRs to keep the
> scope of each PR contained, allowing for more effective reviews.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
