[ https://issues.apache.org/jira/browse/KAFKA-9568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17051503#comment-17051503 ]
Oleg Khoruzhenko commented on KAFKA-9568: ----------------------------------------- We run Kafka Streams application with _Static Membership_ and uses _Interactive Queries_ functionality. Say we have two instances A and B running on HOST_A and HOST_B correspondingly. Then A migrated to another HOST_C. The APPLICATION_SERVER_CONFIG changes to the IP of HOST_C. Issue appears when you need to query remote store on the instance A. In that case {{StreamsMetadata}} doesn't get updated and returns ip address of HOST_A, even though A is already migrated to HOST_C. Let me try to explain the case we are dealing with on example from kafka-streams-example project. # I added static group membership to WordCountInteractiveQueriesExample.java see [https://github.com/o0oxid/kafka-streams-examples/commit/54cb74f40ef3c659588dbedc496bc456123db215] # Started confluent-platform: {noformat} 48a1ede4f588 confluentinc/cp-enterprise-kafka:5.4.0 "/etc/confluent/dock…" 2 hours ago Up 2 hours 0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp kafka-streams-examples_kafka_1 bd6cf95c56b7 confluentinc/cp-zookeeper:5.4.0 "/etc/confluent/dock…" 2 hours ago Up 2 hours 2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:32181->32181/tcp kafka-streams-examples_zookeeper_1 {noformat} # Run first application in static group membership mode - group.instance.id 'client-1' and port '7070' {noformat} > java -cp target/kafka-streams-examples-5.4.0-standalone.jar > io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample > 7070 localhost:9092 client-1 [2020-03-04 10:38:31,071] WARN [main] The configuration 'admin.retries' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2020-03-04 10:38:31,072] WARN [main] The configuration 'admin.retry.backoff.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) Mar 04, 2020 10:38:31 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime WARNING: A provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService will be ignored. [2020-03-04 10:38:49,717] WARN [interactive-queries-example-client-StreamThread-1] [Consumer instanceId=client-1-1, clientId=interactive-queries-example-client-StreamThread-1-consumer, groupId=interactive-queries-example] The following subscribed topics are not assigned to any members: [interactive-queries-example-word-count-repartition] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) {noformat} # Run second application in different terminal - group.instance.id 'client-2' and port '7071' {noformat} > java -cp target/kafka-streams-examples-5.4.0-standalone.jar > io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample > 7071 localhost:9092 client-2 [2020-03-04 10:39:05,188] WARN [main] The configuration 'admin.retries' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2020-03-04 10:39:05,189] WARN [main] The configuration 'admin.retry.backoff.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) Mar 04, 2020 10:39:05 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime WARNING: A provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService will be ignored. {noformat} Querying StreamsMetadata returns two instances 'localhost:7070' and 'localhost:7071': {noformat} curl -s localhost:7070/state/instances | jq [ { "host": "localhost", "port": 7070, "storeNames": [ "windowed-word-count" ] }, { "host": "localhost", "port": 7071, "storeNames": [ "word-count" ] } ] {noformat} Now let's re-run the second application with same group.instance.id 'client-2', but different port '7072' {noformat} >java -cp target/kafka-streams-examples-5.4.0-standalone.jar >io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample > 7072 localhost:9092 client-2 [2020-03-04 10:39:05,188] WARN [main] The configuration 'admin.retries' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2020-03-04 10:39:05,189] WARN [main] The configuration 'admin.retry.backoff.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) Mar 04, 2020 10:39:05 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime WARNING: A provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService will be ignored. {noformat} Querying instance metadata returns stale data. Note the port is the same '7071', while application is running on '7072': {noformat} > curl -s localhost:7070/state/instances | jq [ { "host": "localhost", "port": 7070, "storeNames": [ "windowed-word-count" ] }, { "host": "localhost", "port": 7071, "storeNames": [ "word-count" ] } ] {noformat} So now it's impossible to perform interactive queries against remote machine because metadata points to the old port '7071'. Issue is not reproducible with dynamic membership. > Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership > ------------------------------------------------------------------------ > > Key: KAFKA-9568 > URL: https://issues.apache.org/jira/browse/KAFKA-9568 > Project: Kafka > Issue Type: Bug > Components: streams > Affects Versions: 2.4.0 > Reporter: David J. Garcia > Priority: Major > > A kstreams application with static membership, and > StreamsConfg.APPLICATION_SERVER_CONFIG set, will NOT update old server config > upon restart of application on new host. > Steps to reproduce: > > # start two kstreams applications (with same consumer group) and enable > static membership (and set application server config to <ip-addr>:<port>) > # kill one of the applications and restart it on a new host(with new ip) > before timeout ends (so that rebalancing doesn't occur). > # the other kstreams application will now have an invalid > application_server_config > Possible fix: > If an application restarts with a new host/identity..etc, it could trigger a > "light-rebalance" where the other applications in the consumer group don't > change partition assignments ,but instead just get their configuration > updated. -- This message was sent by Atlassian Jira (v8.3.4#803005)