[GitHub] [kafka] ableegoldman commented on a change in pull request #8662: HOTFIX: skip listOffsets request for newly created changelog topics
ableegoldman commented on a change in pull request #8662: URL: https://github.com/apache/kafka/pull/8662#discussion_r425314470 ## File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopicManager.java ## @@ -169,6 +173,9 @@ public void makeReady(final Map topics) { log.error(timeoutAndRetryError); throw new StreamsException(timeoutAndRetryError); } +log.debug("Completed validating internal topics and created {}", newlyCreatedTopics); Review comment: I do agree it would be useful though. Feel free to create a ticket :P This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #8662: HOTFIX: skip listOffsets request for newly created changelog topics
ableegoldman commented on a change in pull request #8662: URL: https://github.com/apache/kafka/pull/8662#discussion_r425314263 ## File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopicManager.java ## @@ -169,6 +173,9 @@ public void makeReady(final Map topics) { log.error(timeoutAndRetryError); throw new StreamsException(timeoutAndRetryError); } +log.debug("Completed validating internal topics and created {}", newlyCreatedTopics); Review comment: I think this race condition was particularly severe since we do the listOffsets request pretty much immediately after creating the topics, whereas whatever we're doing with that topic next will not be until the rebalance was completed. AFAIK we've never had any users report subsequent operations failing after the first rebalance due to not-yet-fully-created topics, but it could have just slipped past us This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #8662: HOTFIX: skip listOffsets request for newly created changelog topics
ableegoldman commented on a change in pull request #8662: URL: https://github.com/apache/kafka/pull/8662#discussion_r424751772 ## File path: streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java ## @@ -1862,6 +1866,35 @@ public void shouldThrowIllegalStateExceptionIfAnyTopicsMissingFromChangelogEndOf assertThrows(IllegalStateException.class, () -> partitionAssignor.assign(metadata, new GroupSubscription(subscriptions))); } +@Test +public void shouldSkipListOffsetsRequestForNewlyCreatedChangelogTopics() { +adminClient = EasyMock.createMock(AdminClient.class); +final ListOffsetsResult result = EasyMock.createNiceMock(ListOffsetsResult.class); +final KafkaFutureImpl> allFuture = new KafkaFutureImpl<>(); +allFuture.complete(emptyMap()); + +expect(adminClient.listOffsets(emptyMap())).andStubReturn(result); +expect(result.all()).andReturn(allFuture); + +builder.addSource(null, "source1", null, null, null, "topic1"); +builder.addProcessor("processor1", new MockProcessorSupplier(), "source1"); +builder.addStateStore(new MockKeyValueStoreBuilder("store1", false), "processor1"); + +subscriptions.put("consumer10", + new Subscription( + singletonList("topic1"), + defaultSubscriptionInfo.encode() + )); + +EasyMock.replay(result); Review comment: It gets replayed during configuration (at the end of `configureDefault` below) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org