FrankChen021 opened a new pull request, #19817:
URL: https://github.com/apache/druid/pull/19817
### Description
This fixes a race in the embedded Kafka test resource that was exposed by
the failed `KafkaIndexFaultToleranceTest` run in #19754. The failure is
unrelated to that PR's dependency changes.
After increasing a topic's partition count, `createPartitions()` can
complete before the new partition leaders are ready to serve requests. The test
helper immediately created an idempotent producer and published records, which
could receive `NOT_LEADER_OR_FOLLOWER` followed by repeated
`OUT_OF_ORDER_SEQUENCE_NUMBER` responses. Because the helper ignored the
futures returned by `KafkaProducer.send()`, it could return after silently
dropping records; the ingestion test then timed out waiting for rows that never
reached Kafka.
This PR:
- waits for a successful latest-offset lookup through every partition leader
before returning from `increasePartitionsInTopic`;
- waits for every non-transactional producer send result, so delivery
failures are surfaced;
- adds a regression test that expands a topic from two to four partitions,
immediately publishes 1,000 records, and verifies all records reached all four
partitions.
There is no end-user behavior change.
##### Key changed classes
- `KafkaResource`
- `KafkaResourceTest`
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code
wherever it would not be obvious.
- [x] added or updated tests to cover the changed behavior.
- [x] run the exact embedded test that originally failed.
### Tests
```text
mvn -ntp test -pl extensions-core/kafka-indexing-service -am \
-Dtest="org.apache.druid.indexing.kafka.simulate.KafkaResourceTest" \
-Dsurefire.failIfNoSpecifiedTests=false -Pskip-static-checks \
-Dweb.console.skip=true -T1C
mvn -ntp test -pl embedded-tests -am \
-Dtest="org.apache.druid.testing.embedded.indexing.KafkaIndexFaultToleranceTest#test_supervisorRecovers_afterChangeInTopicPartitions"
\
-Dsurefire.failIfNoSpecifiedTests=false -Pskip-static-checks \
-Dweb.console.skip=true -T1C
```
Both pass locally. The embedded test runs both transactional and
non-transactional parameterizations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
