AshenScribe opened a new pull request, #23402:
URL: https://github.com/apache/kafka/pull/23402
### **PR Description**
#### **Summary**
This PR removes redundant offset commits from `EndToEndLatency.java` to
make the benchmarking tool completely stateless. This ensures that if
the process is abruptly terminated (hard-killed), it does not leave
behind stale checkpoints on the broker that could skew latency metrics
upon restart.
#### **The Problem**
If `EndToEndLatency` is hard-killed, the consumer group might have
uncommitted messages. Upon restart, the consumer resumes from the last
committed checkpoint rather than the tail of the log. Because it
immediately processes the backlog already sitting on the broker, it
erroneously reports near-zero "ghost" latency.
#### **The Solution**
Since the tool is stateless and only measures live end-to-end
performance, we do not need to save the consumer's progress.
* Removed `consumer.commitSync()` from the success path in `execute()`.
* Removed `consumer.commitSync()` from the failure path in
`commitAndThrow()`.
By avoiding offset commits entirely, the consumer group has no recorded
state on the broker. Upon restart, it is forced to obey
`auto.offset.reset = latest` (and the programmatic `seekToEnd` on
startup), jumping directly to the log end to await newly produced
messages.
---
### **How to Test**
1. **Start a local Kafka broker** and create a test topic.
2. **Run the benchmark**, then abruptly terminate it mid-run (`Ctrl +C`):
```
./bin/kafka-run-class.sh org.apache.kafka.tools.EndToEndLatency
--bootstrap-server localhost:9092 --topic test-topic --num-records 50000
--producer-acks all --record-size 100
```
3. **Restart the benchmark immediately.**
* **Verification:** The initial print (at index `0`) will report
realistic startup latency (e.g., `~30-40ms` for metadata and handshake
bootstrap overhead) instead of an immediate near-zero reading. This
proves the consumer is correctly ignoring any backlog and waiting for
new messages.
Conclusion: The latency recorded on the first loop (0 36.269337 ms)
indicates real client connection handshakes, JIT compilation, and
metadata initializatoin overhead.
Reviewers: Chia-Ping Tsai <[email protected]>, Ken Huang
<[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
