315157973 commented on a change in pull request #9860:
URL: https://github.com/apache/pulsar/pull/9860#discussion_r591563148
##########
File path:
pulsar-client-tools/src/main/java/org/apache/pulsar/client/cli/CmdConsume.java
##########
@@ -404,4 +427,35 @@ public void close() {
private static final Logger log =
LoggerFactory.getLogger(ConsumerSocket.class);
}
+
+ private static class DataConsistencyChecker {
+ private final ConcurrentHashMap<String, AtomicLong> messagesPerKey =
new ConcurrentHashMap<>();
+ private final boolean failOnDuplicateKey;
+
+ public DataConsistencyChecker(boolean failOnDuplicateKey) {
+ this.failOnDuplicateKey = failOnDuplicateKey;
+ }
+
+ public void messageReceived(String key) throws Exception {
+ AtomicLong count = messagesPerKey.computeIfAbsent(key, k -> new
AtomicLong());
+ long newValue = count.incrementAndGet();
+ if (failOnDuplicateKey && newValue == 2) {
+ String message = "Key '"+key+"' has been received more than
once";
+ log.error(message);
+ throw new Exception(message);
Review comment:
As long as there are duplicate messages, an exception will be thrown,
there is no way to count, there will always be only one result.
Can some unit tests be added?
##########
File path:
pulsar-client-tools/src/main/java/org/apache/pulsar/client/cli/CmdConsume.java
##########
@@ -404,4 +427,35 @@ public void close() {
private static final Logger log =
LoggerFactory.getLogger(ConsumerSocket.class);
}
+
+ private static class DataConsistencyChecker {
+ private final ConcurrentHashMap<String, AtomicLong> messagesPerKey =
new ConcurrentHashMap<>();
Review comment:
There is ConcurrentBitSet in Pulsar
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
