Hi there,
I realized that consuming messages is implemented as polling on protocol
level.
Is there a reason why the client has to ask for messages?
Wouldn't it make more sense to push messages to consumers - at least to
high-level ones?
I would think about it like:
Broker receives a new message from a produce.
It could push the message first to all replicas and then (bulked) to all
consumers (at least high-level ones with consumer group). [bulk:
offsetOfFirstMsg, list(msgs)]
Consumers could ack the (last) message (with the new offset) just after
processing it. [newOffset]
After the last message is acked the broker would push the next (bulked)
messages to consume. And so on..
Pros:
This way it seams like the protocol and the high-level consumer would be
simplified.
Clients have a more natural control over the offset and could ack per
message or per bulk as needed or performance allows.
Additionally the stream processing path over Kafka would be faster.
Best Regards
Alexander Binzberger
--
Alexander Binzberger
System Designer - WINGcon AG
Tel. +49 7543 966-119
Sitz der Gesellschaft: Langenargen
Registergericht: ULM, HRB 734260
USt-Id.: DE232931635, WEEE-Id.: DE74015979
Vorstand: thomasThomas Ehrle (Vorsitz), Fritz R. Paul (Stellvertreter), Tobias
Treß
Aufsichtsrat: Jürgen Maucher (Vorsitz), Andreas Paul (Stellvertreter), Martin
Sauter
