Hi, multiple answers to this question:
1) it depends of you send messages sync or async to the brokers. Producers do buffer messages in-memory for more efficient writes to the brokers. If messages are successfully sent to the brokers, you can get an acknowledgment back the you can check on the producer (how acks work and what guarantees they provide is configurable). 2) broker also buffer records in-memory before they flush them to disk. It's also configurable how the broker behaves. To guard against data loss, brokers can replicate data to multiple nodes, and you can configure how many replicates must be successfully written before a broker send and ack back to the producer. Thus, even if data is not written to disk yet, it's replicated and won't be lost if a broker fails (after an ack was sent back to the producer). 3) For writes, there is no such thing as commit (by default). Data can be consumed immediately by the consumer. There is some dependency on the replication though and consumer only read data after the messages are successfully replicated. 4) Kafka also supports transactional writes. For those write, the consumer can be configured to only read committe or both committed an and uncommitted data. Hope this helps. -Matthias On 4/4/18 2:41 PM, Victor L wrote: > Can someone explain how message queueing in partition(s) work? > Specifically: > Is there any buffering before message is committed to log (written to disk) ? > Does the term "partition" refer to sequence of messages written to disk, or > are the messages stored in memory queue and written to disk only after that > memory queue is full? If there's no buffering, can message be lost before > it's committed? > Does the message needs to be committed to the log before it can be > consumed? > Thank you >
signature.asc
Description: OpenPGP digital signature
