GitHub user sujitn created a discussion: send_messages should return
base_offset and partition_id
Following the Kafka wire protocol transport review, send_messages returning no
offset was identified as a blocker for any Kafka compatibility work. This also
benefits Iggy's own SDK users knowing the offset of what you just wrote is a
fundamental primitive for exactly-once semantics, deduplication, and position
tracking.
**Current behavior**
send_messages returns an empty OK response. The offset is computed inside
append_messages_to_local_partition (messages.rs:332), used to stamp messages
and update partition state, then discarded. ShardResponse::SendMessages is a
unit variant.
**Proposed wire format change**
Before: [status:4 LE (0)][length:4 LE (0)]
After: [status:4 LE (0)][length:4 LE (12)][base_offset:8 LE][partition_id:4
LE]
This is backwards compatible for existing SDK clients. The SDK response path in
tcp_client.rs:245 reads length bytes then returns them. Current send_messages
callers do .await?; Ok(()) — they ignore the body. An old SDK against a new
server will silently discard the 12 extra bytes. A new SDK against an old
server sees length == 0 and can return a default.
**What changes**
Server (internal, no external impact):
- append_messages_to_local_partition → return Result<u64, IggyError> (the
current_offset local variable that already exists)
- ShardResponse::SendMessages → SendMessages { base_offset: u64 }
- send_messages_handler.rs → send_ok_response(&[offset_bytes, partition_bytes])
instead of send_empty_ok_response()
SDK trait (separate release):
- async fn send_messages(...) -> Result<(), IggyError> → Result<AppendInfo,
IggyError>
- AppendInfo { base_offset: u64, partition_id: u32 }
- If response body is empty (old server), return AppendInfo { base_offset:
u64::MAX, partition_id: 0 } or use Option<AppendInfo>
Questions for maintainers
1. Is changing the response body length for command 101 acceptable given it's
technically backwards compatible?
2. Should partition_id be included in the response? It's known client-side for
explicit partitioning but not for balanced/key-based routing.
3. Any concerns about the foreign SDK bindings (Python, Go, Java, Node)?
Existing versions would keep working — they'd just ignore the extra bytes.
GitHub link: https://github.com/apache/iggy/discussions/3044
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
