GitHub user mlevkov added a comment to the discussion: Proposal: Generic HTTP
Sink Connector
## Feature idea: Per-message HTTP header forwarding
While working on the HTTP sink PR (#2925), a question came up about dynamic
per-message headers. Today, all HTTP headers are static (config-derived, same
for every request). But there are real use cases for per-message headers:
**Use cases:**
- `X-Iggy-Offset` / `X-Iggy-Topic` / `X-Iggy-Partition` — routing and dedup at
the HTTP endpoint without body parsing
- Forwarding Iggy user-defined message headers (`ConsumedMessage.headers`) as
HTTP headers — e.g., a producer attaches `X-Correlation-ID` and the sink
forwards it to the downstream endpoint
**Proposed design — additive layering:**
```rust
// Static config headers (pre-built once in open(), cloned per-request)
let mut request = build_request(self.method, client, &self.url)
.headers(self.request_headers.clone())
.header("content-type", content_type);
// Per-message dynamic headers (opt-in via config)
if self.include_metadata_headers {
request = request
.header("x-iggy-offset", offset.to_string())
.header("x-iggy-topic", &topic_metadata.topic)
.header("x-iggy-partition", partition_id.to_string());
}
// Forward Iggy user headers as HTTP headers (opt-in)
if self.forward_iggy_headers {
if let Some(headers) = &message.headers {
for (key, value) in headers {
request = request.header(
format!("x-iggy-{}", key.to_string_value()),
value.to_string_value(),
);
}
}
}
```
**Config surface:**
```toml
[plugin_config]
# Include iggy metadata (offset, topic, partition) as HTTP headers
include_metadata_headers = false
# Forward iggy user-defined message headers as HTTP headers (prefixed with
x-iggy-)
forward_iggy_headers = false
```
This preserves the current pre-built `HeaderMap` optimization for static
headers while layering dynamic per-message headers on top. Both options are
opt-in and disabled by default.
Thoughts? This could be a follow-up PR if there is interest.
GitHub link:
https://github.com/apache/iggy/discussions/2919#discussioncomment-16300152
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
