GitHub user mlevkov added a comment to the discussion: Proposal: Generic HTTP
Sink Connector
Thank you @spetz — that's a great perspective on where this can go.
We designed the HTTP layer with that kind of reuse in mind, even if it's
currently packaged as a single sink plugin. The internals are already separated
into composable pieces:
- **Connection management**: `reqwest::Client` with configurable pool sizing,
TCP keep-alive (30s for cloud LB detection), and pool idle timeout (90s) — all
independent of the Sink trait
- **Retry engine**: Exponential backoff with `Retry-After` header parsing,
transient status classification (429/5xx), and configurable max delay — generic
enough for any HTTP call
- **Batch serialization**: 4 modes (individual, ndjson, json_array, raw)
covering the common patterns for both JSON and binary payloads
- **Metadata envelope**: Optional `{metadata, payload}` wrapping with Iggy
context (stream, topic, offset, timestamps)
Extracting the HTTP client + retry + serialization into a shared
`iggy-http-client` crate that other sinks/sources can depend on feels like a
natural next step. The Elasticsearch and Quickwit sinks already speak HTTP — a
shared component would give them connection pooling, retry, and observability
for free.
Happy to explore that direction once the base sink lands and stabilizes.
GitHub link:
https://github.com/apache/iggy/discussions/2919#discussioncomment-16128176
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
