GitHub user mlevkov edited a discussion: Proposal: Generic HTTP Sink Connector
Hi team,
We've been running Iggy in production for our ad mediation platform since
January 2026 (migrated from Fluvio) and would like to contribute a generic HTTP
sink connector back to the ecosystem.
The connector framework has 6 sinks today, but no generic HTTP sink for
arbitrary endpoints — webhooks, Lambda functions, REST APIs, SaaS integrations.
Every HTTP-using sink (Quickwit, Elasticsearch, Iceberg) re-implements its own
client logic, error handling, and retry strategy independently.
We've put together a detailed design document (attached) covering the full
proposal: four batch modes (individual, ndjson, json_array, raw), configurable
metadata envelopes with proper u128/binary serialization, exponential backoff
retry with transient error classification, and an opt-in health check. The
design follows patterns from the existing sinks — reqwest from Quickwit, retry
logic from PostgreSQL/MongoDB, AtomicU64 counters from MongoDB. No new
workspace dependencies required.
During our analysis of the runtime and existing sinks, we also identified two
issues in the connector runtime that affect
all sinks — details in the attached document under "Runtime Issues Discovered
During Analysis." Happy to file these separately and contribute fixes if the
team agrees they should be addressed.
We'd love your feedback on the design before we start implementation. A few
specific questions:
1. Should we open a tracking issue first, or go directly to a PR?
2. Are there any design preferences we should be aware of (e.g., batch mode
naming, config field conventions)?
3. Any thoughts on the runtime issues — should we file those as separate
issues?
Looking forward to your feedback.
[IDEA-010-iggy-http-sink-github-discussion.md](https://github.com/user-attachments/files/25916346/IDEA-010-iggy-http-sink-github-discussion.md)
GitHub link: https://github.com/apache/iggy/discussions/2919
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
