GitHub user mlevkov created a discussion: Proposal: Generic HTTP Sink Connector
Hi team,
We've been running Iggy in production for our ad mediation platform since
January 2026 (migrated from Fluvio) and would like
to contribute a generic HTTP sink connector back to the ecosystem.
The connector framework has 6 sinks today, but no generic HTTP sink for
arbitrary endpoints — webhooks, Lambda functions,
REST APIs, SaaS integrations. Every HTTP-using sink (Quickwit, Elasticsearch,
Iceberg) re-implements its own client logic,
error handling, and retry strategy independently.
We've put together a detailed design document (attached) covering the full
proposal: four batch modes (individual, ndjson,
json_array, raw), configurable metadata envelopes with proper u128/binary
serialization, exponential backoff retry with
transient error classification, and an opt-in health check. The design
follows patterns from the existing sinks — reqwest
from Quickwit, retry logic from PostgreSQL/MongoDB, AtomicU64 counters from
MongoDB. No new workspace dependencies required.
During our analysis of the runtime and existing sinks, we also identified two
issues in the connector runtime that affect
all sinks — details in the attached document under "Runtime Issues Discovered
During Analysis." Happy to file these
separately and contribute fixes if the team agrees they should be addressed.
We'd love your feedback on the design before we start implementation. A few
specific questions:
1. Should we open a tracking issue first, or go directly to a PR?
2. Are there any design preferences we should be aware of (e.g., batch mode
naming, config field conventions)?
3. Any thoughts on the runtime issues — should we file those as separate
issues?
Looking forward to your feedback.
[IDEA-010-iggy-http-sink-github-discussion.md](https://github.com/user-attachments/files/25916346/IDEA-010-iggy-http-sink-github-discussion.md)
GitHub link: https://github.com/apache/iggy/discussions/2919
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
