GitHub user mlevkov edited a comment on the discussion: Proposal: Generic HTTP Sink Connector
## PR and Issues Filed The HTTP sink connector implementation is ready for review: - **PR**: #2925 — `feat(connectors): add generic HTTP sink connector` - 4 batch modes (`individual`, `ndjson`, `json_array`, `raw`), retry with exponential backoff, metadata envelope - 60 unit tests + 7 integration tests (WireMock in Docker) - Connection pooling with TCP keep-alive (30s) and pool idle timeout (90s) - Comprehensive README (810 lines): config reference, runtime model, deployment patterns, message flow - 4 rounds of automated code review (42 findings addressed) - Follows all patterns from existing sinks (MongoDB, PostgreSQL, Elasticsearch) - No new dependencies — all workspace deps While analyzing the runtime to inform the design, we found two issues affecting **all existing sinks**. Filed separately: - **Issue #2927** — `consume()` return value discarded in sink runtime (`process_messages()` in `runtime/src/sink.rs`) - **Issue #2928** — `PollingMessages` auto-commit commits offsets before sink processing ### Integration tests The PR includes 7 integration tests in `core/integration/tests/connectors/http/` using WireMock to receive and verify HTTP deliveries. Tests cover: - Single message delivery with metadata envelope verification - Metadata field toggles (checksum, origin timestamp) - All 3 batch modes (ndjson, json_array, individual) - Raw byte passthrough - Multi-topic delivery with per-topic metadata accuracy ### Test counts | Category | Count | |----------|-------| | Unit tests | 60 | | Integration tests | 7 | | Code review rounds | 4 (double-review protocol) | | Review findings fixed | 40 | | Review findings deferred (with issues) | 2 | GitHub link: https://github.com/apache/iggy/discussions/2919#discussioncomment-16104535 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
