CryoThrust commented on issue #11304: URL: https://github.com/apache/seatunnel/issues/11304#issuecomment-5539538365
Following the narrowed phase-1 proposal, here is a concrete design outline for review before implementation. ## Phase 1: Mem0 sink design ### Goals - Add a connector-local sink for a self-hosted Mem0-compatible REST API. - Support batch hydration and CDC-driven freshness through `upsert` and `delete` operations. - Keep the change entirely within the connector module: no engine, core, or connector SPI changes. - Use the existing HTTP connector base and existing SeaTunnel retry/backpressure behavior where possible. ### Non-goals - No shared `MemorySink` abstraction in core for the first backend. - No conversational ingestion protocol or agent-runtime integration. - No embedding/model dependency in the connector; records arrive after any existing transform/embedding stage. - No bidirectional source or archival sink in phase 1. ### Record and request mapping The sink would accept a normal SeaTunnel row and use explicit options for the Mem0 scope fields: - `operation`: `upsert` or `delete`, with a configured default for batch jobs and an optional row-field override for CDC pipelines. - `user_id`: required scope field, either a literal option or a row path. - `agent_id` and `run_id`: optional literal or row paths. - `memory`: the text/content row path for upsert. - `metadata`: optional JSON object row path; connector-owned metadata must not overwrite user metadata. - `memory_id`: required for delete unless the configured delete mode targets a scope. The connector should reject ambiguous configurations at startup (for example, a delete operation without `memory_id` or an explicit scope-delete mode) rather than producing a partially valid request. ### Delete and idempotency semantics - A row-level delete maps to Mem0's memory-delete endpoint using `memory_id`. - Scope deletion is a separate, explicit mode and must not be inferred from a null memory value. - Retries must reuse the same request identity. The connector should derive an idempotency key from a stable source event identifier when one is available, otherwise from a deterministic hash of operation, scope, memory ID, and content. - The key must be attached through the existing HTTP request-header extension point if Mem0 supports it; otherwise the connector should document the at-least-once behavior instead of claiming exactly-once delivery. ### Failure and retry contract - Retry transient transport failures, HTTP 408/429, and 5xx responses using the existing connector policy. - Do not retry authentication/authorization or invalid-request responses; surface the response status and sanitized body through the connector error path. - For a partially accepted batch, report the rejected records with their operation and scope identifiers. The connector must not acknowledge the whole batch as successful. - Preserve ordering only within the configured request batch where required by the source semantics; cross-batch ordering is not guaranteed in phase 1. ### Testing and acceptance criteria - Unit tests cover option validation, upsert/delete request serialization, scope mapping, idempotency-key generation, status classification, and sanitized error handling. - Connector integration tests use a local Mem0-compatible mock server for deterministic request/response assertions. A full self-hosted Mem0 Testcontainers profile can be added if its image and startup time are acceptable to CI. - Two runnable examples are required: JDBC -> Mem0 batch hydration and MySQL-CDC -> Mem0 upsert/delete. - Documentation must state the delivery guarantee, supported operation modes, required scope fields, and the fact that embeddings are performed upstream. ### Compatibility and rollout The first PR should add only the connector, tests, examples, and documentation. A later second-backend evaluation can determine whether a shared memory contract is justified. If this boundary looks acceptable, I can turn it into a STIP-style document with the exact option names and request examples, then submit the connector as a separate implementation PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
