DanielLeens commented on issue #11304: URL: https://github.com/apache/seatunnel/issues/11304#issuecomment-4880124928
This is an interesting direction, but it needs to be evaluated as a larger design topic rather than as a normal connector request. The strongest part of the proposal is the problem framing: SeaTunnel is naturally much stronger at the data-movement and lifecycle side of agent memory than at the agent runtime side. The hard part is scope and abstraction boundaries. Right now the proposal still combines several different problem families: 1. memory hydration / backfill 2. CDC freshness and invalidation 3. deletion / compliance propagation 4. archiving / analytics export 5. backend migration / backup Those are related, but they do not necessarily need one shared abstraction in the first step. Before this can become a strong STIP-style discussion, it would help to narrow the phase-1 target and make these parts explicit: - which concrete memory backend should be the first integration target - whether phase 1 is source -> memory sync, memory -> lakehouse export, or delete propagation, instead of all of them together - what reusable contract would live in SeaTunnel itself versus what stays connector-specific - compatibility impact on existing connector APIs, config options, and engine behavior - dependency and licensing expectations for any memory-backend SDKs A good next step would be to turn this into a smaller MVP proposal, for example: 1. one concrete backend 2. one direction of data flow 3. one small set of operations, such as upsert + delete 4. one validation plan with docs and examples If you can refine the scope that way, this can become a much stronger design discussion for the community. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
