JiriOndrusek opened a new issue, #9003: URL: https://github.com/apache/camel-quarkus/issues/9003
Vector stores cannot be enumerated for "what did I already ingest", and LangChain4j has no equivalent of LangChain-Python's indexing API (`RecordManager`/`cleanup`), so today any ingestion pipeline is either append-only (restarts re-embed the whole corpus, edits join their previous vectors instead of replacing them) or wipe-and-reload (loses everything between passes). Upstream context showing this gap is real and repeatedly asked for, never fully closed: - langchain4j/langchain4j#1085 — "Does langchain4j have indexing API?" (closed after only the removal-support half shipped; the record-manager/indexing API itself never followed) - langchain4j/langchain4j#2931 — "Incremental update of the knowledge base" (open; maintainer confirms "there is currently no such functionality in LangChain4j, you need to track that yourself") - quarkiverse/quarkus-langchain4j#473 — "Easy RAG: strategies to control whether data should be ingested or not" (closed; a checksum/diff re-ingestion idea was floated and called "awesome if we could do" but never built) A `SyncLedger` SPI + `JdbcSyncLedger` JDBC implementation now exist on the `feature/sql-ladger` branch (`extensions-support/langchain4j`), with a behavioural contract test (`SyncLedgerContract`) verified against both H2 and real PostgreSQL (`integration-tests/langchain4j-sync-ledger`, JVM + native). It's currently experimental and internal (not yet wired into a user-facing extension) — this issue tracks turning it into a real PR and, longer term, wiring it into a `langchain4j-ingest` extension. Note: this is deliberately a stopgap. If langchain4j ever ships its own record-manager/indexing API upstream (per langchain4j/langchain4j#1085 / langchain4j/langchain4j#2931 above), our `SyncLedger`/`JdbcSyncLedger` are candidates for removal in favor of an adapter over the upstream mechanism — `SyncLedger` is designed as a swappable SPI for exactly this reason, so this feature carries no long-term compatibility promise. Co-Authored-By: Claude Sonnet 5 <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
