JiriOndrusek opened a new issue, #9015:
URL: https://github.com/apache/camel-quarkus/issues/9015
Ingesting documents into a vector store for RAG currently means writing the
plumbing by hand: a consumer route, splitting, embedding, and storing, repeated
in every application. This proposes a Camel Quarkus extension that declares
that pipeline instead of coding it.
A pipeline is configured with a directory:
```properties
quarkus.camel.ai.ingest.products.source.directory=/var/data/product-docs
quarkus.camel.ai.ingest.products.embedding-store=products
```
or with any Camel consumer at all, which is the general form — there is no
hand-curated list of source types. The Endpoint DSL is the way to reach one, so
a connector's options are typed and checked by the compiler:
```java
@Ingest("product-docs")
IngestPipeline productDocs() {
return
IngestPipeline.from(Source.endpoint(aws2S3("product-docs").region("eu-west-1"))
.documentId("CamelAwsS3Key"))
.embeddingStore("products");
}
```
What ingestion needs beyond transport is a stable document id, and where
that lives is the consumer's business: `document-id` names the header
(`CamelAwsS3Key`, `CamelKafkaKey`) or gives an expression. Every segment
written carries `camel_quarkus_pipeline` and `camel_quarkus_document_id`, so
retrieval can cite the document an answer came from. Combined with the RAG
augmentor bridge (#9013), an `@RegisterAiService` interface then answers from
the store a pipeline filled, without glue code.
## Scope of this first increment
- file and endpoint sources, the `@Ingest` builder API with a configuration
twin
- build-time validation: unknown component with an add-extension hint,
`@Ingest` method shape, splitter bounds, conflicting source declarations
- integration tests covering a directory, a pushed endpoint, a
builder-declared pipeline, plus S3 (MinIO) and Kafka against containers
## Deliberately out of scope, to follow separately
- the synchronising engine — this increment writes what it is given and
keeps no record, so a document ingested twice leaves two copies and a restart
re-reads the directory. Skipping unchanged documents, replacing edited ones and
removing deleted ones needs the tracker (#9003) and the sync engine on top of it
- document parsing: content is read as text, so PDF and DOCX must be
converted first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
