nitinware opened a new issue, #40047:
URL: https://github.com/apache/beam/issues/40047
### What would you like to happen?
## Motivation
Beam's Python ml/rag ingestion module currently supports Milvus and Qdrant
(`sdks/python/apache_beam/ml/rag/ingestion/`). Chroma is one of the most
widely-adopted open-source vector databases, especially for local and
embedded RAG workloads, and is the missing piece in Beam's OSS vector-DB
coverage.
## Proposal
Add `sdks/python/apache_beam/ml/rag/ingestion/chroma.py` implementing a
`WriteToChroma` PTransform matching the shape of the existing Milvus and
Qdrant connectors:
- Accepts `Chunk` records from the existing RAG chunking transforms
- Supports both `chromadb.HttpClient` (remote) and
`chromadb.PersistentClient`
(embedded / local) client modes
- Configurable collection name, embedding + metadata mapping, batch size
- Standard retry/backoff
## Benefits
- Completes Beam's coverage of the OSS local-first vector-DB tier
- Chroma is Apache-2.0 licensed — no dependency-license concerns
- **`PersistentClient` runs in-process**, so integration tests do not need
Docker or a remote service — a meaningful advantage over Milvus/Qdrant
which require running servers in CI
## Testing plan
- Unit tests mirroring `qdrant_test.py`
- Integration test using `chromadb.PersistentClient` against a tmpdir
(no external service required)
- Docs + example consistent with existing Milvus / Qdrant docs
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
