P.S. The updates to Qdrant and the clients are backwards compatible. Further reducing any maintenance overhead.
On Tue, 16 Jan, 2024, 7:29 pm Anush Shetty, <anush.she...@qdrant.com> wrote: > We'd gladly add the integration tests along with the mock tests that are > currently in place and since there's no difference in running a self hosted > image and the cloud offering, comprehensive integration tests with the > Docker image should suffice. > > As you said, would appreciate any thoughts from the community. > > On Tue, 16 Jan, 2024, 7:11 pm Jarek Potiuk, <ja...@potiuk.com> wrote: > >> BTW: Dashboard links here: >> >> https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards >> >> On Tue, Jan 16, 2024 at 2:39 PM Jarek Potiuk <ja...@potiuk.com> wrote: >> >> > I'd love to hear from others in the community who already use Qdrant >> what >> > they think :) ? >> > >> > Few comments to Anush: >> > >> > I did a bit of review of the links and did some usual research. >> > >> > 1) Re: requirements it does not introduce any big issues. Urllib3 < 2 >> is a >> > bit strange (but we are anyhow limited by botocore now, so not a big >> issue, >> > I hope it can be removed in the future. >> > >> > Requires-Dist: fastembed (==0.1.1) ; (python_version < "3.12") and >> (extra >> > == "fastembed") >> > Requires-Dist: grpcio (>=1.41.0) >> > Requires-Dist: grpcio-tools (>=1.41.0) >> > Requires-Dist: httpx[http2] (>=0.14.0) >> > Requires-Dist: numpy (<1.21) ; python_version < "3.8" >> > Requires-Dist: numpy (>=1.21) ; python_version >= "3.8" and >> python_version >> > < "3.12" >> > Requires-Dist: numpy (>=1.26) ; python_version >= "3.12" >> > Requires-Dist: portalocker (>=2.7.0,<3.0.0) >> > Requires-Dist: pydantic (>=1.10.8) >> > Requires-Dist: urllib3 (>=1.26.14,<2.0.0) >> > >> > 2) Open source version seems to be fully supported and alive. This >> looks >> > pretty cool after looking at the information provided. The code is small >> > and literally calling the library QdrantClient, so it does not seem like >> > something that might require a lot of maintenance, >> > >> > My concerns are with testability and future-proof maintenance. This is a >> > fast-pacing area. There will be breaking changes. Yes. There are unit >> > tests and system tests there. But we have no time/possibility to run our >> > tests against real quadrant serve and especially against one run in the >> > cloud "by hand". >> > >> > So, two points: >> > >> > 1) Open-source version: Similar to Kafka provider - seems Qdrant has a >> > nicely dockerized version that can be installed from officially released >> > images (https://qdrant.tech/documentation/quick-start/) - seems like >> > perfect candidate to run integration tests with it on our CI. If that is >> > there, this means that we can both - easily make sure it continues to >> work, >> > but also - equally easily bump the version of Qudrant when new >> major/minor >> > release is out and have our tests run automatically in our CI. And it >> will >> > nicely run in Breeze with `breeze --integration qdrant` when someone >> wants >> > to run the integration tests locally: See >> > >> https://github.com/apache/airflow/tree/main/tests/integration/providers/apache/kafka >> > and >> > >> https://github.com/apache/airflow/blob/main/scripts/ci/docker-compose/integration-kafka.yml >> > - I think that shoudl be condition of approving it >> > >> > 2) Cloud version: It would also help if you could (especially if you >> want >> > to run the system tests against your cloud) that you get similar >> dashboards >> > as we have for Amazon and other LLM providers (maintained by Astronomer) >> > which would show the status of system tests you run with main version. >> > >> > Are you ok with extending the PR and adding integration tests and >> > committing to maintaining such a dashboard? >> > >> > If there are voices from the community "yeah it's useful" - and the >> points >> > 1) and 2) are addressed, I am quite positive about accepting the >> provider :) >> > >> > J >> > >> > >> > >> > On Tue, Jan 16, 2024 at 1:41 PM Anush Shetty <anush.she...@qdrant.com> >> > wrote: >> > >> >> Hello, Airflow community, >> >> >> >> I am Anush - an Integrations engineer at Qdrant. This discussion >> proposes >> >> to include Qdrant as a supported provider for Airflow. >> >> Following up on >> https://lists.apache.org/list.html?dev@airflow.apache.org >> >> . >> >> >> >> Qdrant - https://github.com/qdrant/qdrant, is an open-source vector >> >> search >> >> engine and database, governed by the Apache-2.0 license, allowing >> complete >> >> freedom for commercial usage and redistribution. >> >> >> >> Proposed provider PR: https://github.com/apache/airflow/pull/36805 >> >> >> >> Qdrant ranks amongst the most performant and most used vector databases >> >> available today. >> >> - https://qdrant.tech/benchmarks/ >> >> - https://ossinsight.io/collections/vector-search-engine/ >> >> >> >> We believe Qdrant would be a valuable addition for Airflow users to >> have >> >> as >> >> an option when building DAGs. >> >> >> >> Qdrant can be deployed by users on their own or via Qdrant's cloud >> >> offering. >> >> >> >> The proposed provider supports interfacing with Qdrant instances >> through >> >> both REST and GRPC interfaces without any restrictions on the mode of >> >> deployment used. >> >> >> >> As part of our commitment, the Qdrant team is willing to undertake the >> >> responsibility of maintaining and updating the provider as per user >> >> requests or any identified needs. >> >> >> >> Anush >> >> >> > >> >
