> self hosted image and the cloud offering, comprehensive integration tests with the Docker image should suffice.
I am fine with that. On Tue, Jan 16, 2024 at 3:00 PM Anush Shetty <anush.she...@qdrant.com> wrote: > P.S. The updates to Qdrant and the clients are backwards compatible. > Further reducing any maintenance overhead. > > On Tue, 16 Jan, 2024, 7:29 pm Anush Shetty, <anush.she...@qdrant.com> > wrote: > > > We'd gladly add the integration tests along with the mock tests that are > > currently in place and since there's no difference in running a self > hosted > > image and the cloud offering, comprehensive integration tests with the > > Docker image should suffice. > > > > As you said, would appreciate any thoughts from the community. > > > > On Tue, 16 Jan, 2024, 7:11 pm Jarek Potiuk, <ja...@potiuk.com> wrote: > > > >> BTW: Dashboard links here: > >> > >> > https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards > >> > >> On Tue, Jan 16, 2024 at 2:39 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > >> > I'd love to hear from others in the community who already use Qdrant > >> what > >> > they think :) ? > >> > > >> > Few comments to Anush: > >> > > >> > I did a bit of review of the links and did some usual research. > >> > > >> > 1) Re: requirements it does not introduce any big issues. Urllib3 < 2 > >> is a > >> > bit strange (but we are anyhow limited by botocore now, so not a big > >> issue, > >> > I hope it can be removed in the future. > >> > > >> > Requires-Dist: fastembed (==0.1.1) ; (python_version < "3.12") and > >> (extra > >> > == "fastembed") > >> > Requires-Dist: grpcio (>=1.41.0) > >> > Requires-Dist: grpcio-tools (>=1.41.0) > >> > Requires-Dist: httpx[http2] (>=0.14.0) > >> > Requires-Dist: numpy (<1.21) ; python_version < "3.8" > >> > Requires-Dist: numpy (>=1.21) ; python_version >= "3.8" and > >> python_version > >> > < "3.12" > >> > Requires-Dist: numpy (>=1.26) ; python_version >= "3.12" > >> > Requires-Dist: portalocker (>=2.7.0,<3.0.0) > >> > Requires-Dist: pydantic (>=1.10.8) > >> > Requires-Dist: urllib3 (>=1.26.14,<2.0.0) > >> > > >> > 2) Open source version seems to be fully supported and alive. This > >> looks > >> > pretty cool after looking at the information provided. The code is > small > >> > and literally calling the library QdrantClient, so it does not seem > like > >> > something that might require a lot of maintenance, > >> > > >> > My concerns are with testability and future-proof maintenance. This > is a > >> > fast-pacing area. There will be breaking changes. Yes. There are unit > >> > tests and system tests there. But we have no time/possibility to run > our > >> > tests against real quadrant serve and especially against one run in > the > >> > cloud "by hand". > >> > > >> > So, two points: > >> > > >> > 1) Open-source version: Similar to Kafka provider - seems Qdrant has a > >> > nicely dockerized version that can be installed from officially > released > >> > images (https://qdrant.tech/documentation/quick-start/) - seems like > >> > perfect candidate to run integration tests with it on our CI. If that > is > >> > there, this means that we can both - easily make sure it continues to > >> work, > >> > but also - equally easily bump the version of Qudrant when new > >> major/minor > >> > release is out and have our tests run automatically in our CI. And it > >> will > >> > nicely run in Breeze with `breeze --integration qdrant` when someone > >> wants > >> > to run the integration tests locally: See > >> > > >> > https://github.com/apache/airflow/tree/main/tests/integration/providers/apache/kafka > >> > and > >> > > >> > https://github.com/apache/airflow/blob/main/scripts/ci/docker-compose/integration-kafka.yml > >> > - I think that shoudl be condition of approving it > >> > > >> > 2) Cloud version: It would also help if you could (especially if you > >> want > >> > to run the system tests against your cloud) that you get similar > >> dashboards > >> > as we have for Amazon and other LLM providers (maintained by > Astronomer) > >> > which would show the status of system tests you run with main version. > >> > > >> > Are you ok with extending the PR and adding integration tests and > >> > committing to maintaining such a dashboard? > >> > > >> > If there are voices from the community "yeah it's useful" - and the > >> points > >> > 1) and 2) are addressed, I am quite positive about accepting the > >> provider :) > >> > > >> > J > >> > > >> > > >> > > >> > On Tue, Jan 16, 2024 at 1:41 PM Anush Shetty <anush.she...@qdrant.com > > > >> > wrote: > >> > > >> >> Hello, Airflow community, > >> >> > >> >> I am Anush - an Integrations engineer at Qdrant. This discussion > >> proposes > >> >> to include Qdrant as a supported provider for Airflow. > >> >> Following up on > >> https://lists.apache.org/list.html?dev@airflow.apache.org > >> >> . > >> >> > >> >> Qdrant - https://github.com/qdrant/qdrant, is an open-source vector > >> >> search > >> >> engine and database, governed by the Apache-2.0 license, allowing > >> complete > >> >> freedom for commercial usage and redistribution. > >> >> > >> >> Proposed provider PR: https://github.com/apache/airflow/pull/36805 > >> >> > >> >> Qdrant ranks amongst the most performant and most used vector > databases > >> >> available today. > >> >> - https://qdrant.tech/benchmarks/ > >> >> - https://ossinsight.io/collections/vector-search-engine/ > >> >> > >> >> We believe Qdrant would be a valuable addition for Airflow users to > >> have > >> >> as > >> >> an option when building DAGs. > >> >> > >> >> Qdrant can be deployed by users on their own or via Qdrant's cloud > >> >> offering. > >> >> > >> >> The proposed provider supports interfacing with Qdrant instances > >> through > >> >> both REST and GRPC interfaces without any restrictions on the mode of > >> >> deployment used. > >> >> > >> >> As part of our commitment, the Qdrant team is willing to undertake > the > >> >> responsibility of maintaining and updating the provider as per user > >> >> requests or any identified needs. > >> >> > >> >> Anush > >> >> > >> > > >> > > >
