Yes. The main thing there is that this is a chicken-egg problem. WHEN we release 2.8.0 this provider MUST be already released (airflow will not install otherwise). So we MUST release the common.io provider before Airflow 2.8.0. Actually even long before - when we release alpha or beta, common.io will have to be released already. otherwise it will be impossible to test alpha/beta etc.
We could of course utilise --pre-release flag when we install alphas/betas/rcs of airflow - but there is a problem with that flag that it impacts ALL packages, so for example it could install a new pre-release version of (say SQLALCHEMY). This is a missing feature in `pip` to be able to specify pre-release packages selectively. So there is no other solution currently - we have to release common.io before Airflow 2.8 even if it is not yet usable in any released version of Airflow. We could of course try to coordinate and release it a day, or an hour, or a week before. Or now. Does not matter - but it will make the whole release process, coordination more complex and will put also extra risks and dependencies between the two independent release processes. Releasing it now allows us to test the release process for common.io as well. J. On Sat, Nov 11, 2023 at 3:31 PM Elad Kalif <elad...@apache.org> wrote: > Hi Kaxil > > We release providers when they are ready. This is no different than the > openlienage provider that we needed to release before 2.7 and no one could > use it before 2.7 is out. > From providers release process point of view we are ready with AIP-58 - > thus all left is for core release manager to choose when to cut 2.8 > > On Sat, Nov 11, 2023 at 3:46 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Hmm interesting we already had 1.0 release for that provider so releasing > > 1.0.1 is no different :shrug: > > > > I rescind my -1 (binding), to +0 (binding) --- because I don't see how > that > > provider is going to be used without Airflow 2.8 and the code is still > > getting worked out > > > > On Sat, 11 Nov 2023 at 13:44, Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > > FYI: It contains Airflow 2.8 as requirement which is good but really > this > > > provider right now, doesn't make much sense as the code path can change > > > until Airflow 2.8 is cut > > > > > > On Sat, 11 Nov 2023 at 13:42, Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > > >> -1 (binding) on common.io Provider > > >> https://pypi.org/project/apache-airflow-providers-common-io/1.0.1rc1/ > > >> > > >> It isn't going to work with Airflow 2.7.*, it needs code that is main > > >> (airflow.io.store.*) > > >> > > >> > > >> > > >> On Sat, 11 Nov 2023 at 12:57, Jarek Potiuk <ja...@potiuk.com> wrote: > > >> > > >>> One thing to add to my +1. > > >>> > > >>> I have not mentioned it before, but I promised to the team > contributing > > >>> the new LLM providers to review the double check licences / > > dependencies > > >>> of > > >>> our new LLM providers. > > >>> > > >>> Just to confirm, it all looks good. > > >>> > > >>> We do not have a strict requirement to adhere to a permissive licence > > for > > >>> providers, because they are "optional" components of Airflow but we > are > > >>> better than this: > > >>> > > >>> * All the direct libraries we use belong to "Category A" > > >>> https://www.apache.org/legal/resolved.html#category-a. T > > >>> * There are is one transitive dependency (*tqdm*) that is in > "Category > > B" > > >>> https://www.apache.org/legal/resolved.html#category-b -> this i > > >>> "weak-copyleft" but allowed by the ASF - Mozilla Public licence. It > has > > >>> proper attribution in the "openai" client, so we do not have to > mention > > >>> them (because they do). > > >>> > > >>> Also as a side comment: > > >>> > > >>> Currently our automated licence check only checks if our sources have > > >>> appropriate licensing information. But in the future, when we > complete > > >>> SBOM > > >>> implementation (part of our Security Improvements work) - we will be > > able > > >>> to automate those licence checks, because the tool we are using > > (cdxgen) > > >>> to > > >>> generate SBOM has also all the information about licences used by the > > >>> dependencies of Airflow, so we will be able to verify compliance with > > >>> licences in automated way for any new dependencies we will be using. > > >>> > > >>> This is a nice side effect of the security-related improvements we > work > > >>> on > > >>> :). Something not immediately visible for users and even > contributors, > > >>> but > > >>> it puts very strong foundations to Airflow's stability and > "enterprise" > > >>> readiness - both security processes and licensing compliance being > > >>> expected > > >>> from Airflow by the enterprise users. > > >>> > > >>> J. > > >>> > > >>> On Fri, Nov 10, 2023 at 6:27 PM Aritra Basu < > aritrabasu1...@gmail.com> > > >>> wrote: > > >>> > > >>> > +1 (non binding) > > >>> > Tested out against some test dags > > >>> > > > >>> > -- > > >>> > Regards, > > >>> > Aritra Basu > > >>> > > > >>> > On Fri, 10 Nov 2023, 6:35 pm Pankaj Koti, < > pankaj.k...@astronomer.io > > >>> > .invalid> > > >>> > wrote: > > >>> > > > >>> > > +1 (non-binding) > > >>> > > > > >>> > > Concurring with Rahul Vats on the testing of the mentioned > > providers. > > >>> > > > > >>> > > A big thank you to Elad for skillfully overseeing and > synchronizing > > >>> this > > >>> > > release > > >>> > > with impeccable timing. We (Astronomer) truly value this effort > and > > >>> are > > >>> > > enthusiastic > > >>> > > about the establishment of a placeholder and initial space in > > >>> Airflow > > >>> > for > > >>> > > the new > > >>> > > LLM providers (Cohere, OpenAI, Pinecone, pgvector and Weaviate). > > >>> > > We anticipate that this development will be advantageous for the > > >>> > community > > >>> > > and > > >>> > > eagerly invite contributions from members of the Airflow + LLM > > >>> community > > >>> > to > > >>> > > enhance the robustness and feature completeness of these > providers. > > >>> > > > > >>> > > Best regards, > > >>> > > > > >>> > > *Pankaj Koti* > > >>> > > Senior Software Engineer (Airflow OSS Engineering team) > > >>> > > Location: Pune, Maharashtra, India > > >>> > > Timezone: Indian Standard Time (IST) > > >>> > > Phone: +91 9730079985 > > >>> > > > > >>> > > > > >>> > > On Fri, Nov 10, 2023 at 5:53 PM Hussein Awala <huss...@awala.fr> > > >>> wrote: > > >>> > > > > >>> > > > +1 (binding) I checked the checksums, the signatures, the > > licences > > >>> and > > >>> > > the > > >>> > > > sources, all looks good. > > >>> > > > > > >>> > > > On Wed 8 Nov 2023 at 22:10, Elad Kalif <elad...@apache.org> > > wrote: > > >>> > > > > > >>> > > > > Hey all, > > >>> > > > > > > >>> > > > > I have just cut the new wave Airflow Providers packages. This > > >>> email > > >>> > is > > >>> > > > > calling a vote on the release, > > >>> > > > > which will last for 72 hours - which means that it will end > on > > >>> > November > > >>> > > > 11, > > >>> > > > > 2023 21:10 PM UTC and until 3 binding +1 votes have been > > >>> received. > > >>> > > > > > > >>> > > > > > > >>> > > > > Consider this my (binding) +1. > > >>> > > > > > > >>> > > > > Airflow Providers are available at: > > >>> > > > > https://dist.apache.org/repos/dist/dev/airflow/providers/ > > >>> > > > > > > >>> > > > > *apache-airflow-providers-<PROVIDER>-*.tar.gz* are the binary > > >>> > > > > Python "sdist" release - they are also official "sources" > for > > >>> the > > >>> > > > provider > > >>> > > > > packages. > > >>> > > > > > > >>> > > > > *apache_airflow_providers_<PROVIDER>-*.whl are the binary > > >>> > > > > Python "wheel" release. > > >>> > > > > > > >>> > > > > The test procedure for PMC members is described in > > >>> > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-pmc-members > > >>> > > > > > > >>> > > > > The test procedure for and Contributors who would like to > test > > >>> this > > >>> > RC > > >>> > > is > > >>> > > > > described in: > > >>> > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors > > >>> > > > > > > >>> > > > > > > >>> > > > > Public keys are available at: > > >>> > > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > >>> > > > > > > >>> > > > > Please vote accordingly: > > >>> > > > > > > >>> > > > > [ ] +1 approve > > >>> > > > > [ ] +0 no opinion > > >>> > > > > [ ] -1 disapprove with the reason > > >>> > > > > > > >>> > > > > > > >>> > > > > Only votes from PMC members are binding, but members of the > > >>> community > > >>> > > are > > >>> > > > > encouraged to test the release and vote with "(non-binding)". > > >>> > > > > > > >>> > > > > Please note that the version number excludes the 'rcX' > string. > > >>> > > > > This will allow us to rename the artifact without modifying > > >>> > > > > the artifact checksums when we actually release. > > >>> > > > > > > >>> > > > > The status of testing the providers by the community is kept > > >>> here: > > >>> > > > > https://github.com/apache/airflow/issues/35540 > > >>> > > > > > > >>> > > > > You can find packages as well as detailed changelog following > > the > > >>> > below > > >>> > > > > links: > > >>> > > > > > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-amazon/8.11.0rc1/ > > >>> > > > > > > >>> > > > > >>> > > https://pypi.org/project/apache-airflow-providers-apache-spark/4.4.0rc1/ > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/7.9.0rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-cohere/1.0.0rc1/ > > >>> > > > > > > >>> > > > https://pypi.org/project/apache-airflow-providers-common-io/1.0.1rc1/ > > >>> > > > > > > >>> > > > https://pypi.org/project/apache-airflow-providers-databricks/5.0.0rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-discord/3.4.1rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-docker/3.8.1rc1/ > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > https://pypi.org/project/apache-airflow-providers-elasticsearch/5.1.1rc1/ > > >>> > > > > > > https://pypi.org/project/apache-airflow-providers-ftp/3.6.1rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-google/10.11.1rc1/ > > >>> > > > > > > https://pypi.org/project/apache-airflow-providers-http/4.7.0rc1/ > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > https://pypi.org/project/apache-airflow-providers-microsoft-azure/8.2.0rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-openai/1.0.0rc1/ > > >>> > > > > > > >>> > > > > >>> > > https://pypi.org/project/apache-airflow-providers-openlineage/1.2.1rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-pgvector/1.0.0rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-pinecone/1.0.0rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-postgres/5.8.0rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-qubole/3.4.3rc1/ > > >>> > > > > > > >>> > > > https://pypi.org/project/apache-airflow-providers-salesforce/5.5.1rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-slack/8.4.0rc1/ > > >>> > > > > > > >>> > > > https://pypi.org/project/apache-airflow-providers-snowflake/5.1.1rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-weaviate/1.0.0rc1/ > > >>> > > > > > > >>> https://pypi.org/project/apache-airflow-providers-yandex/3.6.0rc1/ > > >>> > > > > > > >>> > > > > Cheers, > > >>> > > > > Elad Kalif > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >> > > >
